Shazam for software libraries

Nowadays its’ common to use package managers to describe, fetch and install 3rd party open source dependencies. But that was not always the case and there are many legacy projects out there in the Enterprises of this world there 3rd party libraries are stored in a “lib” directory without any documentation. It’s not a rare case that there are dependencies like a “beanutils.jar”. But which version is it? Which license does it have? Are there any known security vulnerabilities? Nobody knows!

The VersionEye API can identify such components by their SHA values. If you generate the SHA value for your JAR file and send it to the VersionEye API then you will get back the coordinates of that JAR file in Maven. That way you will know the exact GroupId, ArtifactID and Version of the component and based on that you can find out the license and known security vulnerabilities!

Of course it’s a lot of work to do that by hand, especially if you have a couple hundred files in your lib directory. That’s why we developed a command line tool to automate that. The veye_checker is implemented in Rust and we have binaries for Windows, Linux and Mac OS X which run out of the box! Simply download the binary from here, make the binary executable and run this command:

./veye_checker resolve ~/lib -a "YOUR_API_KEY" -o inventory.csv

With “-a” you specify your API key for the VersionEye API. With “-o” you specify the output file. The default the veye_checker outputs everything as CSV to the console. The results is a CSV which shows you exactly the files, their coordinates in the corresponding package manager, their license, the number of security vulnerabilities and a link to their VersionEye page. Here a screenshot:

os_inventory

Currently this works for Maven (Java), Nuget (C#), NPM (Node.JS) and PyPi (Python). Right now we have almost 10 Million SHA values in our database.

This is still a very early version of the “veye_checker”, but please try it out and give us feedback. We would love to hear from you!

Identifying components by SHA values

The public VersionEye API is exposing a lot of information about open source projects. If you know the coordinates of a component (software library) in a certain package manager, you can use the VersionEye API fetch all kind of meta information about the component. Meta information like available versions, licenses, security vulnerabilities and many more.

But what if you don’t know the coordinates of a component? What if you have somewhere on your disk a “beanutils.jar” and you don’t know which version it is or to which Maven GroupId it belongs to? You don’t know which license it has and you don’t know if it is vulnerable or not! For this problem the VersionEye API has a solution now. Simply create the SHA1 value for the “beanutils.jar” file. In Ruby it’s a one liner:

sha1 = Digest::SHA1.file "beanutils.jar"

Now take the SHA1 value and use the new SHA API Endpoint at VersionEye.

screen-shot-2017-02-08-at-08-54-36

With Curl it would be this command:

curl -X GET --header 'Accept: application/json' 'https://www.versioneye.com/api/v2/products/sha/427662b038bd8f52097f783f6ea163e45851b2a1?api_key=YOUR_API_KEY'

As response you get back the coordinates of the component. Now you know the Maven GroupId & ArtifactId and the version of your JAR file. That information you can use to gather even more information about the component from the VersionEye API. Simply use the products API Endpoint:

Screen Shot 2017-02-08 at 08.55.35.png

With Curl it would be this command:

curl -X GET --header 'Accept: application/json' 'https://www.versioneye.com/api/v2/products/Java/commons-beanutils%3Acommons-beanutils?prod_version=1.9.0&api_key=YOUR_API_KEY'

With that request you get back the dependencies, licenses and security vulnerabilities of the component. Now you know that your “beanutils.jar” is licensed under the Apache-2.0 license and it has at least 1 security vulnerability.

As you know the Maven coordinates of your “beanutils.jar” you could also simply visit the corresponding VersionEye page in the browser.

Screen Shot 2017-02-08 at 08.56.58.png

Here you see as well the license and the security vulnerability. And in the right upper corner you can see that there is a newer version of the component available, version 1.9.3. By clicking on that number you will see that the newest version of the component is not vulnerable.

The VersionEye database has SHA values for more than 90K .NET projects, more than 170K Java projects, almost 400K Node.JS projects and more than 88K Python projects. Altogether we have more than 9 Million SHA values.

screen-shot-2017-02-22-at-08-48-40

Every version of a component has at least 1 SHA value. In Maven a version of a component contains multiple files, like for example “*.jar”, “-javadoc.jar”, “-sources.jar” and “.pom”. In this case every file has his own SHA value, that means to 1 artefact we have multiple SHA values in our database.

The SHA method for Maven components and Node.JS modules is always “SHA1”. For Python projects we store the MD5 hash value. For .NET components we either have a base64 encoded SHA256 or SHA512 value. If you have a .NET Nuget package simply create a base64 encoded SHA256 value for it and fire it against our API. If the result is empty repeat the process with a SHA512 value. Here an example how it would look like in Ruby:

Base64.encode64 Digest::SHA512.digest File.read('Newtonsoft.Json.9.0.1.nupkg')

For more than 99% of all Nuget packages we have a SHA value.

Currently we have only SHA values for Java components from Maven central. That’s the official central repository for Java components. VersionEye is crawling a couple other Maven repositories as well, but for those we don’t have SHA values right now. But we are working on it.

Right now we do NOT have SHA values for Ruby but we are working on it!

Try out the new API and let us know how you like it.

5 Years of VersionEye

In February 2012 I founded the VersionEye GmbH in Mannheim / Germany. Unbelievable, but that’s already 5 years ago. Back then in 2012 VersionEye was only tracking versions for Java, Ruby and Python libraries, all together less than 150K components. There was no GitHub integration, no Bitbucket & Stash integration, no API and almost no traffic on the VersionEye.com page.

In the last 5 years the product evolved a lot. Now VersionEye is tracking licenses & security vulnerabilities as well besides versions for more than 1.2 Million components. There is a very good integration for GitHub, Bitbucket and Atlassian Stash. There is a public REST API which is heavily used by many native plugins and AddOns. The product became open source last year and is used as on-premise installations at big enterprises. The usage of VersionEye is reflected by the growing traffic on VersionEye.com. Here are 2 screenshots from the VersionEye Google Analytics account:

screen-shot-2017-02-06-at-10-49-38

screen-shot-2017-02-06-at-10-50-00

By the way: Since January 2014 we spent exactly 0 EUR for Ads and Marketing. All the traffic we are getting is organic.

Many thanks to all the people who are using the product, giving feedback and helping improving it. There are a couple interesting new features and integrations planed for 2017. Stay tuned and follow VersionEye on Twitter 😉

New API Endpoints

The VersionEye API is heavily used! In the mean white the VersionEye API is serving more HTTP requests then the web application. That’s why made it even more awesome 🙂

New SwaggerUI

For almost 2 years we didn’t update the swaggerUI, shame on us! The old swaggerUI was a bit buggy. Many times it did not submit input fields to the API, that was specially the case if there was another input field with the same name, higher up on the page. That and many other small issues are solved now with the newest swaggerUI. Beside that the new swaggerUI shows for every request the corresponding CURL command, which is super awesome because that way you can run the commands directly in the command line.

screen-shot-2017-02-03-at-10-39-19

/sessions/login

We added a new API Endpoint for sessions/login. Now you can send your user/email and password via HTTP POST to this Endpoint and you will receive back your user object with your user API key. With the user API key you can then use other Endpoints of the API.

screen-shot-2017-02-03-at-10-42-24

That is especially useful if you want to build a login screen for a mobile App like iOS or Android 😉

/organisations

We added a whole bunch of new Endpoints just for the organisation object.

screen-shot-2017-02-03-at-09-53-52

With “/organisations” you get back a list of all organisations where you are in the Owners team. The response includes the organisation API key as well.

screen-shot-2017-02-03-at-10-51-24

/organisations/orga_name/teams

With this Endpoint you get back a list of all the teams in the organisation and some basic information about team members …

screen-shot-2017-02-03-at-10-54-17

… and projects which are assigned to the team.

screen-shot-2017-02-03-at-10-54-43

/organisations/orga_name/projects

With this Endpoint you get back a list of all projects of the organisation. The list contains basic information about the projects. The more detailed information about projects please use the “/projects” Endpoint.

screen-shot-2017-02-03-at-10-58-36

/organisations/orga_name/inventory

This Endpoint returns the dependency inventory of the organisation. That is a list of all used dependencies overall projects in the organisation.

screen-shot-2017-02-03-at-11-03-37

More information about the data structure can be found here and here is another blog post about the inventory feature.

Leave a comment here if you have questions to this new API Endpoints or if you need another one 😉

 

CSV Export for OS Inventory

VersionEye monitors your projects and notifies you about out-dated dependencies, security vulnerabilities and license violations. There are many ways to create a VersionEye project and to keep it in sync with the dependencies from the daily software development. As VersionEye knows all the dependencies from all your projects it can easily display you the inventory list of all your dependencies over all your projects, in real time. You find that  inventory link in your organisation on the left side.

screen-shot-2017-02-03-at-08-21-35

That way you know exactly which and how many open source dependencies you are using and you can see immediately the licenses. Beside that the list shows you which OS dependency & version you are using in which of your projects. In the screenshot above I can see for example that the Java dependency “com.rabbitmq:amqp-client” is used in 2 of my Java projects, in the “maven-indexer” and the “versioneye-maven-crawler” project. In both projects the dependency is used in the newest version. Good for me 🙂

By default you get a complete list of ALL your dependencies overall your projects. But as you can see, there are some filters above the list which can be used to filter down the list by teams, language, version and some other criteria. Maybe you only want to see the inventory of a specific team or maybe you are only interested in the PHP inventory list of your company.

This inventory list exist already since a couple months is used heavily by Enterprise clients. If you scroll down the list you will see a CSV export link. That’s new!

screen-shot-2017-02-03-at-08-22-44

Now you can export that inventory list as CSV file. Here is an example how it looks like.

Screen Shot 2017-02-03 at 08.20.58.png

The first part of this CSV export shows exactly which dependency is used in which version, what would be the newest version, the license of the used version, the number of their known security vulnerabilities and the VersionEye project ID there this dependency is used in.

screen-shot-2017-02-03-at-08-34-04

The second part of the export shows some details about your projects where the dependency is used in, like the VersionEye project ID, project name, your project version (if available) and in case it’s a Maven project the export is showing the GroupdID & ArtifactID.

screen-shot-2017-02-03-at-08-34-18

The inventory list with all the filters is also available as API Endpoint.

screen-shot-2017-02-03-at-08-39-11

That way you can fetch the data as JSON as well and create your custom Inventory report. You could use this API Endpoint to create a custom inventory PDF report. Check out the VersionEye API.

By the way. Everything on VersionEye.com is 100% open source. The source code is on GitHub and pullrequests are welcome 😉

Read Only API Key

The VersionEye API is serving every week several Million HTTP requests and there are many plugins and AddOns out there using the VersionEye API. At the beginning only users could have an API key but in the mean while we have API keys which are attached to an organisation at VersionEye. With an organisation API key all CRUD operations inside that corresponding organisation can be performed. Some people requested an read only API key for organisations. And here it is!

screen-shot-2017-02-01-at-15-26-56

Now every organisation at VersionEye has 2 API keys. 1 “normal” one which is valid for write and delete operations and a read only one which will NOT work for HTTP POST and HTTP DELETE operations.

That way you can give somebody access to all your projects in your organisation without allowing them to change something.

Let us know how you like this feature. If you wann give feedback leave a comment here.

Simplified component whitelist

With VersionEye you can ensure a license policy by assigning a license whitelist to your project. Beside that there is a feature called component whitelist, with that you can whitelist certain components for your project. That makes totally sense for dependencies which are for example unknown to VersionEye. Another use case would be if the dependency has an unknown license, in that case you could also put the dependency on the component whitelist.

The component whitelist is a list of “LANGUAGE:PROD_KEY:VERSION” values. Until now you had to know which value you have to put on your component whitelist. For many people it was confusing. Now we simplified that. In the project view in the license tab beside each dependency which is unknown or violates the license whitelist you will find a “+” icon. But clicking on it the dependency from that row will be added to the component whitelist:

screen-shot-2017-01-31-at-16-35-11

The “+” icon only shows up if are an Admin or an Owner of the organisation AND a component whitelist is assigned to the project.

That should make your life much easier 😉