New Feature – Open Source License Compliance & Security

Default License Whitelist

In VersionEye it’s very easy to setup a license whitelist. A license whitelist describes licenses which are allowed in your organisation. You can even have multiple license whitelists per organisation. That way different projects can have different license whitelists. That makes totally sense because licenses have different obligations. Some licenses can be used in a cloud environment but not for mobile apps.

However, most people don’t know much about software licenses. They simply don’t know what to put on a license whitelist and what not. That’s why VersionEye has a default license whitelist now. It contains a small set of software licenses which can be used in any environment. The default license whitelist currently contains this licenses:

Apache-1.0
Apache-1.1
Apache-2.0
BSD
BSD-2-Clause
BSD-3-Clause
BSD-4-Clause
BSD-4-Clause-UC
CC0
CC0-1.0
ISC
MIT
Public Domain
WTFPL

This license whitelist has always the name “default_lwl” and for newly created organisations it’s marked as default license whitelist. That means it gets assigned to all new created projects and all the project dependencies are compared against that whitelist.

Screen Shot 2017-06-12 at 19.10.11

Of course you can edit the “default_lwl” any time. You can remove licenses from it and you can add new licenses to it any time. It’s just a suggestion to start with.

Let us know how this works out for you.

New project view with more details

VersionEye can monitor your software project and notify you about out-dated dependencies, license violations and security vulnerabilities. Up to now we had for each of them a separate tab in the project view page. That was bit confusing and many users even didn’t noticed that there are different tabs. That’s why improved that. Now there is one unified table view which shows all the desired information. Here an example:

Screen Shot 2017-05-10 at 09.09.47

Everything what is red is an issue. In the above example you can see immediately which dependencies have security vulnerabilities, which are outdated and which are violating your license whitelist! There is no need of switching between tabs anymore!

The header was refactored, too. Now you can download all the available project exports from the same page. By the way the version.pdf export is new 😉 Beside that you can download here the license.pdf and security.pdf for your project.

versioneye-exports

The skipped the license tab completely but kept the security tab because it’s still a nice summary for the security vulnerabilities!

Let us know how you like this new view.

Rust Support

Rust is a blazingly fast new language which guarantees thread safety! Our latest project, a command line tool to identify random components by their SHA values, is implemented in Rust. As we use Rust for our own purposes we want to track Rust dependencies of course 🙂 That’s why added support for Rust now! VersionEye is tracking more than 9K Rust packages from crates.io now and they are all available through our public search.

Screen Shot 2017-05-04 at 08.59.58

Beside that VersionEye can parse cargo.toml and cargo.lock files. If you are using cargo to manage your Rust dependencies then you can let VersionEye monitore those files to receive notifications about new releases of your dependencies.

Screen Shot 2017-05-04 at 09.09.45

The above screenshot shows a Rust project which is monitored directly on GitHub.

This integration is still very fresh. Please try it out and give us feedback!

Shazam for software libraries

Nowadays its’ common to use package managers to describe, fetch and install 3rd party open source dependencies. But that was not always the case and there are many legacy projects out there in the Enterprises of this world there 3rd party libraries are stored in a “lib” directory without any documentation. It’s not a rare case that there are dependencies like a “beanutils.jar”. But which version is it? Which license does it have? Are there any known security vulnerabilities? Nobody knows!

The VersionEye API can identify such components by their SHA values. If you generate the SHA value for your JAR file and send it to the VersionEye API then you will get back the coordinates of that JAR file in Maven. That way you will know the exact GroupId, ArtifactID and Version of the component and based on that you can find out the license and known security vulnerabilities!

Of course it’s a lot of work to do that by hand, especially if you have a couple hundred files in your lib directory. That’s why we developed a command line tool to automate that. The veye_checker is implemented in Rust and we have binaries for Windows, Linux and Mac OS X which run out of the box! Simply download the binary from here, make the binary executable and run this command:

./veye_checker resolve ~/lib -a "YOUR_API_KEY" -o inventory.csv

With “-a” you specify your API key for the VersionEye API. With “-o” you specify the output file. The default the veye_checker outputs everything as CSV to the console. The results is a CSV which shows you exactly the files, their coordinates in the corresponding package manager, their license, the number of security vulnerabilities and a link to their VersionEye page. Here a screenshot:

Currently this works for Maven (Java), Nuget (C#), NPM (Node.JS) and PyPi (Python). Right now we have almost 10 Million SHA values in our database.

This is still a very early version of the “veye_checker”, but please try it out and give us feedback. We would love to hear from you!

Identifying components by SHA values

The public VersionEye API is exposing a lot of information about open source projects. If you know the coordinates of a component (software library) in a certain package manager, you can use the VersionEye API fetch all kind of meta information about the component. Meta information like available versions, licenses, security vulnerabilities and many more.

But what if you don’t know the coordinates of a component? What if you have somewhere on your disk a “beanutils.jar” and you don’t know which version it is or to which Maven GroupId it belongs to? You don’t know which license it has and you don’t know if it is vulnerable or not! For this problem the VersionEye API has a solution now. Simply create the SHA1 value for the “beanutils.jar” file. In Ruby it’s a one liner:

sha1 = Digest::SHA1.file "beanutils.jar"

Now take the SHA1 value and use the new SHA API Endpoint at VersionEye.

screen-shot-2017-02-08-at-08-54-36

With Curl it would be this command:

curl -X GET --header 'Accept: application/json' 'https://www.versioneye.com/api/v2/products/sha/427662b038bd8f52097f783f6ea163e45851b2a1?api_key=YOUR_API_KEY'

As response you get back the coordinates of the component. Now you know the Maven GroupId & ArtifactId and the version of your JAR file. That information you can use to gather even more information about the component from the VersionEye API. Simply use the products API Endpoint:

Screen Shot 2017-02-08 at 08.55.35.png

With Curl it would be this command:

curl -X GET --header 'Accept: application/json' 'https://www.versioneye.com/api/v2/products/Java/commons-beanutils%3Acommons-beanutils?prod_version=1.9.0&api_key=YOUR_API_KEY'

With that request you get back the dependencies, licenses and security vulnerabilities of the component. Now you know that your “beanutils.jar” is licensed under the Apache-2.0 license and it has at least 1 security vulnerability.

As you know the Maven coordinates of your “beanutils.jar” you could also simply visit the corresponding VersionEye page in the browser.

Screen Shot 2017-02-08 at 08.56.58.png

Here you see as well the license and the security vulnerability. And in the right upper corner you can see that there is a newer version of the component available, version 1.9.3. By clicking on that number you will see that the newest version of the component is not vulnerable.

The VersionEye database has SHA values for more than 90K .NET projects, more than 170K Java projects, almost 400K Node.JS projects and more than 88K Python projects. Altogether we have more than 9 Million SHA values.

screen-shot-2017-02-22-at-08-48-40

Every version of a component has at least 1 SHA value. In Maven a version of a component contains multiple files, like for example “*.jar”, “-javadoc.jar”, “-sources.jar” and “.pom”. In this case every file has his own SHA value, that means to 1 artefact we have multiple SHA values in our database.

The SHA method for Maven components and Node.JS modules is always “SHA1”. For Python projects we store the MD5 hash value. For .NET components we either have a base64 encoded SHA256 or SHA512 value. If you have a .NET Nuget package simply create a base64 encoded SHA256 value for it and fire it against our API. If the result is empty repeat the process with a SHA512 value. Here an example how it would look like in Ruby:

Base64.encode64 Digest::SHA512.digest File.read('Newtonsoft.Json.9.0.1.nupkg')

For more than 99% of all Nuget packages we have a SHA value.

Currently we have only SHA values for Java components from Maven central. That’s the official central repository for Java components. VersionEye is crawling a couple other Maven repositories as well, but for those we don’t have SHA values right now. But we are working on it.

Right now we do NOT have SHA values for Ruby but we are working on it!

Try out the new API and let us know how you like it.

New API Endpoints

The VersionEye API is heavily used! In the mean white the VersionEye API is serving more HTTP requests then the web application. That’s why made it even more awesome 🙂

New SwaggerUI

For almost 2 years we didn’t update the swaggerUI, shame on us! The old swaggerUI was a bit buggy. Many times it did not submit input fields to the API, that was specially the case if there was another input field with the same name, higher up on the page. That and many other small issues are solved now with the newest swaggerUI. Beside that the new swaggerUI shows for every request the corresponding CURL command, which is super awesome because that way you can run the commands directly in the command line.

screen-shot-2017-02-03-at-10-39-19

/sessions/login

We added a new API Endpoint for sessions/login. Now you can send your user/email and password via HTTP POST to this Endpoint and you will receive back your user object with your user API key. With the user API key you can then use other Endpoints of the API.

screen-shot-2017-02-03-at-10-42-24

That is especially useful if you want to build a login screen for a mobile App like iOS or Android 😉

/organisations

We added a whole bunch of new Endpoints just for the organisation object.

screen-shot-2017-02-03-at-09-53-52

With “/organisations” you get back a list of all organisations where you are in the Owners team. The response includes the organisation API key as well.

screen-shot-2017-02-03-at-10-51-24

/organisations/orga_name/teams

With this Endpoint you get back a list of all the teams in the organisation and some basic information about team members …

screen-shot-2017-02-03-at-10-54-17

… and projects which are assigned to the team.

screen-shot-2017-02-03-at-10-54-43

/organisations/orga_name/projects

With this Endpoint you get back a list of all projects of the organisation. The list contains basic information about the projects. The more detailed information about projects please use the “/projects” Endpoint.

screen-shot-2017-02-03-at-10-58-36

/organisations/orga_name/inventory

This Endpoint returns the dependency inventory of the organisation. That is a list of all used dependencies overall projects in the organisation.

screen-shot-2017-02-03-at-11-03-37

More information about the data structure can be found here and here is another blog post about the inventory feature.

Leave a comment here if you have questions to this new API Endpoints or if you need another one 😉

CSV Export for OS Inventory

VersionEye monitors your projects and notifies you about out-dated dependencies, security vulnerabilities and license violations. There are many ways to create a VersionEye project and to keep it in sync with the dependencies from the daily software development. As VersionEye knows all the dependencies from all your projects it can easily display you the inventory list of all your dependencies over all your projects, in real time. You find that inventory link in your organisation on the left side.

screen-shot-2017-02-03-at-08-21-35

That way you know exactly which and how many open source dependencies you are using and you can see immediately the licenses. Beside that the list shows you which OS dependency & version you are using in which of your projects. In the screenshot above I can see for example that the Java dependency “com.rabbitmq:amqp-client” is used in 2 of my Java projects, in the “maven-indexer” and the “versioneye-maven-crawler” project. In both projects the dependency is used in the newest version. Good for me 🙂

By default you get a complete list of ALL your dependencies overall your projects. But as you can see, there are some filters above the list which can be used to filter down the list by teams, language, version and some other criteria. Maybe you only want to see the inventory of a specific team or maybe you are only interested in the PHP inventory list of your company.

This inventory list exist already since a couple months is used heavily by Enterprise clients. If you scroll down the list you will see a CSV export link. That’s new!

screen-shot-2017-02-03-at-08-22-44

Now you can export that inventory list as CSV file. Here is an example how it looks like.

Screen Shot 2017-02-03 at 08.20.58.png

The first part of this CSV export shows exactly which dependency is used in which version, what would be the newest version, the license of the used version, the number of their known security vulnerabilities and the VersionEye project ID there this dependency is used in.

screen-shot-2017-02-03-at-08-34-04

The second part of the export shows some details about your projects where the dependency is used in, like the VersionEye project ID, project name, your project version (if available) and in case it’s a Maven project the export is showing the GroupdID & ArtifactID.

screen-shot-2017-02-03-at-08-34-18

The inventory list with all the filters is also available as API Endpoint.

screen-shot-2017-02-03-at-08-39-11

That way you can fetch the data as JSON as well and create your custom Inventory report. You could use this API Endpoint to create a custom inventory PDF report. Check out the VersionEye API.

By the way. Everything on VersionEye.com is 100% open source. The source code is on GitHub and pullrequests are welcome 😉

Read Only API Key

The VersionEye API is serving every week several Million HTTP requests and there are many plugins and AddOns out there using the VersionEye API. At the beginning only users could have an API key but in the mean while we have API keys which are attached to an organisation at VersionEye. With an organisation API key all CRUD operations inside that corresponding organisation can be performed. Some people requested an read only API key for organisations. And here it is!

screen-shot-2017-02-01-at-15-26-56

Now every organisation at VersionEye has 2 API keys. 1 “normal” one which is valid for write and delete operations and a read only one which will NOT work for HTTP POST and HTTP DELETE operations.

That way you can give somebody access to all your projects in your organisation without allowing them to change something.

Let us know how you like this feature. If you wann give feedback leave a comment here.

Simplified component whitelist

With VersionEye you can ensure a license policy by assigning a license whitelist to your project. Beside that there is a feature called component whitelist, with that you can whitelist certain components for your project. That makes totally sense for dependencies which are for example unknown to VersionEye. Another use case would be if the dependency has an unknown license, in that case you could also put the dependency on the component whitelist.

The component whitelist is a list of “LANGUAGE:PROD_KEY:VERSION” values. Until now you had to know which value you have to put on your component whitelist. For many people it was confusing. Now we simplified that. In the project view in the license tab beside each dependency which is unknown or violates the license whitelist you will find a “+” icon. But clicking on it the dependency from that row will be added to the component whitelist:

screen-shot-2017-01-31-at-16-35-11

The “+” icon only shows up if are an Admin or an Owner of the organisation AND a component whitelist is assigned to the project.

That should make your life much easier 😉

Improved license recognition

VersionEye is monitoring now more than 1.2 Million open source projects and collecting all kind of meta information to this projects. One kind of the meta information is the corresponding license. Currently the VersionEye database contains licenses to more than 8 Million artefacts.

However, sometimes the maintainers of a project didn’t provide the license information that way that it’s easy to read and recognise for machines. That is specially the case for Python and the .NET platform. Take for example the gpkit Python library. In the license field they provided the full license text, not just the license name.

python-gpkit-00

That doesn’t match very well with the license whitelist in VersionEye 😉 That’s why we improved it now!

All together we have 11335 Python licenses like that in our database and they belong to 1989 unique projects on PIP. Our new license crawler could match 9933 licenses to SPDX identifiers. That are 1799 unique projects on PIP we could assign a SPDX identifier to, which didn’t had one before. For more than 90% of this projects we could recognise and identify an SPDX identifier. And now the same library on VersionEye looks like that:

python-gpkit-01

You see clear that it’s MIT license and now this works well together with our license whitelist 🙂

In Nuget, the package manager for the .NET platform, many license names look like this.

“Nuget unknown”. That is the case if the maintainers provided a license link but didn’t provide a license name. Our new license crawler is now following this links and with machine learning it tries to identify a known license. If the similarity to a known license text is higher that 90% we assign the corresponding SPDX identifier to the software library in our database. And now the same package looks like this and you can see immediately that it’s the MIT license!

That also helps to use these packages together with our license whitelist. For the .NET packages our recognition system was not quiet as good as for Python. For .NET we could only identify 65% of the licenses of packages with a link but without a license name. Stil not bad and much better than before 😉

This are just the first results. Of course this is also work in progress and the recognition will become better in future.

Your feedback is very welcome.