java – Open Source License Compliance & Security

Identifying components by SHA values

The public VersionEye API is exposing a lot of information about open source projects. If you know the coordinates of a component (software library) in a certain package manager, you can use the VersionEye API fetch all kind of meta information about the component. Meta information like available versions, licenses, security vulnerabilities and many more.

But what if you don’t know the coordinates of a component? What if you have somewhere on your disk a “beanutils.jar” and you don’t know which version it is or to which Maven GroupId it belongs to? You don’t know which license it has and you don’t know if it is vulnerable or not! For this problem the VersionEye API has a solution now. Simply create the SHA1 value for the “beanutils.jar” file. In Ruby it’s a one liner:

sha1 = Digest::SHA1.file "beanutils.jar"

Now take the SHA1 value and use the new SHA API Endpoint at VersionEye.

screen-shot-2017-02-08-at-08-54-36

With Curl it would be this command:

curl -X GET --header 'Accept: application/json' 'https://www.versioneye.com/api/v2/products/sha/427662b038bd8f52097f783f6ea163e45851b2a1?api_key=YOUR_API_KEY'

As response you get back the coordinates of the component. Now you know the Maven GroupId & ArtifactId and the version of your JAR file. That information you can use to gather even more information about the component from the VersionEye API. Simply use the products API Endpoint:

Screen Shot 2017-02-08 at 08.55.35.png

With Curl it would be this command:

curl -X GET --header 'Accept: application/json' 'https://www.versioneye.com/api/v2/products/Java/commons-beanutils%3Acommons-beanutils?prod_version=1.9.0&api_key=YOUR_API_KEY'

With that request you get back the dependencies, licenses and security vulnerabilities of the component. Now you know that your “beanutils.jar” is licensed under the Apache-2.0 license and it has at least 1 security vulnerability.

As you know the Maven coordinates of your “beanutils.jar” you could also simply visit the corresponding VersionEye page in the browser.

Screen Shot 2017-02-08 at 08.56.58.png

Here you see as well the license and the security vulnerability. And in the right upper corner you can see that there is a newer version of the component available, version 1.9.3. By clicking on that number you will see that the newest version of the component is not vulnerable.

The VersionEye database has SHA values for more than 90K .NET projects, more than 170K Java projects, almost 400K Node.JS projects and more than 88K Python projects. Altogether we have more than 9 Million SHA values.

screen-shot-2017-02-22-at-08-48-40

Every version of a component has at least 1 SHA value. In Maven a version of a component contains multiple files, like for example “*.jar”, “-javadoc.jar”, “-sources.jar” and “.pom”. In this case every file has his own SHA value, that means to 1 artefact we have multiple SHA values in our database.

The SHA method for Maven components and Node.JS modules is always “SHA1”. For Python projects we store the MD5 hash value. For .NET components we either have a base64 encoded SHA256 or SHA512 value. If you have a .NET Nuget package simply create a base64 encoded SHA256 value for it and fire it against our API. If the result is empty repeat the process with a SHA512 value. Here an example how it would look like in Ruby:

Base64.encode64 Digest::SHA512.digest File.read('Newtonsoft.Json.9.0.1.nupkg')

For more than 99% of all Nuget packages we have a SHA value.

Currently we have only SHA values for Java components from Maven central. That’s the official central repository for Java components. VersionEye is crawling a couple other Maven repositories as well, but for those we don’t have SHA values right now. But we are working on it.

Right now we do NOT have SHA values for Ruby but we are working on it!

Try out the new API and let us know how you like it.

VersionEye Maven Plugin 3.11.1

Version 3.11.1 of the VersionEye Maven Plugin is out and here are the change logs. This release brings a couple improvements and BugFix!

BugFix

Assume you have a Maven multi module project with 3 modules. Now you add the VersionEye Maven Plugin to the parent pom and run mvn versioneye:create. That will create a new project with subprojects in VersionEye. Now on your CI system on each build you run maybe this goal: mvn versioneye:securityAndLicenseCheck. That will always update the VersionEye project with the current dependencies from your Maven project and check them for security vulnerabilities and license violations. Everything works so far.

Now you are adding a new Maven module to your Maven reactor build and the next execution of mvn versioneye:securityAndLicenseCheck will fail, because it will try to update an non existing module on the VersionEye server.

This bug is fixed now! Version 3.11.1 of the VersionEye Maven Plugin can handle newly added Maven modules. If a new module is added it will automatically create a new subproject for that on the VersionEye server.

Improved Merging

For Maven multi module projects one project is created for the parent pom and for each module a project is created on the VersionEye server and merged into the project for the parent pom. This merging process was working through the GAV coordinates of the parent pom. That was leading to problems if the same project was monitored twice on a VersionEye server. The new merge mechanism is independent from the GAV coordinates, it only relies on the project ID from the VersionEye API.

Support for Project License

The VersionEye Maven Plugin is resolving all dependencies locally and only sends a pom.json file to the VersionEye API, which contains only the needed information. Now this pom.json file also contains the license from the pom.xml file and the project license is displayed in the web interface.

Screen Shot 2017-01-12 at 13.31.55.png

VersionEye Maven Plugin 3.10.2

Today we released the VersionEye Maven Plugin 3.10.2. This patch release is fixing a bug which occurred on big multi module projects, where sometimes the project_id couldn’t be found. Beside that the logging is improved. Now the plugin will log out at which places it is looking for the project_id. That way it’s easier do debug if something goes wrong.

VersionEye Maven Plugin – Transitiv Dependencies

Version 3.10.0 of the popular VersionEye Maven Plugin is out! This new release brings a new feature to the plugin. Now with this optional configuration parameter:

<transitiveDependencies>true</transitiveDependencies>

he plugin will resolve ALL direct AND transitive dependencies and send all of them to the VersionEye API. By default this is turned off. If this new option is turned of the plugin will only send the direct dependencies to the VersionEye API and the result might look similar to this one.

VersionEye-Direct-Dependencies

If the transitivDependencies option is turned on the scopes are ignored. The plugin will create 2 scopes, 1 for the direct dependencies and 1 for the transitive dependencies. The result will look similar to this one:

VersionEye-Transitiv-Dependencies

This configuration option also has some implications to other configuration options. If transitiveDependencies is true then the configurations optionsignoreDependencyManagement and skipScopes are ignored.

Try out this new release and feel free to open a ticket on the GitHub repository if you find any issues.

VersionEye Maven Plugin 3.9.1

VersionEye Maven Plugin 3.9.1 is out and it has a lot of new features!

Out-dated dependencies

Now the Mavne golas mvn versioneye:create and mvn versioneye:update output the out-dated dependencies directly in the console. Like in this example:

Screen Shot 2016-03-02 at 09.05.20

ignoreDependencyManagement

This new configuration option can be used to ignore dependencies which are defined in the dependencyManagement section of the pom.xml. By default the value of this switch is false. That means by default the dependencies which are defined in the dependencyManagement section of the pom.xml are processed and send to the server. If you don’t want that, just set the switch to false!

API Key

The way how the API key is fetched was completely refactored! Now it’s possible to keep the API key in the environment variable VERSIONEYE_API_KEY. That is specially useful for setups on a central build server like Jenkins for example!

The plugin will look for the API Key at this places in that order:

Environment variable VERSIONEYE_API_KEY
~/.m2/versioneye.properties
src/qa/resources/versioneye.properties
src/main/resources/versioneye.properties
pom.xml file

That means if the environment variable VERSIONEYE_API_KEY is set, it can be overwritten by the~/.m2/versioneye.properties file. And that can be overwritten by the settings insrc/qa/resources/versioneye.properties and so on.

Proxy Settings

Until now the proxy settings could be configured directly in the pom.xml file in the plugin configuration part. Now the configuration for the proxy can be placed at ~/.m2/versioneye.properties as well. That way it’s centralised and not all projects need to configure it on their own. Just put the proxy settings into the ~/.m2/versioneye.properties file like this:

proxyHost=PROXY_HOST
proxyPort=PROXY_PORT
proxyUser=PROXY_USER
proxyPassword=PROXY_PASSWORD

If a project configures the proxy in the pom.xml in the configuration part of the plugin, that values are taken for that project. But otherwise the plugin will look for proxy settings at~/.m2/versioneye.properties.

mvn versioneye:delete

Now there is new goal for deleting a project from the server.

mvn versioneye:delete

The goal above will delete the project and all it’s modules from the VersionEye server and also remove alle versioneye.properties files from the source code. That is specially handy for big Java projects with many modules!

VersionEye Maven Plugin 3.7.0

Release 3.7.0 of the VersionEye Maven Plugin is out and it brings a couple cool new features!

Support for Organisations & Teams

If a new project is created, by default, it is attached to the user account who created the project. In the Web interface the project can be moved into an organisation and then a team can be assigned to it. Until now that was a lot of manual work. Now with this release it can be automated. The new version of the VersionEye Maven Plugin has 2 more configuration options. Now the organisation and the team can be defined directly in the plugin configuration, like in this example:

<plugins>
    <plugin>
        <groupId>com.versioneye</groupId>
        <artifactId>versioneye-maven-plugin</artifactId>
        <version>3.7.0</version>
        <configuration>
            <apiKey>API_KEY</apiKey>
            <organisation>versioneye</organisation>
            <team>BackendDevs</team>
        </configuration>
    </plugin>
</plugins>

By creating the project with this command:

mvn versioneye:create

The project will be created at VersionEye and automatically assigned to the organisation, in this example to the organisation “versioneye”. And the team also gets assigned to the project, in the example above it would be “BackendDevs”. The organisation and team names are case sensitive! Please take care of that, otherwise it will not work!

This feature also assumes that the owner of the API key is part of that organisation and has the permissions to attach projects to that organisation and the permission to assign teams to projects! If that is not the case or the organisation doesn’t exist, the project still gets created, but then it belongs to the user who owns the API key. Just like the old behaviour!

securityCheck

Similar to the licenseCheck goal, there is a new goal “securityCheck”!

mvn versioneye:securityCheck

This goal checks how many of the project dependencies have known security vulnerabilities! If the count is greater than 0, this goal will exit with an non zero exit code! If you run this goal on a CI server, this will break the build if there are security vulnerabilities!

securityAndLicenseCheck

Similar to the securityCheck, this goal is checking security AND license violations!

mvn versioneye:securityAndLicenseCheck

If a dependency has a known security vulnerability OR a dependency is violating the license whitelist on the server, this goal will exit with an non zero exit code! Means, that will break your build if you run it on a CI server!

Try out this cool new features and leave a feedback comment here if you like 😉

Security Alerts for Java & Python

A couple months ago VersionEye started to track security vulnerabilities for PHP packages. A couple weeks ago the feature was rolled out for Ruby & Node.JS dependencies as well. And now it’s rolled out for Java & Python!

Now you will see the security tab in your Java & Python projects as well. Just like in the example here.

If you click on the dependency link above you will come to the package detail page where more details to the security issues are visible.

If your project dependencies are affected the dependency badge turns to “insecure”, showing everybody that some of the dependencies have security issues.

The security feature is available via the VersionEye API as well. You can filter by language and prod_key. Feel free to build your own integration on it 🙂

VersionEye notifies you about security vulnerabilities independently from the version & license notifications. The security notification emails are going out on each Tuesday.

Currently VersionEye is crawling 6 different security sources for this feature. For Java & Python we are using the victims db, which claims to have 0 false positives. Please contribute to this db if you know about a Java or Python security vulnerability and help to make the world a safer place.

Do you know more good security databases which you would like to see integrated with VersionEye? If so contact me on Twitter please.

Support for Apache Maven Changelog Plugin

The Apache Maven Changelog Plugin is a plugin for the Java build tool Maven. It generates reports regarding the recent changes in your SCM. To generate the reports for the changes between two git tags/releases you have to add the plugin with this configuration to your pom.xml file.

<reporting>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-changelog-plugin</artifactId>
      <version>2.3</version>
      <configuration>
        <type>tag</type>
        <tags>
          <tag implementation="java.lang.String">3.5.0</tag>
          <tag implementation="java.lang.String">3.5.1</tag>
        </tags>
      </configuration>
    </plugin>
  </plugins>
</reporting>

This will generate a report for alle SCM changes between the tags 3.5.0 and 3.5.1 of the VersionEye Maven Plugin. The connection to the SCM has to be configured in the scm tag, like this for example:

<scm>
  <connection>scm:git:https://github.com/versioneye/versioneye_maven_plugin.git</connection>
  <developerConnection>scm:git:https://github.com/versioneye/versioneye_maven_plugin.git</developerConnection>
  <url>https://github.com/versioneye/versioneye_maven_plugin.git</url>
</scm>

To generate the report simply run:

mvn site

That will generate the default reports and the changes reports. Beside that the command generates a file “target/changelog.xml”. This XML file contains all the information about the changes and is the ground for the HTML reports which are generated by “mvn site”.

Now VersionEye has a support for changelog.xml files. There is a new Endpoint at the VersionEye API which allows to push a changelog.xml file and to assign it to a specific software artefact.

I generated the changelog.xml file for the VersionEye Maven Plugin version 3.5.1. The changelog.xml file can be found in this gist. Now that VersionEye has the SCM changes it can display the changes directly on the package page.

Check it out online here.

The revision numbers are even clickable and lead directly to the GitHub commit. That is a little bit magic because the changelog.xml file doesn’t contain any information about the SCM itself. If you want to have the revision numbers clickable you have to add 1 additional line to the changelog.xml with the “revision_base” tag. Like in my changelog.xml file in line 3.

This feature was mainly build for VersionEye Enterprise and is very useful as change management if you have a lot of closed source components in your company.

Your feedback is highly appreciated.

VersionEye SBT Plugin

Probably you know already the VersionEye Maven Plugin. Now there is a similar plugin for Scala SBT as well. The VersionEye SBT Plugin is maintained by Mark Paluch. He basically implemented the whole SBT plugin in 1 single day, because he is awesome!

Currently the plugin is not pushed to Maven Central or JCenter. Not yet! If you want to try it out, please clone the repository with git, navigate into the directory and run this to install the plugin into your local repository.

sbt publish

After that you can use the plugin in your own SBT projects. Simply follow the instructions on the README page here.

The cool thing is, that the plugin has a licenseCheck goal. This

sbt versioneye:licenseCheck

will update an existing VersionEye project with your current dependencies and check all dependencies against a central license whitelist on the server. If one of your dependencies is violating the license whitelist on the server, the plugin will exit with an Exception. If you run that goal on a CI server it will break your build if there is a license violation. If you work in a commercial team and you want to ensure that nobody is accidentally introducing a copyleft license, like GPL for example, then that’s the way to go 😉

VersionEye Maven Plugin 3.5.0

Version 3.5.0 of the VersionEye Maven Plugin is out. It offers another configuration option to skip certain dependencies with a defined scope.

By default the VersionEye Maven Plugin is resolving ALL dependencies and sending ALL dependencies to the VersionEye API. The VersionEye Server is then checking the dependencies and is sending out emails. But sometimes you get notifications for a dependency which is under test scope or provided scope. Something you don’t really ship with your application. Ideally you want VersionEye to ignore those dependencies. Now you can configure which scopes should be skipped by the plugin. Simply add this line to your plugin configuration.

<skipScopes>test,provided</skipScopes>

The line above will ignore all dependencies which have test or provided scope.