Shazam for software libraries

Nowadays its’ common to use package managers to describe, fetch and install 3rd party open source dependencies. But that was not always the case and there are many legacy projects out there in the Enterprises of this world there 3rd party libraries are stored in a “lib” directory without any documentation. It’s not a rare case that there are dependencies like a “beanutils.jar”. But which version is it? Which license does it have? Are there any known security vulnerabilities? Nobody knows!

The VersionEye API can identify such components by their SHA values. If you generate the SHA value for your JAR file and send it to the VersionEye API then you will get back the coordinates of that JAR file in Maven. That way you will know the exact GroupId, ArtifactID and Version of the component and based on that you can find out the license and known security vulnerabilities!

Of course it’s a lot of work to do that by hand, especially if you have a couple hundred files in your lib directory. That’s why we developed a command line tool to automate that. The veye_checker is implemented in Rust and we have binaries for Windows, Linux and Mac OS X which run out of the box! Simply download the binary from here, make the binary executable and run this command:

./veye_checker resolve ~/lib -a "YOUR_API_KEY" -o inventory.csv

With “-a” you specify your API key for the VersionEye API. With “-o” you specify the output file. The default the veye_checker outputs everything as CSV to the console. The results is a CSV which shows you exactly the files, their coordinates in the corresponding package manager, their license, the number of security vulnerabilities and a link to their VersionEye page. Here a screenshot:

os_inventory

Currently this works for Maven (Java), Nuget (C#), NPM (Node.JS) and PyPi (Python). Right now we have almost 10 Million SHA values in our database.

This is still a very early version of the “veye_checker”, but please try it out and give us feedback. We would love to hear from you!

Identifying components by SHA values

The public VersionEye API is exposing a lot of information about open source projects. If you know the coordinates of a component (software library) in a certain package manager, you can use the VersionEye API fetch all kind of meta information about the component. Meta information like available versions, licenses, security vulnerabilities and many more.

But what if you don’t know the coordinates of a component? What if you have somewhere on your disk a “beanutils.jar” and you don’t know which version it is or to which Maven GroupId it belongs to? You don’t know which license it has and you don’t know if it is vulnerable or not! For this problem the VersionEye API has a solution now. Simply create the SHA1 value for the “beanutils.jar” file. In Ruby it’s a one liner:

sha1 = Digest::SHA1.file "beanutils.jar"

Now take the SHA1 value and use the new SHA API Endpoint at VersionEye.

screen-shot-2017-02-08-at-08-54-36

With Curl it would be this command:

curl -X GET --header 'Accept: application/json' 'https://www.versioneye.com/api/v2/products/sha/427662b038bd8f52097f783f6ea163e45851b2a1?api_key=YOUR_API_KEY'

As response you get back the coordinates of the component. Now you know the Maven GroupId & ArtifactId and the version of your JAR file. That information you can use to gather even more information about the component from the VersionEye API. Simply use the products API Endpoint:

Screen Shot 2017-02-08 at 08.55.35.png

With Curl it would be this command:

curl -X GET --header 'Accept: application/json' 'https://www.versioneye.com/api/v2/products/Java/commons-beanutils%3Acommons-beanutils?prod_version=1.9.0&api_key=YOUR_API_KEY'

With that request you get back the dependencies, licenses and security vulnerabilities of the component. Now you know that your “beanutils.jar” is licensed under the Apache-2.0 license and it has at least 1 security vulnerability.

As you know the Maven coordinates of your “beanutils.jar” you could also simply visit the corresponding VersionEye page in the browser.

Screen Shot 2017-02-08 at 08.56.58.png

Here you see as well the license and the security vulnerability. And in the right upper corner you can see that there is a newer version of the component available, version 1.9.3. By clicking on that number you will see that the newest version of the component is not vulnerable.

The VersionEye database has SHA values for more than 90K .NET projects, more than 170K Java projects, almost 400K Node.JS projects and more than 88K Python projects. Altogether we have more than 9 Million SHA values.

screen-shot-2017-02-22-at-08-48-40

Every version of a component has at least 1 SHA value. In Maven a version of a component contains multiple files, like for example “*.jar”, “-javadoc.jar”, “-sources.jar” and “.pom”. In this case every file has his own SHA value, that means to 1 artefact we have multiple SHA values in our database.

The SHA method for Maven components and Node.JS modules is always “SHA1”. For Python projects we store the MD5 hash value. For .NET components we either have a base64 encoded SHA256 or SHA512 value. If you have a .NET Nuget package simply create a base64 encoded SHA256 value for it and fire it against our API. If the result is empty repeat the process with a SHA512 value. Here an example how it would look like in Ruby:

Base64.encode64 Digest::SHA512.digest File.read('Newtonsoft.Json.9.0.1.nupkg')

For more than 99% of all Nuget packages we have a SHA value.

Currently we have only SHA values for Java components from Maven central. That’s the official central repository for Java components. VersionEye is crawling a couple other Maven repositories as well, but for those we don’t have SHA values right now. But we are working on it.

Right now we do NOT have SHA values for Ruby but we are working on it!

Try out the new API and let us know how you like it.

VersionEye Maven Plugin 3.11.1

Version 3.11.1 of the VersionEye Maven Plugin is out and here are the change logs. This release brings a couple improvements and BugFix!

BugFix

Assume you have a Maven multi module project with 3 modules. Now you add the VersionEye Maven Plugin to the parent pom and run mvn versioneye:create. That will create a new project with subprojects in VersionEye. Now on your CI system on each build you run maybe this goal: mvn versioneye:securityAndLicenseCheck. That will always update the VersionEye project with the current dependencies from your Maven project and check them for security vulnerabilities and license violations. Everything works so far.

Now you are adding a new Maven module to your Maven reactor build and the next execution of mvn versioneye:securityAndLicenseCheck will fail, because it will try to update an non existing module on the VersionEye server.

This bug is fixed now! Version 3.11.1 of the VersionEye Maven Plugin can handle newly added Maven modules. If a new module is added it will automatically create a new subproject for that on the VersionEye server.

Improved Merging

For Maven multi module projects one project is created for the parent pom and for each module a project is created on the VersionEye server and merged into the project for the parent pom. This merging process was working through the GAV coordinates of the parent pom. That was leading to problems if the same project was monitored twice on a VersionEye server. The new merge mechanism is independent from the GAV coordinates, it only relies on the project ID from the VersionEye API.

Support for Project License

The VersionEye Maven Plugin is resolving all dependencies locally and only sends a pom.json file to the VersionEye API, which contains only the needed information. Now this pom.json file also contains the license from the pom.xml file and the project license is displayed in the web interface.

Screen Shot 2017-01-12 at 13.31.55.png

 

VersionEye Maven Plugin 3.9.1

VersionEye Maven Plugin 3.9.1 is out and it has a lot of new features!

Out-dated dependencies

Now the Mavne golas mvn versioneye:create and mvn versioneye:update output the out-dated dependencies directly in the console. Like in this example:

Screen Shot 2016-03-02 at 09.05.20

ignoreDependencyManagement

This new configuration option can be used to ignore dependencies which are defined in the dependencyManagement section of the pom.xml. By default the value of this switch is false. That means by default the dependencies which are defined in the dependencyManagement section of the pom.xml are processed and send to the server. If you don’t want that, just set the switch to false!

API Key

The way how the API key is fetched was completely refactored! Now it’s possible to keep the API key in the environment variable VERSIONEYE_API_KEY. That is specially useful for setups on a central build server like Jenkins for example!

The plugin will look for the API Key at this places in that order:

  • Environment variable VERSIONEYE_API_KEY
  • ~/.m2/versioneye.properties
  • src/qa/resources/versioneye.properties
  • src/main/resources/versioneye.properties
  • pom.xml file

That means if the environment variable VERSIONEYE_API_KEY is set, it can be overwritten by the~/.m2/versioneye.properties file. And that can be overwritten by the settings insrc/qa/resources/versioneye.properties and so on.

Proxy Settings

Until now the proxy settings could be configured directly in the pom.xml file in the plugin configuration part. Now the configuration for the proxy can be placed at ~/.m2/versioneye.properties as well. That way it’s centralised and not all projects need to configure it on their own. Just put the proxy settings into the ~/.m2/versioneye.properties file like this:

proxyHost=PROXY_HOST
proxyPort=PROXY_PORT
proxyUser=PROXY_USER
proxyPassword=PROXY_PASSWORD

If a project configures the proxy in the pom.xml in the configuration part of the plugin, that values are taken for that project. But otherwise the plugin will look for proxy settings at~/.m2/versioneye.properties.

mvn versioneye:delete

Now there is new goal for deleting a project from the server.

mvn versioneye:delete

The goal above will delete the project and all it’s modules from the VersionEye server and also remove alle versioneye.properties files from the source code. That is specially handy for big Java projects with many modules!

Support for Apache Maven Changelog Plugin

The Apache Maven Changelog Plugin is a plugin for the Java build tool Maven. It generates reports regarding the recent changes in your SCM. To generate the reports for the changes between two git tags/releases you have to add the plugin with this configuration to your pom.xml file.

<reporting>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-changelog-plugin</artifactId>
      <version>2.3</version>
      <configuration>
        <type>tag</type>
        <tags>
          <tag implementation="java.lang.String">3.5.0</tag>
          <tag implementation="java.lang.String">3.5.1</tag>
        </tags>
      </configuration>
    </plugin>
  </plugins>
</reporting>

This will generate a report for alle SCM changes between the tags 3.5.0 and 3.5.1 of the VersionEye Maven Plugin.  The connection to the SCM has to be configured in the scm tag, like this for example:

<scm>
  <connection>scm:git:https://github.com/versioneye/versioneye_maven_plugin.git</connection>
  <developerConnection>scm:git:https://github.com/versioneye/versioneye_maven_plugin.git</developerConnection>
  <url>https://github.com/versioneye/versioneye_maven_plugin.git</url>
</scm>

To generate the report simply run:

mvn site

That will generate the default reports and the changes reports. Beside that the command generates a file “target/changelog.xml”. This XML file contains all the information about the changes and is the ground for the HTML reports which are generated by “mvn site”.

Now VersionEye has a support for changelog.xml files. There is a new Endpoint at the VersionEye API which allows to push a changelog.xml file and to assign it to a specific software artefact.

versioneye_changelog_xml

I generated the changelog.xml file for the VersionEye Maven Plugin version 3.5.1. The changelog.xml file can be found in this gist. Now that VersionEye has the SCM changes it can display the changes directly on the package page.

VersionEye-SCM-Changes

Check it out online here.

The revision numbers are even clickable and lead directly to the GitHub commit. That is a little bit magic because the changelog.xml file doesn’t contain any information about the SCM itself. If you want to have the revision numbers clickable you have to add 1 additional line to the changelog.xml with the “revision_base” tag. Like in my changelog.xml file in line 3.

This feature was mainly build for VersionEye Enterprise and is very useful as change management if you have a lot of closed source components in your company.

Your feedback is highly appreciated.

VersionEye Maven Plugin 3.5.0

Version 3.5.0 of the VersionEye Maven Plugin is out. It offers another configuration option to skip certain dependencies with a defined scope.

By default the VersionEye Maven Plugin is resolving ALL dependencies and sending ALL dependencies to the VersionEye API. The VersionEye Server is then checking the dependencies and is sending out emails. But sometimes you get notifications for a dependency which is under test scope or provided scope. Something you don’t really ship with your application. Ideally you want VersionEye to ignore those dependencies. Now you can configure which scopes should be skipped by the plugin. Simply add this line to your plugin configuration.

<skipScopes>test,provided</skipScopes>

The line above will ignore all dependencies which have test or provided scope.

VersionEye Maven Plugin 3.2.1

Just released version 3.2.1 of the VersionEye Maven Plugin.

This version is a patch release! If you run “mvn versioneye:create" in a multi module project on the parent it might be that the parent pom.xml doesn’t has any dependencies. The default behaviour of VersionEye is to skip projects with 0 dependencies. In a multi module project this behaviour causes the error that the submodules can’t be merged into the parent project, because the parent was skipped because it has 0 dependencies.
This Bug is fixed now!

VersionEye Maven Plugin 3.2.0

This week version 3.2.0 of the VersionEye Maven Plugin was released!

You need the VersionEye Maven Plugin if you want to track multi module builds with VersionEye. The plugin runs inside your Maven build lifecycle and resolves all dependencies & variables natively with the maven-core project. It is the best way to track Maven projects on VersionEye!

You can use the plugin by simply adding this code snippet to your pom.xml file.

<build>
  <plugins>
    <plugin>
      <groupId>com.versioneye</groupId>
      <artifactId>versioneye-maven-plugin</artifactId>
      <version>3.2.0</version>
      <configuration>
        <apiKey>MY_SECRET_API_KEY</apiKey>
      </configuration>
    </plugin>
  </plugins>
</build>

You can find more information about the API Key here.

The current version brings many small improvements and much more configuration options! Find here the change logs. I want to point out 2 changes.

Scopes

The plugin resolves ALL dependencies & variables locally and creates a pom.json document, with all the relevant information for VersionEye, and sends it to the VersionEye API. Now the scopes of the dependencies are send to the server as well! And in the Web interface the dependencies are divided by scopes.

Screen Shot 2015-02-25 at 11.13.50

Set parent project explicitly

Executing “mvn versioneye:create” on a multi module project will merge all submodules into the parent project on the server, by default. By default the parent project is determined from the pom.xml. Here is an example. The maven project on GitHub has a parent pom.xml file in the root and a couple submodules. The pom.xml files of the submodules reference as parent the pom.xml file in the root directory.

Screen Shot 2015-02-25 at 11.23.13

This is kind of normal. Many multi module projects have this structure. Executing the VersionEye Maven Plugin on this project would create a single project on VersionEye and each module would be a “subproject” on VersionEye. The advantage of this is that you have everything in one single view and the numbers for dependencies and licenses are summed up over all submodules. It looks like this:

Screen Shot 2015-02-25 at 11.35.59

If you wish to remain each submodule as separate project on VersionEye you can turn of the merging with this line in the plugin configuration:

<mergeAfterCreate>false</mergeAfterCreate>

This is all kind of default. But what happens if one of your submodules references a parent project which is in a complete different git repository? It’s not part of the current maven build. And maybe the referenced parent project is even not available at VersionEye. In that case the merging would fail because VersionEye tries to merge an existing project into a non existing project. In this case you could turn of the merge with the configuration above. That would be a quick fix.

But what if you want to merge each submodule into the maven project which is described with the pom.xml in the root directory? That is not the parent which is set as parent in the submodules pom.xml. But maybe you still want to enforce this logical structure on VersionEye! With the current version of the plugin you can enforce into which project on VersionEye the submodules should be merged, by simply setting the parent GA explicitly in the plugin configuration.

<parentGroupId>your.parent.groupid</parentGroupId>
<parentArtifactId>your.artifactid</parentArtifactId>

This is not a theoretical case. It’s a real problem which happened to some VersionEye users. With the current version of the plugin this problem is solved now!

Let me know if you have questions to this. I’m happy to help!

Improved Crawlers for Maven – jCenter, Atlassian, Adobe

VersionEye is crawling a bunch of repositories like RubyGems, NPMRegistry, Packagist and many more. For Java we crawled for long time only search.maven.org. Simply because it provides an maven index to download and that makes it really easy to get the data into our database! Other maven repositories don’t provide a maven index and that means we need to do some heavy xml parsing! Simply doubling the number of our EC2 instances is not an option because currently we don’t swim in Money! In the last weeks we worked hard on performance improvements for the non-maven-index maven crawlers. And because the work was successful we are happy to announce that VersionEye supports this maven repositories as well:

Now VersionEye tracks 123K Java projects with 1.37 million artefacts!

VersionEye Maven Plugin 3.1.1

Version 3.1.1 of the VersionEye Maven Plugin is out. It brings improvements for multi module projects and a better handling of error messages.

  • Better error messages: Until now the plugin did not handle error messages from the server. If the API was not available the build just failed with a random Exception. The current version handles the response code from the API. If the server responds with an error message it will be displayed in the command line as well!
  • Multi Module Builds: In a multi module project the plugin created a separate project for each module. This behaviour changed in this version. Since VersionEye supports projects with multiple files the plugin creates always only 1 single project! If it is a multi module project, each module is a “subproject” in the VersionEye project.

You can find the source code on GitHub: https://github.com/versioneye/versioneye_maven_plugin.

Give it a try and let us know what you think.