API – Open Source License Compliance & Security

New API Endpoints

The VersionEye API is heavily used! In the mean white the VersionEye API is serving more HTTP requests then the web application. That’s why made it even more awesome 🙂

New SwaggerUI

For almost 2 years we didn’t update the swaggerUI, shame on us! The old swaggerUI was a bit buggy. Many times it did not submit input fields to the API, that was specially the case if there was another input field with the same name, higher up on the page. That and many other small issues are solved now with the newest swaggerUI. Beside that the new swaggerUI shows for every request the corresponding CURL command, which is super awesome because that way you can run the commands directly in the command line.

screen-shot-2017-02-03-at-10-39-19

/sessions/login

We added a new API Endpoint for sessions/login. Now you can send your user/email and password via HTTP POST to this Endpoint and you will receive back your user object with your user API key. With the user API key you can then use other Endpoints of the API.

screen-shot-2017-02-03-at-10-42-24

That is especially useful if you want to build a login screen for a mobile App like iOS or Android 😉

/organisations

We added a whole bunch of new Endpoints just for the organisation object.

screen-shot-2017-02-03-at-09-53-52

With “/organisations” you get back a list of all organisations where you are in the Owners team. The response includes the organisation API key as well.

screen-shot-2017-02-03-at-10-51-24

/organisations/orga_name/teams

With this Endpoint you get back a list of all the teams in the organisation and some basic information about team members …

screen-shot-2017-02-03-at-10-54-17

… and projects which are assigned to the team.

screen-shot-2017-02-03-at-10-54-43

/organisations/orga_name/projects

With this Endpoint you get back a list of all projects of the organisation. The list contains basic information about the projects. The more detailed information about projects please use the “/projects” Endpoint.

screen-shot-2017-02-03-at-10-58-36

/organisations/orga_name/inventory

This Endpoint returns the dependency inventory of the organisation. That is a list of all used dependencies overall projects in the organisation.

screen-shot-2017-02-03-at-11-03-37

More information about the data structure can be found here and here is another blog post about the inventory feature.

Leave a comment here if you have questions to this new API Endpoints or if you need another one 😉

CSV Export for OS Inventory

VersionEye monitors your projects and notifies you about out-dated dependencies, security vulnerabilities and license violations. There are many ways to create a VersionEye project and to keep it in sync with the dependencies from the daily software development. As VersionEye knows all the dependencies from all your projects it can easily display you the inventory list of all your dependencies over all your projects, in real time. You find that inventory link in your organisation on the left side.

screen-shot-2017-02-03-at-08-21-35

That way you know exactly which and how many open source dependencies you are using and you can see immediately the licenses. Beside that the list shows you which OS dependency & version you are using in which of your projects. In the screenshot above I can see for example that the Java dependency “com.rabbitmq:amqp-client” is used in 2 of my Java projects, in the “maven-indexer” and the “versioneye-maven-crawler” project. In both projects the dependency is used in the newest version. Good for me 🙂

By default you get a complete list of ALL your dependencies overall your projects. But as you can see, there are some filters above the list which can be used to filter down the list by teams, language, version and some other criteria. Maybe you only want to see the inventory of a specific team or maybe you are only interested in the PHP inventory list of your company.

This inventory list exist already since a couple months is used heavily by Enterprise clients. If you scroll down the list you will see a CSV export link. That’s new!

screen-shot-2017-02-03-at-08-22-44

Now you can export that inventory list as CSV file. Here is an example how it looks like.

Screen Shot 2017-02-03 at 08.20.58.png

The first part of this CSV export shows exactly which dependency is used in which version, what would be the newest version, the license of the used version, the number of their known security vulnerabilities and the VersionEye project ID there this dependency is used in.

screen-shot-2017-02-03-at-08-34-04

The second part of the export shows some details about your projects where the dependency is used in, like the VersionEye project ID, project name, your project version (if available) and in case it’s a Maven project the export is showing the GroupdID & ArtifactID.

screen-shot-2017-02-03-at-08-34-18

The inventory list with all the filters is also available as API Endpoint.

screen-shot-2017-02-03-at-08-39-11

That way you can fetch the data as JSON as well and create your custom Inventory report. You could use this API Endpoint to create a custom inventory PDF report. Check out the VersionEye API.

By the way. Everything on VersionEye.com is 100% open source. The source code is on GitHub and pullrequests are welcome 😉

Pull request support for GitHub

77% of all our signed up users are coming from GitHub and they use VersionEye to monitor their GitHub repositories. For a while now there is a VersionEye service hook in GitHub which calls back the VersionEye API on every git push event and updates the corresponding project in VersionEye with the latest dependencies from GitHub. But now you can use VersionEye to test your changes directly in a Pull Request and see the results of those changes in the GitHub Pull Request UI. This way you can even enforce rules that would prevent a Pull Request merge if a dependency is blacklisted or it is flagged with a security vulnerability.

Given that GitHub is moving away from Service Hooks towards native WebHooks, we changed our integration to do the same. If you are logged in to VersionEye with your GitHub account you can simply click the link “Create from GitHub” and navigate to a repository where you will see all supported project files and the corresponding branches. To monitor a project file simply flip the switch directly next to it.

Screen Shot 2016-08-16 at 14.05.48

In the example above the Gemfile in the develop branch is monitored by VersionEye and it works like this: Every time a new project file is imported from GitHub, VersionEye will automatically create a web hook on that GitHub repository, which will call back the VersionEye API on each “git push” event and if the monitored files are affected, VersionEye will re-parse the files and update the view in VersionEye immediately.

The cool thing is that this also works with pull requests! Assume somebody is forking your project, adding new dependencies and sends back a pull request. In that case the VersionEye web hook kicks in and checks if some of the dependencies have security vulnerabilities or unknown licenses. For those cases, VersionEye will respond to the pull request with a “failed” status message and you can see the result directly on the GitHub Pull-Request tab, like this:

GitHub-pullrequest-VersionEye-check

By clicking on the “Details” link you can see a more specific report at VersionEye for that pull request.

A report like this is generated on every new commit which is added to the existing pull request. It’s similar to a build at TravisCI. Now you can still merge the pull request if you like but at least you are informed about security vulnerabilities and license issues.

In the VersionEye UI you will find a new sub menu “Pull requests” in the organisation view. Here are all commits from all pull requests are listed which belong to your organisation.

Screen Shot 2016-08-16 at 14.22.50

The interesting part here is how VersionEye is combining dynamic and static analysis. Your project is checked for security vulnerabilities and licenses on each commit and each pull request but also regularly independent from that. Even if there are no commits and no pull requests for a couple days, VersionEye will check your project once a day by default and send you an email if a new security vulnerability was found for one of your dependencies.

With this new feature the developers are getting instant feedback to their changes. In real time the dependencies are analysed and the developers informed. That saves a lot of Time & Money. We think it’s important to run this kind of analysis as early in the development process as possible. If your software is already bundled and published to a binary repository it’s too late for this kind of analysis because your software might be distributed already to a high number of devices/clients. Run your analysis as early and as often as you can 😉

And by the way, that’s all open source 😉 You can run the software on your own infrastructure and it works well together with GitHub Enterprise 🙂

This is a pretty new feature. If you find a bug or you have some kind of feedback, simply open a ticket here.

Security Vulnerabilities and Licenses via the VersionEye API

VersionEye can monitor your software dependencies and notify you about out-dated versions. But that’s not all. VersionEye can help you with licenses and security vulnerabilities as well. You can setup a license whitelist for your projects and VersionEye will notify you as soon some of your dependencies are violating your license whitelist. For PHP projects you can even receive notifications about known security vulnerabilities.

Now the license and security vulnerabilities are available via the API as well. With this Endpoint it’s possible to fetch information about an existing project.

The response is a JSON object with an Array of dependencies. Now each element in the dependencies Array has a new variable called “licenses”.

If it is empty that means that VersionEye has no license information about this dependency. Otherwise it contains a list of licenses like this.

“on_whitelist” shows if the license is on the license whitelist or not. If the value is “null”, that means that there is no license whitelist assigned to the project. “on_cwl” stands for on component whitelist. The values are equivalent to the “on_whitelist” field.

Beside that each element in the “dependencies” Array contains a new field “security_vulnerabilities”. That is an Array of known security vulnerabilities to the dependency.

The versioneye-php package is a wrapper around the VersionEye API and it implemented this new features already in the newest version.

Now the VersionEye API is even more powerful. Take advantage of it 😉

API rate limits

Most of the functionalities of VersionEye are available via the API as well. The VersionEye API becomes more and more popular. There are already a lot of plugins and add ons build on top of the VersionEye API. You can find a list of this plugins on the API page at the bottom.

In the last couple months the API responded to more than 17 Million API requests. Because of the growing traffic I introduced some rate limits.

10 calls per hour for non authenticated calls.
50 calls per hour for authenticated calls with an API key.

If you want to have a higher rate limit please write a email to support@versioneye.com with your use case.

NPM Module for VersionEye

Now there is a NPM Module for the VersionEye API. The versioneye-update NPM Module was developed by Onwerk, a Software Service Provider from Mannheim (South Germany). They develop web & mobile applications with Node.JS. Like this interactive Jackpot game, built with iPad, XBox Kinect, Raspberry Pi and NodeJS.

The Onwerk engineers like to stay ahead of cutting edge technology. They want to keep their dependencies up-to-date to get bug & security fixes ASAP into their applications. And of course they want to take advantage of new features as soon as possible.

VersionEye has a very good Integration for GitHub and Bitbucket. If your source code is on one of this cloud SCMs, VersionEye can monitor your package.json directly via the GitHub/Bitbucket API and you get notifications about out-dated dependencies automatically.

But the use case for Onwerk is different. They do BIG Software Projects for LARGE customers and because of NDAs und German privacy laws they are not allowed to give out the source code to anybody else. That’s why they are using the VersionEye API to get notified about out-dated dependencies.

And because they wanted to automate the whole process they developed the versioneye-update NPM module, which gets executed on each build on their private Jenkins CI Server. The process looks like this:

The NPM module versioneye-update is running on each build on the Jenkins. The module is sending the current package.json file to the VersionEye API to update an existing VersionEye project. That way VersionEye nows which dependencies are used in the project right now. VersionEye will compare the version numbers from the package.json file with the newest versions in the VersionEye database to find out-dated dependencies. If there is at least 1 out-dated dependency or at least 1 license violation VersionEye will send out an email notification to the project owner and the project collaborators.

That way the whole process is automated. The engineers don’t have to execute wired commands in the console and they are not in risk to forget something. Beside that the source code stays in house. VersionEye never has access to the source code. The only file which has to be shared with VersionEye is the package.json and that doesn’t get stored on the server! After parsing it once the file object becomes a victim of the garbage collection.

Try the versioneye-update module by yourself and give feedback. Beside this NPM module there are many other Plugins and AddOns build on top of the VersionEye API. Check them out on the API site.

Merge Projects via the VersionEye API

Round about one month ago we released a new feature which supports multiple files in one project. That is useful for almost every modern software project. Some package managers, such as Bundler, Composer, CocoaPods and others, generate anyway 2 files per project. Beside that many modern software projects are splittet up in frontend and backend. And many times there are different technologies and different package managers involved to handle the open source dependencies. That’s why it makes a lot of sense for VersionEye to support multiple files per project.

In the Web Interface you can merge already existing projects into another project. And of course you can “unmerge” them if you like. Now this 2 functions are available via the API as well.

Assume you have 2 projects with each 1 file. A Java project (ID: 222) with a simple pom.xml file for your backend and a JavaScript project (ID: 333) with a bower.json file. But actually it’s the same project and you would like to merge them on VersionnEye to have everything together in one single view. With this call on the API you would merge them:

/api/v2/projects/222/merge/333?api_key=YOUR_API_KEY

In that case the Java project would be the parent and the JavaScript project would be the child. If you merge them the first time the order doesn’t madder. The other way around would work the same way.

If you would like to split the 2 projects again you could do it with this call:

/api/v2/projects/222/unmerge/333?api_key=YOUR_API_KEY

Easy! Right?

There is even a third new endpoint. This one:

This endpoint is specially for the VersionEye Maven Plugin. On a multi module project the VersionEye Maven Plugin gets executed several times and there are multiple lifecycle runs involved. Because of the maven architecture the VersionEye Maven Plugin doesn’t know always the VersionEye ID of the parent project. That’s why this endpoint got introduced, to identify the parent by groupId and artifactId. These 2 coordinates are always available in the Maven lifecycle.

Try out the new Endpoints and leave a comment if you have questions.

VersionEye – Slack

Slack is one of the fastest growing startups in the Bay Area. Slack offers a real-time messaging, archiving and search tool that facilitates better communication within a team.

They just closed another investment round with $120M. Congrats! 🙂

Since last week there is VersionEye – Slack integration. It enables you to get VersionEye notifications directly in Slack.

Many Thanks to Tom Graham for the great work. You can checkout the VersionEye-Slack project here. Feedback is welcome 🙂

VersionEye API Updates

I just updated the code for the GitHub Endpoints.

GET /v2/github/sync

Now “/v2/github/sync” is without “enforce” parameter and it works asynchronously. It returns the status code “running” or “done”.

{ "status": "running" }

POST /v2/github/{repo_key}

“/v2/github/{repo_key}” has a new parameter “file”. In the previous version this Endpoint was only able to import a Gemfile. Now it is possible to define the project file. The “file” parameter is optional. If it’s not specified the default value is “Gemfile”.

DELETE /v2/github/{repo_key}

Now “/v2/github/{repo_key}” has a “file” parameter as well. It works the same way as described for “POST /v2/github/{repo_key}”.

GET /v2/github/search

This Endpoint was removed because it was kind of redundant and out of scope. After “/v2/github/sync” you can filter down your repositories with the “GET /v2/github” Endpoint.

Let me know if you questions or suggestions to this.

VersionEye API private/public

Just added 2 new fields to the project resource at the VersionEye API. This is an example response.

The 2 new fields are “public” and “private_scm”.

“public” can be true or false and basically says if the project on VersionEye is private/public. If it is private, only the owner and the collaborators have read access to it. If it is public, everybody can see the dependencies and licenses of the project. Even random users who are not signed up at VersionEye!

“private_scm” can be true, false or “null” and basically says if the SCM (Source Control Management) of the project is private or public. Possible SCMs are GitHub and Bitbucket. If the project was created via our GitHub/Bitbucket integration, this attributes tells you if the repo behind the project is private/public.