Improved license recognition

VersionEye is monitoring now more than 1.2 Million open source projects and collecting all kind of meta information to this projects. One kind of the meta information is the corresponding license. Currently the VersionEye database contains licenses to more than 8 Million artefacts.

However, sometimes the maintainers of a project didn’t provide the license information that way that it’s easy to read and recognise for machines. That is specially the case for Python and the .NET platform. Take for example the gpkit Python library. In the license field they provided the full license text, not just the license name.

python-gpkit-00

That doesn’t match very well with the license whitelist in VersionEye 😉 That’s why we improved it now!

All together we have 11335 Python licenses like that in our database and they belong to 1989 unique projects on PIP. Our new license crawler could match 9933 licenses to SPDX identifiers. That are 1799 unique projects on PIP we could assign a SPDX identifier to, which didn’t had one before. For more than 90% of this projects we could recognise and identify an SPDX identifier. And now the same library on VersionEye looks like that:

python-gpkit-01

You see clear that it’s MIT license and now this works well together with our license whitelist 🙂

In Nuget, the package manager for the .NET platform, many license names look like this.

csharp-00.png

“Nuget unknown”. That is the case if the maintainers provided a license link but didn’t provide a license name. Our new license crawler is now following this links and with machine learning it tries to identify a known license. If the similarity to a known license text is higher that 90% we assign the corresponding SPDX identifier to the software library in our database. And now the same package looks like this and you can see immediately that it’s the MIT license!

csharp-01.png

That also helps to use these packages together with our license whitelist. For the .NET packages our recognition system was not quiet as good as for Python. For .NET we could only identify 65% of the licenses of packages with a link but without a license name. Stil not bad and much better than before 😉

This are just the first results. Of course this is also work in progress and the recognition will become better in future.

Your feedback is very welcome.

VersionEye Maven Plugin 3.10.2

Today we released the VersionEye Maven Plugin 3.10.2. This patch release is fixing a bug which occurred on big multi module projects, where sometimes the project_id couldn’t be found. Beside that the logging is improved. Now the plugin will log out at which places it is looking for the project_id. That way it’s easier do debug if something goes wrong.

Now VersionEye has support for .NET

VersionEye had already support for 12 package managers and now we are adding Nuget, the package manager for the .NET platform from Microsoft! As Microsoft Windows is all about continuous updating, Microsoft will like VersionEye very much, because VersionEye is continuous updating for .NET packages 🙂

With fixed releases and snapshots together VersionEye is currently monitoring more than 70K open source .NET packages on Nuget. Here are the numbers from .NET / Nuget compared with package managers from other languages.

Screen Shot 2016-07-26 at 15.23.36

The above chart can be viewed on the VersionEye statistic page and is updated once a week. The numbers representing projects in official package mangers. Random OS projects on GitHub/Bitbucket are not taken into the calculation! Currently the C# .NET community has less open source packages than the Python, Ruby, PHP, Java or Node.JS community. As Microsoft changed their open source strategy a couple months ago, this might change in future. But this is the status quo!

Every .NET package has his own page at VersionEye, like this one for example.

Screen Shot 2016-07-26 at 14.20.57

Now you can follow any .NET package at VersionEye and you will receive an email notification as soon a new release for that package is available. But that’s not all. VersionEye is showing the most important KPIs to the package in the right upper corner. KPIs like average release time, external references, the license of the package and a dependency badge which can be integrated into README pages at GitHub, GitLab and Bitbucket.

VersionEye is also creating author profiles for the contributors of the open source libraries. By clicking on an author icon on a package detail page the author page shows up. Like this one for example.

Screen Shot 2016-07-26 at 14.48.58

On the author page you can see to which other .NET packages the author contributed to and what his/her keywords are. The keywords are clickable as well. This is for example the keyword page for JSON.

Screen Shot 2016-07-26 at 15.35.56

The keyword page can be filtered down by language and now C# is on the filter as well.

But the coolest feature is that VersionEye can monitor a Nuget project in your Git repository and notify you about out-dated dependencies and license violations. Now VersionEye can monitor project.json and *.nuspec files on GitHub, Bitbucket and Stash.

Screen Shot 2016-07-26 at 15.41.34

For monitoring one of this files, simply flip the switch in our Git integration view. The project report looks like that.

Screen Shot 2016-07-26 at 15.42.30

It’s that easy to get notified about out-dated dependencies and license violations 🙂 Projects like this one can be created and update via the VersionEye API as well. There already many native VersionEye plugins for build tools and package managers like Maven, SBT, Gradle and NPM which are using the VersionEye API. I’m looking forward to see some native VersionEye plugins for the .NET platform as well 🙂

TODOs

2 Things are still not finished for the .NET integration and are still ongoing.

  1. Release Dates: On some packages you will see “Release 116 years ago”. Of course that is wrong 🙂 On some packages the release date was not set correctly. Currently a background job is running which is fetching the release dates from an alternative source.
  2. License Names: The Nuget API doesn’t expose license names, only links to licenses. Currently we are working on it to resolve the links to meaning full names. For a couple thousand packages this is already done but for the majority this process is still ongoing.

Feedback

The .NET integration in VersionEye is pretty fresh. We rely on your feedback! Please test it and let us know if something is wrong. You can reach us on Twitter or simply open a ticket at GitHub. By the way. VersionEye itself is open source. Pull requests are welcome 😉

Open Ticket System for VersionEye

We at VersionEye are always interested into your Feedback! Right from the beginning the feedback icon was part of the main menu at VersionEye.

Screen Shot 2015-02-12 at 10.34.33

By clicking on the feedback icon a modal dialog appeared where you could send a message to the VersionEye Team. We received your feedback as email and it was not visible to anybody else. Many times we received a bug/feature report multiple times. Simply because many users found the same issue.

A couple days ago we changed the behaviour of the feedback icon. Instead of opening a modal dialog it redirects the user to the ticket system on Bitbucket.

Screen Shot 2015-02-12 at 10.37.10

Everybody can sign up for free at Bitbucket and open a ticket for VersionEye. The tickets are visible for everybody! Even for users who are not singed up a Bitbucket!

Why Bitbucket? Why not GitHub?

The code for VersionEye is splittet up in over 10 git repositories. For long time we hosted everything on GitHub and we used internal the GitHub ticket system for ourselves. There are 2 reasons why we moved some repositories to Bitbucket.

  • On Bitbucket you can make the ticket system public and at the same time keep the source code private.
  • The Ticket system on Bitbucket is a little bit more advanced. Specially the “vote” feature is very useful for communities. That way users can vote for a ticket and we can sort the tickets by votes. That is very useful for new feature requests. Now we don’t have to guess anymore which feature to implement first. We simply implement the features first with the most votes!

Screen Shot 2015-02-12 at 11.38.42

We still have code on GitHub and we are still paying customers of GitHub. Simply because the performance of GitHub is better. A ‘git push’ or ‘git checkout’ works on GitHub always a couple seconds faster than on Bitbucket.  With git you can have multiple backends without a problem 🙂 That’s why we all love git!

We hope you will embrace our new open ticket system and we will see many features requests and votes in future 😉

Projects with multi file support

In modern software development it is not uncommon that you use more than 1 package manager for a software project. Maybe you are using Ruby for the backend and JavaScript for the front end. In that case you would use 2 package managers. Bundler/Rubygems for the backend and maybe bower for the front end. You would have at last 3 files for describing your dependencies. A Gemfile and a Gemfile.lock for your backend and a bower.json file for your front end.

Up to now a project on VersionEye was always a representation of 1 single file. For example the Gemfile. So if you wanted to monitor all 3 files you had to have 3 projects on VersionEye. Now a project on VersionEye can have multiple files!

Screen Shot 2014-12-23 at 11.21.47
If you click on a Gemfile in our GitHub integration, VersionEye will check if there is a corresponding Gemfile.lock in the same directory. If so, it will parse that Gemfile.lock as well and create a new project with 2 files.

VersionEye-Multi
In the dependency tab, the corresponding files are displayed above the dependency table. In this example the Gemfile is selected and we see the dependencies of the Gemfile. By clicking on the Gemfile.lock we can switch to the dependencies of the Gemfile.lock. It’s all in the same project view.

The overall numbers in the head of the page are summed up from all files in the project. That means over the 2 files there are 58 unique dependencies and 33 of them are out-dated. Also the dependency badge is summed up over all files. The dependency badge only turns green if ALL dependencies in ALL files are up-to-date.

But that’s not all. How do we get the bower.json file into this project? Let’s assume the bower.json file is already a seperate project on VersionEye. Now we can merge projects into each other. This is how it works. Let’s navigate to the desired project we would like to merge into our Ruby project. Navigate to the “settings” tab. Here there is a new input field which looks like this:

Screen Shot 2014-12-23 at 12.13.53
Here we can pick a “parent” project to merge in. By clicking on the merge button the current project will be merged as subproject into the selected project.

VersionEye-Multi-3Now we have a project with 3 files. Gemfile, Gemfile.lock and a bower.json file. The overall numbers are updated. Overall 3 files we have 59 dependencies and 33 are out-dated.

Under the file names there is a link “unmerge bower.json”. By clicking on that link the bower.json can be unmerged from this project. That means it will be removed from this project and be again a seperate project.

Pretty cool! Right? This is just the first step. In the next update we will bring this feature to the API and to the plugins as well 😉

Let us know what you think about this feature, either here in the comments or on Twitter.