Category: Package Manager

Rust Support

Rust is a blazingly fast new language which guarantees thread safety! Our latest project, a command line tool to identify random components by their SHA values, is implemented in Rust. As we use Rust for our own purposes we want to track Rust dependencies of course 🙂 That’s why added support for Rust now! VersionEye is tracking more than 9K Rust packages from crates.io now and they are all available through our public search.

Screen Shot 2017-05-04 at 08.59.58

Beside that VersionEye can parse cargo.toml and cargo.lock files. If you are using cargo to manage your Rust dependencies then you can let VersionEye monitore those files to receive notifications about new releases of your dependencies.

Screen Shot 2017-05-04 at 09.09.45

The above screenshot shows a Rust project which is monitored directly on GitHub.

This integration is still very fresh. Please try it out and give us feedback!

Improved license recognition

VersionEye is monitoring now more than 1.2 Million open source projects and collecting all kind of meta information to this projects. One kind of the meta information is the corresponding license. Currently the VersionEye database contains licenses to more than 8 Million artefacts.

However, sometimes the maintainers of a project didn’t provide the license information that way that it’s easy to read and recognise for machines. That is specially the case for Python and the .NET platform. Take for example the gpkit Python library. In the license field they provided the full license text, not just the license name.

python-gpkit-00

That doesn’t match very well with the license whitelist in VersionEye 😉 That’s why we improved it now!

All together we have 11335 Python licenses like that in our database and they belong to 1989 unique projects on PIP. Our new license crawler could match 9933 licenses to SPDX identifiers. That are 1799 unique projects on PIP we could assign a SPDX identifier to, which didn’t had one before. For more than 90% of this projects we could recognise and identify an SPDX identifier. And now the same library on VersionEye looks like that:

python-gpkit-01

You see clear that it’s MIT license and now this works well together with our license whitelist 🙂

In Nuget, the package manager for the .NET platform, many license names look like this.

csharp-00.png

“Nuget unknown”. That is the case if the maintainers provided a license link but didn’t provide a license name. Our new license crawler is now following this links and with machine learning it tries to identify a known license. If the similarity to a known license text is higher that 90% we assign the corresponding SPDX identifier to the software library in our database. And now the same package looks like this and you can see immediately that it’s the MIT license!

csharp-01.png

That also helps to use these packages together with our license whitelist. For the .NET packages our recognition system was not quiet as good as for Python. For .NET we could only identify 65% of the licenses of packages with a link but without a license name. Stil not bad and much better than before 😉

This are just the first results. Of course this is also work in progress and the recognition will become better in future.

Your feedback is very welcome.

VersionEye Maven Plugin 3.10.2

Today we released the VersionEye Maven Plugin 3.10.2. This patch release is fixing a bug which occurred on big multi module projects, where sometimes the project_id couldn’t be found. Beside that the logging is improved. Now the plugin will log out at which places it is looking for the project_id. That way it’s easier do debug if something goes wrong.

Now VersionEye has support for .NET

VersionEye had already support for 12 package managers and now we are adding Nuget, the package manager for the .NET platform from Microsoft! As Microsoft Windows is all about continuous updating, Microsoft will like VersionEye very much, because VersionEye is continuous updating for .NET packages 🙂

With fixed releases and snapshots together VersionEye is currently monitoring more than 70K open source .NET packages on Nuget. Here are the numbers from .NET / Nuget compared with package managers from other languages.

Screen Shot 2016-07-26 at 15.23.36

The above chart can be viewed on the VersionEye statistic page and is updated once a week. The numbers representing projects in official package mangers. Random OS projects on GitHub/Bitbucket are not taken into the calculation! Currently the C# .NET community has less open source packages than the Python, Ruby, PHP, Java or Node.JS community. As Microsoft changed their open source strategy a couple months ago, this might change in future. But this is the status quo!

Every .NET package has his own page at VersionEye, like this one for example.

Screen Shot 2016-07-26 at 14.20.57

Now you can follow any .NET package at VersionEye and you will receive an email notification as soon a new release for that package is available. But that’s not all. VersionEye is showing the most important KPIs to the package in the right upper corner. KPIs like average release time, external references, the license of the package and a dependency badge which can be integrated into README pages at GitHub, GitLab and Bitbucket.

VersionEye is also creating author profiles for the contributors of the open source libraries. By clicking on an author icon on a package detail page the author page shows up. Like this one for example.

Screen Shot 2016-07-26 at 14.48.58

On the author page you can see to which other .NET packages the author contributed to and what his/her keywords are. The keywords are clickable as well. This is for example the keyword page for JSON.

Screen Shot 2016-07-26 at 15.35.56

The keyword page can be filtered down by language and now C# is on the filter as well.

But the coolest feature is that VersionEye can monitor a Nuget project in your Git repository and notify you about out-dated dependencies and license violations. Now VersionEye can monitor project.json and *.nuspec files on GitHub, Bitbucket and Stash.

Screen Shot 2016-07-26 at 15.41.34

For monitoring one of this files, simply flip the switch in our Git integration view. The project report looks like that.

Screen Shot 2016-07-26 at 15.42.30

It’s that easy to get notified about out-dated dependencies and license violations 🙂 Projects like this one can be created and update via the VersionEye API as well. There already many native VersionEye plugins for build tools and package managers like Maven, SBT, Gradle and NPM which are using the VersionEye API. I’m looking forward to see some native VersionEye plugins for the .NET platform as well 🙂

TODOs

2 Things are still not finished for the .NET integration and are still ongoing.

  1. Release Dates: On some packages you will see “Release 116 years ago”. Of course that is wrong 🙂 On some packages the release date was not set correctly. Currently a background job is running which is fetching the release dates from an alternative source.
  2. License Names: The Nuget API doesn’t expose license names, only links to licenses. Currently we are working on it to resolve the links to meaning full names. For a couple thousand packages this is already done but for the majority this process is still ongoing.

Feedback

The .NET integration in VersionEye is pretty fresh. We rely on your feedback! Please test it and let us know if something is wrong. You can reach us on Twitter or simply open a ticket at GitHub. By the way. VersionEye itself is open source. Pull requests are welcome 😉

Open Ticket System for VersionEye

We at VersionEye are always interested into your Feedback! Right from the beginning the feedback icon was part of the main menu at VersionEye.

Screen Shot 2015-02-12 at 10.34.33

By clicking on the feedback icon a modal dialog appeared where you could send a message to the VersionEye Team. We received your feedback as email and it was not visible to anybody else. Many times we received a bug/feature report multiple times. Simply because many users found the same issue.

A couple days ago we changed the behaviour of the feedback icon. Instead of opening a modal dialog it redirects the user to the ticket system on Bitbucket.

Screen Shot 2015-02-12 at 10.37.10

Everybody can sign up for free at Bitbucket and open a ticket for VersionEye. The tickets are visible for everybody! Even for users who are not singed up a Bitbucket!

Why Bitbucket? Why not GitHub?

The code for VersionEye is splittet up in over 10 git repositories. For long time we hosted everything on GitHub and we used internal the GitHub ticket system for ourselves. There are 2 reasons why we moved some repositories to Bitbucket.

  • On Bitbucket you can make the ticket system public and at the same time keep the source code private.
  • The Ticket system on Bitbucket is a little bit more advanced. Specially the “vote” feature is very useful for communities. That way users can vote for a ticket and we can sort the tickets by votes. That is very useful for new feature requests. Now we don’t have to guess anymore which feature to implement first. We simply implement the features first with the most votes!

Screen Shot 2015-02-12 at 11.38.42

We still have code on GitHub and we are still paying customers of GitHub. Simply because the performance of GitHub is better. A ‘git push’ or ‘git checkout’ works on GitHub always a couple seconds faster than on Bitbucket.  With git you can have multiple backends without a problem 🙂 That’s why we all love git!

We hope you will embrace our new open ticket system and we will see many features requests and votes in future 😉

Projects with multi file support

In modern software development it is not uncommon that you use more than 1 package manager for a software project. Maybe you are using Ruby for the backend and JavaScript for the front end. In that case you would use 2 package managers. Bundler/Rubygems for the backend and maybe bower for the front end. You would have at last 3 files for describing your dependencies. A Gemfile and a Gemfile.lock for your backend and a bower.json file for your front end.

Up to now a project on VersionEye was always a representation of 1 single file. For example the Gemfile. So if you wanted to monitor all 3 files you had to have 3 projects on VersionEye. Now a project on VersionEye can have multiple files!

Screen Shot 2014-12-23 at 11.21.47
If you click on a Gemfile in our GitHub integration, VersionEye will check if there is a corresponding Gemfile.lock in the same directory. If so, it will parse that Gemfile.lock as well and create a new project with 2 files.

VersionEye-Multi
In the dependency tab, the corresponding files are displayed above the dependency table. In this example the Gemfile is selected and we see the dependencies of the Gemfile. By clicking on the Gemfile.lock we can switch to the dependencies of the Gemfile.lock. It’s all in the same project view.

The overall numbers in the head of the page are summed up from all files in the project. That means over the 2 files there are 58 unique dependencies and 33 of them are out-dated. Also the dependency badge is summed up over all files. The dependency badge only turns green if ALL dependencies in ALL files are up-to-date.

But that’s not all. How do we get the bower.json file into this project? Let’s assume the bower.json file is already a seperate project on VersionEye. Now we can merge projects into each other. This is how it works. Let’s navigate to the desired project we would like to merge into our Ruby project. Navigate to the “settings” tab. Here there is a new input field which looks like this:

Screen Shot 2014-12-23 at 12.13.53
Here we can pick a “parent” project to merge in. By clicking on the merge button the current project will be merged as subproject into the selected project.

VersionEye-Multi-3Now we have a project with 3 files. Gemfile, Gemfile.lock and a bower.json file. The overall numbers are updated. Overall 3 files we have 59 dependencies and 33 are out-dated.

Under the file names there is a link “unmerge bower.json”. By clicking on that link the bower.json can be unmerged from this project. That means it will be removed from this project and be again a seperate project.

Pretty cool! Right? This is just the first step. In the next update we will bring this feature to the API and to the plugins as well 😉

Let us know what you think about this feature, either here in the comments or on Twitter.

Improved Support for Gradle

Currently VersionEye supports 9 Package Managers. One of them is Gradle, the build system written in Groovy!

gradle-icon-512x512

Gradle is becoming more and more popular and we are getting more and more emails to our Gradle support. If you work with Gradle you define your dependencies usually in a “build.gradle” file. A dependency definition can look like this:

compile group: 'org.apache.pdfbox', name: 'pdfbox', version: '1.8.5'

This definition works since longer with VersionEye. What didn’t worked was for example this:

//    compile group: 'com.artofsolving', name:'jodconverter', version:'2.2.1'

If you commented out a line in your gradle file VersionEye parsed it anyway. Since today this is fixed!

Another cool feature in Gradle is that you can define variables. For example to hold version numbers. Here are some variable definitions:

project.ext.hibernateCoreVersion = '4.1.9.Final'
project.ext.jerseyVersion = '1.17'
project.ext.springDataJpaVersion = '1.4.1.RELEASE'

And here is another way to define dependencies as Arrays:

[group: 'org.hibernate', name: 'hibernate-core', version: hibernateCoreVersion],
[group: 'org.hibernate', name: 'hibernate-entitymanager', version: hibernateCoreVersion],

In the above example you can see that instead of using a static version number, we use the variable we defined before. This variable declaration and defining dependencies in an Array is now supported by VersionEye as well!

Gradle VersionEye Plugin

Beside the improved support for Gradle files we recommend to use the Gradle VersionEye Plugin. It resolves the dependencies locally and is well suited for more complex Gradle build files, for example with submodules. The plugin sends the locally resolved dependencies to the VersionEye API to create/update a project on VersionEye. Our Enterprise clients are using this (Or the Maven) Plugin to keep their projects up-to-date. 

You can run the plugin every night on your CI System to update the corresponding VersionEye project with the current dependencies. VersionEye will check the dependencies and send out notification emails if some dependencies are out-dated. 

Let me know if you have questions to this. Either here in the comments or on Twitter. 

Problems with Package Managers – Bower Edition

Are you a front-end developer, working every day with HTML, CSS and JavaScript? Then you have probably heard already about Bower, the package manager for frontend JavaScript. You can install the client via NPM. Bower is based on Node.JS but doesn’t use NPM as registry, it’s a completely separated project with it’s own registry. You can use Bower as a Ruby, Python or PHP guy as well. Installing a package works like this:

bower install backbone#1.1.0

Which will download backbone version 1.1.0 and his dependencies into the directory ./bower_components/backbone/. All Bower packages are downloaded into ./bower_components.

Bower is handling versions and dependencies but It does NOT wire the JS components into your project. And that’s OK! It doesn’t force you to a specific directory structure. That’s all up to you.

Last week we finished the bower integration for VersionEye.  More than 5K JavaScript packages are now available through our search, powered by ElasticSearch and can now easily be followed. If you want to get notified about a new version of your favorite JavaScript package, simply follow it on VersionEye and you will receive an email notification on the day they release the new tag/version.

During our integration work we had to crawl the bower registry and GitHub to get the meta data in our MongoDB / ElasticSearch backend. Here are some interesting numbers I would like to share with you.

All registered bower packages are available through this JSON file:

https://bower.herokuapp.com/packages

Each element in the array contains the name of a bower package and the URL to his repository. At this time the list has 8504 entries. Only 47 URLs don’t link to GitHub.

bower_github

Some of the 47 URLs link to Bitbucket and funny entries like 192.168.0.131 🙂

We consider a bower package valid if both of these conditions are true:

  • The URL still exists and is publicly available.
  • The repository contains a valid bower.json or component.json file.

From the 8504 packages 2515 we couldn’t validate. That means either the repository doesn’t exist anymore or we couldn’t find any bower.json file or the the bower.json file was not valid JSON. 5989 packages had valid bower.json files.

bower_valid

More than 900 registered repositories simply don’t provide a bower.json file. Other ones provide a bower.json file but it contains errors. The most common error is that there is a comma to much or to less in an array definition.

From the 5989 packages we could validate, 1194 don’t have any version tags in the associated git repository. 4795 packages provide at least one tag.

bower_tags

Having a package registered on a package manager without fixed version/tag is a bit useless. From all 8504 registered packages only 4795 offer a valid bower.json file and some tags on their repositories.

bower_comp_valid

Specification

The bower specification describes exactly how a valid package has to look like. One of the points for the name is: “Lowercase, a-z, can contain dash or dot but not start/end with them“. But actually 495 packages do have uppercase characters in their name. That alone is not dramatic. But some packages are registered with the same name. For example the “scroller” package. It is registered as “scroller” and “Scroller”. Two different GitHub repositories with almost the same name. The only differentiator is the lowercase/uppercase S in the beginning. We were able to find 34 names where collisions occur if you force lowercase names.

Validation

There is a very simple reason for this numbers. There is currently almost no validation on the submissions. But user authentication and package validation is already on the todo list of the Bower team. I expect many improvements in the next months. However, launching a package manager without any kind of validation was maybe not the best idea.

Conclusion

I don’t wrote this article to keep you away from Bower. The people behind the bower project are smart guys and very responsive on Twitter. They are continuously improving. And by the way, it’s all open source. Everybody can send a pull-request to improve the validation. I already send a pull-request for the name validation.

Having some kind of versioning for frontend development is very important. Otherwise you just download some CSS and JS files from the Internet and put them into your project. After a couple weeks most likely you don’t even remember which version of backbone your are using. Imagine doing that with your ruby gems or python libraries! Developers should take versioning of their frontend code as seriously as they do for their backend development.

If you are a Ruby on Rails developer you should check out this page https://rails-assets.org/ for rails – bower integration.

If you wanna know more about how to monitor a bower.json file or how to get a dependency badge for your bower project you should check out this blog post about our Bower integration.

What are your experiences with Bower? Leave a comment here or contact me on Twitter. I would like to hear your opinion to this topic.