Improved Mute Feature

VersionEye shows you which of your project dependencies are out-dated. Sometimes you have good reasons not to update. Sometimes you want to stick to an out-dated version, because the newest version is buggy or insecure. On the other side you don’t want to get the notification emails from VersionEye every day for that library. You know that it’s out-dated and you have your reasons to stick to it. In that case you can “mute” that specific artefact.

The mute feature was improved again. Until now the “mute” icon was in the left column of the table beside the dependency name. That was a bit confusing because the “mute” icon has nothing to do with the name of the dependency. It is related to the newest version. That’s why the icon moved into the newest column, before the “newest” version string.

dep_icon

Now you can read it like “mute 2.7.1″.

But that’s not all. If you click on the mute icon a modal dialog will show up. Like this here.

MuteDialog

Here you can type in a reason why you want to mute this specific version of the dependency. The text input field is not mandatory. You can leave it empty if you want. But if you leave a message here your collaborators can see why you muted this artefact, by simply hover over the “mute” icon.

MuteMessage

Beside that in the project header you will see the number of muted dependencies.

Screen Shot 2015-07-22 at 17.14.07

Let me know how you like the new mute feature or if you have any suggestions.

Displaying errors from the backend systems in the UI

VersionEye can monitor your project file (Gemfile, package.json and so on) on GitHub/Bitbucket and notify you about out-dated dependencies, license violations and security vulnerabilities.

The process of fetching a file from the GitHub API is very time consuming and many things can go wrong. Doing this inside of a HTTP request/response cycle would mean to block the web process for several seconds or even minutes. That’s why this job is done by background workers. In the project detail view a “re parse” job can be triggered by hitting the “Re Parse Now” button. Until the background job is running a loading bar is displayed like on this image.

Screen Shot 2015-06-07 at 08.22.18

If the job is done the page reloads.

By dealing with external APIs many things can go wrong. Maybe the API is not reachable or there is another error and the project file can not be parsed. The background worker is throwing some Exceptions, but they never show up in the UI because the workers are running on completely different machines. That problem is solved now!

Here is a drawing of the architecture. Every square on the image is a different machine with his own IP address. They can even be in different geographical regions.

RabbitMQ-Architecture

VersionEye is using heavily RabbitMQ to distribute work. If somebody hits the “Re Parse Now” button, the request goes to a Rails controller of one of the web application servers. The Rails controller sends a message with the project id to RabbitMQ and returns immediately the response to the Browser. That takes only a couple milli seconds and after the response is processed the user can see that the background job is running. See picture 1 for that.

The RabbitMQ server can receive messages from web app servers, but also from a scheduler, which is triggering re parse jobs for projects with daily, weekly and monthly period.

There are several workers who subscribed to a specific channel on RabbitMQ. Some workers are responsible for fetching a users list of repositories from the GitHub API. Some workers know how to search for “supported project files” in a given git repository. And other workers are responsible for fetching a given project file from the GitHub API, parsing it and updating an existing project at VersionEye with the re parsed dependencies.

Every worker is specialised in doing exactly 1 job. The code for a worker is usually less then 100 lines. You can think of it as micro services.

Assume worker1 will get the job which just was triggered by the user. The worker tries to fetch the file from the GitHub API but something goes wrong. The GitHub API returns a error code and the project file can not be fetched. Some Exceptions are thrown and logged somewhere in some log files. Normally the worker would response to RabbitMQ that the job is done. But now the worker is storing error messages in the project object itself. That is a known pattern from ActiveRecord. If you try to persist an object with ActiveRecord and the operation fails it stores the error messages inside of the object. The error messages can be displayed with this command:

.errors.full_messages.to_sentence

The workers at VersionEye are doing it similar. The workers are storing error messages in the “parsing_errors” array of the project model. If this array is not empty, the elements are displayed in the project detail view, like here for example.

Screen Shot 2015-07-14 at 17.56.43

In the project overview page the projects with errors are marked with a “warn” sign. Like the first project in this example.

Screen Shot 2015-07-14 at 17.56.19

Now the users are getting feedback from the background workers. Possible reasons for this errors are:

  • The given URL doesn’t exist anymore.
  • The project file was moved.
  • The git repository was renamed.
  • The credentials for the GitHub/Bitbucket API are not valid anymore.

The most common reasons are the last 2 ones. If the git repository was renamed the VersionEye project has be removed and created newly again. If the credentials for the GitHub/Bitbucket API are not valid anymore then it can be fixed by going to settings and disconnecting from GitHub/Bitbucket and then connecting again to GitHub/Bitbucket. That they VersionEye is getting new API tokens for the logged in user.

This feature is quiet new, just a couple hours old. Feedback is welcome ;-)

New search results for VersionEye

The search results on VersionEye just have changed. Now the license information and the latest release time are displayed directly in the search results. Until now this information was only available in the detail view.

Assume you are a mobile developer and you need a library for QR codes. Let’s type in “QR” into the search field and filter the results for Objective-C libraries. Here are the search results:

Screen Shot 2015-07-09 at 17.49.35

You can see the licenses immediately. And you can see that 2 results are marked red because their newest version is older then 1 year. The risk that they are not maintained actively anymore is high in that cases. QRCoder and QRCodeReader have been released in the last 30 days. It seems that they are actively maintained projects. I would check them out.

Here is another Example. Assume you are Java developer and you are searching for “log4J”. Now the results look like that:

Screen Shot 2015-07-09 at 17.50.16

Now you see immediately that the first result for log4j is an old library because the newest version of the library was release more than 3 years ago. That’s why it’s marked red. The 2nd result of log4j is the current brach and the newest version on that was released 2 months ago. You can also see immediately the licenses of all libraries, without going into a detail page.

The new search results should help you to see the relevant information more quickly.

Responsive Design for VersionEye

Maybe you noticed it already. VersionEye changed the CSS implementation and is responsive now. It was a switch from Boostrap 2 to Bootstrap 3.3.5. Now the layout is floating around depending on the viewport of the device and on mobile you will see the typical burger menu in the right upper corner.

IMG_1208

Specially the search results and the product detail pages have been optimised for mobile.

IMG_1209 IMG_1210 IMG_1211

And the user profile pages as well.

IMG_1213 IMG_1212

It was a lot of work migrating from Bootstrap 2 to Bootstrap 3.3.5, but it was worth it. The pages are looking much cleaner now and they are usable on mobile. If something looks odd please open a ticket here.

Another lesson learned with MongoDB

MongoDB is the primary database at VersionEye. Currently VersionEye is crawling more than 600K open source projects on a daily basis. Some of the crawlers are implemented in Java, others in Ruby. You can follow a library at VersionEye and as soon the next version comes out you get a email notification. Today I got this email from VersionEye.

Screen Shot 2015-06-25 at 18.20.59

As you can see the version information is missing for the Java libraries. The email template was not touched in the last couple days. Obviously the crawlers for Maven repositories are implemented in Java :-) and they get updated more frequently. The error must be somewhere in the Java crawlers.

The version object is an embedded document in the product object. Every time a crawler finds a new version it adds a new version object to the corresponding product object. The code for that looks like that.

BasicDBObject productMatch = new BasicDBObject();
productMatch.put(Product.LANGUAGE, language);
productMatch.put(Product.PROD_KEY, prodKey);

BasicDBObject versionObj = version.getDBObject();
versionObj.put(Version.VERSION, version.getVersion());

BasicDBObject versionsUpdate = new BasicDBObject();
versionsUpdate.put("$push", new BasicDBObject(Version.VERSIONS, versionObj));

So far so good. In the next lines the product object is updated with the current time.

DBObject verUpdate = getDBObjectByKey(language, prodKey);
verUpdate.put(Product.UPDATED_AT, new Date());
getCollection().update(productMatch, verUpdate);

And of course there is a unit test case for this code and the test case is always green. On production sometimes the new version just disappears. Not always! Just sometimes. At first I thought I found a bug in MongoDB, but this only happened to the Java crawlers, never to the Ruby crawlers. So the root of all eval must be the implementation. I needed a whole day to figure it out!

On production MongoDB is running in a Replica Set on multiple hosts and 2 days ago I changed the read preference of the mongodb driver to “secondary”. That means that the read operations are distributed to ALL nodes in the Replica Set. And this is what happened.

The first code snippet always runs through and adds a new version to the product. But then the 2nd code snippet is reloading the product object from the db and executing an update.

DBObject verUpdate = getDBObjectByKey(language, prodKey);
verUpdate.put(Product.UPDATED_AT, new Date());
getCollection().update(productMatch, verUpdate);

If the changes are not yet distributed in the whole Replica Set and the read operation goes to a node which doesn’t has the new version yet, a product object is loaded without the new version. On this object the “updated_at” field is updated and stored back to the database. But the “update” method on the java driver doesn’t update only the changed field, it updates the whole object. And so it comes that it stores the object without the new version.

There are different solutions to this. First of all I could change the read preference back to “primary” again. But there is a better solution. Actually there is a way to only update single properties in a document in MongoDB. That works like this.

DBObjectnewValues = getDBObjectByKey(language, prodKey);
newValues.put(Product.UPDATED_AT, new Date());
BasicDBObject set = new BasicDBObject("$set", newValues);
getCollection().update(productMatch, set);

The big difference is in line 3. That tells the java driver to only update the changed properties. On day headache for a one liner! I hope I can save somebody else 1 day headache with this blog post.

VersionEye Maven Plugin 3.5.0

Version 3.5.0 of the VersionEye Maven Plugin is out. It offers another configuration option to skip certain dependencies with a defined scope.

By default the VersionEye Maven Plugin is resolving ALL dependencies and sending ALL dependencies to the VersionEye API. The VersionEye Server is then checking the dependencies and is sending out emails. But sometimes you get notifications for a dependency which is under test scope or provided scope. Something you don’t really ship with your application. Ideally you want VersionEye to ignore those dependencies. Now you can configure which scopes should be skipped by the plugin. Simply add this line to your plugin configuration.

<skipScopes>test,provided</skipScopes>

The line above will ignore all dependencies which have test or provided scope.

VersionEye Update

VersionEye can monitor your project on GitHub/Bitbucket and notify you about out-dated dependencies and license violations. The integration via the GitHub/Bitbucket API works very well and is very convenient.

However. Not everybody is using GitHub/Bitbucket. Through the VersionEye API you can create/update projects as well and take advantage of the VersionEye infrastructure. Assume you created already a project through file upload or via the URL feature. Now you want to update the project every day automatically with your current project file to get the notifications. For that purpose there is a new open source project, versioneye_update at GitHub.  It’s a very simple shell script which is using CURL to upload a project file to the VersionEye API. In the first lines of the script the variables need to be configured.

#!/bin/bash 

VERSIONEYE_SERVER=https://www.versioneye.com
API_KEY=<YOUR_SECRET_API_KEY> 
PROJECT_ID=<YOUR_PROJECT_ID>

If that is done, it’s dead easy to update an existing project. Simply run:

./update.sh <PROJECT_FILE>

For example:

./update.sh composer.lock

To update an existing project with your current composer.lock file. The script will output the number the dependencies, number of out-dated dependencies and the number of license violations for the case you have a license whitelist assigned to the project at VersionEye.

update-sh

This project is meant to be executed on a Continuous Integration Server. Ideally you update your VersionEye project with the current project file on each build.

Let me know how this works for you. You are also welcome to open a ticket on the repo or a to send a pull request with improvements ;-)