VersionEye can monitor your project file (Gemfile, package.json and so on) on GitHub/Bitbucket and notify you about out-dated dependencies, license violations and security vulnerabilities.
The process of fetching a file from the GitHub API is very time consuming and many things can go wrong. Doing this inside of a HTTP request/response cycle would mean to block the web process for several seconds or even minutes. That’s why this job is done by background workers. In the project detail view a “re parse” job can be triggered by hitting the “Re Parse Now” button. Until the background job is running a loading bar is displayed like on this image.
If the job is done the page reloads.
By dealing with external APIs many things can go wrong. Maybe the API is not reachable or there is another error and the project file can not be parsed. The background worker is throwing some Exceptions, but they never show up in the UI because the workers are running on completely different machines. That problem is solved now!
Here is a drawing of the architecture. Every square on the image is a different machine with his own IP address. They can even be in different geographical regions.
VersionEye is using heavily RabbitMQ to distribute work. If somebody hits the “Re Parse Now” button, the request goes to a Rails controller of one of the web application servers. The Rails controller sends a message with the project id to RabbitMQ and returns immediately the response to the Browser. That takes only a couple milli seconds and after the response is processed the user can see that the background job is running. See picture 1 for that.
The RabbitMQ server can receive messages from web app servers, but also from a scheduler, which is triggering re parse jobs for projects with daily, weekly and monthly period.
There are several workers who subscribed to a specific channel on RabbitMQ. Some workers are responsible for fetching a users list of repositories from the GitHub API. Some workers know how to search for “supported project files” in a given git repository. And other workers are responsible for fetching a given project file from the GitHub API, parsing it and updating an existing project at VersionEye with the re parsed dependencies.
Every worker is specialised in doing exactly 1 job. The code for a worker is usually less then 100 lines. You can think of it as micro services.
Assume worker1 will get the job which just was triggered by the user. The worker tries to fetch the file from the GitHub API but something goes wrong. The GitHub API returns a error code and the project file can not be fetched. Some Exceptions are thrown and logged somewhere in some log files. Normally the worker would response to RabbitMQ that the job is done. But now the worker is storing error messages in the project object itself. That is a known pattern from ActiveRecord. If you try to persist an object with ActiveRecord and the operation fails it stores the error messages inside of the object. The error messages can be displayed with this command:
.errors.full_messages.to_sentence
The workers at VersionEye are doing it similar. The workers are storing error messages in the “parsing_errors” array of the project model. If this array is not empty, the elements are displayed in the project detail view, like here for example.
In the project overview page the projects with errors are marked with a “warn” sign. Like the first project in this example.
Now the users are getting feedback from the background workers. Possible reasons for this errors are:
- The given URL doesn’t exist anymore.
- The project file was moved.
- The git repository was renamed.
- The credentials for the GitHub/Bitbucket API are not valid anymore.
The most common reasons are the last 2 ones. If the git repository was renamed the VersionEye project has be removed and created newly again. If the credentials for the GitHub/Bitbucket API are not valid anymore then it can be fixed by going to settings and disconnecting from GitHub/Bitbucket and then connecting again to GitHub/Bitbucket. That they VersionEye is getting new API tokens for the logged in user.
This feature is quiet new, just a couple hours old. Feedback is welcome 😉
Open Source License Compliance & Security 