Security Alerts for Ruby Gems

Since a couple weeks VersionEye shows security issues for PHP projects. Now this feature works the same way for NodeJS and Ruby packages. If VersionEye is monitoring a Gemfile for you, then you will see the “Security” tab in the project view. Just like here in this example.

VersionEye-Ruby-Security

In the “Security” tab all known security vulnerabilities are listed for your 3rd party dependencies. If there is a security issue the dependency badge turns red! By clicking on the package name the package detail page comes up with a more detailed description of the security vulnerability.

VersionEye-Ruby-Security_2

On the detail the page a detailed description of the security vulnerability shows up and a link to the original source. That way it’s easy to reproduce the security vulnerability.

Now there is now reason not to use VersionEye. You get notifications about:

  • out-dated dependencies
  • license violations
  • security vulnerabilities

This feature is pretty new, but already good tested through the PHP community. Your feedback is anyway welcome either here in the comments or on Twitter.

Caching Rails assets with NGinx

Nginx is a very popular high performance web server. It’s very easy to configure and many Ruby on Rails developers are using it as a load balancer for Rails applications. Usually the setup looks like this:

Screen Shot 2015-08-18 at 10.13.19

Typically the rails application is running on Mongrel, Unicorn or Puma. These ruby application servers can deliver static assets (.png, .jpg, .js, .css) as well, but there are not really optimised for that. Nginx is much faster in delivering static assets like images, javascript and css files. Ideally your ruby application server should only handle requests to ruby controllers and generate dynamic content.

This is the Nginx configuration for load balancing between 2 ruby application servers, or any other application servers:

 
upstream unicorns {
  server 192.168.1.11:8080;
  server 192.168.1.12:8080;
}
 
server {
  listen 80;
  location / {
        proxy_redirect off;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_pass http://unicorns;
    }
}

At Rails all static assets are delivered under the route “/assets/”. Now we will tell Nginx to cache files from “/assets/” for 120 minutes. For that we need this 2 lines at the beginning of our nginx.conf file.

proxy_cache_path /tmp/nginx levels=1:2 keys_zone=veye_zone:10m inactive=120m;
proxy_cache_key "$scheme$request_method$host$request_uri";

The first line defines the directory where the files for the cached asserts should be stored. In this case it would be “/tmp/nginx”. We define the name space “veye_zone” with 10 MB for this cache. The 2nd line defines the structure of the cache key, which has to be unique.

Now we can add another location for “/assets” to our configuration.

server {
  listen 80;

  location /assets/ {
        proxy_redirect off;
        proxy_pass_header Cookie;
        proxy_ignore_headers Set-Cookie;
        proxy_hide_header Set-Cookie;
        
        proxy_cache veye_zone;
        proxy_cache_valid 200 302  120m;
        proxy_cache_valid 404      1m;

        proxy_pass http://unicorns;
  }

  location / {
        proxy_redirect off;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_pass http://unicorns;
    }
}

The 3 most important lines for the caching are these here:

        proxy_cache veye_zone;
        proxy_cache_valid 200 302  120m;
        proxy_cache_valid 404      1m;

They reference to the “veye_zone” cache, which we defined in the top of our nginx.conf. The 2nd line tells Nginx to cache everything with a response code of 200 and 302 for 120 minutes. And everything with a response code of 404 for 1 minute.

All HTTP requests to “/assets/” first will go to one of the ruby application servers, but the responses will be cached at Nginx for 120 minutes. All further requests to “/assets/” will not hit the ruby app servers, because Nginx can deliver the content from the cache.

The beauty at this setup is that it’s seamless to the deployment of the ruby application. The deployment process doesn’t have to copy the assets files to the Nginx server. Nginx is just automatically caching all assets.

Displaying errors from the backend systems in the UI

VersionEye can monitor your project file (Gemfile, package.json and so on) on GitHub/Bitbucket and notify you about out-dated dependencies, license violations and security vulnerabilities.

The process of fetching a file from the GitHub API is very time consuming and many things can go wrong. Doing this inside of a HTTP request/response cycle would mean to block the web process for several seconds or even minutes. That’s why this job is done by background workers. In the project detail view a “re parse” job can be triggered by hitting the “Re Parse Now” button. Until the background job is running a loading bar is displayed like on this image.

Screen Shot 2015-06-07 at 08.22.18

If the job is done the page reloads.

By dealing with external APIs many things can go wrong. Maybe the API is not reachable or there is another error and the project file can not be parsed. The background worker is throwing some Exceptions, but they never show up in the UI because the workers are running on completely different machines. That problem is solved now!

Here is a drawing of the architecture. Every square on the image is a different machine with his own IP address. They can even be in different geographical regions.

RabbitMQ-Architecture

VersionEye is using heavily RabbitMQ to distribute work. If somebody hits the “Re Parse Now” button, the request goes to a Rails controller of one of the web application servers. The Rails controller sends a message with the project id to RabbitMQ and returns immediately the response to the Browser. That takes only a couple milli seconds and after the response is processed the user can see that the background job is running. See picture 1 for that.

The RabbitMQ server can receive messages from web app servers, but also from a scheduler, which is triggering re parse jobs for projects with daily, weekly and monthly period.

There are several workers who subscribed to a specific channel on RabbitMQ. Some workers are responsible for fetching a users list of repositories from the GitHub API. Some workers know how to search for “supported project files” in a given git repository. And other workers are responsible for fetching a given project file from the GitHub API, parsing it and updating an existing project at VersionEye with the re parsed dependencies.

Every worker is specialised in doing exactly 1 job. The code for a worker is usually less then 100 lines. You can think of it as micro services.

Assume worker1 will get the job which just was triggered by the user. The worker tries to fetch the file from the GitHub API but something goes wrong. The GitHub API returns a error code and the project file can not be fetched. Some Exceptions are thrown and logged somewhere in some log files. Normally the worker would response to RabbitMQ that the job is done. But now the worker is storing error messages in the project object itself. That is a known pattern from ActiveRecord. If you try to persist an object with ActiveRecord and the operation fails it stores the error messages inside of the object. The error messages can be displayed with this command:

.errors.full_messages.to_sentence

The workers at VersionEye are doing it similar. The workers are storing error messages in the “parsing_errors” array of the project model. If this array is not empty, the elements are displayed in the project detail view, like here for example.

Screen Shot 2015-07-14 at 17.56.43

In the project overview page the projects with errors are marked with a “warn” sign. Like the first project in this example.

Screen Shot 2015-07-14 at 17.56.19

Now the users are getting feedback from the background workers. Possible reasons for this errors are:

  • The given URL doesn’t exist anymore.
  • The project file was moved.
  • The git repository was renamed.
  • The credentials for the GitHub/Bitbucket API are not valid anymore.

The most common reasons are the last 2 ones. If the git repository was renamed the VersionEye project has be removed and created newly again. If the credentials for the GitHub/Bitbucket API are not valid anymore then it can be fixed by going to settings and disconnecting from GitHub/Bitbucket and then connecting again to GitHub/Bitbucket. That they VersionEye is getting new API tokens for the logged in user.

This feature is quiet new, just a couple hours old. Feedback is welcome 😉

Merge Projects via the VersionEye API

Round about one month ago we released a new feature which supports multiple files in one project. That is useful for almost every modern software project. Some package managers, such as Bundler, Composer, CocoaPods and others, generate anyway 2 files per project. Beside that many modern software projects are splittet up in frontend and backend. And many times there are different technologies and different package managers involved to handle the open source dependencies. That’s why it makes a lot of sense for VersionEye to support multiple files per project.

In the Web Interface you can merge already existing projects into another project. And of course you can “unmerge” them if you like. Now this 2 functions are available via the API as well.

Screen Shot 2015-02-05 at 11.42.12

Assume you have 2 projects with each 1 file. A Java project (ID: 222) with a simple pom.xml file for your backend and a JavaScript project (ID: 333) with a bower.json file. But actually it’s the same project and you would like to merge them on VersionnEye to have everything together in one single view. With this call on the API you would merge them:

/api/v2/projects/222/merge/333?api_key=YOUR_API_KEY

In that case the Java project would be the parent and the JavaScript project would be the child. If you merge them the first time the order doesn’t madder. The other way around would work the same way.

If you would like to split the 2 projects again you could do it with this call:

/api/v2/projects/222/unmerge/333?api_key=YOUR_API_KEY

Easy! Right?

There is even a third new endpoint. This one:

Screen Shot 2015-02-05 at 11.55.14

This endpoint is specially for the VersionEye Maven Plugin. On a multi module project the VersionEye Maven Plugin gets executed several times and there are multiple lifecycle runs involved. Because of the maven architecture the VersionEye Maven Plugin doesn’t know always the VersionEye ID of the parent project. That’s why this endpoint got introduced, to identify the parent by groupId and artifactId. These 2 coordinates are always available in the Maven lifecycle.

Try out the new Endpoints and leave a comment if you have questions.

Projects with multi file support

In modern software development it is not uncommon that you use more than 1 package manager for a software project. Maybe you are using Ruby for the backend and JavaScript for the front end. In that case you would use 2 package managers. Bundler/Rubygems for the backend and maybe bower for the front end. You would have at last 3 files for describing your dependencies. A Gemfile and a Gemfile.lock for your backend and a bower.json file for your front end.

Up to now a project on VersionEye was always a representation of 1 single file. For example the Gemfile. So if you wanted to monitor all 3 files you had to have 3 projects on VersionEye. Now a project on VersionEye can have multiple files!

Screen Shot 2014-12-23 at 11.21.47
If you click on a Gemfile in our GitHub integration, VersionEye will check if there is a corresponding Gemfile.lock in the same directory. If so, it will parse that Gemfile.lock as well and create a new project with 2 files.

VersionEye-Multi
In the dependency tab, the corresponding files are displayed above the dependency table. In this example the Gemfile is selected and we see the dependencies of the Gemfile. By clicking on the Gemfile.lock we can switch to the dependencies of the Gemfile.lock. It’s all in the same project view.

The overall numbers in the head of the page are summed up from all files in the project. That means over the 2 files there are 58 unique dependencies and 33 of them are out-dated. Also the dependency badge is summed up over all files. The dependency badge only turns green if ALL dependencies in ALL files are up-to-date.

But that’s not all. How do we get the bower.json file into this project? Let’s assume the bower.json file is already a seperate project on VersionEye. Now we can merge projects into each other. This is how it works. Let’s navigate to the desired project we would like to merge into our Ruby project. Navigate to the “settings” tab. Here there is a new input field which looks like this:

Screen Shot 2014-12-23 at 12.13.53
Here we can pick a “parent” project to merge in. By clicking on the merge button the current project will be merged as subproject into the selected project.

VersionEye-Multi-3Now we have a project with 3 files. Gemfile, Gemfile.lock and a bower.json file. The overall numbers are updated. Overall 3 files we have 59 dependencies and 33 are out-dated.

Under the file names there is a link “unmerge bower.json”. By clicking on that link the bower.json can be unmerged from this project. That means it will be removed from this project and be again a seperate project.

Pretty cool! Right? This is just the first step. In the next update we will bring this feature to the API and to the plugins as well 😉

Let us know what you think about this feature, either here in the comments or on Twitter.

Most referenced packages

We just updated our language pages. Up to now we only displayed the top 10 followed packages and the 10 currently updated packages. Now we display also the top 10 referenced packages from that language. Here is an example for the Ruby page.

Screen Shot 2014-01-06 at 18.49.42

The first columns shows the top 10 packages with the most references. Rspec is the most referenced package in Ruby! 19498 other GEMs are referencing it. It has more references then Ruby on Rails.

Feel free to checkout the other language pages as well:

https://www.versioneye.com/php
https://www.versioneye.com/Clojure
https://www.versioneye.com/Node.JS
https://www.versioneye.com/Objective-C

Let us know what you think!