Pull request support for GitHub

77% of all our signed up users are coming from GitHub and they use VersionEye to monitor their GitHub repositories. For a while now there is a VersionEye service hook in GitHub which calls back the VersionEye API on every git push event and updates the corresponding project in VersionEye with the latest dependencies from GitHub. But now you can use VersionEye to test your changes directly in a Pull Request and see the results of those changes in the GitHub Pull Request UI. This way you can even enforce rules that would prevent a Pull Request merge if a dependency is blacklisted or it is flagged with a security vulnerability.

Given that GitHub is moving away from Service Hooks towards native WebHooks, we changed our integration to do the same. If you are logged in to VersionEye with your GitHub account you can simply click the link “Create from GitHub” and navigate to a repository where you will see all supported project files and the corresponding branches. To monitor a project file simply flip the switch directly next to it.

Screen Shot 2016-08-16 at 14.05.48

In the example above the Gemfile in the develop branch is monitored by VersionEye and it works like this: Every time a new project file is imported from GitHub, VersionEye will automatically create a web hook on that GitHub repository, which will call back the VersionEye API on each “git push” event and if the monitored files are affected, VersionEye will re-parse the files and update the view in VersionEye immediately.

The cool thing is that this also works with pull requests! Assume somebody is forking your project, adding new dependencies and sends back a pull request. In that case the VersionEye web hook kicks in and checks if some of the dependencies have security vulnerabilities or unknown licenses.  For those cases, VersionEye will respond to the pull request with  a “failed” status message  and you can see the result directly on the GitHub Pull-Request tab, like this:

GitHub-pullrequest-VersionEye-check

By clicking on the “Details” link you can see a more specific report at VersionEye for that pull request.

gh-veye-02.png

A report like this is generated on every new commit which is added to the existing pull request. It’s similar to a build at TravisCI. Now you can still merge the pull request if you like but at least you are informed about security vulnerabilities and license issues.

In the VersionEye UI you will find a new sub menu “Pull requests” in the organisation view. Here are all commits from all pull requests are listed which belong to your organisation.

Screen Shot 2016-08-16 at 14.22.50

The interesting part here is how VersionEye is combining dynamic and static analysis. Your project is checked for security vulnerabilities and licenses on each commit and each pull request but also regularly independent from that. Even if there are no commits and no pull requests for a couple days, VersionEye will check your project once a day by default and send you an email if a new security vulnerability was found for one of your dependencies.

With this new feature the developers are getting instant feedback to their changes. In real time the dependencies are analysed and the developers informed. That saves a lot of Time & Money. We think it’s important to run this kind of analysis as early in the development process as possible. If your software is already bundled and published to a binary repository it’s too late for this kind of analysis because your software might be distributed already to a high number of devices/clients. Run your analysis as early and as often as you can 😉

And by the way, that’s all open source 😉 You can run the software on your own infrastructure and it works well together with GitHub Enterprise 🙂

This is a pretty new feature. If you find a bug or you have some kind of feedback, simply open a ticket here.

NPM Module for VersionEye

Now there is a NPM Module for the VersionEye API. The versioneye-update NPM Module was developed by Onwerk, a Software Service Provider from Mannheim (South Germany). They develop web & mobile applications with Node.JS. Like this interactive Jackpot game, built with iPad, XBox KinectRaspberry Pi and NodeJS.

The Onwerk engineers like to stay ahead of cutting edge technology. They want to keep their dependencies up-to-date to get bug & security fixes ASAP into their applications. And of course they want to take advantage of new features as soon as possible.

Onwerk+NPM=VersionEyeUpdate

VersionEye has a very good Integration for GitHub and Bitbucket. If your source code is on one of this cloud SCMs, VersionEye can monitor your package.json directly via the GitHub/Bitbucket API and you get notifications about out-dated dependencies automatically.

But the use case for Onwerk is different. They do BIG Software Projects for LARGE customers and because of NDAs und German privacy laws they are not allowed to give out the source code to anybody else. That’s why they are using the VersionEye API to get notified about out-dated dependencies.

And because they wanted to automate the whole process they developed the versioneye-update NPM module, which gets executed on each build on their private Jenkins CI Server. The process looks like this:

Onwerk-VersionEye-Integration

The NPM module versioneye-update is running on each build on the Jenkins. The module is sending the current package.json file to the VersionEye API to update an existing VersionEye project. That way VersionEye nows which dependencies are used in the project right now. VersionEye will compare the version numbers from the package.json file with the newest versions in the VersionEye database to find out-dated dependencies. If there is at least 1 out-dated dependency or at least 1 license violation VersionEye will send out an email notification to the project owner and the project collaborators.

That way the whole process is automated. The engineers don’t have to execute wired commands in the console and they are not in risk to forget something. Beside that the source code stays in house. VersionEye never has access to the source code. The only file which has to be shared with VersionEye is the package.json and that doesn’t get stored on the server! After parsing it once the file object becomes a victim of the garbage collection.

Try the versioneye-update module by yourself and give feedback. Beside this NPM module there are many other Plugins and AddOns build on top of the VersionEye API. Check them out on the API site.

Open Ticket System for VersionEye

We at VersionEye are always interested into your Feedback! Right from the beginning the feedback icon was part of the main menu at VersionEye.

Screen Shot 2015-02-12 at 10.34.33

By clicking on the feedback icon a modal dialog appeared where you could send a message to the VersionEye Team. We received your feedback as email and it was not visible to anybody else. Many times we received a bug/feature report multiple times. Simply because many users found the same issue.

A couple days ago we changed the behaviour of the feedback icon. Instead of opening a modal dialog it redirects the user to the ticket system on Bitbucket.

Screen Shot 2015-02-12 at 10.37.10

Everybody can sign up for free at Bitbucket and open a ticket for VersionEye. The tickets are visible for everybody! Even for users who are not singed up a Bitbucket!

Why Bitbucket? Why not GitHub?

The code for VersionEye is splittet up in over 10 git repositories. For long time we hosted everything on GitHub and we used internal the GitHub ticket system for ourselves. There are 2 reasons why we moved some repositories to Bitbucket.

  • On Bitbucket you can make the ticket system public and at the same time keep the source code private.
  • The Ticket system on Bitbucket is a little bit more advanced. Specially the “vote” feature is very useful for communities. That way users can vote for a ticket and we can sort the tickets by votes. That is very useful for new feature requests. Now we don’t have to guess anymore which feature to implement first. We simply implement the features first with the most votes!

Screen Shot 2015-02-12 at 11.38.42

We still have code on GitHub and we are still paying customers of GitHub. Simply because the performance of GitHub is better. A ‘git push’ or ‘git checkout’ works on GitHub always a couple seconds faster than on Bitbucket.  With git you can have multiple backends without a problem 🙂 That’s why we all love git!

We hope you will embrace our new open ticket system and we will see many features requests and votes in future 😉

New license crawler finds licenses in git repositories

VersionEye is crawling a whole bunch of package managers + some GitHub Repositories. Many open source libraries provide their license directly within the package manager. On search.maven.org the license information is even mandatory. If somebody submits a new package and the license information is not included in the pom.xml file the package will be declined and not published. That’s a good thing! Unfortunately at most other package managers the license is not mandatory and so it comes that we have many open source libraries in our database without a license 😦

opensource_logo

But many projects on GitHub provide a LICENSE file. Other projects copy & paste the license text directly into their README page. That’s why I developed a new crawler which is looking for a LICENSE file in the root of a git repository. If some kind of LICENSE file was found the crawler tries to recognize the license text. Is it an MIT text, a GPL 3 text or some other license text? If the crawler recognizes the license text it saves the information into the VersionEye database. Otherwise not.
If there is no LICENSE file the crawler tries to find some license text directly in the REAMDE file. If the crawler couldn’t find anything it skips the repo and goes on to the next one.

The license crawler doesn’t crawl whole GitHub. It crawls only git repositories of open source packages which are already in the VersionEye database and have a corresponding link to GitHub. It basically completes meta data to already existing open source packages in the database. And right now there are 463.873 open source libraries in the VersionEye database.

Results

In the past couple days the license crawler was able to find license information for 22.694 open source libraries. 5.684 of this open source libraries are JavaScript libraries registered on Bower, without any license information. One popular example is angular-mocks. The repository provides a bower.json file without a license! The license text of angular-mocks is directly embedded into the README.md file. VersionEye found & recognized the license as MIT and displays it now on the corresponding VersionEye page. There are many more examples like this one.

Next Step

These works already pretty good. But the next logical step is to write a crawler which recognizes licenses in source code. That requires a bit more computing power and a bit more time to write it. Nothing for 2014. But a good goal for 2015 😉

Update Single Git Repository

VersionEye has a very good integration for GitHub, Bitbucket and Stash. It can monitor project files from different repositories and multiple branches. If VersionEye is connected to a GitHub Account it fetches meta data about that account via the official API. Meta data such as repo names, branch names and file names. If a branch gets added/removed from a repository this meta data is out-of date. Until now you had to hit the “Reimport all repositories” link to reimport ALL repositories with current meta data. If you have many repositories that is a bit time consuming and annoying.

Now we added in each repository view a link “Update meta-data about this repository”, which only updates branches and file names of the current repository. That’s much more efficient 🙂

UpdateSingleGitRepo

How to connect VersionEye Enterprise with Atlassian Stash

The VersionEye Enterprise VM is currently in private beta. It runs in your private network and can crawl your private repositories, such as Artifactory Pro, Nexus Pro and others. And it can be connected to your private Git repository, such as GitHub Enterprise or Atlassian Stash.

The Atlassian Stash integration is brand new. Connecting it to Atlassian Stash means that VersionEye Enterprise can use Stash as OAuth SSO (Single Sign On) provider. If setup correctly users can login to VersionEye Enterprise with their existing Stash Accounts, by simply clicking the “Login with Stash” button.

Login with Stash

This is how you set it up. Check out this video on YouTube or keep reading forward.

First login in to VersionEye Enterprise. The default login is “admin” / “admin”.

VersionEyeEnterpriseLogin

Then click the “Stash Settings” link on the left side. The site should look similar to this one.

VersionEye Enterprise Stash Settings

The WEB URL needs to point to the domain where Stash is currently running. The OAuth process in Stash requires a private and a public RSA certificate. This is how you create the private RSA certificate.

 openssl genrsa -out mykey.pem 1024

This stores the private RSA certificate in mykey.pem. The next command generates the public RSA certificate and stores it in mykey.pub.

 openssl rsa -in mykey.pem -pubout >> mykey.pub

Now copy & paste the text value from mykey.pem into the “Private RSA Key” input field in VersionEye Enterprise.

VersionEye Enterprise + Atlassian Stash

Now login as admin to Stash and navigate to the “Application Links”. Unfortunately there is no easy way to generate an OAuth consumer key in Stash. We have to go through the “Application Links” process. Just add a new “Application Link”.

Stash-AppLink

You will get an error message similar to this one.

Screen Shot 2014-10-05 at 21.41.55

That’s OK. Just ignore it and click “Continue”. In the next form only the first 2 input fields are important. As Application Name type in “VersionEye” and as Application Type select “Generic Application”. All other input fields are not important and can be filled with random text.

Screen Shot 2014-10-05 at 21.43.09

That will create a new Application Link in the list. Click on “Edit” to set it up correctly. In the upcoming mask click on “Incoming Authentication”. Here you can fill in the “Consumer Key” from VersionEye.

Screen Shot 2014-10-05 at 21.44.18

And further down the public key from the “mykey.pub” file.

Screen Shot 2014-10-05 at 21.45.28

And that’s it.

Now you can logout from Stash and VersionEye. Navigate to the “Sign in” page on VersionEye Enterprise and you will see the “Login with Stash” button. By clicking on it you should get this dialog from Stash.

Screen Shot 2014-10-05 at 21.47.38

After clicking allow you are logged in into VersionEye Enterprise.