Merge Projects via the VersionEye API

Round about one month ago we released a new feature which supports multiple files in one project. That is useful for almost every modern software project. Some package managers, such as Bundler, Composer, CocoaPods and others, generate anyway 2 files per project. Beside that many modern software projects are splittet up in frontend and backend. And many times there are different technologies and different package managers involved to handle the open source dependencies. That’s why it makes a lot of sense for VersionEye to support multiple files per project.

In the Web Interface you can merge already existing projects into another project. And of course you can “unmerge” them if you like. Now this 2 functions are available via the API as well.

Screen Shot 2015-02-05 at 11.42.12

Assume you have 2 projects with each 1 file. A Java project (ID: 222) with a simple pom.xml file for your backend and a JavaScript project (ID: 333) with a bower.json file. But actually it’s the same project and you would like to merge them on VersionnEye to have everything together in one single view. With this call on the API you would merge them:

/api/v2/projects/222/merge/333?api_key=YOUR_API_KEY

In that case the Java project would be the parent and the JavaScript project would be the child. If you merge them the first time the order doesn’t madder. The other way around would work the same way.

If you would like to split the 2 projects again you could do it with this call:

/api/v2/projects/222/unmerge/333?api_key=YOUR_API_KEY

Easy! Right?

There is even a third new endpoint. This one:

Screen Shot 2015-02-05 at 11.55.14

This endpoint is specially for the VersionEye Maven Plugin. On a multi module project the VersionEye Maven Plugin gets executed several times and there are multiple lifecycle runs involved. Because of the maven architecture the VersionEye Maven Plugin doesn’t know always the VersionEye ID of the parent project. That’s why this endpoint got introduced, to identify the parent by groupId and artifactId. These 2 coordinates are always available in the Maven lifecycle.

Try out the new Endpoints and leave a comment if you have questions.

Projects with multi file support

In modern software development it is not uncommon that you use more than 1 package manager for a software project. Maybe you are using Ruby for the backend and JavaScript for the front end. In that case you would use 2 package managers. Bundler/Rubygems for the backend and maybe bower for the front end. You would have at last 3 files for describing your dependencies. A Gemfile and a Gemfile.lock for your backend and a bower.json file for your front end.

Up to now a project on VersionEye was always a representation of 1 single file. For example the Gemfile. So if you wanted to monitor all 3 files you had to have 3 projects on VersionEye. Now a project on VersionEye can have multiple files!

Screen Shot 2014-12-23 at 11.21.47
If you click on a Gemfile in our GitHub integration, VersionEye will check if there is a corresponding Gemfile.lock in the same directory. If so, it will parse that Gemfile.lock as well and create a new project with 2 files.

VersionEye-Multi
In the dependency tab, the corresponding files are displayed above the dependency table. In this example the Gemfile is selected and we see the dependencies of the Gemfile. By clicking on the Gemfile.lock we can switch to the dependencies of the Gemfile.lock. It’s all in the same project view.

The overall numbers in the head of the page are summed up from all files in the project. That means over the 2 files there are 58 unique dependencies and 33 of them are out-dated. Also the dependency badge is summed up over all files. The dependency badge only turns green if ALL dependencies in ALL files are up-to-date.

But that’s not all. How do we get the bower.json file into this project? Let’s assume the bower.json file is already a seperate project on VersionEye. Now we can merge projects into each other. This is how it works. Let’s navigate to the desired project we would like to merge into our Ruby project. Navigate to the “settings” tab. Here there is a new input field which looks like this:

Screen Shot 2014-12-23 at 12.13.53
Here we can pick a “parent” project to merge in. By clicking on the merge button the current project will be merged as subproject into the selected project.

VersionEye-Multi-3Now we have a project with 3 files. Gemfile, Gemfile.lock and a bower.json file. The overall numbers are updated. Overall 3 files we have 59 dependencies and 33 are out-dated.

Under the file names there is a link “unmerge bower.json”. By clicking on that link the bower.json can be unmerged from this project. That means it will be removed from this project and be again a seperate project.

Pretty cool! Right? This is just the first step. In the next update we will bring this feature to the API and to the plugins as well 😉

Let us know what you think about this feature, either here in the comments or on Twitter.

New Single Page App for GitHub/Bitbucket integration

Yesterday we launched the new Single Page App for the VersionEye – GitHub/Bitbucket integration. For sure you know the old one. This here.

OldSPA_Github

It had a lot of disadvantages. The UI was confusing and cluttered with a lot of information, which is not important for the VersionEye use case. And specially if you had a lot of branches and project files in a repository you had to scroll in the little fixed div area. That was not very comfortable.

The new SPA (Single Page App) is a completely rewrite, with ReactJS.  The UI is much simpler, there is nothing you can do wrong. This is the initial page.

Screen Shot 2014-07-31 at 10.10.40

You get ALL your repositories listed, without any other useless information. With the text input field you can filter the list and quickly get to your desired repository. By clicking on a repository name you come to the next page, there we import ALL the branches and the supported project files.

Screen Shot 2014-07-31 at 10.12.36

Here you can take advantage of the full size of your screen. If you want to monitor a file, simply flip the switch on. After the file is parsed the file name will turn into a link which leads to the VersionEye project page, where you can see all the dependencies of the file.

This SPA is much simpler than the old one. There is nothing you can do wrong and it works much faster. Beside the SPA we also refactored the backend services, which now take more advantage of RabbitMQ. You can try the new SPA in the login area here for GitHub:

https://www.versioneye.com/user/projects/github_repositories

And here for Bitbucket:

https://www.versioneye.com/user/projects/bitbucket_repositories

Let me know how you like it. Either here in the comments or on Twitter.

Problems with Package Managers – Bower Edition

Are you a front-end developer, working every day with HTML, CSS and JavaScript? Then you have probably heard already about Bower, the package manager for frontend JavaScript. You can install the client via NPM. Bower is based on Node.JS but doesn’t use NPM as registry, it’s a completely separated project with it’s own registry. You can use Bower as a Ruby, Python or PHP guy as well. Installing a package works like this:

bower install backbone#1.1.0

Which will download backbone version 1.1.0 and his dependencies into the directory ./bower_components/backbone/. All Bower packages are downloaded into ./bower_components.

Bower is handling versions and dependencies but It does NOT wire the JS components into your project. And that’s OK! It doesn’t force you to a specific directory structure. That’s all up to you.

Last week we finished the bower integration for VersionEye.  More than 5K JavaScript packages are now available through our search, powered by ElasticSearch and can now easily be followed. If you want to get notified about a new version of your favorite JavaScript package, simply follow it on VersionEye and you will receive an email notification on the day they release the new tag/version.

During our integration work we had to crawl the bower registry and GitHub to get the meta data in our MongoDB / ElasticSearch backend. Here are some interesting numbers I would like to share with you.

All registered bower packages are available through this JSON file:

https://bower.herokuapp.com/packages

Each element in the array contains the name of a bower package and the URL to his repository. At this time the list has 8504 entries. Only 47 URLs don’t link to GitHub.

bower_github

Some of the 47 URLs link to Bitbucket and funny entries like 192.168.0.131 🙂

We consider a bower package valid if both of these conditions are true:

  • The URL still exists and is publicly available.
  • The repository contains a valid bower.json or component.json file.

From the 8504 packages 2515 we couldn’t validate. That means either the repository doesn’t exist anymore or we couldn’t find any bower.json file or the the bower.json file was not valid JSON. 5989 packages had valid bower.json files.

bower_valid

More than 900 registered repositories simply don’t provide a bower.json file. Other ones provide a bower.json file but it contains errors. The most common error is that there is a comma to much or to less in an array definition.

From the 5989 packages we could validate, 1194 don’t have any version tags in the associated git repository. 4795 packages provide at least one tag.

bower_tags

Having a package registered on a package manager without fixed version/tag is a bit useless. From all 8504 registered packages only 4795 offer a valid bower.json file and some tags on their repositories.

bower_comp_valid

Specification

The bower specification describes exactly how a valid package has to look like. One of the points for the name is: “Lowercase, a-z, can contain dash or dot but not start/end with them“. But actually 495 packages do have uppercase characters in their name. That alone is not dramatic. But some packages are registered with the same name. For example the “scroller” package. It is registered as “scroller” and “Scroller”. Two different GitHub repositories with almost the same name. The only differentiator is the lowercase/uppercase S in the beginning. We were able to find 34 names where collisions occur if you force lowercase names.

Validation

There is a very simple reason for this numbers. There is currently almost no validation on the submissions. But user authentication and package validation is already on the todo list of the Bower team. I expect many improvements in the next months. However, launching a package manager without any kind of validation was maybe not the best idea.

Conclusion

I don’t wrote this article to keep you away from Bower. The people behind the bower project are smart guys and very responsive on Twitter. They are continuously improving. And by the way, it’s all open source. Everybody can send a pull-request to improve the validation. I already send a pull-request for the name validation.

Having some kind of versioning for frontend development is very important. Otherwise you just download some CSS and JS files from the Internet and put them into your project. After a couple weeks most likely you don’t even remember which version of backbone your are using. Imagine doing that with your ruby gems or python libraries! Developers should take versioning of their frontend code as seriously as they do for their backend development.

If you are a Ruby on Rails developer you should check out this page https://rails-assets.org/ for rails – bower integration.

If you wanna know more about how to monitor a bower.json file or how to get a dependency badge for your bower project you should check out this blog post about our Bower integration.

What are your experiences with Bower? Leave a comment here or contact me on Twitter. I would like to hear your opinion to this topic.

Bower Integration

We finished our bower integration! Yeah 🙂 So many of you requested this integration and now it’s done. Now I will get a bit fewer emails per week 🙂

BowerJS Bird

Through the bower integration we have now additional 5K+ JavaScript and 500+ CSS libraries on VersionEye, you can follow. Check out the JavaScript page to see the most followed libraries. Currently jquery-mobile, jquery and backbone are the most followed JavaScript libraries.

Thanks to Bower we can now show dependencies on the JavaScript pages. Just like for Ruby, PHP, NodeJS and other languages. Beside the dependencies we show you some bower code snippets and a download link.

Screen Shot 2014-01-28 at 17.26.01

And since we have the dependencies now, we can display dependency badges for JavaScript libraries too 🙂

dep_up-to-date dep_out-of-date dep_none

I’m looking forward to see them on the README pages on GitHub 🙂

Beside the follow feature VersionEye can actively monitor your bower.json file on GitHub and notify you about out-dated dependencies. In the login area just navigate to the repository and turn on the switch beside the package file.

Screen Shot 2014-01-28 at 17.58.08

Just turn it on and you will receive an email notification as soon their are updates available for your bower.json file. Stay passive & up-to-date 😉

Screen Shot 2014-01-28 at 17.58.39

The bower integration is kind of new. If you see anything that doesn’t seems to be OK, then just contact me on twitter or submit a message via the contact form at VersionEye.

VersionEye started its first Meetup group “Geek2Geek”

ImageGeek2Geek is a monthly Berlin Tech Talk. Our goal is to bring software developers with a diverse background together. We noticed that the coding community is often isolated, meaning that PHP devs. are only going to PHP UG’s and Java devs. solely to Java UG’s

However, we believe that we could achieve great things if we connect. If you’re a open source developer, start-up, techie, or geek using Java, Ruby, Python, Node.JS, PHP, JavaScript, R or Clojure, this group is for you!

Our first meetup is on July 23, 2013 and we’re pleased to announce that the speakers for our first Geek2Geek meetup are Christoph Beckmann from KaufDA and Tobias Balling from BLINKIST. This time we will focus on “IT infrastructure for DevOps”.

Christoph Beckmann
Christoph is team lead at KaufDA. He’s developing preferably with the Grails framework and is a DevOp expert. Over the past 2 1/2 years he has helped build the international KaufDA IT team. Previously, he gained experience at a consulting firm in Cologne and founded Germany’s first toilet search engine. Christoph will show how KaufDA manages its infrastructure with puppet in 5 countries.

Tobias Balling
Tobias is CTO at BLINKIST. He’s is currently thinking about the perfect presentation topic. So, more details are coming soon!

Thanks to VersionEye, snacks and beer are available for free, while supplies last. We’re looking forward to meeting you!