Which programming language has the best package manager?

I have to work with a lot (9) of different package managers at my daily work at VersionEye. Part of our mission is it to make manual updating of dependencies extinct, because it’s a manual and time consuming task which nobody enjoys. That’s why we are building a notification system for open source software libraries to make Continuous Updating easy and fun. And since we support several programming languages – 8 at this point! – I get to write crawlers and parsers for all of them. To give you a better overview over the strengths and weaknesses of these package managers, I picked the most popular one for each language and will compare them. The contenders are:

  • RubyGems / Bundler (Ruby)
  • PIP / PyPI (Python)
  • Packagist / Composer (PHP)
  • NPM (Node.JS)
  • Bower (JS, CSS, HTML)
  • CocoaPods (Objective-C)
  • Maven (Java)
  • Lein (Clojure)

What are package managers?

Package managers are tools for software developers to help them easily share and consume software libraries and of course to manage dependencies! Specially transitive dependencies. If you are a software developer and you are still downloading software libraries with your browser and placing them into the right directory by hand, then you are doing something wrong! That is totally 1999!

Nowadays every software developer uses at least 1 package manager at work. That means you define the software libraries (dependencies) you want to use in your project in a project file. The package manager is then  responsible for

  • downloading the software libraries from a repository
  • placing the downloaded libraries into the right place and linking them correctly into the project
  • resolving transitive dependencies

A package manager always has 2 parts:

  • a Repository of binaries or source code
  • a client which communicates with the repository and performs some work on the client side. This is in most cases a command line tool

The advantages of such a system is pretty clear. It’s much easier to stay in control of dependencies since you just change a number in a text file and execute a command. The package manager is doing all the work for you. Another big advantage is that you don’t need to checkin the actual software libraries into your SCM. You only checkin the project file from your package manager.

RubyGems / Bundler (Ruby)

RubyGems is the package manager for Ruby. Libraries that are bundled as such are also called gems. A gem contains a directory with source code and a *.gemspec file. The *.gemspec file contains meta information to the gem, such as name, version, author, license and of course the dependencies of the gem. A gem can be build with the command line tool “gem” like this:

gem build rails.gemspec

And published to the central gem repository like this:

gem push rails-1.0.0.gem

In an non library (gem) project the dependencies are mostly managed by bundler, a very popular dependency manager for RubyGems. Bundler manages dependencies defined in a Gemfile, a simple text file. Here is an example:

source 'https://rubygems.org'

gem 'rails' , '3.2.16'
gem 'jquery-rails' , '2.2.1'

group :test do
  gem 'rspec' , '2.14.0'
end

The file starts with defining the repository. Which is rubygems.org, the central repository for all ruby gems. Each line in the file defines a gem, with name and version. Gems can be grouped together for different environments. E.g. gems in the test group are not required on production, but in test environments.

You don’t need to define an exact version number. You can also define a range of versions and take advantage of pre defined operators. Everything well documented on bundler.io.

Just execute the command in the directory where the Gemfile is placed and it will fetch all dependencies from the repository, resolves transitive dependencies and places them into the right directory.

bundle install

This command also creates or updates the file Gemfile.lock. This file contains the actually locked versions of the dependencies in the Gemfile and their transitive dependencies. The locked versions in the Gemfile.lock are specially important when you work with pessimistic version constraints. The Gemfile.lock is something you checkin to your SCM and never change by hand! More on that on bundler.io.

Pros

  • All gems are stored centralized on RubyGems.org. Really all ruby open source developers are using it and pushing their gems to it.
  • Learning curve is very low, very easy to understand how it works.
  • It’s very easy to publish new gems on rubygems.org. It takes less then a minute!
  • It has a very good REST JSON API.
  • Besides gems, a tag on a git repository can be defined as a dependency. That makes it very easy to fork & patch a software library.
  • Bundler supports semantic versioning and the pessimistic version constraints specifier enables to fetch the latest patches without breaking to support fetching always the newest patch/minor version of a gem.
  • There is a mirror at http://mirror1.prod.rhcloud.com/mirror/ruby/

Cons

  • Usually gems are not cryptographically signed! This can lead to security issues! It is possible to sign gems, but it’s not mandatory and most developers don’t sign their gems.
  • Defining a license for a gem is not mandatory. But most of the gems are using the MIT license.

You will not find much negative writing about it on the internet, besides the points which are listed here.

NPM (Node.JS)

NPM is the package manager for Node.JS. Libraries are stored as tgz files in the central Node.JS repository, which is npmjs.org. An NPM package is a zipped directory with source code and a package.json file, which contains all the meta information about the package, such as name, version, description, dependencies and so on. A package can be published like this:

npm publish

Here is an example for a package.json file from a well known Node.JS library. I modified the file a bit to get it shorter.

{
  "name": "request",
  "description": "Simplified HTTP request client.",
  "version": "2.31.1",
  "author": "Mikeal Rogers",
  "repository": {
    "type": "git",
    "url": "http://github.com/mikeal/request.git"
  },
  "bugs": {
    "url": "http://github.com/mikeal/request/issues"
  },
  "engines": [
    "node >= 0.8.0"
  ],
  "main": "index.js",
  "dependencies": {
    "qs": "~0.6.0",
    "json-stringify-safe": "~5.0.0",
    "forever-agent": "~0.5.0",
    "node-uuid": "~1.4.0",
    "mime": "~1.2.9"
  },
  "optionalDependencies": {
    "tough-cookie": "~0.10.0",
    "form-data": "~0.1.0",
    "tunnel-agent": "~0.3.0",
    "http-signature": "~0.10.0",
    "oauth-sign": "~0.3.0",
    "hawk": "~1.0.0",
    "aws-sign2": "~0.5.0"
  },
  "scripts": {
    "test": "node tests/run.js"
  }
}

By executing the following command in a directory with a package.json file, NPM will download all dependencies from the package.json file, resolve transitive dependencies and place them into the right place.

npm install

Pros

  • All packages are centralized at npmjs.org.
  • Learning curve is very low.
  • It’s very easy to publish new packages on npmjs.org.
  • It has a very good REST JSON API.
  • There is an NPM mirror in Europe.
  • Besides NPM packages, a tag on a git repository can be defined as a dependency. That makes it very easy to fork & patch a software library.
  • NPM supports semantic versioning and has an own operator which supports fetching always the newest patch/minor version of a package.

Cons

  • The NPM packages are not signed! That might lead to security issues!
  • Defining a license for a package is not mandatory.

NPM is a very young package manager. They learned from the failures of other package managers. It’s almost perfect!

Packagist / Composer (PHP)

Composer is the new package manager for PHP. It was written to replace PEAR. There are still less then 1.000 packages on Pear, but already more then 25K packages on packagist.org, the central repository for Composer packages.

Composer is similar to NPM. Dependencies are defined in a JSON file, called composer.json. Here is a very simple example:

{
  "name": "your-vendor-name/package-name",
  "version": "1.0.0",
  "require": {
    "php": ">=5.3.0",
    "another-vendor/package": "1.*"
  }
}

Executing this command in a directory with composer.json will install all dependencies into your project and generate a composer.lock file, same as in Ruby the Gemfile.lock.

php composer.phar install

One big difference to NPM is that packagist.org doesn’t host any files. It is completely based on Git/Svn/Hn tags. Submitting a package to packagist.org means submitting a link to a public Git, Subversion or Mercurial repository. packagist.org expects a composer.json file in the root directory of the master branch AND tags on the repository. The name of the tag is the version number displayed on packagist.org.

Pros

  • All packages are centralized at packagist.org.
  • Learning curve is very low.
  • It’s very easy to publish new packages.
  • It has a very good REST JSON API.
  • Licenses are mandatory.
  • It supports semantic versioning and has an own operator which supports fetching always the newest patch/minor version of a package.
  • Composer has a very unique and cool feature called “minimum stability”, which allows to specify the minimum stability for a requested dependency. More on that on the official documentation.

Cons

  • There are no mirrors. If packagist.org is down you can not fetch packages anymore.
  • The packages are not signed! It might lead to security issues!

The Composer / Packagist project is even younger then NPM. It is a big step forward for the whole PHP community. All big PHP frameworks moved already to Composer / Packagist.

PyPI (Python)

PyPI is the central repository for python packages. Dependencies are defined in a setup.py file, which is pretty much pure Python code. Here is an example:

 
from setuptools import setup
import os

setup(name='giki',
    version='0.1pre',
    description='a Git-based wiki',
    author='Adam Brenecki',
    author_email='adam@brenecki.id.au',
    url='',
    packages=['.'.join(i[0].split(os.sep))
        for i in os.walk('giki')
        if '__init__.py' in i[2]],
    install_requires=[
        'dulwich==0.8.5',
        'argparse', 'requests>=1.1.0,<1.3.0', 'colorama',
        'jinja2==2.6',
        'Werkzeug==0.8.3',
        'markdown2==2.0.1',
        'docutils==0.9.1',
        'textile==2.1.5',
    ],
    extras_require = {
        'test':  [
            'nose==1.1.2',
            'WebTest==1.4.0',
        ],
        'faster_markdown':  [
            'misaka==1.0.2',
        ],
    },
    entry_points = {
    'console_scripts':
        ['giki = giki.cli:main'],
    },
)

Installing the dependencies works like this:

python setup.py install

The packages at PyPI are hosted as “*.tar.gz” files. Each packages contains the source code and a setup.py file with meta informations, such as name, version and dependencies.

With a little bit preparation a package can be published to PyPI with this 2 commands:

 python setup.py register -r PyPI
 python setup.py sdist upload -r PyPI

Pros

  • All packages are centralized at PyPI.
  • Learning curve is very low.
  • It’s easy to publish new packages on PyPI.
  • There are multiple mirrors for fail-over scenarios.

Cons

  • The packages are not signed! That can lead to security issues.
  • Defining a license for a package is not mandatory.
  • No build-in support for semantic versioning.

PyPI is a robust package manager! There is room for improvements, that for sure. But the core commiters behind the project are aware of that and new features are in the pipeline.

CocoaPods (Objective-C)

CocoaPods.org is the central repository for CocoaPods, a package manager for Objective-C software libraries and mainly used by iOS developers. The CocoaPods command line tool is implemented in Ruby and hosted on RubyGems.org. It can be installed like this:

gem install cocoapods

Dependencies are defined in a simple text file called Podfile. Here a simple example:

platform :ios
pod 'JSONKit', '1.4'
pod 'Reachability', '3.0.0'

Executing this command in the root directory will install the dependencies:

pod install

Currently CocoaPods is completely relying on GitHub as backend. There is one CocoaPods/Specs repository with all Pod specifications available on CocoPods.org. Submitting a new pod package works via pull-request. Each pull-request gets reviewed and merged by a human after passing automated tests. This doesn’t scale infinitely but it scales for the current size and guarantees a high quality for the pod specs.

Pros

  • All packages are centralized at CocoaPods/Specs.
  • Learning curve is very low.
  • Very good documentation.
  • It’s easy to publish new packages, with a pull-request.
  • License information is mandatory! Pull-requests for Pods without license definition will not be merged.
  • It supports semantic versioning and has an own operator which supports fetching always the newest patch/minor version of a package.

Cons

  • The packages are not signed! That can lead to security issues.
  • No mirrors available.

CocoaPods is a young project. The quality of the Pod specs is very high, because of the human review. If they grow to the size of RubyGems this workflow will not scale anymore. But for right now it’s good enough.

Bower (Frontside JS & CSS)

Bower.io is a package manager for frontside JavaScript and CSS. Pretty much for everything what you are loading from the server to the browser. Packages like jQuery and Bootstrap for example.

The bower command line tool is available as package on NPM. It can be installed globally like this:

npm install -g bower

Installing a bower package works like this:

bower install <package>#<version>

It downloads the package and it’s dependencies into a directory called bower_components.

Every bower package is described in a bower.json file. Here’s an example:

{
  "name": "my-project",
  "version": "1.0.0",
  "main": "path/to/main.css",
  "ignore": [
    ".jshintrc",
    "**/*.txt"
  ],
  "dependencies": {
    "<name>": "<version>",
    "<name>": "<folder>",
    "<name>": "<package>"
  },
  "devDependencies": {
    "<test-framework-name>": "<version>"
  }
}

Bower is completely Git based and works without any user authentication. Everybody can register new packages like this:
bower register <my-package-name> <git-endpoint>
Bower expects the root of the git-endpoint to contain a bower.json file, describing the bower package. And it expects that there are some tags on the Git repository. The names of the tags are the version numbers.

To unregister a package you have to ask the maintainers in the longest GitHub issue in the history of mankind.

Pros

  • All packages are centralized at bower.io.
  • Learning curve is very low.
  • It’s easy to publish new packages. Works even without registration.

Cons

  • The packages are not signed! That can lead to security issues.
  • No mirrors available.
  • The process for unregistering a package is a No-Go.
  • License informations are not mandatory.
  • Many registered packages doesn’t provide a bower.json in their git repository. The package registration process doesn’t check if the package is valid, if it really contains a bower.json file or if there are some tags on the repository.

Bower is kind of cool, but the quality of the packages is many times bad. Unregistering a package is a pain. And a couple hundred registered packages even doesn’t provide a bower.json file in their repositories. User authentication and package validation would be necessary to improve this.

Maven (Java)

Maven is the package manager for Java. The central Maven repository is Search.maven.org.
A project and it’s dependencies are described in an pom.xml file. Here is an example.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

  <modelVersion>4.0.0</modelVersion>
  <groupId>org.ploin.pmf</groupId>
  <artifactId>ploinMailFactory</artifactId>
  <packaging>jar</packaging>
  <version>1.4.1</version>
  <name>ploinMailFactory</name>
  <url>http://www.ploinmailfactory.org</url>
  <description>Loadbalancing for Mail Sending</description>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>${junitVersion}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>commons-logging</groupId>
      <artifactId>commons-logging</artifactId>
      <version>1.1</version>
      <scope>compile</scope>
    </dependency>
  </dependencies>

</project>

Each dependency is identified by:

  • groupId
  • artifactId
  • version

Other properties like scope are optional. As you can see you need at least 5 lines to define 1 single dependency. Dependencies are checked, fetched, resolved and installed on each compile time.

mvn compile

The packages on the Maven Repository Server are pretty much maintained in a pre defined directory structure, based on groupId, artifactId and version of the artifact. The artifacts themselves are binaries, mostly *.jar files. You can see it on this mirror from ibiblio.

Publishing an artifact on the central maven repository takes only 1 week. No! I’m not kidding you! First of all you have to signup at JIRA and open a ticket. The whole process is described here. And be prepared to scroll 🙂

Pros

  • The artifacts are singed!
  • Licenses are mandatory for publishing!
  • There are many mirrors available all over the world.

Cons

  • Pushing artifacts to the maven central repository is not as easy as it should be. Because this process is a pain in the ass ALL big Java Frameworks and Companies are maintaining their own Maven Repository Servers. They push their artifacts first to their own MVN Server and then 1 or 2 weeks later the artifact appears on search.maven.org.
  • Not really centralized. Because of the previous point. There are at least 30 different public maven repository servers out there, who are not mirroring search.maven.org, but host other artifacts who are not available on search.maven.org. Java developers have to google for the right repository server with the desired artifacts.
  • Maven is a very complex tool! A pom.xml file can inherit from another pom.xml file. And a pom.xml file can include other pom.xml files. All that leads to high complexity and sometimes to resolution errors. Another side effect of this architecture is that resolving transitive dependencies this way is very slow, because different xml files have to be downloaded and parsed to build a complete model for 1 single artifact.
  • Maven violates the “Single Responsibility” pattern. Maven is not only doing dependency management! It is doing all kind of things which has nothing to do with dependency management. For example executing tests, generating reports, generating JavaDoc, doing deployments and many other things which are, in other languages, done by other tools.
  • Not sure why an artifact/package needs a “GroupId” AND an “ArtifactID” AND a version number. All other package managers are satisfied with a name and a version. KIS!
  • Having 5 lines of XML code for 1 single dependency is kind of overkill!

Maven is by far the most complex package manager I know. And know many of them!

Lein (Clojure)

Leiningen is the package manager for Clojure. It uses a maven repository as backend. Lein itself has the same scope as Maven. It’s not just doing dependency management! At the same time it’s a build tool. Dependencies are defined in a project.clj file. Here an example:

(defproject leiningen.org "1.0.0"
  :description "Generate static HTML for http://leiningen.org"
  :dependencies [[enlive "1.0.1"]
                 [cheshire "4.0.0"]
                 [org.markdownj/markdownj "0.3.0-1.0.2b4"]]
  :main leiningen.web)

Defining a dependency fits here into 1 line. And “GroupId” is not required for Clojure packages! Leiningen is using 2 sources for the dependency resolution.

The dependencies in a project.clj file can be explicitly fetched with this command:

lein deps

But they will be fetched anyway at compile time.

Publishing a package on clojars.org is pretty easy, after you signed up.

lein pom
scp pom.xml mylib.jar clojars@clojars.org:

Pros

  • The artifacts are signed!
  • It simplifies the dependency definition of maven.
  • It’s easy to publish new artifacts.
  • It’s easy to learn how it works.
  • It allows to reuse Java artifacts from other maven repositories.

Cons

  • It violates the “Single Responsibility” pattern. Same as Maven.
  • Licenses are not mandatory.
  • No Mirrors.

Leiningen is not perfect, but it is doing many things right.

Maven based package managers

There are many languages running on the JVM and most of them have their own package managers. But all of them are more or less based on Maven repositories. That means they all use a Maven Repository as backend and just have a better client (command line tool) and different conventions to deal with the backend.  These are the package managers from this family:

Ivy, SBT and Gradle support own repository types, beside Maven Repositories. But it kind of makes sense that they all access Maven Repositories, because they all want to reuse existing Maven artifacts.

Conclusion

All discussed package managers are open source and can be hosted internally as well. All of them are easy to get started with, except for Maven which is more complex and less fun. I’ve created a comparison matrix to highlight the differences with the most important attributes on the x axis. Unfortunately, there is no single package manager that offers every desired feature:

Comparison Package Managers

There is no clear winner and imho, there is no perfect package manager 😦 There are of course more package managers, such as CPAN (Perl) and Nuget (.NET), and we hope to be able to cover them in the future. You can find a complete list of package managers here. We are working continuously to add additional languages to our API and make VersionEye available for every software developer in the world!

What is your favorite package manager? And why? And which features are missing in current package managers? I’d love to hear your thoughts! Leave a comment or send me a tweet.

Follow the discussion on hacker news and Reddit.

if you read this far, you might want to monitor your packages on VersionEye 😉

Don’t miss tomorrow’s Tech Talk: “Continuous updating of software libraries in the cloud”

VersionEye founder & CEO, Robert Reiz, will talk about „Continuous updating of software libraries in the cloud” on August 6th in Berlin Mitte. The forum discussion is presented by SIBB Forum Saas & Cloud Computing.

All software developers who want to know how they can keep their software projects up to date, and want to find out quickly and easily what dependencies are out-dated, should learn more about our convenient SaaS solution. Using VersionEye you will be informed about new versions of your software libraries to be always up to date.

Computer scientist Robert Reiz founded his third start-up “VersionEye” in Silicon Valley in 2012. Previously, he was Chief Architect and Managing Director for “WildGigs” in San Francisco and founder & CEO of the Java consulting house “PLOIN” in Mannheim. Today VersionEye has two locations in Berlin and San Francisco and a team around the globe.

We are looking forward to your forum participation! Snacks and beer are available for free, while supplies last. Please register here: http://www.amiando.com/COYTZWA.html

date: August 6, 2013
time: 5 pm to 7 pm
location: VersionEye GmbH, Brunnenstr. 181, 1. HH. 1. OG, 10119 Berlin

 

Image

VersionEye at “meet the cloud” – meet & greet with open it berlin

ImageVersionEye founder & CEO, Robert Reiz, will talk about „Continuous Updating” on August 22nd at the “meet the cloud” meet & greet, presented by open it berlin.

All software developers who want to know how they can keep their software projects up to date, and want to find out quickly and easily what dependencies are out-dated, should learn more about our convenient SaaS solution. Using VersionEye you will be informed about new versions of your open source software libraries to be always up to date.

We are looking forward to seeing you there! The participation is free of charge, but please send an email to Michael Stamm for your registration: stamm@tsb-berlin.de or register here: http://www.open-it-berlin.de/veranstaltungen/meet-greet-mit-open-it-berlin-meet-cloud

date: August 22, 2013
time: 3 pm to 6 pm
location: TSB Innovationsagentur Berlin GmbH, Ludwig Erhard Haus, 5. OG, Fasanenstr. 85, 10623 Berlin

GitHub Single Page App

VersionEye stands for Continuous Updating. Therefore we did some more work on our GitHub Single Page App. The layout looks a little different and you are able to see more repositories on your screen. With a single click you can display the branches and with one more click you can add them to your VersionEye project list.

Beside this we changed the sorting once again, descending to the last commit in the repositories.

Let us know what you think!

Image

New date for event: „Continuous updating of software libraries in the cloud”

New date for event: „Continuous updating of software libraries in the cloud”

VersionEye founder & CEO, Robert Reiz, will talk about „Continuous updating of software libraries in the cloud” on August 6th. The forum discussion is presented by SIBB Forum Saas & Cloud Computing.

All software developers who want to know how they can keep their software projects up to date, and want to find out quickly and easily what dependencies are out-dated, should learn more about our convenient SaaS solution. Using VersionEye you will be informed about new versions of your software libraries to be always up to date.

Computer scientist Robert Reiz founded his third start-up “VersionEye” in Silicon Valley in 2012. Previously, he was Chief Architect and Managing Director for “WildGigs” in San Francisco and founder & CEO of the Java consulting house “PLOIN” in Mannheim. Today VersionEye has two locations in Berlin and San Francisco and a team around the globe.

We are looking forward to your forum participation! Snacks and beer are available for free, while supplies last. Please send an email to Astrid Vieth for your registration: astrid.vieth@sibb.de

date: August 6, 2013
time: 5 pm to 7 pm
location: VersionEye GmbH, Brunnenstr. 181, 10119 Berlin

Neuer Termin: “Continuous Updating für Software-Bibliotheken aus der Cloud”

Neuer Termin: “Continuous Updating für Software-Bibliotheken aus der Cloud”

VersionEye Gründer und Geschäftsführer, Robert Reiz, spricht am 6. August 2013 zum Thema „Continuous Updating für Software-Bibliotheken aus der Cloud“, präsentiert vom SIBB Forum Saas & Cloud Computing.

Alle Software-Entwickler, die wissen wollen, wie sie ihre Software-Projekte up to date halten können und schnell und unkompliziert herausfinden möchten, welche Abhängigkeiten out-dated sind, sollten mehr über unsere praktische SaaS-Lösung erfahren. Mithilfe von VersionEye werden Sie über neue Versionen Ihrer Software-Bibliotheken informiert und sind so immer auf dem neuesten Stand.

Diplom-Informatiker Robert Reiz gründete sein drittes Start-up „VersionEye“ 2012 im Silicon Valley. Zuvor war er Chief Architect und Managing Director für „WildGigs“ in San Francisco und Gründer und Geschäftsführer des Java Consulting Hauses „PLOIN“ in Mannheim. Heute hat VersionEye zwei Standorte in Berlin und San Francisco und wird von einem internationalen Team unterstützt.

Kommen Sie vorbei, wir freuen uns auf Sie und Ihre Fragen! Snacks und Bier stehen kostenlos zur Verfügung, solange der Vorrat reicht. Bitte melden Sie sich per e-Mail bei Astrid Vieth an: astrid.vieth@sibb.de

Datum: 06. August 2013
Zeit: 17:00 bis 19:00 Uhr
Ort: VersionEye GmbH, Brunnenstraße 181, 10119 Berlin