Improved Crawlers for Maven – jCenter, Atlassian, Adobe

VersionEye is crawling a bunch of repositories like RubyGems, NPMRegistry, Packagist and many more. For Java we crawled for long time only search.maven.org. Simply because it provides an maven index to download and that makes it really easy to get the data into our database! Other maven repositories don’t provide a maven index and that means we need to do some heavy xml parsing! Simply doubling the number of our EC2 instances is not an option because currently we don’t swim in Money! In the last weeks we worked hard on performance improvements for the non-maven-index maven crawlers. And because the work was successful we are happy to announce that VersionEye supports this maven repositories as well:

Now VersionEye tracks 123K Java projects with 1.37 million artefacts!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s