Status of cpanmetadb

2011.11.07

Status of cpanmetadb

Google AppEngine went out of beta and changed its pricing model today.

cpanmetadb the backend database that cpanm uses, is affected by this change. Basically because it heavily uses GAE datastore, it begins to throw OverQuotaErrors this morning.

I've made a change to it to use MetaCPAN API instead of data store. It seems to work well for most of the modules, but because MetaCPAN handles modules and scripts a little differently from just parsing PAUSE 02packages file, there are some mismatches for some queries. Namely, there are some modules that do not return expected results, such as version or warnings returning the perl-5.14.2 as its distribution, etc.

I dont' have time to fully migrate this over to full ElasticSearch based query, or to have another cpan packages DB running on some other host - so far cpanminus has an option to fallback to search.cpan.org screen scraping, which used to work fine before our switch to cpanmetadb.

I will keep you posted as I make more updates to stablize everything.

UPDATE: Nov 9th 3pm PST: I set up a new instance of cpanmetadb using Perl, Tatsumaki and Twiggy and on-memory database (aka perl hash :-)) and run it on my linode instance (might take a bit while before DNS update gets propagated), and switched cpanmetadb to redirect all the requests to the instance.

This seems to be working well for most clients including cpanm. If your client doesn't handle redirections automatically, you might have to change that to follow redirects (like -L option for curl). Let me know if you have any troubles with this switch.

Nov 7, 2011 1:51:33 PM

4 Comments

I bet Google keeps on updating and changing their Google AppEngine to make it more fast and reliable. It's simply like searching for the best Tutor for your child.

Posted by: Account Deleted | 2011.11.10 at 21:31