bulknews.typepad.com

Starman 0.2000, delayed and --preload-app

I just shipped Starman 0.2000 to CPAN. This release contains bug fixes from Graham Barr and Patrick Donelan on rare 100% cpu spin loops and a bug in Keep-alive handling on HTTP/1.0, but also has a major incompatible feature: preloading apps.

Preloading applications

plackup and all of its handlers, by default loads your application before running it on the server. So if you say:

  plackup -s Starlet myapp.psgi

the myapp.psgi file is compiled in the master (parent) process and then the Starlet server preforks off children to run the compiled application. This saves a lot of memory in copy-on-write friendly environment, but might cause problems, if you want to initialize resources in the myapp.psgi compilation time, such as opening sockets or doing database connections.

Delayed loader

To solve this problem and make Starman's default safer, I introduced a new loader for Plack: Delayed. This is close to the Shotgun loader but it delays the compilation of the application until the first request time. It also passes the callback web servers can call to initialize the application earlier than the runtime, which Starman 0.2000's starman exeuctable now makes use of by default.

All plackup handlers still use the default loader by default, so if you want to use this loader with Starlet, you should say:

  plackup -L Delayed -s Starlet myapp.psgi

but Starman's starman command automatically adds -L Delayed by default, so you don't need to. To continue preloading the app in the master to save memory (and you're sure that loading your app doesn't cause any shared resource issues), you can say:

  starman --preload-app myapp.psgi

to keep the behavior pre 0.2000. Alternatively you can also say:

  starman -MCatalyst -MDBIx::Class myapp.psgi

to load modules in the master, but delays the compilation of application until the per-children init hook.

BTW with this change the default behavior of Starman becomes the same with Rack's Unicorn. Read more about this in starman -h.

cpanminus 1.0 is coming

I'm now in Vienna, Austria to join the Perl QA Hackathon 2010.

The hackathon is sponsored by Vienna perl mongers and 123people, and all the accommodation and flights are paid by them. I appreciate their help to make CPAN QA/Testing toolchain better.

On day 1 and 2 i've been working on fixing bugs and wishlists for cpanminus supposedly, and just now i shipped cpanminus 0.999_01, the first release candidate for cpanminus 1.0. I'll tell you what's new for this release.

Less Features

The name cpan "minus" has just started as a counter joke for the CPANPLUS client, and adding features to what's supposed to be config-free and deps-free and tiny/light/whatever just doesn't sound right. So i deleted all the developers-oriented features from cpanminus, such as:

• plugins support
• --recent command

Some of you, like only 8-10 people on the planet who enjoyed the early adaptation of the plugin system might be disappointed that the plugins are not supported anymore. No worries, i'm starting to write another CPAN client (temporary codenamed "jam" :)), that works on top of cpanminus and supports all of the plugins we developed for cpanminus, just as they are. 

Until that happens, you can continue using the cpanminus 0.9934, which is the last version of the cpanm that supports plugins.

More metadata retrieval

I refactored the code base so cpanm always resolves the package name to distribution using CPAN Meta DB and then use search.cpan.org's metadata index service before downloading the tarball. This makes lots of command run faster, especially the --skip-installed option.

Prompts on failures

One of the thing that annoys you when installing CPAN modules is the test failures. I mean, testing is a great culture of Perl and CPAN but when the test fails, CPAN clients refuses to install without the 'force' mode, and it's so annoying.

cpanminus default behavior is consistent to other clients as CPAN.pm or CPANPLUS: the module is not installed when the test fails (unless --force is specified). 

In 1.0 we add --prompt option, which gives you the chance to fix the failures. When you run cpanm with --prompt option (which you can set by default if you set it in PERL_CPANM_OPT environment variable), and the test fails, it asks you:

  Testing CPAN-Test-Dummy-Perl5-Build-Fails-1.03 failed.
  You can s)kip, r)etry, f)orce install or l)ook ? [s] 

It might be obvious, but "skip", the default choice is to skip installing the module and proceed with other dependencies. "retry" re-runs the test again to see if it was a temporary failure. "force" would, obviously, force install the module, which is very handy if you're really sure that it's just a bad test failure that doesn't matter.

Also, the "look" command is pretty nice. It allows you to open the working directory with the shell, pretty much like CPAN.pm's 'look' command, to see if there's any missing configuration in your system, and you can fix there. And you exit the sub shell and type 'r' to retry installing the module.

Of course you can hit Control-c in the prompt to bail out the entire installation process.

Bailing out when dependencies fail

This is related to the previous section, but cpanminus 1.0 changed the behavior when one ore more dependencies failed to install.

Say, you're trying to install Plack and all of its dependencies. Imagine one of its dependencies, like HTTP::Body, failed the test and hence not installed.

Previously, cpanminus still continues the build process of Plack itself, and if Plack doesn't define an unit test that uses HTTP::Body, the test accidentally passes and cpanminus succeeds to install Plack. The end result is that Plack is installed to your system, however when you're trying to use Plack::Request, you'll get a runtime error like "Can't locate HTTP/Body.pm in @INC ...".

This is a broken installation and should be avoided.

cpanminus 1.0 will bail out the build process when one or more dependencies failed to install, by default. With --prompt option enabled, cpanminus will ask you:

  Installing the following dependencies failed:
  ==> HTTP::Body
  Do you want to continue building Plack-0.9928 anyway? [n] 

So if you're really sure that the dependency failure doesn't matter, you can still continue installing the module by hitting 'y'. Of course you can also use the --force option to force install anything, like you've always been able to.

(Note that Plack thing is just an example, and since Plack has tests that uses HTTP::Body, it won't happen :D)

Minor improvements

These are the major improvements and changes in cpanminus 1.0 but there are more minor improvements. Such as setting a proper exit code, better error messages, uninstalling shadows by default and being able to specify your favorite mirror with --mirror. See the Changes file or the newly added `man cpanm` to see all the details.

I'll start working on a new CPAN client, as well as specifying cpanminus API so that the new client can make use of it. Stay tuned :) 

Sunaba - Perl web apps in the cloud

It is a great pleasure that my 4 hour hack last night lets me now introduce: Sunaba the perl sandbox environment to run any PSGI/Plack applications.

Since we started the Plack project, it has always been my hope that someday we can port this PSGI interface on the "Cloud" infrastructure like Google AppEngine or Heroku for Ruby Rack applications.

The hardest part to implement that is to create a sandbox perl runtime environment, and fortunately Dan Kogai did the hardest part already. Like an year ago he created this lleval API which runs your code in a sandbox (chroot + FreeBSD ptrace) and Shibuya.pm folks tried to jailbreak it, with some interesting results. You can read the slides how he implemented this sandbox.

I wouldn't say this is completely secure, but it works pretty well - you can't open local files, run arbitrary XS code nor system("rm -fr /"). while(1) would be killed in 1 second timeout. Meanwhile there are some popular modules besides core modules installed such as LWP, YAML and JSON.

My Sunaba application is just a web frontend with Tatsumaki + Twiggy + SQLite, that basically allows you to create and edit applications on the browser, as well as a runner PSGI application that serializes the current PSGI $env and sends it into Dan's box to get the response back.

All the service restrictions apply, and NO WARRANTY. But it's a good playground to toy with the PSGI interface. Well, Sunaba means "sandbox" in Japanese so you can play and make your hands dirty :)

I can think of more enhancements to this, such as minimal/tiny framework support (i wonder what's the source code size limit of Dan's service - maybe 8K or something?) and the ability to install middleware components etc. etc. Also, for now Kogai-san's server is the best I can think of, but theoretically we can run the apps on other sandbox environments such as CodePad or Ideone. The only difference is that Kogai-san is my friend and it's less likely to be banned for the excessive use :)

Frequently Asked Questions

• Is there a limit for the code size? - Sunaba doesn't have any limit on the code size at this moment but the backend server appears to limit the source code size up to something like 4K.
• How do you determine who can edit the application? - It's currently based on your IP address and User-Agent string. Yes, it's not a very good authentication.
• Are the application URLs permanent? - No, I plan to clean up applications periodically. It's a sandbox after all, you know :)