PSGI - Perl WSGI

Info: if you come from Google and want to know what PSGI and Plack are: take a look at our PSGI/Plack project web page that has links to PSGI spec and FAQ, and then Plack git repo for the actual code.

HTTP::Engine is the best thing that happened in the Perl web application development land lately. It's a port of Catalyst::Engine and takes all the goodness from Python's WSGI and Ruby's Rack interface, as well as using Moose's Role-base abstraction but using Any::Moose under the hood so it runs fast at runtime under Moose but still boots up fast with less memory in CGI using Mouse as a backend. This should be great for everyone.

Well.

Did you smell something fishy about the last paragraph? If your friend cooks something for you and says "Yeah I learned a great technique from Chinese cookbook and trying to apply that with raw fish Sushi stuff mixed into some mexican Chicken Tacos thing so this should be pretty good." you'll probably want to avoid eating it. HTTP::Engine doesn't do the exact same thing but something similar.

A few months ago I was writing AnyEvent-based HTTP::Engine adaptor and swore a couple of times during coding it: there are too much abstractions going on and as an adaptor implementor you need to know which hook you should implement and which parameters should pass when, but the class hierarchy is still monolithic so your adaptor code totally depends on the HTTP::Engine's implementation. 

That really sucks.

The basic motto of HTTP::Engine - "WSGI/Rack equivalent for Perl" is quite wrong. WSGI/Rack has the specification which is just a protocol between adaptors and an application. WSGI implementation calls the application, which is just a Python's callable object (could be an iterator, callable or just a method) with single big 'env' hash that contains everything and start_response callback. Rack omits the start_response and instead excepts an array of 3 values as a response (because Ruby has this wonderful thing called Iterator), but the basic design is the same.

HTTP::Engine is monolithic and the adaptor's implementation is so much tied to its sometimes badly designed API or over-engineered implementation. Last night tokuhirom, Yappo and I brainstormed this thing on IRC and agreed to drop those engine specific drivers (which is called Interface:: and Builders in HTTP::Engine) from HTTP::Engine and clone WSGI/Rack hash and callback based protocol as much as possible. HTTP::Engine API will stay the same, but now HTTP::Engine will become a thin wrapper on top of PSGI (Perl WSGI - a little goofy but straightforward name, right?). PSGI is a specification (gateway protocol), and we'll probably create PSGI implementation for HTTP::Engine, probably named Plack.

The separation of spec and implementation is important here: JavaScript's Jack does it pretty nice - JSGI is the spec and Jack is the implementation including middleware. Ruby's Rack does both, which could be sometimes confusing.

Anyway, this should make the whole HTTP::Engine stack much easier to implement and allow us to reuse tests in different environments as much as possible.

I proposed this on #moose and #http-engine IRC channels this morning, and got a response like below:

10:33 rjbs: sounds cool
10:33 mst: miyagawa: good plan.
10:33 confound: tokuhirom++ miyagawa++ excellent
10:35 stevan: tokuhirom++ yappo++ miyagawa++ # very nice
10:48 autarch: overall, I think this is a good idea

So this is promising.

There were some things to bikeshed, based on some buggy reference code and misunderstandings, but that actually proves the benefit of this PSGI thing. If you think the reference implementation sucks, you can write your own. Actually we think PSGIRef should not be used by the real applications and there should be more implementations written by experts on FastCGI/ModPerl/Gearman/Mojo/Perlbal and alike.

We'll keep the HTTP::Engine API compatible, but writing your own API wrapper should be much easier as well, and writing adaptors for frameworks like Catalyst, Jifty, Ark, Angelos and Squatting would still be pretty easy. Also, some of them already use or have an adaptor for HTTP::Engine which doesn't need to change anyway.

By the way, the fact I proposed at and got positive feedbacks from #moose IRC channel might have given you an impression that PSGI has something to do with Moose, e.g. PSGI implementation is based on Moose Roles, or implementations should use Moose. Absolutely not. The fact I did it in #moose is that the channel has lots of perl rockstars and many eyes who are keen and insightful on this kind of topic. PSGI protocol and reference implementations would not be tied to Moose, or more importantly, any modules at all.

In fact, we shouldn't worry about the whole Moose vs non-Moose battle with this design, since those who want to run a small app under CGI can write CGI.pm based plain hash adaptor without buying more RAM, while large scale web apps developer can make Moose and FastCGI based adaptor to take the advantage of their 64bit machine with 4GB of RAM.

tokuhirom++ for bootstrapping the whole thing by JFDI the reference code. We should start dumping out the specification soon, based on this reference: there's a couple of things we need to think about how to adopt Python's callable, Ruby's iterator or streamable IO object etc. but hopefully we can come up with a pretty simple but flexible and extensible API for those things.

Hope we could come up with the result soon in YAPC::Asia 2009 hackathon at Tokyo. Keep your eyes on, or join us on the development on #[email protected]

Comments

[www.simplicidade.org]

Excellent news!

I tried to port HTTP::Engine to AnyEvent to use Catalyst apps inside AnyEvent long-running processes, as an admin interface, or webservices interface, and gave up.

I wrote AnyEvent::Mojo because of that.

Nice to see that I'll have some alternatives.

[miyagawa]

When did you try to port that? I actually implemented HTTP::Engine that runs on AnyEvent: http://github.com/http-engine/HTTP-Engine/commits/async and it's being used in my app Remedie: i use it every day :)

It's not really merged into git master nor released to CPAN for the exact reason I posted in this entry, though.

[Randy J. Ray]

I've been looking at an adaptor for HTTP::Engine to my XML-RPC server class. I'll be watching this effort, to see if I want to use it (either instead or in addition to).

[btrott]

I love this... I think it's exactly the right answer to what I was looking for a couple of weeks ago for a really simple webapp (which we talked about).

[Mark Stosberg]

This approach seems sensible to me, and I look forward to seeing it progress.

[miyagawa]

Mark, glad that you're interested. Over this recent few days we've got a ton of updates: got already working CGI::Application, or rather, any CGI scripts running on PSGI implementation like HTTP::Server::Simple, Mojo or FCGI.

Its implementation is a little hacky just like HTTP::Request::AsCGI to pipe env and redirect standard i/o, but i actually have a plan to update CGI.pm to directly support PSGI for the much better performance. Go take a look at Plack http://github.com/miyagawa/Plack and let us know what you think!

[awwaiid]

I like this... it feels more natural to me and I think I could use it to make a Continuity::Adapt::HttpEngine backend (or perhaps ::Plack?) much more naturally.

--Brock