« Common Lisp "style" redux | Main | IASW-2005 »

2005-08-19

Wilbur and HTTP client access

Let me preface this all by saying that even though most people by now probably think that Wilbur2 is never going to materialize, I have - from time to time - been working on the last missing pieces.

Currently, I am worrying about HTTP client functions (and other "URL access" functions), these are an important part of Wilbur's "Data Source API". Several options to implement this exist:

  1. The "classic" Wilbur implementation has its own HTTP client, a piece of code that's quite complicated. It has been read-time conditionalized for Allegro, MCL, etc., and contains quite a bit of platform-dependent code.

  2. As an alternative to #1, I could take Trivial-HTTP and add some just to make sure that things like chunking and redirects work (OK, so this is essentially what #1 does, except that it also does some fancy header parsing).

  3. Lately, on OpenMCL, I have been using a piece of code that essentially calls curl in a separate process and pipes the output back to the CL process, to be read as a stream. It was trivial to bolt onto the existing interface so that it is call-compatible with #1.

  4. As an alternative to #3, I could use cl-curl, which basically calls libcurl directly. I haven't yet studied the source code enough to understand whether I would have to play with processes (ahem, threads in CL) to get the same nice "asynchronous" behavior I now get with #3. Generally, I am not crazy about linking with foreign libraries if I need to provide/use some "glue code" (there is some in cl-curl); I already do some of that on MacOS X to determine the proxy settings for HTTP, and would prefer that I didn't have to.

There are other alternatives besides the ones above. For example, AllegroServe (well, actually Portable AllegroServe is the one I have used in some projects) has an HTTP client but earlier experience with that was less than pleasant.

I love Common Lisp, but I have to say that Java/Python/etc. developers don't generally have to struggle with these types of issues. The general observation is that we are still missing a lot of stuff that should be considered "standard" (not standard parts of the language, but standard in the sense that every developer does not have to solve these problems over and over again).

Posted by ora at 11:37

Comments

When I switched to SBCL, and found that Wilbur's DB loading didn't work (because it wasn't running on one of the Lisps you mentioned), I hacked up something called wilbur-web. This uses trivial-http (with minor extensions to send Accept: headers), is a mere 40 lines long, and returns a DB from a URL.

It's behaved impeccably for me ever since, though I've only really tested it on SBCL.

http://cliki.net/wilbur-web should have it.

Posted by: Rich at August 19, 2005 07:10 PM