Digg just released a prototype of their optimized data streams.  This this is pretty cool.  It uses MIME multipart HTTP responses to return a stream of responses, dispatching each one as it comes in.  Bugzilla has been using MIME multipart for a while, though only to serve a short “Bugzilla is searching for your bugs” message before returning the actual results.

I’m still digging through it but it looks like a great way to deal with large numbers of resources at pageload time.  As an example, you can batch 50 individual profile images in a single HTTP connection roundtrip to populate your frontpage, versus having to serve 50 individual images, or having to manually stitch them into a single image on the server side.

Assuming you can scale it well on the server, you could potentially multiplex a few long-running API calls on a single stream as well.  As each one is ready, you could then throw it down the pipe and deal with it on the client.

The concept is cool.  There are some limitations obviously, but it’s a fantastic way to deal with bulk data transfer.

More at ReadWriteWeb and Digg the Blog.

Read full post