Archive for the ‘web’ Category

A PubSubHubbub to XMPP Gateway

Wednesday, September 9th, 2009

At first glance, both rssCloud and PubSubHubbub have an interesting shortcoming that makes them difficult to use for desktop feed readers. Since both of them require HTTP callbacks to a publicly accessibly endpoint, a user is required to open up a port on their firewall.

It turns out that a subtle difference in the specifications gives PubSubHubbub a big edge in this case. While rssCloud requires your callback endpoint to live at the IP address you make your request from, PubSubHubbub allows you to subscribe any endpoint you wish by specifying a hub.callback url.

So how do we turn this into a real-time feed for desktop clients? Simple: we implement a PubSubHubbub subscriber on a publicly-available, always-on server that receives PubSubHubbub update events and wraps them in XMPP. The XMPP events are transmitted to the desktop client, where it can then process them as if it received the callbacks directly.

The server application doesn’t need to be smart. Only the “subscribe” and “publish” modes of PubSubHubbub’s protocol are required. All it needs to do is correctly route the update subscriptions to the correct XMPP account. In fact, with Google AppEngine’s new XMPP support, you can this in a few dozen lines of code, as I’ve done here:

A PubSubHubbub to XMPP gateway, hosted on Google AppEngine

Try out the gateway by entering your XMPP ID on the main page. This will give you a callback URL that you can use on Google’s main PubSubHubbub hub. Enter the URL for any PubSubHubbub-enabled field as the topic.

The code is simple, though not very robust:

@SuppressWarnings("serial")
public class Subscribe extends HttpServlet {
	@Override
	protected void doPost(HttpServletRequest req, HttpServletResponse resp)
			throws ServletException, IOException {
		resp.setStatus(204);
		XMPPService xmpp = XMPPServiceFactory.getXMPPService();
		JID jid = new JID(req.getPathInfo().substring(1));

		byte[] buffer = new byte[10 * 1024];
		req.getInputStream().read(buffer);
		xmpp.sendMessage(new MessageBuilder().withBody(
				"Got update: " + new String(buffer))
				.withRecipientJids(jid).build());
	}

	public void doGet(HttpServletRequest req, HttpServletResponse resp)
			throws IOException {
		resp.setStatus(200);
		resp.setContentType("text/plain");

		XMPPService xmpp = XMPPServiceFactory.getXMPPService();
		JID jid = new JID(req.getPathInfo().substring(1));

		if (req.getParameter("hub.mode").equals("subscribe"))
			xmpp.sendMessage(new MessageBuilder().withBody(
					"Subscribing to " + req.getParameter("hub.topic"))
					.withRecipientJids(jid).build());
		else
			xmpp.sendMessage(new MessageBuilder().withBody(
					"Unsubscribing from " + req.getParameter("hub.topic"))
					.withRecipientJids(jid).build());

		resp.getOutputStream().print(req.getParameter("hub.challenge"));
		resp.getOutputStream().flush();
	}
}

Postscript: I really hope that PubSubHubbub gets a new name.

PubSubHubbub vs. rssCloud

Monday, September 7th, 2009

UPDATE: I’ve written a PubSubHubbub-to-XMPP gateway that solves some of the issues of running a real-time feed reader behind a firewall.

UPDATE 2 rssCloud has a serious vulnerability that needs to be addressed in the protocol. I’ve linked some security recommendations here that rssCloud hubs should implement as soon as possible.

These last few months have brought us not one, but two RSS-to-real-time protocols: PubSubHubbub and rssCloud. While rssCloud has been “around” for a while, it never saw much adoption or interest until recently.

As a developer, the important question is: which of these two protocols should I focus on?

When you compare the two protocols technically, you find that there are some similarities (UPDATE: see here for a more in-depth comparison of the APIs):

  • Both PubSubHubbub and rssCloud allow the hub to live on a different server than the server that is providing RSS. This lets the complexity of both of these protocols to live in a black box somewhere else, managed by someone who cares more about getting the details right.
  • Both offer a fairly simple publisher “ping” notification for publishers. An rssCloud client can make a simple POST request to the specified cloud server, which is then verified by the server to ensure that the update was real (alternatively, rssCloud can use XML-RPC or SOAP, neither of which are in fashion right now). PubSubHubbub has a very similar POST operation with very similar semantics.
  • Both offer simple APIs on the hub for subscribing to feeds. PubSubHubbub offers an unsubscribe option, while rssCloud times out subscriptions after 25 hours (the client is expected to re-subscribe after 24).

There are some significant differences between the two protocols, however:

  • PubSubHubbub supports RSS and Atom out of the box. rssCloud does not support Atom right now, as noone has defined how it would look inside of an Atom feed.
  • PubSubHubbub provides “fat pings” to clients, while rssCloud only provides basic notification updates. A PubSubHubbub subscriber can keep tabs on a feed entirely through the ping notifications, allowing it to skip polling of any feed that supports the update protocol. rssCloud requires the subscriber to re-poll the feed after receiving a ping. The “fat ping” has the advantage of saving the feed publisher bandwidth, since clients aren’t downloading the same repeated feed entries time after time, and potentially CPU cycles, since the feed publisher only has to generate a single feed output for the hub rather than for all of its clients (this can be mitigated by caching the generated feed). The fat ping requires more work on the part of the hub, however, as it needs to detect which parts of the feed have changed and push those parts into the subscriber notification dispatch queue.
  • PubSubHubbub lets you subscribe any endpoint you like (with some intelligence to prevent you spamming pings to arbitrary hosts). rssCloud infers your endpoint hostname from the IP address of the request, requiring your subscription logic to live on the same servers as your ping endpoints.

Back to the question: which of these protocols should I focus on? The answer probably depends on what you are doing.

  • If you are a publisher that publishers both RSS and Atom feeds, it’s trivial for you to support pinging rssCloud and PubSubHubbub hubs. There’s nothing stopping you from doing it now – just figure out which hubs to use. If you use FeedBurner and PingShot, Google has already cloud-enabled your blog for you.  If you want to control your own hub, you’ll probably want to pick an off-the-shelf one. PubSubHubbub is likely the best choice here as it both saves you bandwidth and gets you real-time support in FriendFeed.
  • If you are planning on writing a hub, you’ll probably want to start with rssCloud. Its implementation will be simpler than PubSubHubbub as all it does is redistribute ping notifications.
  • If you are a feed reader or a content spider, you’ll probably have to implement both. I believe that PubSubHubbub gives you the biggest bang for the buck now, as it’s supported by nearly all of the Google feed properties: FeedBurner (the Atom/RSS intermediary choice for a significant number of self-hosted blogs), Blogger (millions of blogs) and Google Reader feeds. It’s also supported by LiveJournal (which lists 20+ million blogs on its homepage).  rssCloud is fairly new, but it managed to score a big integration with wordpress.com (7.5 million blogs, according to their own blog). Unfortunately, as not all of the big sites have implemented both, you’ll have to deal with two technologies for the time being.

After researching both of the technologies in-depth, I’d say that PubSubHubbub is the better technology overall.  While more complex to implement for hubs, it offers far more to feed readers and publishers in terms of bandwidth savings and real-time updates.  For companies doing content analysis, PubSubHubbub is a huge win: it brings the power of the Twitter firehose to RSS. No matter which technology you choose, however, you’ll be getting your RSS feed updates far more often.  It might even allow the next real-time technology to be built on an open XML feed rather than a proprietary company’s servers.

A quieter window.name transport for IE

Tuesday, July 28th, 2009

Using window.name as a transport for cooperative cross-domain communication is a reasonably well-known and well-researched technique. I came across it via two blog posts by members of the GWT community that were using it to submit GWT FormPanels to endpoints on other domains.

For our product, I’ve been looking at various ways we can offer RPC for our script when it is embedded in pages that don’t run on servers under our control.  Modern browsers, like Firefox 3.5 and Safari 4.0 support XMLHttpRequest Level 2.  This standard allows you to make cross-domain calls, as long as the server responds with the appropriate Access-Control header.  Internet Explorer 8 supports a proprietary XDomainRequest that offers similar support.

When we’re looking at “downlevel” browsers, like Firefox 2/3, Safari 2/3 and IE 6/7, the picture isn’t as clear. The window.name transport works well in every downlevel browser but IE6 and 7. In those IE versions, each RPC request made across the iframe is accompanied by an annoying click sound. As you can imagine, a page that has a few RPC requests that it requires to load will end up sounding like a machine gun. The reason for this is IE’s navigation sound which plays on every location change for any window, including iframes. The window.name transport requires a POST and a redirect back to complete the communication, triggering this audio UI.

I spent a few hours hammering away on the problem, trying to find a solution. It turns out that IE6 can be fooled with a <bgsound> element that masks the clicking sound. This doesn’t work in IE7, however. My research then lead to an interesting discovery: the GTalk team was using an ActiveX object named “htmlfile” to work around a similar problem: navigation sounds that would play during their COMET requests. The htmlfile object is basically a UI-disconnected HTML document that works, for the most part, the same way as a browser document. The question was now how to use this for a cross-domain request.

The interesting thing about the htmlfile ActiveX object is that not all HTML works as you’d expect it to. My first attempt was to use the htmlfile object, creating an iframe element with it, attaching it to the body (along with an accompanying form inside the htmlfile document) and POSTing the data. Unfortunately, I couldn’t get any of the events to register. The POST was happening, but none of the iframe’s onload events were firing:

if ("ActiveXObject" in window) {
    var doc = new ActiveXObject("htmlfile");
    doc.open();
    doc.write("<html><body></body></html>");
    doc.close();
} else {
    var doc = document;
}

var iframe = doc.createElement('iframe');
doc.body.appendChild(iframe);
iframe.onload = ...
iframe.onreadystatechange = ...

The second attempt was more fruitful. I tried writing the iframe out as part of the document, getting the iframe from the htmlfile and adding event handlers to this object. Success!  I managed to capture the onload event, read back the window.name value and, best of all, the browser did this silently:

if ("ActiveXObject" in window) {
    var doc = new ActiveXObject("htmlfile");
    doc.open();
    doc.write("<html><body><iframe id='iframe'></iframe></body></html>");
    doc.close();
    var iframe = doc.getElementById('iframe');
} else {
    var doc = document;
    var iframe = doc.createElement('iframe');
    doc.body.appendChild(iframe);
}

iframe.onload = ...
iframe.onreadystatechange = ...

I’m currently working on cleaning up the ugly proof-of-concept code to integrate as a transport in the Thrift-GWT RPC library I’m working on. This will allow us to transparently switch to the cross-domain transport when running offsite, without any visible side-effects to the user.

Mix’s Gestalt project is on the wrong track

Wednesday, July 22nd, 2009

I came across the Mix Gestalt project tonight and I thought I’d share some thoughts. It’s a bit of script that effectively sucks code snippets in languages other than Javascript out of your page and converts them to programs running on the .NET platform.

While interesting, it has a number of drawbacks that make it far less interesting than the HTML5-based approach that works in the standards-compliant browsers based on WebKit, Gecko and Opera, as well as the improved IE8.

First of all, it has to bootstrap .NET into Firefox (or whichever browser you are running it in).  This adds a few milliseconds to your page’s cold load time if it’s not already loaded. In the day and age of fast websites, any additional page time is just a no-go.

Once it’s up and running, the code that Gestalt compiles has to talk to the browser over the NPRuntime interface. Imagining pushing the number of operations required to do 3D rendering or real-time video processing becomes very difficult.  To offer a comparison, the Javascript code that runs in Firefox is JIT’d to native code. When the native code has to interact with the DOM, it gets dispatched through a set of much faster quickstubs. For browsers that run plugins out-of-process like Chrome and the future Mozilla, NPRuntime will be even worse!

One of the other claims about Gestalt is that it preserves the integrity of “View Source”. I’d argue that View Source is dead – and it has been for some time now. I rarely trust the View Source representation of the page.The web is still open, but it’s more about inspecting elements and runtime styles and being able to tweak those. I rarely trust the View Source representation of the page. Dynamic DOM manipulation has all but obsoleted it. Firebug provides this for Firefox, while Chrome and Safari come with an advanced set of developer tools out of the box. Even IE8 provides a basic, though buggy set of inspection tools.

The last unfortunate point for the Gestalt project is that it requires a plugin installation on Windows and Mac, and is effectively unsupported under Linux. You won’t see any of these Gestalt apps running on an iPhone or Android device any time soon either.

So where do I see the right path?  HTML5 as a platform is powerful. Between <canvas>,  SVG, and HTML5 <video> you get virtually the same rendering power as the XAML underlying Gestalt, but a significantly larger reach.

As for the scripting languages, Javascript is the only language that you’ll be able to use on every desktop and every device on the market today. Why interpret the <script> blocks on the client when you can compile the Python and Ruby to Javascript itself, allowing it to work on any system?

Regular readers of my blog will know that I’m a big fan of GWT – a project that effectively compiles Java to Javascript. For those interested in writing in Python, Pyjamas is an equivalent project. I’m sure that there must be a Ruby equivalent out there as well.

Javascript is the Lingua Franca of the web, so any project that hopes to bring other languages to it will have to take advantage of it if it.  I’d hope that the Gestalt project evolves into one that leverages, rather than tries to replace the things that the browser does well.

A taste of Firefox Extensions, written in GWT

Monday, July 13th, 2009

UPDATE: It’s live! The open-source project is up on Google Code and I’ve blogged a more about it.

I’m getting closer to having the GWT bindings that we wrote for Firefox ready for public release. What we’ve got is more than enough to write a complex extension. The bindings were even enough to write a prototype of an OOPHM server, itself written in GWT!

For now, just a taste of what extension development is like GWT, complete with strong typing, syntax checks, auto-completion and *hosted mode support*:

protected nsIFile createTempFile() {
    nsIFile file = nsIProperties.getService("@mozilla.org/file/directory_service;1")
        .get("TmpD", nsIFile.iid());
    file.append("logs");
    if (!file.exists()) {
        file.create(nsIFile.DIRECTORY_TYPE, 0777);
    }

    file.append("log.txt");
    file.createUnique(nsIFile.NORMAL_FILE_TYPE, 0666);

    return file;
};

protected void write(String value, nsIFile file) {
    nsIFileOutputStream foStream = nsIFileOutputStream.createInstance("@mozilla.org/network/file-output-stream;1");
    foStream.init(file, 0x02 | 0x08 | 0x10, 0666, 0);
    foStream.write(value, value.length());
    foStream.close();
};

The bindings are all generated from the xulrunner SDK’s IDL files and include documentation, parameter names and constants:

  /**
     * @param file          - file to write to (must QI to nsILocalFile)
     * @param ioFlags       - file open flags listed in prio.h
     * @param perm          - file mode bits listed in prio.h
     * @param behaviorFlags flags specifying various behaviors of the class
     *        (currently none supported)
     */
  public final native void init(nsIFile file, int ioFlags, int perm, int behaviorFlags) /*-{
    return this.init(file, ioFlags, perm, behaviorFlags);
  }-*/;

More on video4all, HTML5 <video> everywhere

Tuesday, July 7th, 2009

If you’re not familiar with video4all, let me start off with a quick intro: It allows you to use the standards-compliant HTML5 <video> tag on any browser, freeing you from the complexity of configuring markup for multiple video formats.

I’ve been tweaking the video4all source a bit since last night’s late release to fix some issues with other browsers and clean up some of the code. Adding support for browsers without binding languages was pretty simple – a setInterval runs and checks for new video elements every few seconds, converting them to flash embeds as needed. It’s not ideal (DOM mutation events would be great here), but it does a decent enough job.

One problem that I’ve run up against is that Safari 4 under windows will actually eat your <video> element’s <source> tags if QuickTime isn’t installed!  They are no longer available in the document once eaten by the parser. In fact, there’s no way that I can find to recover these elements.  I’ve been trying to report a bug to Apple, but their bugreporter fails with a mysterious error every time I try to log in with my ADC credentials. I might consider adding a hack property to the video element to support this ultra-minority browser (-x-safari-win-mp4-src?), but I’ll keep researching ways to rescue the missing tags first.

So, what’s next for video4all? First of all, I’d like to remove the hard-coded FlowPlayer control bar that the player uses. It affects the aspect ratio of the video, making it difficult to size these things properly. Secondly, I’d like to start work on binding the rich video JS interface to the flash control behind the scenes. Even making the simple methods to start and stop the video available would be a big help!

Anyways, if you haven’t seen the demo before, check it out:

If you are interested in helping make this project better, visit us at the video4all project site and join the discussion. I’d love to hear some feedback about potential methods to fix Safari 4′s broken parser, even if they are glorious hacks!

HTML5 <video> support for older browsers

Tuesday, July 7th, 2009

I’ve been working on a small project to bring support for the HTML5 <video> tag to older browsers, hoping to encourage use of this tag.  The idea is to use Flash’s video/mp4 support as a “downlevel” emulator for the video tag.

It uses an HTC binding in IE and an XBL binding in Mozilla to create a flash video in place of the video tag itself. The flash video support is provided by the excellent FlowPlayer, which supports playing mp4 videos out-of-the-box.

Right now, video4all only supports videos that are statically added to your page. I hope to add support for dynamic addition of videos soon. The videos must be encoded in both video/mp4 and video/ogg formats to properly support Firefox, Safari and the Flash fallbacks. You’ll need to ensure that your video sources are properly tagged with the correct MIME types so that the script can pick them up.

The currently released browsers (that I know of) that support <video> are:

  • Firefox 3.5
  • Safari 4
  • iPhone 3.0

This project extends support for <video> to:

  • IE6+
  • Firefox 3.0
  • Safari 2-3
  • Opera (9.x)

For more info, visit the project page.

Here’s a demo (hosted in an iframe):

Digg’s optimized data streams

Wednesday, April 22nd, 2009

Digg just released a prototype of their optimized data streams.  This this is pretty cool.  It uses MIME multipart HTTP responses to return a stream of responses, dispatching each one as it comes in.  Bugzilla has been using MIME multipart for a while, though only to serve a short “Bugzilla is searching for your bugs” message before returning the actual results.

I’m still digging through it but it looks like a great way to deal with large numbers of resources at pageload time.  As an example, you can batch 50 individual profile images in a single HTTP connection roundtrip to populate your frontpage, versus having to serve 50 individual images, or having to manually stitch them into a single image on the server side.

Assuming you can scale it well on the server, you could potentially multiplex a few long-running API calls on a single stream as well.  As each one is ready, you could then throw it down the pipe and deal with it on the client.

The concept is cool.  There are some limitations obviously, but it’s a fantastic way to deal with bulk data transfer.

More at ReadWriteWeb and Digg the Blog.

The Evolution of a Web Technology

Tuesday, April 21st, 2009
  1. Obscurity
  2. Relative obscurity
  3. Robert Scoble gets it
  4. The ?-erati get it
  5. The SEO and marketing “experts” abuse it and annoy everyone
  6. Oprah ?s on her show
  7. Soccer moms and your grandma try ?ing
  8. Charmin Ultra-Soft becomes your friend on ?
  9. Someone in the ? space gets bought
  10. Time passes
  11. Robert Scoble declares that ? is dead

:)

Google eats GWT dogfood

Tuesday, April 14th, 2009

Google’s new profile editor uses GWT, cool!

Looks like they aren’t obfuscating class names yet (a feature that just landed on the tip of trunk, IIRC):

http://www.google.com/s2/gwt/resources/9312606AE1FEAAD063F34A9446584258.cache.js

The name of the project is “com.google.focus” and there are references to codenames: “evergreen” (contact info DB?) and “publicusername”.  They are also pulling some of the code from Google Collections into the final output.