Archive for the ‘development’ Category

More on window.name Cross-Domain Transports

Monday, January 4th, 2010

I previously discussed using window.name as a transport last year, but there wasn’t enough information and theory in the previous post to implement your own transport. I’ll fill in the gaps a bit more here to help developers looking to implement this themselves. You can also see the implementation we’re using at DotSpots in the gwt-rpc-plus project: Cross­Domain­Frame­Transport.java and Cross­Domain­Frame­Transport­Request.java.

The process of making a window.name request is described below. Note that you can inspect the HTML and HTTP traffic on my blog to see how our DotSpots plugin performs the cross-domain requests.

  1. Create the iframe and form to post. In IE, we use ActiveXObject as previously discussed to work around the machine-gun navigation sounds. You’ll need to stash a reference to the ActiveXObject somewhere where IE won’t garbage collect it.
    if ("ActiveXObject" in window) {
        var doc = new ActiveXObject("htmlfile");
        doc.open();
        doc.write("<html><body><iframe id='iframe'></iframe></body></html>");
        doc.close();
        var iframe = doc.getElementById('iframe');
    } else {
        var doc = document;
        var iframe = doc.createElement('iframe');
        doc.body.appendChild(iframe);
    }
    
    var form = doc.createElement('form');
    doc.body.appendChild(form);
    
  2. Set the the iframe’s name to a globally unique request ID. Using a unique request ID is very important, as two frames with the same name in Safari will result in one of them being set to a random name.
    iframe.contentWindow.name = requestId;
    form.target = requestId;
  3. POST the form to the endpoint, including the unique request ID and URL on the host domain. You can usually use /favicon.ico as a local path to return to. This is what gets posted by the DotSpots plugin on my blog:
    serial=0-8908858
    data=... (truncated JSON data)
    type=window.name
    redirect=http%3A%2F%2Fgrack.com%2Ffavicon.ico
  4. At this point, the iframe is now in a different security domain, so onload events won’t fire and window name is inaccessible.
  5. The server generates code to set the window name and redirect back to the original domain. Since the iframe is in a different security domain, the parent page won’t be able to tell that it was loaded. We redirect this iframe back to the URL requested once we’ve set the name.
    <html><body>
    <script>
    window.name='wnr-0-8908858(truncated JSON data)';
    window.location.replace('http://grack.com/favicon.ico');
    </script></body></html>
  6. The browser redirects back. When it gets back to the other domain’s favicon, onload will successfully fire and the window.name previously set by the server will be accessible. As multiple onload and onreadystatechange events might occur during the request/response process, you should always check the window.name for the appropriate prefix.
  7. Clean up the iframe and form. Remove the iframe and form after the request completes, but do this after a short delay. If you remove the iframe immediately after onload fires, the Firefox loading throbber will continue to animate indefinitely.

If you are considering using a transport like this for your own project, you should also consider using postMessage for uplevel browsers that support it. There are two HTTP requests required for window.name, adding a fair bit of latency to each cross-domain request. Using postMessage reduces the number of requests to one, cutting the round-trip latency by 50%.

Your server responses when using postMessage should look like this instead:

<html><body>
<script>
window.parent.postMessage('wnr-0-8908858(truncated JSON data)', '*');
</script></body></html>

If you’re using GWT, all of this work is already done for you in our open-source library, gwt-rpc-plus. We’ve tested it against all of the current desktop browsers (all the way back to IE6) and the mobile browsers too (Android and iPhone).

GWT 2.0 on OSX and the ExternalBrowser RunStyle

Saturday, December 12th, 2009

GWT 2.0 ships with a new HtmlUnit-based testing system that makes testing faster, but isn’t full web-browser under the hood. It can’t perform layout, so any tests that rely on layout measurements won’t succeed.

There are a number of alternate run styles that are useful, however. Manual mode, which outputs a URL that you can connect any browser to, works well for quick testing, but is a pain when you need to keep running it over and over. RemoteWeb and Selenium automate the process of starting and stopping browsers, but require you to start an external server before testing.

There’s another run-style that isn’t well-documented, but I’ve found to be the most useful for testing locally: ExternalBrowser. It requires an executable name in the command-line, the name of the browser to start. On OSX, you can specify the fully-qualified name of the executable beneath the application bundle, or you can use a shell script to launch the browser of your choice.

Place the following script under /usr/bin named ’safari’ and make it chmod a+x. This allows you (and ExternalBrowser) to launch Safari from the command-line’s PATH. The argument to “open” is the name of any browser that lives under your /Applications/ directory, including subdirectories.

#!/bin/sh
open -W -a Safari $1

Add a Java system property “gwt.args” to your testing launch configuration in Eclipse, Ant script or other IDE. You can then specify the run style like so:

-Dgwt.args="-runStyle ExternalBrowser:safari"

Now, when you run your unit tests, you’ll see GWT launch Safari and navigate to the appropriate URL on its own. Tests will leave windows up on your screen after they complete, but you can safely close them after the run terminates.

My Presentation at Google Campfire One

Saturday, December 12th, 2009

I went down to Mountain View last week to show off the company we’ve been building, DotSpots, plug our brand-new Chrome extension and demo some of the great new features of GWT 2.0.

See it directly on YouTube here: http://www.youtube.com/watch?v=JQpuDB2Jxfg.

Kudos to the Google team for putting together such a great event. You don’t realize how much planning and preparation goes into an hour-long event like that.

Absolutizing URLs in Javascript

Tuesday, November 17th, 2009

Occasionally it becomes useful to get the absolute form path of a URL from a relative one. You might be dynamically changing links on a page, or loading scripts from the same location as scripts that have already been loaded.

Computing an absolute URL by hand is problematic: you need to deal with any <base> elements in the document and properly implement the relative path canonicalization that browsers already do.

It turns out that on standards-compliant browsers based on Gecko, WebKit or Opera, the href property of anchor elements is magical. If you assign a URL fragment to it, it comes back as a fully-qualified URL when you read it back.

Ok, that’s great, but what about IE?  Well, it turns out that IE will fully-qualify URLs returned by the href property, but only if that anchor tag was created by the parser. Using createElement(‘a’) and setting href won’t trigger this code path.  There’s a trick we can use to work around this limitation, however.  You can force IE to use the parser to create elements by assigning innerHTML of another element.  This runs the element creation through the magic construction path that correctly sets up the attribute/property mapping.

Here’s a snippet of a function that will correctly canonicalize any URL you pass to it:

<script>
function canonicalize(url) {
    var div = document.createElement('div');
    div.innerHTML = "<a></a>";
    div.firstChild.href = url; // Ensures that the href is properly escaped
    div.innerHTML = div.innerHTML; // Run the current innerHTML back through the parser
    return div.firstChild.href;
}
</script>

You can use this script for other interesting purposes, like determining the base path for the current page (returns “/” for “/foo” and “/foo/” for “/foo/”). It gets the relative URL for the path “_#”, which removes any anchors, query strings or filenames from the current URL.

<script>
function getBasePath() {
    return canonicalize("_#").slice(0,-2);
}
</script>

The above code was tested on Safari, Chrome, Firefox 3.5 and IE6.

Fix for GWT Hosted mode crash with Safari 4.0.4

Monday, November 16th, 2009

My local tests started failing soon after upgrading my machine to Safari 4.0.4. Some research and pointers from the GWT folk pointed me at the root cause, a WebKit bug which is now fixed.

Kelly Norton pointed me at a quick fix:

  1. Download the latest WebKit nightly from here
  2. Save it locally
  3. Add the following environment variable to your testing .launch targets (using the path to your new WebKit.app, of course). Tests run from Ant or the command line will need to use the appropriate tasks or shell commands:
    Name: DYLD_FRAMEWORK_PATH
    Value: /Applications/path-to-your-local-webkit/WebKit.app/Contents/Frameworks/10.5
    
  4. Snow Leopard users will need to use 10.6 in the path above.

The proper fix should arrive in WebKit 4.0.5 at some point, but this will keep you running for now.

Jim Douglas has posted more detailed instructions on how to configure your launch targets on the GWT issue here: Issue #4220: GWT crash (Safari 4.0.4)

Javascript primitive conversion quick-reference

Wednesday, September 16th, 2009

Javascript has three primitive types: number, string and boolean. You can quickly coerce values between the primitive types using some simple expressions.

There are a few different coersion expressions, depending on how you want to handle some of the corner cases.  I’ve automatically generated a list below:

Conversion: To Number To Number To Number To String To Boolean To Boolean
Expression: +x (+x)||0 +(x||0) “”+x !!x !!+x
null 0 0 0 “null” false false
(void 0) NaN 0 0 “undefined” false false
NaN NaN 0 0 “NaN” false false
Infinity Infinity Infinity Infinity “Infinity” true true
-Infinity -Infinity -Infinity -Infinity “-Infinity” true true
0 0 0 0 “0″ false false
“0″ 0 0 0 “0″ true false
1 1 1 1 “1″ true true
“1″ 1 1 1 “1″ true true
2 2 2 2 “2″ true true
“2″ 2 2 2 “2″ true true
[] 0 0 0 “” true false
({}) NaN 0 NaN “[object Object]“ true false
true 1 1 1 “true” true true
“true” NaN 0 NaN “true” true false
false 0 0 0 “false” false false
“false” NaN 0 NaN “false” true false
“” 0 0 0 “” false false
“null” NaN 0 NaN “null” true false

The above table was generated with this code (note: uses some Firefox-specific code).

<table id="results" style="border-collapse: collapse; border: 1px solid black;">
 <tr id="header">
 <th>Conversion:</th>
 </tr>
 <tr id="header2">
 <th>Expression:</th>
 </tr>
</table>

<script>
function styleCell(cell) {
 cell.style.border = '1px solid black';
 cell.style.padding = '0.2em';
 return cell;
}

values = [
null, undefined, NaN, +Infinity, -Infinity, 0, "0", 1, "1", 2, "2",
   [], {}, true, "true", false, "false", "", "null"
]

coersions = [
["To Number", "+x"],
 ["To Number", "(+x)||0"],
 ["To Number", "+(x||0)"],
 ["To String", "\"\"+x"],
 ["To Boolean", "!!x"],
 ["To Boolean", "!!+x"]
]

var results = document.getElementById('results');
var trHeader = document.getElementById('header');
var trHeader2 = document.getElementById('header2');

for (var i = 0; i < coersions.length; i++) {
 var th = trHeader.appendChild(styleCell(document.createElement('th')));
 th.textContent = coersions[i][0]
 th = trHeader2.appendChild(styleCell(document.createElement('th')));
 th.textContent = coersions[i][1]
}

for (var i = 0; i < values.length; i++) {
 var tr = results.appendChild(document.createElement('tr'));
 var rowHeader = tr.appendChild(styleCell(document.createElement('th')));
 rowHeader.textContent = uneval(values[i]);

 for (var j = 0; j < coersions.length; j++) {
 var td = tr.appendChild(styleCell(document.createElement('td')));
 td.textContent = uneval(eval("(function(x) { return "+coersions[j][1]+"})")(values[i]));
 }
}

</script>

PubSubHubbub vs. rssCloud

Monday, September 7th, 2009

UPDATE: I’ve written a PubSubHubbub-to-XMPP gateway that solves some of the issues of running a real-time feed reader behind a firewall.

UPDATE 2 rssCloud has a serious vulnerability that needs to be addressed in the protocol. I’ve linked some security recommendations here that rssCloud hubs should implement as soon as possible.

These last few months have brought us not one, but two RSS-to-real-time protocols: PubSubHubbub and rssCloud. While rssCloud has been “around” for a while, it never saw much adoption or interest until recently.

As a developer, the important question is: which of these two protocols should I focus on?

When you compare the two protocols technically, you find that there are some similarities (UPDATE: see here for a more in-depth comparison of the APIs):

  • Both PubSubHubbub and rssCloud allow the hub to live on a different server than the server that is providing RSS. This lets the complexity of both of these protocols to live in a black box somewhere else, managed by someone who cares more about getting the details right.
  • Both offer a fairly simple publisher “ping” notification for publishers. An rssCloud client can make a simple POST request to the specified cloud server, which is then verified by the server to ensure that the update was real (alternatively, rssCloud can use XML-RPC or SOAP, neither of which are in fashion right now). PubSubHubbub has a very similar POST operation with very similar semantics.
  • Both offer simple APIs on the hub for subscribing to feeds. PubSubHubbub offers an unsubscribe option, while rssCloud times out subscriptions after 25 hours (the client is expected to re-subscribe after 24).

There are some significant differences between the two protocols, however:

  • PubSubHubbub supports RSS and Atom out of the box. rssCloud does not support Atom right now, as noone has defined how it would look inside of an Atom feed.
  • PubSubHubbub provides “fat pings” to clients, while rssCloud only provides basic notification updates. A PubSubHubbub subscriber can keep tabs on a feed entirely through the ping notifications, allowing it to skip polling of any feed that supports the update protocol. rssCloud requires the subscriber to re-poll the feed after receiving a ping. The “fat ping” has the advantage of saving the feed publisher bandwidth, since clients aren’t downloading the same repeated feed entries time after time, and potentially CPU cycles, since the feed publisher only has to generate a single feed output for the hub rather than for all of its clients (this can be mitigated by caching the generated feed). The fat ping requires more work on the part of the hub, however, as it needs to detect which parts of the feed have changed and push those parts into the subscriber notification dispatch queue.
  • PubSubHubbub lets you subscribe any endpoint you like (with some intelligence to prevent you spamming pings to arbitrary hosts). rssCloud infers your endpoint hostname from the IP address of the request, requiring your subscription logic to live on the same servers as your ping endpoints.

Back to the question: which of these protocols should I focus on? The answer probably depends on what you are doing.

  • If you are a publisher that publishers both RSS and Atom feeds, it’s trivial for you to support pinging rssCloud and PubSubHubbub hubs. There’s nothing stopping you from doing it now – just figure out which hubs to use. If you use FeedBurner and PingShot, Google has already cloud-enabled your blog for you.  If you want to control your own hub, you’ll probably want to pick an off-the-shelf one. PubSubHubbub is likely the best choice here as it both saves you bandwidth and gets you real-time support in FriendFeed.
  • If you are planning on writing a hub, you’ll probably want to start with rssCloud. Its implementation will be simpler than PubSubHubbub as all it does is redistribute ping notifications.
  • If you are a feed reader or a content spider, you’ll probably have to implement both. I believe that PubSubHubbub gives you the biggest bang for the buck now, as it’s supported by nearly all of the Google feed properties: FeedBurner (the Atom/RSS intermediary choice for a significant number of self-hosted blogs), Blogger (millions of blogs) and Google Reader feeds. It’s also supported by LiveJournal (which lists 20+ million blogs on its homepage).  rssCloud is fairly new, but it managed to score a big integration with wordpress.com (7.5 million blogs, according to their own blog). Unfortunately, as not all of the big sites have implemented both, you’ll have to deal with two technologies for the time being.

After researching both of the technologies in-depth, I’d say that PubSubHubbub is the better technology overall.  While more complex to implement for hubs, it offers far more to feed readers and publishers in terms of bandwidth savings and real-time updates.  For companies doing content analysis, PubSubHubbub is a huge win: it brings the power of the Twitter firehose to RSS. No matter which technology you choose, however, you’ll be getting your RSS feed updates far more often.  It might even allow the next real-time technology to be built on an open XML feed rather than a proprietary company’s servers.

A quieter window.name transport for IE

Tuesday, July 28th, 2009

Using window.name as a transport for cooperative cross-domain communication is a reasonably well-known and well-researched technique. I came across it via two blog posts by members of the GWT community that were using it to submit GWT FormPanels to endpoints on other domains.

For our product, I’ve been looking at various ways we can offer RPC for our script when it is embedded in pages that don’t run on servers under our control.  Modern browsers, like Firefox 3.5 and Safari 4.0 support XMLHttpRequest Level 2.  This standard allows you to make cross-domain calls, as long as the server responds with the appropriate Access-Control header.  Internet Explorer 8 supports a proprietary XDomainRequest that offers similar support.

When we’re looking at “downlevel” browsers, like Firefox 2/3, Safari 2/3 and IE 6/7, the picture isn’t as clear. The window.name transport works well in every downlevel browser but IE6 and 7. In those IE versions, each RPC request made across the iframe is accompanied by an annoying click sound. As you can imagine, a page that has a few RPC requests that it requires to load will end up sounding like a machine gun. The reason for this is IE’s navigation sound which plays on every location change for any window, including iframes. The window.name transport requires a POST and a redirect back to complete the communication, triggering this audio UI.

I spent a few hours hammering away on the problem, trying to find a solution. It turns out that IE6 can be fooled with a <bgsound> element that masks the clicking sound. This doesn’t work in IE7, however. My research then lead to an interesting discovery: the GTalk team was using an ActiveX object named “htmlfile” to work around a similar problem: navigation sounds that would play during their COMET requests. The htmlfile object is basically a UI-disconnected HTML document that works, for the most part, the same way as a browser document. The question was now how to use this for a cross-domain request.

The interesting thing about the htmlfile ActiveX object is that not all HTML works as you’d expect it to. My first attempt was to use the htmlfile object, creating an iframe element with it, attaching it to the body (along with an accompanying form inside the htmlfile document) and POSTing the data. Unfortunately, I couldn’t get any of the events to register. The POST was happening, but none of the iframe’s onload events were firing:

if ("ActiveXObject" in window) {
    var doc = new ActiveXObject("htmlfile");
    doc.open();
    doc.write("<html><body></body></html>");
    doc.close();
} else {
    var doc = document;
}

var iframe = doc.createElement('iframe');
doc.body.appendChild(iframe);
iframe.onload = ...
iframe.onreadystatechange = ...

The second attempt was more fruitful. I tried writing the iframe out as part of the document, getting the iframe from the htmlfile and adding event handlers to this object. Success!  I managed to capture the onload event, read back the window.name value and, best of all, the browser did this silently:

if ("ActiveXObject" in window) {
    var doc = new ActiveXObject("htmlfile");
    doc.open();
    doc.write("<html><body><iframe id='iframe'></iframe></body></html>");
    doc.close();
    var iframe = doc.getElementById('iframe');
} else {
    var doc = document;
    var iframe = doc.createElement('iframe');
    doc.body.appendChild(iframe);
}

iframe.onload = ...
iframe.onreadystatechange = ...

I’m currently working on cleaning up the ugly proof-of-concept code to integrate as a transport in the Thrift-GWT RPC library I’m working on. This will allow us to transparently switch to the cross-domain transport when running offsite, without any visible side-effects to the user.

Mix’s Gestalt project is on the wrong track

Wednesday, July 22nd, 2009

I came across the Mix Gestalt project tonight and I thought I’d share some thoughts. It’s a bit of script that effectively sucks code snippets in languages other than Javascript out of your page and converts them to programs running on the .NET platform.

While interesting, it has a number of drawbacks that make it far less interesting than the HTML5-based approach that works in the standards-compliant browsers based on WebKit, Gecko and Opera, as well as the improved IE8.

First of all, it has to bootstrap .NET into Firefox (or whichever browser you are running it in).  This adds a few milliseconds to your page’s cold load time if it’s not already loaded. In the day and age of fast websites, any additional page time is just a no-go.

Once it’s up and running, the code that Gestalt compiles has to talk to the browser over the NPRuntime interface. Imagining pushing the number of operations required to do 3D rendering or real-time video processing becomes very difficult.  To offer a comparison, the Javascript code that runs in Firefox is JIT’d to native code. When the native code has to interact with the DOM, it gets dispatched through a set of much faster quickstubs. For browsers that run plugins out-of-process like Chrome and the future Mozilla, NPRuntime will be even worse!

One of the other claims about Gestalt is that it preserves the integrity of “View Source”. I’d argue that View Source is dead – and it has been for some time now. I rarely trust the View Source representation of the page.The web is still open, but it’s more about inspecting elements and runtime styles and being able to tweak those. I rarely trust the View Source representation of the page. Dynamic DOM manipulation has all but obsoleted it. Firebug provides this for Firefox, while Chrome and Safari come with an advanced set of developer tools out of the box. Even IE8 provides a basic, though buggy set of inspection tools.

The last unfortunate point for the Gestalt project is that it requires a plugin installation on Windows and Mac, and is effectively unsupported under Linux. You won’t see any of these Gestalt apps running on an iPhone or Android device any time soon either.

So where do I see the right path?  HTML5 as a platform is powerful. Between <canvas>,  SVG, and HTML5 <video> you get virtually the same rendering power as the XAML underlying Gestalt, but a significantly larger reach.

As for the scripting languages, Javascript is the only language that you’ll be able to use on every desktop and every device on the market today. Why interpret the <script> blocks on the client when you can compile the Python and Ruby to Javascript itself, allowing it to work on any system?

Regular readers of my blog will know that I’m a big fan of GWT – a project that effectively compiles Java to Javascript. For those interested in writing in Python, Pyjamas is an equivalent project. I’m sure that there must be a Ruby equivalent out there as well.

Javascript is the Lingua Franca of the web, so any project that hopes to bring other languages to it will have to take advantage of it if it.  I’d hope that the Gestalt project evolves into one that leverages, rather than tries to replace the things that the browser does well.

@SuppressWarnings on variable declarations

Tuesday, July 14th, 2009

I’ve been using this trick for a while and I thought I’d share it. For those who live by Eclipse’s quickfixes, it’s not entirely obvious that it’s legal.

If you have legacy code like the code below, where foo.list() is a Java 1.4-compatible method returning java.util.List, you’ll normally see an “unchecked cast” warning on the assignment like so:

public void doSomeListStuff(Foo foo) {
    List list = foo.list(); // Warning: unchecked cast
    for (Blah blah : list) {
        frobinate(blah);
    }
}

Normally, Eclipse offers to fix it for you like this:

@SuppressWarnings("unchecked")
public void doSomeListStuff(Foo foo) {
    List list = foo.list(); // Everything is A-OK!
    for (Blah blah : list) {
        frobinate(blah);
    }
}

A better solution takes advantage of Java’s less-touted ability to annotate local variable declarations. By annotating the declaration and assignment instead of the method, the warning suppression is limited in scope to the assignment expression itself:

public void doSomeListStuff(Foo foo) {
    @SuppressWarnings("unchecked")
    List list = foo.list();

    for (Blah blah : list) {
        frobinate(blah);
    }
}

This keeps your code under maximum protection of Java’s generic strong typing. By annotating a whole method with @SuppressWarnings(“unchecked”), you may inadvertently introduce a later, unsafe cast that could cause a bug down the line.