GWT 2.0 on OSX and the ExternalBrowser RunStyle

December 12th, 2009

GWT 2.0 ships with a new HtmlUnit-based testing system that makes testing faster, but isn’t full web-browser under the hood. It can’t perform layout, so any tests that rely on layout measurements won’t succeed.

There are a number of alternate run styles that are useful, however. Manual mode, which outputs a URL that you can connect any browser to, works well for quick testing, but is a pain when you need to keep running it over and over. RemoteWeb and Selenium automate the process of starting and stopping browsers, but require you to start an external server before testing.

There’s another run-style that isn’t well-documented, but I’ve found to be the most useful for testing locally: ExternalBrowser. It requires an executable name in the command-line, the name of the browser to start. On OSX, you can specify the fully-qualified name of the executable beneath the application bundle, or you can use a shell script to launch the browser of your choice.

Place the following script under /usr/bin named ’safari’ and make it chmod a+x. This allows you (and ExternalBrowser) to launch Safari from the command-line’s PATH. The argument to “open” is the name of any browser that lives under your /Applications/ directory, including subdirectories.

#!/bin/sh
open -W -a Safari $1

Add a Java system property “gwt.args” to your testing launch configuration in Eclipse, Ant script or other IDE. You can then specify the run style like so:

-Dgwt.args="-runStyle ExternalBrowser:safari"

Now, when you run your unit tests, you’ll see GWT launch Safari and navigate to the appropriate URL on its own. Tests will leave windows up on your screen after they complete, but you can safely close them after the run terminates.

My Presentation at Google Campfire One

December 12th, 2009

I went down to Mountain View last week to show off the company we’ve been building, DotSpots, plug our brand-new Chrome extension and demo some of the great new features of GWT 2.0.

See it directly on YouTube here: http://www.youtube.com/watch?v=JQpuDB2Jxfg.

Kudos to the Google team for putting together such a great event. You don’t realize how much planning and preparation goes into an hour-long event like that.

Re-enable ‘Install’ button for Mac Chrome Extensions

December 8th, 2009

UPDATE (Jan 6, 2010): The Chrome team has turned on extensions on Mac in the latest dev channel build. Even better, you can install directly from extension pages without this bookmarklet. Hooray!

We just officially launched our extension on the Chrome extension gallery today and Mac users are having trouble installing the extension. The ‘Install’ button for Chrome extensions is disabled if you are running on a Mac.

You can manually download extensions from the Google download site using a URL like so:

https://clients2.google.com/service/update2/crx?response=redirect&x=id%3Danfibeojfgdfejcfflalkebdfgojfbbm%26uc.

The bolded part of the URL above is the extension’s public ID (a hash of the public key).

Here’s a bookmarklet that’ll re-enable the button and allow you to download the extension. It recreates the download URL from the current page you are viewing:

Enable Extension Install

Try it out on the DotSpots extension page here. Click the bookmarklet and the Install button will be activated.

UPDATE: I removed the lang=en-US part of the URL. Some extensions fail to load with that attribute in place.

Note that you’ll need an extension-supported build. The next developer channel build should support it. If you are too anxious to wait, you can install the latest Chromium build from here: http://build.chromium.org/buildbot/snapshots/chromium-rel-mac/34059/.

Absolutizing URLs in Javascript

November 17th, 2009

Occasionally it becomes useful to get the absolute form path of a URL from a relative one. You might be dynamically changing links on a page, or loading scripts from the same location as scripts that have already been loaded.

Computing an absolute URL by hand is problematic: you need to deal with any <base> elements in the document and properly implement the relative path canonicalization that browsers already do.

It turns out that on standards-compliant browsers based on Gecko, WebKit or Opera, the href property of anchor elements is magical. If you assign a URL fragment to it, it comes back as a fully-qualified URL when you read it back.

Ok, that’s great, but what about IE?  Well, it turns out that IE will fully-qualify URLs returned by the href property, but only if that anchor tag was created by the parser. Using createElement(‘a’) and setting href won’t trigger this code path.  There’s a trick we can use to work around this limitation, however.  You can force IE to use the parser to create elements by assigning innerHTML of another element.  This runs the element creation through the magic construction path that correctly sets up the attribute/property mapping.

Here’s a snippet of a function that will correctly canonicalize any URL you pass to it:

<script>
function canonicalize(url) {
    var div = document.createElement('div');
    div.innerHTML = "<a></a>";
    div.firstChild.href = url; // Ensures that the href is properly escaped
    div.innerHTML = div.innerHTML; // Run the current innerHTML back through the parser
    return div.firstChild.href;
}
</script>

You can use this script for other interesting purposes, like determining the base path for the current page (returns “/” for “/foo” and “/foo/” for “/foo/”). It gets the relative URL for the path “_#”, which removes any anchors, query strings or filenames from the current URL.

<script>
function getBasePath() {
    return canonicalize("_#").slice(0,-2);
}
</script>

The above code was tested on Safari, Chrome, Firefox 3.5 and IE6.

Fix for GWT Hosted mode crash with Safari 4.0.4

November 16th, 2009

My local tests started failing soon after upgrading my machine to Safari 4.0.4. Some research and pointers from the GWT folk pointed me at the root cause, a WebKit bug which is now fixed.

Kelly Norton pointed me at a quick fix:

  1. Download the latest WebKit nightly from here
  2. Save it locally
  3. Add the following environment variable to your testing .launch targets (using the path to your new WebKit.app, of course). Tests run from Ant or the command line will need to use the appropriate tasks or shell commands:
    Name: DYLD_FRAMEWORK_PATH
    Value: /Applications/path-to-your-local-webkit/WebKit.app/Contents/Frameworks/10.5
    
  4. Snow Leopard users will need to use 10.6 in the path above.

The proper fix should arrive in WebKit 4.0.5 at some point, but this will keep you running for now.

Jim Douglas has posted more detailed instructions on how to configure your launch targets on the GWT issue here: Issue #4220: GWT crash (Safari 4.0.4)

Packing Chrome extensions in Python

November 9th, 2009

We’re just about to release our DotSpots extension for Chrome and I’ve been working on integrating the CRX packaging into our build process. CRX files are basically ZIP files with an RSA signature and public key tacked on to the front of it. Generating these files requires you to use the Chrome –pack-extension argument (which in turn requires you to deploy the 100MB+ Chrome binaries to your build machine).

The existing code to pack a Chrome extension in Python is pretty dated: it will only generated the insecure CRX version 1 format that doesn’t use a cryptographic signature. There’s some Ruby code to pack a version 2 extension, but it requires a lot of dependencies that aren’t installed by default on OSX or in Fedora.

I’ve written some code in Python that uses openssl under the hood to do the grunt work. It cuts some corners by requiring you to pre-zip your files, but you’ll get better results from 7zip -9 than Python’s internal zip code anyways. Pass it three arguments: The input ZIP file, the PEM key (generated when you manually pack the extension in Chrome for the first time) and the output file.

#!/usr/bin/python
# Cribbed from http://github.com/Constellation/crxmake/blob/master/lib/crxmake.rb
# and http://src.chromium.org/viewvc/chrome/trunk/src/chrome/tools/extensions/chromium_extension.py?revision=14872&content-type=text/plain&pathrev=14872

import sys
from array import *
from subprocess import *

arg0,input,key,output = sys.argv

# Sign the zip file with the private key in PEM format
signature = Popen(["openssl", "sha1", "-sign", key, input], stdout=PIPE).stdout.read();

# Convert the PEM key to DER (and extract the public form) for inclusion in the CRX header
derkey = Popen(["openssl", "rsa", "-pubout", "-inform", "PEM", "-outform", "DER", "-in", key], stdout=PIPE).stdout.read();

out=open(output, "wb");
out.write("Cr24")  # Extension file magic number
header = array("l");
header.append(2); # Version 2
header.append(len(derkey));
header.append(len(signature));
header.tofile(out);
out.write(derkey)
out.write(signature)
out.write(open(input).read())

print "Done."

Javascript primitive conversion quick-reference

September 16th, 2009

Javascript has three primitive types: number, string and boolean. You can quickly coerce values between the primitive types using some simple expressions.

There are a few different coersion expressions, depending on how you want to handle some of the corner cases.  I’ve automatically generated a list below:

Conversion: To Number To Number To Number To String To Boolean To Boolean
Expression: +x (+x)||0 +(x||0) “”+x !!x !!+x
null 0 0 0 “null” false false
(void 0) NaN 0 0 “undefined” false false
NaN NaN 0 0 “NaN” false false
Infinity Infinity Infinity Infinity “Infinity” true true
-Infinity -Infinity -Infinity -Infinity “-Infinity” true true
0 0 0 0 “0″ false false
“0″ 0 0 0 “0″ true false
1 1 1 1 “1″ true true
“1″ 1 1 1 “1″ true true
2 2 2 2 “2″ true true
“2″ 2 2 2 “2″ true true
[] 0 0 0 “” true false
({}) NaN 0 NaN “[object Object]“ true false
true 1 1 1 “true” true true
“true” NaN 0 NaN “true” true false
false 0 0 0 “false” false false
“false” NaN 0 NaN “false” true false
“” 0 0 0 “” false false
“null” NaN 0 NaN “null” true false

The above table was generated with this code (note: uses some Firefox-specific code).

<table id="results" style="border-collapse: collapse; border: 1px solid black;">
 <tr id="header">
 <th>Conversion:</th>
 </tr>
 <tr id="header2">
 <th>Expression:</th>
 </tr>
</table>

<script>
function styleCell(cell) {
 cell.style.border = '1px solid black';
 cell.style.padding = '0.2em';
 return cell;
}

values = [
null, undefined, NaN, +Infinity, -Infinity, 0, "0", 1, "1", 2, "2",
   [], {}, true, "true", false, "false", "", "null"
]

coersions = [
["To Number", "+x"],
 ["To Number", "(+x)||0"],
 ["To Number", "+(x||0)"],
 ["To String", "\"\"+x"],
 ["To Boolean", "!!x"],
 ["To Boolean", "!!+x"]
]

var results = document.getElementById('results');
var trHeader = document.getElementById('header');
var trHeader2 = document.getElementById('header2');

for (var i = 0; i < coersions.length; i++) {
 var th = trHeader.appendChild(styleCell(document.createElement('th')));
 th.textContent = coersions[i][0]
 th = trHeader2.appendChild(styleCell(document.createElement('th')));
 th.textContent = coersions[i][1]
}

for (var i = 0; i < values.length; i++) {
 var tr = results.appendChild(document.createElement('tr'));
 var rowHeader = tr.appendChild(styleCell(document.createElement('th')));
 rowHeader.textContent = uneval(values[i]);

 for (var j = 0; j < coersions.length; j++) {
 var td = tr.appendChild(styleCell(document.createElement('td')));
 td.textContent = uneval(eval("(function(x) { return "+coersions[j][1]+"})")(values[i]));
 }
}

</script>

Time for Atom to Put RSS Out to Pasture?

September 15th, 2009

Is it time to the world to move on from RSS and to its successor, Atom? Some considerations:

Atom has an IETF standard for syndication. Atom has an IETF standard for publication. Atom was designed for modularity. Atom supports rich, well-defined activities within feeds.

RSS is effectively frozen at 2.0:

RSS is by no means a perfect format, but it is very popular and widely supported. Having a settled spec is something RSS has needed for a long time. The purpose of this work is to help it become a unchanging thing, to foster growth in the market that is developing around it, and to clear the path for innovation in new syndication formats. Therefore, the RSS spec is, for all practical purposes, frozen at version 2.0.1. We anticipate possible 2.0.2 or 2.0.3 versions, etc. only for the purpose of clarifying the specification, not for adding new features to the format. Subsequent work should happen in modules, using namespaces, and in completely new syndication formats, with new names.

It is full of legacy tags and archaic design decisions:

The purpose of the <textInput> element is something of a mystery. You can use it to specify a search engine box. Or to allow a reader to provide feedback. Most aggregators ignore it.

We are spending all this time duplicating effort. Every feed reader needs to deal with Atom and RSS. Every blog provides an Atom feed and an RSS feed. Users trying to subscribe to blog feeds are presented with an unnecessary choice.

RSS solved a need at the time, even though it was crufty and difficult to use and difficult to parse (remember when RSS XML didn’t have to be well-formed XML?). It served as an inspiration for millions of sites to open up their content to new methods of reading. It inspired a great successor, Atom, which has surpassed it many times over.

We dropped gopher when its time ran out. It’s time to make Atom the primary format for blogs.

Real-time Search and Frnégtttrdre

September 14th, 2009

Robert Scoble’s accidental tweet (“Frnégtttrdre”) earlier tonite caused a minor ripple: people wondering if he was announcing a secret project, under the influence of alcohol or wandering around with an unlocked iPhone in his back pocket.

It also makes for an interesting test for real-time search.

An hour after his tweet:

Anyone else I’ve missed?

Conclusions: If you tweet a random word, there’s no guarantee that you’ll get indexed right away.  Additionally, not every service tests unicode querystring parameters.

Serious rssCloud Protocol DDoS Vulnerability

September 10th, 2009

UPDATE: There’s a new domain parameter in rssCloud that makes this DDoS far, far worse.  Since there’s no verification (yet) on rssCloud endpoints, you can now subscribe any server to any rssCloud hub’s notifications.

While researching some of the issues of rssCloud running in a shared hosting environment, I came across a serious vulnerability in the protocol. The vulnerability allows someone to cripple a shared web host. Because of the sensitive nature of this vulnerability, I’m not going to share example code or which shared host(s) are vulnerable.  The fix is easy: follow these security recommendations to close the hole.

The inspiration for this vulnerability was discovered by Nick Lothian’s post on FriendFeed. It turns out that many shared hosting providers route incoming and outgoing HTTP requests through different IP addresses. The process of routing the HTTP requests is usually done transparently by a networking gear outside of the web servers themselves.

rssCloud’s specification infers the endpoint from the REMOTE_ADDR CGI variable at the time of the subscription. It would be very difficult to get an rssCloud subscriber working in a shared hosting environment because every subscription request you make goes out on IP address A, but all of your incoming requests come in via port 80 on IP address B. For some shared web providers, the machines that make outgoing requests are also web servers, serving banner messages or redirects to sales sites. Because they are web servers, they are considered valid rssCloud REST endpoints (returning 200 OK for POST requests on some URLs).

When you put these pieces together, it becomes readily apparent that you can now subscribe your shared host’s outgoing HTTP request IP address to any number of feeds. Considering that Wordpress has 7.5 million blogs that speak rssCloud, there’s a significant number of blogs that could end up pinging the machine.

There are probably a number of other interesting vulnerabilities in this area, such as traffic that travels through a proxy, or an anonymizing service such as TOR. It may be possible to knock one of these offline by subscribing it to a large number of feeds.

The problem with rssCloud is that its subscription request only proves that you can make requests via the given IP address, not that the given IP address is willing to receive them. By adding the challenge parameter I suggested in the previous post, you can now guarantee that the endpoint is willing to receive these requests, making it much harder to subscribe an unwilling participant in the protocol.