Archive for the ‘development’ Category

A taste of Firefox Extensions, written in GWT

Monday, July 13th, 2009

UPDATE: It’s live! The open-source project is up on Google Code and I’ve blogged a more about it.

I’m getting closer to having the GWT bindings that we wrote for Firefox ready for public release. What we’ve got is more than enough to write a complex extension. The bindings were even enough to write a prototype of an OOPHM server, itself written in GWT!

For now, just a taste of what extension development is like GWT, complete with strong typing, syntax checks, auto-completion and *hosted mode support*:

protected nsIFile createTempFile() {
    nsIFile file = nsIProperties.getService("@mozilla.org/file/directory_service;1")
        .get("TmpD", nsIFile.iid());
    file.append("logs");
    if (!file.exists()) {
        file.create(nsIFile.DIRECTORY_TYPE, 0777);
    }

    file.append("log.txt");
    file.createUnique(nsIFile.NORMAL_FILE_TYPE, 0666);

    return file;
};

protected void write(String value, nsIFile file) {
    nsIFileOutputStream foStream = nsIFileOutputStream.createInstance("@mozilla.org/network/file-output-stream;1");
    foStream.init(file, 0x02 | 0x08 | 0x10, 0666, 0);
    foStream.write(value, value.length());
    foStream.close();
};

The bindings are all generated from the xulrunner SDK’s IDL files and include documentation, parameter names and constants:

  /**
     * @param file          - file to write to (must QI to nsILocalFile)
     * @param ioFlags       - file open flags listed in prio.h
     * @param perm          - file mode bits listed in prio.h
     * @param behaviorFlags flags specifying various behaviors of the class
     *        (currently none supported)
     */
  public final native void init(nsIFile file, int ioFlags, int perm, int behaviorFlags) /*-{
    return this.init(file, ioFlags, perm, behaviorFlags);
  }-*/;

Stored Procs are Dead – Long Live Stored Procs!

Tuesday, July 7th, 2009

Every developer has a different opinion of stored procedures. If you’ve worked on a few projects that involve databases, you’ll inevitably come across the as part of a project.

I’ve personally had a love/hate relationship with them. They require maintenance scripts like your regular table structure, but they aren’t structure. Generally, databases don’t give you good introspection ability for stored procedures either – fire-and-forget management.

Classically, stored procedures have been a bastardized version of the database’s native SQL. Take the basic SQL, add in some cursors, looping constructs and function invocation – that’s your stored procedure language. Depending on your database vendor, be prepared to deal with different syntaxes, punctuation and functionality.

No matter what, you’ll be writing your application in a different language, making the number of languages involved in any project with stored procs N + 1. If you choose to evaluate a different database vendor for a different project, or version 2.0 of your current project, your previous stored procedure may or may not translate.

Over the last decade and a bit, database vendors have been adding native language bindings to the database itself, allowing developers to write code in languages they are comfortable in, regardless of the database storage technology behind the scenes.

I’ll step back for a bit here – the role of application storage has been changing recently, pushed by a number of factors:

  1. Data-binding frameworks, like Hibernate, SQLAlchemy, Ruby on Rails and a number of others have been creating an abstraction layer over different database brands, as well as managing the flow of data between the live objects in the system.
  2. SQL-free large stores like Google’s BigTable, CouchDB and others have been pushing complexity out of the datastore and into the application itself.
  3. Languages with dynamic typing are in-vogue. Python, Ruby and PHP are popular, Java will be getting dynamic dispatch soon.
  4. New accessible structured storage formats, like Apache Thrift and Google’s Protobuf are allowing developers to create highly versionable “blobs” that are easy to write to, but hard to index and query.

Databases are going to have to evolve over the near future to suit the way that applications and hardware are developing today. Memory is cheap, disks are huge and time-to-market is one of the primary driving forces for developers.

I believe that the ideal role for a database is to be a high-powered host for serialization framework stubs. Instead of having to map application objects from database tables using clumsy SQL generation techniques and reconstructing objects from deep JOINs, the serialization frameworks should be injecting small stubs next to the database process itself. These stubs can coordinate data retrieval with the database itself – managing, grooming and traversing indexes, altering storage as needed for client-side data and sending appropriately structured data to the application.

Drizzle is a large step in the right direction. Instead of providing the smörgåsbord of features like the other DB vendors on the market, it provides a light-weight, modular architecture that allows you to swap and re-plug components like you would your application itself.

Here’s my plea to database developers: Let me compose my database server like I do my Java code, from appropriately tailored components optimized for my use. Let me run my Java/Python/Ruby code on the database server, right next to the data itself, in a language that I’m familiar and comfortable with – running on both sides of the database connection TCP connection.

http://drizzle.org/

More on video4all, HTML5 <video> everywhere

Tuesday, July 7th, 2009

If you’re not familiar with video4all, let me start off with a quick intro: It allows you to use the standards-compliant HTML5 <video> tag on any browser, freeing you from the complexity of configuring markup for multiple video formats.

I’ve been tweaking the video4all source a bit since last night’s late release to fix some issues with other browsers and clean up some of the code. Adding support for browsers without binding languages was pretty simple – a setInterval runs and checks for new video elements every few seconds, converting them to flash embeds as needed. It’s not ideal (DOM mutation events would be great here), but it does a decent enough job.

One problem that I’ve run up against is that Safari 4 under windows will actually eat your <video> element’s <source> tags if QuickTime isn’t installed!  They are no longer available in the document once eaten by the parser. In fact, there’s no way that I can find to recover these elements.  I’ve been trying to report a bug to Apple, but their bugreporter fails with a mysterious error every time I try to log in with my ADC credentials. I might consider adding a hack property to the video element to support this ultra-minority browser (-x-safari-win-mp4-src?), but I’ll keep researching ways to rescue the missing tags first.

So, what’s next for video4all? First of all, I’d like to remove the hard-coded FlowPlayer control bar that the player uses. It affects the aspect ratio of the video, making it difficult to size these things properly. Secondly, I’d like to start work on binding the rich video JS interface to the flash control behind the scenes. Even making the simple methods to start and stop the video available would be a big help!

Anyways, if you haven’t seen the demo before, check it out:

If you are interested in helping make this project better, visit us at the video4all project site and join the discussion. I’d love to hear some feedback about potential methods to fix Safari 4′s broken parser, even if they are glorious hacks!

HTML5 <video> support for older browsers

Tuesday, July 7th, 2009

I’ve been working on a small project to bring support for the HTML5 <video> tag to older browsers, hoping to encourage use of this tag.  The idea is to use Flash’s video/mp4 support as a “downlevel” emulator for the video tag.

It uses an HTC binding in IE and an XBL binding in Mozilla to create a flash video in place of the video tag itself. The flash video support is provided by the excellent FlowPlayer, which supports playing mp4 videos out-of-the-box.

Right now, video4all only supports videos that are statically added to your page. I hope to add support for dynamic addition of videos soon. The videos must be encoded in both video/mp4 and video/ogg formats to properly support Firefox, Safari and the Flash fallbacks. You’ll need to ensure that your video sources are properly tagged with the correct MIME types so that the script can pick them up.

The currently released browsers (that I know of) that support <video> are:

  • Firefox 3.5
  • Safari 4
  • iPhone 3.0

This project extends support for <video> to:

  • IE6+
  • Firefox 3.0
  • Safari 2-3
  • Opera (9.x)

For more info, visit the project page.

Here’s a demo (hosted in an iframe):

Digg’s optimized data streams

Wednesday, April 22nd, 2009

Digg just released a prototype of their optimized data streams.  This this is pretty cool.  It uses MIME multipart HTTP responses to return a stream of responses, dispatching each one as it comes in.  Bugzilla has been using MIME multipart for a while, though only to serve a short “Bugzilla is searching for your bugs” message before returning the actual results.

I’m still digging through it but it looks like a great way to deal with large numbers of resources at pageload time.  As an example, you can batch 50 individual profile images in a single HTTP connection roundtrip to populate your frontpage, versus having to serve 50 individual images, or having to manually stitch them into a single image on the server side.

Assuming you can scale it well on the server, you could potentially multiplex a few long-running API calls on a single stream as well.  As each one is ready, you could then throw it down the pipe and deal with it on the client.

The concept is cool.  There are some limitations obviously, but it’s a fantastic way to deal with bulk data transfer.

More at ReadWriteWeb and Digg the Blog.

Oracle buys Sun (but really MySQL and Java)

Monday, April 20th, 2009

The news that Oracle bought Sun caught me off guard.  I was moderately disappointed when they were courting IBM as a suitor, but I would have been more than happy if they had gone through with the deal.  IBM understands open-source and has a proven track record in Eclipse.

The best case scenario for Java would have been a acquisition by  or a merger with RedHat, though that would be very unlikely considering the size of RedHat itself – Sun is almost four times bigger than RedHat.  Out of some of the big software names large enough to buy and aggregate Sun’s high-end open-source assets, Google would have been a great choice.  They have shown a great deal of leadership in the open-source community.  We would have seen a vibrant community spring up out of this – consider how well the large GWT open-source projects are run.

So, what does the future hold for Java now?  I can’t tell, but the I think the best case is status quo for now.  I hope that Oracle spins Java off into its own, independent organization in the future.

If it comes down to it, the community will route around any damage.  It started down that path once before with Classpath and Apache Harmony, but those didn’t turn out to be necessary at the time.  Who knows – maybe Oracle will change through this whole process?  I’m not holding my breath, however.  :)

The final word on Google Eclipse plugin OSX crashes

Sunday, April 19th, 2009

I’ve blogged about the subject of GWT JVM crashes far too much (here and here).  This is, I hope, the final word on the subject.  I spent some time disassembling the Google Eclipse plugin (did I agree not to do that in one of the EULAs? ;) ) and discovered that they are launching the JVM without using the Eclipse infrastructure.  This means that the JVM arguments are effectively hardcoded, except in one case where -startOnFirstThread is passed for OSX clients.  For those interested, the buggy classes of note are GWTDeploymentParticipant and LaunchUtils.

I posted a note on the contributor’s list and received a response from a Googler suggesting that I wrap my Java executable in a shell script.  Not one to shy away from wrapping system executables with scripts, I decided to give it a shot tonight.  My first shell script attempt was a disaster (dear lazyweb: how can you get bash to respect quoted spaces when using $@?), so I gave it a whack in Python.

This is tested and somewhat guaranteed to work for JVM 1.6 under OSX:

cd /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin
sudo mv java java_wrapped
sudo nano java

And the script:

#!/usr/bin/env python
import sys
import os

print sys.argv
cmd = os.path.dirname(sys.argv[0]) + '/java_wrapped'

args = ['',
        '-XX:CompileCommand=exclude,org/eclipse/core/internal/dtree/DataTreeNode,forwardDeltaWith',
        '-XX:CompileCommand=exclude,org/eclipse/jdt/internal/compiler/lookup/ParameterizedTypeBinding,<init>',
        '-XX:CompileCommand=exclude,org/eclipse/jdt/internal/compiler/lookup/ParameterizedMethodBinding,<init>']

args.extend(sys.argv[1:])

print cmd
print args
print ""

os.execv(cmd, args)

If you have any suggestions on how to improve my Python, I’d be glad to hear them in the comments.

UPDATE: Thanks to Vitali, I got a bash script working as well:

#!/bin/bash
`dirname $0`/java_wrapped \
        "-XX:CompileCommand=exclude,org/eclipse/core/internal/dtree/DataTreeNode,forwardDeltaWith" \
        "-XX:CompileCommand=exclude,org/eclipse/jdt/internal/compiler/lookup/ParameterizedTypeBinding,<init>" \
        "-XX:CompileCommand=exclude,org/eclipse/jdt/internal/compiler/lookup/ParameterizedMethodBinding,<init>" \
        "$@"

AppEngine+Java+OSX “Invalid memory access” on deploy

Friday, April 17th, 2009

UPDATE, Apr 19 2009: This does not fix all compiler crashes in Eclipse.  This is only useful for the compiler crashes where an Eclipse JVM configuration is used.  Please see my later blog post for a better fix and disregard this information.

This is the same error I hit before (see my earlier post), but it happens in Eclipse during the “deploy to AppEngine” phase.  You’ll see it if you’ve got a project configured with a 1.6 JVM on the build path (even if your .class compatibility is set to 1.5).  It’s not obvious how to fix this at first glance – I had to dig around to find the options that Eclipse passes to the JVM.

The solution

You can fix it by adding the same compiler workarounds we explored before to your JVM default arguments.  IMPORTANT NOTE: Add the args to all of your 1.6 level installed JVMs.  I’ve found that Eclipse doesn’t always choose the selected one if there’s an equivalent also installed (ie: it may use your ’1.6′ JVM if you have ’1.6.0′ selected).

-XX:CompileCommand=exclude,org/eclipse/core/internal/dtree/DataTreeNode,forwardDeltaWith -XX:CompileCommand=exclude,org/eclipse/jdt/internal/compiler/lookup/ParameterizedTypeBinding,<init> -XX:CompileCommand=exclude,org/eclipse/jdt/internal/compiler/lookup/ParameterizedMethodBinding,<init>

Add them to your JVMs under Preferences > Java > Installed JREs by clicking the edit button and filling out “Default VM Arguments”:

jvm-args-to-fix-crash

An example of where to place the arguments

Google eats GWT dogfood

Tuesday, April 14th, 2009

Google’s new profile editor uses GWT, cool!

Looks like they aren’t obfuscating class names yet (a feature that just landed on the tip of trunk, IIRC):

http://www.google.com/s2/gwt/resources/9312606AE1FEAAD063F34A9446584258.cache.js

The name of the project is “com.google.focus” and there are references to codenames: “evergreen” (contact info DB?) and “publicusername”.  They are also pulling some of the code from Google Collections into the final output.

GWT 1.6 crashes (and a fix)

Tuesday, April 14th, 2009

There’s a bug in the latest Java 1.6  that Apple provides that gets tickled by the GWT 1.6 compiler.  It manifests as a reproducible HotSpot compiler crash that looks like this:

    [java] Invalid memory access of location 00000000 rip=01160767

When you look into the crash logs that Java produces (found under ~/Library/Logs/CrashReporter/java_*), you can see that HotSpot was compiling a method when it crashed:

 Java VM: Java HotSpot(TM) 64-Bit Server VM (1.6.0_07-b06-57 mixed mode macosx-amd64)
Current thread (0x0000000101843800):  JavaThread "CompilerThread1" daemon [_thread_in_native, id=1756672000, stack(0x0000000168a4b000,0x0000000168b4b000)]
Stack: [0x0000000168a4b000,0x0000000168b4b000]
Current CompileTask:
C2:637      org.eclipse.jdt.internal.compiler.lookup.ParameterizedMethodBinding.<init>(Lorg/eclipse/jdt/internal/compiler/lookup/ParameterizedTypeBinding;Lorg/eclipse/jdt/internal/compiler/lookup/MethodBinding;)V (596 bytes)

The workaround is to disable JIT of some JDT methods using some advanced JVM command-line arguments.  Using Ant, you can tack them on to your GWT compiler task <java> blocks, like so:

<java classname="com.google.gwt.dev.Compiler" classpathref="compileClassPath@{module}" fork="true" failonerror="true">
    ...
    <jvmarg value="-XX:CompileCommand=exclude,org/eclipse/core/internal/dtree/DataTreeNode,forwardDeltaWith" />
    <jvmarg value="-XX:CompileCommand=exclude,org/eclipse/jdt/internal/compiler/lookup/ParameterizedTypeBinding,&lt;init&gt;" />
    <jvmarg value="-XX:CompileCommand=exclude,org/eclipse/jdt/internal/compiler/lookup/ParameterizedMethodBinding,&lt;init&gt;" />

By applying those commands, you ask the HotSpot compiler to skip those methods (relying on the interpreter to run them).  You’ll know it’s working if you see this during your compile:

[java] CompilerOracle: exclude org/eclipse/core/internal/dtree/DataTreeNode.forwardDeltaWith
[java] CompilerOracle: exclude org/eclipse/jdt/internal/compiler/lookup/ParameterizedTypeBinding.<init>
[java] CompilerOracle: exclude org/eclipse/jdt/internal/compiler/lookup/ParameterizedMethodBinding.<init>
[java] Compiling module com.dotspots.ExtensionModule

I’m hoping that Apple eventually gets around to releasing an updated Java 1.6 for OSX.  This bug was fixed in mainline Java a very long time ago!