grack.com

Robert Scoble’s accidental tweet (“Frnégtttrdre”) earlier tonite caused a minor ripple: people wondering if he was announcing a secret project, under the influence of alcohol or wandering around with an unlocked iPhone in his back pocket.

It also makes for an interesting test for real-time search.

An hour after his tweet:

Anyone else I’ve missed?

Conclusions: If you tweet a random word, there’s no guarantee that you’ll get indexed right away.  Additionally, not every service tests unicode querystring parameters.

Read full post

UPDATE: There’s a new domain parameter in rssCloud that makes this DDoS far, far worse.  Since there’s no verification (yet) on rssCloud endpoints, you can now subscribe any server to any rssCloud hub’s notifications.

While researching some of the issues of rssCloud running in a shared hosting environment, I came across a serious vulnerability in the protocol. The vulnerability allows someone to cripple a shared web host. Because of the sensitive nature of this vulnerability, I’m not going to share example code or which shared host(s) are vulnerable.  The fix is easy: follow these security recommendations to close the hole.

The inspiration for this vulnerability was discovered by Nick Lothian’s post on FriendFeed. It turns out that many shared hosting providers route incoming and outgoing HTTP requests through different IP addresses. The process of routing the HTTP requests is usually done transparently by a networking gear outside of the web servers themselves.

rssCloud’s specification infers the endpoint from the REMOTE_ADDR CGI variable at the time of the subscription. It would be very difficult to get an rssCloud subscriber working in a shared hosting environment because every subscription request you make goes out on IP address A, but all of your incoming requests come in via port 80 on IP address B. For some shared web providers, the machines that make outgoing requests are also web servers, serving banner messages or redirects to sales sites. Because they are web servers, they are considered valid rssCloud REST endpoints (returning 200 OK for POST requests on some URLs).

When you put these pieces together, it becomes readily apparent that you can now subscribe your shared host’s outgoing HTTP request IP address to any number of feeds. Considering that Wordpress has 7.5 million blogs that speak rssCloud, there’s a significant number of blogs that could end up pinging the machine.

There are probably a number of other interesting vulnerabilities in this area, such as traffic that travels through a proxy, or an anonymizing service such as TOR. It may be possible to knock one of these offline by subscribing it to a large number of feeds.

The problem with rssCloud is that its subscription request only proves that you can make requests via the given IP address, not that the given IP address is willing to receive them. By adding the challenge parameter I suggested in the previous post, you can now guarantee that the endpoint is willing to receive these requests, making it much harder to subscribe an unwilling participant in the protocol.

Read full post

Copy of an email I sent to http://tech.groups.yahoo.com/group/rss-cloud/messages:

I’ve been looking into the security of rssCloud over the last few days and a number of serious issues have come up. The ones of immediate importance have been reported and fixed, but I’d like to suggest some protocol changes to prevent rssCloud servers from DoS’ing sites or ending up DoS’ing each other.

Here are my implementor guidelines, based on the research over the last few days. These guidelines should mitigate all of the problems I’ve found so far without requiring rssCloud subscribers to make major changes to their code:

  1. An rssCloud implementation MUST validate that the Content-Type of the POST is application/x-www-form-urlencoded.
  2. All rssCloud parameters MUST be read from the body of the HTTP POST (ie: $_POST in PHP or equivalent in other languages). Parameters in the querystring must be ignored.
  3. The path parameter for the callback path MUST begin with a /.
  4. The port parameter, if specified, must be converted to an integer value before constructing the callback URL with it. Any trailing non-digit characters must cause the hub to return a 5xx error.
  5. The subscription callback and all subscription pings MUST include a “challenge” parameter (inspired by PubSubHubbub). The subscriber MUST respond with a response that contains the contents of that challenge parameter as the body of the response. No additional information may appear in the body.

I haven’t taken a look at the SOAP/XML-RPC parts of the protocol yet, but the extra framing should help make them more secure. Someone more familiar with those technologies should run those through their paces.

Feel free to contact me with any rssCloud hub implementation you might have and I’ll run my battery of tests against it.

Read full post

I caught the audio of the rssCloud get-together in Berkeley tonight and it was very enlightening.

One of the first points brought up was the problematic subscription API. The subscription API requires that the endpoint live at the the same IP address as the system making the subscription request. Dave Winer’s response was basically “we can’t and won’t change the protocol because it’s too widely deployed”. He asked that anyone who wanted to change this fork the protocol. Unfortunately, the lack of flexibility in assigning endpoint URLs makes this a very difficult sell for larger organizations where outgoing and incoming HTTP requests are routed differently. I think this was a big mistake on Dave’s part here. There’s a great opportunity to fix the glaring holes in the protocol (those 7.5 million blogs on Wordpress run the same WP rssCloud plugin that I do).

Some other interesting points were brought up, such as the lack of a block button (useless in a distributed web) and ideas for distributed identity. Tunneling was quickly brought up, but the discussion moved on just as quickly.

At one point, someone asked about rssCloud support in Atom. Dave Winer suggested using a namespace for the element and some discussion took place on it. I’m not sure who brought it up, but it will likely be blogged and Dave will point to it to make it official. Noone brought up the already-specified <link rel= tag as an alternative, unfortunately.

Another interesting item brought up was that Wordpress will be supporting Pubsubhubbub as well at some point in the future. It was more convenient to support rssCloud first, so they went with it. Dave Winer and Matt Mullenwag joked that when they do, “just don’t say rsscloud is dead”.

More notes are available from @susiewee’s live twittering.

Read full post

At first glance, both rssCloud and PubSubHubbub have an interesting shortcoming that makes them difficult to use for desktop feed readers. Since both of them require HTTP callbacks to a publicly accessibly endpoint, a user is required to open up a port on their firewall.

It turns out that a subtle difference in the specifications gives PubSubHubbub a big edge in this case. While rssCloud requires your callback endpoint to live at the IP address you make your request from, PubSubHubbub allows you to subscribe any endpoint you wish by specifying a hub.callback url.

So how do we turn this into a real-time feed for desktop clients? Simple: we implement a PubSubHubbub subscriber on a publicly-available, always-on server that receives PubSubHubbub update events and wraps them in XMPP. The XMPP events are transmitted to the desktop client, where it can then process them as if it received the callbacks directly.

The server application doesn’t need to be smart. Only the “subscribe” and “publish” modes of PubSubHubbub’s protocol are required. All it needs to do is correctly route the update subscriptions to the correct XMPP account. In fact, with Google AppEngine’s new XMPP support, you can this in a few dozen lines of code, as I’ve done here:

A PubSubHubbub to XMPP gateway, hosted on Google AppEngine

Try out the gateway by entering your XMPP ID on the main page. This will give you a callback URL that you can use on Google’s main PubSubHubbub hub. Enter the URL for any PubSubHubbub-enabled field as the topic.

The code is simple, though not very robust:

@SuppressWarnings("serial")
public class Subscribe extends HttpServlet {
    @Override
    protected void doPost(HttpServletRequest req, HttpServletResponse resp)
            throws ServletException, IOException {
        resp.setStatus(204);
        XMPPService xmpp = XMPPServiceFactory.getXMPPService();
        JID jid = new JID(req.getPathInfo().substring(1));

        byte[] buffer = new byte[10 * 1024];
        req.getInputStream().read(buffer);
        xmpp.sendMessage(new MessageBuilder().withBody(
                "Got update: " + new String(buffer))
                .withRecipientJids(jid).build());
    }

    public void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws IOException {
        resp.setStatus(200);
        resp.setContentType("text/plain");

        XMPPService xmpp = XMPPServiceFactory.getXMPPService();
        JID jid = new JID(req.getPathInfo().substring(1));

        if (req.getParameter("hub.mode").equals("subscribe"))
            xmpp.sendMessage(new MessageBuilder().withBody(
                    "Subscribing to " + req.getParameter("hub.topic"))
                    .withRecipientJids(jid).build());
        else
            xmpp.sendMessage(new MessageBuilder().withBody(
                    "Unsubscribing from " + req.getParameter("hub.topic"))
                    .withRecipientJids(jid).build());

        resp.getOutputStream().print(req.getParameter("hub.challenge"));
        resp.getOutputStream().flush();
    }
}

Postscript: I really hope that PubSubHubbub gets a new name.

Read full post