FOSDEM

Pieter Hintjens will give a talk about OpenAMQ at FOSDEM 2009.

Could you briefly introduce yourself?

Like Kuifje (Tintin), I search for the truth, defend the weak, and stand up to bullies. Unlike Kuifje, I don't have a white dog, and I get older each year. My work is designing and writing software systems: large architectures that solve complex problems. I'm also a student of human nature and its manifestations in politics, economics, and society. Software design is about overcoming human, not technical, limitations. I've been a free software developer since 1991, and active in various political issues like software patents. I've been writing various books on and off for a decade but those seem to take too much bandwidth, so now I stick to short writing (www.aliopacto.com, www.devilswiki.com).

What will your talk be about, exactly?

I'll talk about three things, at least. First, AMQP, a new protocol, which we'll cover further in this interview. Second, OpenAMQ, a free implementation of AMQP, which is interesting for a couple of reasons that I'll keep as a surprise. Lastly, RestMS and Zyre, which are respectively a specification for, and implementation of, a RESTful messaging service that interoperates with AMQP. However, this won't simply be a technical briefing, I'll present a more holistic view of what it takes to make a project like OpenAMQ work.

What do you hope to accomplish by giving this talk ? What do you expect?

I want to promote our open source products and try to convince those listening that there are good reasons to look at AMQP, OpenAMQ, RestMS, Zyre, and the other designs we've made. Sturgeon's Law applies to software as to no other field, and it's often hard to be heard above the cacophony of "Bigger! Better! Newer!", especially for small teams like ours.

So it's a privilege to get a chance to speak at FOSDEM. I'll do my best to keep the sales talk off the stage and instead stick to useful lessons we learned, which could help other teams.

Which problems does AMQP solve?

It is, today, relatively cheap to make applications. It's still expensive to connect them together. AMQP is the first serious attempt at a standard protocol that does that. Until AMQP, each product that promised connectivity did it their own way. So AMQP does what any free and open digital standard should do: gives customers maximum choice of suppliers. It should do to messaging what SQL did to databases.

Why would a large company, being able to afford commercial messaging products, choose for an open source AMQP product?

Even - especially - large companies love to save money. AMQP offers choice of suppliers which means the lowest possible costs for licenses, support, and material. The competition means that AMQP products are faster and need less hardware. The openness means that AMQP skills are cheaper. No more Wizards. The euro difference between traditional commercial messaging and OpenAMQ is as much as a hundred times.

Switching messaging infrastructure can be very risky. We're talking of critical systems, like Dow Jones Company, who are now running the Dow Jones Industrial Average on OpenAMQ. One really does not want data to be lost, received late, or more than once! The risk is offset by the huge gains, so much that we've seen firms where OpenAMQ was the first ever open source product they used.

If the AMQ model for a business messaging server is so simple, why hasn't it been invented before?

In fact AMQP is not as simple as it could be. This summer I spent a long time looking at complexity that remains in the protocol, and designing ways to remove that. The result is RestMS, which does much the same as AMQP but is considerably simpler.

Simplicity is thus the destination, not the departure point. Initial designs are complex, since complexity is cheap. Any fool can build a complex system. It takes skill, and time, to build a simple system (which does real work, of course).

AMQP solves problems until recently only seen by large firms. The whole concept of joining dozens or hundreds of arbitrary applications together does not interest most protocol designers or FOSS developers. So AMQP could only emerge when large banks like JPMorganChase decided they needed it, and paid experts like us to make it.

Despite appearances, software technology moves slowly, and things happen when the time is right. AMQP's time is now right: there is a market for this technology.

One advantage of AMQP is that it's language-independent. In which programming languages do AMQP implementations exist? How does it compare with Stomp, which has implementations in more than ten languages?

Stomp was actually written as a response to AMQP, and is the wire level format of a single product, ActiveMQ. James Strachan worked for a couple of weeks with me on AMQP at the very start of the project, then decided to make Stomp. There is one Stomp implementation (the ActiveMQ server) and clients in ten languages. Stomp belongs to ActiveMQ, and reflects the needs and capabilities of that product. A business that uses Stomp does not get any real benefit, they are dependent on a single vendor.

AMQP was written without regard for any particular product, and we've seen five or six independent server implementations of AMQP plus clients in C, C++, Python, Ruby, Java, C#. AMQP has larger ambitions than Stomp, and lives independently from any implementation. Businesses that use AMQP immediately get a choice of multiple competing products.

This is the main reason for a free and open digital standard: choice. A documented format is better than none, but it's not the same as a free and open standard.

What is the unique selling point of OpenAMQ compared to other AMQP implementations? And what parts of AMQP are not implemented in OpenAMQ?

OpenAMQ does not implement persistent messaging (queues on disk). We consider that to be complex, unreliable, and for most cases, unnecessary. We focus on speed and reliability. Recently we found what appears to be the last stability bug: after 2^31 socket operations, the server would crash due to an array index error. Always use uint for array indices! But 2^31... that's a lot of messages. We hit that after six hours at one client. The server now runs forever without any issues, and can process around 600,000 512-byte messages per second (in and out combined).

This kind of robustness comes from simplicity. It comes from understanding where traffic builds up, and preventing that. It comes from very fast multithreaded client APIs. It comes from not doing certain things, and so in OpenAMQ we're not driven to add features but rather to make what we have more stable, faster, and easier to use.

So, federation and failover work with a single command line option each. I like this kind of essential simplicity. Also, OpenAMQ spreads into areas like RestMS, which opens the door for whole new messaging possibilities. Imagine: connect your AJAX web application to an AMQP data feed, with no special programming.

How big is the OpenAMQ community? How many persons of this community are working for iMatix?

On the dev mailing list I count 85 people, but few of them are contributors to the core code. There are reasons for that, mainly that we use a lot of meta programming and that keeps people out. When we get a subproject where people want to contribute we spin it off as an independent project on Github. And people help a lot with bug reports, suggestions, the usual stuff. The iMatix OpenAMQ team is about 5 developers, and several more support people. All of us work on multiple projects, not just OpenAMQ.

You are a regular speaker at FOSDEM, how was the response to your talks the preceding years?

Oh, people find the talk of software patents to be depressing. It's like when you ask someone "how are you?" and they tell you their cancer story. Sure, it's important, but it's not the stuff of parties.

Still, I found the FOSDEM participants to be surprisingly tolerant of this guy who keeps popping up and saying, "BEWARE! They want to steal your software!" For that, thanks! I promise to not do it again.

This interview is licensed under a Creative Commons Attribution 2.0 Belgium License.