One of our biggest technical challenges is making Aardvark available on every communication channel, especially IM. We think of Aardvark as a contact, a friend who always knows who to send your question to. We want Aardvark to live wherever people are already talking to their friends, so that it’s always an easy and natural experience to ask and answer questions.

You can already talk to Aardvark on Gtalk (and Gmail chat), MSN messenger (Windows Live Messenger), AIM, Twitter, and email… but one important member of the IM family we’ve been missing is Yahoo! Messenger.

We’re happy to announce that as of today, Aardvark is fully operational on Yahoo! Messenger! We’ve just launched a major upgrade to our IM infrastructure — so in addition to Yahoo! support, we’ve also improved response-time across all IM networks and solved a few lingering problems with MSN logoffs.

I want to talk a bit about the technical challenges of integrating IM networks into the Aardvark application, for the benefit of other developers…

Here at Aardvark we use Ruby on Rails (RoR) and embrace the ecosystem around it. This allows us to move fast, with agility, while keeping bugs and production incidents rare. (Read more about our approach to engineering.)

Integrating IM newtorks in a RoR stack was challenging, especially because each IM network uses a different protocol (often proprietary). A common way to interoperate with the networks is to bridge all protocols over to XMPP, using something like Openfire and its Transport plugin, and then a Ruby XMPP client (the main choice being xmpp4r). That’s what we’ve been doing up until now.

That solution had a series of problems. First, we had to integrate conventional web calls with “IM calls”, without having to write a RoR implementation for IM and without consuming too many request handlers. (As an example: an incoming IM message makes a web call to RoR, which decided to send an IM message back - that’s 3 handlers interlocked right there!)

The second problem with bridging protocols to XMPP is that the protocols are continually changing. This is a well known cat and mouse game — for instance, just recently, Yahoo! Messenger changed their login protocol. It appears to us that the fastest-running cat these days is the Pidgin team and the fantastic work they put into Libpurple - but with our bridging-to-XMPP strategy, we were not taking advantage of that.

And last but not least, when it comes to dealing with a lot of small IM XMPP messages, the Ruby stack we had was having trouble (potentially because of its interactions with the Ruby garbage collector), since most of the Ruby XMPP libraries were meant for smaller deployments — not sending millions of messages per month! Memory leaks threatened to make some components unusable, so it was inherently un-scalable.

So we had to rethink this core component of Aardvark. Ultimately, we took the best of the RoR stack (which allows for a well-tested and agile environment), and the best of an I/O-centric C approach. We designed a asynchronous, mono-threaded C/C++ component that exports ActiveResources to our RoR stack and communicates with it via AR as well (ie through a RESTful API). It speaks many dialects of client IM protocols both with local servers for vark.com as well as remote servers for AIM and the like (natively via Libpurple). We used Lighttpd on the incoming side, and libcurl on the outgoing side.

The end result is a pretty simple, robust, and highly scalable component (we can run tons of these in parallel) — with hugely improved performance over the previous approach. It’s well-isolated from Ruby, while being naturally integrated (thanks to the use of REST and ActiveResource). We’re excited about how this bridges a typical web-app with the legacy IM networks — especially in the context of the discussions about web APIs, XMPP and web hooks models.

I hope others find this helpful! If you have other thoughts on how to deal with these issues, we’d love to hear them.