Stack Glue Using Node.js & Redis

One of our requirements at Kicksend is to get messages proxied out from our Rails app to the XMPP messaging server on-demand during a HTTP request. This is trickier than you would expect. A typical XMPP session consists of a multi-step authentication and then usually a persistent connection per-user, something that doesn’t play well within a typical short HTTP request cycle. While most of Kicksend’s messaging triggers are on the client-side, there are a few instances when we need to communicate server-side from the webapp to the XMPP server (for example, when adding a new connection), and doing an authentication each time produces a noticeable slowdown that degrades user-experience.

Xmpp_proxy_1

The solution to this problem involves setting up a XMPP component that hooks into the XMPP server directly, acting as a trusted proxy. Any messages you forward to this component will get passed on directly to the server. Indeed, many of the bigger XMPP-backed apps you’ve used have some custom components running in the background in addition to the XMPP server. 

XMPP components can be written in any language as long as they conform to the spec. At the same time, they should be able to handle a flood of messages that might occur at any time and also be easy enough to connect to from other sections of your app. While I was doing some poking around on Github, I noticed xmpp.js, an XMPP component library for Node.js – after doing a quick test, we realized this worked really well.

Next up, we need a way to reliably get messages to the new proxy-component. The naive implementation would be to maintain a persistent connection to the proxy-component from each instance of the Rails app – but this is a horrible idea in practice. Honestly, the less things you have connected persistent to your Rails instances, the better. We were already using Redis in our stack, so it made sense to use a Redis list as a queue instead of setting up another direct connection. Now the new architecture is as shown in the image below.

Xmpp_proxy_2

Now, whenever we need to pass a message to the XMPP server from the webapp, we stick it into a special Redis list. The proxy-component is now connected to the XMPP server and also connected to Redis. Using Redis’ BLPOP feature, the proxy-component ‘listens’ to the list, and forwards any new messages to the XMPP server. BLPOP is especially suited for this setup since it blocks the redis connection till a new item shows up in the list, making it quite zippy (otherwise you’d have to poll the list every few seconds, not as great or fast). 

We have this running internally on Kicksend, and it’s proving to be quite reliable. Sure, architecture-wise this adds a new point of failure but we’ve ensured that it’s isolated to a smaller section of our app and constantly monitored. At the same time, we’re also making sure that only messages that can afford to be lost run through this, and our users see direct speedup results in some of the more important UX elements of our app. We’re all about optimizing user-experience, so this setup is a definite win in our eyes.

Kicksend is beautiful, powerful realtime filesharing for your family and friends. Signup here.

PS: XMPP folks who want to use this setup, there are a couple of gotchas. Make sure your XMPP server isn’t checking “from” attributes in the component configuration. Also, older versions of Redis have issues with maintaining BLPOP connections alive. Make sure you’re on the latest.

3 Comments

  • Reply May 6, 2011

    Julien Nakache

    Great post, thanks. I’ve never noticed BLPOP and now I wonder why Resque is now using that feature instead of polling Redis repeatedly…Anyway, as you’re using Rails, I guess you developed your component using Xmpp4r. I’d love to know what your think about this gem, esp. regarding load handling.Cheers,Julien

  • Reply May 9, 2011

    gautamrege

    Nice post. I was wondering if you looked at the Redis PubSub feature?If you want your Rails stack, any websocket and any other components (not necessarily XMPP components) to get push messages, you could use this. Since it has wild-card support for the events along with Json payload, I have found it very scalable.What do you think?

  • Reply May 9, 2011

    Julien Nakache

    Sorry… I just re-read the first part of the article (and the title) and looks like your using xmpp.js… Then, I have another questions : is redis the only bridge between your xmpp component and rails ? If yes, how do you communicate back in between xmpp.js and rails ? something like Resque ? if not, do you also access your main database (mysql?) with js drivers, or something else ?Thx

Leave a Reply

Leave a Reply