VMware vFabric Blog: 2 Lessons on RabbitMQ from HighLoad – Extending the server with Plugins and HA Queues

VMware vFabric Blog: 2 Lessons on RabbitMQ from HighLoad – Extending the server with Plugins and HA Queues:
Recently, I presented at the HighLoad++ conference in Moscow Russia. Many of the developers I spoke with were interested in two pieces of information about RabbitMQ, and I wanted to share these with a broader audience:

• Extending through Plugins
• High Availability Queuing

If you haven’t heard of HighLoad++, the conference is pretty special since it focuses only on high traffic websites (mostly from Russia). The main point of the conference is to talk about new architectures and approaches for highly complex systems and covers things like:

• Designing scalable, fault-tolerant architectures
• Queuing systems
• Sharding and clusters
• Management and system administration

You get the chance to learn directly from the developers of apps like Yandex and Mail.ru. At the same, time, you get to meet and learn from core developers of products like MySQL, PostgreSQL, and MariaDB.

RabbitMQ Plug-Ins, Exchanges, and Authentication Mechanisms

In my case, I was invited to talk about RabbitMQ. In particular, I introduced several not so well known features about RabbitMQ: plugins, different kinds of exchanges, and authentication mechanisms. The talk showcased how easy it is to extend RabbitMQ with your own plugins.
In covering plugins, there are a few things of value to mention. If you didn’t already know this, RabbitMQ has an extensible plugin system where you can write your own plugins. To give you an idea of what you can do with plugins, the application developers maintain several plugins like:

• LDAP authentication
• A management/monitoring API over HTTP and browser-based UI
• A shovel to move messages from a queue on one broker to an exchange on another broker
• A gateway to expose AMQP via STOMP

As you can see, this type of plugin system exposes a lot of capabilities – from a design and architecture perspective, the way third parties can extend this is really well thought out. There are many “hooking points” to extend from – you can add new message routing algorithms, add your own full-blown management interface, access functionality outside of AMQP and integrate a new protocol like STOMP or MQTT.
A topic that is quite interesting on its own is message persistence. Developers often want to know if it’s possible to plug in your own message store with RabbitMQ – perhaps with something like PostgreSQL. Well, you can add your own message store by using plugins, but you might want to read this first.
In short, it’s up to you how many knobs you want to tweak on RabbitMQ to get the most out of this messaging and queuing server.

RabbitMQ and HA Queues

When I spoke with some of the attending architects and developers later on, I learned they had a lot of interest in learning more about RabbitMQ support for HA Queues, consistent hashing exchanges, and flow control support – features that make RabbitMQ a really scalable message queue server. This topic deserves it’s own post, but let me call out a few things for anyone that is looking into the topic and also tee up an upcoming post.
RabbitMQ’s model is fairly simple:

• A producer sends messages.
• A consumer receives messages.
• A queue is a buffer – a “mailbox” for messages. Many producers can send messages to a queue and many consumers can receive from a queue. Basically, the queue holds a list of tasks to be done later.
• An exchange receives messages from producers and pushes it to queues.  There are a few different types like Fanout – this one broadcasts all messages to all the queues it knows. You can learn about Direct, Topic, Headers, and more about Fanout exchanges here.
• Bindings define the connection between exchanges and queues.

We normally look at two key requirements with high availability queuing.

• The message should not get lost if the consumer dies.
• The message should not get lost if the RabbitMQ server dies.

The concepts to look into include durable queues, persistent messages, wrapping the (producer’s) publish code in a transaction, and clustering RabbitMQ. This documentation on highly available queues is a great place to start, but you can find all these topics in the documentation or by searching online.
The RabbitMQ HighLoad++ Slides — Coming to London Soon

One of the attending developers told me after the talk: “Today, I’m presenting about a server we created for messaging. Your talk showed me that, sadly, we re-implemented parts of RabbitMQ.”
All in all it was a very nice experience and a conference worth adding to your calendar.  From November 5-7, I’ll be presenting again at the Erlang Factory Lite in London. I’d love an opportunity to discuss RabbitMQ with you, please stop by. Here are the slides:

>> For more information on RabbitMQ:

• Read my book
• Download a trial
• View Documentation
• Find more Resources

About the Author: Alvaro Videla is a software developer on the Cloud Foundry team within VMware in Switzerland. Before moving to Europe, he worked in Shanghai to build one of the largest German online dating sites. Alvaro Co-wrote the book “RabbitMQ in Action” for Manning Publishing. Some of the RabbitMQ open source projects can be found here . In addition to programming, Alvaro enjoys music, books, and traveling with his wife.