How to set up a RabbitMQ cluster on Azure

When I was exploring IOT, I bumped into some interesting protocols which I found very useful for certain type of applications. My requirement was to keep all mobile clients in sync. The protocols like MQTT and  AMQP seemed to be a perfect fit. I started experimenting with public brokers provided by Arduino and Adafruit. Then realised that my application needs all its architecture components to reside in our environment. Continued exploring open source options. Explored Mosquitto,VerneMQ and then finally RabbitMQ. It didn’t take long to figure out that RabbitMQ is the better messaging broker.

For those of you who are beginners to Azure and messaging brokers, here are some links to get you started

  1. https://azure.microsoft.com/en-in/overview/what-is-azure/
  2. https://en.wikipedia.org/wiki/Message_broker

A single RabbitMQ node on a single VM is sufficient to achieve synchronisation among mobile clients. I chose to opt for a cluster of RabbitMQ nodes to handle higher number of connections and message load. Clustering is achieved by connecting all the RabbitMQ nodes. This way the nodes will share users, virtual hosts, queues exchanges and runtime parameters. The advantage of a cluster is if one of the VM’s go down due to excess load, the connections and messages is replicated in other VM’s.

In this blog I will explain how I set up a RabbitMQ cluster on a Linux Virtual Machine in Azure.

Pre-requisites:

  1. Microsoft Azure account
  2. Basic knowledge of Linux commands
  3. Understanding of messaging broker

I picked Linux VM over Windows due to my inclination towards command line interface and ease of installing applications in Linux. There are two ways a RabbitMQ cluster can be set up.

  1. Create individual Linux VM’s, Install RabbitMQ on it, Connect the RabbitMQ nodes installed in each VM.
  2. Install RabbitMQ Cluster package provided by Bitnami in Azure.

The first approach is time consuming as there are multiple steps involved. Microsoft Azure makes our life easy by providing a readymade package which takes care of all the dirty work of creating VM’s, installing RabbitMQ

Login to Azure portal and search for RabbitMQ Cluster in the Azure Marketplace. Select the cluster option from Bitnami and click on create. Below is the screenshot of the Marketplace.

You will be presented with a screen where you need to fill the specifications of the cluster. You can select the number of slaves (VM’s) you need for your architecture and proceed with the form. The decision on VM size is based on your expected application load. I suggest you create a separate resource group for this cluster to keep a tab on the costs incurred. Below is the screenshot of the RabbitMQ cluster creation form.

After the process of buying the cluster is complete you will can view the resources generated in the group. I chose 2 slaves. Below is the screenshot of my resource group.

I have totally 3 VM’s created. One acting as a master and 2 as slaves. Below are the names of each VM that got generated for me.

  1. rabbitmq1
  2. rabbitmq2
  3. rabbitmq3

Note that an IP will be assigned only to a master VM by default. You can choose to assign IP’s to the other VM’s if you intend to access them independently. Also by default SSH will be enabled on port 22 for all VM’s. You need to add inbound security rules to the network security group which gets created along with the cluster creation to be able to access the VM’s using different protocols. My application needs a few ports open. So, I added a few rules to the security group. Below is the screenshot for the same.

Now we can SSH into the VM’s using the IP address assigned to them. An SSH Key is required to login. Below is the command:

ssh –i <ssh_key_filename> bitnami@<ip>

Replace the <ssh_key_filename> and <ip> with relevant values.

Now that we have logged into the VM, enable the management console. Below is the command:

rabbitmq-plugins enable rabbitmq_management

Download “rabbitmqadmin” file from “http://<ip>/15672/cli” and FTP the file into your master VM home directory.

Then copy the file into /usr/local/bin. Below is the command:

sudo cp rabbitmqadmin /usr/local/bin/

Then modify permission. Below is the command:

sudo chmod +x /usr/local/bin/rabbitmqadmin

You can access the management console in the below url:

http://:15672

The http api is available in the below url:

http://:15672/api

You can edit the RabbitMQ config file here:

sudo nano /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.config

To start and stop bitnami service, below is the command:

sudo service bitnami start

sudo service bitnami stop

To enable MQTT protocol on the VM, below is the command:

sudo rabbitmq-plugins enable rabbitmq_mqtt

You can check the cluster status with below command:

sudo rabbitmqctl cluster_status

Queues within a RabbitMQ cluster are located on a single node by default. They need to be mirrored across multiple nodes. Each mirrored queue consists of one master and one or more slaves, with the oldest slave being promoted to the new master if the old master disappears for any reason. Messages published to the queue are replicated to all slaves. Consumers are connected to the master regardless of which node they connect to, with slaves dropping messages that have been acknowledged at the master. Queue mirroring therefore enhances availability, but does not distribute load across nodes (all participating nodes each do all the work). This solution requires a RabbitMQ cluster, which means that it will not cope seamlessly with network partitions within the cluster and for that reason, is not recommended for use across a WAN. Queues have mirroring enabled via policy. Policies can change at any time. It is valid to create a non-mirrored queue, and then make it mirrored at some later point (and vice versa). More on this are documented in RabbitMQ site. For this example, we will replicate all queues by executing below command:

sudo rabbitmqctl set_policy ha-all “” ‘{“ha-mode”:”all”,”ha-sync-mode”:”automatic”}’

That should be it. The RabbitMQ cluster is now up and running.

 

 

Recent Posts

Recent Comments

    Archives

    Categories

    Meta

    Akshay Kunila Written by:

    Be First to Comment

    Leave a Reply

    Your email address will not be published. Required fields are marked *