How many messages does the technology in your organization deal with on any given day? How many different applications, software programs and hardware devices do you utilize in processing those messages? If the answer is ‘a lot’, it may be time to ensure all those variables can talk to one another as efficiently as possible. An open source Message Broker could be the answer.

“A Message Broker is really a translator. It’s a computer program module that sits between the messaging protocol of the sender and the messaging protocol of the receiver,” said Ray Collins, TDK Technologies Technical Lead. “Middleware like this can customize information for each individual user efficiently.”

How Message Queuing Works

Collins noted that the first message brokers solved a big problem for stock traders, whose workstations in the 1980s were cluttered with multiple terminals connected by mountains of cable to each type of information the trader needed to do their job. Teknekron Software Systems (which eventually became TIBCO) saw an opportunity to replace the various terminals and their siloed applications with a software bus developed by company founder Vivek Ranadive.  The concept utilized a ‘publish–subscribe’ messaging pattern. Publishers send messages to categories rather than to specific recipients. Subscribers receive messages only from categories that are of interest.

“The result was that a single workstation could display programs by plugging into the software bus. The traders could then subscribe to the information they wanted to see,” Collins said.  “The systems consolidated a hodgepodge of data traffic within a single desktop computer.”

Teknekron TIBCO MQ software dominated the financial services industry initially. Then IBM developed competing message queuing software, with its WebSphere MQ commercial platform.  Collins said these proprietary systems have pricing structures which are barriers for many organizations. They also often require different products to service different applications. For example, if an application subscribing to information on a TIBCO MQ suddenly needed to consume messages from an IBM MQ, it couldn’t easily be done. Those factors, and others, led to alternatives including Java Message Service (JMS) and then open source message broker applications.

Open Source Message Brokers: Two of Many Options

Open source software products have become attractive options in developer circles. They are generally free to download thanks to the collaborative and public way the software is developed. There are many open source message queuing options available.

RabbitMQ was one of the first and is the mostly widely deployed of the open source options. Collins said RabbitMQ is very low cost to set up and to operate.  Since it can be set up on virtually any kind of server, it’s very attractive for small businesses, start-ups, or large companies. It supports multiple messaging protocols, message queuing, delivery acknowledgement, flexible routing to queues, and multiple exchange types.

“Because of the way Rabbit MQ is set up, the producer is not aware of the consumer. And the consumer is not aware of the producer. They are only aware of the middleware they are connected to,” Collins said. “Delivery acknowledgement is really important.  Rabbit MQ needs to know that the right message has been delivered to the right consumer.”

RabbitMQ is also extremely fast, capable of handling more than one million messages per second. However, Collins noted it is not designed to store messages long-term. 

Apache Kafka is another open source option. This software started out at LinkedIn to connect different internal systems. At the time, LinkedIn was moving to a more distributed architecture and needed to break away from monolithic approaches to data integration and real-time stream processing. Apache Kafka is designed for high volume publish-subscribe messages and streams. It is meant to be durable, fast, and scalable. In addition, Apache Kafka provides message storage. But it is expensive, in part because it requires multiple servers.

“Apache Kafka has message persistence built into it, which can be a key consideration,” Collins said. “Apache Kafka is especially attractive for large scale enterprise applications”.

Message brokers are popular in an increasing number of applications, especially where several different servers are interconnected with tightly-coupled consumers. This solution consolidates information from all kinds of sources down to one and provides a way to communicate with one another without consumers being tightly-coupled.  The use of data analytics or the implementation of the Internet of Things (IoT) into business operations are also applications where a centralized translator of information and messages could be considered.

An IT pro who'll take the time to learn my business. Is that too much to ask?