Ever since Dr. Peter Rodgers first coined the term "Micro-Web-Services" during a presentation at the Web Services Edge conference in 2005, the IT industry slowly shifted from monolithic design to microservices.
Netflix made the decision to break their monolith into microservices back in 2009. Best Buy started to transform their monolithic architecture to a microservice architecture in 2010. eBay introduced microservices in 2011.
If you go back to 2001, Amazon’s retail website was a large architectural monolith. At Amazon’s re:Invent 2015 conference, Rob Brigham explained the change. By separating their backend into loosely-coupled services, isolating the services irrelevant in terms of their tech stack, and having smaller dev teams focus on a well-defined service, they could deliver software more efficiently and at scale.
Today, backends almost always follow the microservices architectural design pattern. While message queue-based inter-process or inter-thread communication dates back to the early 1980s with UNIX System V API or perhaps other real-time OS kernels, the birth of message brokers operating in a network of different computer systems as we know it primarily started at 2000s.
RabbitMQ v1.0.0 was released on 1 July 2007. Kafka was originally developed at LinkedIn, and was subsequently open sourced in early 2011. Amazon SQS went into beta in late 2004 and became generally available in mid 2006.
Once a microservice architecture is chosen as the path for the development of the backend, a decision must be made for communication between the services.
Synchronous protocol: like HTTP (REST, SOAP or any other RPC)
Asynchronous protocol: message queue protocols like AMQP(RabbitMQ), Apache Kafka, STOMP, MQTT, etc.
A synchronous approach is more error-prone, harder to debug, and harder to recover. Many industry professionals pass this option over.
On the other hand, the asynchronous approach gives the architecture a single, solid message bus. This makes it easier to debug, less error-prone, and more secure. Data is delivered between services in a reliable fashion, making asynchronous the obvious choice.
That is, unless you have legacy services in an older programming language, or have an older infrastructure that can’t be easily modernized. Moving into the cloud-native era however, usually requires businesses to choose an asynchronous design.
Once the asynchronous approach is chosen, the architects need to decide what message broker software and the message queue protocol to use.
My journey of understanding the similarities and differences began with the objective of adding support of several message queue protocols like AMQP, Apache Kafka, Redis etc. to Mizu. Mizu is a Kubernetes-aware deep inspection tool that lets you inspect the traffic between your microservices.
Implementing the protocol dissectors for such message queue protocols gave me in-depth knowledge for both the protocols and the broker software.
Here is a list of prominent message broker software and cloud services in the year 2021:
Here is a message queue protocol comparison matrix based on the client libraries' perspective:
Note: headers indicates a dictionary/map with arbitrary amount of keys while attrs indicates a limited set of key-value pairs.
Note: We didn't include XMPP into this list of comparison because it's not so popular in the area of microservices.
Similarities:
Differences:
The choice of the message queue protocol is actually more important than the choice of the broker software. This is because if you choose a more generic protocol, it's easy to replace the broker with something else.
Advanced Message Queue Protocol (AMQP) is a TCP-based binary protocol, which has become an ISO and OASIS standard. It's primarily used by RabbitMQ. There are different versions of the AMQP specification and they are, in some cases, wildly different from the previous one. RabbitMQ uses AMQP version 0-9-1 making that version the most commonly-used. The other versions are 0-8, 0-9, 0-10 and 1.0. AMQP version 1.0 is the most common among the cloud service providers and RabbitMQ also supports 1.0 via a plugin. 1.0 is the OASIS standard.
Apache Kafka is both the name of the message broker software and the protocol itself. As of 2021, there are 12 versions of the protocol and client libraries implement all of them at once.
It's a project developed and maintained by the Apache Software Foundation, written in Scala and Java.
The Apache Kafka team chooses to define their own protocol, instead of adopting AMQP or STOMP for example. This is because they hold that the protocol defines the implementation, therefore it reduces their freedom for creating a truly distributed messaging system and performing certain optimizations.
Much like AMQP, Streaming Text Oriented Messaging Protocol (STOMP) is a text-based protocol not specific to a certain message broker software. But it lacks lot of functionality and optimizations that other protocols have. On the other hand, the protocol's simplicity makes it easier to adopt. Therefore, there are many client libraries for this protocol and RabbitMQ supports STOMP via a plugin as well. It's logical to use STOMP for simpler use cases or for fast prototyping.
Message Queuing Telemetry Transport (MQTT) is the message queue protocol for the Internet of Things (IoT). It's an ISO and OASIS standard. The most notable brokers that support MQTT are Eclipse Mosquitto and HiveMQ.
Redis Serialization Protocol (Redis/RESP) is the protocol of Redis, an in-memory key–value database. Redis is not technically a message queue software, but through some client libraries, it can be used for that purpose. It's mainly the choice for the people who are so in love with Redis in their tech stack that they also use Redis as the message bus.
The message queue services of cloud providers are the way to go if you don't want to maintain the infrastructure, you want to automatically scale, and you already use cloud services directly as your backend such as Firebase or serverless functions. They are all HTTP-based, so don't expect to have the efficiency of a dedicated message queue protocol.
In conclusion, go with AMQP (RabbitMQ) or Apache Kafka in general. They work and scale well.
Don't go with STOMP; it's a wasted potential. It hasn’t been adopted well and it’s not really optimized in any way.
If you're an IoT company then go with MQTT. It's cut out for IoT.
If you're in love with Redis and can't handle adding a new technology to your stack then go with Redis.
If you're not maintaining your own infrastructure and you’re heavily dependent on cloud services already then go with a cloud service for the message queue solution as well.
UP9 recently released Version 0.14.0 of its open source API Traffic Viewer, Mizu. Read more about the changes here.
The most notable update was increased protocol support. This has brought support for Kafka, Redis, and AMQP (aka RabbitMQ). These protocols are in addition to REST and gRPC.
Additionally, you can now use Mizu to view API traffic between microservices that are using numerous protocols to communicate. For a quick guide on how to see multiple-protocol support in action, go here.
Mizu is available to download as an executable binary or as source code directly from its GitHub repository.
With high quality products and considerate service, we will work together with you to enhance your business and improve the efficiency. Please don't hesitate to contact us to get more details of best message queue.