An overview of message brokers

Introduction

The message brokers have earn a lot of popularity during the past years, some companies have been increasing the efforts to provide a message queuing solution and get a part of the cake. But what are message brokers / message queues? In this article I’ll show you what they are meant for and some solutions that you can find outside.

How does the application communication works?

First of all, before getting into details and understand why they exists, we need to know how the different applications can communicate and exchange information.

Figure #1. Regular application communication

Message Broker

The message broker will help us to avoid the issues associated to the questions shown before. The message broker acts here like a middle man in which it receive events from a source and forward those to the destinations and this message broker has its own storage to save events if the destinations are not available. Some of the benefits is to provide High availability, avoid the “spoken language” limitation as well as working properly with Firewalls.

Figure #2. Message broker communication
Figure #3. Destination is down
Figure #4. Network is slow

Message Queues

Now that we understood how the message broker works, let’s take a deep dive in the Message queues, which are required to use the Brokers. The message queues are First In First Out concept (FIFO) in which the first message that arrives to the queue will be the first in being sent. Just like being on a red traffic light of a single lane road, all the cars will be doing a queue and the first one in that queue will be the first one to get out of there.

Figure #5 Message Queues
Figure #6 Competing consumers

Topics & Subscriptions

Maybe these are the most important concepts when building distributed systems. They follow the same logic as the queues we have just discussed but there are a few differences! and they can also be different between different message brokers.

Figure #7. Topic & Subscriptions

Publish & Subscribe

Now that we have understood the basics of the Message Brokers, the next topic to understand are the publish and subscribe, these are the basics for the Publish / subscribe Pattern, which basically allow us to decouple senders and receivers of messages.

  • The consumers (receiving application) only needs to know the topic to subscribe to.

Distributed Log topics

When working on distributed topics we do not use the subscriptions, here the things work a little bit different. First of all when we are working on distributed topics we are talking about clusters in which we have different servers running our message brokers, for example Apache Kafka.

Figure #8. Reading from a topic
Figure #9. Topic partitions
Figure #10. Consumer groups.

--

--

Sr. Security software engineer working in the DevSecOps area. CompTIA Sec+, C|EH

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Josué Carvajal

Sr. Security software engineer working in the DevSecOps area. CompTIA Sec+, C|EH