Importance of Message Brokers
what is a Message broker?
A message broker is a software that allows applications, systems, and services to communicate and share data with each other. This is accomplished by the message brokers by converting messages between formal messaging protocols. This enables interdependent services to communicate directly with one another, even though they are written in different languages or run on different platforms.
Message brokers often use a substructure called a message queue to provide reliable message storage and assured delivery. The message queue stores and orders the messages until the consuming applications can process them. Messages are placed in a message queue in the same order in which they were sent and remain there until receipt is verified.
moreover, Asynchronous messaging was also made possible by the message brokers. It prevents data loss and allows systems to perform even though they are experiencing intermittent connectivity or latency problems, which are typical on public networks. Asynchronous messaging ensures that messages will be delivered once (and once only) in the correct order relative to other messages.
Why we need Message Brokers?
Think about the HTTP request we make, when a client sends a request, server will respond immediately. can we use this same kind of communication system to the application which runs on different platforms and implemented by multiple programming languages, to communicate with each other?
- What happens when a message you sent to another endpoint of yours, takes time to respond back because it needs to do some computations or ask another endpoint for a result?.
- what to do if the server is not active when we send the data?
- If a server sent a message when we are offline, how to get that message after we come online?
It could be difficult to maintain integrity and availability when handling interactions between our applications.
Therefore, to avoid these issues, as a solution, message brokers were introduced. Message brokers are responsible for validating, storing, routing, and delivering messages to their appropriate destinations. They act as intermediaries between senders and receivers, allowing senders to send messages without knowing where the receivers are, whether they are active or not, or how many are receiving.
Types of Message broker Models?
1. Point-to-point messaging
This is the distribution pattern used in message queues where the sender and receiver have a one-to-one relationship. Each message in the queue is only sent to one receiver and consumed once. When a message must be acted on only once, point-to-point messaging is used.
Examples: payroll and financial transaction processing. here, both senders and receivers need a guarantee that each payment will be sent once and once only.
2. Publish/subscribe messaging
In this message distribution pattern, the producer publishes each message to appropriate topics. Multiple consumers, on the other side, subscribe to topics from which they want to receive messages. All messages published to a topic are sent to all applications that have subscribed to it. This is a broadcast-style distribution method in which the publisher of the message has a one-to-many relationship with the consumer of the message.
for example, an airline provides updates about the landing times, multiple parties such as ground crews performing aircraft maintenance and refueling, baggage handlers etc could make use of the information.
Let’s Discuss an available Message broker briefly
There are plenty of Message brokers available to use. For Example:
- Apache Kafka, Apache ActiveMQ, RabbitMQ, ZeroMQ etc.
Let’s look at one of the above briefly
Apache Kafka
what is Apache Kafka?
Apache Kafka is a real-time distributed data streaming platform that allows you to publish, subscribe to, store, and process streams of records. It’s designed to manage data streams from a variety of sources and deliver them to a variety of consumers. In other words, it transfers vast quantities of data not just one-to-one, but also one-to-many.
How Apache Kafka works?
Kafka is a distributed system that uses a high-performance TCP network protocol to communicate between its servers and clients. It can be deployed on bare metal, virtual machines, and containers in both on-premise and cloud environments.
Servers: Kafka is run as a cluster of one or more servers that can span multiple datacenters or cloud regions. These servers import and export data as event streams on a continuous basis in order to integrate Kafka with our existing systems, such as relational databases and other Kafka clusters. The Kafka cluster is highly scalable and fault-tolerant. In case if any of its servers fails, the other servers will take over their work to ensure continuous operations without any data loss.
clients: They allow you to build distributed applications and microservices that read, write, and process streams of events in parallel, at scale, and fault-tolerantly, even when there are network or system failures.
When to use Apache Kafka?
- Real-time data processing
Ex: in the finance domain, it is important to block fraudulent transactions instantly when they occur. we can do this by Apache kafka while real-time transaction are processing on.
- Application activity tracking
Each event that occurs in the application can be published to the dedicated Kafka topic. For Ex: User clicks, registrations, likes, time spent by users on certain pages, etc. — all these events can be sent to Kafka’s topics. consumers can subscribe to those topics and they can receive data for different purposes such as monitoring, analysis and so on.
- Logging and/or monitoring system
It is possible to publish logs into Kafka topics. The logs can be stored in a Kafka cluster for some time.
