AMQP: Introducing the Advanced Message Queuing Protocol

Services must be able to communicate with each other. Information is exchanged and data transmitted. To many people it sounds easier than it actually is, because there are several difficulties to overcome with such conversations between different applications: In IT, for example, there are language barriers (in the form of various programming languages) and you have to stick to a protocol so that it does not end up in chaos.

The Advanced Message Queuing Protocol, or AMQP for short, promises a solution: a common protocol that can transport the information through a mediator. How does it work? And what else can AMQP do?

The Advanced Message Queuing Protocol has been developed since 2003 and, by way of exception, does not originate from a dedicated tech company, but was first conceived by JPMorgan Chase, a US bank. There, they decided to continue designing the project together with others. However, in addition to IT companies such as Cisco, it was the financial sector that continued to be primarily interested in AMQP. Why is that? Because, there more than anywhere, time is money! The transmission of information plays a major role at a bank, a credit card company or an exchange. Several hundred thousand messages are exchanged per second there. If news arrives too late or not at all, it can be expensive.

Since no commercial product was able to cope satisfactorily with the requirements at that time, project manager John O'Hara chose to conceive his own, separate protocol. O'Hara’s focus was on open standards such as TCP/IP and he decided to also make AMQP freely available, in order to advance the development of the protocol. Meanwhile, the OASIS working group, a non-profit organization, was working to develop the protocol.

AMQP single-handedly solves several problems: the protocol (in cooperation with a messaging broker) ensures a robust data transfer, while also allowing messages to be stored in a queue. This in turn enables an asynchronous communication: sender and receiver do not have to work around each other. The recipient (consumer) of the message does not need to directly accept the information, process it, and confirm receipt to the sender (producer). Instead, they collect the message from the queue when they have capacity available. Meanwhile, this gives the producer the opportunity to continue working - so there is no idle time.

The success of the relatively recent protocol also has to do with interoperability. The Advanced Message Queuing Protocol establishes a common basis of its own. This even allows the different applications to be written in different programming languages. In this way, even programs in different organizations can easily communicate with each other. And as AMQP is freely available, any company can use the protocol at no extra cost.

In the OSI model AMQP acts on the application layer, meaning it has direct contact with the various programs. IMAP (for email), FTP (for transferring files) and IRC (for instant messaging) are also active on this layer. To transmit the messages, the protocol relies on mediators, or messaging brokers. These take over the distribution of messages to different recipients according to defined rules. AMQP also controls the behavior of these mediation servers.

The Advanced Message Queuing Protocol therefore refers both to the communication between various participants and to the behavior of the brokers themselves. These contain their instructions from the messages.

In the cosmos of AMQP there are three participants and one item:

• The message is the core element of the whole communication.
• The producer creates a message and sends it.
• The messaging broker distributes the message to different queues according to defined rules.
• The consumer takes the message out of the queue that they can access, and edits it.

The mediation of the message is paused again with the broker. The exchange accepts the messages and routes the data to the correct queue. The exchange learns which queue the message belongs to from the binding. There are four different ways in which the exchange forwards messages.

The first kind, the direct exchange, sends messages to exactly one recipient and works with routing keys. One of these keys is given to the message. A queue in turn has a binding key. This identifies the queue to the exchange. If the routing key and binding key match, the message can be forwarded to the queue and therefore to the recipient of the message. It is also possible that a queue has several binding keys and therefore also comes into question for several routing keys. Conversely, multiple queues can also share one binding key, which is called multiple binding. The exchange replicates the message and sends it to multiple recipients.

The fanout exchange works similarly. However, here the broker completely ignores the routing key. Instead, the exchange routes a message to all available queues and thereby duplicates the information. The topic exchange works in a different way. Similar to the direct exchange, routing key and binding keys are matched against each other, but there doesn’t have to be an exact match present. Instead, you use wildcards. This makes it possible to specifically provide messages for multiple queues.

Finally, the headers exchange acts with the header of a message, instead of using a routing key. This contains values that are matched with the binding. The argument with the name x-match determines if all values must match (value: all) or only one must match the binding (value: any). While the former corresponds to the direct exchange, the latter can produce the same effect as a topic exchange.

A frame is the basic unit with AMQP. A connection consists of the ordered sequence of frames. Order in this case means that the last frame must not arrive at the receiver until all other frames have first reached their destination. Each frame can be divided into three segments (in version 1.0):

• Frame header: This mandatory header has a size of 8 bytes. Here you will find information that determines the routing of the message.
• Extended header: This area is optional and has no set scope. It serves to expand the header in the future with further information.
• Frame body: The body contains the actual data to be transferred. The size is freely selectable. However, this area can also be left empty, then the frame only serves to maintain the connection.

The body of a frame, in turn, can take nine different forms:

• open: Negotiates the connection parameters between broker and client.
• begin: Indicates that a connection is starting.
• attach: The message is appended with a link that is necessary in order to use the data transfer.
• flow: Changes the status of a link.
• transfer: The actual message is transmitted with the transfer frame.
• disposition: A disposition frame provides information on changes to the information delivery.
• detach: Removes the link.
• end: Indicates that the connection will be terminated.
• close: Terminates the connection and declares that no further frames will be sent.

Each queue has its own name, which identifies it to the other participants. Either a client or the broker determines the name. A queue is realized by a memory, which can either permanently sit on a hard disk or briefly in RAM. The permanent variant guarantees that the queue will continue to exist even after restarting a broker. But it does not guarantee that messages are also permanently secured: here, it depends on the message as to whether they are still available after a restart.

What happens when a consumer cannot recall the message in queue, because, for example, the client or the connection has broken down? It is possible to decide whether a consumer must properly acknowledge the receipt of a message or whether the mere delivery is sufficient for success. If the first variant is chosen and the consumer does not need to send back a message, the broker tries to send the message to another consumer or to reach the actual recipient again. However, if the variant is activated without confirmation and the consumer does not retrieve the message, it will be lost.

But it is also possible for a client to deliberately reject the acceptance of a message. This may be useful if message processing does not work. The feedback from the consumer causes the broker to either completely delete the message or incorporate it back into the queue.

There are currently two completely independent versions of Advanced Message Queuing Protocols. Version 1.0 was developed by the OASIS group. However, the slightly older version 0-9-1 is often used, especially with RabbitMQ. They are not compatible with each other. Version 1.0 differs primarily by the reduced importance of brokers, bindings and exchanges. 0-9-1 doesn't need this, but neither does it prohibit the use of such middlemen.