QMQP: The Quick Mail Queuing Protocol

March 20, 2002

QMQP is used to send an email message from one computer to to another over a TCP network. This functions like SMTP but faster, and of course, more secure. Dan Bernstein developed QMQP with Qmail and its primary use was to allow one host to offload delivery of a message to another host. Since QMQP is enormously faster than its SMTP counterpart, it can be used to shuffle messages from one system to another without loss of speed,

QMQP is used in a Mini-Qmail environment to allow email to “pass though” (generally a firewall) one computer to another. The Mini-Qmail Server accepts a SMTP connection from external clients and opens a QMQP protocol connection to its designated QMQP server machine, passing the message into the queue of the primary mail handler. By doing this the Mini-Qmail system is the only machine exposed to the internet for security issues.

Qmail-qmqpc, the QMQP client program is a drop in replacement for qmail-queue and maintains the exact interface. The difference is that qmail-queue places a message into the queue of the local qmail instance, while qmail-qmqpc opens a connection to another qmail server (where its counterpart qmail-qmqpd is listening on port628) to place the message in the queue of the other server. That is, qmail-qmqpd invokes qmail-queue to place the messge in its queue.

You will find that for a QMQP environment like Mini-Qmail, the qmail-queue program has been replaced with a symbolic link to qmail-qmqpc. This makes all programs use QMQP to send mail. Talk QMQP to me! QMQP uses Bernstein’s netstring format to deliver the message. This format is safe in terms of funny CRLF (Carriage-return + line-feed character) line separators used by SMTP, and also prevents unexpected characters from upsetting the conversation.

The format of the netstring is simple: string_len + “:” + string + “,” where the string_len is the ASCII (think “human-readable”) representation of the string length, and never with leading zeroes. The zero-length string is represented by “0:,” as its netstring, and “string” is represented as “6:string,”. The trailing comma is used to verify that the string ended as expected.

The format of the QMQP conversation is simple. The client opens the connection, sends a single netstring (containing nested netstrings of the message, sender, and recipients). After the full string is received by the QMQP Daemon, it speak a single character: ‘K’ for success, ‘Z’ for a temporary deferral, or ‘D’ for a permanent error. That’s all. So a basi conversation is of the form:

(Client) 393:30:Message body...,15:Fsender_address...,26:Trecipient_address,,

Note that the lengths are just demonstratable numbers, not the actual lengths. Also, like qmail-queue (and most of qmail), the addresses are all prefixed by the status character: ‘F’ for from address, and ‘T’ for the “to:” address. Finding the QMQP Service Qmail has a pretty simple method to find your QMQP server... actually a little too simple! In the $QMAIL_HOME/control/qmqpservers control file, there is a list of IP addresses, one server per line. Qmail starts with the first server on the list, and if it doesn’t respond, tries the next one, and so on. The servers are expected to be in a dotted-quad format.. you know, like 127.0.0.1 but with your ip address. I don’t think you’d want to use the loop-back address Conclusion That’s about all there is to a QMQP service. Heres a basic cronological order of events:

1. Application Program has a mail to send
2. Qmail-Qmqpc is forked (though program may not know that it is calling qmail-qmqpc through symbolic link via qmail-queue)
3. Application Speaks to Qmail-qmqpc using the
4. qmail-queue protocol qmail-qmqpc reads the message, sender, and recipient list into memory (yes, memory!)
5. qmail-qmqpc read the
6. control/qmqpservers file for the list of IP addresses to try to send the message to qmail-qmqpc Connects to the first responding server in the file
7. qmail-qmqpd is started via
8. tcpserver listening on the remote machine, and accepts the connection qmail-qmqpc sends a netstring containing the message netstring, sender netstring and recipient netstrings.
9. qmail-qmqpd invokes qmail-queue and writes the message into its queue
10. qmail-qmqpd returns the status character to the client
11. qmail-qmqpc accepts and processes the status code returned from the server
12. qmail-qmqpd finishes.
13. qmail-qmqpc finishes.
14. The Application progam resumes it logic.

QMQP exposes some of the more useful operations you can make to qmail in setting up a scalable operations center or email server farm. For starters, I am concerned about the in-memory netstring containing the while message plus its recipient list. The problem is the qmail-queue protocol reads the message before its recipients, and that the entire communication is a sngle (nested) netstring. Given messages with ver large bodies/attachments and/or recipient lists, you may blow out on memory if you are not expecting it. Of course, the only other option is to cache it to a temporary disk file first, and that buys more time and disk activity.

Next, we can create a very simple load balancing servive from a Mini-Qmail setup to a series of qmail-handling machines. Perhaps the solution here is to hack the qmqpservers list to start at a random line in the file, and wrap around to the top of the list back through the starting machine to find the server to deliver the message to. This can spread the load around, given sufficient incoming messages to give a good distribution. I’m sure Bernstein would have a much better approach... and I would to if I had a chance to think harder about it.

The mail server running the QMQPD Daemon service could monitor itself, its queue, etc., and temporarily shut down its listener if it gets too busy. This allows other servers to take the load while it “catches up”.

Domain mapping can be utilized here as well to determine which machine to deliver to. Maybe you can read a file listing domains and servers to connect to. Yes, its a little extra overhead, but it may be worth it.

In the EZMLM mail list manager for qmail, ezmlm-send has a special patch(?) for qmqpc that allows it to send the designated destination QMQP server as a server name. Have to double check this though!