The Developer’s Guide to IMAP
Common pitfalls developers should avoid when using IMAP.
Tasia Potasinski | May 30, 2019
So you want to build an IMAP integration? Easy!
from imaplib import IMAP4_SSL with IMAP4_SSL(host) as session: session.login(login, password) session.select('INBOX') status, data = session.search(None, 'ALL') for sequence in data.split(): status, body = session.fetch(sequence, '(BODY[TEXT])') process(body)
Looks good, doesn’t it? Or have we celebrated too soon? While a library like this one might give the impression that integrating with IMAP is as simple as login, fetch, log out, underneath the surface IMAP contains nuances that can cause any developer massive headaches. If your mission is to build a reliable integration, it behooves you to learn about what lies underneath.
In this blog, we’ll uncover:
- IMAP: The Protocols, Commands, and Scopes
- What to Look Out for When Integrating with IMAP
- A Better Way to Integrate With IMAP
I. IMAP: Protocols, Data Models, and Commands
The IMAP model assumes a centralized server and a multitude of transient, possibly low-bandwidth clients. These clients may examine what messages are on the server without downloading them in their entirety; therefore they are under no obligation to delete the messages after fetching them, as is the custom with POP.
Therefore, a client does not need to have any storage at all (though in practice most at least maintain a cache)! Though this architecture seems unremarkable today, in the late 1980s the concept of using one’s mailbox as a central database rather than a queue was revolutionary.
The IMAP Protocol: The IMAP protocol itself is line-oriented and typically occurs over a long-running session per user-agent (there is no asynchronous do-this-and-wake-me-when-it’s-done functionality). A client will connect, authenticate, and select a particular mailbox to issue commands against.
Authentication can occur in two ways: with the `LOGIN` command (simple password) or `AUTHENTICATE` (any SASL mechanism e.g. OAuth, Kerberos…). This session based approach has significant implications for the IMAP server’s performance.
IMAP Data Models: Messages, annotated with several attributes, are organized under a hierarchy of mailboxes `(inbox/finance/receipts`). The attributes include metadata like date, size, and optional flags, but also two identifiers: the sequence number from 1…n in the mailbox and a UID. Either can be used to refer to the message, and both are defined only within the space of that particular mailbox (more on this below).
The UID is particularly unstable, and can actually be reset *during the IMAP session*. The unique identifier of a message MUST NOT change during the session, and SHOULD NOT change between sessions.
That’s one of the promises that IMAP makes—when you open a session on a specific folder, your UIDs will remain the same for the duration that that session is held open. But if you open a new session, all bets are off—you have to check the UIDVALIDITY value returned from the command (SELECT) that opens the session, and if it’s higher than it was previously the UIDs may all be completely different.
The structure of the message envelopes themselves is formally defined outside of IMAP. But the spec does call for an attempt to parse out the headers/body of the message so that they may be queried.
Commands: The bulk of the IMAP RFC is devoted to commands. They can broadly be divided into two categories: connection management (e.g. `STARTTLS`, `CLOSE`) and CRUD functionality (e.g. `FETCH`, `STORE`, `SEARCH`). Modern extensions have been bolted on to the spec to enable things like delta updates and push notifications, but specific implementations often leave much to be desired. A given server’s support for these is revealed in the `CAPABILITY` command, which is typically announced by the server after authentication.
Out of Scope: It is also important to understand what IMAP doesn’t define:
- Any particular lifecycle for messages (While flags like `/Deleted` do have particular semantics for individual commands, it is largely up to the client to define his or her workflow)
- Labels, categories, colors, or tags. These are extensions implemented by individual providers.
- Global UIDs
- Any interpretation of what the message body is (this is left to MIME)
- Any facility to send the messages (Sent/SMTP problem)
II. What to Look Out for When Integrating with IMAP
1. Volatile UIDs
With IMAP, a message’s UID is useless for most practical purposes. It is not unique across folders. It will change when moved to another folder. It can change when a folder is renamed (depending on the configuration of your server). It can change between sessions. It can change when the server decides it should change (reindexing, etc). This instability is why some providers like Gmail provide [their own global ID scheme] (https://developers.google.com/gmail/imap/imap-extensions#access_to_the_gmail_unique_message_id_x-gm-msgid) as an extension.
But outside of Gmail, how can a message be uniquely identified? IMAP does reveal a `UIDVALIDITY` number which can be queried per mailbox. This mechanism, acting like a timestamp, provides a number to associate alongside the UID. A client can query the `UIDVALIDITY` and detect a change, indicating that it should reindex its cache of that folder. Thus a message can be uniquely identified with the tuple of (`UID`, `UIDVALIDITY`, `folder_name`).
`UIDVALIDITY` does not address all of the mentioned shortcomings of a message’s UID – particularly it still offers no way to track a message across folders. A scheme to uniquely index messages is left as an exercise to the reader.
2. Storing Deleted and Purging Expunged Messages
When a client “deletes” a message, they are typically performing a soft delete – marking the message with the `DELETED` flag which the client might show in a special Trash folder. This operation is actually performed with the `STORE` command. A client can furthermore `EXPUNGE` the mailbox which actually deletes the messages from the client’s view – whether or not the server holds onto a copy is left as an implementation detail. Be warned – the innocuous sounding` CLOSE` command also `EXPUNGE`s by default.
**A special Trash/Deleted folder is not part of the IMAP spec. Servers implement this differently – for some it is a virtual folder which simply shows all messages marked as `DELETED`. Others require messages to be explicitly moved there as part of message deletion.
***The name of the Trash/Deleted folder is also not standard. Clients must query for this – Gmail, for instance, localizes the name of the folder and provides a mapping with an extension to the LIST command.
How do you know when a message has arrived? For the simplest use cases a poller will work, assuming that neither latency nor performance overhead is particularly important. When the integration needs to scale, however, IMAP provides a few extensions: `IDLE` and `NOTIFY`. `IDLE` keeps a connection open so that the server can drip feed events to the client; this approach only supports a single folder, however. `NOTIFY` is an evolution that allows the client to define what mailboxes and events it wants to subscribe to so that only a single connection is needed.
When a client reconnects to a server, how does it know what has changed? Does it need to compare its cache with every message in every folder? Luckily, IMAP provides two extensions to enable delta updates: `CONDSTORE` and `QRESYNC`. Both extensions introduce a modification sequence (similar to the `UIDVALIDITY` message) which is associated with each message. This has the additional use case of preventing multiple clients from clobbering each other’s changes when manipulating the same message (or mailbox).
One caveat to extensions: If you want your IMAP integration to support all IMAP implementations that you’ll run into in the wild you unfortunately can’t assume that these extensions exist—most servers support them, but some still don’t.
A Better Way to Integrate with IMAP
Integrating IMAP into your app can take weeks or months of time just for the initial build, and the ongoing maintenance and support costs scale as you add more users. The Nylas API provides an easier way to integrate with IMAP that’s secure and low-maintenance.
With Nylas, you get all the features of integrating directly with IMAP and every other email service provider in the world in as little as four lines of code. Nylas comes out-of-the-box with enterprise-grade security features, industry-leading SLAs and quick-start SDKs to make integrating as simple as possible.