Calendar management
Embedded contextual email
Scheduling automation
Automated outreach
Telehealth
Marketplaces
Recruiting
Document management
Real estate
Communications software
Docs
SDKs
Tutorials
API status
Changelog
Support
Blog
Events
Guides
Newsroom
Partners
Build vs. buy
Customer stories
Trust & security
Why NylasThe Developer’s Guide to Integrating with IMAP
Common pitfalls developers should avoid when using IMAP.
Common pitfalls developers should avoid when using IMAP.
So you want to build an IMAP integration? Easy!
from imaplib import IMAP4_SSL
with IMAP4_SSL(host) as session:
session.login(login, password)
session.select('INBOX')
status, data = session.search(None, 'ALL')
for sequence in data[0].split():
status, body = session.fetch(sequence, '(BODY[TEXT])')
process(body)
Looks good, doesn’t it? Or have we celebrated too soon? While a library like this one might give the impression that integrating with IMAP is as simple as login, fetch, log out, underneath the surface IMAP contains nuances that can cause any developer massive headaches. If your mission is to build a reliable integration, it behooves you to learn about what lies underneath.
In this blog, we’ll uncover:
The IMAP model assumes a centralized server and a multitude of transient, possibly low-bandwidth clients. These clients may examine what messages are on the server without downloading them in their entirety; therefore they are under no obligation to delete the messages after fetching them, as is the custom with POP.
Therefore, a client does not need to have any storage at all (though in practice most at least maintain a cache)! Though this architecture seems unremarkable today, in the late 1980s the concept of using one’s mailbox as a central database rather than a queue was revolutionary.
The IMAP Protocol: The IMAP protocol itself is line-oriented and typically occurs over a long-running session per user-agent (there is no asynchronous do-this-and-wake-me-when-it’s-done functionality). A client will connect, authenticate, and select a particular mailbox to issue commands against.
Authentication can occur in two ways: with the LOGIN command (simple password) or AUTHENTICATE (any SASL mechanism e.g. OAuth, Kerberos…). This session based approach has significant implications for the IMAP server’s performance.
IMAP Data Models: Messages, annotated with several attributes, are organized under a hierarchy of mailboxes `(inbox/finance/receipts`). The attributes include metadata like date, size, and optional flags, but also two identifiers: the sequence number from 1…n in the mailbox and a UID. Either can be used to refer to the message, and both are defined only within the space of that particular mailbox (more on this below).
The UID is particularly unstable, and can actually be reset *during the IMAP session*. The unique identifier of a message MUST NOT change during the session, and SHOULD NOT change between sessions.
That’s one of the promises that IMAP makes—when you open a session on a specific folder, your UIDs will remain the same for the duration that that session is held open. But if you open a new session, all bets are off—you have to check the UIDVALIDITY value returned from the command (SELECT) that opens the session, and if it’s higher than it was previously the UIDs may all be completely different.
The structure of the message envelopes themselves is formally defined outside of IMAP. But the spec does call for an attempt to parse out the headers/body of the message so that they may be queried.
Commands: The bulk of the IMAP RFC is devoted to commands. They can broadly be divided into two categories: connection management (e.g. STARTTLS, CLOSE) and CRUD functionality (e.g. FETCH, STORE, SEARCH). Modern extensions have been bolted on to the spec to enable things like delta updates and push notifications, but specific implementations often leave much to be desired. A given server’s support for these is revealed in the CAPABILITY command, which is typically announced by the server after authentication.
Out of Scope: It is also important to understand what IMAP doesn’t define:
1. Volatile UIDs
With IMAP, a message’s UID is useless for most practical purposes. It is not unique across folders. It will change when moved to another folder. It can change when a folder is renamed (depending on the configuration of your server). It can change between sessions. It can change when the server decides it should change (reindexing, etc). This instability is why some providers like Gmail provide [their own global ID scheme] (https://developers.google.com/gmail/imap/imap-extensions#access_to_the_gmail_unique_message_id_x-gm-msgid) as an extension.
But outside of Gmail, how can a message be uniquely identified? IMAP does reveal a UIDVALIDITY number which can be queried per mailbox. This mechanism, acting like a timestamp, provides a number to associate alongside the UID. A client can query the UIDVALIDITY and detect a change, indicating that it should reindex its cache of that folder. Thus a message can be uniquely identified with the tuple of (UID, UIDVALIDITY, folder_name).
UIDVALIDITY does not address all of the mentioned shortcomings of a message’s UID – particularly it still offers no way to track a message across folders. A scheme to uniquely index messages is left as an exercise to the reader.
2. Storing Deleted and Purging Expunged Messages
When a client “deletes” a message, they are typically performing a soft delete – marking the message with the DELETED flag which the client might show in a special Trash folder. This operation is actually performed with the STORE command. A client can furthermore EXPUNGE the mailbox which actually deletes the messages from the client’s view – whether or not the server holds onto a copy is left as an implementation detail. Be warned – the innocuous sounding CLOSE command also EXPUNGEs by default.
**A special Trash/Deleted folder is not part of the IMAP spec. Servers implement this differently – for some it is a virtual folder which simply shows all messages marked as DELETED. Others require messages to be explicitly moved there as part of message deletion.
***The name of the Trash/Deleted folder is also not standard. Clients must query for this – Gmail, for instance, localizes the name of the folder and provides a mapping with an extension to the LIST command.
3. Notifications
How do you know when a message has arrived? For the simplest use cases a poller will work, assuming that neither latency nor performance overhead is particularly important. When the integration needs to scale, however, IMAP provides a few extensions: IDLE and NOTIFY. IDLE` keeps a connection open so that the server can drip feed events to the client; this approach only supports a single folder, however. NOTIFY is an evolution that allows the client to define what mailboxes and events it wants to subscribe to so that only a single connection is needed.
When a client reconnects to a server, how does it know what has changed? Does it need to compare its cache with every message in every folder? Luckily, IMAP provides two extensions to enable delta updates: CONDSTORE and QRESYNC. Both extensions introduce a modification sequence (similar to the UIDVALIDITY message) which is associated with each message. This has the additional use case of preventing multiple clients from clobbering each other’s changes when manipulating the same message (or mailbox).
One caveat to extensions: If you want your IMAP integration to support all IMAP implementations that you’ll run into in the wild you unfortunately can’t assume that these extensions exist—most servers support them, but some still don’t.
Integrating IMAP into your app can take weeks or months of time just for the initial build, and the ongoing maintenance and support costs scale as you add more users. The Nylas API provides an easier way to integrate with IMAP that’s secure and low-maintenance.
With Nylas, you get all the features of integrating directly with IMAP and every other email service provider in the world in as little as four lines of code. Nylas comes out-of-the-box with enterprise-grade security features, industry-leading SLAs and quick-start SDKs to make integrating as simple as possible.
Start building for free, or learn more from one of our platform specialists.
Please enter your email address and receive the latest updates.
