Real-time ETL for Email and Communications Data with Nylas Streams

Discover how Nylas Streams provides the missing link between unstructured communications data and the rest of your workflow. Extract, transform and load email data into cloud databases like Google BigQuery and AWS SNS—all in real time, with little to no code.

Ash Ryan Arnwine | January 26, 2022

As product managers, application developers, and data analysts, we’re all looking for ways to better serve our customers and run more effective organizations through personalized experiences, data-driven business intelligence, and real-time delivery of… well, all the data to everywhere it’s needed.

The truth is that there’s an ocean of untapped data sitting in our communications tools, like email, but most companies aren’t utilizing it. If we could hook into that resource for our customers and organizations, we could:

Want a PDF of this article?

Share it with a friend or save it for later reading.

  • Turbocharge our e-commerce applications with personalization
  • Improve real-time shipping delivery tracking for our customers
  • Stream sales emails and Zoom transcripts into data warehouses for real-time analytics
  • Get always-current analytics on customer support emails, and much more…

The trick, of course, is how to go about setting this all up.

The challenges of doing it yourself

If you want to take this on yourself, you’ll encounter at least a couple of challenges along the way in what amounts to creating your home-cooked Extract-Transform-Load (or, ETL for short) workflows.

Traditional APIs aren’t designed for real-time ETL

For the first challenge, let’s consider the Extract and Load phases of the ETL workflow. Building it yourself, you’ll need to allocate development time to adding on application features that pull the data from one place and send it to another. A couple of common paths are using REST APIs or Webhooks.

Traditional REST APIs, while amazing for many things, are not ideal for approximating real-time. With REST, you’ll likely end up building an automatic polling mechanism, or a user-triggered pulling feature. Polling and pulling can be wastefully expensive in terms of running costs, while risking a clunky user experience. 

Webhooks and Deltas can offer an improvement over basic REST APIs in terms of speed and convenience, but the burden is still left on the application developer to write the application-side features to handle them.

Either way, whether you go for REST or Webhooks, you’re adding on new features to your application, all while potentially coming up short on the real-time experience you want to deliver to your users.

Transforming data from providers is expensive

For the second challenge, let’s consider massaging the actual data into something useful—the Transform phase of ETL. 

Communications data is semi-structured; what’s available in the data and how it’s structured will vary by provider. The problem is that your data warehouse will expect predictable structures to work with, leading to the need to transform your data.

This transformation of data is usually left as an exercise for the developer or engineer. It’s tedious, vendor-dependent, error-prone, and becomes an eternal maintenance issue.

What if you could offload ETL for communications data altogether, while gaining the benefits of real-time data flow?

Nylas Streams: Real-time ETL for communications data

Nylas Streams is our event streaming offering that lets you do a little configuration to run continuous ETL on email and communications data in real-time. 

An animation showing how Nylas Streams extracts, transforms and loads communications data

Nylas Streams lets application developers and data engineers connect to email communications data from providers like Google and Microsoft. From there, Nylas Streams will transform that communications data into optimal formats and send it to destination cloud databases like Google Big Query, Google Pub/Sub, AWS SNS, and Snowflake—all in real time.

If you’re looking for a simple, code-free way to directly connect your data to your data warehouse and ensure that the data is transformed to an optimal structure along the way, Streams is the way to go. 

Give Nylas Streams a try

Are you ready to connect Gmail data to Google Pub/Sub? Or Microsoft email data to AWS SNS?

Try Nylas Streams now, a simple and powerful way to get real-time communications data, optimally transformed for your needs and streamed to your data warehouse.

Ash Ryan Arnwine

Ash is Director of Developer Relations for Nylas. He loves building, leading, and serving teams who enable and inspire developers to create great things on API platforms. Ash is a JavaScript fan, photography enthusiast 📸, and recovering beatmaker. He is lucky to be the husband of an ikebana floral designer 🌷 and dad of the feistiest 5-year-old girl to ever 🦖 the 🌎.