Low Level Electron Debugging - Nylas

Low Level Electron Debugging

Looking under the hood.

The Nylas Team | March 23, 2017

We’re big fans of Electron here at Nylas. It’s allowed us to iterate quickly across platforms using the best of modern web standards. Since Electron is built on top of Chromium we get some great debugging tooling from Chrome Developer Tools. Usually these are enough for our purposes, but when bugs get weird, we need to go one level down, and look under the hood.

This post will focus on a recent bug with SQLite and how we utilized LLDB to find the root cause of Nylas Mail mysteriously and intermittently crashing.

Setting up Electron

First you need Electron without symbols stripped. If you run lldb on the prebuilt downloaded Electron, you’ll get hex garbage in your stacks.

Start by building Electron from source. This isn’t as bad as it sounds! Follow the instructions for Mac, Win, or Linux.

On Mac it’s as simple as:

script/bootstrap.py -v && script/build.py -c D

Just make sure you’re on the latest OSX and have the latest XCode properly installed.

Once you have Electron built, you can forevermore use your new debug executable to launch your app instead of a precompiled one.

Setting up native Node modules

If you have plain javascript dependencies, you can debug them as normal through the inspector panel. However, if you’re using native modules, sometimes the issue can be deep inside the compiled code of that module.

In our case, we had a strong hunch that the source of our bug was in SQLite. Code executing here doesn’t show up in the inspector console (except as an unhelpful grey bar). Even with our debug version of Electron, we need to make sure that our native node module also has symbols. By default when you npm install native node modules, they’ll strip symbols making low-level debugging almost impossible.

You need to rebuild native modules with debug flags.

Here’s how we did it for SQLite:

$ cd node_modules/sqlite3 
$ [vi|emacs|nano] package.json

And change
"install": "node-pre-gyp install --fallback-to-build"
to
"install": "node-pre-gyp install --debug --fallback-to-build"

$ NPM_CONFIG_TARGET=1.4.15 NPM_CONFIG_ARCH_x64=NPM_CONFIG_TARGE_ARCH=x64 NPM_CONFIG_DISTURL=https://atom.io/download/electron NPM_CONFIG_RUNTIME=electron NPM_CONFIG_BUILD_FROM_SOURCE=true npm install

All those environment variables we set before the npm install are necessary to make sure we’re using Electron’s headers. Please read up about Using Native Node Modules if that’s foreign to you.

Now we’re ready to launch the app & have all debug symbols available! 🎉

Start debugging (now with more symbols)

Launch your app with the debug version of Electron we previously built:

$ electron/out/D/Electron.app/Contents/MacOS/Electron myElectronAppFolder

Now let’s attach lldb or gdb to your app!

The first trick is finding the correct process ID to attach to. Your app will likely have two or more processes. In our case, we knew our bug was coming from a particular process because it would blow the memory sky high. 😬

Activity Viewer

Now start lldb:

$ lldb -p 5600

Once attached, it’ll bring your process to a screeching halt. At this point you can look at the backtrace, explore stack frames, and much more. Read the full LLDB documentation to find out everything you can do.

Catching trouble

The trick was to have lldb stop at just the right time when our bug happened. For intermittent bugs this is frequently difficult. While you can use a combination of chrome inspector breakpoints and lldb breakpoints, our bug had an (un)fortunate property of causing the whole app to mysteriously crash when it reared its head. When this happened, we attached lldb.

Now that we’re in lldb, attached to the right process, and stopped in the middle of our mysterious crash, let’s look around.

$ (lldb) thread backtrace all

Since we rebuilt SQLite with debug flags, we now get obviously sqlite-related stacktraces in some of our threads and frames. Let’s dive into those further:

(lldb) thread backtrace
* thread #1: tid = 0xbc0d2e,
...
frame #11: 0x0000000116334d25 node_sqlite3.node`Nan::imp::Factory::New(value="


 


 

About the Author

Nylas was founded in 2013 with a mission to power the communications layer of the modern technology stack. Our universal APIs for email, calendar, and contacts are used by tens of thousands of developers across more than 25 countries.

Ready to Start Building?

Connect up to 10 accounts for free today. No credit card required.