Low Level Electron Debugging

Looking under the hood


We’re big fans of Electron here at Nylas. It’s allowed us to iterate quickly across platforms using the best of modern web standards. Since Electron is built on top of Chromium we get some great debugging tooling from Chrome Developer Tools. Usually these are enough for our purposes, but when bugs get weird, we need to go one level down, and look under the hood.

This post will focus on a recent bug with SQLite and how we utilized LLDB to find the root cause of Nylas Mail mysteriously and intermittently crashing.

Setting up Electron

First you need Electron without symbols stripped. If you run lldb on the prebuilt downloaded Electron, you’ll get hex garbage in your stacks.

Start by building Electron from source. This isn’t as bad as it sounds! Follow the instructions for Mac, Win, or Linux.

On Mac it’s as simple as: script/bootstrap.py -v && script/build.py -c D . Just make sure you’re on the latest OSX and have the latest XCode properly installed.

Once you have Electron built, you can forevermore use your new debug executable to launch your app instead of a precompiled one.

Setting up native Node modules

If you have plain javascript dependencies, you can debug them as normal through the inspector panel. However, if you’re using native modules, sometimes the issue can be deep inside the compiled code of that module.

In our case, we had a strong hunch that the source of our bug was in SQLite. Code executing here doesn’t show up in the inspector console (except as an unhelpful grey bar). Even with our debug version of Electron, we need to make sure that our native node module also has symbols. By default when you npm install native node modules, they’ll strip symbols making low-level debugging almost impossible.

You need to rebuild native modules with debug flags.

Here’s how we did it for SQLite:

$ cd node_modules/sqlite3
$ [vi|emacs|nano] package.json

And change
"install": "node-pre-gyp install --fallback-to-build"
to
"install": "node-pre-gyp install --debug --fallback-to-build"

$ NPM_CONFIG_TARGET=1.4.15 NPM_CONFIG_ARCH_x64=NPM_CONFIG_TARGE_ARCH=x64 NPM_CONFIG_DISTURL=https://atom.io/download/electron NPM_CONFIG_RUNTIME=electron NPM_CONFIG_BUILD_FROM_SOURCE=true npm install

All those environment variables we set before the npm install are necessary to make sure we’re using Electron’s headers. Please read up about Using Native Node Modules if that’s foreign to you.

Now we’re ready to launch the app & have all debug symbols available! 🎉

Start Debugging (Now with more symbols)

Launch your app with the debug version of Electron we previously built:

$ electron/out/D/Electron.app/Contents/MacOS/Electron myElectronAppFolder

Now let’s attach lldb or gdb to your app!

The first trick is finding the correct process ID to attach to. Your app will likely have two or more processes. In our case, we knew our bug was coming from a particular process because it would blow the memory sky high. 😬

Activity Viewer

Now start lldb:

$ lldb -p 5600

Once attached, it’ll bring your process to a screeching halt. At this point you can look at the backtrace, explore stack frames, and much more. Read the full LLDB documentation to find out everything you can do.

Catching Trouble

The trick was to have lldb stop at just the right time when our bug happened. For intermittent bugs this is frequently difficult. While you can use a combination of chrome inspector breakpoints and lldb breakpoints, our bug had an (un)fortunate property of causing the whole app to mysteriously crash when it reared its head. When this happened, we attached lldb.

Now that we’re in lldb, attached to the right process, and stopped in the middle of our mysterious crash, let’s look around.

$ (lldb) thread backtrace all

Since we rebuilt SQLite with debug flags, we now get obviously sqlite-related stacktraces in some of our threads and frames. Let’s dive into those further:

(lldb) thread backtrace
* thread #1: tid = 0xbc0d2e,
...
frame #11: 0x0000000116334d25 node_sqlite3.node`Nan::imp::Factory<v8::String>::New(value="<!DOCTYPE html><html><head><title></title><meta charset=\"UTF-8\"><meta content=\"IE=edge,chrome=1\" http-equiv=\"X-UA-Compatible\"><meta content=\"telephone=no\" name=\"format-detection\"><style type=\"text/css\">.globalTable th{padding:0px}</style><style type=\"text/css\">@media only screen and (min-width:481px) {.wild_full{width:100%;}}</style><link href='https://d11civ2ku1dhdc.cloudfront.net/img/bz4cdvckpa/8bu2ska6n3/fonts.css' rel='stylesheet' type='text/css'><style type=\"text/css\">.holderMax{width:100% !important;max-width:589px !important;}@media only screen and (max-width:480px) {.smallFull{display:block;width:100%}.out_hide{display:none!important;}.cellFull{max-width:100%!important;display:block!important;width:100%!important;}.noMoreHeight{min-height:none!important;height:auto!important;}.blockClass{display:block!important;width:auto!important;}.blockClassPadding{display:block!important;width:100%!important;}.noWidthMax{max-width:100%!important;}.gallery_2{max-width:100%!important;display:inline-block!important;w"..., length=90017) + 37 at nan_implementation_12_inl.h:269
    frame #12: 0x0000000116348a31 node_sqlite3.node`Nan::imp::Factory<v8::String>::return_t Nan::New<v8::String, char const*, unsigned long>(arg0="<!DOCTYPE html><html><head><title></title><meta charset=\"UTF-8\"><meta content=\"IE=edge,chrome=1\" http-equiv=\"X-UA-Compatible\"><meta content=\"telephone=no\" name=\"format-detection\"><style type=\"text/css\">.globalTable th{padding:0px}</style><style type=\"text/css\">@media only screen and (min-width:481px) {.wild_full{width:100%;}}</style><link href='https://d11civ2ku1dhdc.cloudfront.net/img/bz4cdvckpa/8bu2ska6n3/fonts.css' rel='stylesheet' type='text/css'><style type=\"text/css\">.holderMax{width:100% !important;max-width:589px !important;}@media only screen and (max-width:480px) {.smallFull{display:block;width:100%}.out_hide{display:none!important;}.cellFull{max-width:100%!important;display:block!important;width:100%!important;}.noMoreHeight{min-height:none!important;height:auto!important;}.blockClass{display:block!important;width:auto!important;}.blockClassPadding{display:block!important;width:100%!important;}.noWidthMax{max-width:100%!important;}.gallery_2{max-width:100%!important;display:inline-block!important;w"..., arg1=90017) + 33 at nan_new.h:214
...

Next we pick the thread and frame that has sqlite in it:

$ (lldb) thread select 1
..
$ (lldb) thread backtrace
..
$ (lldb) frame select 14
frame #14: 0x00000001163460df node_sqlite3.node`node_sqlite3::Statement::Work_AfterAll(req=0x00007fcf5416ac78) + 399 at statement.cc:551
   548                         Rows::const_iterator it = baton->rows.begin();
   549                         Rows::const_iterator end = baton->rows.end();
   550                         for (int i = 0; it < end; ++it, i++) {
-> 551                             Nan::Set(result, i, RowToJS(*it));
   552                             delete *it;
   553                         }
   554
$ (lldb)  l
   555                         Local<Value> argv[] = { Nan::Null(), result };
   556                         TRY_CATCH_CALL(stmt->handle(), cb, 2, argv);
   557                     }
   558                     else {
   559                         // There were no result rows.
   560                         Local<Value> argv[] = {
   561                             Nan::Null(),

We zero in on: node_sqlite3::Statement::Work_AfterAll . By taking a quick look through the sqlite source code, that function stood out as one that likely has frame variables that can tell us what we want.

Finally, we use the fact that lldb is fully interactive and use existing sqlite functions to print out the value of suspicious variables. In our case we wanted to know what query was running when the app hung.

$ (lldb) print sqlite3_sql(stmt->_handle)
(const char *) $0 = 0x00007fcf95498720 "SELECT * FROM `messages`;"

AH HA! That’ll do it… Selecting several GB of message data at once will hang sqlite and crash the app when it runs out of memory. Some piece of our code unexpectedly, and intermittently, queried sqlite with invalid limits.

Final Thoughts

LLDB, chrome developer tools, and sound development practices are all parts of our toolkit. Each one serves a slightly different purpose and we’re always learning something new about how to improve our debugging skills and the improving the quality of Nylas Mail. Going forward we have a couple of ideas on where to take this next.

How to connect Xcode Instruments

Instruments is a very powerful debugging tool. It’s possible to connect this same electron stack and get timelines of memory allocation, disk access, and much more.

Could we build an Electron debugger?

All of this setup could be automated and generalized for any Electron app. Building an Electron debugger wouldn’t be too far off. Imagine having a tool like Instruments but specifically built for Electron Apps. We think that would be amazing.

Thanks to Mark Hahnenberg for contributing the lldb expertise for this work!

Written by Tomasz Finc

Subscribe to Engineering Blog Updates

Start Developing Today

Connect up to 10 accounts (email, calendar, and contacts) for free today.

Get Started