Our website uses cookies to improve your experience. You can learn more about how we use cookies in our Privacy Policy.

Low Level Electron Debugging

Looking under the hood.

The Nylas Team | March 23, 2017

We’re big fans of Electron here at Nylas. It’s allowed us to iterate quickly across platforms using the best of modern web standards. Since Electron is built on top of Chromium we get some great debugging tooling from Chrome Developer Tools. Usually these are enough for our purposes, but when bugs get weird, we need to go one level down, and look under the hood.

This post will focus on a recent bug with SQLite and how we utilized LLDB to find the root cause of Nylas Mail mysteriously and intermittently crashing.

Setting up Electron

First you need Electron without symbols stripped. If you run lldb on the prebuilt downloaded Electron, you’ll get hex garbage in your stacks.

Start by building Electron from source. This isn’t as bad as it sounds! Follow the instructions for Mac, Win, or Linux.

On Mac it’s as simple as:

script/bootstrap.py -v && script/build.py -c D

Just make sure you’re on the latest OSX and have the latest XCode properly installed.

Once you have Electron built, you can forevermore use your new debug executable to launch your app instead of a precompiled one.

Setting up native Node modules

If you have plain javascript dependencies, you can debug them as normal through the inspector panel. However, if you’re using native modules, sometimes the issue can be deep inside the compiled code of that module.

In our case, we had a strong hunch that the source of our bug was in SQLite. Code executing here doesn’t show up in the inspector console (except as an unhelpful grey bar). Even with our debug version of Electron, we need to make sure that our native node module also has symbols. By default when you npm install native node modules, they’ll strip symbols making low-level debugging almost impossible.

You need to rebuild native modules with debug flags.

Here’s how we did it for SQLite:

$ cd node_modules/sqlite3 
$ [vi|emacs|nano] package.json

And change
"install": "node-pre-gyp install --fallback-to-build"
to
"install": "node-pre-gyp install --debug --fallback-to-build"

$ NPM_CONFIG_TARGET=1.4.15 NPM_CONFIG_ARCH_x64=NPM_CONFIG_TARGE_ARCH=x64 NPM_CONFIG_DISTURL=https://atom.io/download/electron NPM_CONFIG_RUNTIME=electron NPM_CONFIG_BUILD_FROM_SOURCE=true npm install

All those environment variables we set before the npm install are necessary to make sure we’re using Electron’s headers. Please read up about Using Native Node Modules if that’s foreign to you.

Now we’re ready to launch the app & have all debug symbols available! 🎉

Start debugging (now with more symbols)

Launch your app with the debug version of Electron we previously built:

$ electron/out/D/Electron.app/Contents/MacOS/Electron myElectronAppFolder

Now let’s attach lldb or gdb to your app!

The first trick is finding the correct process ID to attach to. Your app will likely have two or more processes. In our case, we knew our bug was coming from a particular process because it would blow the memory sky high. 😬

Activity Viewer

Now start lldb:

$ lldb -p 5600

Once attached, it’ll bring your process to a screeching halt. At this point you can look at the backtrace, explore stack frames, and much more. Read the full LLDB documentation to find out everything you can do.

Catching trouble

The trick was to have lldb stop at just the right time when our bug happened. For intermittent bugs this is frequently difficult. While you can use a combination of chrome inspector breakpoints and lldb breakpoints, our bug had an (un)fortunate property of causing the whole app to mysteriously crash when it reared its head. When this happened, we attached lldb.

Now that we’re in lldb, attached to the right process, and stopped in the middle of our mysterious crash, let’s look around.

$ (lldb) thread backtrace all

Since we rebuilt SQLite with debug flags, we now get obviously sqlite-related stacktraces in some of our threads and frames. Let’s dive into those further:

(lldb) thread backtrace
* thread #1: tid = 0xbc0d2e,
...
frame #11: 0x0000000116334d25 node_sqlite3.node`Nan::imp::Factory<v8::String>::New(value="<!DOCTYPE html><html><head><title></title><meta charset=\"UTF-8\"><meta content=\"IE=edge,chrome=1\" http-equiv=\"X-UA-Compatible\"><meta content=\"telephone=no\" name=\"format-detection\"><style type=\"text/css\">.globalTable th{padding:0px}</style><style type=\"text/css\">@media only screen and (min-width:481px) {.wild_full{width:100%;}}</style><link href='https://d11civ2ku1dhdc.cloudfront.net/img/bz4cdvckpa/8bu2ska6n3/fonts.css' rel='stylesheet' type='text/css'><style type=\"text/css\">.holderMax{width:100% !important;max-width:589px !important;}@media only screen and (max-width:480px) {.smallFull{display:block;width:100%}.out_hide{display:none!important;}.cellFull{max-width:100%!important;display:block!important;width:100%!important;}.noMoreHeight{min-height:none!important;height:auto!important;}.blockClass{display:block!important;width:auto!important;}.blockClassPadding{display:block!important;width:100%!important;}.noWidthMax{max-width:100%!important;}.gallery_2{max-width:100%!important;display:inline-block!important;w"..., length=90017) + 37 at nan_implementation_12_inl.h:269
 frame #12: 0x0000000116348a31 node_sqlite3.node`Nan::imp::Factory<v8::String>::return_t Nan::New<v8::String, char const*, unsigned long>(arg0="<!DOCTYPE html><html><head><title></title><meta charset=\"UTF-8\"><meta content=\"IE=edge,chrome=1\" http-equiv=\"X-UA-Compatible\"><meta content=\"telephone=no\" name=\"format-detection\"><style type=\"text/css\">.globalTable th{padding:0px}</style><style type=\"text/css\">@media only screen and (min-width:481px) {.wild_full{width:100%;}}</style><link href='https://d11civ2ku1dhdc.cloudfront.net/img/bz4cdvckpa/8bu2ska6n3/fonts.css' rel='stylesheet' type='text/css'><style type=\"text/css\">.holderMax{width:100% !important;max-width:589px !important;}@media only screen and (max-width:480px) {.smallFull{display:block;width:100%}.out_hide{display:none!important;}.cellFull{max-width:100%!important;display:block!important;width:100%!important;}.noMoreHeight{min-height:none!important;height:auto!important;}.blockClass{display:block!important;width:auto!important;}.blockClassPadding{display:block!important;width:100%!important;}.noWidthMax{max-width:100%!important;}.gallery_2{max-width:100%!important;display:inline-block!important;w"..., arg1=90017) + 33 at nan_new.h:214
...

Next we pick the thread and frame that has sqlite in it:

$ (lldb) thread select 1
..
$ (lldb) thread backtrace
..
$ (lldb) frame select 14
frame #14: 0x00000001163460df node_sqlite3.node`node_sqlite3::Statement::Work_AfterAll(req=0x00007fcf5416ac78) + 399 at statement.cc:551
    548     Rows::const_iterator it = baton->rows.begin();
    549     Rows::const_iterator end = baton->rows.end();
    550      for (int i = 0; it < end; ++it, i++) {
->  551        Nan::Set(result, i, RowToJS(*it));
    552       delete *it;
    553     }
    554
$ (lldb) l
    555     Local<Value> argv[] = { Nan::Null(), result };
    556     TRY_CATCH_CALL(stmt->handle(), cb, 2, argv);
    557   }
    558   else {
    559     // There were no result rows.
    560     Local<Value> argv[] = {
    561       Nan::Null(),

We zero in on: node_sqlite3::Statement::Work_AfterAll . By taking a quick look through the sqlite source code, that function stood out as one that likely has frame variables that can tell us what we want.

Finally, we use the fact that lldb is fully interactive and use existing sqlite functions to print out the value of suspicious variables. In our case we wanted to know what query was running when the app hung.

$ (lldb) print sqlite3_sql(stmt->_handle)
(const char *) $0 = 0x00007fcf95498720 "SELECT * FROM `messages`

AH HA! That’ll do it… Selecting several GB of message data at once will hang sqlite and crash the app when it runs out of memory. Some piece of our code unexpectedly, and intermittently, queried sqlite with invalid limits.

Final Though

LLDB, chrome developer tools, and sound development practices are all parts of our toolkit. Each one serves a slightly different purpose and we’re always learning something new about how to improve our debugging skills and the improving the quality of Nylas Mail. Going forward we have a couple of ideas on where to take this next.

How to connect Xcode Instrumen

Instruments is a very powerful debugging tool. It’s possible to connect this sameelectron stack and get timelines of memory allocation, disk access, and much more.

Could we build an Electron debugger?

All of this setup could be automated and generalized for any Electron app. Building an Electron debugger wouldn’t be too far off. Imagine having a tool like Instruments but specifically built for Electron Apps. We think that would be amazing.

Thanks to Mark Hahnenberg for contributing the lldb expertise for this work!

About the Author

Nylas was founded in 2013 with a mission to power the communications layer of the modern technology stack. Our universal APIs for email, calendar, and contacts are used by tens of thousands of developers across more than 25 countries.

Ready to Start Building?

Connect up to 10 accounts for free today. No credit card required.