Minidumps: Giving Weight to Your Electron Bug Reports

Sentry introduced (welcomed) support for minidump crash reports earlier this year. In this post, Tim Fish, an essential contributor and co-maintainer of Sentry’s Electron SDK, explores his own experience with the intersection of minidumps and Electron.

A couple of years ago I landed my first large contract as a freelance developer replacing a much loved, but neglected, Windows data capture application with a modern cross-platform alternative. It would be entirely my responsibility to replace the app and resolve any issues — so I was keen to have as much information available to me as possible.

I had also decided that an Electron app was the best solution, and as it was unlike anything I’d worked on before, proper error reporting was doubly important.

Crashes of Unknown Origins

After months of work, beta builds were released to my customer, and I started getting reports of some native crashes. In the renderer process of an Electron app, these crashes manifest themselves as a completely blank window — not something that would go unnoticed! I wasn’t sure whether these crashes were caused by Electron, the customer-supplied native drivers, or my (possibly) naive calls into them via node-ffi.

My initial stopgap solution was to at least ensure that the app could recover. Specifically, when the renderer process crashed, I forced it to restart, and because the JavaScript state is persisted, the process continued from where it left off — with minimal data loss. Unfortunately, one of the downsides to this approach was that I immediately stopped receiving reports of native crashes from beta testers.

Much of my previous native crash resolution was based around reproducing an issue locally with a debugger attached. In this case, I didn’t have the time or resources to attempt to replicate these issues, especially on platforms I wasn’t hugely familiar with. I needed to know if the crashes were my responsibility or if I should report them upstream to my customer, and ASAP.

Minidumps to the Rescue

As it turns out, minidumps were the answer. Conceived initially by Microsoft for crash reports on Windows, minidumps have become an accessible format for all sorts of native crashes. Google has since developed a popular library called Breakpad and its successor Crashpad, which is able to generate minidumps on a wide variety of operating systems and CPU architectures. Fortunately, Electron has already done the work of integrating it and making it available with only a few lines of JavaScript.

const { crashReporter } = require('electron')

crashReporter.start({
  productName: 'YourName',
  companyName: 'YourCompany',
  submitURL: 'https://your-domain.com/url-to-submit',
  uploadToServer: true
})

Why did Google choose a Windows crash reporting format for Chrome? The minidump file format specifies a highly generic way to carry parts of a crashing processes memory. Since such processes can get quite large — gigabytes, in fact — this is reduced to the essentials only: CPU register values, the call stack of all threads, and some select custom memory regions.

The minidump format really does live up to its name. Electron's crash reports are usually just around a few hundred kilobytes. Yet, they contain full stack traces comprising drivers, custom native extensions, and Electron's source itself. Given debug information, crash reporting tools can reconstruct debugger-like information from minidumps and provide advanced error analysis on top. In fact, debuggers use the same mechanism to display stack traces, called "stack walking."

Using minidumps in Electron is straightforward: the built-in crash reporter automatically creates minidumps when a renderer or even the main process crashes. The Sentry SDK then adds useful context information and submits it to Sentry, where it shows up seconds later in the issues stream. And thanks to debug information, I can even see the original source lines in inlined stack frames (which are optimized out by the compiler).

So where were my native crashes coming from? Some were originating in Electron, but the vast majority were from the drivers I’d been supplied. This was to be expected as the company had only recently ported them to Mac and Linux. Mission accomplished?

Minidumps Give Bug Reports Context

I wasn’t off the hook yet. What if I was passing garbage parameters? What if my confidence was misplaced and calling native code from JavaScript was a terrible idea? Reports of “your driver is broken when I call it from JavaScript” would be met with skepticism. My bug reports would only carry weight if they came with minimal C reproductions.

At this point, having context and breadcrumbs with the minidumps became super helpful. I cobbled together an ES6 Proxy, which intercepts every node-ffi native call and prints it to the console with all the parameters and this ends up in the breadcrumbs. (Note: this has terrible performance and fills the breadcrumbs, so it’s only enabled via command line argument.)

function traceMethodCalls(obj) {
    const handler = {
        get(target, propKey, receiver) {
            const targetValue = Reflect.get(target, propKey, receiver);
            if (typeof targetValue === 'function') {
                return function (...args) {
                    console.log('Call', propKey, args);
                    return targetValue.apply(this, args); // (A)
                }
            } else {
                return targetValue;
            }
        }
    };
    return new Proxy(obj, handler);    
}

Now my native crash reports could include a list of all the native calls preceding it, and I had everything I needed to report to my customer. Not only could I supply stack traces where debug symbols were available, but in many cases, I could also provide C code to reproduce issues.

If you haven’t already (or even if you have), check out Sentry’s Electron and minidump documentation. Use it. Break things. Repair them. Break them some more. Repair them again. Break them one more time. Repair — you get the idea.

Don’t hesitate to let us know what you think about using Electron with Sentry (or about anything else Sentry related). Post feedback in our forum, bugs in our issue tracker, or shout out to our support engineers for help.