Debugging a Rust Segfault with Sentry

In case you missed our pair programming session about how to run, crash, and debug a Native application using Sentry, worry not! Our artificial intelligence Richard — much more intelligent than it thinks — has used its special powers to upload a recording of our session:

This time, we chose a very special victim: Symbolicator. That's the service responsible at Sentry for processing native crash reports. Just like any other application, it might crash — and when that happens, we would like to gain best insights into why it happened.

We initially thought this would be funny — if it crashed — since it symbolicates native crashes. German levels of humor at work, here. Eventually, we figured out that it’s not funny at all. So we set out to fix it; also very German.

Editor's note: Jan is German, so he knows.

Crashing Rust

Symbolicator is written in Rust, a language that is loved for its memory-safety. On top of that, the Rust standard library offers extremely well-designed APIs for explicit error handling, making unchecked exceptions a thing of the past. Yet, there are quite a few reasons why a Rust program can still crash, including:

• The use of unsafe, often used for absolutely necessary optimization

• Interop with C and C++, which is inherently unsafe

• Stack overflows or out-of-memory situations

We chose the first option to introduce a very reliable and unfortunately too well-known crash: The infamous segmentation fault. Look at this beauty:

unsafe { *(0 as *mut u32) = 42; }

This is a slightly modified version of the code we used in the recorded session. It dereferences the null pointer and tries to assign it a value of 42. Totally unsafe, and guaranteed to crash. Fun fact: depending on the optimization level, this line might either cause a segmentation fault or an illegal instruction.

Symbolicator already uses our Rust SDK. It is able to report errors and panics to Sentry, can record breadcrumbs, and can even manage concurrent scopes. However, it cannot handle hard application crashes. For that, we need to haul out the big guns.

Using the Native SDK

Our Native SDK to the rescue! We chose the Crashpad distribution, which creates a minidump when the application crashes. The SDK builds a dynamic library that can be linked to our Rust application. To instruct Cargo, we added a few lines to the build script:

println!("cargo:rustc-link-lib=sentry_crashpad");
println!("cargo:rustc-link-search=path/to/lib");
println!("cargo:rustc-cdylib-link-arg=-Wl,-rpath,@executable_path/.");

This tells the build system to link the dynamic library of our SDK, where to find it during the build, and where to find it at runtime. From now on, the dynamic library needs to be distributed alongside the main executable.

The Native SDK exposes a C API that is declared in the sentry.h header file. We chose to compile a small C file that initializes the SDK and exports a single function to our Rust code:

#include <sentry.h>

void init_native(void)
{
    sentry_options_t *options = sentry_options_new();
    sentry_options_set_dsn(options, "__SENTRY_DSN__");
    sentry_options_set_debug(options, 1);
    sentry_options_set_handler_path(options, "crashpad_handler");
    sentry_init();
}

To build and link this program, we added a call to the cc crate in our build script:

cc::Build::new()
    .file("native/src/init.c")
    .include("native/include")
    .compile("native")

Now, we are able to call the C function to initialize the Native SDK directly from Rust code. To do so, we declared the function signature in an extern "C" block. Calling a C function requires the use of unsafe, since the compiler can no longer reason about the side effects and contents of the function's body.

extern "C" {
    fn init_native();
}

unsafe {
    init_native();
}

Alternatively, you can call functions from the Native SDK directly. This requires FFI bindings for each function, which can be created conveniently using the bindgen crate at build time. It generates a Rust file from the C header. While we did not do this in our coding session — to keep it short — we generally recommend to use Sentry's API directly from Rust.

Wrapping Up

Finally, we could upload debug information of our new Symbolicator build to Sentry. This is normally done as part of our deployment process, but since we were not going to deploy this obviously broken version of Symbolicator — not that we've ever done that! — we had to upload manually using sentry-cli (paths abbreviated for readability):

$ sentry-cli upload-dif --include-sources symbolicator symbolicator.dSYM

The --include-sources flag makes source code available to Sentry. In the live session, we also used --wait to ensure that the files are processed by Sentry before we crash for the next time. Now, once the application crashes, the SDK sends a Minidump crash report to Sentry, which points right to the bug. See for yourself:

With debug information available, the application performance monitoring tool Sentry resolves Rust function names and even source context. In this case, you can clearly see how the stack trace points to our unsafe line of code. What a shame that not all issues will be this easy to debug.

Adding the Native SDK to one of our production Rust services was a fun exercise for us. We hope that you enjoyed this short excursion into the world of Rust with us, and would love to learn about your use cases. Targeting C and C++, Android NDK, and virtually any other language that compiles to machine code, the sentry-native is becoming one of our most versatile SDKs.

As always, you are most welcome to post feedback in our forum, or ask our support engineers for help.