How HTTP Toolkit Debugs Netlify Errors with Sentry

Tim Perry, the creator of HTTP Toolkit (a suite of developer tools for debugging, testing and building with HTTP), recently detailed his experience debugging Netlify function errors with Sentry in his aptly named blog post, Debugging Netlify Function Errors with Sentry.

Netlify functions are a quick, easy and powerful tool, but like most serverless platforms, they can be even more difficult to debug and monitor than traditional server applications. It's a hard environment to precisely recreate locally, there's no machine you can SSH into in a pinch, and no built-in error notifications.

Your code is going to break eventually, and you need the tools to fix it.

HTTP Toolkit uses Netlify functions under the hood to manage user account information and Paddle checkout callbacks. If we hit errors here, people's payments will fail, or they'll stop being given access to paid features, so this can be pretty bad! I need to be able to catch errors immediately, debug and work out why they're happening, and confirm that my fixes work.

Debugging and fixing issues here is a big topic, but one of the first steps is knowing exactly when and how errors happen. There's a few tools for this, but personally I've had a lot of success on projects recently with Sentry. They've got a generous free plan (5k errors a month), built-in integrations for almost everything, and some good and detailed error reporting tools too.

If you can get Sentry set up, you'll get emails every time there's an error in your function, and you can explore the errors themselves in detail to work out exactly what failed. Perfect, but the setup for Netlify functions has a few extra steps.

Start reporting errors to Sentry

I'm going to be using JavaScript here, and I'm assuming you've already got a working Netlify function set up. To add Sentry reporting from there, you need to:

• Create a Sentry account

• Create a Sentry project in that account for your functions

• Take the DSN for your Sentry project and set it as a SENTRY_DSN variable in your Netlify build

• npm install --save @sentry/node^4.6

• Initialize the Sentry SDK:

const Sentry = require("@sentry/node");
Sentry.init();
// Sentry will pickup your DSN from process.env.SENTRY_DSN
// Alternativly you can do this Sentry.init({ dsn: process.env.DSN });

With just this in place, uncaught errors and rejections are now reported automatically!

Unfortunately though, errors/rejections in handlers are caught and swallowed, so we'll need to catch those too. In addition, AWS Lambda (the service behind Netlify functions) doesn't behave exactly as you'd expect, so some error reports will be delayed or lost when your function is shut down after execution.

Catching handler errors

Let's detect handler function errors first. To start with, create an convenient reportError method you can call to report errors to Sentry, which will wrap the extra logic that we'll need in a minute.

// Don't use this example quite yet! It's not complete - see below.
function reportError(error) {
  console.warn(error);
  if (typeof error === "string") {
    Sentry.captureMessage(error);
  } else {
    Sentry.captureException(error);
  }
}

Then add a wrapper around each of your function handlers. The wrapper needs to call the function handler as normal, but catch any errors or promise rejections, and report them to Sentry. It then needs to rethrow the error too, so that an HTTP error is still returned:

// Don't use this example quite yet! It's not complete - see below.
function catchErrors(handler) {
  return async function() {
    try {
      return await handler.call(this, ...arguments);
    } catch (e) {
      // This catches both sync errors & promise
      // rejections, because we 'await' on the handler
      reportError(e);
      throw e;
    }
  };
}

Reliable reporting with Sentry & AWS Lambda

A Lambda function runs until completion, and then will be frozen. Later calls may start it up again, or it might be disposed of, and the whole process created afresh. That means that any Sentry requests that haven't been sent when your function responds might be lost.

Fortunately, we can fix this. We need to do two things: wait for reported errors to be fully sent, and ensure that Sentry doesn't interfere with normal Lambda shutdown.

Change your report error function to the below:

async function reportError(error) {
  console.warn(error);
  if (typeof error === "string") {
    Sentry.captureMessage(error);
  } else {
    Sentry.captureException(error);
  }
  await Sentry.flush();
}

Lastly, to stop Sentry callbacks interfering with normal Lambda lifecycle, we need to set context.callbackWaitsForEmptyEventLoop to false.

We can do this in our handler wrapper, and we also need to update that wrapper to wait on the reportError call too, to make sure that it's completed.

Change your catchErrors wrapper to:

function catchErrors(handler) {
  return async function(event, context) {
    context.callbackWaitsForEmptyEventLoop = false;
    try {
      return await handler.call(this, ...arguments);
    } catch (e) {
      // This catches both sync errors & promise
      // rejections, because we 'await' on the handler
      await reportError(e);
      throw e;
    }
  };
}

All done! With this in place, all handler errors will be reliably reported to Sentry, and you can rest safe in the knowledge that your functions are working nicely (or at least, that you know exactly how much they're failing).

Bonus Extensions

There's two optional extra steps I'd like to mention, to help you debug your issues more easily.

First, extra reporting is super useful. Sentry.addBreadcrumb for example lets you record extra events that will be included in any later exceptions. You can also call our reportError function from anywhere else in your code to immediate report errors, even if you don't actually throw them and fail (but do remember to wait on the returned promise).

Second, include your function's git commit as your Sentry release, so you always know which version of the code threw which errors. Netlify provides this as a COMMIT_REF environment variable, but this sadly isn't available in the runtime Lambda environment, so we need to make sure we bake it in at build time. To do that, first extend the default webpack config:

// webpack.js:
const webpack = require("webpack");
const { COMMIT_REF } = process.env;

module.exports = {
  plugins: [
    new webpack.DefinePlugin({
      "process.env.COMMIT_REF": JSON.stringify(COMMIT_REF)
    })
  ]
};

You'll need to change your build script to pass -c ./webpack.js to your netlify-lambda build command to use this.

Then, change the initial Sentry setup to pass this variable on to Sentry:

const Sentry = require("@sentry/node");
const { COMMIT_REF } = process.env;
Sentry.init({ release: COMMIT_REF });

And voila, automated error reports for Netlify functions:

Want to see a complete example of this in action? Take a look at HTTP Toolkit's accounting internals.