Getting to That Elusive “Inbox Zero” With Custom Alerts and Codeowners

Forethought is a leading AI company providing customer service solutions that transform the customer experience. As a high-growth startup with a fast-expanding engineering org, teams had to deal with compounding complexity, leading to challenges measuring the impact and health of their services.

Forethought’s core engineering team maintains common services between other internal teams, infrastructure, data, and tools — and, as they added more engineers, the original team split into five. This made it hard to determine ownership of issues and contributed to a growing backlog of Jira tickets.

Our Jira backlog became increasingly noisy, especially as we added multiple projects, which impacted our ability to monitor release health.

-- Jad Chamoun, Engineering Manager, Forethought.

Ownership, alerts and getting the right info to the right people on time

Jad’s team identified ownership as the first step to working through their backlog of issues, so they integrated their GitHub codeowners file with Sentry. This way they could divide up and assign responsibility to specific teams, giving them greater autonomy and speeding up resolution. Now errors are filtered by team, automatically triaged, and assigned.

Next they implemented quarterly bug-bashes — think of these as a zero-inbox for errors where everyone tackles everything and wipes the slate clean. Okay, so quarterly isn’t exactly a cadence you’d expect to resolve a backlog of any sort, and neither did they. Jad started running weekly audits using custom alerts to see the volume of errors over time, highlighting those with the highest impact on users and tackling them first. This lets them allocate engineering time more strategically, ultimately freeing up time to manage their backlog.

By attributing distinct identifications to organizations within their ecosystem, Forethought can see which customers are impacted by a particular error, how many times a failure on an API call happened, and any fluctuations in transaction volumes. They’ve enriched this process by setting up alerts to fire when a failure rate hits a custom threshold, then quickly identifying where the issue came from and how to replicate it. Once a threshold has been hit, a new issue is opened and routed through code owners to the core services team.

Custom alert thresholds

I had to be able to balance my engineers’ time between fixing bugs and building new features, for that to happen we needed a solution that helped us stay on top of our backlog.

At this point, Sentry’s Slack integration comes in handy, as an error notification is sent directly to the channel of the responsible team – instead of everyone getting all the alerts in a single channel, trying to figure out who’s going to tackle it – where members can communicate within the channel and resolve it fast.

Jad’s team also leverages the Sentry API to investigate issues with custom-built connectors for internal tools, like Zendesk. This lets them deep link into specific issues, saving them time by reducing the amount of ad-hoc questions they have to process from outside teams, and helping them resolve and debunk issues themselves.

Enriching internal tools using the Sentry API

With a clear focus on reducing their Jira backlog, Forethought focused on increasing ownership and customizing alerts to improve reliability. In doing so, Jad also gave his engineers more autonomy and set them up with solutions that could scale as the business continues to grow.

If you’d like to learn more about Forethought’s workflow, check out our full conversation with Jad here.