← Back to Blog Home

How Distributed Tracing Works: Traces, Spans, and Trace Context

Distributed tracing captures the flow and timing of requests as they move through a system. By linking operations across multiple services, it shows you how a request ran, where it failed, and where it slowed down. In this blog post, we will cover the benefits of distributed tracing, explore its relevance for different application architectures, and show how it operates behind the scenes.

Additionally, we’ll examine the essential components of distributed tracing and discuss the propagation of trace context for achieving distributed tracing. By the end, you’ll know what’s actually inside a trace and how it stays connected across services.

What is Distributed Tracing? Benefits and Applications

Distributed tracing is the process of capturing the flow and timing of requests as they pass through a system. It links the operations that happen across multiple services, so you can debug a problem that started in one service and surfaced in another, and see where time is being spent.

So, what are the benefits of distributed tracing? Firstly, it provides you with a clear view of the entire operational flow of a specific action, from the frontend or mobile interface to the database. When something breaks, you can follow the chain of events that led to the error instead of guessing which service is responsible. When something is slow, the same view shows which operation took the time. Either way, you get to the root cause faster and improve the reliability of your service.

Now, let’s discuss who needs distributed tracing. If your application is built on microservices or serverless architecture, distributed tracing is essential. These architectures bring their own advantages but also introduce challenges such as reduced visibility. Distributed tracing serves as a perfect remedy in such cases.

However, even if your application is not built on microservices or serverless architecture, the benefits of distributed tracing still matter. Background jobs, third-party APIs, and AI agents that call models and tools all add hops that are hard to debug from a stack trace alone. And every developer wants to find performance bottlenecks, regardless of their application’s architecture.

Deep Dive: The Inner Workings of Distributed Tracing

Now, let’s explore the inner workings of distributed tracing and understand how it operates behind the scenes.

To begin, let’s take a look at a diagram that illustrates the various components involved in distributed tracing. At the core of it all is the trace, which represents a log of events occurring during the execution of a program. Think of it as a chain of operations performed during a specific task.

Diagram illustrating how a trace is composed of multiple transactions, and each transaction is composed of multiple spans.

Diagram illustrating how a trace is composed of multiple transactions, and each transaction is composed of multiple spans.

Every trace is uniquely identified by a UUID (Universally Unique Identifier). This trace identifier allows you to differentiate between different traces. Moving on, you have transactions. A trace consists of multiple transactions, each representing a different part of your application, such as the frontend, backend, or database. A transaction describes the operations carried out within a specific entity. For instance, it could be an API handler on the back-end.

Within each transaction, you have spans. Spans are the atomic units of work and represent individual resources being loaded, UI component lifecycle events, file I/O operations, requests made to external services, and more. Since transactions follow a tree structure, top-level spans can be further divided into smaller spans, mirroring the way one function may call several smaller functions.

Diagram illustrating the parent-child relationship between spans within a single transaction.

Diagram illustrating the parent-child relationship between spans within a single transaction.

Imagine the root span as a function calling three different child spans, each potentially representing a separate function. These child spans can then call four more functions, and so on. Each span requires its own span identifier, created as a UUID when the span begins its operation.

Now, let’s examine a real-world example to better understand how distributed tracing works. Consider a page load event triggered by the browser when a user lands on a page. In this scenario, you have three services involved: the browser, the backend, and the database — each service has its own transaction.

The browser, besides rendering the page, sends an API call to the backend to fetch the page’s data in JSON format. To retrieve the data, the backend sends a query to the database server. Finally, the database returns the requested data to the backend, which then transforms it into JSON format and sends it back to the page. Throughout this process, transactions are created for each service involved.

Now that we’ve covered what a trace consists of, let’s explore how we make it distributed. To achieve distributed tracing, our application must propagate the trace context between transactions. So in our previous example, the browser will send the trace context to the backend through the headers, so the backend will know which trace it should contribute spans to. The backend can also send the trace context to any other services it uses, or in this case, just measure the db queries using spans for each of them. At the end, the backend doesn’t necessarily need to return the trace context back to the client, because that’s where it originates from.

The trace context includes three values: the trace identifier, the parent identifier (which corresponds to the span identifier of the parent span), and the sampled value. The sampled value determines whether a particular transaction should be included in the trace. It can be set as no value (deferred decision), 0 (do not sample), or 1 (sampled). By passing the trace context, our services know to which trace they should append their transactions and spans.

And that’s a wrap! If you’d like to learn more, check out the distributed tracing video series I created here.

Syntax.fm logo

Listen to the Syntax Podcast

Of course we sponsor a developer podcast. Check it out on your favorite listening platform.

Listen To Syntax