A Software Observability Roundup

January 02, 2025

I spent some time recently catching up on my #to-read saves in Obsidian. More than a few of these were blog posts from 2024 about software observability. Talk of "redefining observability", "observability 2.0", and "try Honeycomb" had caught my eye in a few spaces, and so I had been hoarding links on the topic.

After spending a few days immersing myself in those articles and branching out to others, I decided to write this bullet-form roundup:

1. for myself, as a way of solidifying my current understanding
2. in public, as a way to invite corrections and improvements (drop a comment below or @parente.dev on Bluesky!)
3. with my colleagues in mind, as a new way to approach and discuss an ever-green question:

As our issue space changes and grows, and our solutions adapt and scale in response, what (else) should we do today so that we can readily address unknown-unknowns tomorrow?

Overview

The seventeen references I surveyed offer perspectives on observability as it pertains both to software systems and organizations around them. They cover what observability is, what problems it solves, how it is and should be implemented. There's alignment from the authors on the state of affairs, learned best practices, and a direction in which the industry should head. Shared terminology and goals are works in progress.

Origins

• According to control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.¹
• The discipline of software engineering (distributed computing, site reliability engineering, et al) has not settled on a single definition. One that stays close to the control theory original is that software observability measures how well a system's state can be understood from the obtained telemetry.¹
• Metrics, logs, and traces caught on as the three kinds of telemetry required to observe a software system—the so-called "three pillars of observability."
  • ... perhaps because they helped build a shared vocabulary at the 2017 Distributed Tracing Summit.²
  • ... perhaps because they do provide a comprehensive way for engineers to monitor systems for known problems and hint at where the issue lies.³
  • ... perhaps because solutions for monitoring systems using metrics, logs, and traces are what vendors had to sell.⁴

Problems and Limitations

• The task of analyzing disjoint metrics, logs, and trace data falls on humans when using three-pillar systems designed primarily for monitoring.⁵
  • Moving beyond investigation of known-knowns is difficult without data and tooling designed to support correlations and experimentation.⁶
  • Use of monitoring tools leads to org reliance on the intuition of a few system experts resulting in cognitive costs and bus-factor risks. Low visibility slows development and reduces team confidence.⁸
  • Using CloudWatch logs, CloudWatch metrics, and X-Ray traces together, for example, requires users to infer answers to questions from their mental model of the system, incomplete data, disparate views, and reading of code.⁹
• The three-pillar data model constrains the types of questions that can be asked and answered, with an almost exclusive focus on engineering concerns. Even mature observability programs will struggle to answer questions of greater interest and value to the business³, such as:
  • What's the relationship between system performance and conversions, by funnel stage, broken down by geo, device, and intent signals?
  • What's our cost of goods sold per request, per customer, with real-time pricing data of resources?
  • How much does each marginal API request to our enterprise data endpoint cost in terms of availability for lower-tiered customers? Enough to justify automation work?
• There are many sources of truth when disparate formats (metrics, logs, traces) and/or tools are in play, with decisions made at write-time about how the data will be used in the future. ¹⁰
• The value of metrics, logs, and (un-sampled) traces does not scale with the costs required to collect, transfer, and store them.⁵ As the bill goes up, the value stays constant at best, and more likely decreases.⁸
  • Logs get noisier and get slower to search with greater volume.
  • Custom metrics require more forethought and auditing as the set grows over time.

• "At the end, the three pillars of observability do not exist. It's not something we should be relying on."⁹

  • The coexistence of metrics, logging, and tracing is not observability. They are telemetry useful in monitoring systems.⁷

Better Practices

• Instrument applications to emit "wide events" (or "canonical logs" or "structured logs") as your telemetry data.

  • Wide events have high-dimensionality (many attributes) and attributes with high-cardinality (many possible unique values) making them context-rich (everything about the event is attached to it).¹¹
  • "High-dimensionality" roughly equates with hundreds of attributes at present. Metadata about hosts, pods, builds, requests, responses, users, customers, timing, errors, teams, services, versions, third-party vendors, etc. are all fair game.¹²

• Have a single source of truth which stores the wide events as they are emitted.

  • Do no aggregation at write-time. Make decisions at read-time about how to query and use the data.^{10 11}
  • Wide events from a service continuously handling 1000 requests per second—about 1 million events per day—can compress to about 80 MB in columnar formats like Parquet and cost pennies to retain for a few months in typical object stores.¹²
  • Custom metrics are effectively infinite as costs no longer increase linearly (thanks to columnar data storage) and the ability to cross-correlate increases as more event attributes are added. Intelligent sampling can control volume costs associated with these structured events when scale demands it.⁸
  • Storing event data in one place lends itself to the application of AI-tools which are good at correlating and summarizing¹⁴, perhaps continually in the background.⁹

• Adopt tooling that lets you explore quickly and cheaply—to discover emergent behaviors, answer new questions, prepare for unknown unknowns.

  • Proper tooling allows engineers to investigate any system, regardless of their experience with it or its complexity, in a methodical and objective manner.¹³
  • The waterfall view of traces, root spans, nested spans, and the like is not sufficient. Users need the ability to "dig" into data however they deem necessary.¹⁴
  • You will never ask the same question twice. Something is different since you last asked it.¹⁵
  • There is a natural tension between a system’s scalability and its feature set. You can afford more powerful observability features at scales orders of magnitude smaller than Google.⁵

Looking Forward

• Confusion abounds about what observability really is¹⁴ to the point that folks are actively redefining it^{15 3} or versioning it^{4 16} to improve clarity.

  • "Pretty much everything in business is about asking questions and forming hypotheses, then testing them." That's observability.³
  • The cognitive systems engineering definition of observability—feedback that provides insight into a process and refers to the work needed to extract meaning from available data—may be a better starting point for software engineering.¹⁵
  • "Observability is the process through which one develops the ability to ask meaningful questions, get useful answers, and act effectively on what you learn." It is not a tooling problem but rather a strategic capability akin to business intelligence.¹⁵
  • "Observability 2.0 has one source of truth, wide structured log events, from which you can derive all the other data types." The benefit to the full software development lifecycle, the cost model, and the adoption by a critical mass of developers make observability 2.0 inevitable.¹⁰
  • "Observability 1.0 gave us lots of useful answers, observability 2.0 gives us the potential to ask meaningful questions, and observability 3.0 is going to give us the ability to act effectively on what we learn."¹⁶

• There is consensus on the direction in which software observability should head: toward the better practices mentioned earlier. Discussion continues to establish shared language and goals.

  • "Observability 3.0 will be measured by the value that non-engineering functions in the business are able to get from it."¹⁶
  • "The success of Observability 2.0 will be measured by how well engineering teams can understand their decisions and describe what they do in the language of the business."¹⁷

References