Key Takeaways
- As infrastructure becomes increasingly cloud native, distributed, and AI-driven, legacy monitoring tools are cracking under the scale and velocity of workloads. What began as a way to give engineers much-needed visibility into systems has become one of the most painful and expensive parts of the modern infrastructure stack
- The shift from monolithic systems to microservices and now AI has turned observability from a search problem—finding the needle in the haystack—into a real-time analytics challenge that requires correlating billions of signals to form a unified view of system behavior. Many legacy systems have hit fundamental scaling and cost limits
- Modern observability stacks are being rewritten on real-time analytics engines such as ClickHouse that are designed to deliver full-fidelity visibility across massive telemetry streams at a fraction of the cost and latency of legacy systems
- Above this foundation, a new class of AI-driven observability platforms is emerging. These systems seek to leverage LLMs and AI agents to detect, diagnose, and remediate issues in real time —transforming observability from reactive firefighting into proactive intelligence that keeps systems continuously reliable, performant, and secure
Today’s applications generate more telemetry than ever, pushing legacy monitoring tools to their limits. As infrastructure becomes increasingly cloud native, distributed, and AI-driven, the traditional observability stack is cracking under the volume and velocity of modern workloads. What began as a way to give engineers confidence when systems failed has become one of the most painful and expensive layers in today’s infrastructure stack.
To address this, a new generation of observability software is emerging, powered by real-time data infrastructure and AI-native automation. Moving beyond passive monitoring, these systems aim to use live telemetry to detect issues, diagnose their root causes, and take autonomous action to maintain system health. In this shift, observability is evolving from a reactive cost center to a dynamic source of intelligence and competitive advantage.
The Two Fractures in Traditional Observability
At the data layer, the shift to cloud, microservices, and now AI has dramatically changed the shape and velocity of telemetry. Modern systems now consist of thousands of distributed components, each emitting their own stream of logs, metrics, and traces. The result is an explosion in data volume and complexity, transforming the core observability workload from a search problem—finding the proverbial needle in the haystack—into a real-time analytics challenge, where billions of signals must be correlated to form a coherent picture of system behavior.
Most legacy architectures, built for a simpler era, have hit fundamental scaling and cost limits. Coupled with their rigid ecosystems and ingest-based pricing, customers are often forced into a difficult tradeoff: observe less or spend more. Coinbase reportedly spent $65 million on Datadog in 2022,1 while OpenAI’s annual observability bill is rumored to approach $170 million.2 Security teams face similar strain, paying seven- to eight-figure Splunk bills just to retain logs.3
At the same time, the operational capacity of DevOps and security operations centers is approaching a breaking point. Incident response teams can be paged at 2 a.m. to search oceans of logs to pinpoint the source of failures, while security analysts are buried under anxiety-inducing dashboards and a constant stream of alerts, many of which are false positives or redundant. This isn’t just a visibility problem; it’s an execution crisis, where the surface area of infrastructure has outpaced what reasonable staffing levels can realistically maintain and secure.
Together, these trends reveal two core fractures in observability: many traditional data architectures are foundering under the scale of today’s workloads, while DevOps and security teams often feel trapped in a cycle of reactive firefighting, manual triage, and operational fatigue.
The Next Generation Observability Stack
THE REAL-TIME FOUNDATION
At its core, the next-generation observability stack is being built around real-time analytics infrastructure that’s purpose built for modern telemetry. Unlike architectures of the past, real-time analytics engines such as ClickHouse combine columnar architecture, high-throughput ingestion, and petabyte-scale storage efficiency with the goal of delivering ultra-fast queries across continuous streams of structured data. This allows teams to analyze vast volumes of telemetry in real time, potentially achieving full visibility at a fraction of the cost and latency of traditional systems.
AI-NATIVE APPLICATION LAYER
But visibility alone isn’t enough. The volume of services, alerts, and incidents far exceeds what human staffing levels can manage. What’s emerging at the application layer is a new class of AI-native platforms capable of automating the triage, analysis, and response workflows of infrastructure teams. Instead of relying on engineers to sift through telemetry, these systems deploy AI agents that are designed to autonomously detect, diagnose, and remediate issues in real time. Early examples include:
- AI SRE (Site Reliability Engineering) – Companies such as Traversal, Rootly AI, Resolve AI, and Cleric build autonomous incident responders that traverse dependency graphs, correlate logs and metrics, and identify root patterns in seconds—seeking to collapse incident response times and turn firefighting into continuous optimization.
- AI SOC (Security Operations Centers) – Platforms such as Dropzone, Prophet Security, Torq, and Tines transform SOCs from reactive command centers into proactive defense layers. They use agents that are built to triage, enrich, and correlate alerts to draft reports and automate remediation workflows, implementing robust internal controls around cash management, including independent verification for wire transfers.
In this new paradigm, DevOps and SOC teams are largely freed from the burden of manual investigation, allowing them to focus on higher-impact system engineering challenges.
FROM VISIBILITY TO INTELLIGENCE
At Adams Street, we’re excited about this next-generation observability stack: companies building real-time data infrastructure and intelligent applications that can elevate observability from reactive firefighting to AI-driven automation and continuous improvement.
At the data layer, we’re investors in ClickHouse, whose real-time engine already underpins observability pipelines for some of the world’s most data-intensive organizations—powering petabyte-scale monitoring at OpenAI, secure model-training observability for Anthropic, Tesla’s internal metrics platform, and Netflix’s logging system. As we enter the AI-native era, this real-time substrate is increasingly indispensable as telemetry becomes the raw ingredient for intelligence. As agents explore data, they will generate constant streams of iterative queries, introducing an entirely new access pattern for databases and demanding architectures that can sustain thousands of concurrent, low-latency interactions.
Above this foundation, we’re energized by teams building the AI-native application layer, helping to transform observability from a reactive cost center into a continuous source of reliability and insight. As agents begin to anticipate issues, optimize performance, and keep systems aligned with business goals, observability returns to being a true layer of confidence, which should enable real-time data and intelligent automation to work together to keep infrastructure continuously healthy and secure.
1. The Pragmatic Engineer, Datadog’s $65M/year customer mystery solved, May 12, 2023
2. Investing.com, Guggenheim downgrades Datadog on fears that OpenAI will cut spending, July 8, 2025
3. Splunk Announces Fiscal Fourth Quarter and Full Year 2024 Financial Results
Important Considerations: This information (the “Paper”) is provided for educational purposes only and is not investment advice or an offer or sale of any security or investment product or investment advice. Offerings are made only pursuant to a private offering memorandum containing important information. Statements in this Paper are made as of the date of this Paper unless stated otherwise, and there is no implication that the information contained herein is correct as of any time subsequent to such date. All information has been obtained from sources believed to be reliable and current, but accuracy cannot be guaranteed. References herein to specific sectors, general partners, companies, or investments are not to be considered a recommendation or solicitation for any such sector, general partner, company, or investment. Any such highlighted investments have been listed for illustrative purposes only, however, there can be no guarantee that similarly attractive investments will be available in the future; subject to confidentiality obligations, a complete list of Adams Street investments within the related portfolio is available upon request. This Paper is not intended to be relied upon as investment advice as the investment situation of individuals is highly dependent on circumstances, which necessarily differ and are subject to change. The contents herein are not to be construed as legal, business, or tax advice, and individuals should consult their own attorney, business advisor, and tax advisor as to legal, business, and tax advice. Past performance is not a guarantee of future results and there can be no guarantee against a loss, including a complete loss, of capital. Certain information contained herein constitutes “forward-looking statements” that may be identified by the use of forward-looking terminology such as “may,” “will,” “should,” “expect,” “anticipate,” “estimate,” “intend,” “continue,” or “believe” or the negatives thereof or other variations thereon or comparable terminology. Any forward-looking statements included herein are based on Adams Street’s current opinions, assumptions, expectations, beliefs, intentions, estimates or strategies regarding future events, are subject to risks and uncertainties, and are provided for informational purposes only. Actual and future results and trends could differ materially, positively or negatively, from those described or contemplated in such forward-looking statements. Moreover, actual events are difficult to project and often depend upon factors that are beyond the control of Adams Street. No forward-looking statements contained herein constitute a guarantee, promise, projection, forecast or prediction of, or representation as to, the future and actual events may differ materially. Adams Street neither (i) assumes responsibility for the accuracy or completeness of any forward-looking statements, nor (ii) undertakes any obligation to update or revise any forward-looking statements for any reason after the date hereof. Also, general economic factors, which are not predictable, can have a material impact on the reliability of projections or forward-looking statements. Adams Street Partners, LLC is a US investment adviser governed by applicable US laws, which differ from laws in other jurisdictions.
