Platform Engineering at Google (First Early Release) (James Brookbank, Leah Hargreaves etc.)（Z-Library）

Platform Engineering at Google Foundational Insights for Software Systems With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles. Curated by James Brookbank, Leah Hargreaves, and Salim Virji

Platform Engineering at Google Curated by James Brookbank, Leah Hargreaves, and Salim Virji Copyright © 2027 Google, LLC. All rights reserved. Published by O’Reilly Media, Inc., 141 Stony Circle, Suite 195, Santa Rosa, CA 95401. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Louise Corrigan Development Editor: Rita Fernando Production Editor: Clare Laylock Interior Designer: David Futato Interior Illustrator: Kate Dullea December 2026: First Edition Revision History for the First Edition 2026-06-10: First Release See https://oreilly.com/catalog/errata.csp?isbn=9781098169435 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Platform Engineering at Google, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have

used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-098-16937-4 [LSI]

Brief Table of Contents (Not Yet Final) Chapter 1. What Is Platform Engineering? (unavailable) Chapter 2. Platform Engineering Inside Google (unavailable) Chapter 3. Platform Engineering on Google Cloud (unavailable) Chapter 4. A History of Platforms at Google (unavailable) Chapter 5. The Monolithic Ideal: Inside Google3 (unavailable) Chapter 6. The Composable Ecosystem: Google Cloud’s IDP (unavailable) Chapter 7. Just Enough Platform Engineering (unavailable) Chapter 8. Modular Abstractions (unavailable) Chapter 9. Coupling (unavailable) Chapter 10. Layers (unavailable) Chapter 11. Quality Attributes (unavailable) Chapter 12. Deep Dive: Efficiency Chapter 13. Deep Dive: Security (unavailable) Chapter 14. Deep Dive: Reliability (unavailable) Chapter 15. Ecosystem Types (unavailable) Chapter 16. Shared Fate and Responsibility Chapter 17. AI with Platform Engineering (unavailable) Chapter 18. Socio-Technical Platforms (unavailable) Chapter 19. Education and Platform Engineering Chapter 20. Shifting Down (unavailable)

Chapter 21. Why Developer Experience Matters (unavailable) Chapter 22. Platform as a Product (unavailable) Chapter 23. Platform ROI (unavailable) Chapter 24. Golden Paths (unavailable) Chapter 25. Techniques for Building on Cloud Platforms (unavailable) Chapter 26. Antipatterns for Cloud Platforms (unavailable) Chapter 27. Measuring Platform Success (unavailable) Chapter 28. Platform Evolution (unavailable) Chapter 29. Cloud Platform Migrations (unavailable) Chapter 30. How to Get Started (unavailable) Chapter 31. Conclusion (unavailable)

Chapter 1. Deep Dive: Efficiency By Chris Kennelly A NOTE FOR EARLY RELEASE READERS With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles. This will be the 12th chapter of the final book. Please note that the GitHub repo will be made active later on. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at rfernando@oreilly.com. Software ecosystems provide leverage for understanding and improving the performance of the applications that operate within them. This leverage multiplies the investment in the ecosystem. Common foundations make it practical to understand the resource usage of software. These in turn create opportunities for improving these elements to reduce the cost of operation. Creating the Virtuous Efficiency Cycle: Lessons from Production Google’s cluster management system, Borg, abstracts job scheduling decisions away from its users. Rather than requiring users to manage the mapping of workloads by hand, Borg manages job colocation to allow the machines to run at higher utilization reliably.

A single team’s workloads might be more CPU-centric while another team’s could be more RAM-heavy. If the two ran separately, they would need differently-shaped machines to avoid resource waste. Rather than provision spare machines to accommodate failures on a team-by-team basis, Borg provides redundancy at the data center level. These lessons of running Borg in production influenced the design of Kubernetes. As part of Borg, each machine has a small number of common processes that oversee system health, configuration, and manage running jobs. This commonality allowed us to build a profiler that could run on each machine, enabling teams to debug performance issues while providing a powerful, aggregated view. We named this infrastructure Google-Wide Profiling (GWP). Aggregated data created a new set of opportunities for improving job performance automatically. For Google’s most performance-critical and resource-intensive workloads, teams had used profile-guided optimization (PGO) features in the compiler to improve their performance. At the time, PGO required these teams to collect profiles of their application in order to provide them to the compiler when making optimization decisions. Realizing these gains was not an easy process. Each team had to set up a loadtest to replay production-like traffic. On each new version of their software, they would need to first build an instrumented version, run their loadtest, gather the resulting profile, and provide that to a second build phase to actually optimize their program. Using GWP, we could aggregate performance counters collected across Google’s fleet to record which functions were called and which conditionals were taken. We called this “AutoFDO.” This allowed us to synthesize a profile for each team in Google and their workloads. By dispensing with the need to develop a reliable and accurate load test and the extra build steps, it was far easier for teams to adopt with a single line change to their build

configurations. Automatic and ready-to-use profile generation unlocked substantial resource savings, since the prior required effort deterred PGO adoption. Further, while common infrastructure is required to keep the profiles up to date, it can be maintained centrally with far less engineering effort than replicating it across teams. In the first year, AutoFDO doubled the fraction of Google’s fleet using PGO; and today, it is the majority of our PGO-optimized compute usage. The centralized telemetry of GWP allowed us to better understand our resource costs across the entire fleet of applications. In 2015, our memory allocator, TCMalloc, used approximately 8% of Google’s fleetwide usage, making it larger in aggregate than any single application. The common libraries that make up our ecosystem are an essential source of leverage. We use them for better developer productivity. Their influence on performance creates a second source of leverage: A central optimization to these libraries is a rising tide that lifts all boats, without requiring each development team at Google to evaluate whether their usage is optimal. We could build up expertise in each of these areas–memory allocation, compilers, and so on– that would be hard for any single application team to develop while concerned with the code they steward. Shifting Optimizations Down Shifting optimizations into the platform allows us to leverage it, avoiding each team from having to rediscover the same lessons for themselves. While this doesn’t fully relieve teams from having to think about performance from time to time, it allows them to focus their mental budget on situations unique to them. As business needs and use cases change, centralized performance improvements let engineers focus on those outcomes, rather than considering how the latest computer architecture will affect them and prepare for it. On-by-Default

In developing these optimizations, we generally strived for an “on by default” approach. After prototyping, canarying with major workloads, and evaluation, we aimed to enable the optimizations for everyone so it would be taken up with their release cycle. This avoided the pitfall of building it and having no one come, as individual teams didn’t need to discover for themselves each new optimization we developed. The expertise of the engineers developing changes was also best suited to identify when and where to deploy each optimization, rather than having engineers ramp up to become (temporary) allocator experts, and then (temporary) compiler experts, and so on for each change. By making improvements centrally, these optimizations quickly flow to everyone as software is released. In Google’s monorepo, teams routinely branch from head and create new releases of their software so it can be deployed to production. Since teams would be releasing new software anyways, receiving these improvements requires no additional effort and it can happen behind the scenes. After discovering TCMalloc was our largest software component as a proportion of CPU usage in our data centers, we began iteratively improving it. Profiles allowed us to find hotspots, patterns of high usage, so we could identify opportunities to improve performance, implement them, and deploy them to the fleet. By leveraging the compiler and Linux kernel, we developed and upstreamed novel features like sized deallocation and per-CPU caches to improve performance. Over time, we pivoted towards developing better allocation policies that improved application performance by changing data layout. Small changes to layout might have negligible impact on the cost of allocating memory itself, but could have large impacts on how the application performed as it ran its business logic. In adopting this approach, our aim was to improve all of the applications Google runs. This is not always possible, since in a large software ecosystem, there are some edge cases where an optimization has a negative

impact on performance. Rather than block on the presence of any regression, we manage these to optimize for our overall fleet performance: If the regressions are minor, we absorb them in the interest of delivering an aggregate benefit. We choose this over the alternative of fine-tuning each application. Reduced configurability reduced complexity, allowing us to sustain a high cadence of improvements. Options increase the state-space that has to be considered with every future change, making it more difficult to reason about, test, and successfully land new features in production. For otherwise good optimizations with large regressions for some, we provide an opt-out mechanism. New optimizations can be typically turned off on an application-by-application basis if they introduce unforeseen, but significant regressions. These were generally rare, but we have sought to remove them expeditiously. Opt-outs can tell us about what edge cases have proven to be truly important and need more attention, leading to further optimization insights. For example, a single, long-standing opt-out from Temeraire informed two distinct follow-on improvements. While it can be tempting to only make changes exactly where they’re beneficial, an imperceptible regression or the toil of an opt-out for one team might be better than the opportunity cost of failing to deploy a major win for hundreds of others. Spreading Best Practices Another way to build optimizations is to take the best practices of teams and shift them into the ecosystem for all to benefit from. For example, Abseil’s hash table implementation, SwissMap, originated out of a partnership between our C++ core libraries team and two Zurich-based engineers responsible for maintaining Google’s search index. They had set out to make an improved hash table that used modern hash table techniques,

improved the design to reduce memory usage, and added API features to soften common sharp performance edges. For the indexing team’s purposes, most of the benefits for their application could be obtained by replacing their hottest hashtables and declaring victory. Rather than stopping there, we opted to drive widespread adoption of the superior hashtable, replacing other usage throughout Google’s monorepo with SwissMap, using our large-scale change infrastructure to refactor C++ code at scale. This allowed every other team to get the benefit of the improved implementation, reducing our CPU and RAM requirements substantially. Without the scaled, centrally-driven migration, we would not have realized these savings. The broad rollout produced ongoing ecosystem dividends: “Just use SwissMap” instead of considering several other hashtable implementations was good advice because of its implementation quality, but widespread usage of it made it easier for new code to reach for it. For an engineer trying to develop a new feature, it is easier to use the existing types than to migrate everything first, or carefully benchmark among several choices. Even if today’s choice is optimal, changes in usage can reverse the decision, adding to the toil each team has to sustain to keep up. When we had tried to retrofit optimizations in previous implementations, we usually hit stumbling blocks from Hyrum’s Law: Software had developed inadvertent dependencies on iteration order, requiring those dependencies be removed before we could move forward with the change. Drawing on those lessons, we introduced randomization to prevent code from relying on the order of iteration. This made it easier to land subsequent optimizations under the hood, allowing us to iteratively improve the hash function by changing its algorithm.

Because of our implementation freedom, we were able to add telemetry features like built-in profiling. This has allowed us to find and fix bad hash functions, identify optimizations in the implementation, and improve application usage of their hashtables. Working Further up the Stack While we worked to make the core libraries, compilers, and production environment our platform uses as fast as possible, sometimes the best solution to improve performance was to work further up the stack. At the lowest level, a function like `memcpy` needs to copy data from one location to another. This operation has to be done unconditionally: There’s no opportunity to identify that the copy doesn’t actually need to happen. Some of the calls to `memcpy` might be part of a much larger, logical copy operation. By analyzing the source code, we might identify that the source of the copy could be reused, letting us elide the operation entirely. The fastest copy is the one we don’t make. This is where code analysis tools ranging from Clang-Tidy to LLM coding tools have been helpful for both identifying and capturing these opportunities. “Something is expensive” is not necessarily actionable, but combining this observation with automated analysis of the code allows us to greatly improve the signal-to-noise ratio to use engineer time productively. Since many of these performance pitfalls fall into a set of archetypes, we’ve sought to prepare code changes (through automated tools) and sent them teams for review. This minimizes friction, since central teams can quickly develop and apply the same context for each change they make, rather than requiring each team to get up to speed each and every time for each optimization. Where possible, we’ve aimed to make these tools an ambient part of our coding ecosystem. Detecting and preventing at code review time allows us to avoid regressions altogether as part of our efforts to shift left. Shifting Left

“Shifting left” is often applied in testing for correctness, but it is also a vital strategy for avoiding regressions that impact efficiency. It is easier to find and fix correctness bugs earlier in the development lifecycle. The same is true for performance. Making sure a new feature is efficient upfront means it’s less likely to be delayed by a lack of compute capacity or scalability limits as users begin to use it. The following are a number of techniques we’ve adopted a number of techniques to find (or prevent) performance issues from persisting: Once changes have rolled out, we use our fleetwide telemetry to look for performance anomalies and raise alerts where regressions have occurred in specific functions, applications, or libraries. As software updates are rolled out, the changes are canaried by the teams making them. They update a subset of the servers running their software and pause, giving an opportunity to perform automated reliability and performance checks to compare the old with the new. Teams can specify small or large scale benchmarks. By running these periodically as changes are made to the code base, they can help catch issues that have landed, but before they are widely deployed to production. Similarly, engineers can run them on their changes as part of the development process, allowing potential issues to be identified earlier and mitigated. Automated tooling can assist with the code review process. Common performance antipatterns can be flagged, and often recommended code provided, to engineers authoring and reviewing changes. Carefully designed libraries in the ecosystem make the efficient path well-lit. Besides improving productivity, straightforward libraries make it easier to ask high-level questions rather than worrying about whether something will work at all.

Creating Performance Observability The consistent infrastructure of our ecosystem has allowed us to steadily improve observability, making it possible to debug, understand, and improve the efficiency of our ecosystem. These tools work along a spectrum from individual processes to fleet wide performance: Monitoring tools for observing system health [SRE book] Profiling tools for developing a fleetwide picture [GWP] Tools for understanding a distributed system [Dapper, Census] For example, having a common RPC system for communicating between services made it possible to add these tools over time. Tracing features could be implemented in one place and immediately benefit from network effects. Our profiling services had a consistent touchpoint for retrieving profiles, making it possible to expand the set of profile types over time. These observability features created visibility into the performance of our systems, allowing them to be steadily improved. This, in turn, fed into more investments into observability. To gather a new piece of data, performance engineers didn’t have to pay the full bootstrapping costs for the entire ecosystem. A new idea for our memory allocator’s telemetry could ride along existing rails, rather than being built from scratch. Telemetry for commonly used data structures like SwissMap required more work, but still far less than would be required without the ecosystem’s heavy usage of certain data structures and the collection infrastructure. Without this centralized infrastructure, data would need to be gathered in an ad hoc fashion. Profiles gathered from a handful of workloads can be informative, but we might fall victim to a streetlamp effect, looking at the already best characterized, understood, and optimized workloads rather than where the largest opportunities were, simply because they are easiest for us to look at. As we use these tools to identify ever larger efficiency opportunities, we’ve worked to connect widely available signals to Google’s businesses.

Connecting to Business Value We don’t buy servers to allocate memory in TCMalloc, put protocol buffers into other protocol buffers, or to handle branch mispredictions by our processors. These low-level factors can be easily measured, but ease of measurement does not imply they are meaningful to the business. Improving efficiency for users of our ecosystem requires understanding which changes constitute improvements, and which don’t. Google manages a vast fleet of servers to handle search queries, AI queries, process log records, and transcode videos. Maximizing value and minimizing costs are the outcomes we ultimately aim for. Engineers optimizing performance are ultimately trying to maximize the useful things Google does for our users (serve search queries, videos on YouTube, etc.) and minimize the things Google has to buy (CPUs, RAM, disks, etc.) to do so. While measuring a job’s resource consumption is easy, it’s harder to tell just how much useful work it’s accomplishing without help. A service might use more CPU or RAM because it has suffered a regression, but another equally plausible answer is that it is simply busier. We cannot determine from casual inspection what caused the increase in CPU usage, whether this is from an increase in workload (serving more videos per unit time) or a decrease in efficiency (some added, needless protocol conversion per video). Over time, we’ve leveraged other parts of our ecosystem so that services can report how much useful work they are performing over time. If we know the number of videos processed, we can easily determine whether we are getting more, or less, real work done per CPU-second (or byte-second of RAM, disk operation, or hardware accelerator time). We refer to these metrics as application productivity metrics, or APMs. At first glance, distilling “business value” for each internal service into a single metric is a tall order. By leveraging other parts of our platform, we could provide reasonable, out-of-the-box defaults for most users: RPC-

focused servers would export total requests, data processing pipelines would export total records, and so on. With further analysis, we were able to identify outliers: A service that processes batches of work might prefer to count the total number of elements processed, not the number of batches. The interlocking observability tools allowed us to find these systematically, uncovering the occasional misconfigured service whose resource usage did not correlate its metrics. While perfect correlation was not a direct goal–reporting resource usage as “useful work” would be perfectly correlated–it gave us a basis for finding outliers. Without APMs, certain classes of efficiency optimizations wouldn’t be possible to measure accurately. Our TCMalloc work to optimize layout didn’t make the allocator itself any faster, but instead sped up the rest of the application as a result of its decisions. In relative terms, this made memory allocation look more expensive, even though applications could now process more with fewer resources. Over time, we described our memory allocator as “faster than light”: We sped up applications with intelligent allocation decisions by far more than we would have saved by naively reducing the cost of memory allocation. While the relative costs of a library are often a good proxy, this underscored the importance of connecting things back to as close to a source of truth as possible. Understanding Criticality In the course of handling requests across servers, there’s work that happens independently from user requests: A search query from a human is valuable. A search query from a prober used for monitoring is important insofar as it helps with monitoring the service’s reliability; but search exists for humans, not for monitoring. A human might no longer be interacting with the system, but it continues to use resources anyways. For example, a YouTube

session might continue to play for several minutes after someone has left their computer. The automatic playback pause after inactivity cuts off a flow of unimportant video plays. Request hedging helps give users consistent latency by sending duplicative requests: One of the requests and the work done might prove to be unnecessary. Taken to an exaggerated extreme, sending multiple hedged requests unconditionally is wasteful: Users don’t get meaningfully better latency and production uses resources profligately. Choosing the right time to hedge can let us strike a balance. Deadline propagation allows child requests to backends to time out early, rather than continue to perform work believing the requestor will still use it. This is an important technique for minimizing unproductive work, especially during overload. Load balancing and load shedding do not contribute directly to request processing, but they are critical to running high reliability services at practical utilization levels. A datacenter with requests haphazardly sent from one server to another with no overload protections would need to be mostly idle to keep hot spotting from overwhelming part of the system. Load testing a machine or service to the point of overload produces valuable knowledge for capacity planning and validates that load shedding mechanisms work. The requests do not directly answer user requests; but without this knowledge, the user experience would eventually suffer as regressions go unnoticed. Similarly, loadshedding infrastructure might see limited usage day-to-day, but it is critical to reliable operation when capacity is degraded. Many of these activities are effectively a cost of doing business. It’s hard to run a reliable, low-latency service distributed across numerous servers without them. Nonetheless, it’s important to not let the tail wag the dog here: Monitoring or load testing are not an end unto themselves.

Understanding how these uses fit into the bigger picture helps you run with fewer idle resources while realizing higher levels of reliability. Being able to shed unessential requests during a datacenter disruption lets us avoid overprovisioning. Background work can be shuffled to off-peak hours. Selecting Good Proxies Along the way, there are proxy metrics that can help with telling us that our optimization idea is on the right track, or help to explain the causal connection to top-level metrics. We want to align with the business problem, without boiling the ocean every time we make a small change and want to assess it. Measurement has its own return on investment too, and the benefits of additional precision is quickly outweighed by the cost of obtaining it. Similar to how we can carefully use small scale benchmarks to predict larger, more complex benchmarks to predict production, we can select proxy metrics to measure success. This lets us align with business goals, especially harder to measure ones, while still having an effective yardstick for day-to-day work. Goodhart’s Law reminds us that “when a measure becomes a target, it ceases to be a good measure.” Escaping this completely is challenging, but analysis is easier the more closely aligned the metric is with what we’re optimizing. Lessons Learned Take the Long View Opinionated ecosystems may appear to be an impediment to optimization, but it is easy to miss the opportunity costs of catering too much to specific use cases or flexibility. The costs of the former tend to be visible, while the latter is not. A healthier, more sustainable ecosystem can deliver broad optimizations that help each participant in it.

For example, the engineers who designed SwissMap made it randomize its iteration order. They had seen the previous challenges when valuable optimizations were delayed because, despite the name “unordered,” code had accreted dependencies on the ordering. This randomization carries slight, direct costs that are challenged occasionally. Teams using SwissMap would sometimes ask for randomization to be disabled to simplify their own testing infrastructure or make copies a bit faster. The indirect costs are less obvious: Turning off the randomization feature would make it harder to add further optimizations to common operations. This would hurt not just the ecosystem, but also the individual users for whom randomization was a pain point. Configurability can be a short-term boon, but long-term it is a double-edged sword. The options need to be discoverable to be useful, otherwise the options may never be used. Options increase the state-space that has to be considered with every future change, making it more difficult to reason about, test, and successfully land new features. What was initially a great expedient becomes a tax on velocity that delays improvements. Don’t Assume Immutability Teams working on a single application sometimes treat the underlying ecosystem and platform as immutable, and solve problems within that constraint. The platform itself is rarely immutable if there’s sufficient value to be gained from an ecosystem-wide change. Even if it makes sense to work around a problem for reasons of expediency, discovery of these pain points can raise awareness. A frequent problem might look rare if everyone works around it rather than seeing it get fixed in the platform once and for all. This becomes more practical with the economies of scale from centralization, since needed expertise can be built up. Create Leverage and Use It