I’ve found that the biggest lever is the environment you build around the agent. The constraints, feedback loops, and evaluation criteria that shape what the agent produces. Birgitta Böckeler writes about this process (called harness engineering) in detail1.

The harness

A harness is everything surrounding the agent. What it can see, what feedback it gets, what constraints it operates under. You define what “good” looks like, and the harness tells the agent when it’s not there yet.

Think of reconciliation loops in Kubernetes. You declare the target state. A reconciler works toward it. You don’t micromanage the steps. You define what “good” looks like and let the system figure out how to get there.

Harness engineering works the same way. You encode your judgment into the harness and the agent reconciles toward it.

Fitness functions as a harness

I recently built the first version of an AI-native architectural analysis tool. I started by building the evaluation layer.

First, I defined fitness functions. These encode my architectural judgment about what “good” means for this specific system. One example: the tool sends a text summary of the codebase to an LLM, so I defined a token cost fitness function. It measures the token count of the generated summary and fails if it exceeds 12,000 tokens. That threshold encodes a judgment: the summary must fit in a single LLM context window with room for the analysis prompt. Other fitness functions cover findings quality, schema conformance, and parser coverage.

Then I built a dashboard to compare approaches against these functions.

Only then did I let AI agents implement solutions. The harness was already in place.

The dashboard lets you compare solutions against your judgment. When an approach scores poorly on a fitness function, you see it immediately.

Fitness Functions are target state contracts.

When you put your architectural judgment into the evaluation layer, every AI-generated solution gets measured against it automatically. You can fan out multiple agents exploring different approaches and let the harness pick the winner.

The judgment is still yours. You chose the fitness functions, you decided what “good” looks like. Now that judgment is encoded in something durable and repeatable.

This comes with a risk. Fitness functions can encode the wrong judgment. If you optimize for token cost alone, the agent will produce summaries that are short but miss critical architectural details. The harness is only as good as the judgment you put into it. That’s why you need multiple fitness functions that balance each other, and why you should revisit them as you learn more about your problem.

Build the evaluation before the implementation. Let the harness carry your judgment. Evaluate constantly.

This leaves the question: what is left for us software architects?

The answer is judgment. And what I’ve found is that AI can serve as something unexpected: a flight simulator for developing it.

Why judgment stays human

Let me go back to that tenant-isolation example. The AI laid out three strategies with their technical characteristics: latency, throughput, isolation level, implementation cost. All useful. But I was the one who recognized that the relevant dimension to evaluate wasn’t any of those. It was the fact that one strategy handled high load by rejecting requests to protect other tenants. In this specific business context, that’s unacceptable. The AI couldn’t know that.

That was a judgment call requiring understanding of consequences across the whole system and its relationship to the business context. I knew which question to ask, out of all the possible questions. I brought the lens that mattered. LLMs can surface technical characteristics, but they cannot understand your business context well enough to make the trade-off decision for you.

The flight simulator

So if judgment is our core contribution, how do we get better at it?

Researchers on expert intuition tell us that two things matter: a domain with learnable patterns, and enough practice with timely feedback. Kahneman and Klein call this the foundation of “naturalistic decision-making”. It’s how firefighters, chess players, and surgeons build expert intuition1.

I tried to create exactly that environment using AI. Working conversationally with Claude, I had it suggest possible solutions and implement whichever one I chose. I made the architectural decisions; Claude handled the code. This let me iterate on decisions in minutes instead of days, stripping away implementation time so I could focus purely on architectural thinking. It’s a flight simulator for architects.

An important caveat

This approach works best for quality attributes you can measure quickly and holistically. In my case, I was trying to improve latency, so I had an objective measure to optimize for: load and performance tests. When I deployed a bad decision, I immediately felt it in declining numbers. When I got it right, the improvement was just as visible. That tight feedback loop is what makes the practice effective.

Other attributes are harder. Maintainability across 17 teams? That feedback loop doesn’t fit in a practice project. But for the attributes you *can* measure (like latency, throughput or availability) this is the fastest feedback loop an architect has ever had.

Before AI-assisted coding, spinning up a full practice environment (Docker, Kafka, Flink, a load generator with out-of-order events) would have eaten all the time I had for actual learning. Now, I spun up an entire practice project with minimum effort. Using that, I only implemented the parts with the highest learning payoff. For streaming systems this was choosing watermarks, windowing strategies and implementing channels. What I automated was all the boilerplate: setting up a docker swarm, installing Kafka and Flink as well as implementing a sample application which emits a high load of events (with some of them being out-of-order).

Project-based learning helps discover gaps in your knowledge much better than passive reading … and now you can engineer the learning environment yourself with an AI agent acting as a feedback loop.

For example, the learning project forced me to decide between different watermark strategies. Even though I knew what watermarks were from my distributed systems classes at university, I did not have a deep and thorough understanding of them. The problem forced me to read up on it in more detail in order to be able to make a decision. I implemented a bounded-out-of-orderness watermark and asked Claude Code to review my approach. The tool asked me what would happen if one partition stalled entirely. I hadn’t considered that. It forced me to find a solution. This is something that normal reading or coursework cannot do.

The Setup

Here’s how the system behind that interaction works. I set up a CLAUDE.md which describes a learning project and different stages so the coding agent can guide me along the way.

I instruct Claude Code to help me in certain areas (boilerplate) but to challenge me in others. This way, Claude Code can generate a learning environment for me to build out exactly those parts that help me learn fastest. The most counterintuitive part of the setup is telling a coding tool to actually refuse to code.

# Learning Project

## Purpose
This is a deliberate learning project for [here comes a list of learning goals]. 
The goal is NOT to produce working code fast. The goal is for the developer to build
genuine understanding by writing the hard parts themselves.

## Current Stage
Stage: 1

Do not help with any stage beyond the current stage number above.

## Stage Definitions

### Stage 1 — Basic Windowed Aggregation
[...]

### Stage 2 — Event Time and Watermarking
[...]

### Stage 3 — Stateful Fraud Signals
[...]

### Stage 4 — Kill Tests
[...]

### Stage 5 — Latency Budget Under Pressure
[...]

## Protected Areas — Never Implement These
Even if asked directly, never write implementations for:
- [list of forbidden areas and concepts]

If the developer asks you to implement any of the above, respond with
one Socratic question that identifies the conceptual gap. Do not write
code. Do not hint at the answer.

## When the Developer is Stuck on a Protected Area
1. Ask one question to identify what they think should happen
2. Name the relevant concept or class — do not explain it in full
3. Wait for them to try again

## When the Developer Shares Working Code in a Protected Area
Review it for production risks only. Specifically check:
- [some common mistakes and failure modes]

Respond with risks as questions: "What happens if...?"

## What You Can Always Do Without Restriction
- Generate boilerplate, Docker config, pom.xml, serializers, Kafka
  source/sink configuration, model classes, build files
- Explain concepts when asked
- Review working code for production risks
- Answer questions about concepts

Apart from the more general instructions above, I configured specific commands for Claude Code which I then used at specific points in time during my learning process.

The “Review” command

This command lets Claude Code review what I just implemented. Because it's configured with specific things to check for, it asks follow-up questions and guides me towards a better solution.

The developer has implemented: $ARGUMENTS

They believe it works. Find what would break in production.

Check for:
1. [list of important concepts and caveats]

Do NOT rewrite their code. 
For each risk, ask: "What happens if...?"

The “Stuck” command

This command makes me "unstuck” when I run into problems by asking leading questions and giving hints instead of coding out the solution.

The developer is stuck on: $ARGUMENTS

Do NOT provide a code solution.

1. Ask them to describe what they think should happen in plain English
2. Ask what they tried and what the actual behavior was  
3. Identify the specific concept gap — explain only that concept, not 
   the implementation
4. Name the relevant Flink API class or doc section

Never write the implementation.

The “Predict” command

This command forces me to predict what will happen during a failure scenario before I run it, then helps me reconcile any gap between prediction and reality. The below example is specific to the technology I was learning about (streaming systems).

The developer is about to run Kill test: $ARGUMENTS

Ask them these four questions. Do not continue until all are answered:

1. What state exists right now, and where does it live?
2. What was the last checkpoint, and what does it contain?
3. What events have arrived since the last checkpoint?
4. What do you expect the output topic to contain after recovery?

Once they answer all four, say only: "Run it."

After they report the outcome, help them reconcile any difference 
between their prediction and reality.

Future Plans

I plan on developing this out into a ready-to-use template for future learning projects which I will open source. I will keep you updated on the progress over the following weeks.

#1 The Autonomy Slider

When building autonomous (AI) agents, we have control over the level of autonomy we give those agents. A self-driving car, for example, might assist you in driving around corners, parking sideways or changing lanes. You might also outsource the entire job of transportation to it: bring me from A to B.

Now, the idea of an autonomy slider (as Andrej Karpathy calls it in the talk linked below) is exactly that: giving users a choice over how autonomous AI agents should be. I mostly use Cursor for AI-augmented development and I do so on multiple “levels”: using code completion features to write/refactor individual functions all the way up to using the chat interface to write bigger chunks of code (the autonomy slider is turned up fully). This can even be extended by using tools like RooCode, which fully automates feature development including design and testing.

Now, I don’t want to argue in favor of one autonomy level over the other in coding; I use the example to illustrate autonomy sliders. Andrej stresses the importance of including autonomy sliders when building useful AI agents.

Check out the talk here:

#2 The Slow Productivity Engineer

There are few authors who have influenced by thinking as much as Cal Newport. In his newest book, Slow Productivity, he defines pseudo-productivity as “using visible activity as a crude proxy for actual productivity." Being the dominant way we think about knowledge work since the 21st century, he states that

Long work sessions that don't immediately produce obvious contrails of effort become a source of anxiety--it's safer to chime in on email threads and "jump on" calls than to put your head down and create a bold new strategy.

Instead, Newport defines an alternative: Slow Productivity. He argues that by focusing on different things, we actually increase the outcomes of the effort we put in.

A philosophy for organizing knowledge work efforts in a sustainable and meaningful manner, based on the following three principles:

1. Do fewer things.

2. Work at a natural pace.

3. Obsess over quality.

In InfoQ’s Engineering Culture Podcast, Holly Cummins made some remarks about productivity metrics and software engineering that reminded me a lot of Newport’s ideas.

#3 A Surprising Book on Vibe Coding

What do artists like Neil Young, U2, Metallica, Red Hot Chili Peppers, AC/DC, System of a Down, Eminem and many more have in common? Every one of them had at least one record produced by Rick Rubin.

As I enjoyed his book The Creative Act (get the hardcover, the binding is great), I was surprised to learn that he teamed up with Anthropic to write a book about vibe coding. Check it out here: https://www.thewayofcode.com/

#1 Are LLMs inhibiting human intelligence?

Your brain grows by solving hard problems and struggling with them. That’s the point of neuroplasticity.

For some time, I’ve been worried that my extensive use of LLMs in problem solving challenges inhibits my ability to learn and prevents my skills and knowledge from adapting and improving. Evidence now seems to be emerging about this, indicating that we need to think more about when to use those tools to preserve cognitive ability.

A study I’ve discovered this week states the following:

AI tools, while valuable for supporting performance, may unintentionally hinder deep cognitive processing, retention, and authentic engagement with written material. If users rely heavily on AI tools, they may achieve superficial fluency but fail to internalize the knowledge or feel a sense of ownership over it.

What’s also interesting about this study is that it shows diminishing ownership over LLM-assisted text. I was wondering whether this could extend to development teams. Could teams using AI for coding extensively feel less ownership about it? This would be an interesting research question.

#2 The GitHub Monolith

When watching Gergely Orosz’ interview with GitHub CEO Thomas Dohmke, one thing stuck out to me: GitHub is still operating mostly on a monolithic system. I would be interested in digging deeper into how they do this reliably at such a scale.

When I teach Microservices I emphasize two main criteria which should make you think about including deployment boundaries: (1) supporting varying load patterns (i.e. part A has to scale at a different rate/frequency than part B) and (2) improving team independence (i.e. teams want to use different technologies and deploy independently from other teams). GitHub seems to have done this by splitting out the Copilot API (which clearly has different load patterns than the main GitHub platform) and GitHub Actions.

Anyway, check out the full interview here:

Although this story is widely disputed, starlings are a magnificent and interesting species of birds. What makes them particularly fascinating is their collective behavior in flocks. Starling flocks contain up to thousands of birds and provide mesmerizing, coordinated flight patterns (called murmurations) - all while individual birds follow simple rules like maintaining distance, aligning with neighbors, and steering towards the flock’s center.

What is so intriguing about these phenomena is that they work without central control, relying instead on communication between individual birds. To bring these ideas back to software systems, today I want to discuss a concept called choreography.

Choreography

Imagine your starling birds now as microservices. Choreography is a communication strategy that integrates parts without requiring central control. Individual services execute a workflow by transmitting information to each other in a decentralized manner. This is done by emitting events, which are notifications of something that happened, without knowing who will handle them. I have written about events in “Event-Driven Communication and Its Effects on Team Independence”:

Events are statements of facts that occurred. The emitting module releases an event, letting others know something happened, like "new order placed" or "order canceled". They never induce a direct reply because replying to statements of facts would not make much sense. Events have an effect that commands or queries do not. The producer has no knowledge of the receiving modules, their behavior, or what they do with the data. It releases the event and considers its job done, reducing the coupling between modules. Events are often used in combination with the publish-subscribe pattern. […] Usually, any module interested in a specific type of event can subscribe to it.

What happens in choreographed workflows is that the overall workflow emerges as a pattern of independent reactions between services, rather than being explicitly defined in a single place. Because each service's activation logic is embedded locally, the overall workflow behavior becomes distributed across services.

While this emergent behavior seen in choreographed systems and starling flocks benefits the autonomy of individual services, the lack of explicit workflow logic representation in a single location can sometimes cause issues, such as errors that are hard to trace. This is why an alternative approach to choreography exists, which is known as orchestration.

Orchestration

Placing workflow logic within a single microservice, which then controls the invocation of other services, is known as orchestration, and the central microservice is correspondingly called an orchestrator.

While the invoked services have no knowledge about the orchestrator, the other way around is (obviously) the case. In this way, orchestrators are able to centralize workflows and complex invocation logic - with all the trade-offs this creates.

Conclusion

Clearly, neither choreography nor orchestration is inherently and objectively superior. As I personally lean towards choreography, especially when the communication crosses teams and bounded contexts, I acknowledge that orchestration provides value in depicting workflows which require some tasks to be executed in a certain order. However, I always try to build my module boundaries around these consistency/order requirements such that I use orchestration more often within those bounded contexts. This way, the communication overhead that comes with extending orchestrated systems does not cross team boundaries and the knowledge of the process coordination remains local and within the same team.

Organizational distance describes how far apart two teams are within a communication network. Teams sharing the same office space and interacting daily have a small organizational distance. In contrast, teams situated in separate units—or even different organizations—and located on different continents have a large organizational distance.

Measuring Organizational Distance

Quantifying organizational distance precisely is challenging, but it’s possible to get an intuitive sense of it by considering a few key questions:

How often do you interact with the other team?
Frequent interactions, such as daily conversations, indicate a smaller organizational distance. Less frequent communication typically suggests greater distance.

How much physical or digital space do you share with the other team?
Sharing more space generally reduces organizational distance. Physical space includes working on the same office floor, using common interaction areas, or having shared lunch spaces. Digital spaces might involve using shared chatrooms, forums, or collaboration tools. Typically, physical interaction implies a smaller organizational distance compared to purely remote interaction.

How much cultural overlap exists between your team and the other team?
Teams with significant cultural overlap often find it easier to develop a common understanding of problems and solutions. Conversely, cultural differences, particularly when teams are spread across multiple time zones or geographic regions, usually increase organizational distance.

The organizational distance between teams significantly influences the type of integration they should select. The figure below illustrates that cross-team integration has multiple dimensions, represented here along three distinct axes:

Technical Axis

Synchronous, direct communication (e.g., calling a REST interface) is most suitable for teams with close organizational distance, as they can more easily collaborate on evolving their APIs together. For teams separated by greater distance, techniques like API versioning can help manage changes more effectively.

In contrast, asynchronous and indirect communication—often implemented using middleware—decouples the runtime behavior of interacting components. Asynchronous messaging reduces coupling by using flexible, immutable event schemas or backward-compatible message formats and by eliminating the need for immediate responses. This approach allows each team evolve its components more independently, increasing autonomy and minimizing mutual disruptions.

Domain Axis

Domain-Driven Design outlines several integration patterns that are strongly influenced by organizational distance.

Patterns such as Shared Kernel or Partnership are most suitable for teams with close organizational ties, due to the tight coordination and shared understanding required. In contrast, teams separated by larger organizational distances benefit from patterns like Open Host Service or Anticorruption Layer, as these explicitly introduce translation layers that manage differences between the domain models of each team.

Interaction Axis

The book Team Topologies introduces clear interaction modes to describe team relationships. For example, the collaboration mode implies close organizational distance and is typically recommended only for short-term, intensive interactions. In contrast, the X-as-a-Service mode supports a more stable, long-term relationship and is better suited to teams separated by larger organizational distances.

Conclusion

Teams with shorter organizational distances typically benefit from synchronous, direct integration patterns combined with close collaboration. Conversely, teams separated by greater organizational distances should prefer asynchronous communication patterns and explicit domain-model translations, such as an anti-corruption layer:

In general, a larger area in the diagram indicates greater organizational distance. Therefore, when integrating with a component owned by a distant team, it is advisable to avoid patterns in the center of the graph, instead favoring those toward the outer layers.

So I wanted to use this week’s article to make up for this. Here are four books I think you should read:

Team Topologies: Organizing Business and Technology Teams for Fast Flow

This book applies agile principles to organizational design. It introduces four different team types and interaction modes, which includes stream-aligned teams and platform teams.

Read this book if you are interested in organizational aspects influencing software architecture or if you are currently in a role where you drive organizational change.

Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations

Written by members of the DevOps Research and Assessment (DORA) team, this book includes powerful insights into the delivery performance of software development teams. They not only write about hypothetical scenarios but present a wide array of empirical data to support their arguments. One of the most prominent concepts from the book are the four key metrics: Deployment Frequency, Lead Time for Changes, Change Failure Rate, Time to Restore Service

Read this book if you are in senior technical roles and want to advance your organization’s value delivery.

Building Evolutionary Architectures: Automated Software Governance

This book not only introduces the powerful idea of architectural fitness functions but also discusses useful patterns and strategy to build systems which can evolve long-term in changing contexts.

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems

This book is one of the most-read books for building highly scalable, distributed information systems. Topics include distributed database clusters, data streaming and message-oriented communication.

Read this book if you are in any way involved in building distributed systems, especially with high reliability or performance requirements.

When a system’s components become too interdependent, even a small change can have wide-ranging effects. For example, if a modification in one component triggers changes in several others, it signals architectural entanglement. This issue is especially problematic when different teams are responsible for different components, because any change requires extensive communication and coordination across teams, disrupting priorities and backlogs. Such complexity not only increases technical overhead but also adds organizational and communication burdens, ultimately resulting in longer lead times and a higher likelihood of errors.

[…]

In software architecture, autonomy means that a system’s component can operate and evolve independently, reducing the need for frequent (inter-team) coordination.

Autonomy is not binary; it exists on a spectrum from total dependency to complete independence. Instead of viewing autonomy as something you either have or do not have, individual entities should always strive to become more autonomous. Therefore, if your team is working on a component that ranks low on the scale, you should seek ways to increase its autonomy rather than accepting its dependency.

In the article, I also outlined several distinct aspects of component autonomy, summarized here:

• Functional Autonomy: Each component should have a clearly defined and distinct responsibility, ensuring its functionality is clearly separated from other system parts. This minimizes coordinated changes across components.

• Data Autonomy: Components should independently manage their internal data without relying on shared mutable data or schemas, preventing data-driven changes from cascading across components.

• Runtime Autonomy: Components should operate independently at runtime, remaining functional even when related services become temporarily unavailable. They should include mechanisms enabling continued operation without immediate external input.

• Activation Autonomy: Component with high activation autonomy decide independently when to perform actions instead of depending on external triggers or orchestrators. This self-driven behavior enhances responsiveness and reduces external dependency.

• Scaling Autonomy: Autonomous components can handle increased loads independently without external bottlenecks limiting their performance. Their scalability primarily depends on their own design and allocated resources.

• Deployment Autonomy: Teams should be able to update or redeploy their components independently, without requiring synchronized changes in other system areas. This autonomy supports frequent releases and minimizes system-wide disruptions.

• Solution Autonomy: Solution autonomy allows teams responsible for a component to independently choose technologies, tools, and strategies. This freedom fosters innovation and rapid adaptation to changing business needs.

After discussions with some of you who’ve sent me feedback, I have added another aspect - you’ll find it in the original article but I also will summarize it here:

Infrastructure Autonomy

Component teams with low infrastructure autonomy rely on manual work by other teams to provision infrastructure they need. Conversely, teams with high infrastructure autonomy can independently provision and manage their infrastructure, often supported by platform or infrastructure teams providing self-service capabilities. Teams with high infrastructure autonomy rarely trigger additional manual tasks in other teams, as they utilize provided self-service infrastructure tools:

As part of my regular architecture reviews with clients, I pay special attention to component autonomy. To facilitate this, I've developed heuristics to assess component autonomy systematically. In this article, I share these heuristics, structured as key questions designed to help you evaluate autonomy across all aspects. Throughout the listing, the component being evaluated will be referred to as "Component A".

Functional Autonomy

When you make changes to Component A, do you typically need to adjust other components to maintain the overall correctness of the system? If so, how many components are affected?
A high number of affected components indicates low functional autonomy. If you cannot modify Component A without simultaneously adjusting and redeploying other components, it suggests that Component A's functional boundaries are not well-defined. This issue might arise from unclear or overlapping responsibilities or from components bypassing clearly defined interfaces and directly depending on Component A's internal implementation.

Do changes in Component A often cascade into larger system modifications, suggesting a lack of clear boundaries in functionality?
If changing Component A frequently requires adjustments to multiple other components, this signals unclear functional boundaries and a higher level of architectural coupling. Ideally, modifications should have a limited impact radius, affecting as few additional components as possible.

Data Autonomy

When modifying the data stored by Component A or its associated schema, do you also need to adapt or update other components?
If the answer is yes, it suggests that Component A relies on shared mutable data, significantly reducing its autonomy. Dependence on shared mutable data typically requires synchronized changes and coordinated deployments across multiple components to prevent system errors. Ideally, components should manage their own data independently, minimizing cross-component dependencies.

Runtime Autonomy

When a related service or dependency becomes (temporarily) unavailable, can Component A continue operating effectively?
To assess this, first identify all components that Component A communicates with, including data providers, synchronous and asynchronous services, orchestrators, and data sinks. Next, evaluate how Component A behaves when each of these components is unavailable. Can it still function normally and continue processing requests without immediate external input?

Discuss these scenarios with your team or leverage Chaos Engineering (TODO link https://en.wikipedia.org/wiki/Chaos_engineering) techniques to simulate and test component resilience under failure conditions.

Activation Autonomy

Does Component A independently decide when to act, or does it rely on external triggers?
Assess the conditions under which Component A initiates tasks. If other components control or heavily influence when Component A executes its tasks, this indicates lower activation autonomy.

When changes occur in Component A, how many other components require modifications?
Frequent simultaneous updates to other components alongside Component A suggest lower activation autonomy. This happens because other components have excessive knowledge of, or dependency on, Component A's execution logic, increasing the system’s coupling.

Do many of your team's tasks depend on changes in components beyond Component A?
If the internal logic or interfaces of other components frequently trigger changes within Component A, it demonstrates a lack of isolation in activation logic, further reducing activation autonomy.

Scaling Autonomy

Are there other components (or infrastructure) that have the same scaling behavior as Component A?
If other components share the same scaling pattern as Component A, it indicates they must handle similar throughput levels. This dependency can become problematic if any of these components cannot meet the required throughput, potentially creating bottlenecks and limiting Component A’s ability to scale independently.

Does Component A perform non-business-related actions (like committing transactions or cleaning up data) solely to maintain consistency?
These actions suggest there may be cross-component consistency constraints, especially in distributed systems. Such constraints typically introduce additional overhead, requiring components to perform extra work that can impede their independent scalability.

Deployment Autonomy

When deploying Component A, do you have to deploy other components at the same time? Does deploying Component A require stopping, restarting, or redeploying other components?
Consider the extent to which deploying Component A affects other parts of the system. If deployment of Component A frequently involves modifying or coordinating with other components—or requires collaboration with other teams—this indicates lower deployment autonomy.

Does your team have the authority and flexibility to update and release Component A independently, without obtaining approval or coordination from other teams or central entities?
Frequent and efficient deployments depend heavily on a team's autonomy. Cross-team communication and coordination can significantly slow down deployments and introduce unnecessary overhead.

Infrastructure Autonomy

Does the need for infrastructure changes by Component A induce manual work in other teams?
If infrastructure changes for Component A routinely depend on manual tasks from other teams (such as platform or infrastructure teams), this indicates low infrastructure autonomy. For example, if Component A requires new data storage, low autonomy would mean another team needs to provision it manually. In contrast, high infrastructure autonomy would allow your team to directly self-serve and quickly integrate pre-configured infrastructure resources, minimizing dependency and delays.

Solution Autonomy

Does the team responsible for Component A have the freedom to adopt technological solutions or make decisions about Component A’s design and evolution without needing extensive coordination with other teams?
Two key aspects influence this: First, Component A must be architecturally isolated enough to enable technological independence, allowing the team to freely select programming languages, databases, or frameworks. Second, the organizational culture should empower teams by granting them sufficient freedom to explore, choose, and implement their own solutions, rather than strictly regulating the technologies and processes they use.

Summary

Below, you’ll find all aspects of autonomy and their associated review questions:

When a system’s components become too interdependent, even a small change can have wide-ranging effects. For example, if a modification in one component triggers changes in several others, it signals architectural entanglement. This issue is especially problematic when different teams are responsible for different components, because any change requires extensive communication and coordination across teams, disrupting priorities and backlogs. Such complexity not only increases technical overhead but also adds organizational and communication burdens, ultimately resulting in longer lead times and a higher likelihood of errors.

To mitigate these challenges, it is essential to design systems where each component is as autonomous as possible. This lays the groundwork for long-term adaptability, evolvability, and reliability.

What is Autonomy?

Autonomy comes from the greek words autos (self) and nomos (law). It refers to an entity’s ability to make its own decisions and govern its actions independently. In other words, when an entity is autonomous, the decisions of others do not directly affect its behavior. For example, if one entity acts and another is forced to respond immediately, then at least one of them is not fully autonomous. (strong emphasis on the words “forced” and ”immediately”. If another entity has chosen to react and can do so eventually and on their own terms, it is still autonomous)

In software architecture, autonomy means that a system’s component can operate and evolve independently, reducing the need for frequent (inter-team) coordination.

Autonomy is not binary; it exists on a spectrum from total dependency to complete independence. Instead of viewing autonomy as something you either have or do not have, individual entities should always strive to become more autonomous. Therefore, if your team is working on a component that ranks low on the scale, you should seek ways to increase its autonomy rather than accepting its dependency.

Let's consider the following example (shown in the figure below): Component A is called by an orchestrator. Whenever the orchestrator deems it necessary to call Component A, it will decide to do so (1). Once invoked, Component A contacts Component C because it needs C to update its internal state (2). Component A can only report back after confirming that Component C’s task is complete. In addition, Component A reads data from a data schema that it shares with Component B (3).

How autonomous is Component A? Not very much. Consider these issues:

• Component A does not decide when to be invoked; the orchestrator makes that decision.

• Component A depends on Component C being available to respond to the orchestrator.

• Component A can only scale to the degree that Component C can.

• If Component B changes the shared data schema, Component A’s code must be adapted and redeployed.

Aspects of Autonomy

It is not only the case that autonomy isn’t binary, it’s also multi-dimensional. Rather than being defined by a single attribute, autonomy encompasses multiple aspects that capture its various facets. Let’s look at each in turn.

Functional Autonomy

Components must have clear responsibilities and well-defined boundaries that separate them from one another. When these boundaries are fuzzy or unclear, components end up sharing responsibilities, which means that changes in one component often force changes in others.

The example on the left shows low functional autonomy. Component B is shared across contexts, meaning that both rely on the same domain model contained within the component. In domain-driven design, this is called a shared kernel. Although shared kernels can be advantageous in some cases, they inherently hinder functional autonomy. This is especially problematic when two different teams are responsible for the two bounded contexts. In such situations, implementing shared kernels requires high communication bandwidth between teams—a significant drawback if the teams are separated by a large organizational distance (I have written about this in “Connecting Modules Across Teams: Selecting Integration Mechanisms Based on Team Distance”).

The example on the right has clear boundaries of responsibility. In this scenario, the shared Component B is replaced by an explicit integration point between the two bounded contexts, resulting in clear boundaries of responsibility.

Here are some strategies to increase functional autonomy:

• Components should expose only what is necessary through explicit interfaces while keeping internal details hidden. Interaction between components should occur solely through these interfaces, with no side channels. This interface might take the form of an API or use an intermediary such as a message queue. If the integration crosses deployment boundaries, I recommend using asynchronous and indirect communication, as I have arguerd in "Network Communication And Leaky Abstractions".

• A component should be as small as necessary to fulfill its purpose. This idea aligns with the Unix philosophy of “do one thing and do it well” and is reflected in the Single Responsibility Principle.

Data Autonomy

Shared data often undermine a component's autonomy—especially when components rely on a common, mutable data schema. When multiple components depend on a shared schema, any changes to that schema force every dependent component to adapt. To enhance data autonomy, ensure that each component manages its internal state independently, without relying on shared data.

One important caveat is that shared data does not harm autonomy if it is immutable. For example, if several components read from an append-only storage—where existing data cannot be altered and new data can only be appended—then data autonomy is maintained. In summary, avoid using shared mutable data or schemas to ensure each component can operate independently.

Runtime Autonomy

Consider two scenarios involving separately deployed services that communicate over a network (both are depicted in the figure below): In the example on the left, the order service makes a synchronous call to the inventory service before approving an order. This means the order is approved only after the inventory has reserved the products. In this case, the order service needs the inventory service to be available during its operation, resulting in lower runtime autonomy.

In the example on the right, the order service informs the inventory service about an incoming order asynchronously—using an event, for example. Once the event is committed to the middleware, the order service continues processing without blocking and waiting for a response. If inventory is insufficient, the inventory service can asynchronously notify the order service, which then decides how to proceed. Here, the order service does not depend on the inventory service’s availability for the transaction, which significantly increases its runtime autonomy.

You can increase runtime autonomy by applying a few strategies:

• Deploy Components Independently: Ensure components run as separate processes. If components share the same process, they inherently lack runtime autonomy.

• Decouple the runtime behavior of components by employing asynchronous communication such as notifications, state transfers, or event sourcing. I have detailed out these approaches in “Event-Driven Communication and Its Effects on Team Independence”.

• Handle Responses Optimistically: Avoid waiting for immediate answers from dependent services. Instead, resolve issues optimistically. For example, if the inventory is insufficient, the system might automatically trigger a restock or, alternatively, the order service could notify the user of a potential delay.

Activation Autonomy

High activation autonomy means that a component should determine on its own when to act rather than relying on other components to decide when it is invoked. In other words, only the component itself should know when specific actions need to be performed.

This approach contrasts with orchestration patterns, where an orchestrator is responsible for triggering actions in other components. Contrary, in systems designed with high activation autonomy, each component reacts to environmental cues and decides independently whether to take action.

Consider an example: suppose you need to build a workflow where bookings over $1,000 must be reported. One component processes the bookings, and a separate component handles reporting.

Low Activation Autonomy (Left Example):

Here, the reporting component depends on both its internal logic and on an external orchestrator to determine when it should run. As a result, any change in the activation logic requires modifying both the reporting component and the orchestrator.

High Activation Autonomy (Right Example):

In the right example, all the knowledge about reporting sits within the reporting component. There is no need for coordination and one singular point of change. The difference is that communication over self-contained facts that are modelled after the domain (events) gives the receiving component the power to decide whether they need to take action. The activation logic is isolated within the component instead of distributed over multiple components, giving it activation autonomy.

Scaling Autonomy

In “Beyond Linear Scaling: How Software Systems Behave Under Load”, I discussed factors that limit the scalability of components. Two factors are particularly relevant in this context: The first one are bottlenecks or resources shared with other components, which can inhibit the throughput of each individual component. Another is the additional work that a system needs to perform to maintain consistency and coherency. As I wrote in the article:

In certain systems, adding more load can actually decrease throughput. Let's examine the following system:

The three subtasks each have their own data storage, either persistently through separate data stores or ephemerally via caches. Let's assume the system must maintain consistency to respond correctly. Imagine one subtask updating data in its own storage; the system must then propagate these changes to other storages using a transaction.

This has significant consequences. Not only do the subtasks need to stop responding until a consistent state is restored, but the process of re-establishing this state requires additional work before the system can resume service. This overhead negatively affects throughput: as more load is added, the throughput actually decreases—a phenomenon often observed in real-world throughput metrics.

If a component depends on other components to scale up its throughput—whether through resource contention, bottlenecks, or the extra work needed for consistency—it exhibits low scaling autonomy.

Here are strategies to increase scaling autonomy:

• Lower resource contention by providing sufficient resources and enable components to acquire the resources they need through self-service interfaces (for example, using cloud infrastructure).

• Minimize coherency overhead by avoiding to enforce strict consistency across components when possible, as doing so introduces extra work and tightly couples components’ scaling behaviors.

Deployment Autonomy

High deployment autonomy is evident when teams can deploy components independently without impacting other deployments. For example, if deploying Component A requires a team to stop, restart, or redeploy Component B, the system exhibits low deployment autonomy. This low autonomy forces teams into frequent, error-prone coordination and hinders their ability to deploy updates on their own schedules.

Solution Autonomy

One factor often overlooked when focusing solely on technical details is the autonomy of the team responsible for a component. In “Designing Effective Cross-Team Dynamics For Platform Teams”, I argued that team autonomy is one of the three pillars for building successful and motivated teams:

Adopting a team-first perspective is essential for resolving these challenges. We need to focus on designing valuable and productive interactions between platform and application teams. A useful framework for this is outlined in Steven Pinker’s book “Drive: The Surprising Truth About What Motivates Us”. Pinker identifies three key factors that motivate individuals and teams:

Autonomy: The ability to make decisions, reflect on them, and adapt as needed.

Mastery: The opportunity to perform work and receive feedback to improve skills.

Purpose: The sense of working towards something achievable and worthwhile.

[…]

Autonomy

Just as I prefer choosing my own route to a destination, autonomous teams need control over their solution space and the authority to make decisions within it. They must also be able to gather and incorporate both external and internal feedback, as well as prioritize their work within their areas of responsibility.

Solution autonomy describes the socio-technical dynamic in which teams can decide how to build, test and deploy the components they are responsible for. This means to fully empower development teams to build each component using appropriate patterns, technologies, tools, and programming languages—rather than being constrained by a pre-defined set of rules imposed by external architecture or governance teams.

Importance of Aspects

Not all aspects of autonomy carry the same weight; development teams must determine which are more or less important in their context. For example, runtime, scaling, and deployment autonomy are essential when prioritizing reliability, while functional, data, and solution autonomy are key to ensuring long-term evolvability.

It is important to consider all autonomy aspects because autonomous components can make your system more dynamic in the future. Make clear trade-off decisions only if certain aspects play only a minor role in your context (which is rare). For instance, you might decide, “We trade off invocation autonomy for the significant benefit of monitoring orchestrated workflows.” Keep in mind that such trade-offs can incur substantial costs in your system’s evolvability and long-term adaptability across many dimensions. I have written about some of those dimensions in “Reactive Systems: Designing Systems for Multi-Dimensional Change”.

Improved Example

In the example at the beginning of the article, I have outlined reasons for low autonomy. Having introduced some vocabulary, we can now map those issues to the autonomy aspects:

• Component A does not decide when to be invoked; the orchestrator makes that decision. (low activation autonomy).

• Component A depends on Component C being available to respond to the orchestrator. (low runtime autonomy).

• Component A can only scale to the degree that Component C can. (low scaling autonomy).

• If Component B changes the shared data schema, Component A’s code must be adapted and redeployed. (low data and deployment autonomy).

To improve the autonomy of both Component A and Component C, I made the following design adjustments:

(1) Increase Activation Autonomy: Component A now independently decides which system events warrant a reaction.

(2) Increase Runtime and Activiation Autonomy: Instead of waiting synchronously for a response from Component C, Component A now notifies C about the necessary changes and then handles responses asynchronously. This adjustment boosts Component A’s runtime autonomy and also enhances Component C’s activation autonomy.

(3) Increase Data and Deployment Autonomy: Components A and B now maintain redundant data storage and separate data schemas. They communicate necessary state transitions through a message queue, which increases both components’ data and deployment autonomy.

Conclusion

Autonomy is not guaranteed. Autonomy is something you need to buy. And I mean that in the literal sense, because there is a cost. For example, achieving data autonomy might require additional storage for redundant data, and increasing activation autonomy could reduce the convenience of explicit, sequential monitoring. Every design decision involves trade-offs, and your task is to balance these within your context.

My advice is simple: increase the autonomy of each component as much as possible, and only compromise when absolutely necessary. I have seen too many systems become brittle over time because developers and architects failed to consider autonomy. If you aim for long-term adaptability, evolvability and reliability, ensure to take all aspects of autonomy into account.

Service A sends a request (1) and then waits for Service B’s response. Service B processes the request (2) and sends back a response (3). Then Service A receives the response and resumes (4). In a local call, the CPU jumps directly to the function’s memory address. Over a network, though, messages travel over unreliable links. They might be delayed, dropped, or even lost if Service B is down.

Synchronous network calls mask network issues by mimicking local procedure calls. This is a leaky abstraction: the network’s unreliability eventually shows up in your code. If Service B is unavailable or slow, Service A must handle retries, timeouts, or implement circuit breakers. This adds complexity and creates tight runtime coupling between services.

Instead, asynchronous and indirect communication makes network properties explicit. With asynchronous communication, Service A sends a request and continues processing other tasks, handling the response later through a callback. Indirect communication means that neither the caller nor the receiver is directly connected; they interact through an intermediary, such as a message queue or append-only storage.

To approach why the networks nature is more explicit in this sort of communication, let’s look at two examples and list out the kinds of assumptions the calling service makes about the underlying network. This is a proxy for the degree of coupling, since it shows how much the calling service needs to now about the network and the called service.

The first example is synchronous and direct communication between services. Like in the figure above, let’s assume that Service A contacts Service B. It has the following assumption about the communication with Service B:

• “I know that Service B exists”

• “I know where Service B is located so I can directly contact it”

• “I will get a response from Service B in time”

If any of these assumptions fail, Service A must deal with the error. Service B is not available? Service A needs to retry the request. Service B takes a long time to respond? Service A needs to track a timeout. Either way, there is a lot of burden on Service A to handle fallback mechanisms. Network failures leak into the implementation of Service A, increasing its complexity and its coupling to Service B.

Now, let’s look at asynchronous and indirect communication, more specifically a request/reply pattern. This discussion similarly applies to other forms of messaging, like publish/subscribe or event mechanisms.

Here are the assumptions that Service A has about the communication with Service B:

• “I send a message to Message Queue 1 and someone will handle it.”

• “Eventually, I will receive a response on Message Queue 2 which my internal callback will handle.”

There are much less assumptions which leak failure modes into the implementation of Service A. Since it is already implemented in a way where it handles responses eventually, the unreliability of the network is already baked into the design of the communication.

Using this form of communication forces you to design for failure. It includes all failure modes of network connections as first-class in the design of the communication protocol. Designing a service that sends requests and handles responses asynchronously means that you are forced to design the service in a way where responses come back at an arbitrary time and the service should be able to handle it. This mitigates potential network failures like increased latency or dropped connections and puts solutions to those failure modes in the design of the communication.

Another interesting example where asynchronous communication shines is that it helps to decouple individual services over the dimension of time.

(De-)coupling in Time

Think about cooking soup. Some tasks, like cutting onions and heating water, can happen simultaneously, but other steps must occur in a strict order, like adding ingredients only after preparation is complete. Similarly, software systems are often built in a way where certain parts/routines need to be executed before other parts can be executed. We call this temporal coupling. An example I witnessed recently was a data pipeline that stored data in a shared database and another pipeline read out of that shared database. Both pipelines were coupled temporally because one pipeline needed to execute before the other one.

Similarly, synchronous communication over networks always leads to temporal coupling. When Service A call Service B, it is dependent on the latter’s timing and availability. If Service B is down or responds slowly, due to the synchronous call’s blocking nature, Service A has to wait and delay potentially waiting upstream services.

The goal should be to design distributed systems where individual services can operate independently over time, instead of coupling them by relying on certain execution orders. Using asynchronous communication helps here. As I have mentioned above, designing systems in this way better reflects the reality of potentially unreliable network communication.

Let’s look at an example. Say you have an order service which checks an inventory service to check for availability of products before confirming the order to the user. The order service looks like this internally:

function processOrder(order) {
  // Synchronous call: wait for the response
  const inventoryStatus =   inventoryService.checkAvailability(order.items);
  
  if (inventoryStatus.available) {
    console.log('Inventory available, processing order.');
    // Continue processing the order
  } else {
    console.log('Item out-of-stock, notifying customer.');
    // Handle out-of-stock scenario
  }
}

With asynchronous communication, we again use an intermediary. Let’s implement two message queues in this example, one for sending requests and one for responses:

The order service uses a callback which is being invoked as soon as a response is sent over the responding message queue:

function processOrder(order) {
  // Asynchronous call: continue processing without waiting
  inventoryService.checkAvailabilityAsync(order.items, (inventoryStatus) => {
    if (inventoryStatus.available) {
      console.log('Inventory available, processing order.');
      // Continue processing the order
    } else {
      console.log('Item out-of-stock, notifying customer.');
      // Handle out-of-stock scenario
    }
  });
  
  // Other independent tasks can be performed here without waiting
  console.log('Order received; processing will continue once inventory is confirmed.');
}

The second example has lower temporal coupling since the order service has no expectations about the availability of the inventory service. All it is concerned about is sending requests to a message queue and receiving responses from another one which invokes its callback.

Drawbacks of Asynchronous Communication

While this post advocates for asynchronous communication, it’s important to note that it comes with its own challenges—such as managing message ordering, debugging flows, and handling eventual consistency.

I don’t want to convince you to use asynchronous and indirect communication in every case but to use its strength in making network properties more explicit in the system’s architecture. One rule of thumb I have had good experience with is to incorporate domain boundaries: using asynchronous, indirect communication when communicating across bounded contexts (usually event-driven communication) but being more open to synchronous, direct communication within bounded contexts. If you want to know more about bounded contexts, check out my post “A Case Against 'One Model to Rule Them All’”.

By designing with asynchronous, indirect communication, you build systems that naturally account for network unreliability, reduce tight coupling, and allow services to operate independently—making your distributed architecture more robust, scalable and adaptable.

Although this example might seem trivial, it illustrates an interesting dynamic also present in software development: certain teams provide infrastructure and operations (the Autobahn) for other teams, who then choose to utilize—or bypass—this infrastructure to achieve their specific goals.

We typically distinguish between two types of teams. There are teams that produce code that represents the end user’s domain. In this post, we call these sorts of teams application teams. In contrast, there are teams that produce an environment for application teams to operate in, most often by providing the infrastructure needed to execute their code. In a sense, they are the teams building and maintaining the highway, while application teams are responsible for transporting people efficiently. We call the second type of team platform teams.

When Platform Teams Become a Bottleneck

The dynamics between application teams and platform teams are often complex. While in some smaller contexts, application teams build and operate their software solutions without needing other teams to help, they usually develop their software on some platform - be it on a cloud platform, embedded platform or operating system. The responsibility of platform teams is to manage these platforms.

Platform teams can be internal (within the same company) or external, such as when application teams utilize a public cloud platform. For the remainder of this post, we'll focus on the dynamics between application teams and internal platform teams, though much of the discussion also applies to external ones.

Platform teams play a pivotal role within organizations by delivering central infrastructure, which can inadvertently position them as potential bottlenecks. To prevent this, platform teams must navigate particular challenges and avoid specific pitfalls.

TicketOps

Organizations should avoid delays caused by application teams waiting on platform teams. This often happens when platform teams rely excessively on manual processes to deliver infrastructure and services. For example, an application team may need to release a new service version but must first submit a ticket to the platform team to reconfigure the production environment. Waiting for the ticket to be resolved creates significant friction within the organization. This not only leads to finger-pointing and debates over prioritization and misalignment but also slows down the application team's delivery speed.

This reliance on tickets and manual work for the interaction between application and platform teams is often framed as TicketOps.

Governance

While it is understandable that platform teams fear application teams adopting a wide variety of tools, frameworks, and programming languages—which can lead to a disorganized and unmanageable ecosystem—the primary method used to address this complexity is often governance. Governance typically involves imposing strict rules to limit technological diversity, primarily through safelisting approved tools and technologies.

However, this approach has two significant drawbacks:

First, strict enforcement of current technologies can severely hinder the adoption of new tools and methodologies, disregarding the specific needs of individual teams. This rigidity transforms the platform from a value-adding resource into a burden for development teams, ultimately stifling innovation and limiting the organization's ability to adapt to emerging trends.

Second, teams with unique requirements that do not fit within the governed solutions may find the platform overly restrictive. This can lead them to seek workarounds or wait for platform teams to implement necessary adaptations, which can slow down delivery speed and reduce overall efficiency.

Over-Engineering and Thick Platforms

When platform teams attempt to implement and support a wide array of technologies, infrastructure, and configuration options, they can quickly become overwhelmed by the sheer number of capabilities integrated into the platform. This not only makes the platform harder to use but also increases the burden on platform teams to maintain the complexity, keep everything up-to-date, and respond to a broad range of change requests from application teams.

Addressing Challenges in Team Dynamics

The issues discussed are primarily social in nature, presenting themselves as technical symptoms exacerbated by technical inefficiencies. To effectively tackle these problems, the central question should be: What do application and platform teams truly need?

Adopting a team-first perspective is essential for resolving these challenges. We need to focus on designing valuable and productive interactions between platform and application teams. A useful framework for this is outlined in Steven Pinker’s book “Drive: The Surprising Truth About What Motivates Us”. Pinker identifies three key factors that motivate individuals and teams:

• Autonomy: The ability to make decisions, reflect on them, and adapt as needed.

• Mastery: The opportunity to perform work and receive feedback to improve skills.

• Purpose: The sense of working towards something achievable and worthwhile.

Let’s explore how we can address each of these areas to enhance team dynamics.

Autonomy

Just as I prefer choosing my own route to a destination, autonomous teams need control over their solution space and the authority to make decisions within it. They must also be able to gather and incorporate both external and internal feedback, as well as prioritize their work within their areas of responsibility.

Often, the dynamics between application and platform teams can impede the autonomy of both, especially when responsibilities are divided between development and operations. For instance, application teams building software and then handing it off to platform teams for operation can undermine the responsibility and autonomy of all involved.

Let’s look at two strategies to increase autonomy:

Self-Service

Self-service interfaces should be the primary mode of interaction between application and platform teams. By leveraging self-service tools, development teams can provision the necessary artifacts and infrastructure provided by the platform team—ranging from databases and middleware to container infrastructure, documentation, and entire application/service blueprints. Self-service interfaces have two essential properties: first, invoking the self-service interface does not require or cause manual work from the platform team. Second, application teams manage the lifecycle of the provisioned artifacts, including creation, updating, and deletion. The platform team provides blueprints and tools for the application team to make handling those artifacts easier.

In contrast to the TicketOps approach, where application teams depend on platform teams to perform tasks and wait for ticket resolution, self-service interfaces clearly delineate responsibilities. Platform teams focus on providing a seamless developer experience, while application teams handle their own infrastructure needs using the provided tools - the self-service interface is then merely a tool they use to make their jobs easier.

Team APIs

Team APIs are a valuable tool for making interactions between teams more explicit, thus increasing autonomy. As I have written in “The Power of Team APIs: Streamlining Cross-Team Interactions”:

By formulating a Team API, development teams can publish typical and deliberately designed interaction points for other teams. It contains everything about the team that other teams need to know to contact it and provides context to make interactions more productive. Usually, this means

• describing what the team's responsibility is: what modules or features are they responsible for?

• answering whether the team provides value to end-users or development teams (e.g. as a platform or enabling team)

• listing what the team provides for other teams. This could be technical concerns like an API or social aspects such as consulting.

• stating how to reach the team

• describing what the team is currently working on and what other teams it relies on to achieve its goals.

(Check out the post for an example of a Team API).

Platform and application teams can use Team APIs to define their interactions and establish a clear-cut separation of responsibility. Platform teams specifically should mention what self-services they offer and when application teams should contact them, e.g. to communicate improvement ideas for the platform.

Mastery

Creating exceptional products that teams are proud of and users love is one of the most fulfilling parts of software development. Achieving mastery, both individually and collectively as a team, is crucial for reaching these goals. Here are two approaches to foster mastery within application and platform teams:

Lower Cognitive Load

Cognitive load refers to the amount of information an individual or team can process at any given time. It exists on an individual but also on a team basis. Each team has a limit of complexity they handle reliably and sustainably. Multiple things increase team cognitive load, for example the complexity coming from programming languages and frameworks (also called intrinsic cognitive load), the complexity of tools a team uses to approach tasks such as continuous deployment or monitoring (also called extraneous cognitive load) and the complexity of mastering the domain (also called germane cognitive load).

To enhance mastery, teams should minimize intrinsic and extraneous cognitive load to foxcus on germane cognitive load. Teams should be as good as possible in mastering their domain since this is the area where the users of their product derive the most value from. A team that is heads over in details of their technical setup is much less effective than a team that can focus on writing value-adding business code.

Therefore, both application and platform teams profit from focusing on their domain. For application teams, this is mostly the business domain of the application they are building and for platform teams this is the technical domain of the infrastructure they are maintaining. Further, platform teams can help in lowering the cognitive load of application teams (the extraneous cognitive load specifically).

Your Platform Is a Product

Platform teams should treat the platform they build as a product, with developers as their customers. Just as those maintaining a highway strive to provide the fastest and safest routes, platform teams should aim to maximize the developer experience.

Adopting a product mindset involves

• gathering requirements for the product: engaging with application teams, observing they issues they are facing and looking for opportunities to lower their cognitive load

• developing a product ownership: prioritizing tasks and assigning product ownership roles to ensure the platform evolves in line with user needs.

• gathering and incorporating feedback: Continuously seeking and integrating feedback to improve the platform, as detailed in my post “Course Correction: Implementing Effective Feedback”.

Purpose

Purpose in software development arises from striving towards meaningful goals and recognizing the positive impact on users. Application teams often experience this directly by seeing how their software enhances users' lives. Platform teams, while their users are developers, should also find purpose in making their developers happy and effective. There are two important things that platform teams can keep in mind and use as tools to improve their sense of purpose.

Making developers happy

Platform teams should monitor developer satisfaction using metrics like the Net Promoter Score. This involves regularly surveying developers to gauge their likelihood of recommending the platform to other teams, providing holistic feedback on the platform’s effectiveness and areas for improvement.

Thinnest Viable Platform

Based on the term “minimum viable product”, platform teams should strive to create the thinnest viable platform that delivers value to users. Instead of building a huge platform up-front and only later recognizing that developers don’t want to use it, strive to build a platform iteratively and incrementally: start with the smallest, most impactful features that address user needs, build, gather feedback, and refine the platform incrementally. … rinse and repeat.

By doing that platform teams will find that they are much more effective in building a platform their users want - with constantly and continuously delivering value along the way.

An overview

To summarize the strategies and tools discussed:

Conclusion

Designing effective interfaces between platform and application teams is challenging, but crucial for organizational success. By carefully crafting self-service tools and fostering autonomy, mastery, and purpose, platform teams can transform from potential bottlenecks into powerful enablers. Implementing the strategies outlined in this post will help you move closer to achieving a harmonious and efficient team dynamic, ultimately driving innovation and productivity across your organization.

Further Reading

If you want to learn more about cognitive load and the dynamics between application and platform teams, I recommend the book “Team Topologies” (they use the term stream-aligned teams as a more specific term for application teams who are aligned to a value stream).

I have compiled a few ideas and practices in the realm of platform engineering over at my companies website (in German 🇩🇪).

If the system has a sufficient size, those software modules are likely not maintained by one single team. Usually, multiple teams work together and often separate their areas of responsibility by allocating specific modules to certain teams. The structure of the software system and the communication structure between the development teams mirror each other, as I have elaborated in “The Homomorphic Force: When Software and Team Structures Reinforce Each Other”.

In that article, I also discussed how carefully crafting explicit cross-team communication channels along with technical communication plays a crucial role in aligning teams to the system’s structure:

If you have decoupled development teams, make sure to decouple the modules themselves as well. Use technical patterns like asynchronous communication, independently deployable services, and decentralized data storage. Use communication patterns to decouple teams. Co-locate teams that need to work tighter together and actively separate teams that don’t. Make sure that only the communication channels that are necessary due to product dependencies are the ones being actively used. Not all communication is good; what we want is focused communication between specific teams.

However, as we have many tools to describe technical cross-module communication (like OpenAPI or EventCatalog), we barely have any tools to describe social cross-team communication. A few years back, while reading the book Team Topologies, I discovered Team APIs - a tool that helps us make those social communication channels more explicit.

What is a Team API?

By formulating a Team API, development teams can publish typical and deliberately designed interaction points for other teams. It contains everything about the team that other teams need to know to contact it and provides context to make interactions more productive. Usually, this means

• describing what the team's responsibility is: what modules or features are they responsible for?

• answering whether the team provides value to end-users or development teams (e.g. as a platform or enabling team)

• listing what the team provides for other teams. This could be technical concerns like an API or social aspects such as consulting.

• stating how to reach the team

• describing what the team is currently working on and what other teams it relies on to achieve its goals.

The following figure shows an example of a Team API for a hypothetical development team at a music streaming service. It is based on the "Team API Template".

Conclusion

Defining technical communication explicitly is crucial. Social communication lacks the same methodological profoundness and toolchain. Team APIs are an effective solution to make team responsibility and cross-team communication more transparent and explicit.

[Shared resources] could be computational (e.g., CPU), network-related (e.g., bandwidth), or infrastructure (e.g., a lock on a common table). If one subtask exhausts the shared resource, the other subtasks slow down significantly or may even need to stop and wait until enough resources become available again.

[…] As resources become increasingly saturated, subtasks effectively interfere with each other, and the gain in throughput per additional load unit slows down.

In this article, I want to discuss contention in greater detail and outline several strategies to mitigate its adverse effects. Let’s begin by examining the problem more closely…

What is contention?

In general, contention occurs when multiple processes compete for the same constrained resource or architectural bottleneck. The bottleneck effectively sets the system’s throughput, so overall throughput cannot exceed the bottleneck’s capacity.

Bottlenecks may appear in various forms, including resource limitations or architectural congestion points.

Resource Bottlenecks

When a system (and all its parts) depend on a limited resource pool, contention arises as these resources become depleted. This can be computing power (like CPU), bandwidth on a network, or locks on a common database table. As the load grows, the resource pool shrinks, forcing processes to wait longer and slowing throughput growth. Eventually, resources can be exhausted to the point where no further increase in throughput is possible.

Congestion points

Contention can also occur independently of available resources due to congestion points in the system’s architecture. If a particular bottleneck caps a system’s throughput—for instance, as in the figure below, three units—then no matter how much extra load you add, the maximum throughput remains three.

These congestion points can be services within a process flow or infrastructural elements like message queues. They are common in distributed architectures. Consider multiple services relying on a single downstream service: as concurrency rises, the downstream service becomes overwhelmed, leading to longer response times and overload. This problem is especially severe in systems that rely on synchronous communication since congestion in one component can cascade through the entire system.

There are two broad strategies to lower contention: reducing demands on the bottleneck or increasing its capacity. Let’s discuss both in more detail.

Reducing Demands

Contention arises when bottlenecks face excessive load. One way to address this issue is by lowering those demands. The following sections describe four approaches to accomplish this.

Approach #1: Reduce Resource Bottlenecks by Shifting Resource Usage

If the bottleneck involves a shared resource, consider shifting the workload to another type of resource. For example, you could alleviate a network bandwidth bottleneck by compressing data before sending it. This reduces network demand but increases CPU load due to the additional overhead of executing a compression algorithm. Following this example is a good idea if network bandwidth is constrained but CPU capacity is available more broadly.

Approach #2: Reduce Congestion Points by Replicating State

Let’s examine the following system.

There is substantial communication both between Service 1 and Service 3 and between Service 2 and Service 3. Service 3 holds data that the other two services need. As a result, for every task these services perform, they must query Service 3, making it a congestion point. Once Service 3 becomes overloaded, neither Service 1 nor Service 2 can complete their tasks or fulfill requests.

An alternative approach is to replicate the data. By having Service 1 and Service 2 each store their own copy of the data previously held solely by Service 3, the system now looks as follows:

With this change, Service 1 and Service 2 are less dependent on Service 3. Even if Service 3 fails, both can continue serving requests. Communication between the services is now reduced and focuses primarily on maintaining eventual consistency across the replicated data.

Approach #3: Avoid Unnecessary Calls by Using Backoff and Circuit Breakers

Repeatedly retrying calls to an already overloaded service only puts more burden on it. If every requesting service immediately retries failed requests, the stressed service may eventually crash.

One particular strategy to mitigate this issue is retrying with an exponential backoff. Instead of using a fixed retry interval (e.g., every 100 milliseconds), the requesting service gradually increases the waiting period between retries. By doing so, it reduces the incoming load on the already strained service and gives it a chance to recover.

However, exponential backoff alone is not always sufficient. Consider a scenario with three services: A, B, and C. Service A calls B, and B calls C to fulfill the request. If C becomes unreliable, then B retries with exponential backoff, delaying A’s requests further. As discussed in the section “What is contention?” this is a typical behavior in distributed systems, where failure in one service cascades through large parts of the system.

Circuit breakers help address this issue by halting communication with unresponsive services. They do this by incorporating one of three states:

Closed state

When everything is normal, the circuit breakers remained closed, and all the request passes through to the services as shown below. If the number of failures increases beyond the threshold, the circuit breaker trips and goes into an open state.

Open state

In this state circuit breaker returns an error immediately without even invoking the services. The Circuit breakers move into the half-open state after a timeout period elapses. Usually, it will have a monitoring system where the timeout will be specified.

Half-open state

In this state, the circuit breaker allows a limited number of requests from the service to pass through and invoke the operation. If the requests are successful, then the circuit breaker will go to the closed state. However, if the requests continue to fail, then it goes back to Open state.

Examples of circuit breaker implementations are Hystrix and Sentinel.

Approach #4: Use Asynchronous Processing

Most of the issues described above stem from the synchronous nature of communication. One service invokes another and must wait for its response before continuing its work.

Asynchronous processing can decouple this runtime behavior, for example, by using message queues to buffer requests before they reach a bottleneck. I once worked on a system that needed to handle numerous 3D-rendering requests in parallel, but we could only render four images at a time due to licensing constraints on the rendering engine. To mitigate congestion, we introduced a message queue in front of the engine. Although users still had to wait for their rendering tasks to complete, we could immediately return a meaningful response as soon as we placed their request in the queue.

Increasing Capacity

Besides reducing demands on bottlenecks, you can use the second broad strategy mentioned above: increasing the bottleneck’s throughput.

Approach #5: Optimize Throughput By Performance Tuning

This might seem obvious, but it’s still worth emphasizing. Concentrate your performance optimization efforts on known bottlenecks: refine database queries, add read replicas, or improve code efficiency in areas especially prone to congestion. This provides the best return on investment for your performance-tuning initiatives.

Approach #6: Improve Elasticity By Autoscaling

For addressing bottlenecks, elasticity and efficient resource utilization are crucial. As I wrote in “Reactive Systems: Designing Systems for Multi-Dimensional Change”, systems are elastic if they

[…] can scale up or down in small increments, without manual intervention, to support increasing and decreasing load patterns.

The article also provided the following figure to illustrate the idea of elasticity:

Leverage autoscaling capabilities offered by platforms like Kubernetes or public cloud providers and ensure sufficient resource availability to handle usage spikes.

Approach #7: Improve Elasticity By Splitting Bottlenecks

It is not only autoscaling that enables elasticity but also the system’s architecture. Smaller, more modular, and individually deployable components are easier to scale than large, monolithic deployments with long startup times. I also discussed this in “Reactive Systems: Designing Systems for Multi-Dimensional Change”:

Applications deployed as a single process (deployment monoliths) are harder to scale horizontally due to longer start times, resource intensity, and reliance on session state.

Meanwhile, splitting applications helps them become reactive to changes in load. Two things are important here. First, you can split along functionality and arrive at smaller deployables, which then form the entire system by integrating over network communication (e.g., microservices). Second, you can partition the input range among deployables. Narrower responsibilities (e.g., handling customers only with last names beginning with A-C) reduce the load on individual nodes and make smaller-scale increments possible. Combining both strategies leads to the best outcomes in terms of elasticity, i.e., the ability to scale up and down, on demand, and in small increments.

Conclusion

The approaches in this article are only a subset of all strategies to lower the effects of contention. Because resolving one bottleneck often exposes another, it’s best to iteratively refine your system rather than attempting perfect upfront planning. Identify the most restrictive bottleneck, apply the approaches outlined above, and then move on to the next one. Repeat this process until you’ve reduced system contention to an acceptable level. In a sufficiently complex environment, you may never eliminate contention entirely—but you can make it manageable.

The figure shows linear scaling, a line growing and shooting infinitely to the top. For each added load unit, you get an equal gain in throughput (e.g., the number of users or data records the system can handle). There are certain (and very specific) situations where scalability works like that; for example, independently parallelizable tasks like text search, where each subtask looks for keywords in separate parts of the entire text. The result is then a matter of combining outputs from the previously executed subtasks.

However, this is different from how scalability works in general. As soon as you look at entire, complex systems instead of individual, parallelizable algorithms, the issue gets more complicated…

Concurrency

The first issue such a system often encounters is hitting a resource limit — for example, when subtasks share a resource pool and reach its capacity. Initially, the throughput increases approximately linearly, but as it approaches the resource limit, the gain in throughput gradually diminishes and fades out.

In these cases, the system’s throughput is limited by its concurrency and may hit an upper bound once it exhausts the available resource pool.

Shared Resources

Subsystems competing for resources may also inhibit throughput gain. Let's look at the following system:

Although the subtasks can run concurrently, they depend on a shared resource. This shared resource could be computational (e.g., CPU), network-related (e.g., bandwidth), or infrastructure (e.g., a lock on a common table). If one subtask exhausts the shared resource, the other subtasks slow down significantly or may even need to stop and wait until enough resources become available again.

This slows down scalability and disrupts the linear increase in throughput compared to the example before. As resources become increasingly saturated, subtasks effectively interfere with each other, and the gain in throughput per additional load unit slows down.

In this scenario, the system's throughput is limited by its subparts contention over shared resources. As you can see, the gain of throughput per additional load unit slows considerably.

Maintaining Consistency

In certain systems, adding more load can actually decrease throughput. Let's examine the following system:

The three subtasks each have their own data storage, either persistently through separate data stores or ephemerally via caches. Let's assume the system must maintain consistency to respond correctly. Imagine one subtask updating data in its own storage; the system must then propagate these changes to other storages using a transaction.

This has significant consequences. Not only do the subtasks need to stop responding until a consistent state is restored, but the process of re-establishing this state requires additional work before the system can resume service. This overhead negatively affects throughput: as more load is added, the throughput actually decreases—a phenomenon often observed in real-world throughput metrics.

Thus, the throughput of the system is limited by the amount of coherency it needs to maintain.

The Universal Scalability Law

The dynamics I laid out above are described in a model called the Universal Scalability Law, formulated by researcher Neil Gunther back in 1993:

C(N) represents the relative throughput given a load of N. It is relative in the sense that X(N) defines the absolute throughput, and C(N) = X(N) / X(1).

Gunther refers to the scalability effects from above as "The Three Cs":

• Concurrency-limited scalability is proportionally linear because α = 0 and β = 0 (see Figure 1). You should expect a scaling limit, as shown in Figure 3, but the equation above does not explicitly contain one. (Note: To illustrate effects such as reaching a resource limit more intuitively, the illustrations above use the absolute throughput (X(N)) on their vertical axis instead of the relative throughput (C(N)) discussed in this section. Gunther also provides a generalized form of the equation for X(N).)

• Contention-limited scalability is the effect of slowing down throughput gains, as shown in Figure 5. It occurs when α > 0 and β still equals 0.

• Coherency-limited scalability is the negative throughput gain due to extra work to keep the system consistent, as shown in Figure 7. It is the case when α and β are both greater than 0.

How to Use the Universal Scalability Law

Suppose you have made a few measurements by subjecting the system under a particular load and measuring its throughput. You can use the equation above to extrapolate further data points. This helps you assess load scenarios that you have not measured yet or are not able to measure. Gunther states that you need at least around six data points to do this.

You could also take measurements, fit a curve from the above equation, and estimate values for contention (α) and coherency (β). These estimates, alongside the system's throughput, could serve as holistic fitness functions evaluating the scalability properties of your system. In "Architectural Fitness Functions: Building Effective Feedback Mechanisms for Quality Attributes", I have written about holistic fitness functions:

We need effective and (if possible) automated [feedback] mechanisms, especially for quality attributes. The book "Building Evolutionary Architectures: Automated Software Governance" introduced a term for feedback mechanisms on quality attributes borrowed from evolutionary algorithms: (architectural) fitness functions.

In short, an (architectural) fitness function evaluates the system with regard to a quality attribute. You can distinguish fitness functions by the scope of their evaluation:

[…]

Fitness functions can also be holistic. They test the entire system or large parts of it and assess specific quality attributes. For example, they might measure response times or observe the system after being put under load. Holistic fitness functions are usually more complex and expensive to build.

Conclusion

The Universal Scalability Law not only provides deeper intuitions into how scalability works but also aids in analyzing a system’s behavior under load. It serves as both a mental model and a practical tool for evaluating a system’s scalability. You can use the insights generated by the model to discover and prioritize improvements to reduce contention and coherency overhead as needed.

Further Reading and References

If you want to understand the Universal Scalability Law in more detail and discover further usage scenarios and tools, I recommend this article by Neil Gunther.

For a more elaborated discussion on scalability models and how to obtain the graphs from above, see “Getting in the Zone for Successful Scalability”.

This article aims to define an reactive system as a socio-technical system (involving both the technical system as well as the individuals and teams maintaining it) that is able to sense changes from multiple dimensions and transform accordingly, either immediately or over time. It is likely that not all dimensions are relevant for every system, so it is the responsibility of developers and architects to identify the most important ones and design the system accordingly.

In this article, I will discuss several dimensions that I have been brainstorming over the last few days. I will also lay out broadly what properties a system should have to be adaptive to each dimension.

Functionality

Functionality is the dimension where change occurs most frequently. That’s because the main purpose of software systems is to deliver value to users by providing functionality. Demands on functionality change often. New usage ideas and things like regulations typically drive these changes. Additionally, users, often subtly, start using the software differently due to changes in the context in which it is embedded.

Developers should be able to implement changes to functionality without much ceremony. Once central roles, groups, or committees need to acknowledge work, break it down to teams, and coordinate implementation across many parts of the system, the system loses certain reactive properties. Instead of a single cell in the system needing to react to a particular change, adaptions lead to a shock across the entire system. This leads to complex agreements and time dependencies. Teams then often must build solutions together and deploy all changes simultaneously.

Use Domain-driven Design, especially bounded contexts, to decouple modules. This will assign each module a clear boundary of functionality. In addition, as I have written in “A Case Against 'One Model to Rule Them All’”, you should consider the responsibilities of development teams as well:

One very important thing is that a bounded context must be small enough to be clearly assignable to a development team. Don't spread the responsibility of bounded contexts over multiple teams; this only leads to unclear responsibility and makes it harder to establish a ubiquitous language within the bounded context because more people are involved. If you have a bounded context too large for a single team, consider splitting it.

Check out this post for more information on bounded contexts. Extending that idea to include the autonomy of teams, Team Topologies introduces the concept of stream-aligned teams:

[…] the team is empowered to build and deliver customer or user value as quickly, safely, and independently as possible, without requiring hand-offs to other teams to perform parts of the work.

As you’ll discover in the rest of the post, I believe team autonomy serves as one of the main pillars of managing all dimensions of change.

Reactive Property

A system that is reactive in this dimension allows teams to implement most functionality as independently as possible and with little coordination. If coordination is needed, it is explicit and takes place over well-defined social and technical interfaces.

Usage and Load

Load patterns are inherently difficult to predict. Imagine your task is to predict how much hardware you need to support a given workload. You are almost guaranteed to guess too high or too low and end up over- or under-provisioning hardware. Some solutions, especially in cloud computing, counteract this problem by providing increasingly smaller scale-units, with newer technologies that can scale individual functions (Function as a Service). In combination with self-service APIs, these infrastructures can be spun up and torn down at the touch of a button. This enables autoscaling functionality, which scales the infrastructure based on the demand - for example, the current load the system is experiencing.

Besides infrastructure, you need to consider other aspects as well to support increasing and decreasing load patterns. Applications deployed as a single process (deployment monoliths) are harder to scale horizontally due to longer start times, resource intensity, and reliance on session state.

Meanwhile, splitting applications helps them become reactive to changes in load. Two things are important here. First, you can split along functionality and arrive at smaller deployables, which then form the entire system by integrating over network communication (e.g., microservices). Second, you can partition the input range among deployables. Narrower responsibilities (e.g., handling customers only with last names beginning with A-C) reduce the load on individual nodes and make smaller-scale increments possible. Combining both strategies leads to the best outcomes in terms of elasticity, i.e., the ability to scale up and down, on demand, and in small increments.

Reactive Property

A system that is reactive in this dimension can scale up or down in small increments, without manual intervention, to support increasing and decreasing load patterns.

Technological Foundation

Innovation in cross-cutting technical foundations includes not only frameworks, programming languages, or libraries but also patterns, cloud services, and hardware specifications. There is something new or different seemingly every day. How do you cope with this, especially in large applications?

Smaller deployables help here since they are more reactive to changes in their technological foundation. Teams maintaining individual services can experiment with and adopt new concepts without necessarily affecting the entire application. While a change in programming language or underlying frameworks probably requires rework in larger parts of monolithic applications, individual services communicating over a network can completely differ in their implementation or technology. This means experimenting with technological innovations in a single, isolated service is much easier and less risky.

Reactive Property

A system that is reactive in this dimension can absorb technological innovations by experimenting with them within a subset of the system and then, if the innovations prove beneficial, gradually adopt them across the entire system.

Methodology

Similar to technological foundations, changes to methodology can be more easily adopted by autonomous cells within the system rather than by the entire system at the same time. In this context, these autonomous cells are teams.

Innovation in methodology is related to ways of working and communication. This includes adopting team processes (like switching from Scrum to Kanban) or introducing new ways of working, such as conducting regular internal team reviews. Autonomous teams can decide to try out methodologies and adapt them if they prove successful.

Teams know best what works in their day-to-day communication and collaboration, so it is crucial that they be allowed to decide autonomously how to facilitate that. Organizations should not prevent variation in methodology but support cross-team communication so that teams can share their experiences and inspire ideas among other teams.

Reactive Property

A system that is reactive in this dimension allows multiple development teams to use different methodologies. This includes the team's internal workflows as well as communication principles and patterns used when interacting with other teams.

Conclusion

This list of change dimensions is not complete. There are nuances to the dimensions mentioned above, as well as completely different dimensions that I did not mention but which may play a role, often only within specific contexts. What’s important is that you are aware of change dimensions in your particular context and design a system that is reactive to those.

Further Reading

If you want to learn more about microservices (as mentioned in the section ”Functionality”), I recommend Building Microservices by Sam Newman.

About 15-20 years before the fraud's 2016 revelation, the bank recognized significant growth potential in cross-selling products. Cross-selling is a method in which companies focus on existing clients and offer incentives for purchasing additional products instead of seeking new customers. In banking, this means, e.g., to convince checking account clients to add savings accounts or mortgages.

Wells Fargo began tracking its success in cross-selling by measuring the average number of products held by customers. While cross-selling itself is not fraudulent, Wells Fargo executives were so eager to maximize revenues that they pressured sales employees to boost this metric (the number of banking products customers held). To meet this increasingly unattainable sales target, employees began opening accounts using customers' data without consent. Research by the American Bankruptcy Institute Journal revealed that employees "opened as many as 1.5 million checking and savings accounts, and more than 500,000 credit cards, without customers' authorization.”

The focus on a specific metric, combined with immense pressure to reach targets within the same metric, led to unintended behavior. Wells Fargo's story is not a singular occurrence. Similar examples include urban legends such as the Soviet nail factory, where factories met production quotas by making either tiny or massive nails that were useless, depending on whether they were measured by quantity or weight. Another urban legend is India's "cobra effect," where a bounty on cobras led people to breed them for profit, ultimately increasing the cobra population.

In 1975, the statistician Charles Goodhart described this effect in an article. His idea was later coined as Goodhart's Law and is often paraphrased as:

When a measure becomes a target, it ceases to be a good measure.

Measuring the number of average banking products per customer is not wrong in itself. In software development, we do this all the time. We track technical aspects like CPU load, network latency, and lines of code as well as functional aspects like daily active users, purchases, and registrations. There's nothing wrong with gathering data and using it to make informed decisions, right?

Goodhart's Law targets the intention with which metrics are used. Measuring is beneficial, but using metrics as targets and goals can be problematic. Let's dive deeper into this dynamic.

Metrics in Software Development

Software development is product development. Every software team builds a software product that others use to derive value. It follows that the quality of this product matters since users prefer products of higher quality. In software, this usually concerns things users recognize, like performance, reliability, and usability. It also includes things developers observe, like maintainability, extensibility, and operability.

These aspects are often subjective and difficult to quantify. Think of a product you like using, such as your phone or bike. Why do you like it? What you will likely come up with is a gut feeling rather than hard, data-based evidence. However, we like to use metrics to make quality aspects measurable and to implement a baseline that we can improve on. In "Architectural Fitness Functions: Building Effective Feedback Mechanisms for Quality Attributes", I wrote:

You can gather feedback on multiple aspects. One of the most crucial are important quality attributes, like performance or reliability: Do we need to build a software service that is both highly scalable and highly available? Is high performance crucial for our product? We need to assess whether the system we build holds up to those aspects.

We need effective and (if possible) automated mechanisms, especially for quality attributes. The book "Building Evolutionary Architectures: Automated Software Governance" introduced a term for feedback mechanisms on quality attributes borrowed from evolutionary algorithms: (architectural) fitness functions.

In short, an (architectural) fitness function evaluates the system with regard to a quality attribute.

These fitness functions can act as metrics to circle back evaluative or corrective information (i.e., feedback) to the development team, helping to detect areas for improvement.

It is precisely in this usage of feedback as corrective information where Goodhart's Law becomes relevant. As soon as you use metrics as a framework for evaluation and decision-making, issues can start to arise. Focusing on a few metrics tends to narrow your view of the system. People start asking, "Why is the metric down?" instead of reasoning about the actual quality of the entire system. We use metrics because the aspects we care about, such as subjective quality, are often fuzzy and not directly measurable. While we aim to achieve something intangible, we rely on metrics as proxies for that goal. When we conflate the two, we risk prioritizing actions that improve the metric at the expense of the actual objective. Metrics are not the goal itself.

Take test coverage, for example. It measures the proportion of code statements exercised by the unit tests in a test suite. This is a proxy metric because the goal we want to achieve is software that functions correctly and test coverage that helps approximate the correct functionality of the software. However, taking it as a goal can be problematic. Incentivizing teams and individuals to hit 90% test coverage can encourage writing "easy" tests that cover a lot of code, especially under time pressure. Think of testing getters and setters. Other examples of proxy metrics are measuring cyclomatic complexity or the load placed on a particular service.

Holistic Metrics

You can classify fitness functions on a scale from atomic to holistic. I provided an overview of this characterization in "Architectural Fitness Functions: Building Effective Feedback Mechanisms for Quality Attributes":

Atomic fitness functions evaluate a part of the system on a specific quality attribute, like measuring the length of methods within classes or monitoring the status of nodes within a system. Those fitness functions are easier to build, but they only give you a narrow view of the system's quality.

Fitness functions can also be holistic. They test the entire system or large parts of it and assess specific quality attributes. For example, they might measure response times or observe the system after being put under load. Holistic fitness functions are usually more complex and expensive to build.

A rule of thumb: The more the fitness function (or metrics in general) leans towards the atomic side of the scale, the more it is prone to Goodhart's Law. To discuss this effect in detail, we must first discuss the difference between outputs and outcomes.

Outputs are the deliverables of our software project - specifically, its code and executables. Outcomes are the effects these deliverables create in the world, like users interacting with the system or developers interacting with the code. The more you focus on measuring outputs instead of outcomes, the more you are building fitness functions on the atomic side. Meanwhile, focusing on measuring outcomes leads to producing holistic fitness functions since you are more likely to measure the actual effect of the system acting in its context.

Measurements on outcomes do have issues as well. Not only are they more challenging to obtain, but they are also more prone to bias and distortion since not all aspects that affect the measurement are the ones you intend to measure. Take a holistic, outcome-focused measurement like the number of daily active users. It's a metric that is less prone to Goodhart's Law because it is harder to game. However, if daily active users go down, what is the root cause? Is the system's performance or usability degrading? Do newer features not prove valuable for users? Or is it because alternative, competing products outperform ours? Figuring this out is complex, and the metric alone can't give us an answer, only an indication.

Gaming Is a Systemic Symptom, Not the Fault of Individuals

Above, I often phrased actions as “gaming the system.” I want to clarify that I do not believe metric gaming is the fault of teams or individuals. “Gaming” is a response to a flawed system of measurements and metrics. If leaders put pressure on developers to achieve a certain percentage of test coverage, they should not be surprised if the quality and purpose of some automated tests decline. Blame the flawed metric and incentive, not the individuals.

Conclusion

When building metrics and fitness functions, it is important to consider Goodhart’s Law. Avoid using atomic feedback mechanisms as targets; instead, focus on improving holistic measurements. Use atomic mechanisms as useful indicators to identify where to dig deeper after observing degradation in holistic metrics.

Further Reading

If you want to read more about the Wells Fargo cross-selling scandal, check out the article on Wikipedia.

That approach leads to local optimizations. You build and release features over a longer period of time, always focusing on the next increment. After a while, you might find that your system grew and became unmaintainable, fragile, and unreliable. To some degree, we need strategic decisions, principles, and rules to keep this in check. However, too much of this can result in over-engineering and rigidity.

Let’s discuss how to strike a balance between short-sightedness and over-engineering.

Product Increments and Quality Attributes

We can’t reason about a successful software product by only focusing on functionality and features. Each system must uphold certain properties like reliability, performance, or security. We call those properties quality attributes. You can only fulfill quality attributes by following foundational concepts or principles, often cutting across many parts of your system. Need to support millions of users simultaneously? You could rely on established and scalable infrastructure from a public cloud vendor. Need to provide calculation results within a few milliseconds? Maybe you front-load expensive calculations by caching intermediate results across all calculation modules. Need to make sure that specific functionality is easily modifiable by developers? Your team could decide to design a bounded context around the functionality and extract it into its module.

When you develop a particular product increment, you plan on delivering value to your users. So you need to deal with some quality attributes immediately, while others won’t play a role in the upcoming increment. For example, if you plan to deliver new calculation modules, you may want to design a caching solution to reach your performance goals. But, if your product does not yet need to support millions of users, you could delay decisions about this quality attribute.

Product Increments and Strategic Decisions

Your first consideration should always be: “What do we need to consider in the upcoming increments to fulfill the quality attributes we aim for?” If this is the first iteration where you need one service to talk to another, you must at least decide which communication mechanism you want to use.

You could delay other considerations, e.g., how to communicate with that third-party API if the corresponding user story is four iterations down the road. You only have to make sure that your current architecture does not impede that implementation when it comes up. That's where having an architectural vision (a rough idea of the target architecture you are currently aiming for) helps reduce risks and mitigate issues that might come up down the road.

After identifying a highly relevant decision for your next product increment, you want to identify potential options to solve that question. Having many options is a good thing because it increases your decision flexibility.

Options and How to Deal With Them

Your knowledge of a particular option may affect your confidence in building an increment that meets certain quality attributes. Choosing an option you are less optimistic about is okay if you can easily migrate to another option later on.

Figure 1 shows that. The vertical dimension shows your confidence in the option, while the horizontal dimension depicts the cost of migrating away from it to another option. Each field in the quadrant shows how to deal with a given option.

Just do it

Example: A team chooses a familiar message bus and knows it achieves the required throughput, which is critical for achieving performance goals. The message bus also supports an open standard (Java Message Service). That makes it interchangeable with other messaging solutions if the throughput nevertheless turns out to be insufficient.

Options where you have high confidence their implementation will achieve your quality attributes with low migration costs are ideal. They have shallow risks, and you should implement them directly.

Evaluate fast

Example: A team must choose a relational database. First, it will find out which DBMS best supports its load-heavy query patterns. It decides to hide the database behind a Java Persistence API abstraction and tires out different database management systems in production.

When you are unsure about the outcome of implementing the option but it is easily exchangeable, you should evaluate it as fast as possible. You can even assess multiple options, e.g., by trying one solution and switching it out for another. You could also release one solution to half of your users while releasing another solution to the other (A/B testing).

Evaluate and close migration options

Example: All development teams in the company are familiar with Kotlin. Kotlin is known to be appropriate for building readable code and supporting maintainability. A newly formed team responsible for implementing mathematical models decides on the programming language they want to use. Most team members favor Kotlin. But some worry about using a JVM-based language for complex mathematical models, fearing it may hurt performance. So, they decided to build a Proof of Concept in Kotlin, validating whether performance criteria are met. After it turned out to be the case, they did not evaluate other options further and continued working with Kotlin.

With high confidence in an option but high migration costs, you should figure out the riskiest parts of implementing the option and evaluate them quickly. Afterward, don’t actively keep migration paths open; go with the chosen option. (Note: that holds if requirements for quality attributes are believed to stay stable. If they change and evolve, it might be reasonable to keep migration paths open nevertheless).

Evaluate and keep migration options open

Example: Two development teams want to split functionality between their microservices. They came up with an idea of which functionality should belong to which service but remain skeptical about whether this split is a good idea. They decide to implement the solution but ensure future refactorings remain possible. So, they aim for a consistent tech stack to ease moving code between the services (or merging the services into one).

In this case, switching is both challenging and likely. The way to go here is to think strategically about migration paths and actively keep them open until the confidence in the solution rises.

Conclusion

Strategically handling decisions and their options is essential in agile product development. Address necessary questions for upcoming iterations. At the same time, actively keep necessary migration paths open and delay decisions you do not have to make right now. Figure 2 summarizes this idea.

"The tactical result of an engagement forms the base for new strategic decisions because victory or defeat in a battle changes the situation to such a degree that no human acumen is able to see beyond the first battle. In this sense one should understand Napoleon's saying: "I have never had a plan of operations."

Therefore no plan of operations extends with any certainty beyond the first contact with the main hostile force."

- Field Marshal Helmuth von Moltke the Elder

I started professional software development sometime around 2013. It was at a time when Agile methodologies like Scrum were already quite established in teams and organizations. It seemed to me that it had morphed into a default choice in software development. Working like that made sense to me. Instead of doing exhaustive planning up front and shipping software after long development and testing phases, we went for shorter feedback cycles that enabled us to incorporate customer feedback earlier. Being new to software development, I thought this was a good thing (spoiler alert: today, I still think it is).

Nevertheless, my understanding was too superficial to assume that Agile methodologies are a good idea simply based on the fact they were the default choice for most teams I worked in. I wanted to understand the mindset and reasoning behind this way of working. I wanted to dig deeper.

Software Development and Complexity

Before we discuss Agile as a mindset and its methodologies, let's take a step back by looking at a conceptual model from complexity theory: The Cynefin Framework categorizes problems into five buckets: simple, complicated, complex, chaotic, and disorder. Its purpose as a sense-making framework is to contextualize problems and choose appropriate solution strategies.

The first section contains clear problems. In this area, solutions to problems are predictable and repeatable. Thus, we have existing best practices (or checklists) to solve them. Complicated problems are a little more difficult. They require expert knowledge to analyze, but we can follow a clear plan to arrive at a solution.

There are some problems that we can grasp only in hindsight. Here, we face complex problems. Outcomes are unpredictable. We must experiment to find out what works. There are also chaotic problems where we cannot gather enough understanding to conduct experiments in the first place. The space of confusion sits in the middle. That means our problem should reside in one of the other spaces, but determining which one presents challenges, so we rely on familiar solution patterns that may not be the best choice.

Software development lives in the space of complex problems. Why is that? First, we do not know beforehand how our users will use the system's functionality and in what combination they will do so. Second, we do not know what the best approach to solving the problem is in the first place. Also, we work in a fast-changing software development ecosystem. New frameworks and dependency versions appear daily. All the reasons above make it very hard to predict what will happen after we have started developing software systems.

Martin Fowler argues that software development is primarily a design activity, a creative process that is hard to plan. Unlike civil engineering's plan-driven approaches, more predictable "construction work" is a small part of software development.

Classical approaches in our discipline heavily relied on up-front plans and up-front design. That neglected the complex nature we are dealing with. It treats software development as a complicated problem without considering the unpredictability of user needs, the context in which the system operates, and the evolution of the software development ecosystem.

Agile Is About Responding to Change

Responding to change is a central pillar. In the face of uncertainty, it makes sense to work in small increments and decrease the amount of up-front planning since the plan would need to change anyway. A better approach is to conduct experiments, fail as fast as possible, and learn from the assumptions we have made. A consequence of this line of thought is that we want to deliver our system early and regularly to get feedback as fast as possible.

But what changes do we need to respond to in the first place? First, requirements change. They change because it is much easier to argue about a system, its features, and the usage workflow when customers can try and experience the system in the first place. Even when our users would perfectly know about the features and how they will use them, we would still need to factor in changing requirements. The reason for that is the evolving world around us. Our company might decide to enter a new market... or new competitors might be emerging. Changes in the business world are highly unpredictable, as every stock trader would tell you.

Second, we have to deal with an evolving software development ecosystem. Operating systems, container orchestrators, cloud environments, frameworks, and libraries are only examples of tools we use in our day-to-day development. All of them evolve independently and sometimes in unexpected directions. Some of them have security issues that need to be fixed urgently. On top of that, new tools emerge, often unpredictably. Could you have predicted the impact of Docker before it was released and widely adopted?

Agile Is About People

As we have discussed above, software development is creative work. As with other creative work, individuals are not replaceable and make unique contributions to the software product. Each individual has a background, experiences, strengths, and weaknesses that can uniquely and positively impact the team and its work results. Therefore, we should treat people as first-class citizens in the software development process instead of handling them like replaceable gears in a factory.

Agile Is a Mindset

After the end of World War II, inhabitants of the Melanesian islands in the South Pacific practiced what anthropologists later coined "Cargo Cult." During the war, Allied forces dropped packaged equipment and food items onto military bases stationed on the islands. This profoundly affected the islanders since the military shared some of the goods with them, significantly increasing their standard of living. After the war, when military bases were removed from the islands, inhabitants were short of supply and the abundance they brought.

The (scientifically disputed) theory is that inhabitants of the island believed technologies (such as airplanes, headsets, and landing towers) were worshipping relics to spiritual entities who dropped equipment and food packages onto the island. What happened was that the islanders began to build their airplanes, headsets, and landing towers out of straw and practiced military formations with wooden weapons as a spiritual act. The spiritual worship did not result in packages being dropped from the air.

The message is that implementing Agile methodologies without adopting Agile principles and mindsets is problematic. Using processes like Scrum or XP only works as far as you adapt the holding beliefs as well. Otherwise, you will land on an island full of buzzwords without embracing Agile as what it truly is: a mindset.

Out of that mindset follow the practices: we write tests and embrace continuous delivery because it makes it easier for us to deliver working software faster and more often. We hold daily stand-ups to determine if team members can help each other clear away impediments because we believe Agile is about people first. We conduct experiments to see what works and what doesn't work. And we want to release software more often because it helps us collaborate with our customers in the first place.

Agile Is No Means to an End

One of the fundamental problems with Agile lies in its adoption. Superficial adoption of the methodologies and thinking it might be a silver bullet is a significant obstacle. We need a deep understanding of the mindset behind the methods—organization-wide, not only technical but also from leadership positions and beyond.

What is essential for Agile teams is to have a safe space to try things out, potentially fail, and have room to learn and adapt. Leadership and management must embrace that. It is fundamental not to see those methodologies as a control mechanism but as a mindset that helps to deliver features faster, embrace people as creatives and make room for experimentation and growth.

Further Reading and References

In “The New Methodology”, Martin Fowler wrote about his thoughts about the Agile movement in 2005.

The "Manifesto for Agile Software Development" states the most central principles of Agile software development.

In ”The Cynefin Framework”, its creator David Snowden describes elements and applications.

As I do every year, I have read the report with great interest. The central part of this year’s report covers topics currently in broad discussion, such as the adoption of artificial intelligence in the software delivery workflow and platform engineering.

Below, I have summarized my key highlights. The text draws heavily from the information in the report. After reading my summary, I encourage you to dive deeper by reading the full report.

Adoption of AI leads to both positive and negative effects

An overwhelming proportion of survey participants (75.9%) said they use AI in at least one of their daily work tasks. It is most often used for writing, optimizing, and documenting code, summarizing information, or explaining unfamiliar code. Unsurprisingly, this broad adoption has both positive and negative consequences for individuals and development workflows.

Impact on individuals

Noticeable adoption of AI leads to an increase in flow (the degree of focus individuals achieve), job satisfaction (individuals overall feeling about their job) and productivity (the degree of effectiveness individuals feel in their job).

Interestingly, it also leads to a decrease in time spent doing valuable work. The report describes valuable work as “development-related tasks, like coding, […] while less-valuable, even toilsome, work typically includes tasks associated with organizational coordination, like attending meetings.” The authors hypothesize that AI mainly automates valuable work. This is the development-related work from the description above. Meanwhile, the amount of non-valuable work, like organizational coordination, stays the same.

I want to share a personal observation about AI-generated code. Writing code feels valuable. However, fixing AI-generated code does not. I can count many times when I felt accomplished after implementing a complex algorithm. I would not have felt that if I had just fixed and optimized an AI suggestion.

Impact on development workflows

AI adoption drives documentation and code quality and enhances code review speed. The degree of quality is derived from the subjective evaluation of survey participants, leaving some room for misinterpretation. Yet, I think on this scale and for this order of magnitude, it is a good indicator of spotting trends like this.

Surprisingly, AI adoption leads to lower delivery stability, including the change failure rate and the rate at which teams need to make adjustments after release. So, releases that include AI-generated code tend to be more unreliable. The authors think this might be because AI-generated code increases the batch size of releases. Larger batches are often less reliable, leading to more required changes after release.

Improving delivery speed and product quality is not the primary driver for AI adoption

One thing that surprised me the most is that AI adoption is not mainly driven by the desire to increase quality or delivery speed. It is more driven by external pressure, such that others could improve product quality and delivery speed by leveraging AI, and you might fall behind as an individual or organization. However, even if it is not the primary driver for adoption, AI still increases productivity. Only 10% of participants reported negative impacts; the other 90% reported at least slight productivity increases, with 35% accounting for them as moderate or extreme.

Most developers tweak AI-generated code

Most teams do not release AI-generated code to production, only passing automated checks. Most of the time, however, AI-generated code goes through a human's brain at some point before deployment. Copy-pasting AI-generated code often needs manual adjustments. One participant in the report's interview compared the situation to "the early days of StackOverflow, [when] you always thought people on StackOverflow are really experienced, you know, that they will know exactly what to do. And then, you just copy and paste the stuff, and things explode".

Companies across industries adopt AI at the same rate

Industries adopt technologies at vastly different speeds and timescales. However, AI shows no significant difference in adoption rates across industries. What is notable is that organization size predicts AI adoption, with smaller organizations being more aggressive in AI adoption than larger organizations.

As with AI, the adoption of platform engineering leads to both positive and negative effects

Internal developer platforms positively impact surveyed participants' individual performance and their organizations. The most significant factors noted in the report are:

• Higher levels of individual and team productivity (8% and 10%, respectively). Productivity describes the extent to which an individual or team feels efficient.

• An organization's overall performance rises by 6%. This measures the organization's ability to deliver software and business value quickly.

The negative impacts, however, were surprising to me. The report notes that there are two major adverse effects organizations leveraging internal developer platforms observe:

First, throughput (the time it takes for code to reach production and how often code is deployed to production) decreases by 8% compared to organizations not using a platform. Second, delivery stability (the change failure rate and the rate at which rework is needed after release) decreases by 14%.

Multiple forces are at work here. The decrease in throughput could be explained by additional automation steps performed by the platform that are required before release to production, like security checks and testing. Additionally, if a platform is mandatory for development teams, ill-fitted platform functionality could also decrease productivity and throughput.

The drop in delivery stability may be due to increased automation. It lowers the bar for developers to push code since more automation potentially spots more issues. The report also states that a platform might make rollbacks easier, "empowering teams to experiment and deliver changes, which results in an increased level of change failure and rework."

User-centricity leads to better products

User-centricity is the degree to which teams align their priorities to their users’ needs. User-centricity is positive across the board: 

• It makes developers happy by inducing a sense of purpose in their work. 

• It is the only factor that makes software delivery performance less critical to product success. 

As the report states:

“We find that when organizations know and understand users’ needs, stability and throughput of software delivery are not a requirement for product quality. Product quality will be high as long as the user experience is at the forefront.”

Internal documentation, combined with user-centricity, is powerful

Besides user-centrality, the quality of internal documentation plays a significant role in product quality. As the figure below shows, the combination of both factors has substantial positive implications. The report states that this is because internal documentation can act as a magnifier for user feedback and propagate it across the team.

Unstable organizational priorities lead to decreases in productivity and increases in burnout

The authors write:

“We all know the feeling. You’ve spent the last few months working on a new feature. You know it’s the right thing to build for your users, you are focused and motivated. Suddenly, or seemingly so, the leadership team decides to change the organization’s priorities. Now it’s unclear whether your project will be paused, scrapped, Frankensteined, or mutated.”

Shifting priorities leads to disruptions, and constantly doing so has profound negative effects on teams and individuals, especially the burnout rate. Persistent priority changes and the accompanying uncertainties place a burden on employees.

Moving to the cloud may be harmful if the provided flexibility is not used

Only if organizations take advantage of the main characteristics of cloud computing (on-demand self-service, broad network access, resource pooling, rapid elasticity, and flexible infrastructure) can they make the most out of a transformation to the cloud. The report goes even further and states that organizations moving to the cloud and not using the full potential might be better off staying in a more traditional data center:

“Organizations may be better off staying in the data center if they are not willing to radically transform their application or service. Of course, to accomplish this, it is not simply adopting tools or technologies, but often an entire new paradigm in designing, building, deploying, and running applications.”

Organizations across all industries can achieve high levels of software delivery performance

Software delivery performance and the organization's industry are independent. Even though each industry has its challenges, there is no industry prone to high delivery performance and no industry where delivery performance can't be improved.

Conclusion

I found this year's report especially interesting because it addresses two widely discussed topics: the usage of Artificial Intelligence in software development and platform engineering. I was not surprised that both approaches have obvious drawbacks besides their upsides; however, some particular drawbacks (such as platform engineering being able to lower throughput and delivery stability in some cases) raised my eyebrow. The report hypothesized several reasons, which I found sensible in the first place. However, the topics and issues raised in the report will require me to do more thinking and writing in the future.