My 10 Highlights From the 2024 DORA Report
Discover how the latest DORA report reveals surprising pros and cons of AI and platform engineering.
In late October, the DORA (DevOps Research and Assessment) research program released its latest annual “Accelerate State of DevOps” report, surveying over 90,000 individuals involved in software development organizations. It is the tenth installation in a series of yearly reports summarizing how organizations put in place practices of DevOps as well as related disciplines and the effects they observe.
As I do every year, I have read the report with great interest. The central part of this year’s report covers topics currently in broad discussion, such as the adoption of artificial intelligence in the software delivery workflow and platform engineering.
Below, I have summarized my key highlights. The text draws heavily from the information in the report. After reading my summary, I encourage you to dive deeper by reading the full report.
Adoption of AI leads to both positive and negative effects
An overwhelming proportion of survey participants (75.9%) said they use AI in at least one of their daily work tasks. It is most often used for writing, optimizing, and documenting code, summarizing information, or explaining unfamiliar code. Unsurprisingly, this broad adoption has both positive and negative consequences for individuals and development workflows.
Impact on individuals
Noticeable adoption of AI leads to an increase in flow (the degree of focus individuals achieve), job satisfaction (individuals overall feeling about their job) and productivity (the degree of effectiveness individuals feel in their job).
Interestingly, it also leads to a decrease in time spent doing valuable work. The report describes valuable work as “development-related tasks, like coding, […] while less-valuable, even toilsome, work typically includes tasks associated with organizational coordination, like attending meetings.” The authors hypothesize that AI mainly automates valuable work. This is the development-related work from the description above. Meanwhile, the amount of non-valuable work, like organizational coordination, stays the same.
I want to share a personal observation about AI-generated code. Writing code feels valuable. However, fixing AI-generated code does not. I can count many times when I felt accomplished after implementing a complex algorithm. I would not have felt that if I had just fixed and optimized an AI suggestion.
Impact on development workflows
AI adoption drives documentation and code quality and enhances code review speed. The degree of quality is derived from the subjective evaluation of survey participants, leaving some room for misinterpretation. Yet, I think on this scale and for this order of magnitude, it is a good indicator of spotting trends like this.
Surprisingly, AI adoption leads to lower delivery stability, including the change failure rate and the rate at which teams need to make adjustments after release. So, releases that include AI-generated code tend to be more unreliable. The authors think this might be because AI-generated code increases the batch size of releases. Larger batches are often less reliable, leading to more required changes after release.
Improving delivery speed and product quality is not the primary driver for AI adoption
One thing that surprised me the most is that AI adoption is not mainly driven by the desire to increase quality or delivery speed. It is more driven by external pressure, such that others could improve product quality and delivery speed by leveraging AI, and you might fall behind as an individual or organization. However, even if it is not the primary driver for adoption, AI still increases productivity. Only 10% of participants reported negative impacts; the other 90% reported at least slight productivity increases, with 35% accounting for them as moderate or extreme.
Most developers tweak AI-generated code
Most teams do not release AI-generated code to production, only passing automated checks. Most of the time, however, AI-generated code goes through a human's brain at some point before deployment. Copy-pasting AI-generated code often needs manual adjustments. One participant in the report's interview compared the situation to "the early days of StackOverflow, [when] you always thought people on StackOverflow are really experienced, you know, that they will know exactly what to do. And then, you just copy and paste the stuff, and things explode".
Companies across industries adopt AI at the same rate
Industries adopt technologies at vastly different speeds and timescales. However, AI shows no significant difference in adoption rates across industries. What is notable is that organization size predicts AI adoption, with smaller organizations being more aggressive in AI adoption than larger organizations.
As with AI, the adoption of platform engineering leads to both positive and negative effects
Internal developer platforms positively impact surveyed participants' individual performance and their organizations. The most significant factors noted in the report are:
Higher levels of individual and team productivity (8% and 10%, respectively). Productivity describes the extent to which an individual or team feels efficient.
An organization's overall performance rises by 6%. This measures the organization's ability to deliver software and business value quickly.
The negative impacts, however, were surprising to me. The report notes that there are two major adverse effects organizations leveraging internal developer platforms observe:
First, throughput (the time it takes for code to reach production and how often code is deployed to production) decreases by 8% compared to organizations not using a platform. Second, delivery stability (the change failure rate and the rate at which rework is needed after release) decreases by 14%.
Multiple forces are at work here. The decrease in throughput could be explained by additional automation steps performed by the platform that are required before release to production, like security checks and testing. Additionally, if a platform is mandatory for development teams, ill-fitted platform functionality could also decrease productivity and throughput.
The drop in delivery stability may be due to increased automation. It lowers the bar for developers to push code since more automation potentially spots more issues. The report also states that a platform might make rollbacks easier, "empowering teams to experiment and deliver changes, which results in an increased level of change failure and rework."
User-centricity leads to better products
User-centricity is the degree to which teams align their priorities to their users’ needs. User-centricity is positive across the board:
It makes developers happy by inducing a sense of purpose in their work.
It is the only factor that makes software delivery performance less critical to product success.
As the report states:
“We find that when organizations know and understand users’ needs, stability and throughput of software delivery are not a requirement for product quality. Product quality will be high as long as the user experience is at the forefront.”
Internal documentation, combined with user-centricity, is powerful
Besides user-centrality, the quality of internal documentation plays a significant role in product quality. As the figure below shows, the combination of both factors has substantial positive implications. The report states that this is because internal documentation can act as a magnifier for user feedback and propagate it across the team.
Unstable organizational priorities lead to decreases in productivity and increases in burnout
The authors write:
“We all know the feeling. You’ve spent the last few months working on a new feature. You know it’s the right thing to build for your users, you are focused and motivated. Suddenly, or seemingly so, the leadership team decides to change the organization’s priorities. Now it’s unclear whether your project will be paused, scrapped, Frankensteined, or mutated.”
Shifting priorities leads to disruptions, and constantly doing so has profound negative effects on teams and individuals, especially the burnout rate. Persistent priority changes and the accompanying uncertainties place a burden on employees.
Moving to the cloud may be harmful if the provided flexibility is not used
Only if organizations take advantage of the main characteristics of cloud computing (on-demand self-service, broad network access, resource pooling, rapid elasticity, and flexible infrastructure) can they make the most out of a transformation to the cloud. The report goes even further and states that organizations moving to the cloud and not using the full potential might be better off staying in a more traditional data center:
“Organizations may be better off staying in the data center if they are not willing to radically transform their application or service. Of course, to accomplish this, it is not simply adopting tools or technologies, but often an entire new paradigm in designing, building, deploying, and running applications.”
Organizations across all industries can achieve high levels of software delivery performance
Software delivery performance and the organization's industry are independent. Even though each industry has its challenges, there is no industry prone to high delivery performance and no industry where delivery performance can't be improved.
Conclusion
I found this year's report especially interesting because it addresses two widely discussed topics: the usage of Artificial Intelligence in software development and platform engineering. I was not surprised that both approaches have obvious drawbacks besides their upsides; however, some particular drawbacks (such as platform engineering being able to lower throughput and delivery stability in some cases) raised my eyebrow. The report hypothesized several reasons, which I found sensible in the first place. However, the topics and issues raised in the report will require me to do more thinking and writing in the future.