Bring on the Benchmarks

Publish date:

Businesses like to measure their return on investment in new software. So why not measure the return on investment in new software practices, like DevOps and agile?

The reasons why organizations invest heavily in major software projects are well known—and often easy to quantify. They include decreases in things like costs and time-to-market, and increases in things like productivity, market share, customer retention, and share of wallet. Even “soft” metrics like customer loyalty can be quantified. (One example: net promoter score, which is the percentage of workers or customers who recommend a company to others minus the percentage of those who tell others to stay away.) We also know that the software projects that are easiest to justify are those that invite clear before-and-after comparisons and also those where the relevance of these comparisons to a company’s market value is clear.

That’s why it astounds me that we IT project leaders don’t do more benchmarking of our own performance. Case studies are great, but what would be even more persuasive is if we were to show a potential client clear benchmarks of our firm’s ability to achieve positive outcomes (perhaps through third-party audits). And what about the tools and methodologies we advocate, like DevOps and agile? If the companies that follow our advice expect to see quantifiable outcomes, why don’t we expect the same when it comes to the tools and methodologies we ourselves adopt? What better way to encourage best practices across teams than to highlight successes based on real numbers?

Some Performance Benchmarks

Here’s a sample of the kind of benchmarking I’m talking about. Each year for the past five, Puppet, Inc., a software toolmaker, and DevOps Research and Assessment (DORA), an IT market research firm, has published a survey that compares “high performers” (i.e., IT organizations that fully embrace DevOps) versus low performers (i.e., those that don’t) on a number of statistical measures. For the 2016 State of DevOps Report, over 4,500 IT professionals were surveyed. Compared to low performers, high performers:

•         Have 200 times more frequent code deployments (multiple deploys per day versus one every 1-6 months).

•         Have 2,555 times faster lead times between code commit and code into production (<1 hour versus 1–6 months).

•         Have 24 times faster MTTR, i.e., mean time to repair code changes that failed (<1 hour versus < 1 day).

•         Have one-third the number of change failures (0–15% versus 16–30%).

•         Spend 22% less time on unplanned work and rework.

•         Can spend 29% more time on new work, such as new features or code.

•         Are 2.2 times more likely to be recommended as a great place to work by employees.

Benchmarks like these obviously make a great case for DevOps. After all, something must be causing such clear differences in how well high performers are able to perform. The numbers also provide compelling motivation—as in which group would you like to join? And they provide clear guidance. Low performers don’t become high performers overnight. Having benchmarks enables teams to see where progress is occurring and how fast, and also see if there might be some backsliding into old patterns. Objective evidence also gives team leaders more power to change team members’ behavior than if leaders merely rely on their authority. Team members can actually see for themselves how the changes are making a difference.

This Is Prime Time

Right now is an especially opportune time for SAP consultants to benchmark. The transition to SAP Activate marks the most significant change in years in how we conduct major SAP projects. Over the next couple of years we should start to see significant improvements in key benchmarks as teams gain experience in this new framework. We should therefore be able to document those improvements with clear before-and-after comparisons that show the move from waterfall to agile and from ASAP to SAP Activate is working. For many consultants with long experience in the older methodologies this transition will be especially challenging. Objective evidence that the new methodologies are better may just be the key that opens their door to success.

Just as we do with other tools, we should undertake a formal assessment of our benchmarking capabilities. Such an assessment should address questions as: What benchmarks are relevant? What tools do we already have in place that can capture these benchmarks? Who should do the benchmarking? What about using past projects as a base of comparison? What performance metrics are available on those and how easy would it be to capture them? How much of this work can be automated and where does automation already exist that we can leverage?

Obviously we need to strike a healthy balance between measuring our performance and actually performing. And as in all our activities, we need to see a return on investment. That’s true, but we also can’t expect to win if we don’t start keeping score.

Related Posts


No one likes waiting. With Continuous Delivery, now you don’t have to!

Venky Chennapragada
Date icon March 22, 2021

Automated and on demand – here’s how, why, and when you should adopt Continuous Delivery...


How to liberate your legacy applications to unleash powerful, agile next-gen apps

Erik Haahr
Date icon March 1, 2021

Discarding the burden of an existing traditional applications landscape will bring clearer...


Re-think authority and responsibility in the agile company

Nick van Zuijlen
Date icon February 22, 2021

To be successful, teams need quick turnaround on decisions.