What can data teams learn from Google’s State of DevOps report? | Dataform
Opinion

What can data teams learn from Google’s State of DevOps report?

Google's research shows us that elite engineering teams have a lot in common when it comes to DevOps. We think similar principles apply to data teams or, more precisely, DataOps.

Data teamsEngingeering
picture of author

Dan Lee on October 8, 2020

introduction

For the last few years, Google has released a “State of DevOps” report. The report aims to describe how the most DevOps is done at the most effective engineering teams, and provides a set of benchmarks across industries and verticals. Google claims that there are four key metrics that can be used to categorize developer teams into groups - and they call the very best teams Elite Performers. The metrics provide an actionable way for DevOps teams to understand where they can improve.

Can data teams “steal with pride” and learn something about their own performance using these metrics? I think they can. Whilst they were developed with DevOps in mind, the reality is that the difference between a software engineering team and a data team is not that great (in the abstract). With a little tweaking, the same metrics can be used to rank and benchmark data teams.

The metrics

These are the four metrics used to understand DevOps performance:

  • Lead time: How long does it take to go from code committed to code successfully running in production?
  • Deployment frequency: How often does your organization release code to end users?
  • Time to restore: How long does it generally take to restore service after a defect that impacts users occurs?
  • Change fail percentage: What percentage of releases to users result in degraded service and subsequently require remediation?

How can these be reworded to assess data teams DataOps performance?

Lead time

The key question here is, how long does it take to go from code change made, to code change experienced by end users.

How long does it take to go from a change made to a the definition of a data model, to that data model reflecting the new definition for your end users (e.g. in a BI tool).

Deploy frequency

This one doesn’t need much rewording.

How often does your team release updates to data models to end users?

Time to restore

Defects for data teams are data quality issues, so we can be quite specific here.

How long does it generally take to fix a data quality issue once it has been reported to you?

Change fail percentage

Whilst this may be a little harder to measure, the wording doesn’t need too much of an update.

What percentage of changes to your data model result in a data quality issue, or failing pipeline, that requires remediation?

What does good look like?

For the State of DevOps report, Google carried out significant research, speaking to developer teams at many different organizations. To get a true sense of “what good looks like”, we need to do the same for data teams.

That said, for all of these metrics, the best teams will be aiming as low as possible:

  • Changes to data models with a minimal lead time
  • Changes deployed multiple times per day
  • Issues resolved within hours, not days
  • A low (sub 10%) percentage of releases leading to data quality issues

Take the survey!

If you’d like to contribute to our efforts to create a comprehensive set of benchmarks for data teams, take our survey! We’ll take your email address, and as soon as we’ve got a statistically significant number of results we’ll send you the benchmark data, showing you how your team compares.

Achieving Elite status with Dataform

Whilst we didn’t have the State of Data Ops report in mind when we started developing Dataform, the fundamental idea behind the product was always to bring DevOps best practices to the data team.

Dataform is a one-stop-shop for managing a data warehouse in an automated, dependable and continuously deployed manner. Teams using Dataform are operating at Elite status by definition, because each of these principles is built into the core of the platform.

If you’d like to learn more about how Dataform can help your team reach Elite status, get in touch, join our Slack workspace, or sign up for a free trial and see how Dataform works.

More content from Dataform

Cut data warehouse costs with run caching illustration
Product Update

Cut data warehouse costs with run caching

How to save time and money by using our run caching feature
Learn more
Building the Dataform VS Code extension illustration
Guide

Building the Dataform VS Code extension

How we made our own extension for Visual Studio Code.
Learn more
The startup data stack starter pack (2020) illustration
Opinion

The startup data stack starter pack (2020)

Data collection, integration, warehousing, modeling and visualization. What are the options, best of breed products and how much do they all cost.
Learn more

Learn more about the Dataform data modeling platform

Dataform brings open source tooling, best practices, and software engineering inspired workflows to advanced data teams that are seeking to scale, enabling you to deliver reliable data to your entire organization.