The Rise of DataOps – Atlan

Knowledge is getting even larger, and conventional knowledge administration simply doesn’t work. DataOps is on the rise, promising to tame in the present day’s chaos and context challenges.

Let’s face it — conventional knowledge administration doesn’t work. At the moment, 75% of executives don’t belief their very own knowledge, and solely 27% of knowledge tasks are profitable. These are dismal numbers in what has been known as the “golden age of knowledge”.

As knowledge simply retains rising in measurement and complexity, we’re struggling to maintain it underneath management. To make issues worse, knowledge groups and their members, instruments, infrastructure, and use instances have gotten extra numerous on the similar time. The result’s knowledge chaos like we’ve by no means seen earlier than.

DataOps has been round for a number of years, however proper now it’s on hearth as a result of it guarantees to resolve this downside. Only a week aside, Forrester and Gartner lately made main shifts towards recognizing the significance of DataOps.

On June 23 of this 12 months, Forrester launched the newest model of its Wave report about knowledge catalogs — however as an alternative of being about “Machine Studying Knowledge Catalogs” like regular, they renamed the class to “Enterprise Knowledge Catalogs for DataOps”. Every week later, on the thirtieth, Gartner launched its 2022 Hype Cycle, predicting that DataOps will absolutely penetrate the market in 2-5 years and transferring it from the far left aspect of the curve to its “Peak of Inflated Expectations”.

Google Trends data for "DataOps" searches since 2015
Google Developments knowledge on world searches for “DataOps” since 2015. The y-axis reveals “curiosity over time”, or a normalized model of search curiosity. 100 represents peak reputation for the time period within the given time and area.

However the rise of DataOps isn’t simply coming from analysts. At Atlan, we work with trendy knowledge groups around the globe. I’ve personally seen DataOps go from an unknown to vital, and a few firms have even constructed whole methods, features, and even roles round DataOps. Whereas the outcomes range, I’ve seen unbelievable enhancements in knowledge groups’ agility, velocity, and outcomes.

On this weblog, I’ll break down all the pieces it is best to find out about DataOps — what it’s, why it is best to care about it, the place it got here from, and how you can implement it.

What’s DataOps?

The primary, and maybe most vital, factor to find out about DataOps is that it’s not a product. It’s not a software. The truth is, it’s not something you should purchase, and anybody making an attempt to let you know in any other case is making an attempt to trick you.

As an alternative, DataOps is a mindset or a tradition — a approach to assist knowledge groups and other people work collectively higher.

DataOps generally is a bit arduous to know, so let’s begin with a number of well-known definitions.

DataOps is a collaborative knowledge administration follow centered on enhancing the communication, integration and automation of knowledge flows between knowledge managers and knowledge shoppers throughout a corporation.


DataOps is the flexibility to allow options, develop knowledge merchandise, and activate knowledge for enterprise worth throughout all know-how tiers from infrastructure to expertise.


DataOps is a knowledge administration methodology that emphasizes communication, collaboration, integration, automation and measurement of cooperation between knowledge engineers, knowledge scientists and different knowledge professionals.

Andy Palmer

As you may inform, there’s no normal definition for DataOps. Nevertheless, you’ll see that everybody talks about DataOps by way of being past tech or instruments. As an alternative, they give attention to phrases like communication, collaboration, integration, expertise, and cooperation.

In our thoughts, DataOps is de facto about bringing in the present day’s more and more numerous knowledge groups collectively and serving to them work throughout equally numerous instruments and processes. Its ideas and processes assist groups drive higher knowledge administration, save time, and scale back wasted effort.

Why do you have to care about DataOps?

The quick reply: It helps you tame the information chaos that each knowledge particular person is aware of all too nicely.

Now for the longer, extra private reply…

At Atlan, we began as a knowledge group ourselves, fixing social good issues with large-scale knowledge tasks. The tasks had been actually cool — we set to work with organizations just like the UN and Gates Basis on large-scale tasks affecting hundreds of thousands of individuals.

However internally, life was chaos. We handled each hearth drill that would presumably exist, resulting in lengthy chains of irritating cellphone calls and hours spent making an attempt to determine what went fallacious. As a knowledge chief myself, this was a personally susceptible time, and I knew it couldn’t proceed.

We put our minds to fixing this downside, did a bunch of analysis, and came upon the concept of “knowledge governance”. We had been an agile, fast-paced group, and conventional knowledge governance didn’t look like it match us. So we got here collectively, reframed our issues as “How May We” questions, and began an inside mission to resolve these questions with new tooling and practices. By bringing inspiration from numerous industries again to the information world, we stumbled upon what we now know as DataOps.

It was throughout this time that we noticed what the fitting tooling and tradition can do for a knowledge group. The chaos decreased, the identical large knowledge tasks turned exponentially quicker and simpler, and the late-night calls turned splendidly uncommon. And in consequence, we had been capable of accomplish much more with far much less. Our favourite instance: we constructed India’s nationwide knowledge platform, finished by an eight-member group in simply 12 months, a lot of whom had by no means pushed a line of code to manufacturing earlier than.

We later wrote down our learnings in our DataOps Tradition Code, a set of ideas to assist a knowledge group work collectively, construct belief, and collaborate higher.

That’s finally what DataOps does, and why it’s all the craze in the present day — it helps knowledge groups cease losing time on the infinite interpersonal and technical velocity bumps that stand between them and the work they like to do. And in in the present day’s economic system, something that saves time is priceless.

Atlan's DataOps Culture Code
A part of our DataOps Tradition Code

The 4 elementary concepts behind DataOps

Some folks wish to say that knowledge groups are identical to software program groups, they usually attempt to apply software program ideas on to knowledge work. However the actuality is that they couldn’t be extra totally different.

In software program, you’ve some degree of management over the code you’re employed with. In any case, a human someplace is writing it. However in a knowledge group, you usually can’t management your knowledge, as a result of it comes from numerous supply methods in a wide range of consistently altering codecs. If something, a knowledge group is extra like a producing group, remodeling a heap of unruly uncooked materials right into a completed product. Or maybe a knowledge group is extra like a product group, taking that product to all kinds of inside and exterior finish shoppers.

The way in which we like to consider DataOps is, how can we take one of the best learnings from different groups and apply them to assist knowledge groups work collectively higher? DataOps combines one of the best components of Lean, Product Considering, Agile, and DevOps, and making use of them to the sphere of knowledge administration.

DataOps: The best ideas from Lean, Agile, Product Thinking, and DevOps
The 4 elementary concepts behind DataOps


Key thought: Cut back waste with Worth Stream Mappings.

Although its roots return to Benjamin Franklin’s writings from the 1730s, Lean comes from Toyota’s work within the Nineteen Fifties. Within the shadow of World Struggle II, the auto business — and the world as a complete — was getting again on its ft. For automotive producers in all places, workers had been overworked, orders delayed, prices excessive, and prospects sad.

To unravel this, Toyota created the Toyota Manufacturing System, a framework for conserving sources by eliminating waste. It tried to reply the query, how are you going to ship the best high quality good with the bottom value within the shortest time? One among its key concepts is to remove the eight varieties of waste in manufacturing wherever attainable — from overproduction, ready time, transportation, underutilized employees, and so forth — with out sacrificing high quality.

The TPS was the precursor to Lean, coined in 1988 by businessman John Krafcik and popularized in 1996 by researchers James Womack and Daniel Jones. Lean centered on the concept of Worth Stream Mapping. Similar to you’d map a producing line with the TPS, you map out a enterprise exercise in excruciating element, determine waste, and optimize the method to keep up high quality whereas eliminating waste. If part of the method doesn’t add worth to the client, it’s waste — and all waste must be eradicated.

What does a Worth Stream Mapping really appear like? Let’s begin with an instance in the true world.

Lean: Example Value Stream Mapping
The Worth Stream Mapping for ordering a espresso in a restaurant

Say that you just personal a restaurant, and also you need to enhance how your prospects order a cup of espresso. Step one is to map out all the pieces that occurs when a buyer takes after they order a espresso: taking the order, accepting fee, making the espresso, handing it to the client, and so on. For every of those steps, you then clarify what can go fallacious and the way lengthy the step can take — for instance, a buyer having hassle finding the place they need to order, then spending as much as 7 minutes ready in line as soon as they get there.

How does this concept apply to knowledge groups? Knowledge groups are much like manufacturing groups. They each work with uncooked materials (i.e. supply knowledge) till it turns into a product (i.e. the “knowledge product”) and reaches prospects (i.e. knowledge shoppers or finish customers).

So if a provide chain has its personal worth streams, what would knowledge worth streams appear like? How can we apply these similar ideas to a Knowledge Worth Stream Mapping? And the way can we optimize them to remove waste and make knowledge group extra efficients?

Product considering

Key thought: Ask what job your product is de facto engaging in with the Jobs To Be Carried out framework.

The core idea in product considering is the Jobs To Be Carried out (JTBD) framework, popularized by Anthony Ulwick in 2005.

The simplest method to perceive this concept is thru the Milkshake Idea, a narrative from Clayton Christensen. A quick meals restaurant wished to extend the gross sales of their milkshakes, so that they tried plenty of totally different adjustments, reminiscent of making them extra chocolatey, chewier, and cheaper than rivals. Nevertheless, nothing labored and gross sales stayed the identical.

Subsequent, they despatched folks to face within the restaurant for hours, accumulating knowledge on prospects who purchased milkshakes. This led them to understand that just about half of their milkshakes had been bought to single prospects earlier than 8 am. However why? After they got here again the subsequent morning and talked to those folks, they realized that these folks had a protracted, boring drive to work and wanted a breakfast that they may eat within the automotive whereas driving. Bagels had been too dry, doughnuts too messy, bananas too fast to eat… however a milkshake was good, since they take some time to drink and preserve folks full all morning.

As soon as they realized that, for these prospects, a milkshake’s objective or “job” was to offer a satisfying, handy breakfast throughout their commute, they knew they wanted to make their milkshakes extra handy and filling — and gross sales elevated.

The JTBD framework helps you construct merchandise that folks love, whether or not it’s a milkshake or dashboard. For instance, a product supervisor’s JTBD is likely to be to prioritize totally different product options to realize enterprise outcomes.

How does this concept apply to knowledge groups? Within the knowledge world, there are two foremost varieties of prospects: “inside” knowledge group members who must work extra successfully with knowledge, and “exterior” knowledge shoppers from the bigger group who use merchandise created by the information group.

We are able to use the JTBD framework to grasp these prospects’ jobs. For instance, an analyst’s JTBD is likely to be to offer the analytics and insights for these product prioritization selections. Then, when you create a JTBD, you may create an inventory of the duties it takes to realize it — every of which is a Knowledge Worth Stream, and might be mapped out and optimized utilizing the Worth Stream Mapping course of above.


Key thought: Enhance velocity with Scrum and prioritize MVPs over completed merchandise.

In the event you’ve labored in tech or any “trendy” firm, you’ve most likely used Agile. Created in 2001 with the Agile Software program Growth Manifesto, Agile is a framework for software program groups to plan and observe their work.

The core thought in Agile is Scrum, an iterative product administration framework based mostly on the concept of making an MVP, or minimal viable product.

Right here’s an instance: in case you wished to construct a automotive, the place do you have to begin? You may begin with conducting interviews, discovering suppliers, constructing and testing prototypes, and so forth… however that may take a very long time, throughout which the market and world can have modified, and it’s possible you’ll find yourself creating one thing that folks don’t really like.

Agile: Benefits of a Minimum Viable Product (MVP)
Six ways in which an MVP helps enhance improvement

An MVP is about shortening the event course of. To create an MVP, you ask what the JTBD is — is it actually about making a automotive, or is it about offering transportation? The primary, quickest product to resolve this job could possibly be a motorbike slightly than a automotive.

The purpose of Scrum is to create one thing as fast as attainable that may be taken to market and be used to assemble suggestions from customers. In the event you give attention to discovering the minimal resolution, slightly than creating the best or dream resolution, you may study what customers really need after they check your MVP — as a result of they normally can’t categorical what they really need in interviews.

How does this concept apply to knowledge groups? Many knowledge groups work in a silo from the remainder of the group. When they’re assigned a mission, they’ll usually work for months on an answer and roll it out to the corporate solely to study that their resolution was fallacious. Possibly the issue assertion they got was incorrect, or they didn’t have the context they wanted to design the fitting resolution, or perhaps the group’s wants modified whereas they had been constructing their resolution.

How can knowledge groups use the MVP method to scale back this time and are available to a solution faster? How can they construct a delivery mindset and get early, frequent suggestions from stakeholders?

Agile can be utilized to open up siloed knowledge groups and enhance how they work with finish knowledge shoppers. It will possibly assist knowledge groups discover the fitting knowledge, carry knowledge fashions into manufacturing and launch knowledge merchandise quicker, permitting them to get suggestions from enterprise customers and iteratively enhance and adapt their work as enterprise wants change.


Key thought: Enhance collaboration with launch administration, CI/CD, and monitoring.

DevOps was born in 2009 on the Velocity Convention Motion, the place engineers John Allspaw and Paul Hammond offered about enhancing “dev & ops cooperation”.

The standard considering on the time was that software program moved in a linear movement — the event group’s job is so as to add new options, then the operations group’s job is to maintain the options and software program secure. Nevertheless, this discuss launched a brand new thought: each dev and ops’ job is to allow the enterprise.

DevOps turned the linear improvement movement right into a round, interconnected one which breaks down silos between these two groups. It helps groups work collectively throughout two numerous features through a set course of. Concepts like launch administration (imposing set “delivery requirements” to make sure high quality), and operations and monitoring (creating monitoring methods to alert when issues break), and CI/CD (steady integration and steady supply) make this attainable.

DevOps toolchain
The DevOps toolchain (created by Kharnagy on Wikipedia)

How does this concept apply to knowledge groups? Within the knowledge world, it’s simple for knowledge engineers and analysts to perform independently — e.g. engineers handle knowledge pipelines, whereas analysts construct fashions — and blame one another when issues inevitably break. As an alternative of options, this simply results in bickering and resentment. As an alternative, it’s vital to carry them collectively underneath a standard purpose — making the enterprise extra data-driven.

For instance, your knowledge scientists might depend upon both engineering or IT now to deploy their fashions—from exploratory knowledge evaluation to deploying machine studying algorithms. With DataOps, they’ll deploy their fashions themselves and carry out evaluation rapidly — no extra dependencies.

Going from DevOps to DataOps
Going from DevOps to DataOps

Observe: I can not emphasize this sufficient — DataOps isn’t simply DevOps with knowledge pipelines. The issue that DevOps solves is between two extremely technical groups, software program improvement and IT. DataOps solves complicated issues to assist an more and more numerous set of technical and enterprise groups create complicated knowledge merchandise, all the pieces from a pipeline to a dashboard or documentation. Study extra.

How do you really implement DataOps?

Each different area in the present day has a centered enablement perform. For instance, SalesOps and Gross sales Enablement give attention to enhancing productiveness, ramp time, and success for a gross sales group. DevOps and Developer Productiveness Engineering groups are centered on enhancing collaboration between software program groups and productiveness for builders.

Why don’t now we have an analogous perform for knowledge groups? DataOps is the reply.

Establish the top shoppers

Reasonably than executing knowledge tasks, the DataOps group or perform helps the remainder of the group obtain worth from knowledge. It focuses on creating the fitting instruments, processes, and tradition to assist different folks achieve success at their work.

Key consumers of DataOps
The important thing shoppers of DataOps

Create a devoted DataOps perform

A DataOps technique is best when it has a devoted group or perform behind it. There are two key personas on this perform:

  • DataOps Enablement Lead: They perceive knowledge and customers, and are nice at cross-team collaboration and bringing folks collectively. DataOps Enablement Leads usually come from backgrounds like Info Architects, Knowledge Governance Managers, Library Sciences, Knowledge Strategists, Knowledge Evangelists, and even extroverted Knowledge Analysts and Engineers.
  • DataOps Enablement Engineer: They’re the automation mind within the DataOps group. Their key power is sound data of knowledge and the way it flows between methods/groups, appearing as each advisors and executors on automation. They’re usually former Builders, Knowledge Architects, Knowledge Engineers, and Analytics Engineers.
How WeWork structured their DataOps function around 2 personas
How WeWork structured their DataOps perform round two key personas

Map out worth streams, scale back waste, and enhance collaboration

Firstly of an organization’s DataOps journey, DataOps leaders can use the JBTD framework to determine widespread knowledge “jobs” or duties, also referred to as Knowledge Worth Streams. Then, with Lean, they’ll do a Worth Stream Mapping train to determine and remove wasted effort and time in these processes.

In the meantime, the Scrum ideology from Agile helps knowledge groups perceive how construct knowledge merchandise extra effectively and successfully, whereas concepts from DevOps present how they’ll collaborate higher with the remainder of the group on these knowledge merchandise.

Learnings from the 4 movements behind DataOps
What knowledge groups can study from the 4 actions behind DataOps

Making a devoted DataOps technique and performance is much from simple. However in case you do it proper, DataOps has the potential to resolve a few of in the present day’s greatest knowledge challenges, save time and sources throughout the group, and improve the worth you get from knowledge.

In our subsequent blogs, we’ll dive deeper into the “how” of implementing a DataOps technique, based mostly on finest practices we’ve seen from the groups we’ve labored with — how you can determine knowledge worth streams, how you can construct a delivery mindset, how you can create a greater knowledge tradition, and extra. Keep tuned, and let me know when you’ve got any burning questions I ought to cowl!

To get future DataOps blogs in your inbox, join my e-newsletter: Metadata Weekly

Header picture by Chris Liverani on Unsplash

Similar Posts

Leave a Reply

Your email address will not be published.