Analytics Engineering Everywhere

By Jason Ganz, Special Adviser, Data for Progress.

Analytics Engineering — An Introduction

There’s a quiet revolution happening in the world of data. For years we have been blasted with nonstop articles about “The Sexiest Job of the 21st Century” — a data scientist. A data scientist, we have been taught, is a figure of almost otherworldly intelligence who uses quasi-mystical arts to perform feats of data wizardly. But these days, if you talk to the people who watch the data space most closely — there’s a different data role that has them even more excited.

To be clear, there are some very real and very cool applications of data science that can allow organizations to do things with data that can completely transform how their organization operates. But for many orgs, particularly smaller organizations without millions of dollars to invest, data science initiatives tend to fall flat because of the lack of a solid data infrastructure to support them.

While everyone was focused on the rise of data science, another discipline has been quietly taking shape, driven not by glitzy articles in Harvard Business Review but by the people working in the trenches in data-intensive roles. They call it the analytics engineer.

An analytics engineer is someone who brings together the data-savvy and domain knowledge of an analyst with software engineering tooling and best practices. Day to day, that means spending in a suite of tools that is becoming known as “The Modern Data Stack” and particularly dbt. These tools allow analytics engineers to centralize data and then model it for analysis in a way that is remarkably cheap and easy compared to how the ETL of traditional Business Intelligence teams of the past operated.

While data scientists are seen by some as wizardry, the attitude of the analytics engineer is a little different. You’ll hear them refer to themselves as everything from “humble data plumbers’’ to “just a pissed off data analyst.” The work of an Analytics Engineer seems easy to understand, almost banal. They combine data sources, apply logic, make sure there are clean and well-modeled materializations to analyze.

It turns out analytics engineering is a goddamn superpower. Anyone that has worked in, well, basically any organization knows that a tremendous amount of effort goes into standardizing data points that feel like they should be a no-brainer to pull, while more complex questions just sit unanswered for years. Analytics Engineering allows you to have data systems that just work.

A good analytics engineer is hugely impactful for an org, with each analytics engineer being able to help build a truly data-driven culture in ways that would be challenging for a team of people using legacy tools. While in the past there was tremendous repetitive work to do any simple analysis, Analytics Engineers can build complex data models using tools like dbt and have analysis-ready data tables built on any schedule. While before it was impossible to get anyone to agree on standard definitions of metrics, Analytics Engineers can simply build them into their codebase. And in the past, people struggled with incomplete and messy data, and Analytics Engineers… still struggle with incomplete and messy data. But at least we can have a suite of tests on our analytics systems to know when something has gone wrong!

The Rise of Analytics Engineering

You might think that this development would be scary for people working in data — if one analytics engineer is substantially more impactful than a data analyst, won’t our jobs be at risk? Could an org replace five data analysts with one Analytics Engineer and come out ahead?

But the fact of the matter is that no data analyst, anywhere, has ever come close to performing all of the analysis they think could be impactful at their organization — the opposite is far more likely to be the problem. Most data orgs are begging for additional headcount.

As analytics engineers increase the amount of insight organizations can find from data, it actually becomes more likely that these orgs will want to hire additional data talent (both analytics engineers and analysts). In his fantastic post The Reorganization of the Factory, Erik Bernhardsson makes the case that as the toolsets for software engineers has become ever more efficient, the demand for software engineers has counterintuitively grown — as there are more and more use cases where it now makes sense to build software rather than a manual process. This point not only holds for data, but I think it actually is more true for data.

While every organization needs software, not every organization needs software engineers. But every organization needs to learn from their data, and since the ways in which the data needs to be understood will be unique at every organization, they will all need analytics engineers. Software is commonly said to be eating the world — analytics engineering will be embedded in the world. As the incremental value of each data hire rises, there are substantial new areas where data insights and learnings could be applied that they aren’t today. And even if you aren’t interested in becoming an analytics engineer, having well modeled and accurate data makes data analysts and data scientists more effective. It’s a win across the board.

That does not necessarily mean that every analytics engineering role will be doing good for the world. Having more powerful data operations allows you to question, seek insights, and look for new strategies. It can also allow an organization new ways to monitor their employees, surveil, or discriminate. One needs only look at the myriad of public issues in the tech and data science industries right now to see the ways that powerful tech can be misused. It is important to recognize the potential dangers as well as the new opportunities.

If it feels like we’re at a real inflection point for Analytics Engineering — it’s because we are. What was very recently the domain of a few adventurous data teams is quickly becoming industry standard for tech organizations — and there’s every reason to think that other types of organizations will be following along shortly. The impact is just too high.

We’re about to see a huge expansion in the number of and types of places where you can find employment as an analytics engineer. The coming boom in opportunities for analytics engineers will take place across three rough domains, with each having different challenges and opportunities.

• More and more large enterprises, both tech and non-tech organizations, are going to adapt to the modern data stack. As analytics engineering is brought into the most complex legacy data systems, we’ll begin to see what patterns develop to support analytics engineering at scale. If you are interested in really figuring out what the large-scale data systems of the future look like, this will be the place to go.
• Just about every new company is going to be searching for an analytics engineer to lead their data initiatives. This will give them a step up against any competition that isn’t investing in their core data. Being an early analytics engineer at a fast-growing company is tremendously fun and exciting, as you are able to build up a data organization from scratch and see firsthand how analytics engineering can change the trajectory of an organization.
• Finally, many organizations outside the tech business world are going to begin seeing the impact that analytics engineering can bring. You might not have quite the same tech budget, and you might have to learn to advocate for yourself a little more but it might be the area where analytics engineering has the most potential to do good for the world. City governments will use analytics engineering to monitor programs and ensure that government resources are being used effectively. Academic institutions will use analytics engineering to create datasets, many of them public, that will aid in scientific and technological development. The possibility space is wide open.

Analytics engineering is fundamentally a discipline that’s about making sense of the world around us. It’s about allowing everyone in an organization to see a little bit further in their impact on the org and how their work connects to it. Right now, analytics engineering is still a new discipline — pretty soon, it will be everywhere.

Original. Reposted with permission.

Related:

• Why You Should Consider Being a Data Engineer Instead of a Data Scientist
• 9 Skills You Need to Become a Data Engineer
• Data Engineering — the Cousin of Data Science, is Troublesome