With the rapid growth of data, its tooling and its applications, numerous roles have emerged to manage data and derive value from it. One of the more recent roles within data teams is the analytics engineer, a.k.a the bridge between data engineering and data analysis.
In this blog, Analytics Engineer Yirong Lo answers the most frequently asked questions about what it means to be an analytics engineer:
What is the difference between a data engineer, data analyst, and analytics engineer?
One of the most frequently searched questions is how an analytics engineer differs from a data engineer or a data analyst.
Analytics engineers are data professionals tasked with ensuring that data is structured and transformed in a way that is easily decipherable by analysts, data scientists and/or business users.
Analytics engineers fill the gap between data engineering and data analysis by bringing modern engineering practices to the analytics workflow, which reflects the ever-changing landscape of this industry.
Data is a relatively new field, and therefore does not have a set way of working standards (yet). The complexity required to deliver impactful results has drastically increased. The addition of analytics engineering to this field is a natural evolution that comes with data maturity.
We saw a similar evolution in the role of a software engineer, where one role, i.e. an HTML developer, in the early internet days used to involve building static websites, but now involves backend engineers, frontend engineers, UX designers, QA testers, and others roles required to develop web and mobile applications.
To further illustrate the difference between a data engineer, data analyst and analytics engineer, let’s picture your organization’s data infrastructure as a library:
Data engineers lay the foundations of the library by constructing the physical building, and ensuring that it’s structurally sound and can house an ever-growing, vast collection of books. They are the architects that design and implement the layout, make sure books are arriving, and ensure that the library is equipped with all necessary amenities for visitors, from the entrance to the reading areas.
Analysts and data scientists are the library’s regular visitors, coming almost everyday. As avid readers, they delve deep into the content, gain insights and craft stories/solutions to the world outside. Then there are occasional visitors - business users - who may not always have the time to read in depth, but who value and rely on the library's resources.
Okay, so there’s a library and its various visitors, who could be missing?
The librarians, of course!
In our analogy, this role is filled by analytics engineers. Similar to a librarian organizing, ordering, and governing books, analytics engineers are tech-savvy librarians who use the latest tools to recommend books, summarize stories quickly, and ensure that readers have the right access to the right information.
So, what exactly does an analytics engineer do?
Analytics engineers transform raw data into usable formats used for analytical purposes. Analytics engineers are curators of the so-called "semantic layer", “gold zone” and/or “data marts” — a unified view on data that provides a consistent and business-relevant perspective on data assets.
The analytics engineer plays an essential role by ensuring that data is reliable and (re-)usable, improving data quality, maintaining trust in data, encouraging collaboration and streamlining data workflows.
This often requires mastery in tooling, such as a data manipulation language (i.e. SQL, Python, R), ELT-related tooling (i.e. Airflow, dbt), data discovery tools (i.e. Power BI, Tableau, DataHub), and a solid understanding of data warehousing (i.e. AWS, Azure, Snowflake). The goal of an analytics engineer is to enable self-service analytics from which analysts, data scientists and other business users can extract valuable insights.
What are signs that you need an analytics engineer in your organization?
It has become increasingly important to not only make data accessible, but also reliable, (re-)usable and trusted by businesses. This is where the analytics engineer shines.
Some tell-tale signs to recognize that perhaps you need an analytics engineer include:
- There seems to be a lack of data governance. Analysts and business have a hard time finding quality essential data, and questions such as the following are repeatedly asked: Where can I find this data? What does this table/column/value/metric mean? Where does this data come from? How is this metric calculated?
- My data engineers are swamped with ad hoc requests from the business, while they are also tasked to build/maintain/migrate the data platform
- My analysts depend heavily on data engineers and do not feel empowered
- My analysts or data scientists spend more time cleaning and prepping data than extracting insights or developing ML models
- My analysts spend more time fixing business-critical dashboards than generating new insights
- There are duplications of data, insights or dashboards and they show different numbers
- Due to quality and/or speed of insights, the business is unable to make data-informed decisions
- We have all this data, but we don’t know what to make out of it!
Many big players have recognized the value of Analytics Engineers and have incorporated the role into their teams. Examples include Meta, Google, Airbnb, Netflix, Shopify, Amazon, eBay, Stripe, Adyen, to name a few.
Xomnia's analytics engineering team help organizations like The City of Amsterdam, HEMA, APG, and others achieve their data-driven potential. Interested in learning more about what an analytics engineer can do for your organization? Contact Xomnia
What are signs that analytics engineering is the suitable role for you?
Let’s be honest, choosing a career is hard, and it is even harder if there are various different titles associated with similar tasks. Some tasks of an analytics engineer are also hidden in other roles, such as that of a data analyst, data engineer or any other type of business insight professionals.
All this aside, here are some reasons to consider becoming an analytics engineer:
- You are excited to dive deep into technology and data tools: Analytics engineers incorporate various tooling in their work. This includes being comfortable with or eager to learn programming languages like SQL and Python, and understanding databases and data platforms.
- You have a strong desire to optimize: “It works” is not sufficient for you. You strive for “it works efficiently.” Making outputs re-usable and streamlining workflows is what gives you joy.
- You have a collaborative spirit: Analytics engineers act as a bridge. Good communication and teamwork skills are essential. As much as you like coding and learning technologies on your own, you’re also very much curious and aware of what people around you are doing and what they may or may not need from you.
- You like to be at the intersection of business and technology: Perhaps you are currently an analyst who would like to incorporate engineering practices into your work or you are a (data) engineer who wants to move more towards the business.
As a former data analyst, I often found myself cleaning data, figuring out what table and column names mean, and navigating through data requirements across the business. Oftentimes, the required data was either non-existent or still in the collection phase, leading me to extensive liaising between data engineers and business stakeholders. The constant back-and-forth, along with often unclear governance procedures to secure access to the data meant that the actual analysis - the heart of an analyst role - took a backseat. This impacts the business’ ability to make informed decisions, even when you seemingly have the technology and people to do so. I’ve seen peers and other roles with various titles express similar frustrations. I realized that this gap can be filled by an analytics engineer, which is why I made the transition.
As an analytics engineer I’ve pivoted my focus towards making the entire data analytics workflow more efficient and scaling analytics more effectively. I often reflect and think that my ‘data analyst-self’ would have greatly benefited from collaborating with an analytics engineer. And now, I get to be that person!