Every week, there’s a new hot jobs list. Data engineering, data science, or data analytics roles tend to appear near the top. Amid the “Great Resignation,” people across industries and career paths are considering what’s next: a new company, a new career, back to school to learn new skills, or maybe a different option. Constant innovation in the technology industry coupled with the shift in workforce leaves many gazing a bit deeper in the crystal ball to see what’s next, what’s hottest, and what will take us by surprise.
It wasn’t long ago that “Data Scientist” was the perennial darling to top the charts of hot jobs, and it’s not surprising to see the title bouncing around the top today. Understanding and using data is critical for more industries than ever before, with new use cases and tools developing daily. Thus, data jobs will stay at the forefront for years to come. But, what term follows “data” will evolve.
Meet data engineers. Well, it may not be a brand-new introduction; the title first popped up in the mainstream in the late 2010s. But, as we look towards the mid-2020s, data engineers will hold the keys to the future of business.
For those working with data tools and analytics tools daily, who live ELT/ETL and dream in SQL or Python, perhaps the nuances in the titles are apparent. But, for those considering the career shift, return to school, or even online classes such as Coursera or Udacity to acquire the proper skills, you may be asking: What’s data engineering vs. data science vs. data analytics?
Data Engineer Job Description: Enabling Repeated Processes
So, what do data engineers do? Data engineering is the institutional process of making sure data science and data analytics can happen. It’s the work that gets done to set up repeatable processes around data in an organization. Typically, the work spans multiple actors across the organization rather than being a solitary task.
- Function/Tasks: data ingestion, data transformation, data preparation, data profiling, automation & orchestration of data pipelines
- Common Tools: Trifacta, Alteryx (for on-premises), Fivetran, Matillion, Talend, AWS Glue DataBrew, Google Data Fusion, Tableau Dataprep, Snowflake
- Key Responsibilities: collecting, collating, extracting, moving, transforming, cleaning, integrating, organizing, representing, storing, and processing data
Gartner Research recently published a note titled “How to Build a Data Engineering Practice That Delivers Great Consumer Experiences,” in which the firm explains that organizations with mature data engineering practices see benefits such as:
- Faster time to delivery when adding new data to existing analytics and data science models
- The ability to incorporate third-party data more quickly than their peers
- Easier fulfillment of regulatory requirements to meet data transparency expectations
- Business teams that are empowered with composable D&A applications
Skills for data engineering involve planning for things that happen over time and ongoing processes instead of a one-time exploration of a data set, which might be more like data science.
Data Science: Sophisticated Statistical Analysis
Data science is a relatively intensive mathematical or statistical look at the data, sometimes for building AI, machine learning, or predictive models. People in these roles use data tools to forecast what will happen in the future, whether that means recommending products to users based on what you think they’re going to do or detecting fraud. Data scientists look at situations where the truth isn’t known, so they use a predictive model.
- Primary Function: extract insights from data, machine learning, AI/ML
- Common Tools: Apache Spark, Jupyter Notebooks, Python, R, SAS, Vendor products including Trifacta, DataRobot, Dataiku, Amazon SageMaker, Google Vertex AI, Microsoft Azure ML
- Key Responsibilities: Exploratory analysis, Determining correct data sets, Validating and interpreting data, Devising and applying data models
Data science is part machine learning and AI, part sophisticated statistical analysis. Often, people say they are doing data science, and they don’t mean either of those things. They usually give a data engineer job description or a description of data analytics.
Data Analytics: Visually Extracting Insights
Data analytics is like the extended version of business intelligence (BI). These are the people that are building charts, dashboards, and reports with specialized data tools. Data analysts take data and extract insights in a digestible way for business users. These are the outputs shown to decision-makers when they say, “Show me the numbers.”
- Primary Function: data visualization, analysis, transformation, cleansing
- Common Tools: Tableau, Power BI from Microsoft, Excel,
- Key Responsibilities: creating charts, graphs, extracting insights, data mining
There are a few main points when comparing and contrasting the data engineer job description vs. that of a data scientist or a data analyst. Generally, data analysts may have started in the line of business or have business training such as an MBA. Data scientists often come from more technical science training and skills and have degrees in computer science or statistics. Finally, data engineers often come from the software engineering world or enhance their skills from a background in data analytics.
Hierarchy of Data Needs from Data Engineering to Data Science
No matter the title, a few steps must occur before any data worker can perform meaningful data tasks. Consider Maslow’s Hierarchy of Needs. It’s not a new concept to apply this to the data lifecycle, but it is becoming increasingly critical as companies have more access to more data than ever before.
In the data space, machine learning and artificial intelligence are at the top of the pyramid. It’s wonderful to do, but data scientists can only be truly influential if the pyramid base is solid. What’s the point of having data scientists and tools if the data isn’t usable?
At the base of the data hierarchy is compute and storage. The cloud, a.k.a. Cloud Data Warehouses have made the bottom layer much more accessible. With its nearly infinite compute, storage, and scale, the cloud has made the bottom layer easily accessible for any requirement. With the cloud becoming table-stakes, flexibility, scalability, and ease-of-use have become the new order.
In the middle of the pyramid is data engineering. The data engineer job description is moving the data out of the cloud and making it usable to people doing data analytics and data science. Data engineers ensure there is fresh, clean, usable data. It can’t just be data from years ago, but rather repeatable pipelines of usable data daily or weekly.
Then comes data analytics, with data science being the most advanced use at the top of the pyramid.
https://www.simplypsychology.org/maslow.html
The Evolution of the Data Engineer Job Description
Data engineers, data scientists, and data analysts all have a role in the modern data world, and the key to highly effective data management is collaboration. It’s helpful to understand how these roles evolved to better understand the future.
Data is a massive opportunity to extract value and be competitive, and this way of thinking came to the forefront over the past two decades. Amazon, Netflix, Google, and many others including new entrants such as Uber are leading the way in the effective use of massive amounts of data. These are just a few examples of organizations relying on data to make informed decisions to scale their business. This led to personalized recommendation systems and retention prediction systems, all based on AI and machine learning. The way organizations began to look at data was somewhere on the boundary between analytics and data science.
As data science’s popularity grew, the story was clear: there would not be enough data scientists to keep up with demand, and that would cause companies to lose to the competition. In turn, universities focused on data science. Workers with backgrounds in physics or biology began to train in data science. But, soon, organizational leadership discovered that many data scientists couldn’t actually deliver value in the way they envisioned because the data wasn’t ready for them. Without crisp data preparation and data operations, data scientists couldn’t enact repeatable processes.
This led to a period where the common concern was the amount of time data scientists would need to spend on data wrangling, leading to data engineering emerging as a key skill. Data engineer skills are different from the ones that data scientists use for their goals. Organizations and universities began to realize all at once that data engineering was the first thing on companies’ minds.
What do data engineers do? It includes data preparation, understanding who owns the data and permissions to use it and establishing data pipelines. The data engineer job description can include dataops and dataprep topics, including ensuring complete data, correct formatting, and ample permissions to access the data.
Data analysts have long held a crucial role in making sense of data and extracting valuable insights. That role remains and becomes even more accessible as the modern data stack continues to evolve.
Building the Data Engineer Job Description
Effective data engineering begins with a commonplace in the cloud where people with different skills can come together and share the work. Businesses need a smooth handoff between the people who generate the data and manage pipelines and those who will use that data. There needs to be a shared environment in which people can collaborate without restrictions. Highly technical engineers can turn to code, while business-focused individuals can manipulate data with low code and no code visualizations. Even technical people should be constantly immersed in the visuals of the data, so they are aware of the state of the data without having to write code to see a chart. This sort of ideal balance can be achieved with Designer Cloud, the data engineering cloud.