ETL Dashboarding

Business Problem

How many rows do your extract, transform, load (ETL) operations generate every day? How often does something go wrong? How long does it take before you can detect a problem, such as a load failure? When you extract thousands of records a day from multiple data sources, transform them with business rules, and load them into a data store, you hope nothing goes wrong. If something does go wrong, you hope you can correct it before someone down the line makes a decision that relies on the error. ETL saves you huge amounts of labor every day, but it shouldn’t be a black box that you continually hope is running right. And you shouldn’t need an entire IT project to implement tools that show you what’s happening inside that black box.

Analytics Solution

With analytics and workflows, you can orchestrate your own ETL processes and create dashboards and alerts. The dashboard gives database administrators and business intelligence managers high-level insight into the status of the many operations your data undergo on the way to the data store. It monitors how rows are filtered, sorted, aggregated, joined, cleaned, deduplicated, and validated, and sends alerts in case of load failures. Managers can specify the performance indicators they’re most interested in, such as total number of loads, rows written, total data stored, and load time. They can use analytics to generate reports that list the data fields with errant values, along with recommended corrections. To support user training and process improvement, they can design reports that identify trends in defects.