A passenger charter airline and aircraft lessor was facing the following challenges with their existing ETL jobs:
- Testing large volume of data coming from disparate data sources into Data warehouse
- Continuous addition of new data sources and data elements leading to data quality issues in existing data warehouse
- Lack of audit trail and reporting providing visibility on data quality
The client wanted:
- End-to-end visibility of data quality in data warehouse environment
- A data health dashboard providing detailed insight about data loaded and its quality
- The ability to identify unusual patterns during ETL loads
- ETL testing automation processes which are simple to enhance and deploy
- Improved data quality
- Reduced time spent on debugging data related issues due to close monitoring of data health
Y&L:
- Implemented ETL automated testing via Informatica Data Validation option
- Analyzed ETL transformation rules and converted them into Test cases
- Created test cases and scenario development for several data health checks
- Designed and developed ETL testing jobs and automated the data comparison process
- Developed data heath dashboards
- Implemented Performance Testing Automation
- Utilized Informatica, SQL Server, Tableau, and Query Surge