Group Lead - Data Quality Engineer
DP World
- Location
- Gurgaon, Haryana, India
- Job type
- Full-time
Required skills
- API
- Azure
- backend
- communication skills
- compliance
- Databricks
- end-to-end
- product lifecycle
- Spark
- SQL
About the role
DP World
Website:
dpworld.com
Job details:
Job Description
KEY ACCOUNTABILITIES
- Data Quality Policy & Framework Implementation
- Define and operationalize enterprise Data Quality policies, procedures, and standards.
- Establish standardized data quality dimensions and certification frameworks.
- Implement scalable validation frameworks across ingestion, transformation, and serving layers.
- Embed “quality-by-design” principles into data product lifecycle.
- Data Observability Platform Development
- Design and implement end-to-end data observability capabilities including:
- Data freshness and SLA monitoring
- Volume and distribution anomaly detection
- Schema drift and pipeline health monitoring
- Data lineage validation and reliability tracking
- Develop automated alerting and incident detection mechanisms.
- Custom Data Applications (DataApps) Development
- Build custom Data Quality and Observability applications using:
- Databricks native capabilities
- Streamlit / Databricks Apps
- Python-based backend services
- Develop user interfaces enabling:
- Data quality rule configuration
- Dataset certification workflows
- Quality score visualization
- Issue tracking and remediation workflows
- Enable self-service quality monitoring for engineering and analytics teams.
- Azure & Databricks Platform Integration
- Implement data quality checks within Azure-based data pipelines and Databricks workflows.
- Integrate monitoring with:
- ADLS Gen2
- Databricks Lakehouse architecture
- Batch and streaming pipelines
- Develop reusable frameworks leveraging Spark and Delta Lake.
- Optimize performance and scalability of quality validation workloads.
- Automation & Engineering Excellence
- Integrate DQ checks into CI/CD and deployment pipelines.
- Develop metadata-driven quality monitoring solutions.
- Implement automated remediation and self-healing workflows where applicable.
- Ensure auditability, traceability, and governance compliance.
- Metrics, Reporting & Adoption
- Define enterprise Data Quality KPIs and reliability SLAs.
- Build dashboards tracking platform-wide data trust scores.
- Drive adoption of standardized DQ practices across engineering teams.
- Support audit and compliance reporting initiatives.
- Data Quality Score
- Leadership & Collaboration
- Act as technical lead for Data Quality and Observability engineering.
- Mentor engineers on best practices for data reliability.
- Collaborate with Data Engineering, Governance, and Platform Architecture teams.
- Contribute to long-term evolution of the enterprise data platform.
Qualifications, Experience And Skills
Education
- Bachelor’s or master’s degree in computer science, Data Engineering, Information Systems, or related field.
Experience
- 8+ years of experience in Data Quality engineering roles within Data Platforms/Data Engineering teams.
- Proven experience building custom applications on Databricks or data platforms.
- Experience designing enterprise Data Quality or Data Observability solutions.
- Hands-on experience developing internal data tools or platform applications.
Technical Skills (Required)
Strong expertise in: - Microsoft Azure
- Databricks Lakehouse platform
- ADLS Gen2
- Distributed data processing using Spark
- Application & DataApp Development
- Experience building DataApps using:
- Streamlit
- Databricks Apps or notebook-based applications
- Python backend development
- Experience designing UI-driven data engineering tools or internal platforms.
- Data Quality & Observability
- Experience implementing data validation frameworks.
- Strong SQL and Python programming skills.
- Knowledge of anomaly detection, monitoring, and data reliability concepts.
- Engineering & Integration
- CI/CD integration for data pipelines.
- REST API integrations and automation workflows.
- Metadata-driven architectures and lineage concepts.
Core Competencies
- Platform-first engineering mindset.
- Strong problem-solving and analytical thinking.
- Ability to translate governance requirements into scalable technical solutions.
- Strong stakeholder collaboration and communication skills.
- Ownership mindset with ability to lead initiatives end-to-end.
Preferred
- Experience with Great Expectations, Deequ, Soda, or similar frameworks.
- Experience with streaming data validation.
- Exposure to AI-driven data observability or anomaly detection.
- Experience building enterprise internal developer platforms.
Click on Apply to know more.
This page is fully interactive when JavaScript is enabled. Please enable JavaScript to apply or browse related roles.