Position Overview
The Architect owns the end-to-end delivery of the data engineering work on a flagship multi-year enterprise data transformation program. The program is building a unified, governed data foundation on Azure across multiple business domains, with real-time CDC ingestion, master data management, and AI-ready analytics. This is a builder-leader role: you act as the technical bridge between the customer's senior technology leadership and the Aubrant delivery team, write modeling decisions, get hands-on with Databricks and pipeline code as you lead a data engineering team, and pressure-test the team's QA approach yourself.
Delivery Ownership & Execution
- Own end-to-end delivery of the data transformation against agreed architecture, requirements, and schedule
- Translate the architecture and Unified Data Model into an executable plan: source onboarding, ingestion patterns, ELT design, serving patterns, and quality gates
- Drive sprint planning, milestone tracking, and execution across the program's phased delivery
- Identify risks, dependencies, and blockers early; drive resolution and manage scope and timeline commitments
Customer & Stakeholder Engagement
- Act as the day-to-day technical point of contact for customer leadership and engineering on progress, blockers, decisions, and solution alternatives
- Run technical working sessions, design reviews, and walkthroughs that move decisions forward
- Translate business context into technical implications, and technical complexity into clear leadership-ready summaries
Architecture, Modeling & Engineering
- Hold a working understanding of the full target tech stack and validate that implementation choices stay consistent with the reference architecture
- Lead and contribute to data modeling across the core enterprise domains; review modeling work for identity, SCD, CDC, PII, and survivorship correctness
- Build production-grade ETL/ELT pipelines on Azure Databricks (PySpark, Spark SQL) with Delta Lake: ingestion, conformance, survivorship, and quality test layers
- Configure and extend Airbyte connectors for CDC ingestion and integrate API-based sources across SaaS, ERP, HRIS, and operational systems
- Apply Aubrant Workbench accelerators to compress build time and ensure consistency
Infrastructure, DevOps & Quality
- Partner with the cloud and DevOps team on what the data team needs from the platform: workspace topology, network and identity, secret management, observability, and cost guardrails
- Ensure CI/CD pipelines for data assets are in place and used: unit and integration tests, lineage validation, environment promotion, automated deployment, and infrastructure-as-code discipline
- Define the QA approach: data quality rules, test data strategy, regression testing, reconciliation against sources, and acceptance criteria for golden records
- Instruct and review QA work; hold the line on quality gates between Bronze, Silver, Gold tiers and Dev, Test, Prod environments
Team Leadership & Coordination
- Lead and coordinate a cross-functional pod including:
- Support Agile ceremonies, backlog prioritization, and remove blockers
- Mentor Studio Members and codify reusable patterns into the Studio knowledge base and the Aubrant Workbench
Key Qualifications
Experience
- 12+ years in data engineering and data platform delivery, with 5+ years in a Technical Lead or equivalent role on customer-facing engagements
- Multiple end-to-end deliveries of enterprise-scale data platforms, with a track record of delivering against architecture, schedule, and quality
Required Technical Skills
- Azure Databricks (PySpark, Spark SQL), Delta Lake, the Medallion architecture, and ADLS Gen2: hands-on production experience
- Data modeling: conceptual, logical, and physical, including SCD strategy, CDC patterns, PII classification, and survivorship
- CDC and ingestion: production experience with Airbyte, Fivetran, Azure Data Factory, or equivalent, plus API-based source onboarding
- At least one of Azure Synapse, Cosmos DB, or Azure SQL Managed Instance for serving patterns
- CI/CD for data assets and infrastructure-as-code (Terraform, Bicep, or ARM)
- QA approach design and data quality engineering for enterprise data platforms
Leadership & Communication
- Customer-facing presence: able to run a technical conversation with a VP of Technology and walk out with a decision
- Strong written technical communication: design memos, decision logs, and runbooks
- Demonstrated ability to mentor engineers and grow technical capability in a team
Preferred Qualifications
- Databricks Certified Data Engineer Professional or Microsoft Azure Data Engineer Associate / Solutions Architect Expert
- Microsoft Purview or comparable governance and catalog tooling (Collibra, Atlan, Unity Catalog)
- MLflow lifecycle experience or GenAI / LLM integration patterns in production
- Exposure to regulated, compliance-heavy industries (HIPAA, SOC 2, GDPR, PCI DSS)
- Bachelor's or Master's degree in Computer Science, Engineering, or related field
Built on Azure Databricks, Delta Lake, ADLS Gen2, Airbyte, Microsoft Purview, Azure Synapse, Cosmos DB, MLflow, and Power BI.