EliteSquad.AI
Website:
elitesquad.ai
Job details:
Job Description: Data Engineer
Onsite: Seawood, Navi Mumbai
Full time
Share your CV at SANJAY.C@ELITESQUAD.AI
Role Overview
The Data Engineer is a hands-on implementation specialist responsible for building the actual "plumbing" of the database-to-database pipelines. This role focuses on the technical nuances of connectivity, memory-efficient transformations, and robust error handling to ensure zero data loss.
Key Responsibilities
- Pipeline Development: Build secure JDBC connections and develop the worker memory logic required for complex schema mappings (e.g., transforming Oracle DATE types to BigQuery DATETIME).
- DLQ & Fault Tolerance: Implement Dead Letter Queues (DLQ) within the Dataflow architecture to capture and preserve data that fails validation, ensuring a "Fail" path exists for future reprocessing without stopping the pipeline.
- Runtime State Management: Develop and maintain the BigQuery-based runtime state store for watermark persistence, ensuring that extraction points are accurately tracked and advanced only upon successful load.
- Audit & Metadata Enrichment: Programmatically inject audit metadata including extraction_timestamp and source_system_id to maintain full traceability.
- Reliability Engineering: Configure retry and restart behaviors that are safe for Raw layer appends, preventing data duplication during mid-stream failures.
Must-Have Technical Skills
- Data Processing: Strong Java or Python skills for Cloud Dataflow development.
- Storage & SQL: Proficient in BigQuery SQL and managing state in BigQuery metadata tables.
- Python : Strong proficiency in Python (3.x), with experience writing clean, modular, and testable code.
- Security: Experience applying "Least Privilege" principles using GCP IAM service accounts.
- Connectivity: Practical experience configuring JDBC drivers for Oracle and SAP DB.
- ETL Framework Design
- Data Migration Architect
Minimum Experience
- 6 years in data engineering.
- Proven experience implementing Oracle-to-BigQuery ingestion patterns and managing Dataflow-based DLQs.
Click on Apply to know more.