AgileEngine
Website:
agileengine.com
Job details:
🔥 Data Engineer (Senior) – Mumbai / Pune / Bangalore
🚀 Hybrid Opportunity | 6-8 Years Experience | Financial Data & Google Cloud Platform
We're looking for a strong Data Engineer to join a globally strategic data modernisation programme at one of the world's leading investment intelligence firms. You'll design, build and maintain state-of-the-art data pipelines on GCP as part of a platform that powers investment decision tools used across the globe.
This is a high ownership, high impact role — not just another pipeline job.
---
✅ Must-Have Skills:
• 6-8 years of hands-on data engineering experience
• Strong Python programming — pipelines, transformation logic and automation
• Proficient in SQL with strong hands-on BigQuery experience — partitioning, clustering, materialised views and time-series query patterns at scale
• Hands-on experience with Cloud Composer (Apache Airflow) — DAG authoring, SLA alerting, retry logic and dependency management
• Working knowledge of Dataproc (Apache Spark) — batch ingestion, Delta Lake merge operations and incremental data processing
• Familiarity with GCP technologies — Cloud Storage, Pub/Sub, Datastream, Cloud Monitoring, IAM and VPC Service Controls
• REST API experience — consuming external vendor APIs and building service integrations
• Git based collaboration — branching strategies, PR workflows and pipeline-as-code
• AI assisted development tools — GitHub Copilot, Cursor or equivalent
• Strong sense of ownership across ingestion, QA, correction management and audit trails
• Excellent communication skills — you'll work with global cross functional teams across engineering, compliance and business
💼 Key Responsibilities:
• Build and maintain scalable distributed data pipelines on GCP including BigQuery based lakehouse layers and Dataproc driven Delta Lake workflows
• Design and implement bitemporal data models on BigQuery to support certified regulatory grade time-series datasets
• Build and maintain software testing frameworks — unit, non-regression and user acceptance — for pipelines and transformation logic
• Acquire, normalise, transform and release large volumes of financial market data through the OMDP data factory
• Support AI solution integration using Vertex AI — including AI assisted ingestion, anomaly detection and semantic search over the lakehouse
• Collaborate actively with stakeholders across data engineering, compliance and business teams globally
• Contribute to shared platform services — this is a platform role, not a vertical specific one
➕ Good to Have:
• Experience with pandas, PySpark or equivalent data manipulation libraries
• Familiarity with Dataplex for data discovery, lineage, policy tagging and data quality rule management
• Understanding of Change Data Capture patterns using Datastream for replicating transactional data into BigQuery
• Understanding of bitemporal data modelling concepts within BigQuery's append optimised design
• Knowledge of financial reference data — equities, fixed income, corporate actions or index composition
• BigQuery cost management — slot reservations, query cost controls and workload isolation
• Exposure to CI/CD pipelines and infrastructure as code using Terraform for GCP deployments
• Prior experience with LLMs and Agentic AI using Vertex AI — anomaly detection, semantic search or natural language querying over structured data is a strong plus!
---
📋 Quick Check Before You Apply:
6-8 years in data engineering with strong Python, SQL, and hands-on GCP experience — specifically BigQuery, Cloud Composer, and Dataproc? Comfortable working with large volumes of financial data in a global cross-functional environment? Yes to all — apply. No GCP or BigQuery hands-on experience? This one's not for you.
---
📩 Interested candidates, please share:
1. Email ID
2. Relevant Experience
3. CCTC / ECTC
4. Notice Period
⚠️ Please apply only if your experience aligns with the requirements. Candidates with GCP and financial data experience will be prioritised.
Click on Apply to know more.