RBST Consulting
Company:
https://www.linkedin.com/company/rbst-consulting
Industries: Business Consulting and Services
Job details:
GCP Data Engineer
Level: Manager
Experience: 10–13 Years
Location: Any Metro Location
Notice Period: Immediate to 30 Days / Currently Serving Notice
Compensation: Up to ₹35.5 LPA Fixed
Role Overview
We are looking for an experienced GCP Data Engineer / Data Engineering Manager with 10–13 years of experience in data engineering and strong expertise in the Google Cloud Platform (GCP) ecosystem.
The role involves designing and delivering scalable cloud-based data platforms, optimizing large-scale Spark and Dataproc workloads, managing data orchestration platforms, and providing technical leadership to engineering teams.
The ideal candidate should combine strong hands-on expertise in GCP, Python, Spark, Dataproc, BigQuery and Cloud Composer/Airflow with experience in performance engineering, cost optimization, architecture and stakeholder management.
Key Responsibilities
GCP Data Engineering & Cloud Platform
- Design, develop and implement scalable data engineering solutions on Google Cloud Platform.
- Build data solutions using Cloud Run, Cloud Functions with Python, Cloud SQL PostgreSQL, Pub/Sub, GCS and BigQuery.
- Develop and maintain robust data pipelines and cloud-native data processing solutions.
- Design and manage orchestration workflows using Cloud Composer / Managed Airflow.
- Perform Airflow/Composer optimization, capacity planning and performance tuning.
- Drive FinOps and cloud cost optimization initiatives across data workloads.
- Support Databricks-based data processing and integration.
- Design and integrate REST APIs across cloud applications and data platforms.
- Ensure data platforms are scalable, reliable, secure and cost-efficient.
Performance Engineering & Dataproc Optimization
- Execute workload testing and performance optimization activities for large-scale data workloads.
- Enhance and optimize Python-based execution templates.
- Troubleshoot Spark, YARN and Dataproc failures using Spark UI and platform diagnostics.
- Analyze workload profiles and establish cost-performance benchmarks.
- Define N1/N2 mapping strategies and translate custom machine requirements into standard GCP machine types.
- Recommend appropriate VM sizing, schema design, partitioning and performance improvements.
- Identify bottlenecks across Spark, Dataproc and underlying GCP infrastructure.
- Optimize workloads for both performance and cloud cost efficiency.
Technical Leadership
- Provide technical leadership and architecture guidance to data engineering teams.
- Review technical designs, solutions and implementation approaches.
- Mentor engineers and drive engineering best practices.
- Work closely with business stakeholders, architects and cross-functional teams.
- Own technical delivery and ensure solutions meet performance, scalability and business requirements.
- Participate in solution design, estimation, planning and delivery governance.
Required Technical Skills
Mandatory
- 10–13 years of overall experience in Data Engineering.
- Strong hands-on experience with Google Cloud Platform (GCP).
- Strong expertise in:
- Python
- Apache Spark
- Dataproc
- Google Cloud Storage (GCS)
- BigQuery
- Cloud Data Architecture
- Experience with:
- Cloud Run
- Cloud Functions
- Cloud SQL – PostgreSQL
- Pub/Sub
- Cloud Composer / Apache Airflow
- Strong understanding of Spark performance tuning and debugging.
- Experience troubleshooting YARN and Dataproc issues.
- Exposure to workload benchmarking, capacity planning and cloud cost optimization.
- Experience with Airflow administration and optimization.
- Strong understanding of data pipeline architecture and large-scale data processing.
Good to Have
- Experience with Databricks.
- Experience designing and consuming REST APIs.
- Knowledge of GCP FinOps / cloud cost governance.
- Experience with custom machine sizing and workload-to-machine-type mapping.
- Experience with enterprise-scale cloud data migration or modernization programs.
Leadership & Behavioral Skills
- Strong stakeholder management and communication skills.
- Ability to lead technical discussions with senior stakeholders.
- Strong analytical and problem-solving abilities.
- Ability to work independently and lead engineering teams.
- Strong solution-design and architecture mindset.
- Ability to balance performance, scalability, reliability and cost.
Candidate Profile
We are looking for candidates who have progressed beyond traditional ETL development and possess strong expertise in cloud-native data engineering and performance optimization.
Candidates with strong hands-on experience in GCP + Spark + Dataproc + Python + BigQuery + Cloud Composer/Airflow will be preferred.
Click on Apply to know more.
