Website:
scoutit.in
Job details:
We're looking for
Data Engineers!
Responsibilities
- Design, develop, and maintain scalable big data solutions using Spark and PySpark.
- Build and optimize ETL/ELT pipelines for large-scale data processing.
- Work with Google Cloud Platform (GCP) services including DataProc, BigQuery, Cloud Storage, Compute Engine, and Cloud Functions.
- Develop workflow orchestration pipelines using Apache Airflow.
- Create and optimize complex SQL queries for data transformation and analytics.
- Process structured and unstructured datasets using Hadoop ecosystem technologies.
- Collaborate with cross-functional teams including Data Analysts, Architects, and Business stakeholders.
- Ensure data quality, reliability, performance, and scalability of data platforms.
- Implement best practices for cloud infrastructure, security, monitoring, and deployment.
- Support CI/CD and containerized deployments using Docker and Kubernetes.
Must-Have Skills
- Spark
- PySpark
- Google Cloud DataProc
- Apache Airflow
- Proficiency in SQL and database optimization.
- Familiarity with Hadoop and Hive.
- Strong knowledge of Python for scripting and data engineering tasks.
- Familiarity with Google Cloud Platform (GCP).
- Proficiency building scalable distributed data processing systems.
Nice to Have
- Experience with real-time data processing.
- Certifications in Google Cloud Platform (GCP).
- Exposure to Agile/Scrum methodologies.
(
*Note: This is a requirement for one of Scoutit's clients)
Skills: google cloud,google cloud platform,data processing
Click on Apply to know more.