Straive
Website:
straive.com
Job details:
About This Job
Straive
Location: Bangalore Urban, Karnataka, India
Work Mode: On-site
Industry: Business Consulting and Services
Job Description
Senior Data Engineer (AWS & Snowflake Specialist)
Experience Level: 5–6 Years
Role Overview
This role is for a data powerhouse who can bridge the gap between high-performance big
data processing on AWS and elite-level data warehousing in Snowflake. You will be
responsible for the entire lifecycle of data, ensuring that every pipeline includes robust
validation and observability.
Key Responsibilities
- Big Data Processing: Leverage AWS EMR (Spark/Hive) for heavy-lift data
processing and Athena for serverless ad-hoc querying of S3 data lakes.
- Data Validation: Implement Great Expectations (GX) or equivalent Python-based
validation libraries to prevent "data silent failure" within EMR and Snowflake ingestion
layers.
- Snowflake Engineering: Architect end-to-end Snowflake environments, utilizing
SnowSQL for scripting and SnowPipe for continuous, automated data ingestion.
- System Integration: Build and maintain Snowflake Integrations (Storage
Integrations with S3, API Integrations, and Notification Services).
- Advanced ETL: Develop robust pipelines that transition data from raw AWS storage
into optimized Snowflake schemas.
Technical Requirements
- Core Languages (Strong Proficiency Required):
○ SQL: Expert mastery of Snowflake-specific SQL extensions, CTEs, and
performance tuning (Micro-partitioning).
○ Python: Advanced scripting for automation, Boto3 for AWS resource
management, and data manipulation.
- Data Quality: Proven experience implementing automated data testing and
validation frameworks within production pipelines.
- AWS Stack: Extensive experience with EMR, Redshift, Athena, S3, and IAM
security policies.
- Snowflake Mastery: * Proven experience with Snowflake Data Engineering
features (Streams, Tasks, and External Tables).
○ Hands-on usage of SnowPipe for real-time or near-real-time data loading.
Click on Apply to know more.