21K School
Website:
21kschool.com
Job details:
We are looking for a highly capable and experienced Senior Data Engineer with strong Databricks expertise to architect, build, and scale our enterprise data platform. This is a Contract-to-Hire (C2H) position. The role will take ownership of our Data Lake, ETL/ELT frameworks, data migration initiatives, and data integration architecture across multiple systems (ERP, CRM, LMS, Finance, third-party tools, etc.). The ideal candidate should be comfortable working in a scalable cloud-based environment and designing analytics- and AI-ready data infrastructure.
Key Responsibilities
1. Data Platform Architecture & Databricks Implementation
● Design and implement enterprise-grade Data Lake architecture.
● Build and manage scalable data solutions using Databricks.
● Design ingestion frameworks for structured, semi-structured, and unstructured data.
● Architect scalable, high-performance, and cost-optimized storage solutions.
● Define metadata management, data cataloging, and lineage strategies.
2. ETL / ELT Framework Ownership
● Architect and build scalable ETL/ELT pipelines using Databricks and Spark.
● Develop reusable pipeline components and transformation logic.
● Implement monitoring, logging, alerting, and failure recovery mechanisms.
● Optimize performance of large-scale distributed data processing workflows.
● Enable batch and near real-time data processing capabilities.
3. Data Migration & Transformation Leadership
● Lead large-scale data migration initiatives from legacy systems to modern data platforms.
● Plan validation, reconciliation, and rollback strategies.
● Ensure data integrity and compliance during migrations.
● Work closely with engineering teams for system transitions.
4. Data Modeling & Analytics Enablement
● Design dimensional data models (star/snowflake schemas).
● Build analytics-ready datasets for BI, dashboards, and executive reporting.
● Prepare ML/AI-ready datasets in collaboration with data science teams.
5. Integration & Cross-System Data Flow
● Integrate data from ERP, CRM, LMS, payment systems, APIs, and third-party vendors.
● Design robust data contracts between services.
● Ensure reliability and consistency across distributed systems.
6. Governance, Security & Quality
● Establish data governance standards and data quality frameworks.
● Implement validation, monitoring, and reconciliation mechanisms.
● Ensure compliance with data protection and security policies.
7. Leadership & Collaboration
● Provide technical mentorship to junior engineers.
● Participate in architecture reviews and strategic data planning.
● Drive best practices in documentation and coding standards.
Required Skills & Experience
● 3–5 years of experience in Data Engineering.
● Strong hands-on expertise in Databricks.
● Extensive experience with Spark / PySpark.
● Strong experience in designing scalable ETL/ELT pipelines.
● Proven experience in large-scale data migration projects.
● Strong SQL proficiency.
● Strong Python programming skills for data engineering.
● Experience with cloud platforms (AWS preferred).
● Strong understanding of Data Lake concepts (S3, Data Lake, object storage).
● Experience with workflow orchestration tools (Airflow/Prefect).
● Strong knowledge of data modeling principles.
Good to Have
● Experience with Data Lake architecture.
● Experience with Kafka or streaming pipelines.
● Exposure to CI/CD for data pipelines.
● Experience supporting AI/ML data workflows.
● Familiarity with BI tools (Power BI, Tableau, etc.).
● Experience in SaaS or EdTech domain.
Employment Type Contract to Hire (C2H)
Reporting Structure Reports to: AVP – Technology & AI
Location Indiranagar, Bangalore
Click on Apply to know more.