Website:
bigtapp.ai
Job details:
Job Description – Senior Data Engineer - Pipeline
Location: Chennai / Hyderabad / Bangalore
Experience: 5-7 Years
Job Summary
We are seeking an experienced Senior Data Pipeline Engineer to design, develop, and maintain enterprise-grade data pipelines that support analytics, dashboarding, and AI initiatives.
The role involves integrating data from multiple enterprise systems such as SAP, SCADA, AMI, CMS, and APIs into a centralized analytics platform built on Elasticsearch.
The ideal candidate will have strong expertise in ETL/ELT pipeline development, data transformation, and data integration, ensuring high-quality, scalable, and reliable data solutions for business intelligence and advanced analytics.
Mandatory Skills
- Strong hands-on experience in Python for data engineering
- Experience in designing and implementing ETL/ELT pipelines
- Strong SQL and data modelling skills
- Experience working with REST APIs and enterprise data integrations
- Hands-on experience with Elasticsearch (indexing, querying, optimization)
- Experience integrating with SAP systems (RFC/BAPI/ODATA or APIs)
- Knowledge of pipeline orchestration and scheduling tools
- Strong debugging, troubleshooting, and performance optimization skills
Key Responsibilities
Data Pipeline Development
- Design, develop, and maintain scalable ETL/ELT pipelines
- Integrate data from enterprise systems such as:
- SAP
- SCADA
- AMI / HES
- CMS / QMS
- Portal applications
Data Integration & Processing
- Build pipelines using Python, SQL, and API-based frameworks
- Implement data validation, error handling, logging, and monitoring
- Optimize pipelines for performance, scalability, and reliability
Analytics Engine Support
- Develop business logic for:
- KPI computation
- SLA calculations
- Operational analytics
- Implement transformation logic for enterprise dashboards
Elasticsearch Platform
- Design and manage indices, mappings, and queries
- Optimize search performance and indexing strategies
- Manage data lifecycle and query efficiency
AI / ML Data Enablement
- Build pipelines to support AI/ML model training and deployment
- Ensure availability of high-quality, structured data
Collaboration & Support
- Work with analytics engineers, BI teams, and business stakeholders
- Support testing, validation, troubleshooting, and enhancements
- Maintain documentation and ensure knowledge sharing
Qualification
- Bachelor’s degree in Computer Science, IT, Data Engineering, or related field
Technical Skills
- Programming: Python, SQL
- Data Engineering: ETL/ELT pipelines, Data Modeling
- Integration: REST APIs, SAP integration (BAPI/RFC/ODATA)
- Platforms: Elasticsearch
- Orchestration: Scheduling tools (Airflow / similar)
- Analytics: KPI engines, SLA computation
Soft Skills
- Strong analytical and problem-solving skills
- Excellent communication and stakeholder collaboration
- Ability to work in fast-paced enterprise environments
- Strong documentation and knowledge-sharing mindset
Good to Have
- Experience with Kibana or Elastic-based dashboards
- Exposure to AI/ML data pipelines
- Experience with cloud platforms (AWS / Azure / GCP)
- Experience with streaming data processing (Kafka / Spark Streaming)
- Knowledge of data quality frameworks and monitoring tools
- Experience handling large-scale operational datasets
Compensation & Benefits
- Competitive salary and annual performance-based bonuses
- Comprehensive health and optional Parental insurance
- Optional retirement savings plans and tax savings plans
KRA (Key Result Areas)
- High-quality and scalable data pipeline delivery
- Reliable and accurate data integration across systems
- Efficient KPI and analytics engine implementation
- Strong system performance and data availability
KPI (Key Performance Indicators)
- On-time delivery of data pipelines
- Data accuracy and pipeline success rate
- Reduction in pipeline failures and incidents
- Performance optimization (latency, throughput)
- Stakeholder satisfaction and usability of data
Contact: hr@bigtapp.ai
Click on Apply to know more.