Pyspark, Spark, Hive,Unix - Pan India Location

Job Summary

Job Title: Senior Data Engineer (Technical Lead) Location: Chennai Job Summary: We are seeking a highly skilled and experienced Senior Data Engineer (Technical Lead) to lead the design, development, and management of data systems and pipelines that enable data-driven decision-making. In this role, you will oversee a team of data engineers, drive technical architecture decisions, and work closely with data scientists, analysts, and other stakeholders to ensure the smooth integration of data solutions across various platforms. You will be responsible for optimizing data flows, ensuring data quality, and driving innovation in how we store, process, and utilize large datasets. Key Responsibilities: Lead a Data Engineering Team: Provide technical leadership, mentorship, and guidance to a team of data engineers, ensuring they follow best practices in coding, data modeling, and pipeline development. Design Data Systems and Pipelines: Architect and build scalable, efficient, and reliable data pipelines and ETL processes to ingest, transform, and load data from various sources into data warehouses and other data stores. Data Integration & Architecture: Collaborate with cross-functional teams to design and integrate data systems that support data analytics, machine learning, and business intelligence. Data Quality & Governance: Ensure the accuracy, consistency, and quality of data throughout the pipeline. Implement best practices for data governance, security, and compliance. Optimize Data Performance: Identify and resolve bottlenecks in data pipelines and optimize system performance for large-scale data processing. Technical Strategy & Innovation: Lead the selection of tools, frameworks, and technologies for data engineering and data management. Stay up-to-date with industry trends and propose innovative solutions to enhance data processes. Collaboration: Work closely with stakeholders, including data scientists, analysts, and software engineers, to understand business needs and deliver tailored data solutions. Documentation & Standards: Develop clear documentation for data workflows, data models, and technical processes. Ensure code and processes align with industry best practices. Code Reviews & Quality Assurance: Conduct thorough code reviews and enforce best practices to ensure the highest quality and maintainability of data engineering code. Required Skills & Qualifications: Experience 10+ years of hands-on experience in data engineering or a related field, with at least 2+ years in a technical lead or leadership role. Should have extensively worked on migration projects, leading teams and interacting with Architect roles and Client for a minimum of 2 projects Technical Skills: Strong proficiency in programming languages such as Spark or Python or Java, or Scala. Deep experience with Spark SQL / SQL, data warehousing, ETL frameworks, and cloud-based data platforms (e.g., AWS, GCP, Azure). Big Data Technologies: Experience with bigdata tools such as Cloudera Hive, Apache Spark, Hadoop, Kafka, and other distributed computing technologies. Data Warehousing & Databases: Expertise in relational, MPP and NoSQL databases, data modeling, and data warehousing solutions such as Teradata, Snowflake, Redshift, or BigQuery. Cloud Platforms: Hands-on experience with cloud platforms (AWS, GCP, or Azure) and cloud-based data services. Data Pipeline Development: Proven track record in building, optimizing, and managing data pipelines for large datasets and high-availability systems. Team Leadership: Experience leading a team of data engineers and collaborating with cross-functional teams. Problem Solving: Strong analytical and troubleshooting skills with the ability to resolve complex data issues and improve system performance. Communication Skills: Excellent communication and collaboration skills, with the ability to present technical concepts to non-technical stakeholders and lead team discussions. Preferred Qualifications: Data Governance & Security: Familiarity with data privacy, security protocols, and compliance frameworks (e.g., GDPR, HIPAA). Machine Learning & AI: Experience working with data scientists and supporting machine learning or AI initiatives. DevOps & CI/CD: Familiarity with DevOps practices, CI/CD pipelines, and automation in the data engineering context. Visualization Tools: Knowledge of data visualization tools (e.g., Tableau, Power BI) and their integration with data pipelines. Education: Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related field (or equivalent experience).