Senior Data Engineer

Senior Data Engineer

This is a full-time position with D Square Consulting Services Pvt Ltd

Required Experience - 5+ years

Location – Bangalore

The candidate who can join within 30 days

Work from office

Must-Have Skills

• Proven experience working with Databricks/Python/Spark/Scala to build and optimize data pipelines and large-scale data processing systems.
• Strong experience working with data warehouse and data lake environments.
• Advanced level in SQL for data extraction, transformation, and analysis.
• Good understanding of data visualization tools like Tableau/Power BI to support data-driven decision-making.
• Basic experience in data modeling, with the ability to design and maintain efficient and scalable data structures.
• Expert-level experience in ETL (Extract, Transform, Load) processes and LET (Load, Extract, Transform), ensuring data is correctly processed and loaded into the target systems.
• Experience working with Microsoft Azure cloud services is preferred.
• Experience working with Consumer-Packaged Goods (CPG) or Fast-Moving Consumer Goods (FMCG) data is a significant advantage.

Good-to-Have Skills:

• Familiarity with syndicated data sources such as Nielsen, Kantar, IRI, Circana, etc., will be highly preferred.
• Experience with Snowflake is preferred but not mandatory.
• Ability to effectively communicate with both technical and non-technical stakeholders, and work collaboratively with cross-functional teams.
• Strong analytical skills and a detail-oriented mindset, with a focus on delivering high-quality results in a fast-paced environment.

Roles & Responsibilities:

• Design, develop, and maintain robust and scalable data pipelines using Databricks, Python, Spark, and Scala.
• Work with data warehousing and data lake solutions to integrate data from various sources, ensuring high data quality and consistency.
• Utilize advanced SQL skills to perform complex data transformations, queries, and optimizations.
• Collaborate with analytics teams to support data visualization efforts using Tableau and Power BI, ensuring data is accurately represented and easily understood.
• Participate in data modeling activities, contributing to the design and maintenance of efficient data structures.
• Lead the development and optimization of ETL/LET processes to ensure reliable data flow across the organization.
• Leverage Microsoft Azure services to enhance data processing and storage capabilities, optimizing the use of cloud resources.
• Work closely with business stakeholders to understand data needs, provide insights, and support decision-making processes.
• Maintain comprehensive documentation of data processes, pipelines, and models to ensure knowledge transfer and system transparency.