Data Engineer-Hyderabad

We are seeking a skilled and experienced Data Engineer to join our team, focusing on building and managing data solutions on the Azure platform. The ideal candidate will be adept in Azure Databricks, PySpark, and other Azure data services, with a strong background in designing, developing, and deploying end-to-end data pipelines. This role requires a proactive individual who can manage the entire data lifecycle, from requirement gathering to production deployment and support. Knowledge in finance and SAP is a plus, as you'll work closely with data from various business domains. Responsibilities: Requirement Gathering: Collaborate with business stakeholders to understand data requirements and document specifications. Data Pipeline Development: Design, develop, and maintain ETL/ELT pipelines using Azure Databricks, PySpark, and Azure Data Factory to load and transform data from diverse sources. Data Storage Management: Utilize Azure Data Lake Storage (ADLS) for structured and unstructured data storage, ensuring optimal organization for analytics. Data Transformation: Implement data transformations using PySpark and Python to enable data analysis and reporting. Code Optimization: Continuously optimize code to improve the performance, reliability, and efficiency of data pipelines and transformations. Deployment and Automation: Create CI/CD pipelines to automate deployment processes and manage version control using best practices. Production Support: Monitor, troubleshoot, and resolve issues in production to ensure data reliability and accuracy. Documentation and Best Practices: Maintain clear documentation of data processes, standards, and best practices. Collaboration and Communication: Work closely with cross-functional teams to align data solutions with business goals and deliver insights effectively. Requirements: Experience: 3+ years in Data Engineering, with a focus on Azure cloud technologies. Technical Skills: Azure Databricks: Expertise in using Databricks for data processing and analytics. PySpark & Python: Strong skills in PySpark for data transformations and Python for scripting and automation. Azure Data Factory (ADF): Experience in designing, deploying, and managing data pipelines with ADF. Azure Data Lake Storage (ADLS): Proficient in managing data storage solutions for structured and unstructured data in ADLS. CI/CD: Familiarity with CI/CD pipelines in Azure DevOps or similar tools for deploying data solutions. Domain Knowledge: Understanding of finance and SAP data models and processes is a plus. Soft Skills: Strong problem-solving abilities and attention to detail. Excellent communication and interpersonal skills to work collaboratively with stakeholders. Self-motivated with a proactive approach to managing tasks and driving projects forward. Preferred Qualifications: Relevant Azure certifications in data engineering or data science.

Dunboxed Solutions Private Limited