Senior Data Engineer

QJD- Senior Data Engineer-2024

About Quation

Welcome to Quation, where data transforms into the currency of informed decisions. We are more than a data analytics company; we are the architects of your data-driven success. Our commitment to precision, innovation, and industry transformation sets us apart. At Quation, we embody the essence of “Powering Data, Empowering Decisions.” We are driven by the belief that data, when harnessed intelligently, has the power to revolutionize industries, driving growth and efficiency. Join us in revolutionizing the world of data analytics, AI, and ML.

Job Description

We are looking for a skilled and passionate Data Engineer to join our growing team. As a Data Engineer, you will play a critical role in designing, developing, and maintaining our data infrastructure, ensuring the smooth flow of data and enabling data-driven decision-making across the organization.

•Years of experience: 3 - 6 years of relevant work experience
•Budget: ₹15 - ₹30 LPA

Responsibilities
• Design, build, and maintain efficient and reliable data pipelines using SQL and Python, leveraging tools like Apache Airflow for orchestration.
• Develop ETL (Extract, Transform, Load) processes to move data between various sources and destinations.
• Implement data quality checks and validation routines to ensure data accuracy and integrity.
• Collaborate with data analysts and scientists to understand their data needs and translate them into actionable data solutions.
• Optimize existing data pipelines for performance and scalability using Airflow’s features for monitoring and optimization.
• Develop and maintain documentation for data pipelines and processes, including Airflow workflows.
• Contribute to the development and maintenance of data governance policies.
• Stay up to date with the latest trends and technologies in data engineering, especially advancements in orchestration tools like Airflow.

Qualifications
Basic Qualifications
• Bachelor’s degree in computer science, Engineering, or a related field.
• 3-6 years of experience as a Data Engineer or in a similar role.
• Strong proficiency in SQL for data querying, manipulation, and analysis.
• Excellent programming skills in Python, including experience with data manipulation libraries (Pandas, NumPy) and data engineering libraries (Dask, Airflow).
• Proficiency in using Apache Airflow for data pipeline orchestration, scheduling, and monitoring.
• Familiarity with cloud platforms (AWS, Azure, GCP) and data warehousing solutions (Snowflake, Redshift).
• Experience with data visualization tools (Tableau, Power BI) is a plus.
• Ability to work independently and as part of a team.
• Strong analytical and problem-solving skills.
• Excellent communication and documentation skills.

Suggested Skills:

• Data Pipeline Development: Experience in building, testing, and maintaining scalable data pipelines (Python, SQL) using frameworks like Apache Spark or Apache Kafka.
• Database Management: Proficient in managing relational databases (PostgreSQL, MySQL, SQL Server) and NoSQL databases (MongoDB, Cassandra).
• ETL Processes & Tools: Skilled in ETL tools like Apache NiFi, Informatica, Databricks/PySpark.
• Data Warehousing: Experience with data warehousing solutions such as Snowflake, Google BigQuery, and Amazon Redshift, as well as data lake architectures.
• Cloud Platforms: Proficiency with cloud storage services such as AWS S3, Azure Blob Storage, and GCP BigQuery.
• Data Modeling: Expertise in designing logical and physical data models.
• Programming Skills: Proficiency in Python or Java and advanced SQL for data manipulation.
• Workflow Orchestration: Familiar with workflow orchestration tools like Apache Airflow and Luigi.
• Data Integration: Knowledge of data integration methods (APIs, batch processing, real-time streaming).

Preferred Skills:

• Big Data Technologies: Experience with large-scale data processing tools like Apache Hadoop, Spark, and Kafka.
• Containerization & Orchestration: Familiar with container orchestration tools like Kubernetes and Docker.
• DataOps & CI/CD: Knowledge of DataOps practices, version control, CI/CD pipelines, and automated testing/deployment of data solutions.
• Machine Learning Engineering: Experience with ML model deployment and MLOps for supporting data scientists.
• Data Quality & Governance: Skilled in data quality frameworks and validation processes.
• Scripting Languages: Familiarity with additional scripting languages (Shell, Bash).
• Visualization Tools: Experience with tools such as Tableau, Power BI, or Looker.
• Real-Time Data Processing: Proficient in real-time data streaming (Apache Kafka, Kinesis, or Google Pub/Sub).

Ideal Soft Skills:
• Problem-solving
• Collaboration
• Adaptability
• Attention to Detail
• Communication Skills
• Time Management
• Continuous Learning
• Project Management

Notice Period: Candidates with an immediate to 30-day notice period are preferred.
Location: Bangalore/Lucknow Work Type: Hybrid

General Recruitment Disclaimer:
Quation Solutions Pvt Ltd is an equal opportunity employer, and all qualified applicants will be considered for employment without regard to race, color, religion, gender, gender identity, sexual orientation, national origin, genetics, disability, age, or veteran status.