Factana® - Industry Solution Provider
Website:
factana.com
Job details:
Job Description
We are looking for a skilled Data Integration Engineer full-time / part-time, with hands-on experience in Databricks to design, develop, and maintain scalable data integration and data processing solutions. The ideal candidate should have strong experience working with data pipelines, ETL/ELT processes, cloud data platforms, and SQL/Python.
Employment Type: Full-Time / Part-Time
Location: Remote
Experience: 2–6 years in Databricks and data integration.
Key Responsibilities
- Design and develop scalable data integration and ETL/ELT pipelines using Databricks.
- Develop and optimize data processing workflows using Databricks, Apache Spark, and PySpark.
- Integrate data from multiple sources such as databases, APIs, files, and cloud platforms.
- Build and maintain reliable data pipelines for batch and near-real-time processing.
- Work with Delta Lake for data storage, transformation, and optimization.
- Develop complex SQL queries for data transformation and analysis.
- Implement data validation, quality checks, and error-handling mechanisms.
- Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
- Collaborate with data engineers, data analysts, and other technical teams to understand data requirements.
- Monitor, troubleshoot, and resolve data pipeline and integration issues.
- Maintain technical documentation for data pipelines and integration processes.
Required Skills
- Hands-on experience with Databricks.
- Knowledge of Apache Spark / PySpark.
- Strong SQL skills.
- Experience in ETL/ELT and data integration.
- Good understanding of Delta Lake and Databricks data processing.
- Experience with Python scripting.
- Knowledge of data warehousing concepts and data modeling.
- Experience working with REST APIs, JSON, CSV, and other data formats.
- Good understanding of cloud data platforms such as Azure, AWS, or GCP.
- Strong troubleshooting and problem-solving skills.
Interested candidates can share their updated profile at: murali.elumalai@factana.com
Click on Apply to know more.
