Thankz Global Staffing
Website:
thankz.com
Job details:
Sr. Data Engineer – Databricks / Azure | Remote | Contract | 2–11 PM IST | Till end of 2026 (possibility of extension)
Design, build, and maintain automated testing frameworks and Python SDKs for data integration, parity verification, and user accessibility in the Industrial IoT / Smart Buildings domain. We're looking for a Senior Data Engineer with strong Python, PySpark, and Spark skills, with proven experience in automated Data Quality frameworks and CI/CD pipelines, delivering Data & Analytics products using Agile methodology.
Must-Have Skills:
- 5+ years hands-on data engineering experience in Azure ecosystem
- PySpark & DataFrames
- Apache Spark internals (Catalyst Optimizer, Logical Plans)
- Java
- Python (building Libraries/SDKs, Py4J)
- Databricks & Delta Lake optimization
- Automated Testing frameworks (PyTest, JUnit) for data pipelines
- Data Quality tools (Great Expectations, Deequ)
- Azure DevOps, GitHub, CI/CD, artifact management
- Data modeling & warehousing on Azure Databricks
- Data governance & security best practices in Azure
Preferred Certifications: Azure Fundamentals, Azure Data Engineer Associate, Databricks Certified Data Engineer Professional
Responsibilities:
- Implement scalable data solutions on Spark and Flink
- Develop Python SDK wrapping core libraries for use in Jupyter Notebooks
- Build an automated Parity Test Suite via CI/CD against Golden Datasets (Spark & Flink)
- Implement Data Quality rules (Completeness, Validity thresholds)
- Manage and optimize Azure & Databricks resources for performance and cost
- Transform, clean, and prepare data using SQL, Python, and Java
- Monitor and tune workloads/pipelines
- Maintain CI/CD pipelines and documentation
- Participate in Agile ceremonies and enforce DE best practices
Click on Apply to know more.