Website:
datahatch.ai
Job details:
About Data Hatch
Data Hatch is a London-based AI consultancy that builds custom data platforms and AI models for enterprise clients across Europe and the UK. Founded by a former AI Capability Lead at Avanade (Accenture x Microsoft), we have delivered projects for clients in financial services, insurance, healthcare, and industrial automation.
We are an official Databricks Partner and operate on a modern Azure-first stack. We are a small, fast-moving team with no micromanagement — everyone is trusted, accountable, and expected to bring real ownership to their work. We share knowledge, support each other’s growth, and occasionally share memes in the group chat.
The Role
We are hiring a Data Engineer to join our remote team in India. This is a permanent, full-time position.
You will be working on a live production project — migrating data from an on-premises SQL database into a modern Databricks Lakehouse architecture for a UK-based insurance and healthcare client. The train is already moving. We need someone who can hop on, contribute immediately, and help drive it forward.
You will work alongside a data architect, a cloud architect, and junior ML engineers. We are looking for someone who can take the lead on data engineering, mentor junior team members, and collaborate with architects to design and deliver a scalable, cost-efficient data platform.
What You Will Do
- Design and build scalable ETL/ELT pipelines on Azure Databricks and Delta Lake
- Implement and maintain Medallion architecture (Bronze, Silver, Gold layers)
- Own data quality frameworks, schema enforcement, and data governance using Unity Catalog
- Write optimised PySpark and SQL code — handling shuffling, broadcast joins, partitioning, and Z-ordering
- Work with Azure Data Factory for orchestration and pipeline scheduling
- Ingest data from structured, semi-structured, and unstructured sources (including real-time event data)
- Build idempotent pipelines with proper MERGE/upsert logic and failure recovery strategies
- Write and maintain unit tests for data transformation functions and UDFs
- Collaborate with data scientists and ML engineers to ensure data is clean and model-ready
- Communicate progress clearly to technical and non-technical stakeholders
- Contribute to sprint planning and take ownership of two-week delivery cycles
What We Are Looking For
- 4+ years of hands-on data engineering experience
- Deep expertise in Azure Databricks — including Delta Lake, Unity Catalog, and the Databricks platform (not just notebooks)
- Strong PySpark skills: you understand lazy evaluation, the difference between RDDs and DataFrames, shuffling, and how to optimise Spark jobs
- Solid SQL: query optimisation, CTEs, star/snowflake schema, normalisation
- Experience building and owning Medallion architecture from scratch
- Experience with data ingestion from multiple source types: on-prem databases, flat files, APIs, real-time streams
- Solid understanding of idempotency in data pipelines and how to design for it
- Familiarity with Azure Data Factory or similar orchestration tools
- Strong communication skills — you can explain data architecture decisions to a non-technical client
- You are curious, ambitious, and want to grow beyond your current lane
Nice to Have
- Experience in insurance, financial services, or healthcare data
- Knowledge of Snowflake and/or DBT
- Familiarity with MLflow or Azure Machine Learning
- Exposure to Docker and containerised environments
- Interest in LLMs, vector databases, or Generative AI pipelines
- Power BI or Streamlit for data visualisation
What We Offer
- Permanent, full-time remote position — based anywhere in India
- Competitive compensation
- No micromanagement — you own your work
- A genuine startup environment: high stakes, high learning, high reward
- Direct access to state-of-the-art AI projects — we do not work on outdated tech
- Support for certifications, courses, and lateral skill development (Azure, Databricks, ML, whatever you want to pursue)
- A team that actually supports each other and has fun doing it
- A pathway to senior and lead roles as we grow
If you are the kind of engineer who has broken something in production, fixed it on the same day, and walked away knowing more than before — we want to talk to you.
Click on Apply to know more.