iKart Solutions LLP
Website:
ikartsolutions.com
Job details:
This is full time remote contract position. (So no freelancing or moonlighting allowed)
You may need to provide overlapping time with US timezone.
This is client facing role hence excellent communication in English language is MUST.
Min. Experience : 8 - 10 years
We need a champion of Databricks platform who can do everything and anything on Databricks.
Overview
This is a 50% platform administration / 50% data engineering role owning the Azure Databricks environment and building production-grade data pipelines on it. The ideal candidate has strong experience with Databricks on Azure, Terraform (IaC), DLT (Delta Live Tables), and Databricks Asset Bundles (DABs), plus solid data engineering and Azure cloud administration skills.
Key Responsibilities (50% Platform)
Databricks Platform Administration & IaC
Infrastructure as Code (Terraform): Use Terraform to provision and manage Azure Databricks workspaces, clusters, Unity Catalog resources, and associated Azure networking/storage.
Workspace Management: Own day-to-day administration including workspace configuration, cluster policies, pools, and permissions.
Identity & Access: Implement and maintain access control (RBAC), SCIM provisioning, and integration with Entra ID (Azure AD) groups.
Compute Governance: Define and manage standard compute patterns (interactive, job clusters, pools) to balance cost, performance, and governance.
Network & Security: Collaborate with cloud teams on network configurations (VNets, private endpoints, NSGs, firewall rules) using automated deployment scripts.
Standardization: Document platform best practices for notebooks, jobs, libraries, and environment management across teams.
Governance, Security & Operations
Data Governance: Implement and maintain Unity Catalog, fine-grained access policies, and metadata tagging/classifications.
Operational Excellence: Define and operate monitoring, alerting, and incident response processes for Databricks jobs and clusters.
Support: Maintain runbooks and SOPs; provide L2/L3 support for platform and pipeline issues.
Key Responsibilities (50% Data Engineering)
Data Engineering & DLT
Pipeline Design: Design and implement data ingestion/transformation pipelines using Delta Lake and Medallion architecture.
DLT Implementation: Develop and operate Delta Live Tables (DLT) pipelines, incorporating expectations, quality rules, and observability.
Optimization: Fine-tune pipelines for performance and cost (partitioning, Z-order, Liquid Clustering, Photon).
DABs & CI/CD / DevOps
Asset Management: Use Databricks Asset Bundles (DABs) to package and deploy jobs, DLT pipelines, and dashboards.
CI/CD Pipelines: Implement environment promotion (Dev/Test/Prod) using Git-based workflows and Azure DevOps.
Config-as-Code: Maintain configuration-as-code for all Databricks assets and enforce standards for version control and code review.
Required Skills & Experience
Platform & Engineering: Strong hands-on experience with Azure Databricks in both admin and engineering capacities.
Infrastructure as Code: Proven experience using Terraform to manage Azure resources and Databricks provider objects.
Core Tech: Solid knowledge of PySpark, SQL, and Delta Lake concepts.
Deployment: Experience with Databricks Asset Bundles (DABs) and DLT.
Azure Ecosystem: Working knowledge of ADLS Gen2, Entra ID (Azure AD), networking basics, and Azure Monitor.
DevOps: Expertise in Git-based workflows and CI/CD tools (Azure DevOps)
Additional Experience
Overall Professional Experience: 10+ years in Information Technology, with a focus on Data or Cloud Infrastructure.
Platform & DevOps: 5+ years of experience in Platform Administration or DevOps, including strong proficiency with Terraform and CI/CD (Azure DevOps/GitHub Actions).
Databricks Expertise: 3+ years of hands-on experience with Azure Databricks, specifically implementing Unity Catalog, DLT, and Databricks Asset Bundles (DABs).
Cloud Foundation: 3+ years of experience managing Azure core services (Entra ID, ADLS Gen2, Networking/VNets).
Nice-to-Have
Certifications:
Databricks: Certified Data Engineer Associate / Professional
Azure: AZ-104 (Administrator), AZ-305 (Solutions Architect).
Advanced Governance: Experience with multi-workspace Unity Catalog setups and fine-grained access controls.
FinOps: Background in cost optimization and usage tracking for cloud data platforms.
Streaming: Exposure to Auto Loader and Structured Streaming.
Click on Apply to know more.