Databricks + Pyspark Architect
Infosys
full-time
Required skills
- Python
- advanced analytics
- compliance
- database
- Databricks
- design patterns
- end-to-end
- ETL
- machine learning
- multi-tenant
- SQL
- Unity
- SDLC
About the role
Infosys
Website:
infosys.com
Job details:
Experience
- 8–14 years of overall IT experience
- 4–6+ years as a Data / Analytics / Platform Architect
- Strong hands-on experience architecting Databricks Lakehouse platforms Databricks Expertise
- Deep experience with: o Databricks Lakehouse architecture o Delta Lake o Unity Catalog o MLflow
- Strong understanding of batch and streaming data architectures
- Experience designing multi-workspace or multi-tenant setups Agentbricks & AI Knowledge
- Hands-on or strong working knowledge of Agentbricks
- Understanding of: o AI agent concepts (tools, memory, reasoning loops) o RAG architectures and vector search o LLM-driven workflows over enterprise data
- Experience integrating agents with data, APIs, and downstream systems Data Engineering & Programming
- Strong SQL expertise
- Proficiency in Python and PySpark
- Experience with ETL/ELT and data orchestration patterns
- Familiarity with APIs and event-driven integration Databricks & Lakehouse Architecture
- Define and own end-to-end Databricks Lakehouse architecture
- Design scalable platforms for: o Structured, semi-structured, and unstructured data o Analytical and AI workloads
- Establish architecture patterns, best practices, and reference designs
- Lead modernization from legacy data platforms to Databricks-based lakehouse Agentbricks & AI Agent Enablement
- Architect and enable AI agent-based solutions using Agentbricks
- Design data and system patterns to support: o AI agents reasoning over enterprise data o Tool-calling and action-oriented agents o Retrieval-Augmented Generation (RAG) workflows
- Define how agents interact with: o Delta tables o Vector embeddings o Feature stores and semantic layers
- Guide teams on agent orchestration, memory, observability, and safety Advanced Analytics & AI Platform Design
- Enable data architectures that support: o Machine learning pipelines o Feature engineering and reuse o Real-time and batch inference
- Integrate Databricks with: o LLM providers and model endpoints o Vector search and embedding storage
- Collaborate with data scientists and AI engineers to productionize AI solutions Databricks Platform Components
- Architect and govern solutions using: o Delta Lake o Unity Catalog (governance, lineage, access control) o Databricks Workflows and Jobs o MLflow (experiments, models, lifecycle)
- Define cluster, compute, and workspace strategies for multi-team environments Security, Governance & Compliance
- Implement secure data and AI access using: o Unity Catalog permissions o Cloud IAM integrations
- Define governance for: o AI agent data access o Model and agent versioning o Auditability and traceability
- Ensure compliance with enterprise security and regulatory requirements Performance, Cost & Operations
- Design for performance optimization and scalability
- Optimize compute usage, storage, and agent execution costs
- Enable monitoring and observability for: o Data pipelines o ML workloads o AI agents
- Partner with platform and FinOps teams on cost control strategies Technical Leadership & Collaboration
- Act as the Databricks and AI architecture authority
- Review solution designs and guide engineering teams
- Mentor data engineers, ML engineers, and architects
- Translate business and AI requirements into scalable technical designs
- Knowledge of architectural design patterns, performance tuning, database and functional designs
- Hands-on experience in Service Oriented Architecture
- Ability to lead solution development and delivery for the design solutions
- Experience in designing high level and low level documents is a plus
- Good understanding of SDLC is a pre-requisite
- Awareness of latest technologies and trends
- Logical thinking and problem solving skills along with an ability to collaborate
Click on Apply to know more.
This page is fully interactive when JavaScript is enabled. Please enable JavaScript to apply or browse related roles.