Website:
impetus.ai
Job details:
Requirements
The ideal candidate is a hands-on AI architect with deep Databricks and GenAI expertise, strong data engineering foundations, and a proven record of taking AI solutions to production at enterprise scale. Key requirements:
- 12–14 years of experience in data and AI/ML solutions, including designing and architecting large-scale (TB/PB) data platforms and production AI/ML systems.
- Deep, hands-on architecture expertise with Databricks — Spark/PySpark, Delta Lake, lakehouse/medallion architecture, Unity Catalog, Databricks Workflows, and Delta Live Tables — with strong performance and cost optimization.
- Proven experience architecting Generative AI / LLM solutions — RAG pipelines, chunking and embedding strategies, vector store design, prompt engineering, evaluation, and guardrails/responsible-AI patterns.
- Strong hands-on experience with native Databricks AI capabilities — Mosaic AI (Model Serving, Vector Search, Agent Framework), Foundation Model APIs, MLflow, Feature Store, and model governance via Unity Catalog.
- Solid ML/MLOps foundations — model lifecycle management, experiment tracking, deployment, monitoring, and drift/quality evaluation in production.
- Expert-level PySpark and Spark SQL, with strong Python for building reusable data and feature pipelines that power AI workloads (batch and streaming).
- Strong hands-on experience with AWS and Azure Cloud, including networking, IAM/security, cost optimization, and AI/ML services; comfortable architecting across cloud data and AI services.
- Experience with LLM orchestration frameworks (LangChain/LangGraph or equivalent), agentic patterns, tool use, and multi-agent workflows at an architecture level.
- Deep expertise in data and AI governance, data quality, lineage, security, and responsible AI — including PII handling, access control (Unity Catalog, RBAC), and audit.
- Proven ability to set architecture standards, lead design reviews, and provide technical leadership and mentorship across multiple engineering and data science teams.
- Strong communication and stakeholder-management skills; able to drive architecture decisions and translate business needs into technical blueprints for both technical and executive audiences.
Good to Have: Knowledge of LLMOps and observability tooling for AI systems; Databricks / cloud / AI certifications; experience leading cloud/data platform migration and modernization to Databricks; experience contributing to POCs, proposals, and RFPs.
Databricks, AI
AI & GenAI Architecture
- Own the end-to-end architecture for AI/ML and GenAI solutions on Databricks — data and feature pipelines, model training/serving, RAG, and agentic applications.
- Design RAG and LLM architectures — retrieval and embedding pipelines, vector store design, prompt/evaluation strategy, and guardrails — using Mosaic AI and Foundation Model APIs.
- Define reference architectures, design patterns, and reusable frameworks for scalable, secure, and cost-optimized AI platforms.
Data & Platform Foundation
- Architect the lakehouse foundation (Delta Lake, medallion, Unity Catalog) and reusable PySpark pipelines that feed AI and analytics workloads.
- Guide implementation using Databricks Workflows and Delta Live Tables, with standards for data quality, incremental loads, and schema evolution.
MLOps, Governance & Reliability
- Define MLOps/LLMOps practices — model lifecycle, MLflow tracking, deployment, monitoring, and drift/quality evaluation in production.
- Establish AI and data governance, cataloging, lineage, access control, and responsible-AI standards (PII handling, guardrails, audit) ensuring security and compliance.
- Define observability, monitoring, and cost-governance practices for reliable and efficient AI systems at scale.
Technical Leadership & Collaboration
- Set and enforce architecture standards and best practices; lead design and code reviews across multiple engineering and data science teams.
- Mentor engineers, ML engineers, and leads, and provide hands-on guidance from prototype to production readiness.
- Collaborate with product owners, engineering leaders, and business stakeholders to translate requirements into AI blueprints, POCs, and roadmaps.
Click on Apply to know more.
