Alegeus
Website:
alegeus.com
Job details:
About the Role
We are looking for a hands-on LLMOps Engineer with 3+ years of experience to help operationalize Generative AI and LLM-powered capabilities across enterprise products and workflows. This role is focused on LLMOps, including prompt lifecycle management, prompt evaluation, LLM observability, prompt/model governance, controlled rollout, production monitoring, quality tracking, and operational support for LLM-powered systems. The role will support Alegeus’ broader AI enablement and governed AI adoption model by helping ensure that prompts, LLM configurations, evaluation datasets, retrieval settings, AI service configurations, and LLM-powered workflows are deployed through repeatable, traceable, secure, and auditable operational patterns. This is not a pure AI Engineer or product feature engineering role. The candidate is not expected to primarily build product features, business APIs, or application workflows. Instead, this role will focus on making LLM-powered capabilities production-ready, measurable, governed, monitored, and supportable. This is also not a data platform engineering role. Enterprise data pipelines, data warehouse operations, and platform-level data engineering will be handled by the data platform team. This role will consume AI-ready data interfaces, prompts, model endpoints, retrieval configurations, evaluation datasets, and foundational AI services to support reliable LLM operations. The ideal candidate should have hands-on exposure to Generative AI systems, Azure OpenAI or similar LLM platforms, prompt engineering, prompt testing, LLM evaluation, observability, CI/CD, monitoring, and production support. Experience with Python and/or C#/.NET is preferred for automation, tooling, and integration support.
Required Qualifications
• 3+ years of professional experience in LLMOps, AI operations, software engineering, DevOps, cloud engineering, MLOps, AI engineering, or related roles.
• Hands-on exposure to Generative AI concepts such as LLMs, prompts, embeddings, RAG, chatbots, document extraction, summarization, classification, or AI service orchestration.
• Familiarity with LLMOps concepts such as prompt versioning, prompt testing, LLM evaluation, AI observability, token/cost tracking, hallucination monitoring, and safe rollout.
• Experience with Azure OpenAI, OpenAI APIs, Azure AI services, or comparable LLM platforms.
• Programming or scripting experience in Python and/or C#/.NET for automation, tooling, and operational workflows.
• Experience with CI/CD automation using Azure DevOps, GitHub Actions, or similar tools.
• Basic understanding of REST APIs, service integration, cloud deployment patterns, and production support.
• Understanding of logging, monitoring, alerting, dashboards, tracing, and operational metrics.
• Familiarity with secure release practices, access control, audit logging, approval workflows, and production governance.
• Strong problem-solving skills and ability to collaborate across AI engineering, data science, platform, product engineering, security, architecture, and governance teams. Preferred Qualifications
• Prior experience operationalizing LLM-powered applications such as chatbots, document extraction, summarization, classification, intelligent search, claims automation, or workflow assistance.
• Experience with Azure AI Document Intelligence, Azure AI Search, Azure App Services, Azure Functions, Azure Machine Learning, AKS, or related Azure services.
• Experience with prompt evaluation frameworks, golden datasets, regression testing, prompt/model comparison, reviewer workflows, or AI quality dashboards.
• Exposure to RAG workflows, vector search, semantic search, grounding, citations, retrieval quality evaluation, or hallucination detection.
• Exposure to Semantic Kernel, LangChain, LlamaIndex, AutoGen, CrewAI, or similar orchestration frameworks from an operations and observability perspective.
• Exposure to MLflow, model registry, experiment tracking, or basic MLOps practices is a plus.
• Exposure to Docker, Kubernetes, AKS, Terraform, secrets management, RBAC, audit logging, and cloud security patterns.
• Experience with regulated enterprise environments such as healthcare, fintech, benefits administration, claims processing, or insurance.
• Familiarity with compliance-aware AI operations, data privacy, human-in-the-loop review, separation of duties, and responsible AI practices.
• Strong communication and collaboration skills, with the ability to work across global teams.
Click on Apply to know more.
