Website:
appxcess.com
Job details:
Job Description: AI Engineer – On-Premises AI, LLM & Infrastructure
Experience: 1–5 Years
Employment Type: Full-Time
Job Overview
We are looking for an AI Engineer with hands-on experience in developing, deploying, and optimizing AI applications on on-premises infrastructure. The candidate should have knowledge of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), intent parsing, tool calling, Model Context Protocol (MCP), AI orchestration, and GPU-accelerated computing.
The role involves working with locally hosted AI models and NVIDIA GPU infrastructure, including NVIDIA RTX 6000 Blackwell and L40S. The candidate will also be responsible for application containerization, Microsoft Hyper-V virtualization, performance optimization, scalability, networking, security, and maintaining reliable on-premises AI environments.
The ideal candidate should be capable of taking AI applications from development and experimentation through deployment, optimization, and production maintenance.
Key Responsibilities
1. LLM Deployment & On-Premises AI
• Deploy, configure, and manage locally hosted LLMs such as Qwen, Mistral, Kimi, and other suitable open-weight models.
• Work with inference frameworks such as Ollama, vLLM, and similar model-serving technologies.
• Deploy AI workloads on NVIDIA RTX 6000 Blackwell, L40S, and compatible GPU infrastructure.
• Optimize model inference, GPU utilization, VRAM consumption, response latency, and throughput.
• Evaluate model performance, tool-calling capabilities, accuracy, and hardware requirements.
• Implement quantization, batching, streaming inference, and other optimization techniques where appropriate.
• Troubleshoot GPU drivers, CUDA compatibility, memory allocation, and model-serving issues.
• Maintain local AI deployments with an emphasis on data governance, privacy, and reduced dependence on external AI services.
2. RAG, Intent Parsing & AI Orchestration
• Develop RAG pipelines for retrieving information from documents, databases, and enterprise data sources.
• Implement document parsing, text extraction, OCR where required, chunking, embeddings, vector search, and reranking.
• Build intent-parsing and routing mechanisms to identify user requirements and determine the appropriate processing workflow.
• Implement LLM-based tool calling and function calling to interact with APIs, databases, and third-party applications.
• Develop multi-step orchestration workflows involving retrieval, tool selection, execution, and response generation.
• Implement MCP servers and clients to connect AI models with compatible tools and data sources.
• Design and maintain centralized tool registries with tool definitions, input schemas, execution handlers, and permissions.
• Implement guardrails, validation, fallback mechanisms, error handling, and controlled tool execution.
• Improve response accuracy, contextual relevance, source attribution, and reliability.
3. Backend Development & Application Integration
• Develop backend services and REST APIs using Python and frameworks such as FastAPI.
• Integrate AI applications with enterprise systems, databases, document repositories, and third-party APIs.
• Build reusable connectors for data retrieval and authorized application actions.
• Implement asynchronous processing, background jobs, session management, and streaming responses.
• Develop modular services for model inference, orchestration, retrieval, and external integrations.
• Write automated tests, technical documentation, and deployment scripts.
• Collaborate with frontend and infrastructure teams to deliver end-to-end AI applications.
4. Real-Time Voice AI & Conversational Systems
• Develop AI-powered conversational applications supporting real-time voice interactions.
• Integrate speech-to-text, LLM inference, text-to-speech, and optional avatar animation.
• Work with technologies such as Faster-Whisper/Whisper, Kokoro TTS, MuseTalk, and NVIDIA Audio2Face.
• Build low-latency audio-processing pipelines and streaming conversational workflows.
• Implement conversation context management, interruption handling, session management, and error recovery.
• Optimize audio processing, inference latency, GPU utilization, and concurrent session handling.
• Integrate voice AI components with backend APIs and application interfaces.
5. Microsoft Hyper-V & On-Premises Infrastructure
• Provision, configure, and manage virtual machines using Microsoft Hyper-V.
• Deploy and maintain Windows Server and Linux environments for AI applications and supporting services.
• Configure virtual CPU, memory, storage, networking, and resource allocation.
• Enable and troubleshoot GPU access for virtualized AI workloads using supported GPU-P or passthrough configurations, depending on hardware and operating-system compatibility.
• Understand GPU virtualization requirements for NVIDIA RTX 6000 Blackwell and L40S platforms.
• Deploy AI inference services and supporting applications across virtual machines and physical servers.
• Configure virtual switches, network segmentation, firewalls, DNS, and secure service communication.
• Troubleshoot virtualization, GPU availability, driver compatibility, storage, and networking issues.
• Maintain infrastructure documentation, backup procedures, and operational runbooks.
6. Containerization & Deployment
• Containerize AI models, backend services, RAG pipelines, MCP services, and voice-processing components using Docker.
• Use Docker Compose or similar technologies to manage multi-service applications.
• Configure GPU access for supported NVIDIA-enabled containers.
• Manage environment variables, secrets, persistent storage, service dependencies, and deployment configurations.
• Deploy applications across Hyper-V virtual machines and on-premises GPU servers.
• Implement health checks, logging, monitoring, service recovery, and deployment automation.
• Maintain reproducible and reliable deployment environments.
7. Performance, Scalability & Reliability
• Benchmark inference latency, time to first token, tokens per second, throughput, and concurrent user capacity.
• Identify CPU, GPU, VRAM, storage, and networking bottlenecks.
• Optimize resource allocation for multiple AI models, applications, and concurrent users.
• Implement caching, asynchronous processing, queues, batching, and streaming where appropriate.
• Design reliable services with timeouts, retries, fallback mechanisms, and graceful error handling.
• Monitor GPU utilization, memory consumption, application health, and inference performance.
• Ensure AI applications remain scalable, maintainable, and reliable as workloads increase.
8. Security & Maintainability
• Implement authentication, authorization, role-based access controls, and secure API integrations.
• Configure TLS, reverse proxies, firewalls, and secure inter-service communication.
• Apply appropriate access restrictions to tools, documents, databases, and connected applications.
• Implement input validation, output validation, audit logging, and secrets management.
• Ensure that RAG retrieval and tool execution respect user permissions and tenant boundaries.
• Maintain clean, modular, testable code and well-documented infrastructure.
• Troubleshoot production issues and continuously improve system security and stability.
Example Use Cases
The engineer may work on use cases such as:
1. Real-Time AI Companion
Develop an AI companion that interacts with users in live, one-on-one sessions and answers queries through natural conversations. This may involve locally hosted LLMs, Faster-Whisper for speech recognition, Kokoro for speech generation, and MuseTalk or NVIDIA Audio2Face for avatar animation. The focus is on low-latency interactions, conversational context, GPU optimization, and concurrent sessions.
2. Multi-Source Enterprise Chatbot
Develop a chatbot that retrieves information from multiple authorized data sources and interacts with connected applications. This may involve Qwen 14B or Mistral models, document parsing, RAG, API integrations, databases, MCP, and a tool registry. The processing workflow may include intent routing, retrieval, tool calling, orchestration, guardrails, and response generation.
Required Technical Skills
• Python programming and backend API development.
• Experience with or understanding of LLMs such as Qwen, Mistral, or Kimi.
• Familiarity with Ollama, vLLM, or similar inference frameworks.
• Knowledge of RAG, embeddings, vector databases, and semantic search.
• Understanding of intent parsing, tool calling, MCP, and AI orchestration.
• Experience with Docker and containerized application deployment.
• Practical knowledge of Microsoft Hyper-V and virtual machine management.
• Understanding of NVIDIA GPUs, GPU memory, inference performance, and model optimization.
• Knowledge of networking, security, and application troubleshooting.
Preferred Skills
• Experience with NVIDIA RTX 6000 Blackwell, L40S, or comparable GPU infrastructure.
• Knowledge of CUDA, NVIDIA drivers, and GPU monitoring.
• Experience with GPU virtualization or passthrough in Hyper-V where supported.
• Familiarity with Windows Server, Linux, and virtual networking.
• Experience with Faster-Whisper, Kokoro, MuseTalk, NVIDIA Audio2Face, or related voice AI
technologies.
• Familiarity with MCP SDKs, LangGraph, or similar orchestration frameworks.
• Knowledge of OCR, reranking, hybrid retrieval, and document-processing pipelines.
• Experience with reverse proxies, TLS, monitoring, logging, and CI/CD.
Candidate Profile
• Strong analytical and problem-solving skills.
• Hands-on approach to AI development, deployment, and troubleshooting.
• Ability to work across LLMs, backend applications, and infrastructure.
• Understanding of the trade-offs between model quality, latency, GPU requirements, and scalability.
• Ability to develop maintainable and secure AI applications.
- Willingness to learn emerging LLM, agentic AI, voice AI, and GPU technologies.
Click on Apply to know more.
