Recro
Website:
recro.io
Company:
https://www.linkedin.com/company/recro-io
Industries: Software Development, Business Consulting and Services, and Technology, Information and Media
Job details:
Experience: 3–5 Years
About the Role
We are looking for an experienced AI Infrastructure Engineer to build and scale the backend infrastructure powering next-generation AI applications. You will be responsible for designing highly available, low-latency systems that orchestrate multiple AI models, manage real-time conversations, and ensure resilient AI inference at production scale.
This role is ideal for engineers who enjoy solving distributed systems challenges, optimizing AI infrastructure, and building production-grade platforms that serve thousands of concurrent users.
Key Responsibilities
AI Infrastructure & Orchestration
- Design and implement asynchronous, event-driven AI orchestration systems.
- Build and maintain multi-agent AI workflows for complex conversational experiences.
- Own end-to-end latency optimization from user request to AI response.
- Develop resilient AI inference pipelines with retries, circuit breakers, and graceful fallback strategies.
- Implement intelligent request routing and load balancing across multiple AI models and providers.
- Migrate AI conversation services from monolithic architecture to scalable microservices.
- Build WebSocket/SSE-based streaming infrastructure for real-time AI responses.
- Optimize prompt execution, context management, batching, and response caching.
- Improve credit/data retrieval pipelines feeding AI conversations through efficient caching strategies.
- Design observability dashboards for latency, throughput, fallback triggers, and system health.
Required Skills & Experience
Backend & Distributed Systems
- 3–5 years of experience building production backend systems serving 10,000+ concurrent users.
- Strong expertise in asynchronous and event-driven architectures.
- Hands-on experience with:
- Kafka, RabbitMQ, or other message queues
- WebSockets or Server-Sent Events (SSE)
- Event streaming architectures
- Experience designing scalable microservices.
AI Infrastructure
- Production experience integrating multiple LLM providers such as:
- OpenAI
- Anthropic
- Gemini
- Azure OpenAI
- Self-hosted models using vLLM, Triton, or TensorFlow Serving
- Experience implementing:
- AI request routing
- Retry mechanisms
- Provider fallback strategies
- Rate-limit handling
- Understanding of:
- Prompt optimization
- Context window management
- Conversation state management
- Multi-turn AI interactions
- Experience troubleshooting production AI inference issues under heavy load.
Performance & Scalability
- Strong knowledge of caching strategies:
- Redis
- In-memory caching
- CDN-based caching
- Experience optimizing latency and throughput for AI applications.
- Familiarity with concurrency, load balancing, and fault-tolerant architectures.
Good to Have
- Experience with Agentic AI frameworks such as:
- LangGraph
- LangChain
- CrewAI
- Custom orchestration frameworks
- Experience building streaming chat applications similar to ChatGPT.
- Previous experience in fintech or payments.
- Experience decomposing monolithic systems into microservices.
- Familiarity with observability and monitoring tools:
- Grafana
- Prometheus
- Datadog
- Startup experience with ownership of end-to-end systems.
Preferred Technology Stack
- Languages: Python, Go, Java, or Node.js
- Caching: Redis
- Messaging: Kafka, RabbitMQ
- Streaming: WebSockets, Server-Sent Events (SSE)
- AI Platforms: OpenAI, Anthropic, Gemini, Azure OpenAI, vLLM, Triton, TensorFlow Serving
- Observability: Grafana, Prometheus, Datadog
- Architecture: Microservices, Event-Driven Systems
What Success Looks Like
- Deliver highly available AI infrastructure with minimal latency.
- Build resilient AI systems that gracefully handle provider failures.
- Improve response times through intelligent routing, caching, and batching.
- Scale AI conversations reliably for thousands of concurrent users.
- Drive engineering excellence through observability, automation, and performance optimization.
Why Join Us?
- Work on cutting-edge AI infrastructure powering real-world applications.
- Solve complex distributed systems and large-scale AI orchestration challenges.
- Own critical technical decisions impacting customer experience and business growth.
- Opportunity to shape and scale the AI platform as the company grows.
- Clear path toward technical leadership within the AI infrastructure team.
Click on Apply to know more.