Website:
agilegridsolution.com
Job details:
About The Company
PwC India is a leading professional services firm renowned for delivering exceptional audit, consulting, and advisory services across diverse industries. Committed to building trust and solving complex business challenges, PwC leverages innovative technology, deep industry insights, and a collaborative approach to help organizations achieve their strategic objectives. With a strong emphasis on integrity, quality, and excellence, PwC India continuously invests in talent development and cutting-edge solutions to stay ahead in a rapidly evolving marketplace.
About The Role
We are seeking a highly skilled LLM Observability Engineer to join our dynamic team based in Bhubaneswar. In this role, you will be responsible for ensuring the optimal performance, reliability, and cost-efficiency of our large language model (LLM) applications. Your expertise will be pivotal in designing performance testing strategies, implementing observability practices, and deriving actionable insights to enhance model quality and system robustness. This position offers an exciting opportunity to work at the forefront of AI and machine learning technologies, collaborating with cross-functional teams to deliver innovative solutions that drive business value.
Qualifications
- Bachelor’s or Master’s degree in Computer Science, Data Science, Software Engineering, or a related field.
- 4+ years of experience in performance testing, observability, or related roles within AI or software development environments.
- Strong proficiency in programming and scripting languages such as Python and Java.
- Hands-on experience with performance testing tools like JMeter, LoadRunner, or Gatling.
- Familiarity with LLM-specific challenges including non-determinism, hallucinations, and prompt sensitivity.
- Experience with observability platforms and tools such as Datadog LLM Observability, Arize AI, or Langfuse.
- Knowledge of telemetry data collection, monitoring, and analysis techniques.
- Excellent analytical, problem-solving, and communication skills.
Responsibilities
- Design and execute comprehensive performance tests to evaluate LLM application scalability, stability, and responsiveness under various load conditions.
- Implement observability practices by instrumenting LLM applications to capture telemetry data, including prompts, responses, token usage, latency, and errors, utilizing tools like Datadog, LangChain, or OpenTelemetry.
- Monitor and analyze key performance metrics in real-time dashboards, identifying trends, anomalies, and areas for improvement.
- Collaborate with development teams to identify bottlenecks, optimize system performance, and troubleshoot issues related to high latency, errors, or resource constraints.
- Develop and maintain automated evaluation processes for assessing model output quality, including hallucination detection, toxicity classification, and relevance scoring.
- Conduct root cause analysis of production issues by analyzing tracing and logging data, ensuring minimal downtime and optimal system performance.
- Monitor security and compliance aspects related to model behavior, including prompt injections and data leaks, ensuring adherence to data protection regulations.
- Track and manage token usage and computational resources to optimize costs, alerting teams to potential overspending or inefficiencies.
- Work closely with data scientists, ML engineers, and QA teams to provide insights, recommendations, and support for model fine-tuning and system architecture improvements.
Benefits
At PwC India, we believe in fostering a supportive and inclusive work environment that encourages professional growth. Employees enjoy competitive compensation packages, comprehensive health benefits, and opportunities for continuous learning and development. We provide flexible work arrangements, wellness programs, and a collaborative culture that values diversity and innovation. Additionally, our team members gain exposure to cutting-edge AI technologies and participate in impactful projects that shape the future of enterprise solutions.
Equal Opportunity
PwC India is an equal opportunity employer committed to fostering a diverse and inclusive workplace. We do not discriminate based on race, gender, age, religion, national origin, disability, or any other protected characteristic. We believe that diverse perspectives and experiences drive innovation and success, and we are dedicated to providing an equitable environment where all employees can thrive and reach their full potential.
Click on Apply to know more.