Senior AI Engineer

About Netomi

Netomi is a leading agentic AI platform transforming enterprise customer experience. We partner with major global brands like Delta Airlines, MetLife, MGM, and United to deliver scalable agentic automation across the entire customer journey. Our no-code platform ensures rapid time-to-market, low total cost of ownership, and straightforward management of AI agents for diverse CX use cases. Supported by prominent investors including WndrCo, Y Combinator, and Index Ventures, Netomi empowers enterprises to enhance efficiency, reduce costs, and provide superior customer experiences.

Join us to be part of the AI revolution and help shape how the world’s largest brands operate!

About The Senior AI Engineer Role

As a Senior AI Engineer at Netomi, you will be instrumental in building automated systems designed to evaluate, test, and significantly improve the performance of conversational AI models, including Large Language Models (LLMs), chatbots, and voice assistants. This pivotal role bridges machine learning, software engineering, and QA automation, ensuring the deployment of high-quality, safe, and dependable AI systems at scale.

Key Responsibilities

• Develop Python-based pipelines for automated quality testing of AI responses.
• Integrate LLMs into sophisticated automated evaluation frameworks, utilizing tools like GPT-based evaluators, embeddings, or custom scoring mechanisms.
• Automate regression and stress testing specifically for conversational AI flows to ensure robustness.
• Define crucial evaluation metrics covering relevance, factuality, coherence, safety, and empathy of AI interactions.
• Implement a blend of rule-based and advanced AI-driven quality checks.
• Proactively monitor model drift, bias, and potential hallucinations through automated workflows.
• Work with APIs, SDKs, and CI/CD pipelines to seamlessly embed automated AI evaluation into production environments.
• Design and develop monitoring dashboards to provide clear visualizations of conversation quality metrics.
• Collaborate closely with ML engineers, product managers, and QA teams to establish an effective feedback loop for continuous improvement.
• Experiment with prompt engineering techniques and automated prompt-testing frameworks.
• Explore and implement advanced methods such as reinforcement learning, self-critique models, or human-in-the-loop automation for ongoing system enhancement.
• Automate compliance and policy adherence checks for enterprise AI systems, ensuring regulatory standards are met.

Requirements

• Strong 4-5 years of experience in Python development, covering automation, scripting, and efficient data handling.
• Proven experience with Large Language Models (LLMs) and Natural Language Processing (NLP) frameworks.
• Solid understanding of MLOps principles and AI deployment pipelines.

Key skills/competency

• Python development
• Large Language Models (LLMs)
• Natural Language Processing (NLP)
• Automated testing
• MLOps
• AI evaluation
• Prompt engineering
• CI/CD pipelines
• Data handling
• Conversational AI