
AI Prompt & Agent Developer
OpenCall.ai (YC W24) · San Francisco, CA
- On site
- Full-time
- $135,000 / year
- San Francisco, CA
Job highlights
- Develop and maintain AI voice agent prompts in production.
- Iteratively ship fixes using real call data daily.
- Build robust evaluation harnesses for prompt optimization.
- Onboard new customers by configuring AI agents.
- Continuously improve AI performance through QA.
About the role
About OpenCall
OpenCall's voice AI handles calls for multi-location medical groups. We’re solving a unique challenge: pushing the limits of AI at millisecond performance to have the best human-like customer service experience at enterprise scale. Our AI is faster, cheaper, more powerful, and more reliable than anything else on the market. We’re looking for versatile developers to help scale our proprietary system from millions of calls to billions of calls annually. We're hiring an AI Prompt & Agent Developer to own behavioral slice(s) of our voice agents. That behavior splits into two categories: behavior shared across every deployment, and behavior specific to a subset of deployments. You're someone who actually enjoys looking at the data, because the data informs everything else. You'll write prompts, design subagent architectures, build evals, and push automation rates up one small, measurable win at a time.Responsibilities
- Write and maintain the prompts that run in production. This includes intent classification, information extraction, availability negotiation, closing phrases, insurance verification flows, objection handling, and edge-case recovery. You own behavior that touches every customer call.
- Ship iteratively against real call data. Every morning, you'll listen to failed calls from yesterday. Every afternoon, you'll deploy a fix. You’ll be using and helping to develop dashboards, call review tooling, and automated agents to accelerate the work.
- Build evaluation harnesses. You'll develop offline eval sets, run automated prompt optimization (we use GEPA-style approaches), and establish the test suites that let us ship changes without breaking live deployments.
- Human-in-the-loop onboarding. New customers come online constantly. You'll work with and iterate on our internal AI agents that translate a practice's intake form, their scheduling rules, and their quirks into an agent configuration. Every week, you'll be designing new evaluation metrics for these customers and helping to improve existing ones.
- QA and continuous improvement. You'll simulate real-world customer scenarios, measure outcomes, and monitor production agent performance so you can catch drift early and fix it fast.
What we're looking for
- You've shipped prompts that broke production. Doesn't matter if it was at OpenAI, a chatbot startup, a research lab, or your own project. What matters is that you've felt the specific pain of a prompt that worked beautifully in dev and broke the second it hit real users.
- You're meticulous and careful. Looking at data for long stretches energizes you, as long as there's a signal. You stay organized when five things are in flight. We deploy multiple times a day, and we also run healthcare workflows where a bad change costs real money for real practices. You know the difference between moving fast and breaking things.
- Writing sensibility. The best prompt engineers are good writers. You notice register, rhythm, and word choice. You can tell why "Hello, cornerside dental? This is Ava, how can I help you out today?" sounds warmer than "Hello, Cornerside Dental, this is Ava. How can I help you out today" out of a TTS.
- Analytical and empirical. You are relentlessly data-driven. Before you make changes, you proactively run experiments and measure. You don't ship because "I think this is better." You justify a change with "this moved booking rate from 78.2% to 81.4% on n=412 calls."
- Comfort with code. You don't need to be a senior engineer, but you should read Python fluently and TypeScript comfortably, and you can get almost any coding task done by pairing with modern AI coding tools.
Requirements
- 2+ years of experience with AI/ML, NLP, or prompt engineering in production
- Strong analytical and problem-solving mindset; comfort with ambiguity
- Excellent written and verbal communication skills
- Bachelor's degree and/or extensive experience in one or more of: Computer Science, Engineering, Math, Philosophy, Linguistics, Cognitive Science, English, Medicine, or a related field
Preferred Qualifications
- Python chops beyond reading: APIs, data pipelines, testing frameworks
- Prior work with voice AI, TTS, ASR, or telephony platforms (Twilio, etc.)
- Contact center, SaaS, or customer-facing tech background
- Healthcare or medical operations experience — you know what an NPI is, you've worked a front desk, you understand the weird chaos of dental scheduling
- Automated prompt optimization experience (DSPy, GEPA, MIPROv2)
- Fine-tuning experience
Key skills/competency
- Prompt Engineering
- AI Agent Development
- Natural Language Processing (NLP)
- Data Analysis
- Python
- TypeScript
- Evaluation Metrics
- Automation
- Voice AI
- Healthcare Technology
Skills & topics
- AI Prompt Engineering
- Agent Development
- NLP
- Python
- TypeScript
- Voice AI
- SaaS
- Healthcare Tech
- Data Analysis
- Software Development
How to get hired
- Tailor your resume: Highlight your experience shipping prompts, data analysis, and Python/TypeScript skills for AI Prompt & Agent Developer roles.
- Showcase your impact: Quantify achievements with metrics, such as "moved booking rate from X% to Y%".
- Demonstrate writing and analytical skills: Provide examples of your meticulous approach to prompt engineering and data-driven decision-making.
- Prepare for technical and behavioral questions: Be ready to discuss prompt failures, your problem-solving process, and collaboration experience.
- Understand OpenCall's mission: Research their AI voice solutions for medical groups and their focus on performance and scale.
Technical preparation
Practice prompt engineering with Python and TypeScript.,Build evaluation harnesses for AI models.,Analyze call data for performance improvements.,Familiarize with voice AI and telephony platforms.
Behavioral questions
Describe a time a prompt failed in production.,How do you approach data-driven decision-making?,Explain your meticulous process for QA and improvement.,How do you balance speed with production stability?
Frequently asked questions
- What does an AI Prompt & Agent Developer at OpenCall.ai do?
- As an AI Prompt & Agent Developer at OpenCall.ai, you will be responsible for writing and maintaining production prompts for their voice AI system, designing subagent architectures, building evaluation harnesses, and improving automation rates. You'll work with real call data to iterate on AI behavior and ensure high-quality customer service experiences for medical groups.
- What kind of experience is OpenCall.ai looking for in an AI Prompt & Agent Developer?
- OpenCall.ai is seeking candidates with at least 2 years of experience in AI/ML, NLP, or production prompt engineering. They value meticulousness, strong analytical and problem-solving skills, excellent written and verbal communication, and comfort with Python and TypeScript. Experience with voice AI and healthcare operations is a plus.
- How important is data analysis for an AI Prompt & Agent Developer at OpenCall.ai?
- Data analysis is critical. The role requires a relentless, data-driven approach where changes are justified by measured improvements in metrics like booking rates. You'll be expected to dive into call data to identify issues and inform prompt development.
- What does 'shipping prompts that broke production' mean for this role?
- This phrase emphasizes the value OpenCall.ai places on practical, real-world experience. They want to know you've encountered the challenges of prompt engineering in a live environment, learned from prompt failures, and understand the critical difference between development and production performance.
- Is this AI Prompt & Agent Developer role remote or on-site?
- This AI Prompt & Agent Developer role is an on-site position located in San Francisco.
- What kind of technical skills are essential for this AI Prompt & Agent Developer job?
- Fluency in reading Python and comfort with TypeScript are essential. Experience with APIs, data pipelines, testing frameworks, and ideally voice AI or telephony platforms like Twilio is also highly beneficial.
- What is the compensation range for an AI Prompt & Agent Developer at OpenCall.ai?
- The compensation for an AI Prompt & Agent Developer at OpenCall.ai ranges from $75,000 to $135,000 annually, plus equity.
- How does OpenCall.ai handle continuous improvement for their AI agents?
- OpenCall.ai focuses on continuous improvement through daily iteration on call data, building evaluation harnesses, simulating real-world scenarios, and monitoring production agent performance to catch and fix issues early.