Principal Data Scientist - Agent Builder
Elastic · Sweden
- Hybrid
- Full-time
- $180,000 / year
- Sweden
Job highlights
- Lead AI strategy for conversational search platform.
- Define and implement LLM evaluation frameworks.
- Develop RAG, agent, and tool quality metrics.
- Collaborate with engineering to productionize AI systems.
- Mentor data scientists on LLM improvement techniques.
About the role
About Elastic, the Search AI Company
Elastic enables everyone to find the answers they need in real time, using all their data, at scale — unleashing the potential of businesses and people. The Elastic Search AI Platform, used by more than 50% of the Fortune 500, brings together the precision of search and the intelligence of AI to enable everyone to accelerate the results that matter. By taking advantage of all structured and unstructured data — securing and protecting private information more effectively — Elastic’s complete, cloud-based solutions for search, security, and observability help organizations deliver on the promise of AI.
The Role: Search Conversational Experiences Team
The Search Conversational Experiences team builds Elastic’s new conversational and agentic platform that lets customers chat with their own data in Elasticsearch. We build the core quality layer for RAG, agents and tools, retrieval and citations, streaming, memory, and the evaluation signals that turn open-ended questions into grounded, reliable answers.
Principal Data Scientist Responsibilities
As a Principal Data Scientist, you will help set the technical direction for how we evaluate, improve, and scale chat quality across Elastic’s agentic platform. You will define the evaluation strategy that guides product decisions, including which models we standardize on, how we route requests across agents, which tools we enable and when, and how we tailor agents to different Elastic use cases in search and beyond. You will work closely with backend engineering, product, UX, and other data scientists to turn ambiguous, cutting-edge problems into measurable product improvements.
You’ll help lead work on frontier problems such as folding RAG and vector search into an agent’s knowledge base, dynamically enriching model context to improve groundedness, shaping reasoning strategies and tool-selection policies, lighting up agent-driven visualizations on top of Elasticsearch data, and exploring multimodality where it can create meaningful user value. This is an applied leadership role: you will prototype, evaluate, influence roadmap direction, and help teams ship improvements that customers can feel.
What You Will Be Doing
- Define the evaluation strategy for conversational and agentic search, including offline and online evaluation, golden datasets, rubrics, LLM-as-judge calibration, groundedness and citation checks, and A/B testing.
- Lead the design of quality metrics and decision frameworks for RAG, agents, tools, model selection, agent routing, prompt behavior, and cost/latency trade-offs.
- Build, compare, and guide improvements across retrieval and re-ranking approaches, including sparse and dense retrieval, vector search, query understanding, semantic rewrites, and context enrichment.
- Turn experimental results into product and business decisions: which models to use, how to route requests efficiently, which tools should be exposed, and how agents should be customized for different Elastic use cases.
- Partner with engineering to productionize evaluation pipelines, telemetry, dashboards, CI guardrails, and regression detection for chat quality, helpfulness, dedication, latency, and cost.
- Influence the roadmap by identifying the highest-leverage quality gaps, proposing practical solutions, and communicating trade-offs clearly to product, engineering, and leadership.
- Mentor other data scientists and engineers in experiment design, evaluation methodology, statistical rigor, and practical approaches to improving LLM-powered systems.
- Share outcomes through clear docs, notebooks, PRs, dashboards, technical proposals, and cross-functional reviews.
What You Bring
- 8+ years of applied DS/ML experience, with deep expertise in IR, NLP, ranking, semantic search, RAG, or LLM-powered product experiences.
- Strong track record defining and leading evaluation for production AI/ML systems, including offline metrics, online experimentation, LLM-as-judge approaches, groundedness, citation quality, and model comparison.
- Experience influencing product and technical strategy through data, especially in ambiguous or emerging domains where the “right” metric or approach is not obvious at the start.
- Hands-on ability with Python, PyTorch/Transformers, Pandas, notebooks, reproducible experiments, versioned datasets, and clean, reviewable code.
- Strong understanding of retrieval systems, including dense and sparse retrieval, re-ranking, vector search, query understanding, and evaluation metrics such as nDCG, MRR, Recall@k, precision, and latency/cost trade-offs.
- Experience collaborating closely with engineering teams to move from prototype to production, including telemetry design, dashboards, CI guardrails, and quality regression tracking.
- Practical Elasticsearch experience, or experience with similar search and distributed data systems. ES|QL familiarity is a plus.
- Excellent written and verbal communication, with the ability to explain complex scientific and technical trade-offs to engineering, product, design, and leadership audiences.
- A collaborative, low-ego style and a strong ability to mentor, raise standards, and develop transparency for others in a distributed team.
Additional Information - We Take Care of Our People
As a distributed company, diversity drives our identity. Whether you’re looking to launch a new career or grow an existing one, Elastic is the type of company where you can balance great work with great life. Your age is only a number. It doesn’t matter if you’re just out of college or your children are; we need you for what you can do.
We strive to have parity of benefits across regions, and while regulations differ from place to place, we believe taking care of our people is the right thing to do.
- Competitive pay based on the work you do here and not your previous salary
- Health coverage for you and your family in many locations
- Ability to craft your calendar with flexible locations and schedules for many roles
- Generous number of vacation days each year
- Increase your impact - We match up to $2000 (or local currency equivalent) for financial donations and service
- Up to 40 hours each year to use toward volunteer projects you love
- Embracing parenthood with a minimum of 16 weeks of parental leave
Security & Privacy Responsibilities
Take ownership of protecting the confidentiality, integrity, and availability of organizational data and systems by following applicable privacy and security policies, standards, and procedures. Ensure that all individual contributions follow Elastic’s Secure Software Development Framework (SSDF). Proactively participate in mandatory role-based training to ensure personal technical execution consistently aligns with the highest standards of data protection, data privacy, and system resilience.
Equal Opportunity Employer
Different people approach problems differently. We need that. Elastic is an equal opportunity employer and is committed to creating an inclusive culture that celebrates different perspectives, experiences, and backgrounds. Qualified applicants will receive consideration for employment without regard to race, ethnicity, color, religion, sex, pregnancy, sexual orientation, gender perception or identity, national origin, age, marital status, protected veteran status, disability status, or any other basis protected by federal, state or local law, ordinance or regulation.
We welcome individuals with disabilities and strive to create an accessible and inclusive experience for all individuals. To request an accommodation during the application or the recruiting process, please email candidate_accessibility@elastic.co. We will reply to your request within 24 business hours of submission.
Applicants have rights under Federal Employment Laws and can view the following posters linked below:
- Family and Medical Leave Act (FMLA) Poster
- Employee Polygraph Protection Act (EPPA) Poster
Elasticsearch develops and distributes technology and information that is subject to U.S. and other countries’ export controls and licensing requirements for individuals who are located in or are nationals of the following sanctioned countries and regions: Belarus, Cuba, Iran, North Korea, Syria, or Russia, including the Ukrainian territories annexed by Russia (The Crimea region of Ukraine, The Donetsk People's Republic (DNR), The Luhansk People's Republic (LNR), Kherson or Zaporizhzhia). If you are located in or are a national of one of the listed countries or regions, an export license may be required as a condition of your employment in this role. Please note that national origin and/or nationality do not affect eligibility for employment with Elastic.
Please see here for our Privacy Statement.
Key skills/competency
- Principal Data Scientist
- Agent Builder
- Search AI
- Conversational AI
- RAG (Retrieval Augmented Generation)
- LLM Evaluation
- NLP (Natural Language Processing)
- Information Retrieval (IR)
- Machine Learning (ML)
- Python
Skills & topics
- Data Scientist
- Machine Learning
- AI
- LLM
- RAG
- NLP
- Information Retrieval
- Elasticsearch
- Python
- Evaluation
How to get hired
- Tailor your resume: Highlight 8+ years of DS/ML experience, IR, NLP, RAG, and LLM evaluation expertise.
- Showcase experience: Detail your track record in leading evaluation for production AI/ML systems and influencing strategy with data.
- Demonstrate technical skills: Emphasize proficiency in Python, PyTorch/Transformers, Pandas, and reproducible research.
- Prepare for interviews: Be ready to discuss retrieval systems, LLM evaluation methodologies, and collaborate with engineering teams.
- Highlight communication: Showcase your ability to explain complex technical trade-offs to diverse audiences.
Technical preparation
Behavioral questions
Frequently asked questions
- What specific experience does Elastic look for in a Principal Data Scientist - Agent Builder?
- Elastic seeks candidates with 8+ years of applied DS/ML experience, deep expertise in IR, NLP, ranking, semantic search, RAG, or LLM-powered product experiences. A strong track record in defining and leading evaluation for production AI/ML systems, including offline metrics, online experimentation, and LLM-as-judge approaches, is crucial. Hands-on ability with Python, PyTorch/Transformers, Pandas, and experience collaborating with engineering teams are also key.
- What are the primary responsibilities of the Principal Data Scientist role at Elastic?
- The Principal Data Scientist will set the technical direction for evaluating, improving, and scaling chat quality on Elastic's agentic platform. This involves defining the evaluation strategy, leading the design of quality metrics for RAG and agents, building and improving retrieval approaches, and influencing product roadmap decisions through experimental results. They will also partner with engineering to productionize evaluation pipelines and mentor other data scientists.
- How does Elastic approach AI/ML system evaluation in this role?
- Elastic emphasizes a comprehensive evaluation strategy for conversational and agentic search. This includes offline and online evaluation, golden datasets, rubrics, LLM-as-judge calibration, groundedness and citation checks, and A/B testing. The role involves designing quality metrics and decision frameworks for various components like RAG, agents, model selection, and routing, focusing on cost/latency trade-offs.
- What technical skills are essential for the Principal Data Scientist position at Elastic?
- Essential technical skills include strong proficiency in Python, PyTorch/Transformers, and Pandas. Candidates should have experience with notebooks, reproducible experiments, versioned datasets, and clean, reviewable code. A deep understanding of retrieval systems (dense and sparse retrieval, vector search, query understanding) and their evaluation metrics is vital. Familiarity with Elasticsearch or similar distributed data systems is also important.
- How does Elastic foster diversity and inclusion in its hiring process for a Principal Data Scientist?
- Elastic is committed to creating an inclusive culture that celebrates different perspectives. They are an equal opportunity employer and consider all qualified applicants without regard to various protected characteristics. Elastic also welcomes individuals with disabilities and provides accommodations during the application and recruiting process by emailing candidate_accessibility@elastic.co.