Product Manager

Overview

The Copilot Offline Evaluation Platform (COEP) is the foundational system that empowers Copilot product teams to run deterministic, reproducible, and representative evaluations at scale. As a Product Manager on the COEP team, you will help define and deliver capabilities that streamline evaluation workflows across our platform components. You will partner closely with engineering, data science, and scenario teams to build the tools and abstractions that ensure Copilot quality keeps pace with rapid product innovation.

This role is ideal for a systems thinker who loves simplifying complexity, building for scale, and driving clarity across deeply technical problem spaces.

Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

Responsibilities

Own Product Areas Within the Evaluation Lifecycle

• Evalset generation and configuration flows (tenant profiles, grounding data, assertions, ingestion graphs)
• Evaluation environment provisioning and reuse
• Integration with EGP, EMP, and C‑Store orchestrators
• Data/DP Pipeline integration for scenario‑specific query pattern generation
• Agentic UX and natural‑language-driven evaluation tooling enabling end-to-end flow without manual engineering intervention

Deeply Understand COEP Customer Personas

You Will Advocate For Users Across The Platform

• Scenario Owners, who define evaluation goals, author prompts/assertions, and drive evaluation demand
• Feature Developers, who execute eval jobs and review metrics without needing deep infra knowledge
• Quality teams and connector owners (e.g., GCO) who depend on precision, recipes, and repeatability for high-quality evaluations

Drive Clarity & Execution Across Engineering and Partner Teams

• Break down ambiguous, cross-system work into crisp requirements, aligned around customer workflows
• Partner with EGP, EMP, DP Pipeline, and C‑Store engineering teams to ship high‑quality platform capabilities
• Contribute to shared COEP reviews, Product Manager team syncs, and cross-org syncs
• Produce specs, requirements, UX flows, and success metrics for your feature areas

Deliver Platform Reliability, Quality, and Usability

• Ensure features support determinism, reproducibility, and tenant coherence
• Define MVP criteria, success KPIs, backward compatibility expectations, and instrumentation needs
• Drive E2E dogfooding sessions, bug bashes, and quality validation efforts for major milestones (e.g., COEP Agent MVP)

Champion Platform Simplification and Scale

• Identify opportunities to reduce friction, automate manual steps, and unify evaluation workflows
• Simplify onboarding and enable self-serve evaluation flows for hundreds of Copilot product teams
• Create documentation, recipes, templates, and customer-ready guidance to accelerate adoption

Qualifications

Required Qualifications

• Bachelor's Degree AND 3+ years experience in product/service/program management or software development OR equivalent experience.

Preferred Qualifications

• Bachelor's Degree AND 8+ years experience in product/service/program management or software development OR equivalent experience.
• 2+ years experience taking a product, feature, or experience to market (e.g., design, addressing product market fit, and launch, internal tool/framework).
• 4+ years experience improving product metrics for a product, feature, or experience in a market (e.g., growing customer base, expanding customer usage, avoiding customer churn).
• 4+ years experience disrupting a market for a product, feature, or experience (e.g., competitive disruption, taking the place of an established competing product).
• Experience with ML/AI systems, evaluation tools, or developer platforms
• Familiarity with Copilot products and workflows.
• Experience building natural‑language or agentic workflows.
• Solid systems thinking, execution skills, and technical aptitude.

Key skills/competency

• Product Management
• Evaluation Platforms
• Systems Thinking
• AI/ML Integration
• Data Pipelines
• User Advocacy
• Cross-functional Leadership
• Scalability Solutions
• Technical Aptitude
• Workflow Automation