AI Workflow Analyst
Sophie Society Co. · America, SD
- Hybrid
- Full-time
- $60,000 / year
- America, SD
Job highlights
- Test AI tools for Amazon tasks.
- Ensure AI output accuracy and safety.
- Write test cases and scoring criteria.
- Identify and fix AI issues.
- Work remotely with flexible hours.
About the role
AI Workflow Analyst at Sophie Society
Sophie Society is building AI tools designed to streamline real Amazon tasks, including campaign audits, search term analysis, and listing reviews. This role is crucial for ensuring our AI outputs are accurate, useful, and safe before they are deployed on live client accounts.
What You'll Do
- Create test cases using real Amazon data such as search term reports, campaigns, and listings.
- Run AI tools and meticulously review their outputs.
- Check for critical issues including: Wrong keyword classification, Bad campaign structure decisions, Missing or incorrect insights, Hallucinated data.
- Develop simple scoring criteria to evaluate the quality of AI outputs.
- Identify the root cause of issues, whether it's the prompt, input data, or the model itself.
- Collaborate with developers to implement fixes and enhance results.
- Approve or block AI tools before their official release.
Requirements
- Strong hands-on experience with ChatGPT and Claude.
- Ability to write clear and effective prompts, refining them through rigorous testing.
- Exceptional attention to detail; the capacity to identify errors that others overlook.
- Excellent clear and structured writing skills.
- Comfortable working with Git, Markdown, and YAML.
- Familiarity with reading basic Python or TypeScript is required.
Nice to Have
- Prior experience with Amazon PPC, including campaigns, search terms, and campaign structure.
- Experience utilizing tools such as LangSmith, Promptfoo, or similar platforms.
- Previous experience working within an agency environment.
What We Offer
- A fully remote work environment, allowing you to work from anywhere.
- Full-time position (40 hours per week) with flexible scheduling.
- Competitive salary range of $35,000–$60,000 USD, commensurate with experience.
Sophie Society has a proven track record, having assisted over 2000 brands, managed more than 2400 product launches, and generated over $200 million in ad revenue. We also proudly host 'The Profitable PPC Challenge', the world's most attended digital event on Amazon PPC.
Key skills/competency
- AI Workflow Analyst
- Prompt Engineering
- AI Testing
- Amazon PPC
- Data Analysis
- Problem Solving
- Quality Assurance
- ChatGPT
- Claude
- Python
Skills & topics
- AI Workflow Analyst
- AI Testing
- Prompt Engineering
- Amazon PPC
- Data Analysis
- Quality Assurance
- ChatGPT
- Claude
- Remote Work
- Full-time
How to get hired
- Tailor your resume: Highlight AI testing, prompt engineering, and Amazon PPC experience.
- Showcase attention to detail: Provide examples of how you've caught critical errors.
- Demonstrate prompt skills: Mention experience with ChatGPT, Claude, and prompt refinement.
- Explain technical comfort: Detail your familiarity with Git, Markdown, YAML, and basic Python/TypeScript.
- Research Sophie Society: Understand their mission and impact on Amazon brands.
Technical preparation
Master ChatGPT and Claude prompt techniques.,Practice creating test cases from raw data.,Familiarize yourself with Git and YAML.,Review basic Python or TypeScript syntax.
Behavioral questions
Describe a time you found a subtle error.,How do you approach improving unclear instructions?,Explain how you collaborate with technical teams.,How do you prioritize tasks with strict deadlines?
Frequently asked questions
- What are the core responsibilities of an AI Workflow Analyst at Sophie Society?
- As an AI Workflow Analyst at Sophie Society, your primary responsibility is to rigorously test AI tools designed for Amazon tasks. This involves creating test cases, evaluating AI outputs for accuracy and safety, identifying issues, and collaborating with developers to resolve them before the tools go live.
- What technical skills are essential for the AI Workflow Analyst role at Sophie Society?
- Essential technical skills include strong hands-on experience with AI models like ChatGPT and Claude, the ability to write and refine prompts, comfort with Git, Markdown, and YAML, and a basic understanding of Python or TypeScript. Experience with Amazon PPC is also highly valued.
- Is this AI Workflow Analyst position fully remote?
- Yes, the AI Workflow Analyst position at Sophie Society is fully remote, offering the flexibility to work from anywhere. The company emphasizes a comfortable work environment regardless of your location.
- What kind of experience is considered 'nice to have' for this AI Workflow Analyst role?
- Experience with Amazon PPC (campaigns, search terms, structure), familiarity with AI testing tools like LangSmith or Promptfoo, and prior experience working in an agency setting are considered beneficial 'nice to have' qualifications for this role.
- How does Sophie Society evaluate the AI outputs in this role?
- AI outputs are evaluated based on accuracy, usefulness, and safety. You will be responsible for checking for issues like incorrect keyword classification, bad campaign structure decisions, missing insights, or hallucinated data, and developing simple scoring criteria to assess their quality.
- What is the typical work arrangement for this AI Workflow Analyst position?
- This is a full-time position, requiring approximately 40 hours per week. Sophie Society offers flexibility around your schedule within this full-time commitment.
- What is the salary range for an AI Workflow Analyst at Sophie Society?
- The salary range for an AI Workflow Analyst at Sophie Society is between $35,000 and $60,000 USD per year, with the final offer based on your experience and qualifications.