Software Engineer- AI Testing + Workflow Valida...
@ Red River

Boston, Massachusetts, United States
On Site
Full Time
Posted 5 days ago

Your Application Journey

Personalized Resume
Apply
Email Hiring Manager
Interview

Email Hiring Manager

XXXXXXXXX XXXXXXXXXXX XXXXXXXX****** @redhat.com
Recommended after applying

Job Details

Job Summary

As a Software Engineer, AI Testing & Prompt Engineering at Red River, you will design, test, and refine prompts that enable AI models to generate code and execute automated workflows. You will work with engineering, QA, and product teams to ensure AI-driven features are accurate, reliable, and aligned with both user and technical requirements.

What You Will Do

  • Design, iterate, and optimize prompts for AI-assisted code generation and workflow execution.
  • Collaborate with product owners, developers, and QA to translate feature requirements into effective prompting strategies and evaluation benchmarks.
  • Validate AI-generated outputs via structured test cases, regression testing, and performance evaluation.
  • Build or adapt internal tooling to streamline prompt testing, versioning, and deployment.
  • Capture insights from prompt performance and document patterns, metrics, and best practices.

What You Bring

  • Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.
  • Hands-on experience with large language models (e.g., GPT, LLaMA, Claude) for code generation or workflow automation.
  • Strong coding skills in Python or similar languages.
  • Familiarity with testing methodologies including unit, integration, and regression testing.
  • Excellent analytical, problem-solving, and communication skills.

Nice-to-Have

  • Experience with AI-assisted development tools or IDE integrations.
  • Knowledge of prompt-engineering best practices, bias mitigation, and output evaluation metrics.
  • Familiarity with model performance metrics such as functional correctness, ROUGE, or BLEU.

Benefits

Comprehensive medical, dental, vision, retirement 401(k) with employer match, PTO, parental leave and additional benefits available for full time, permanent associates in the United States.

Key Skills/Competency

  • AI Testing
  • Prompt Engineering
  • Workflow Validation
  • Python
  • Large Language Models
  • Automated Testing
  • Code Generation
  • Collaboration
  • Problem-solving
  • Documentation
Apply without a personalized resume

How to Get Hired at Red River

🎯 Tips for Getting Hired

  • Customize your resume: Tailor your skills to prompt engineering requirements.
  • Highlight AI projects: Emphasize experience with large language models.
  • Show collaborative mindset: Detail teamwork in technical projects.
  • Prepare for technical tests: Brush up on coding and test methodologies.

📝 Interview Preparation Advice

Technical Preparation

Review Python coding challenges.
Study AI model performance metrics.
Practice prompt optimization techniques.
Familiarize with automated testing frameworks.

Behavioral Questions

Describe a team project challenge.
Explain conflict resolution in projects.
Highlight adaptability in changing tasks.
Showcase communication in technical teams.

Frequently Asked Questions