AI Safety Evaluation Specialist

AI Safety Evaluation Specialist at Keystone Recruitment

Keystone Recruitment is seeking experienced AI Safety Evaluation Specialists on behalf of one of our clients. This project-based, remote contract opportunity focuses on evaluating and strengthening advanced AI systems by identifying vulnerabilities, assessing systemic risks, and contributing to the development of more robust and reliable AI models.

Key Responsibilities

• Identify and document model limitations, vulnerabilities, and failure patterns.
• Annotate outputs and classify risks using structured taxonomies and evaluation frameworks.
• Develop reproducible testing cases and structured reports.
• Conduct adversarial testing across language models and socio-technical risk scenarios.
• Evaluate AI outputs against defined safety benchmarks and guidelines.
• Produce clear, actionable documentation for technical and non-technical stakeholders.
• Collaborate with project teams to improve model robustness and evaluation methodologies.

Required Qualifications

• Experience in AI safety, security research, red teaming, model evaluation, or related technical domains.
• Strong analytical and structured problem-solving skills.
• Ability to document findings clearly and reproducibly.
• Familiarity with risk assessment frameworks or benchmarking methodologies.
• Strong written communication skills.

Preferred Qualifications

• Experience with adversarial testing or model vulnerability research.
• Background in cybersecurity, machine learning, or socio-technical risk analysis.
• Experience working with evaluation datasets or safety benchmarking tools.
• Ability to translate complex technical findings into clear, structured reports.

Engagement Details

This is an independent contractor engagement offering fully remote work with flexible scheduling. It is project-based, with potential for extensions based on needs and performance. Compensation is hourly, ranging from $85–$185 depending on expertise and scope, with weekly payments via secure platforms. Applicants must be legally authorized to provide independent contractor services in their country of residence.

Key skills/competency

• AI Safety
• Model Evaluation
• Risk Assessment
• Adversarial Testing
• Machine Learning
• Cybersecurity
• Structured Problem-Solving
• Technical Documentation
• Evaluation Frameworks
• Language Models