Software Development Specialist

Software Development Specialist (Remote)

Join one of our clients as a Software Development Specialist and play a key role in designing evaluation-ready workflows for advanced AI systems. This contract-based engagement focuses on transforming ambiguous requirements into structured, repeatable artifacts that can be automatically tested.

You will produce clearly specified deliverables—including documentation and scripts—that enable consistent and reproducible assessment of agent performance across scenarios. The work is outcome-oriented, with strong emphasis on determinism, measurable acceptance criteria, and automation reliability.

Key Responsibilities

• Translate high-level objectives into tightly scoped, testable deliverables with clearly defined inputs, outputs, and measurable success criteria.
• Develop structured documentation outlining expected behavior, constraints, assumptions, and edge cases in a reusable format.
• Build lightweight automation scripts to support evaluation workflows, including artifact generation and output validation.
• Write deterministic Python verifier scripts that confirm completion through final state or output validation (files, directories, structured content checks).
• Design prompts and tasks that reliably trigger the intended workflow behavior while preventing instruction leakage.
• Implement strong error handling and provide clear, actionable failure messages within verification tooling.
• Create plausible but ineffective baseline or distractor approaches to validate evaluation discrimination.
• Maintain clean artifact hygiene with version-ready structure, consistent naming, and reproducible execution patterns.

Required Skills and Qualifications

• Advanced Python proficiency, including file system operations, parsing, validation logic, and deterministic execution.
• Experience with automated grading systems, evaluation harnesses, or QA-style verification workflows.
• Familiarity with prompt engineering and LLM evaluation methodologies, including closed-ended task design and leakage avoidance.
• Strong documentation skills using structured formats such as Markdown and YAML.
• Working knowledge of Git, CLI-based workflows, virtual environments, and dependency management.
• Ability to define precise requirements and maintain tight scope control without relying on domain-specific assumptions.
• Clear written communication skills with strong attention to detail.

Preferred Qualifications

• Experience designing negative controls or distractor solutions for evaluation robustness.
• Understanding of embeddings and similarity techniques (e.g., cosine similarity) for relevance testing.
• Background in reliability testing, deterministic system design, or reproducibility-focused workflows.

Additional Role Details

• Deliverables primarily consist of documentation and scripts intended for automated evaluation replay.
• Strong emphasis on determinism, reproducibility, closed-ended outcomes, and validator reliability.
• Tasks must resist superficial shortcuts and confirm that the intended workflow is actually executed.
• Time-sensitive components should include explicit date boundaries where applicable.

Contract and Payment Terms

• Engagement as an independent contractor.
• Fully remote with flexible scheduling.
• Projects may be extended, shortened, or concluded early depending on performance and business needs.
• No access to confidential or proprietary information from any employer, client, or institution.
• Weekly payments via Stripe or Wise based on services rendered.
• Unable to support H1-B or STEM OPT candidates at this time.

Key skills/competency

• Python
• AI Systems Evaluation
• Automated Testing
• Prompt Engineering
• LLM Evaluation
• Documentation (Markdown, YAML)
• Git
• CLI Workflows
• Deterministic System Design
• Reproducibility