About this role
Summary: Designs and runs evaluations of AI models on demanding technical work, turning findings into training signals and next steps. Builds task suites, scoring methods, and repeatable experiment pipelines with subject-matter specialists.
About the company
Our client is an AI company.
The role
Raydar is recruiting for this role on behalf of our client. Design and run evaluations that test how well AI models handle demanding technical work, then turn the findings into training signals and next steps. The job spans task construction, scoring, reinforcement learning environments and repeatable experiment pipelines, carried out with subject-matter specialists.
What you'll do
- Set up task suites that mirror genuine investigator needs, complete with documented methods, reference scores and data withheld from training.
- Turn domain data and specialist procedures into runnable tasks with defined inputs, permitted tools, target results and verification steps.
- Write scoring criteria and automated verifiers, and probe them for vagueness and loopholes a model could exploit.
- Mine error patterns for the training tasks most worth pursuing, and check whether gains hold on unseen problems.
- Stand up repeatable systems that generate tasks, run models, score outputs and summarize results, with logs of inputs, configurations and task revisions.
- Review what models produce and how they invoke tools, then classify each miss as a model shortfall or a flaw in the task, its framing or its scoring.
- Partner with domain specialists so their expertise shapes task construction and checking, and agree on what a successful answer looks like.
- Report on gains, open questions and the next experiment or engineering step to take.
What we're looking for
- Track record in building test suites, scoring methods or data sourcing for model evaluation.
- Hands-on background designing benchmarks and assessing AI systems, ideally from a firm focused on AI data or evaluation.
- Enough machine learning background to judge what a model can and cannot do.
- Comfort with Python and SQL.
Bonus points
- Past lab work in a life sciences, pharmaceutical or materials setting.
- Previous experience in a senior research, applied ML, science lead or project lead role.
- A degree or academic background in biology, biomedical or chemical engineering, physics or a related field, or equivalent biotech or pharma work experience.
- Familiarity with reward schemes that can be checked automatically, agents that call tools, or reinforcement learning across several steps.
- Past assessment of analytical or investigative work performed by models.
- Past creation of tooling that generates tasks automatically, audits quality or coordinates expert labeling.
Compensation and benefits
- Base salary: USD 160,000 to 200,000 per year
- Equity
- Health coverage
Location and work model
- New York, NY, United States
- On-site, 5 days in-office per week
- Full-time