About this role
Summary: Evaluates an AI model that scores audience behavior by building training datasets, running human-rated scoring, and validating results against real audience behavior.
About DDM Group
DDM Group is a digital media company with three properties: Atlanta Black Star, The Shadow League and Finurah. Over 14 years, DDM grew its audience to more than 20 million monthly users at peak entirely through organic distribution, without paid traffic. DDM has turned what it learned into a proprietary model of what captures an audience's attention and what moves people to change their behavior. We are now testing that model rigorously, and building it into an AI-powered service.
The role
You will own the evaluation of the model from raw data to verdict. You will build the dataset from 14 years of content and analytics (nearly 100,000 published articles on Atlanta Black Star alone, each with Google Analytics history), have AI models score the full archive, validate those scores against trained human raters on a sample, test whether those scores predict real audience behavior, and run live experiments on our properties. Your findings decide what we build and where we invest. We want an honest answer, including if the answer is no. If the model holds up, you will turn the evaluation pipeline into the core of the product: the scoring engine, the generator that proposes signals and plans, and the experiment and learning loop.
This is a remote contract role starting immediately. You will report to Neil Nelson, CEO, who owns the model, and work closely with our editorial team, whose producers serve as raters.
What you will do
- Build the evaluation dataset: export story text and metadata from our WordPress sites, pull Google Analytics 4 data by traffic source, and recover older data from other sources where needed.
- Design a balanced, representative sample, including stories that failed, not only the hits.
- Run the scoring: set up blind rating, train and calibrate human raters with the editorial lead, measure how well raters agree, and write and validate the prompts for AI raters against human judgment.
- Test the model: fit weights on one part of the data and validate them on data held back; compare alternative ways of combining scores; control for distribution effects such as platform algorithm changes and traffic source.
- Test early-warning indicators on historical audience data.
- Design, size and analyze live A/B tests on our properties.
- Keep a clean, reproducible record of every dataset, decision, score, prediction and result.
- Report findings plainly to the CEO, against success criteria agreed before the analysis begins.
### Position Requirements
Required qualifications
- 5 or more years in applied machine learning or data science, with at least one project you owned end to end, from raw data to a decision someone acted on.
- Experimentation: designed, sized and analyzed A/B tests or randomized experiments, including power calculations and pitfalls such as peeking and multiple comparisons.
- Statistics for prediction and validation: regression and classification, out-of-sample and time-based validation, and measures of rater agreement such as intraclass correlation or Krippendorff's alpha.
- Data engineering for analysis: fluent SQL; experience with BigQuery or a similar warehouse; Python (pandas, statsmodels or scikit-learn).
- Web analytics: hands-on work with Google Analytics 4, including traffic-source attribution and its limits.
- Language models as measurement tools: prompted large language models through an API to classify or score text at scale (ideally 100,000 documents or more), managed the cost, and checked the output against human judgment.
- Production systems: built and shipped at least one ML or language-model system that other people relied on, including evaluation, monitoring and cost control.
- Judgment and candor: you can explain a result to a non-technical executive in plain language, and you report findings that contradict expectations.
Preferred
- Experience in digital media, publishing, or growth analytics at a consumer company.
- Systems trained on human ratings, such as ranking models or reward models, including pairwise-comparison data.
- Causal inference beyond A/B tests: difference-in-differences, uplift modeling, or similar.
- Natural language processing on large text archives.
- Time-series analysis, especially detecting regime changes or early-warning signals.
- Background in behavioral science, psychometrics or survey research, including designing rating scales.
- Experience working with editorial or creative teams.
- A graduate degree in statistics, economics, computer science, psychology or a related quantitative field, or equivalent experience.