Every score starts with a person.
Plumbline puts your model's answers in front of verified people and turns what they choose into numbers you can train on: who won, by how much, how sure people were, and where it was a tie.
GET /api/v1/data-collection/batches/{id}/results
Which reply explains compound interest more clearly?
From people to numbers in three steps.
You bring the prompts and answers. Plumbline brings the people, runs the checks and does the counting.
Post your dataset
Upload prompt and answer pairs, or any rating task, through the API. Set how many people judge each item and what they need to know.
Verified people judge
Each item goes to several raters with the order balanced. Hidden test questions catch careless work, and raters who miss them are removed.
Fetch scored data
Get votes, agreement, ties and quality flags for every item, or download preference and reward-model files ready for training.
A vote count is a start. A score tells you how much to trust it.
- VotesHow many people chose A, chose B, or called it a tie.
votes - AgreementThe share of raters behind the winning answer. 7 of 9 is 0.78.
agreement - Order balanceProof that each answer was shown first about equally often.
order_shown - Rater qualityHow the raters on this item did on hidden test questions, plus any flags.
quality - Ready to trainWhether the item clears your bar for a preference pair, and which answer is chosen.
dpo
{ "item_id": "item_0412", "status": "complete", "human": { "raters": 9, "votes": { "response_a": 7, "response_b": 2, "tie": 0 }, "majority": "response_a", "agreement": 0.78, "order_shown": { "a_first": 5, "b_first": 4 } }, "quality": { "gold_pass_rate": 0.96, "flags": [] }, "dpo": { "eligible": true, "chosen": "response_a", "rule": "majority >= 0.6 and majority != tie" } }
Example record. Add ?format=dpo_jsonl or ?format=reward_jsonl to get a training file instead.
Models can grade models. People decide what good means.
An automated judge is quick and cheap, and on easy items it is often right. On everyday questions it often isn't sure, and the hard cases are the ones your model learns most from.
Plumbline keeps people on every item, and uses its own AI judge only to spot rater mistakes.
How often an AI judge matched people when it was confident.
The share of everyday prompts where it was confident. The rest needed people.
Votes that went to whichever answer was shown second. People have biases too, so we balance the order.
From Plumbline's own pilot on public preference data, October 2026. Everyday prompts set; small samples, so read these as estimates.
Real raters, known well enough to trust and never by name.
- VerifiedEvery rater is a real, checked person from QuestionPro's research panels, not a bot or a farm.
- Profiled, privatelyAge band, region and field come with every judgement, so you can see where groups disagree. Names and contact details never leave the panel.
- Watched for qualityHidden test questions and speed checks run on every session, and each rater carries a track record.
- Paid fairly, in the openOur platform fee is published and stays below 40% of what raters earn.
Put people on your data.
Start in a sandbox with a simulated crowd, then switch to real raters when your study is ready.