API reference · Plumbline

Examples

A small Ruby client

The examples on this page use this client: plain Ruby, no gems. It sends your token, turns JSON both ways, follows pages and raises on an error. Set PLUMBLINE_TOKEN in your environment rather than in code.

require "json"
require "net/http"
require "uri"

class PlumblineAPI
  class Error < StandardError
    attr_reader :status, :body

    def initialize(status, body)
      @status = status
      @body = body
      super("#{status}: #{body.is_a?(Hash) ? body.dig("error", "detail") : body}")
    end
  end

  def initialize(token:, base_url: "https://plumbline.mikeporter.ai/api/v1")
    @token = token
    @base_url = base_url.chomp("/")
  end

  def get(path, params = {}) = request(Net::HTTP::Get, path, params: params)
  def post(path, body = {}) = request(Net::HTTP::Post, path, body: body)

  # Yields every result on every page, following _links.next.
  def paginate(path, params = {})
    return enum_for(:paginate, path, params) unless block_given?

    page = get(path, params)
    loop do
      page["results"].each { |result| yield result }
      next_url = page.dig("_links", "next", "href") or break
      page = request(Net::HTTP::Get, next_url)
    end
  end

  # PUTs a file to a signed upload URL. The URL is the credential, so no token is sent.
  def put_file(upload_url, path)
    uri = URI(upload_url)
    req = Net::HTTP::Put.new(uri)
    req.body = File.binread(path)
    perform(uri, req)
  end

  private

  def request(verb, path, params: {}, body: nil)
    uri = URI(path.start_with?("http") ? path : "#{@base_url}/#{path}")
    uri.query = URI.encode_www_form(params) if params.any?
    req = verb.new(uri)
    req["Authorization"] = "Token #{@token}"
    if body
      req["Content-Type"] = "application/json"
      req.body = JSON.generate(body)
    end
    perform(uri, req)
  end

  def perform(uri, req)
    res = Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https") { |http| http.request(req) }
    data = res.content_type == "application/json" ? JSON.parse(res.body) : res.body
    raise Error.new(res.code.to_i, data) unless res.is_a?(Net::HTTPSuccess)

    data
  end
end

api = PlumblineAPI.new(token: ENV.fetch("PLUMBLINE_TOKEN"))

4 endpoints

Datasets

A dataset is the file of items you want judged. For preference work each row is one prompt with two answers: item_id, prompt, response_a, response_b, and optionally category and gold_answer (rows with a gold answer become hidden test questions that check raters are paying attention). Create an empty dataset, ask for an upload URL, PUT the CSV or JSONL file to it, then check back until the dataset is READY. Plumbline reports rows it had to reject and the categories it found.

GET/api/v1/data-collection/datasets

Same as Prolific

List the workspace's datasets

Parameters for GET /data-collection/datasets
Parameter	In	Type	About
`workspace_id`	query	string	Defaults to the token's workspace.
`page`	query	integer
`per_page`	query	integer

GET /api/v1/data-collection/datasets

200 The datasets

{
  "results": [
    {
      "id": "ds_01J9X4T7QWB3V8K2M5N6P7Q8RS",
      "name": "string",
      "workspace_id": "string",
      "schema": "string",
      "status": "UNINITIALISED",
      "datapoint_count": 0,
      "validation": {
        "rows_rejected": 0,
        "errors": [
          {
            "row": null,
            "error": null
          }
        ]
      },
      "categories": {
        "key": 0
      },
      "created_at": "2026-10-08T12:00:00Z"
    }
  ],
  "meta": {
    "count": 0
  },
  "_links": {
    "self": {
      "href": "string",
      "title": "string"
    },
    "next": {
      "href": "string",
      "title": "string"
    },
    "previous": {
      "href": "string",
      "title": "string"
    },
    "last": {
      "href": "string",
      "title": "string"
    }
  }
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

api.paginate("data-collection/datasets").map { |dataset| [dataset["name"], dataset["status"]] }

POST/api/v1/data-collection/datasets

Prolific + Plumbline fields

Create an empty dataset in a workspace

POST /api/v1/data-collection/datasets

{
  "name": "helpfulness-pairs-oct",
  "workspace_id": "ws_01J9X3Q8ZK7H2D4M6N8P0R2T4V",
  "schema": "pairwise_preference@v1"
}

201 The new dataset

{
  "id": "ds_01J9X4T7QWB3V8K2M5N6P7Q8RS",
  "name": "helpfulness-pairs-oct",
  "workspace_id": "ws_01J9X3Q8ZK7H2D4M6N8P0R2T4V",
  "status": "UNINITIALISED"
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

ds = api.post("data-collection/datasets", name: "helpfulness-pairs-oct", schema: "pairwise_preference@v1")
ds["status"] # => "UNINITIALISED"

GET/api/v1/data-collection/datasets/{id}

Prolific + Plumbline fields

Get a dataset; poll until READY

Plumbline adds row validation and detected categories.

Parameters for GET /data-collection/datasets/{id}
Parameter	In	Type	About
`id` required	path	string

GET /api/v1/data-collection/datasets/{id}

200 The dataset

{
  "id": "ds_01J9X4T7QWB3V8K2M5N6P7Q8RS",
  "name": "helpfulness-pairs-oct",
  "status": "READY",
  "datapoint_count": 2000,
  "validation": {
    "rows_rejected": 3,
    "errors": [
      {
        "row": 418,
        "error": "response_b is empty"
      }
    ]
  },
  "categories": {
    "chat": 1210,
    "coding": 402,
    "reasoning": 388
  }
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

loop do
  ds = api.get("data-collection/datasets/#{ds["id"]}")
  break unless ds["status"] == "PROCESSING"

  sleep 2
end
ds["status"]     # => "READY" (or "ERROR": the reason is in ds["validation"])
ds["categories"] # => {"chat" => 1210, "coding" => 402, ...}

GET/api/v1/data-collection/datasets/{id}/upload-url/{filename}

Same as Prolific

Get a signed URL to PUT a CSV or JSONL file to

The URL is valid for one hour. PUT the raw file body to it; no checksum headers are needed. Pairwise schema columns: item_id, prompt, response_a, response_b, optional category, optional gold_answer. Limits in v1: 50 MB and 100,000 rows per file.

Parameters for GET /data-collection/datasets/{id}/upload-url/{filename}
Parameter	In	Type	About
`id` required	path	string
`filename` required	path	string

GET /api/v1/data-collection/datasets/{id}/upload-url/{filename}

200 The signed upload URL

{
  "upload_url": "https://api.plumbline.example/…",
  "expires_at": "2026-10-08T12:00:00Z",
  "http_method": "PUT",
  "dataset_id": "string"
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

upload = api.get("data-collection/datasets/#{ds["id"]}/upload-url/pairs.jsonl")
upload["upload_url"] # PUT the file here within the hour (see Uploads)

6 endpoints

Task design

A batch is the task raters see: what to read, what to decide and how. Point it at a dataset, add Prolific-style instructions if you have them, or switch on Plumbline's own pairwise question with the plumbline_task block (ties allowed or not, rubric sliders, a reason box, and answer order swapped for half the raters so position does not sway the result). Setting the batch up freezes it and builds one task per item.

GET/api/v1/data-collection/batches

Same as Prolific

List the workspace's batches

Parameters for GET /data-collection/batches
Parameter	In	Type	About
`workspace_id`	query	string	Defaults to the token's workspace.
`page`	query	integer
`per_page`	query	integer

GET /api/v1/data-collection/batches

200 The batches

{
  "results": [
    {
      "id": "bt_01J9X4T7QWB3V8K2M5N6P7Q8RS",
      "name": "string",
      "workspace_id": "string",
      "dataset_id": "string",
      "status": "UNINITIALISED",
      "task_details": {
        "task_name": "string",
        "task_introduction": "string",
        "task_steps": "string"
      },
      "plumbline_task": {
        "type": "pairwise_preference",
        "allow_tie": true,
        "rubric": [
          "string"
        ],
        "rationale": "optional",
        "counterbalance_order": true
      },
      "total_task_count": 0,
      "created_at": "2026-10-08T12:00:00Z"
    }
  ],
  "meta": {
    "count": 0
  },
  "_links": {
    "self": {
      "href": "string",
      "title": "string"
    },
    "next": {
      "href": "string",
      "title": "string"
    },
    "previous": {
      "href": "string",
      "title": "string"
    },
    "last": {
      "href": "string",
      "title": "string"
    }
  }
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

api.paginate("data-collection/batches").map { |batch| [batch["name"], batch["status"]] }

POST/api/v1/data-collection/batches

Prolific + Plumbline fields

Create the batch that holds the task

A Prolific batch body works unchanged; the plumbline_task block switches on the native pairwise question.

POST /api/v1/data-collection/batches

{
  "name": "Helpfulness pairs, October",
  "workspace_id": "ws_01J9X3Q8ZK7H2D4M6N8P0R2T4V",
  "dataset_id": "ds_01J9X4T7QWB3V8K2M5N6P7Q8RS",
  "task_details": {
    "task_name": "Which answer is better?",
    "task_introduction": "<p>Read the prompt and both answers.</p>",
    "task_steps": "<ol><li>Pick A, B or about the same</li></ol>"
  },
  "plumbline_task": {
    "type": "pairwise_preference",
    "allow_tie": true,
    "rubric": [
      "helpful",
      "accurate",
      "safe"
    ],
    "rationale": "optional",
    "counterbalance_order": true
  }
}

201 The new batch

{
  "id": "bt_01J9X5B2MRD7F9H1K3M5P7R9TV",
  "status": "UNINITIALISED",
  "dataset_id": "ds_01J9X4T7QWB3V8K2M5N6P7Q8RS"
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

bt = api.post("data-collection/batches",
  name: "Helpfulness pairs, October",
  dataset_id: ds["id"],
  task_details: { task_name: "Which answer is better?",
                  task_introduction: "<p>Read the prompt and both answers.</p>" },
  plumbline_task: { type: "pairwise_preference", allow_tie: true,
                    rubric: ["helpful", "accurate"], rationale: "optional" })

GET/api/v1/data-collection/batches/{id}

Same as Prolific

Get a batch; poll after setup until READY

Parameters for GET /data-collection/batches/{id}
Parameter	In	Type	About
`id` required	path	string

GET /api/v1/data-collection/batches/{id}

200 The batch

{
  "id": "bt_01J9X4T7QWB3V8K2M5N6P7Q8RS",
  "name": "string",
  "workspace_id": "string",
  "dataset_id": "string",
  "status": "UNINITIALISED",
  "task_details": {
    "task_name": "string",
    "task_introduction": "string",
    "task_steps": "string"
  },
  "plumbline_task": {
    "type": "pairwise_preference",
    "allow_tie": true,
    "rubric": [
      "string"
    ],
    "rationale": "optional",
    "counterbalance_order": true
  },
  "total_task_count": 0,
  "created_at": "2026-10-08T12:00:00Z"
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

api.get("data-collection/batches/#{bt["id"]}").values_at("status", "total_task_count")

GET/api/v1/data-collection/batches/{batch_id}/instructions

Same as Prolific

List a batch's instructions

Parameters for GET /data-collection/batches/{batch_id}/instructions
Parameter	In	Type	About
`batch_id` required	path	string

GET /api/v1/data-collection/batches/{batch_id}/instructions

200 The instructions, in order

{
  "instructions": [
    {
      "id": "in_01J9X4T7QWB3V8K2M5N6P7Q8RS",
      "type": "multiple_choice",
      "description": "string",
      "options": [
        "string"
      ],
      "position": 0
    }
  ]
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

api.get("data-collection/batches/#{bt["id"]}/instructions")["instructions"].map { |i| i["type"] }

POST/api/v1/data-collection/batches/{batch_id}/instructions

Prolific + one new type

Add instructions to a batch

Prolific's instruction types (multiple_choice, free_text, multiple_choice_with_free_text, file_upload) are accepted unchanged. Plumbline adds pairwise_preference.

Parameters for POST /data-collection/batches/{batch_id}/instructions
Parameter	In	Type	About
`batch_id` required	path	string

POST /api/v1/data-collection/batches/{batch_id}/instructions

{
  "instructions": [
    {
      "type": "pairwise_preference",
      "description": "Which response is more helpful and accurate?",
      "options": [
        "response_a",
        "response_b",
        "tie"
      ]
    },
    {
      "type": "free_text",
      "description": "Briefly say why."
    }
  ]
}

201 The created instructions

{
  "instructions": [
    {
      "id": "in_01J9X4T7QWB3V8K2M5N6P7Q8RS",
      "type": "multiple_choice",
      "description": "string",
      "options": [
        "string"
      ],
      "position": 0
    }
  ]
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

api.post("data-collection/batches/#{bt["id"]}/instructions", instructions: [
  { type: "multiple_choice", description: "Is the prompt clear?", options: ["Yes", "No"] }
])

POST/api/v1/data-collection/batches/{id}/setup

Same as Prolific

Freeze instructions and build tasks from the dataset

Poll the batch until READY. Builds one task per item plus hidden test-question tasks.

Parameters for POST /data-collection/batches/{id}/setup
Parameter	In	Type	About
`id` required	path	string

POST /api/v1/data-collection/batches/{id}/setup

202 Setup started; the batch is PROCESSING

{
  "id": "bt_01J9X4T7QWB3V8K2M5N6P7Q8RS",
  "name": "string",
  "workspace_id": "string",
  "dataset_id": "string",
  "status": "UNINITIALISED",
  "task_details": {
    "task_name": "string",
    "task_introduction": "string",
    "task_steps": "string"
  },
  "plumbline_task": {
    "type": "pairwise_preference",
    "allow_tie": true,
    "rubric": [
      "string"
    ],
    "rationale": "optional",
    "counterbalance_order": true
  },
  "total_task_count": 0,
  "created_at": "2026-10-08T12:00:00Z"
}

200 Setup finished at once; the batch is READY

{
  "id": "bt_01J9X4T7QWB3V8K2M5N6P7Q8RS",
  "name": "string",
  "workspace_id": "string",
  "dataset_id": "string",
  "status": "UNINITIALISED",
  "task_details": {
    "task_name": "string",
    "task_introduction": "string",
    "task_steps": "string"
  },
  "plumbline_task": {
    "type": "pairwise_preference",
    "allow_tie": true,
    "rubric": [
      "string"
    ],
    "rationale": "optional",
    "counterbalance_order": true
  },
  "total_task_count": 0,
  "created_at": "2026-10-08T12:00:00Z"
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

bt = api.post("data-collection/batches/#{bt["id"]}/setup")
until %w[READY ERROR].include?(bt["status"])
  sleep 2
  bt = api.get("data-collection/batches/#{bt["id"]}")
end
bt["total_task_count"] # one task per item

5 endpoints

Studies

A study asks the panel to work through a batch: how many people judge each item, who may take part, quotas, how many test questions to mix in, and whether the AI judge scores items alongside the people. Creating a study gives you a quote; publishing it opens it to raters. You can pause, resume and stop it at any time, and its progress and timeline show how it is filling.

GET/api/v1/studies

Same as Prolific

List the workspace's studies

Parameters for GET /studies
Parameter	In	Type	About
`state`	query	UNPUBLISHED \| SCHEDULED \| PUBLISHING \| ACTIVE \| PAUSED \| AWAITING REVIEW \| COMPLETED	Filter by status.
`page`	query	integer
`per_page`	query	integer

GET /api/v1/studies

200 The studies

{
  "results": [
    {
      "id": "st_01J9X4T7QWB3V8K2M5N6P7Q8RS",
      "name": "string",
      "internal_name": "string",
      "description": "string",
      "status": "UNPUBLISHED",
      "data_collection_method": "string",
      "data_collection_id": "string",
      "total_available_places": 0,
      "places_taken": 0,
      "estimated_completion_time": 0,
      "maximum_allowed_time": 0,
      "reward": 0,
      "currency": "string",
      "filters": [
        {
          "filter_id": "current-country-of-residence",
          "selected_values": [
            null
          ],
          "selected_range": {
            "lower": null,
            "upper": null
          }
        }
      ],
      "published_at": "2026-10-08T12:00:00Z",
      "date_created": "2026-10-08T12:00:00Z",
      "plumbline_review": {
        "raters_per_item": 3,
        "max_raters_per_item": 9,
        "stop_at_lead": 2,
        "items_per_place": 40,
        "panel_tier": "any",
        "quotas": [
          {
            "attribute": "age_band",
            "split": null
          }
        ],
        "gold_rate": 0.05,
        "dpo_rule": {
          "min_majority_share": 0.6,
          "allow_tie_majority": false
        },
        "ai_judge": {
          "include": true
        },
        "sensitive": false
      },
      "plumbline_quote": {
        "participant_pay_cents": 0,
        "platform_fee_cents": 0,
        "total_cents": 0,
        "fee_rate": 0.0,
        "fee_is_placeholder": true,
        "currency": "USD",
        "accepted_at": "2026-10-08T12:00:00Z",
        "raters_per_item": {
          "minimum": 0,
          "maximum": 0
        },
        "places_needed": {
          "at_minimum": 0,
          "at_maximum": 0
        }
      },
      "plumbline_progress": {
        "items_total": 0,
        "items_complete": 0,
        "judgements_collected": 0,
        "flagged_for_review": 0,
        "raters_removed_for_quality": 0,
        "agreement": 0.0,
        "tie_rate": 0.0,
        "judge_spend_cents": 0,
        "judge_budget_cents": 0
      },
      "plumbline_warnings": [
        {
          "code": "filter_not_applied",
          "field": "string",
          "message": "string"
        }
      ]
    }
  ],
  "meta": {
    "count": 0
  },
  "_links": {
    "self": {
      "href": "string",
      "title": "string"
    },
    "next": {
      "href": "string",
      "title": "string"
    },
    "previous": {
      "href": "string",
      "title": "string"
    },
    "last": {
      "href": "string",
      "title": "string"
    }
  }
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

api.paginate("studies/", state: "ACTIVE").map { |study| study["name"] }

POST/api/v1/studies

Prolific + Plumbline fields

Create the panel review request

Prolific fields are kept; filters use Prolific's filter_id shape, mapped to panel profile fields. The plumbline_review block sets panel and quality rules. The response carries plumbline_quote. external_study_url, completion_codes and prolific_id_option are accepted and echoed but not used, because tasks run inside Plumbline.

POST /api/v1/studies

{
  "name": "Helpfulness pairs, October",
  "description": "Compare two AI answers.",
  "data_collection_method": "AI_TASK_BUILDER_BATCH",
  "data_collection_id": "bt_01J9X5B2MRD7F9H1K3M5P7R9TV",
  "total_available_places": 400,
  "estimated_completion_time": 30,
  "reward": 1200,
  "filters": [
    {
      "filter_id": "current-country-of-residence",
      "selected_values": [
        "US"
      ]
    },
    {
      "filter_id": "fluent-languages",
      "selected_values": [
        "en"
      ]
    }
  ],
  "plumbline_review": {
    "raters_per_item": 5,
    "panel_tier": "certified",
    "quotas": [
      {
        "attribute": "age_band",
        "split": {
          "18-34": 0.3,
          "35-54": 0.35,
          "55+": 0.35
        }
      }
    ],
    "gold_rate": 0.05,
    "ai_judge": {
      "include": true
    }
  }
}

201 The new study, UNPUBLISHED, with its quote

{
  "id": "st_01J9X6C0ZPE8G0J2L4N6Q8S0UW",
  "status": "UNPUBLISHED",
  "data_collection_id": "bt_01J9X5B2MRD7F9H1K3M5P7R9TV",
  "plumbline_quote": {
    "participant_pay_cents": 480000,
    "platform_fee_cents": 158400,
    "fee_rate": 0.33,
    "fee_is_placeholder": true,
    "currency": "USD"
  }
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

st = api.post("studies/",
  name: "Helpfulness pairs, October",
  description: "Compare two AI answers.",
  data_collection_method: "AI_TASK_BUILDER_BATCH",
  data_collection_id: bt["id"],
  total_available_places: 400,
  estimated_completion_time: 30,
  reward: 1200,
  filters: [{ filter_id: "fluent-languages", selected_values: ["en"] }],
  plumbline_review: { raters_per_item: 5, gold_rate: 0.05, ai_judge: { include: false } })
st["plumbline_quote"] # => {"participant_pay_cents" => 480000, "platform_fee_cents" => ..., ...}

GET/api/v1/studies/{id}

Prolific + Plumbline fields

Get a study

Status uses Prolific's values: UNPUBLISHED, SCHEDULED, PUBLISHING, ACTIVE, PAUSED, AWAITING REVIEW, COMPLETED. plumbline_progress adds item-level progress.

Parameters for GET /studies/{id}
Parameter	In	Type	About
`id` required	path	string

GET /api/v1/studies/{id}

200 The study

{
  "id": "st_01J9X6C0ZPE8G0J2L4N6Q8S0UW",
  "status": "ACTIVE",
  "total_available_places": 400,
  "places_taken": 233,
  "plumbline_progress": {
    "items_total": 2000,
    "items_complete": 1164,
    "judgements_collected": 6420,
    "flagged_for_review": 41,
    "raters_removed_for_quality": 3
  }
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

st = api.get("studies/#{st["id"]}")
st["status"]                                 # => "ACTIVE"
st["plumbline_progress"]["items_complete"]

POST/api/v1/studies/{id}/transition

Same as Prolific

Publish, pause, resume or stop a study

Prolific's actions and transitions: PUBLISH (UNPUBLISHED to PUBLISHING to ACTIVE), PAUSE (ACTIVE to PAUSED), START (PAUSED to ACTIVE), STOP (to AWAITING REVIEW, then COMPLETED). PUBLISH needs a READY batch, an accepted quote and, if the study turned the AI judge on in plumbline_review.ai_judge, the workspace's consent. A study that never mentioned the judge publishes without consent; its items wait for consent (see plumbline_warnings). A transition that is not allowed answers 400 with the Error body.

Parameters for POST /studies/{id}/transition
Parameter	In	Type	About
`id` required	path	string

POST /api/v1/studies/{id}/transition

{
  "action": "PUBLISH"
}

200 The study after the transition

{
  "id": "st_01J9X6C0ZPE8G0J2L4N6Q8S0UW",
  "status": "ACTIVE"
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

st = api.post("studies/#{st["id"]}/transition/", action: "PUBLISH")
st["status"] # => "ACTIVE"; later "PAUSE", "START" (resume) or "STOP"

GET/api/v1/studies/{id}/timeline

Plumbline only

Hourly progress, forecast to finish and numbered events (Plumbline only)

Feeds the console's charts and event list. The series starts at publishing (t 0), has one point at the end of each clock hour since, and ends with the study as it stands now (or when it finished). The forecast divides the judgements still needed by the recent pace (weighted over the last six hours); its range uses the 20th and 80th percentile of hourly pace. Events are numbered per study.

Parameters for GET /studies/{id}/timeline
Parameter	In	Type	About
`id` required	path	string
`interval`	query	hour

GET /api/v1/studies/{id}/timeline

200 The timeline

{
  "interval": "hour",
  "series": [
    {
      "t": 0,
      "items_complete": 0,
      "judgements": 0,
      "agreement": null
    },
    {
      "t": 30,
      "items_complete": 1164,
      "judgements": 5820,
      "agreement": 0.71
    }
  ],
  "forecast": {
    "finish_in_hours": 22,
    "range_hours": [
      18,
      28
    ]
  },
  "events": [
    {
      "number": 1,
      "t": 0,
      "kind": "customer_action",
      "title": "Published",
      "detail": {
        "action": "publish",
        "actor": "api_token:plb_abc123"
      },
      "cost_cents": null
    },
    {
      "number": 2,
      "t": 8,
      "kind": "detection",
      "title": "Burst of fast submissions",
      "detail": {
        "detector": "fast_burst",
        "summary": "6 of 41 answers in the last hour came in under a third of the median time and were set aside."
      },
      "cost_cents": null
    }
  ]
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

timeline = api.get("studies/#{st["id"]}/timeline", interval: "hour")
timeline["forecast"] # => {"finish_in_hours" => 22, "range_hours" => [18, 28]}

2 endpoints

Results

Responses are the raw answers, one per rater per question, in Prolific's shape. Results are what you actually train on: one record per item with the votes counted, how far raters agreed, ties kept as ties, test-question scores, flags, splits by rater group, the AI judge's view kept apart from the human label, and whether the item is clear enough for a DPO pair. Ask for format=dpo_jsonl or format=reward_jsonl to get a ready-made training file instead. Both use the standard prompt, chosen, rejected fields that common DPO and reward-model trainers read. The DPO file holds only items with a clear winner. The reward file holds every finished item: ties and tied counts have chosen and rejected set to null, so keep the rows where chosen is set for a plain pair file, or use label, soft_label and weight for a tie-aware loss.

GET/api/v1/data-collection/batches/{batch_id}/responses

Same as Prolific (+ optional profile)

Raw responses, one per rater per instruction, in Prolific's shape

Parameters for GET /data-collection/batches/{batch_id}/responses
Parameter	In	Type	About
`batch_id` required	path	string
`include`	query	rater_profile	`rater_profile` attaches the pseudonymous rater profile to each response.
`page`	query	integer
`per_page`	query	integer

GET /api/v1/data-collection/batches/{batch_id}/responses

200 The responses

{
  "results": [
    {
      "id": "rs_01J9X4T7QWB3V8K2M5N6P7Q8RS",
      "batch_id": "string",
      "participant_id": "rtr_01J9X4T7QWB3V8K2M5N6P7Q8RS",
      "submission_id": "string",
      "task_id": "string",
      "response": {
        "instruction_id": "string",
        "type": "multiple_choice",
        "answer": [
          {
            "value": null
          }
        ]
      },
      "rater_profile": {
        "age_band": "string",
        "country": "string",
        "languages": [
          "string"
        ],
        "education": "string",
        "occupation": "string",
        "tier": "string",
        "tasks_completed": 0,
        "gold_accuracy": 0.0,
        "level": "joined",
        "level_since": "2026-10-08T12:00:00Z",
        "identity_checked_at": "string",
        "evidence": {
          "key": "document"
        }
      },
      "created_at": "2026-10-08T12:00:00Z"
    }
  ],
  "meta": {
    "count": 0
  },
  "_links": {
    "self": {
      "href": "string",
      "title": "string"
    },
    "next": {
      "href": "string",
      "title": "string"
    },
    "previous": {
      "href": "string",
      "title": "string"
    },
    "last": {
      "href": "string",
      "title": "string"
    }
  }
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

api.paginate("data-collection/batches/#{bt["id"]}/responses", include: "rater_profile").each do |response|
  puts [response["participant_id"], response.dig("response", "answer", 0, "value"),
        response.dig("rater_profile", "age_band")].join("\t")
end

GET/api/v1/data-collection/batches/{batch_id}/results

Plumbline only

The finished product, one record per item (Plumbline only)

Votes, agreement, ties, quality flags, segments, DPO eligibility and the ai_judge block for each item. ?format=dpo_jsonl or ?format=reward_jsonl returns a training file (one JSON object per line) instead. Both carry the standard prompt, chosen, rejected fields. dpo_jsonl lists only items with a clear winner; reward_jsonl lists every finished item, with chosen and rejected null on ties and tied counts, plus response_a, response_b, label, soft_label, weight and, when the task has one, rubric.

Parameters for GET /data-collection/batches/{batch_id}/results
Parameter	In	Type
`batch_id` required	path	string
`format`	query	json \| dpo_jsonl \| reward_jsonl
`page`	query	integer
`per_page`	query	integer

GET /api/v1/data-collection/batches/{batch_id}/results

200 The results, or a JSONL training file

{
  "results": [
    {
      "item_id": "string",
      "task_id": "string",
      "category": "string",
      "status": "pending",
      "simulated": true,
      "human": {
        "raters": 0,
        "votes": {
          "response_a": 0.0,
          "response_b": 0.0,
          "tie": 0.0
        },
        "majority": "response_a",
        "agreement": 0.0,
        "soft_label": {
          "response_a": 0.0,
          "response_b": 0.0,
          "tie": 0.0
        },
        "rubric_means": {
          "key": {
            "response_a": null,
            "response_b": null
          }
        },
        "order_shown": {
          "a_first": 0,
          "b_first": 0
        },
        "raters_added": 0,
        "contested": true,
        "instructions": [
          {
            "instruction_id": null,
            "type": "multiple_choice",
            "responses": null,
            "counts": null,
            "distribution": null,
            "values": null
          }
        ]
      },
      "segments": {
        "key": {
          "key": "string"
        }
      },
      "quality": {
        "gold_pass_rate": 0.0,
        "flags": [
          "string"
        ]
      },
      "ai_judge": {
        "status": "scored",
        "judge_version": "plumbline-judge-2026.10",
        "rubric": "pairwise-helpful@v1",
        "p": {
          "response_a": 0.0,
          "response_b": 0.0,
          "tie": 0.0
        },
        "confidence": 0.0,
        "calibrated": true,
        "order_check": "consistent",
        "agrees_with_majority": true
      },
      "dpo": {
        "eligible": true,
        "chosen": "string",
        "rejected": "string",
        "rule": "majority >= 0.6 and majority != tie"
      },
      "judgements": [
        {
          "rater_id": "string",
          "answer": "response_a",
          "answers": {
            "key": null
          },
          "order": "a_first",
          "rationale": "string",
          "seconds": 0.0,
          "rubric": {},
          "rater_profile": {
            "age_band": null,
            "country": null,
            "languages": null,
            "education": null,
            "occupation": null,
            "tier": null,
            "tasks_completed": null,
            "gold_accuracy": null,
            "level": "joined",
            "level_since": null,
            "identity_checked_at": null,
            "evidence": null
          }
        }
      ]
    }
  ],
  "meta": {
    "count": 0,
    "complete": 0
  },
  "summary": {
    "agreement_mean": 0.0,
    "tie_rate": 0.0,
    "dpo_eligible": 0,
    "contested": 0
  },
  "_links": {
    "self": {
      "href": "string",
      "title": "string"
    },
    "next": {
      "href": "string",
      "title": "string"
    },
    "previous": {
      "href": "string",
      "title": "string"
    },
    "last": {
      "href": "string",
      "title": "string"
    }
  }
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

api.paginate("data-collection/batches/#{bt["id"]}/results").each do |record|
  puts [record["item_id"], record["status"], record.dig("human", "majority"),
        record.dig("human", "agreement")].join("\t")
end

# Training files: one JSON object per line.
File.write("dpo.jsonl", api.get("data-collection/batches/#{bt["id"]}/results", format: "dpo_jsonl"))
File.write("reward.jsonl", api.get("data-collection/batches/#{bt["id"]}/results", format: "reward_jsonl"))

1 endpoint

Uploads

The upload URL from a dataset points here. It is the one call that takes no token: the signed URL is the credential, valid for one hour and only for that dataset. Send the raw file as the body; no checksum headers are needed.

PUT/uploads/{token}

Plumbline only (the target of upload_url)

Upload the raw dataset file to a signed URL

The target of upload_url. The signed token in the path is the credential, so no Authorization header is needed. The body is the raw CSV or JSONL file. Ingest starts at once; poll the dataset until READY.

No token: the signed URL is the credential.

Parameters for PUT /uploads/{token}
Parameter	In	Type	About
`token` required	path	string

PUT /uploads/{token}

(the raw CSV or JSONL file)

200 Received; the dataset is PROCESSING

{
  "id": "ds_01J9X4T7QWB3V8K2M5N6P7Q8RS",
  "name": "string",
  "workspace_id": "string",
  "schema": "string",
  "status": "UNINITIALISED",
  "datapoint_count": 0,
  "validation": {
    "rows_rejected": 0,
    "errors": [
      {
        "row": 0,
        "error": "string"
      }
    ]
  },
  "categories": {
    "key": 0
  },
  "created_at": "2026-10-08T12:00:00Z"
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

api.put_file(upload["upload_url"], "pairs.jsonl") # => the dataset, now "PROCESSING"

1 endpoint

Keys

Keys are made and revoked in the console, never through the API. This one call lets a script ask what its own key is: its mode, scopes, expiry and rate limits, so a pipeline can fail early with a clear message instead of a 403 halfway through.

GET/api/v1/plumbline/key

Plumbline only

The key that made this request (Plumbline only)

What this key is and may do: its name, mode (test or live), workspace, scopes, expiry and rate limits, so a script can check itself before it starts. Any key may call it. Keys are made and revoked in the console only; no key can make another.

GET /api/v1/plumbline/key

200 The key

{
  "name": "Training pipeline",
  "prefix": "plb_live_7Gx2kQ",
  "mode": "live",
  "workspace_id": "ws_01J9X3Q8ZK7H2D4M6N8P0R2T4V",
  "scopes": [
    "datasets:read",
    "batches:read",
    "results:read",
    "studies:read"
  ],
  "expires_at": "2027-01-31T23:59:59Z",
  "created_at": "2026-11-07T14:02:11Z",
  "last_used_at": "2026-11-07T14:02:11Z",
  "rate_limits": {
    "requests_per_minute": 600,
    "workspace_requests_per_minute": 3000
  }
}

Errors Any error (400, 401, 403, 404, 409, 422, 429, 500, 501), in Prolific's error body. `detail` says what went wrong in words; for a refused key it starts with a reason a script can key on: `key_expired` (401, the key passed its expiry date), `key_revoked` (401, someone revoked it in the console). A 403 names the scope the key is missing. A 429 says which budget ran out (the key's or the workspace's) and carries `Retry-After`.

{
  "error": {
    "status": 401,
    "error_code": 140101,
    "title": "Unauthorized",
    "detail": "Send a valid API token in the Authorization header: Token <token>."
  }
}

key = api.get("plumbline/key")
key["mode"]   # => "test" or "live"
key["scopes"] # => ["datasets:read", ...]; a Full-access key lists all eight

Plumbline API

Start in the sandbox

Base URL and token

Coming from Prolific

Lists, errors and limits

A small Ruby client

Datasets

List the workspace's datasets

Create an empty dataset in a workspace

Get a dataset; poll until READY

Get a signed URL to PUT a CSV or JSONL file to

Task design

List the workspace's batches

Create the batch that holds the task

Get a batch; poll after setup until READY

List a batch's instructions

Add instructions to a batch

Freeze instructions and build tasks from the dataset

Studies

List the workspace's studies

Create the panel review request

Get a study

Publish, pause, resume or stop a study

Hourly progress, forecast to finish and numbered events (Plumbline only)

Results

Raw responses, one per rater per instruction, in Prolific's shape

The finished product, one record per item (Plumbline only)

Uploads

Upload the raw dataset file to a signed URL

Keys

The key that made this request (Plumbline only)

Start here

Start in the sandbox

Base URL and token

Coming from Prolific

Lists, errors and limits

A small Ruby client

Datasets

List the workspace's datasets

Create an empty dataset in a workspace

Get a dataset; poll until READY

Get a signed URL to PUT a CSV or JSONL file to

Task design

List the workspace's batches

Create the batch that holds the task

Get a batch; poll after setup until READY

List a batch's instructions

Add instructions to a batch

Freeze instructions and build tasks from the dataset

Studies

List the workspace's studies

Create the panel review request

Get a study

Publish, pause, resume or stop a study

Hourly progress, forecast to finish and numbered events (Plumbline only)

Results

Raw responses, one per rater per instruction, in Prolific's shape

The finished product, one record per item (Plumbline only)

Uploads

Upload the raw dataset file to a signed URL

Keys

The key that made this request (Plumbline only)