Harbor

Setup

Install Braintrust alongside Harbor. Requires harbor>=0.16.0 and Python 3.12 or later. Docker must be running for Harbor to execute tasks.

Install packages

pip install braintrust harbor

Set environment variables

.env

BRAINTRUST_API_KEY=<your-braintrust-api-key>
OPENAI_API_KEY=<your-model-provider-api-key>

Run evaluations

Pass --plugin braintrust to harbor run to enable the Braintrust plugin. The plugin is discovered automatically through Harbor’s braintrust entry point after installation.

The Harbor integration is activated through the --plugin braintrust flag in the harbor run CLI, not through auto_instrument(). No import or patching is required in your task code.

harbor run \
  --path task/ \
  --agent terminus-2 \
  --model openai/gpt-5-mini \
  --job-name my-eval \
  --plugin braintrust \
  --yes

By default, results are synced to a Braintrust project named Harbor. After the run completes, the plugin creates a sync manifest at jobs/my-eval/braintrust-sync.json.

Configure with environment variables

Set HARBOR_BRAINTRUST_PROJECT to override the default project name on every run:

.env

HARBOR_BRAINTRUST_PROJECT=my-project

Or pass --plugin-kwarg project_name=my-project directly:

harbor run \
  --path task/ \
  --agent terminus-2 \
  --model openai/gpt-5-mini \
  --job-name my-eval \
  --plugin braintrust \
  --plugin-kwarg project_name=my-project \
  --yes

Verifier output

In the default attachments=all mode, the score and verification spans include Harbor’s captured test-stdout.txt, optional test-stderr.txt, and ctrf.json output when present. To capture only structured CTRF output without raw logs, pass attachments=structured. To disable verifier and reward attachments entirely, pass attachments=none.

harbor run \
  --path task/ \
  --agent terminus-2 \
  --model openai/gpt-5-mini \
  --job-name my-eval \
  --plugin braintrust \
  --plugin-kwarg attachments=structured \
  --yes

Or set the environment variable:

.env

HARBOR_BRAINTRUST_ATTACHMENTS=structured

Raw verifier logs are redacted only by redact_patterns, so set it when the evaluation environment holds secrets.

Pass --plugin-kwarg 'redact_patterns=["regex"]' to harbor run, or set HARBOR_BRAINTRUST_REDACT_PATTERNS to a JSON array of regex patterns, to redact matching strings from raw log output.

Backfill a past job

To re-sync a completed Harbor job without re-running the evaluation, use backfill_job:

backfill.py

import asyncio
from pathlib import Path
from braintrust.integrations.harbor import backfill_job

asyncio.run(backfill_job(Path("jobs/my-eval")))  # Uses the same project as the initial sync

Pass project_name to override the project:

asyncio.run(backfill_job(Path("jobs/my-eval"), project_name="my-project"))

Backfill uses deterministic dataset and experiment identities, so it reconciles with existing Braintrust data instead of creating duplicates.

What Braintrust receives

After each Harbor evaluation run, Braintrust receives:

A managed dataset with the task cases used in the evaluation.
An experiment row for each trial, with the agent’s final output as the experiment output and the Harbor task’s reference answer as the expected output.
Scores derived from Harbor verifier rewards, mapped to Braintrust score fields, with verifier output files (test-stdout.txt, test-stderr.txt, and ctrf.json when present) attached to each score span in attachments=all mode (the default).
Agent traces (when trajectory tracing is enabled), showing the agent’s tool calls and model turns using Harbor’s ATIF format.

AI providers

SDKs

Developer tools

Runtime environments

Setup

Run evaluations

Configure with environment variables

Verifier output

Backfill a past job

What Braintrust receives

Resources

​Setup

​Run evaluations

​Configure with environment variables

​Verifier output

​Backfill a past job

​What Braintrust receives

​Resources

Setup

Run evaluations

Configure with environment variables

Verifier output

Backfill a past job

What Braintrust receives

Resources