API Reference

VAULT API Documentation

The VAULT REST API lets you submit access requests, register models, trigger benchmark runs, and retrieve results — all programmatically. Built on standard HTTP with JSON.

Base URL: https://animl.health/apiVersion: v1

Overview

All requests must be made over HTTPS. The API accepts JSON request bodies and returns JSON responses. Timestamps are ISO 8601 strings in UTC. All IDs are CUID strings.

Benchmark participants interact primarily with /api/models, /api/benchmark-runs, and the /api/leaderboard endpoints. Access request submission via /api/applications is public and does not require authentication.

Authentication

Authenticated endpoints require a Bearer API key in the Authorization header. API keys are issued after your access request is approved. Keys are prefixed vault_sk_live_ and scoped to specific operations.

Request Header

Authorization: Bearer vault_sk_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Keys are shown once at creation — store them securely. Use POST /api/api-keys to create new keys and DELETE /api/api-keys?id= to revoke them.

Access Requests

Submit a request for benchmark access. No authentication required.

Submit access request

POST/api/applications

Parameter	Type	Required	Description
firstName	string	required	Applicant first name
lastName	string	required	Applicant last name
email	string	required	Work email address
orgName	string	required	Organization name
participantType	enum	required	VENDOR \| ACADEMIC \| INTERNAL_QA \| OTHER
useCase	string	required	Description of intended use (20–2000 chars)

Example Request

curl -X POST https://animl.health/api/applications \
  -H "Content-Type: application/json" \
  -d '{
    "firstName": "Jane",
    "lastName": "Smith",
    "email": "jane@acmevetai.com",
    "orgName": "Acme Vet AI",
    "participantType": "VENDOR",
    "useCase": "We are evaluating our clinical summarization model for deployment in veterinary practices. We want an independent, third-party benchmark score."
  }'

Response 201

{
  "applicationId": "clxyz1234567890",
  "status": "pending",
  "reviewEta": "1-3 business days",
  "message": "Your access request has been submitted..."
}

Models

A Model represents an inference endpoint or containerized adapter you want to benchmark. Each model belongs to your organization. You can register multiple models.

List models

GET/api/models

Response

{
  "models": [
    {
      "id": "clxyz...",
      "name": "Acme Vet Summarizer v2",
      "endpointType": "REST_API",
      "status": "ACTIVE",
      "createdAt": "2025-04-01T10:00:00Z",
      "lastRunAt": "2025-04-08T14:22:00Z"
    }
  ],
  "pagination": { "page": 1, "limit": 20, "total": 1, "totalPages": 1 }
}

Register a model

POST/api/models

Parameter	Type	Required	Description
name	string	required	Display name (max 200 chars)
endpointType	enum	required	REST_API \| CONTAINER \| ADAPTER \| BATCH
endpointUrl	string	optional	HTTPS URL of your inference endpoint
authHeaderName	string	optional	Header name for authentication (e.g. X-API-Key)
authHeaderValue	string	optional	Header value — stored AES-256 encrypted at rest
description	string	optional	Optional description (max 2000 chars)

Endpoint Contract

# Your endpoint must accept POST requests with this body:
{
  "case_id": "opaque_ref_abc123",   // internal reference only
  "input": {
    "chief_complaint": "...",
    "history": "...",
    "physical_exam": "...",
    "diagnostics": "..."
  }
}

# And return:
{
  "summary": "Clinical summary text here..."
}

# Timeout: 30 seconds per case.
# VAULT calls your endpoint once per case, sequentially.

Get / update / delete a model

GET/api/models/:id

PATCH/api/models/:id

DELETE/api/models/:id

PATCH accepts any subset of the create fields. DELETE soft-deletes (sets status to INACTIVE).

Benchmark Runs

A Benchmark Run evaluates one of your models against a benchmark suite. Runs are queued and executed asynchronously. Poll GET /api/benchmark-runs/:id or use webhooks to be notified when complete.

Create a run

POST/api/benchmark-runs

Parameter	Type	Required	Description
modelId	string	required	ID of the model to evaluate
benchmarkSuiteId	string	required	ID of the benchmark suite to use
label	string	optional	Optional label for this run
notifyWebhook	string	optional	HTTPS URL to notify on completion
requestPublication	boolean	optional	Set true to request leaderboard publication

Example

curl -X POST https://animl.health/api/benchmark-runs \
  -H "Authorization: Bearer vault_sk_live_..." \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: my-unique-run-key-001" \
  -d '{
    "modelId": "clxyz...",
    "benchmarkSuiteId": "clsuite...",
    "label": "v2.1 release candidate",
    "notifyWebhook": "https://my-server.com/hooks/vault"
  }'

Response 201

{
  "runId": "clrun...",
  "status": "queued",
  "totalCases": 5000,
  "estimatedDuration": 10000,
  "message": "Benchmark run queued. Monitor at GET /api/benchmark-runs/clrun..."
}

Poll run status

GET/api/benchmark-runs/:id

Response (in-progress)

{
  "runId": "clrun...",
  "status": "RUNNING_INFERENCE",
  "progress": { "pct": 42.6, "completedCases": 2130, "totalCases": 5000 },
  "timing": { "queuedAt": "...", "startedAt": "...", "completedAt": null },
  "model": { "name": "Acme Vet Summarizer v2" },
  "suite": { "name": "Clinical Summarization", "version": "v1.3" },
  "metrics": null,
  "report": null
}

Response (complete)

{
  "runId": "clrun...",
  "status": "COMPLETE",
  "progress": { "pct": 100, "completedCases": 5000, "totalCases": 5000 },
  "timing": { "queuedAt": "...", "startedAt": "...", "completedAt": "..." },
  "metrics": {
    "composite_score":     0.847,
    "factual_accuracy":    0.891,
    "clinical_relevance":  0.852,
    "completeness":        0.823,
    "chronological_order": 0.841,
    "organization":        0.835,
    "median_latency_ms":   1240
  },
  "publication": { "status": "PRIVATE", "participantConsented": false },
  "report": { "reportId": "clrep...", "generatedAt": "..." }
}

Cancel a run

DELETE/api/benchmark-runs/:id

Only QUEUED runs can be cancelled.

Leaderboard

Public endpoint. No authentication required.

Get leaderboard

GET/api/leaderboard

Parameter	Type	Required	Description
track	string	optional	Filter by track slug (e.g. clinical-summarization)
suite_id	string	optional	Filter by specific suite ID
page	integer	optional	Page number (default: 1)
limit	integer	optional	Results per page, max 50 (default: 25)

API Keys

List keys

GET/api/api-keys

Create a key

POST/api/api-keys

Parameter	Type	Required	Description
name	string	required	Descriptive name for this key
scopes	string[]	required	benchmark:run · benchmark:read · leaderboard:read · model:write · model:read
expiresInDays	integer	optional	Key expiry in days (1–365). Omit for no expiry.

The raw key is returned once in the response. Store it immediately — it cannot be retrieved again.

Revoke a key

DELETE/api/api-keys?id=:id

Webhooks

Register an HTTPS endpoint to receive real-time notifications when benchmark events occur. VAULT signs all deliveries with HMAC-SHA256 using your signingSecret.

Register a webhook

POST/api/webhooks

Parameter	Type	Required	Description
url	string	required	HTTPS URL to deliver events to
events	string[]	required	run.queued · run.started · run.completed · run.failed · report.ready · publication.approved · publication.rejected

Webhook Payload (run.completed)

{
  "event": "run.completed",
  "runId": "clrun...",
  "status": "COMPLETE",
  "modelName": "Acme Vet Summarizer v2",
  "compositeScore": 0.847,
  "timestamp": "2025-04-08T15:22:00Z"
}

Signature Verification (Node.js)

const crypto = require('crypto');

function verifyWebhook(rawBody, signatureHeader, secret) {
  const expected = crypto
    .createHmac('sha256', secret)
    .update(rawBody)
    .digest('hex');
  return crypto.timingSafeEqual(
    Buffer.from(signatureHeader),
    Buffer.from(expected)
  );
}

System Status

Returns the current operational status of all VAULT service components. No authentication required. See the status page for the live dashboard.

GET/api/status

Response

{
  "status": "operational",
  "components": [
    { "id": "api",              "name": "API",              "status": "operational", "latencyMs": 4 },
    { "id": "database",         "name": "Database",         "status": "operational", "latencyMs": 12 },
    { "id": "leaderboard",      "name": "Leaderboard",      "status": "operational", "latencyMs": 8 },
    { "id": "benchmark_queue",  "name": "Benchmark Queue",  "status": "operational", "latencyMs": 6 }
  ],
  "recentActivity": { "activeRuns": 3, "publishedLeaderboardEntries": 23 },
  "meta": { "version": "1.0", "responseTimeMs": 38, "timestamp": "...", "operatedBy": "ANIML Health" }
}

Errors

All errors follow a standard shape. HTTP status codes are standard (4xx for client errors, 5xx for server errors).

Error Response

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Human-readable explanation",
    "details": [ ... ]  // only present for VALIDATION_ERROR
  }
}

Code	HTTP	Meaning
UNAUTHORIZED	401	Missing or invalid API key
FORBIDDEN	403	Valid key but insufficient scopes
NOT_FOUND	404	Resource not found
CONFLICT	409	Duplicate (e.g. email already registered)
VALIDATION_ERROR	400	Request body failed schema validation
INVALID_STATE	422	Operation not valid for current resource state
RATE_LIMITED	429	Too many requests — back off and retry
INTERNAL_ERROR	500	Unexpected server error

Questions?

Contact benchmark@animl.health or visit the Submission Guide for a step-by-step walkthrough.