Building runners with the HEAT Runner SDK

The HEAT Runner SDK lets you build a Runner of your own: a container that executes one or more node templates. You write a processor for each template, test it on your machine against fixtures, run it against a real deployment, and ship it as an image. The SDK talks to HEAT only through the v2 runner routes, so it works the same from your laptop, from your own hosts and inside the cluster.

The development loop

pip install heat-runner-sdk
heat-runner init my-runner                                   # scaffold: processor, config, tests, fixtures, Dockerfile
cd my-runner && pip install -e ".[dev]" && pytest            # the scaffold's tests pass out of the box
heat-runner test my_runner.app fixtures/basic --dry-run      # what would the next task consume?
heat-runner test my_runner.app fixtures/basic                # run one task offline
heat-runner manifest generate && heat-runner manifest check  # manifest.heat matches the code
heat-runner dev my_runner.app                                # run against a real deployment
heat-runner package --image registry.example.org/my-runner:0.1.0

Each step works without the next one. You can write and test a processor with no HEAT deployment at all, and only connect when you want real data.

A processor

A processor implements one node template. Its configuration is a pydantic model, which also becomes the template’s configuration schema in the manifest.

from pydantic import BaseModel, Field
from heat_runner import Processor, TaskContext, ValidationError, processor
 
 
class Config(BaseModel):
    field_name: str = Field("value", description="Field to read from each input record.")
    scale: float = Field(1.0, description="Multiplier applied to every value.")
 
 
@processor("my-runner", config=Config, description="Scale a numeric field of every input record.")
class MyRunner(Processor[Config]):
    def validate(self, ctx: TaskContext[Config]) -> None:
        if ctx.config.scale == 0:
            raise ValidationError("scale must not be zero")
 
    def process(self, ctx: TaskContext[Config]) -> None:
        for batch in ctx.inputs.since_watermark():
            rows = [
                {"input": item.id, "value": item.read_json()[ctx.config.field_name] * ctx.config.scale}
                for item in batch
            ]
            ctx.outputs.write_json({"rows": rows})

Node template names and runner type names use lowercase latin letters, digits and single hyphens, and start with a letter: my-runner, imu-drift-v2. Underscores, capitals and other characters are rejected when the processor is registered, and again when the manifest is published.

The task context gives a processor its typed configuration, its inputs (latest(), all(), since_watermark(), and its own previous output), its outputs (write_json, write_text, write_bytes, write_file, write_stream), progress reporting, a per-task temporary directory and memory checks. A task ends when process returns: completed when it wrote an output, otherwise with complete_without_output() or suspend(). An exception fails the task, and the platform decides whether to retry it.

Incremental processing

Most HEAT data grows during a session, so the SDK makes incremental processing the default:

• Transparent. A node keeps a watermark: the id of the last input it fully consumed. Every task shows its watermark, how many inputs are pending and the batch limits. heat-runner inspect --node <id> shows the same for a live node, and heat-runner test --dry-run prints what the next task would consume.
• Encouraged. since_watermark() hands your processor batches of new inputs, oldest first. Each output you write moves the watermark with it, in one step, so there is no bookkeeping to get wrong.
• Bounded. A task consumes at most a fixed number of inputs and bytes. When it stops at the limit, the platform schedules a follow-up task for the rest, so a large backlog drains as a series of small tasks you can inspect one by one.

The watermark only moves forward, and only to an input of the task. Outputs carry an identifier the SDK derives from the task, so a task retried after a crash does not publish twice.

Testing offline

A fixture is a directory with a node’s context (its configuration and the incremental state) and its inputs. heat-runner test runs your processor against it with no network and writes outputs to fixtures/<name>/out/. It keeps the watermark in the fixture between runs, so you can replay incremental progress. It also reports peak memory against the limit the runner would get in the cluster. heat_runner.testing has helpers to build fixtures and run processors from your own pytest suite.

Debugging against a deployment

With a runner token that has the debug:read scope:

heat-runner inspect --node 812                  # watermark, pending inputs, batch limits, parents
heat-runner capture --node 812 fixtures/live    # copy the node's context and pending inputs
heat-runner test my_runner.app fixtures/live    # reproduce locally
heat-runner dev my_runner.app --once            # claim one real task with readable logs

inspect and capture are read-only. They never claim tasks or move watermarks.

Signing in

While you develop, use your normal HEAT account. heat-runner login signs you in with your username and password, keeps the session fresh, and uses it for every command, including publishing the manifest. Your account needs the Administrator or Developer role, and it can run tasks only for runner types with runtime set to external, so your machine never takes work from runners the platform deploys.

Runners that keep running on their own use a runner token (hrt_...) instead, scoped to their runner types. Runners the platform deploys receive one automatically, as a file that is rotated for them. For a runner you host yourself, issue one with heat-runner token create and store it with heat-runner login --runner-token. See runner tokens in the v2 API reference.

Registering and deploying

heat-runner manifest generate writes manifest.heat from your processors, and heat-runner manifest check fails when code and manifest disagree. heat-runner manifest publish registers the runner type and its node templates with an Administrator or Developer token. Every runner type declares a runtime:

A published node template can then be used in a Session template like any other. Keep changes to a published template additive: session templates already in use depend on its name and configuration.

Related

• HEAT v2 API: runners
• Introduction to building runners
• Runners overview

HEAT documentation team. Last updated 2 October 2026.