Next data service and direct ingestion

The Next dashboard (ui/dashboard) does not read raw runner blobs per widget. It uses a single in-browser Heat data service (HeatDataService in heat-client) that ingests a structured JSON envelope, indexes channels by realm, and serves samples to widgets (timelines, maps, composable charts, KPI panels, and so on) through React hooks.

That envelope is called direct ingestion: data is already in the channel/realm shape the UI expects, without legacy per-widget JSON keys like { "Map": { ... } }.

Why this format exists

Legacy dashboards (ui/legacy) often received one fixed JSON object per widget type from the v1-era pipeline. Each component had its own contract (Map, Timeline, and similar). Adding a second map or a new chart meant new top-level keys and fragile coupling between runners and UI.

The v2 / Next model separates concerns:

Layer	Responsibility
Upstream processing	Runners or platform nodes produce analytics results
`$heat-dataservice`	A shared, versioned description of what data exists (channels, time series, KPIs, events)
Layout	A separate JSON document describing how to render (which component, which channel ids, titles, chart options)

Benefits:

Multiple widgets can bind the same channel (two maps, timeline + composable chart on one series).
Playback and seeking use a common timeMs axis across series, events, and ranges where applicable.
Realms let one dashboard compare runs (trainees, files, scenarios) without duplicating channel ids in layout.
Validation is centralized (C# on the platform path, TypeScript in the app) so integrators get predictable errors.

Platform and runner docs for producing payloads: dashboard-v2 upstream contract, system-tabular-to-dataservice, system-arbex-js.

End-to-end flow

A dashboard-v2 or heat-system-next-dimension node persists a JSON root (often including $heat-dataservice and optional dashboard_users).
The v2 API returns that payload plus layout for the session dimension.
HeatDataService.ingestSessionData detects $heat-dataservice, validates it, and loads channels into the DataStore (keyed by realm:channelId:name).
Each layout column lists channel ids (and widget-specific options). Components sample data at the current playback time.

What is a direct ingestion payload?

A direct ingestion payload is the object under the $heat-dataservice key on the session/dimension JSON root.

Minimal wrapper:


{
  "$heat-dataservice": {
    "version": "1.0",
    "groups": [{ "id": "metrics", "name": "Metrics" }],
    "realms": [
      {
        "name": "default",
        "channels": [
          {
            "id": "heart-rate",
            "name": "Heart rate",
            "groupId": "metrics",
            "shape": "series",
            "data": [
              { "timeMs": 0, "value": 72 },
              { "timeMs": 1000, "value": 75 }
            ]
          }
        ]
      }
    ]
  },
  "dashboard_users": []
}

The UI checks isHeatDirectIngestionData(root) (presence of $heat-dataservice). Types live in direct-ingestion-types.ts.

Direct ingestion contrasts with legacy automatic mapping inside heat-client, where older session shapes are converted via JMESPath-style rules in data-mapping-config.ts. New Next work should always target $heat-dataservice.

Realms

A realm is a namespace for channel data on one dashboard dimension.

Stored as realms[].name in the payload (for example default, pilot-a, file-metrics-csv).
The in-memory store keys samples as realm:channelId:name (see DataStore.getKey).
Layout includes realms: ["default", "pilot-a", ...] and per-widget configuration.defaultRealm / showRealmFilterDropdown.

Why realms exist: they answer “which run of data am I looking at?” without changing layout channel ids. The same layout can show heart-rate for trainee A or trainee B by switching realm. Typical uses:

One CSV upload per realm (realmPerFile on system-tabular-to-dataservice)
A column in tabular data (realmColumn)
Explicit multi-realm payloads from system-arbex-js heat.dataservice builder (spec.realm)

Important: id + name in realm A is not the same data as the same id + name in realm B. Widgets must pass the selected realm into hooks (useDataServiceData, ComposableChartDS, and so on).

Soft cap: platform normalization recommends at most 50 realms per payload. Prefer more channels or separate dimensions when cardinality is high.

Groups

Groups (groups[].id, groups[].name) are UI organisation for channel pickers and registry entries. Every channel must reference a valid groupId.

Groups do not partition data the way realms do. They are labels (for example ops, metrics, biometrics) so authors and operators can find channels in complex dashboards.

Channels

A channel is one logical dataset inside a realm: a single id, name, groupId, shape, and data.

Field	Role
`id`	Stable identifier used in layout `configuration.channels` and timeline `channelId`
`name`	Display name (also part of the internal store key)
`groupId`	Links to `groups[].id`
`shape`	Determines `data` structure and how widgets consume it
`data`	Payload for that shape (required, never null)
`metadata`	Optional hints (`units`, `channelType`, column definitions for KPI widgets, and so on)

Channel id must be unique within a realm. Layout references channels by id (and optional URI forms like realm:channelId:name in advanced composable chart bindings).

Channel shapes

`shape`	Meaning	Typical `data`	Widget examples
`series`	Time-varying samples	`[{ timeMs, value }]`	TimelineChart, ComposableChart area/scatter, playback
`timestamps`	Labelled instants	`[{ timeMs, annotation }]`	Timeline markers
`value`	Non-time-series or structured snapshot	`{ value: <any JSON> }`	StatsSummary, BarChart, arbitrary KPIs
`events`	Boolean events over time	`[{ timeMs, occurred }]`	Event lanes, ComposableChart events_lane
`ranges`	Intervals / phases	`[{ startTimeMs, endTimeMs, durationMs }]`	Range lanes, phase bars

Not every shape supports playback interpolation. value channels are usually static for the session. series drives the shared timeline clock.

Additional shapes (flightpath, mapdisplay, responseTime) exist for specialised widgets. The platform contract for system-utils nodes standardises on the five shapes above.

Layout vs data

Layout (layoutConfiguration on the dimension) is stored beside $heat-dataservice, not inside each channel.

Layout	Data (`$heat-dataservice`)
Which React component (`MapDisplay`, `ComposableChart`, …)	What numbers and structures exist
`configuration.channels: ["heart-rate"]`	Channel `id: "heart-rate"` with `shape` and `data`
Titles, tooltips, chart heights, `composableChartItem` slices	Optional `metadata` on channels

This split lets operators change presentation without re-running analytics, and re-run analytics without rewriting layout (as long as channel ids stay stable).

See Next dashboard components for per-widget layout fields.

How the frontend ingests payload

Simplified sequence inside HeatDataService:

Load session/dimension JSON from the v2 API.
If $heat-dataservice is present, run validateDirectIngestionPayload.
convertDirectToChannelData flattens realms[].channels[] into ChannelData[] with realm set on each row.
DataStore.ingest indexes by realm:channelId:name and extracts timestamps for seeking.
ChannelRegistry registers groups for discovery/selection.
Components subscribe via samplers tied to TimelineClock per realm.

Invalid payloads surface a descriptive string from describeDirectIngestionValidationFailure in the client (mirrors server-side checks in HeatSystemNextDimensionPayload on the platform).

Producing payloads in HEAT

Approach	When to use
`dashboard-v2` on dashboard-tools (image `dashboard-utils`)	Integrator runners; you supply `$heat-dataservice` + layout in node config
`heat-system-dataservice-envelope`	Declarative mapping from a parent JSON root (`valueFromBindings`, `seriesFromPath`)
`system-tabular-to-dataservice`	Many CSV uploads on one `input-node`
`system-arbex-js` + `heat.dataservice.buildRoot()`	Custom logic and multi-shape payloads in JavaScript

All paths normalize to the same canonical realms structure before the Next UI loads the dimension.

Next dashboard components: widget list and layout wiring
dashboard-v2 upstream contract: field-by-field producer contract
ComposableChart: multi-slice layout on channels
Anatomy of a dashboard: where dashboards sit in the session pipeline
In-repo types: ui/dashboard/src/lib/data-service/data-lib/heat-client/

HEAT documentation team. Last updated 2 October 2026.

Opens your browser's print dialog. Choose "Save as PDF" as the destination.