Core Utils Runner
The Core Utils runner (core-utils-runner) provides domain-agnostic transforms: normalize tabular data, extract protobuf capture payloads, reshape JSON, fold capture data into entity trees, and build $heat-dataservice channels from entity-state records for Next dashboards. Use it in integrator session templates when tasks are routed to the core-utils deployment.
For platform-only multi-upload tabular → dataservice on system-utils, see system-tabular-to-dataservice.
Node template selection
| Template | Purpose | Limitations | Details |
|---|---|---|---|
tabular-remap | Rename, derive, or filter CSV/TSV columns with Python expressions before downstream analytics. | Single tabular parent; expressions run per row in memory; not for publishing dashboards. | tabular-remap |
hydrate-protobuf | Extract legacy protobuf types (Gaze, FolderUpload, OpenDIS PDU, SimulationFrame) to JSON, CSV, or binary. | Prefer hydrate-protobuf-v2 for capture-aware SimulationFrame slices; v1 SimulationFrame uses component flattening, not envelope metadata. | hydrate-protobuf |
hydrate-protobuf-v2 | Merge all parents and emit SimulationFrame, EntityState, EntityDataEvent, or TrackingEvent records with capture envelope fields; optional component and identity projection; can truncate inline media carried on an event’s content field. | JSON or CSV only (no raw bin); dedupe upload flag is log-only today. | hydrate-protobuf-v2 |
hydrate-protobuf-journal | Live-session hydration: each run publishes only the newly decoded rows as an append-only part; the platform composes parts back into one CSV or NDJSON artefact, so consumers are unchanged and storage is about data size once. | OpenDIS PDU and GazeComponent (CSV) or EntityState, EntityDataEvent, TrackingEvent and SimulationFrame (NDJSON); requires retention.skipArchive; consumers must read latest only. | hydrate-protobuf-journal |
domain-tree-entities | Decode raw capture bytes and fold every observation into one current-state node per entity, with health and position history. | Merges by entity id, so unrelated captures conflate shared ids. Plain JSON, not a dashboard payload. | domain-tree-entities |
domain-tree-hwio | Carry-forward-fill hardware input and output events into one flat current-state object. | Timing is lost once a field is folded. Pair with domain-tree-hwio-events for occurrences. | domain-tree-hwio |
domain-tree-hwio-events | Emit the full chronological hardware event history, one entry per event. | Output grows with event count. Correlate with a symmetric time window. | domain-tree-hwio-events |
tabular-query | Run SQL SELECTs over one CSV/TSV in on-disk SQLite; output named tables as JSON (streamed). | One input table; all columns are TEXT in SQLite; very large result uploads still take time proportional to result size. | tabular-query |
tabular-to-dataservice | Build $heat-dataservice from one tabular parent for dashboard-v2. | Single parent only; for all uploads on one input-node use system-utils system-tabular-to-dataservice. | tabular-to-dataservice |
json-template | Build output JSON from a template tree with JMESPath leaves. | Single parent; strict fails on JMESPath errors. | json-template |
session-users-from-json | Register session users from identities in a parent JSON output (capture origin_uuid resolved to roles, or Auth GUIDs). | Idempotent; never removes users. | session-users-from-json |
json-flatten | Flatten nested JSON (or NDJSON lines) to CSV/TSV. | Single parent; wide sparse tables possible on deep nesting. | json-flatten |
json-merge | Shallow or deep merge of multiple JSON object inputs. | Not a dashboard node; arrays off unless allowArrays is set. | json-merge |
simulation-frame-merge | Pivot long-format SimulationFrame JSON to wide CSV by tick and entity. | Expects hydrate-style simulation_frame arrays; include origin_uuid in groupBy for multi-capture merges. | simulation-frame-merge |
converge-outputs | Re-expose every upstream output on this node without copying blobs (aggregation). | Maps references only; configure whenNoOutputs for empty upstream behaviour. | converge-outputs |
session-summary | Compute configuration-driven summary metrics from entity-state records for a Legacy stats strip. | Emits Legacy v1 combined-custom-top-stats, not $heat-dataservice. Single parent. | session-summary |
threat-zone-detection | Measure distance from a target entity to tracked entities and emit safe and threat phases as a responseTime channel. | Requires realmMapping, which the manifest schema does not declare. Simulation time must be epoch seconds. | threat-zone-detection |
entity-state-to-mapdisplay | Turn entity-state records into a mapdisplay channel of playable tracks for MapDisplay. | Single parent. hideAfterEnd is honoured by MapDisplay only, not TacticalMap. | entity-state-to-mapdisplay |
entity-state-to-chart-data | Extract series, events and timestamps channels for ComposableChart from entity-state records. | Any sub-configuration disables auto-discovery. Single parent. | entity-state-to-chart-data |
entity-state-to-stats-summary | Reduce detection beams and sensor drops to fixed engagement counters as a value channel. | Fixed counter set; requires baseCapturedAt and both platform entity lists. | entity-state-to-stats-summary |
opendis-decode | Reshape hydrate-protobuf OpenDIS PDU output into the pdu list entity-state-to-records reads. | No configuration. Does not decode DIS bytes. | opendis-decode |
entity-state-to-records | Convert decoded EntityStatePdu objects into entity-state records with WGS84 positions. | Only EntityStatePdu is converted. Reads the first payload of one parent. | entity-state-to-records |
entity-state-to-flightpath | Turn entity-state records into flightpath channels for the FlightPath widget. | Single parent. One merged channel with entityNames, or one channel per discovered entity without it. | entity-state-to-flightpath |
entity-state-to-cognitive | Score cognitive load from gaze and pupil data with the bundled XGBoost model and emit series channels. | Falls back to a zero-scoring model when the model paths do not load. Single parent. | entity-state-to-cognitive |
entity-state-to-adaptive-cognitive-load | Score cognitive load against a rolling personal baseline from gaze, pupil and heart-rate data. | Needs at least 10 seconds of samples. Score scale differs from entity-state-to-cognitive. | entity-state-to-adaptive-cognitive-load |
entity-state-to-flight-stats | Reduce one numeric field on one entity to min, max and average as a value channel for StatsSummary. | One metric per instance. Single parent. | entity-state-to-flight-stats |
cognitive-load-breakdown | Reduce a cognitive load series channel to a duration-weighted average and time in zone for CognitiveLoadBreakdown. | Reads one series channel by id from a $heat-dataservice parent. | cognitive-load-breakdown |
entity-state-to-eyetracking | Aggregate gaze samples into left, right, up and down percentages for the EyeTracking widget. | Session total only, one sample per record. Fails without usable gaze data. | entity-state-to-eyetracking |
entity-state-to-viewport-tracking | Reduce gaze samples to duration-weighted compass sectors and vertical zones as two value channels. | Screen-point fallback never reaches side or rear sectors. No component page today. | entity-state-to-viewport-tracking |
entity-state-to-lookout | Detect sharp gaze direction changes and emit them as an events channel. | Needs at least two gaze samples. channelName is not applied today. | entity-state-to-lookout |
Project companion templates
These templates are registered on the core-utils runner but are documented with their project, not on the platform docs.
| Template | Purpose | Limitations | Details |
|---|---|---|---|
entity-state-to-smartchair-flightpath | Project-specific transform, documented with its project. | Project scope only. | Index |
entity-state-to-smartchair-chart-data | Project-specific transform, documented with its project. | Project scope only. | Index |
Example workflow
Entity-state workflow
Most entity-state-to-* templates read the same input: the records array that hydrate-protobuf-v2 emits with extractionTarget: EntityState. Each produces one or more $heat-dataservice channels, and json-merge combines them before dashboard-v2.
Supported tabular formats
| Input | Output | Notes |
|---|---|---|
| CSV | CSV / TSV | Auto-detect delimiter in auto mode |
| TSV | CSV / TSV | Delimiter preserved or converted per config |
Related
Opens your browser's print dialog. Choose "Save as PDF" as the destination.