Ingests

In HEAT, an Ingest is a managed endpoint that triggers the creation or augmentation of a Session. By sending data (e.g., a file, JSON payload, or CSV) to an Ingest’s URL, you begin the data-processing workflow defined by a specified Session Template in a particular Project.

How Ingests Work

1. Programmatic Endpoint
   An Ingest provides an HTTP endpoint (/ingest/...) that you can POST data to. You can define a custom URL suffix for clarity (e.g., /ingest/my-simulator), but all Ingests reside under the /ingest/ prefix.

2. Session Creation or Augmentation
   When you post data to this endpoint, HEAT automatically:

   • Creates a Session based on the assigned Session Template if no existing session is found.
   • Inserts your submitted payload into the input node specified by the template.
   • Optionally updates an existing Session if you’ve configured the Ingest to group multiple data uploads into a single Session.

3. Data Pipeline Kickstart
   Once the payload is placed into the designated input node, HEAT begins executing the data-processing pipeline (DAG) specified in the chosen Session Template. This includes any transformations, analytics, or validation steps defined by the workflow.

What an Ingest Binds Together

Every Ingest is tied to one Project, one Session Template, one input node in that template and one Data Source where the payload is stored. These define where and how incoming data is processed.

An Ingest also declares the payload type it accepts (any data, JSON only, or CSV only) and whether senders must present a secret key or may post anonymously.

Session Augmentation

Ingests can optionally be configured to append data to an existing session. The Ingest names a lookup key, a top-level field in the payload. When a session in the project already carries the same value for that field, the new payload lands in that session. Otherwise a new session is created and the value is recorded so later uploads match. This is useful when multiple data files combine to form a single training session, or when data arrives incrementally.

Creating an Ingest

Ingests are created and edited in the Control Centre. The operator page Ingests walks through every field, the resulting URL, a curl example and what happens when data arrives.

Next Steps

• Learn how nodes reference storage in Data Sources.
• Understand the blueprint behind each session in Session Templates.
• Configure an endpoint in the Control Centre.

HEAT documentation team. Last updated 2 October 2026.