Debugging and Catalyst Replay¶
To simplify the process of debugging in-situ pipelines, catalyst now
supports the serialization of conduit_nodes. During each API call,
users can write the
params argument of each API call out to disk.
catalyst_replay, the nodes will be read back in,
and each API call will be invoked again. This prevents users from
needing to re-run their simulation when debugging.
Serializing Nodes and Writing to Disk¶
To use the
catalyst_replay command, nodes must first be written to disk.
The steps to do this are simple:
- Set the environment variable
CATALYST_DATA_DUMP_DIRECTORYto the directory where the node data for each API invocation should be saved.
- Invoke the stub implementation in your custom API implementation.
This will write the
conduit_node passed into the API call out to
conduit_nodes are written out as
.conduit_bin files. They will follow the general pattern
<stage>is one of
<num_ranks>is the number of MPI ranks that the simulation was run with.
<rank>is the 0 based index of the rank used to generate this file.
Files for the
execute stage will also include the invocation number,
catalyst_execute can be called multiple times. For example,
execute_invc0_params.conduit_bin.2.1 would contain the
into the 0th invocation of
catalyst_execute, which was called by 2nd of
two ranks (since rank indices are 0-indexed).
Replaying API Calls with catalyst_replay¶
After the node data has been written out to disk, the
command can be used to read the node data back into memory and execute the
same API calls. Find the
catalyst_replay executable in the
RUNTIME_OUTPUT_DIRECTORY generated by CMake (this is usually
catalyst_replay with the same number of MPI ranks as the simulation
used to generate the data, and pass the value of
as a command-line argument. This invoke each API method with the corresponding node
data. For an example, see the