Lesson 54. Event sourcing and deterministic replay

1. Today's topic

The idea: record input events for the deterministic core, rather than an arbitrary log, so that the FSM behaviour can later be reproduced exactly on a PC.

REAL DEVICE:
GPIO / ADC decisions / UART EC25 / MQTT / timers
    -> normalized events
    -> production FSM + event recorder
    -> replay file
LINUX HOST:
replay file
    -> same FSM/core
    -> deterministic result

The main idea: there is no need to emulate the whole ESP32, EC25 and internet. Feed the same sequence of meaningful events back into the pure logic.

2. Why this matters

A rare EC25 field failure may depend on the order:

+CEREG:0
MQTT socket lost
AT timeout
+CEREG:1
late OK

An ordinary log does not always let you reconstruct the order. A replay file lets you run the same scenario on a PC and turn the failure into a regression test.

3. Theory

What to record

Record the inputs to the core:

MODEM_LINE
MODEM_URC
MQTT_CONNECTED
MQTT_DISCONNECTED
COMMAND_RECEIVED
GPIO_EDGE
ADC_DECISION
CAN_FRAME
TIMER_EXPIRED
POWER_STATE_CHANGED
CONFIG_LOADED

There is no need to record RTOS internals: task switches, mutex acquisition, malloc or printf.

A timer is also an event

The FSM must not ask for the real esp_timer_get_time(). A timer service generates TIMER_EXPIRED. Replay simply feeds the recorded event.

Logical time

An event contains mono_us, but replay must not wait for 40 real seconds. It can instantly set the logical clock to the event timestamp.

Sequence matters more than timestamp

Several events may have the same timestamp. The sequence defines the primary order. Preserve fragmentation

If the parser is being tested, chunk boundaries must be preserved:

callback 17 bytes
callback 31 bytes
callback 4 bytes

Otherwise, the parser bug may disappear.

Outputs as assertions

Expected actions can be recorded:

INPUT: CEREG_REGISTERED
OUTPUT: SEND_AT QIACT

Replay feeds the stimuli and uses observations as assertions.

State hash

After every event, you can hash a canonical state projection, rather than a raw C structure.

4. Common mistakes

1. Recording text instead of events.
2. Recording only errors, without pre-trigger history.
3. Not preserving the sequence.
4. Using real time during replay.
5. Letting the FSM read UART/GPIO/timers itself.
6. Letting the FSM publish MQTT messages itself.
7. Using a Python model on the PC instead of the production C core.
8. Not preserving UART/TCP fragmentation during parser replay.
9. Replaying raw ADC data when only FSM replay is needed.
10. Not versioning the event schema.
11. Hashing raw structs.
12. Not checking invariants after every event.
13. Letting the recorder itself break timing.
14. Writing every event directly to Flash.
15. Recording secrets in a replay artifact.

5. Practical task

Implement replay for modem_core with these events:

typedef enum {
    MODEM_EV_AT_OK = 1,
    MODEM_EV_AT_ERROR,
    MODEM_EV_CEREG_CHANGED,
    MODEM_EV_TIMER_EXPIRED,
} modem_event_type_t;
typedef struct {
    uint32_t sequence;
    uint64_t mono_us;
    modem_event_type_t type;
    int32_t argument;
} modem_replay_event_t;

Start with textual replay v0:

1 0 CEREG 1
2 1000 TIMER 10
3 1100 OK 0
4 1200 CEREG 0
5 4200 TIMER 11
6 4300 CEREG 1
7 4400 OK 0

After every event, print a state dump and check invariants. Create a late OK scenario: an old transaction times out, a new transaction starts, then AT_OK arrives for the old transaction. If the current model does not distinguish transaction IDs, add one.

6. What to try next

• Record .evlog on a Raspberry Pi through diagnostic UART.
• Convert a field replay into a fuzz seed.
• Add differential replay: compare the old modem FSM with the new modem FSM after refactoring.