1. Today's topic

We are designing a binary protocol over UART, RS-485, TCP, UDP, CAN FD, MQTT binary payloads and a Raspberry Pi HIL link. Main topics:

text 
framing
COBS / delimiter
length
CRC
endianness
versioning
TLV
sequence/request_id
stream parser
resynchronization

The main idea: a transport read() is not a message. TCP and UART deliver a byte stream, so first extract a message from the stream, check its boundaries, lengths and CRC, and only then pass it to the application layer.

2. Why this matters in projects

For HIL, a Raspberry Pi can send:

text 
SET_INPUT A03 HIGH
INJECT_MODEM_URC
READ_DIAGNOSTICS
RESET_FAULT_COUNTERS
START_TEST
GET_TEST_RESULT

A text CLI is convenient for a person, but a binary protocol is more convenient for automated tests:

  • fixed types;
  • smaller size;
  • sequence/correlation ID;
  • CRC;
  • strict limits;
  • easier fuzzing.

For remote input modules, a binary protocol can transmit:

text 
raw input mask
confirmed phase
ADC samples
conflict flags
sequence
measurement timestamp
power status

If one byte is lost, the parser must discard the damaged frame and resynchronise at the next one.

3. Theory

Protocol layers

text 
transport adapter
  UART/TCP/UDP/CAN
frame decoder
  delimiter, COBS, length, CRC
message decoder
  version, type, TLV, schema
application service
  authorization, execution, state changes

The frame parser must not operate GPIO, MQTT, NVS or reset.

Proposed format

For UART/RS-485:

text 
COBS(encoded logical frame) + 0x00 delimiter

After COBS decoding:

text 
Offset Size Field
0      2    Magic = 0x4550 "EP"
2      1    Protocol major version
3      1    Header length
4      2    Message type
6      2    Flags
8      4    Sequence
12     4    Payload length
16     ...  Optional header extensions
...    N    Payload
...    4    CRC32C

Why include magic, delimiter, length and CRC together:

  • the delimiter locates the physical frame boundary;
  • COBS guarantees that the delimiter does not occur inside the payload;
  • magic filters out noise and other protocols;
  • length prevents reads beyond the buffer;
  • CRC filters out damaged data.

Do not transmit C structures directly

Poor approach:

c 
typedef struct {
    uint8_t version;
    uint32_t sequence;
    uint16_t value;
} packet_t;
uart_write_bytes(UART_NUM_1, &packet, sizeof(packet));

Problems: padding, endianness, enum size, ABI, unaligned access and uninitialised bytes. The correct approach is to write fields explicitly, in a fixed order and endianness.

c 
static void write_u32_be(uint8_t *p, uint32_t value)
{
    p[0] = (uint8_t)(value >> 24U);
    p[1] = (uint8_t)(value >> 16U);
    p[2] = (uint8_t)(value >> 8U);
    p[3] = (uint8_t)value;
}

A safe parsing order

text 
1. Minimum size.
2. Magic.
3. Version.
4. Header length.
5. Payload maximum.
6. Exact total size.
7. CRC.
8. Flags/schema.
9. Dispatch.

Application logic is not called before CRC validation.

TLV

TLV = Type-Length-Value.

text 
Type:   uint16 big-endian
Length: uint16 big-endian
Value:  Length bytes

You can reserve the most significant bit of Type:

text 
type & 0x8000 == 0 -> optional unknown TLV can be skipped
type & 0x8000 != 0 -> critical unknown TLV rejects message

4. Common mistakes

  • Treating one uart_read() as one frame.
  • Using packed C structures as the wire format.
  • Passing a network-supplied payload_length directly to malloc().
  • Checking CRC after parsing the payload.
  • Continuing the current frame after a UART overflow.
  • Using only magic, without CRC/length.
  • Using a length prefix on a noisy UART without resynchronisation.
  • Changing an old field's meaning in a minor update.
  • Failing to limit the number of TLVs in the TLV parser.
  • Mistaking sequence numbers for cryptographic protection.

5. A practical task for 30–60 minutes

Create PROTOCOL_SPEC.md:

markdown 
# Embedded Protocol v1
## Transport
UART and RS-485:
- COBS encoded
- 0x00 frame delimiter
TCP:
- same COBS stream for initial implementation
UDP:
- one decoded logical frame per datagram
## Limits
Maximum header: 64 bytes
Maximum payload: 1024 bytes
Maximum decoded frame: 1092 bytes
No runtime allocation in parser
## Byte order
All multi-byte integers are big-endian.
## Validation order
1. COBS
2. Minimum length
3. Magic
4. Version
5. Header length
6. Payload limit
7. Exact total length
8. CRC32C
9. Message schema
10. Application authorization

Implement these modules:

text 
cobs.c / cobs.h
protocol_frame.c / protocol_frame.h
protocol_stream.c / protocol_stream.h

Unit tests:

text 
1. A complete valid frame.
2. The same frame one byte at a time.
3. Two frames in one feed().
4. Empty delimiters.
5. A changed byte -> BAD_CRC.
6. Header length < 16 -> reject.
7. Payload length > max -> reject.
8. After an invalid frame, the next valid frame is accepted.
9. Unknown optional TLV is skipped.
10. Unknown critical TLV is rejected.

6. What to read or try next

  • ESP-IDF UART driver: event queue, RX timeout, FIFO overflow and ring-buffer overflow.
  • STM32 ReceiveToIdle_DMA for UART + DMA.
  • RFC 8949 CBOR if a more flexible format is needed.
  • Protocol Buffers wire format, taking deterministic encoding into account for signatures.
One uart_read() returns half a frame, and the next read returns the rest. Which design matches the lesson?

Criteria: Keep framing and validation ahead of application side effects.

Exercise

A damaged UART frame is followed by a valid frame. Describe the expected parser result and two tests that demonstrate recovery without invoking application logic for the damaged frame.

Self-check criteria: Include bounded validation, resynchronisation and equivalent results across chunk boundaries; CRC rejection must precede dispatch.

Show the supplied answer

Discard the damaged frame, resynchronise at the next frame boundary and accept the following valid frame after length/CRC validation. Test the pair in one feed() and then with the same bytes split across several feed() calls. Only the valid frame should be dispatched.