Lesson 27. Data races, deadlock, and priority inversion

1. Today’s topic

We examine synchronization:

task ↔ task
ISR ↔ task
ESP32 core ↔ another ESP32 core
DMA ↔ CPU
driver ↔ application logic

Main problems:

race condition
deadlock
priority inversion
starvation

The central idea: the best way to protect a shared resource is, where possible, not to share it between tasks at all. Assign one owner and let other tasks send it commands and events.

2. Why this matters

EC25

If modem_task, mqtt_task, sms_task, and the CLI write directly to UART, even a mutex around uart_write() will not save them: responses and URC will mix. Serialize the whole AT transaction or transfer UART ownership to a single modem_task.

Phase detector

If input_task updates red/yellow/green/timestamp while MQTT/CAN/health read concurrently, readers may see “torn state”: some fields are new while others are old.

DMA

DMA does not acquire a mutex. If CPU reads the region DMA is writing, the result will be a mixed packet.

3. Theory

Race condition

counter++ is read-modify-write. Two tasks can read the same value and lose an increment.

volatile is not a mutex

volatile does not guarantee atomicity, structure consistency, inter-core ordering, or protection from ISR. Synchronization requires a mutex, queue, notification, critical section, or atomics.

Mutex vs binary semaphore

A mutex has an owner and priority inheritance. A binary semaphore is a signal. Use a mutex for mutual exclusion between tasks; use a semaphore/notification for ISR→task. Priority inversion

LOW holds a mutex, HIGH waits for it, and MEDIUM preempts LOW. HIGH indirectly waits for MEDIUM. A mutex with priority inheritance temporarily raises LOW to HIGH so it can release the resource. But inheritance does not make a long blocking operation safe. Under a mutex, do not wait for the network, an AT-response, Flash, or a callback, or log large strings.

Deadlock

Task A: lock(CONFIG) → lock(I2C)
Task B: lock(I2C) → lock(CONFIG)

Solution: a global lock order.

1. CONFIG
2. STATE
3. I2C
4. MODEM
5. LOG

Prefer not to acquire two mutexes simultaneously at all.

Owner-task pattern

Instead of:

Task A → mutex → UART
Task B → mutex → UART
Task C → mutex → UART

use:

Task A ─┐
Task B ─┼→ command queue → UART owner task → peripheral
Task C ─┘

4. Common mistakes

• Treating volatile as a mutex.
• Using a binary semaphore for mutual exclusion.
• Using a mutex from ISR.
• Holding a mutex during I/O.
• Nested mutexes without an order.
• Calling a callback under a lock.
• Using vTaskSuspendAll() as a lock on ESP32 SMP.
• Protecting each UART write but not the whole AT transaction.
• Using an event bit where a counter is needed.

5. Practical assignment

Create RTOS_SYNC_POLICY.md.

# RTOS synchronization policy
1. Every peripheral has one explicit owner.
2. ISR never waits for mutex or performs recovery.
3. Mutex protects only short task-context resource access.
4. No network, delay, logging or callback while holding mutex.
5. Multiple locks are acquired only in documented order.
6. Every system mutex has finite timeout and diagnostics.
7. Fast ISR-to-task signalling uses notifications/thread flags.
8. Structured events are passed through queues.
9. Shared structures are read only through atomic snapshot API.
10. volatile is not used as synchronization mechanism.

Add a resource table: EC25 UART, I2C, phase state, logger UART, CAN TX, and config.

6. What to try next

• Prohibit direct access to EC25 UART from every file except modem_transport.c.
• Add modem_command_submit().
• Add lock diagnostics: owner, max_hold_us, and timeout_count.
• Run a HIL test with MQTT publish + SMS + CLI AT simultaneously.