Research note: Embassy and RIOT

Kind: Read-only source analysis, 2026-10-07. Embassy from a sparse shallow clone of embassy-rs/embassy at commit b3e27baf6c20 (embassy-executor, embassy-time, embassy-time-driver, embassy-time-queue-utils, embassy-sync); paths relative to the repository root. RIOT from RIOT-OS/RIOT at commit f9e38576567b, read from the git objects (git show HEAD:<path>) because the sparse checkout was empty; paths relative to the repository root as stored in git. Line numbers drift; paths and symbols are the durable reference. Synthesized in R-001.

Part A: Embassy

A1 Execution model

  • The run queue is an intrusive lock-free stack (cordyceps::TransferStack<TaskHeader>): enqueue sets task.next = old head and atomically swaps the head; dequeue takes the whole batch with take_all and iterates it (reverse order, justified for fairness) (embassy-executor/src/raw/run_queue.rs:45-58, 99-111). Link storage is the run_queue_item field inside every TaskHeader (raw/mod.rs:109-120, run_queue.rs:18, 39-42); targets without pointer atomics fall back to a critical-section-guarded stack (run_queue.rs:195-235).

  • A waker is (pointer to TaskHeader, static VTABLE); clone copies, drop is a no-op, so waking needs no allocation (raw/waker.rs:5-21). wake_task does state.run_enqueue: atomically sets RUN_QUEUED and only if it was clear pushes to the owning executor and calls the pender (raw/mod.rs:630-639, 462-469). task_from_waker recovers the task by comparing vtable addresses so wait queues can store a one-word task pointer (waker.rs:34-49; turbo variant waker_turbo.rs).

  • State bits: STATE_SPAWNED and STATE_RUN_QUEUED (raw/state_atomics.rs:25-27; the ARM variant packs AtomicBools into one u32, state_atomics_arm.rs:14-25, 68-74; critical-section fallback state_critical_section.rs). Spawn is a CAS 0 -> SPAWNED|RUN_QUEUED (state_atomics.rs:42-46); lifecycle diagram at mod.rs:70-108. There is no TIMER_QUEUED bit in this version: timer-queue membership lives in TaskHeader.timer_queue_item (mod.rs:116-117) and “in queue” is encoded as waker.is_some() (embassy-time-queue-utils/src/queue_integrated.rs:21-22, 146-149).

  • Task storage: TaskStorage<F> { raw: TaskHeader, future: UninitCell<F> } is repr(C) so the header is at offset 0 (mod.rs:218-224); TaskPool<F, N> is [TaskStorage<F>; N]; spawn claims the first free slot or returns SpawnError::Busy (mod.rs:383-400, spawner.rs:60-67). The #[task] macro emits a static sized at compile time (src/lib.rs:151-193; README.md:5-6); the future is built in place via a closure (mod.rs:250-256, 330). On Poll::Ready the future is dropped, poll_fn replaced by poll_exited, then despawn (mod.rs:276-291).

  • Poll loop: SyncExecutor::poll is run_queue.dequeue_all(|p| (task.poll_fn)(p)) (mod.rs:487-506). The thread-mode Cortex-M executor is loop { poll(); wfe } with pender sev (src/platform/cortex_m.rs:107-117, 16-21). InterruptExecutor: the pender pends an NVIC IRQ (STIR on v7, cortex_m.rs:23-47), the ISR calls executor.poll() (179-182), start() unmasks the IRQ and returns a SendSpawner (207-223). Preemptive priority levels are several executors on software interrupts at different NVIC priorities; a higher-priority ISR executor preempts a lower one mid-poll, while tasks inside one executor stay cooperative (cortex_m.rs:134-147; README.md:12). The pender may be called from any context and must never call poll re-entrantly (src/pender.rs:10-28, mod.rs:530-535).

  • Optional scheduler-priority and scheduler-deadline sort each dequeued batch into a SortedList (run_queue.rs:127-176; Cargo.toml:210-215; src/metadata.rs:60-94), still with no preemption within an executor.

  • The “context switch” cost is one indirect call poll_fn(p) (mod.rs:498); the waker is mem::forget-ed to avoid a virtual drop (299-301). Stack: one per executor (the thread executor on the main stack, interrupt executors on the exception stack); future state is in the static TaskStorage, not on any stack.

A2 Time

  • Driver trait: fn now(&self) -> u64; fn schedule_wake(&self, at: u64, waker: &Waker) (embassy-time-driver/src/lib.rs:118-139). now() must be monotonic and effectively never overflow; 16- and 32-bit hardware must be extended to 64 bits in software (lib.rs:120-128). Binding is by the linker symbols _embassy_time_now and _embassy_time_schedule_wake from time_driver_impl!, exactly one driver per image (lib.rs:141-177).

  • TICK_HZ is a compile-time const chosen by tick-hz-* features (lib.rs:112-115, src/tick.rs:3-40). Instant and Duration are u64 ticks (embassy-time/src/instant.rs:9-10, duration.rs:10-11); ms and µs conversion uses GCD-reduced constant multiply and divide (duration.rs:31-37, 56-66; src/lib.rs:43-45); Instant + Duration is checked_add and panics on overflow, try_* variants exist (instant.rs:178-211, timer.rs:162-174). No wrap handling at all: 64 bits at 1 MHz is about 584,000 years (embassy-time/README.md:11-13).

  • Timer queue API: schedule_wake(at, waker) -> bool (“re-arm the alarm”) and next_expiration(now) -> u64 (wakes expired, returns the minimum) (embassy-time-queue-utils/src/lib.rs:37-48). The integrated queue is an intrusive singly linked list through the per-task TimerQueueItem; re-scheduling an already queued task only lowers expires_at (queue_integrated.rs:35-37, 76-108, 114-132). The generic queue is a heapless::Vec<(at, Waker)>, default 64 slots; when full it pops and wakes a victim (queue_generic.rs:38-40, 70-77, 103-120).

  • One hardware alarm multiplexing: the recommended driver loop is if queue.schedule_wake(..) { loop { next = queue.next_expiration(now); if set_alarm(next) break } } under a critical section (embassy-time-driver/src/lib.rs:73-84). The CMSDK driver shows it concretely: a 32-bit free-running “timekeeper” whose overflow IRQ bumps periods (embassy-time/src/driver_cmsdk/mod.rs:197-199), one alarm timer, trigger_alarm re-arms from the queue (227-243), set_alarm returns false if the target already passed (245-260). No per-tick interrupt: only overflow and next-deadline IRQs.

  • Timer::at and after store expires_at; poll returns Ready only if yielded_once && expired, otherwise schedule_wake(expires_at) and Pending, so it always yields at least once (timer.rs:128-165, 304-315). block_for busy-waits on Instant::now() (delay.rs:14-17).

  • Ticker is drift-free: next() does expires_at += duration from the previous expiry, not from now (timer.rs:359-375, 470-481; README.md:34). Consequence: if several periods are missed it fires back to back until caught up (no skip); reset* methods re-anchor to now (402-460).

A3 Synchronization

  • RawMutex { const INIT; fn lock<R>(&self, f) -> R } only guarantees no concurrent access and is reentrant-tolerant (embassy-sync/src/blocking_mutex/raw.rs:17-38). CriticalSectionRawMutex is critical_section::with (the portable CS crate, 59-65); NoopRawMutex is f() and is !Sync so it cannot cross executors (120-138); ThreadModeRawMutex asserts ICSR.VECTACTIVE == 0 (167-198); PanicRawMutex detects misuse (88-110).

  • Async Mutex: locked: bool plus one WakerRegistration behind a blocking mutex (mutex.rs:19-47); lock registers the waker if locked (77-96); guard drop clears and wakes (203-215). Not fair or FIFO: a single waker slot; registering a second waiter wakes the first so they re-register in turn (waitqueue/waker_registration.rs:21-43). No ownership or priority concept.

  • Channel<M,T,N>: Deque<T,N> plus receiver_waker and senders_waker (both single-slot) (channel.rs:687-691); send wakes the receiver (773-786), receive wakes senders when it frees a full queue (731-744). MPMC in API but multi-waiter contention resolves via the wake-the-other-waiter trick.

  • Signal: enum State { None, Waiting(Waker), Signaled(T) }; signal overwrites and wakes (signal.rs:44-49, 77-84, 92-112), a latest-value notification for one consumer.

  • PubSubChannel: Deque<(T, remaining_readers)>, u64 message ids, MultiWakerRegistration<SUBS> and <PUBS> (pubsub/mod.rs:304-319); slow subscribers get Lagged(n) (364-369); publish_immediate drops the oldest (354-362); an entry is freed when the last reader consumed it (383-387).

  • Wait queues: WakerRegistration single slot (waker_registration.rs:10-50); AtomicWaker lock-free 2-bit state machine WAITING, REGISTERING, WAKING with wake hand-off to an in-flight register (waitqueue/atomic_waker.rs:62-77, 146-268), aliasing CriticalSectionWaker where no 32-bit CAS exists (282-283); MultiWakerRegistration<N> is a heapless Vec that wakes everyone when full (multi_waker.rs:9-51).

  • zerocopy_channel: SPSC over a caller-provided &mut [T]; SendSlot hands out &mut T into the ring and send_done() commits (zerocopy_channel.rs:39-66, 125-172, 193-217). Pipe: byte RingBuffer<N> with read and write WakerRegistration, implements embedded_io_async (pipe.rs:216-218, 273-275, 550-574).

Embassy: ideas to borrow

  • Static task pools with compile-time sizing and Busy on exhaustion, no heap (raw/mod.rs:383-400, lib.rs:151-193).

  • Waker-style completion: a one-word task pointer plus “enqueue only if RUN_QUEUED was clear” makes wakes idempotent and ISR-safe (mod.rs:630-639; state_atomics_arm.rs:68-74); intrusive run-queue links inside the TCB (run_queue.rs:18).

  • A batch-dequeue run queue (take all then iterate) for O(1) ISR enqueue (run_queue.rs:99-111).

  • One-alarm multiplexing via schedule_wake -> bool plus next_expiration(now), with the “set alarm, verify not in past, retry” loop (embassy-time-driver/src/lib.rs:73-84; driver_cmsdk/mod.rs:227-260); a 64-bit tick timebase so wrap never matters (lib.rs:120-128).

  • Ticker semantics expires_at += period (timer.rs:470-481).

  • The RawMutex abstraction selecting a critical section, a thread-mode assert, or a no-op lock per data-sharing scope, plus the portable critical_section crate (blocking_mutex/raw.rs:30-38, 59-65, 167-198).

  • Lagged-aware broadcast with per-message reader counts (pubsub/mod.rs:306, 364-369); the zero-copy slot channel (zerocopy_channel.rs).

Embassy: weaknesses and limitations

  • No preemption inside an executor; a long poll blocks all peers; priorities only via separate interrupt executors on single-core bare metal (cortex_m.rs:134-147; Cargo.toml:160-165).

  • No priority inheritance or ownership in the async Mutex; single-slot wakers cause waiter thrashing (mutex.rs:19-23; waker_registration.rs:30-38).

  • No isolation: everything is privileged, with unsafe task and pointer casts (mod.rs:218-224); SpawnToken drop panics (spawner.rs:51-56); Instant arithmetic panics on overflow (instant.rs:202-211).

  • Timer queue capacity: the generic queue silently wakes a victim when full (queue_generic.rs:70-77); the integrated queue requires Embassy wakers only (waker.rs:34-38).

  • Debuggability of async state machines relies on an optional trace feature (lib.rs:7-14); Ticker catch-up bursts after long stalls (timer.rs:472-475).

Part B: RIOT

B1 Scheduler

  • sched_runqueues[SCHED_PRIO_LEVELS] (circular singly linked clist, O(1) rpush, lpop, lpoprpush, core/lib/include/clist.h:21-40) plus runqueue_bitcache u32 (core/sched.c:76-77); 16 levels by default, maximum 32 (core/include/sched.h:188-189, sched.c:60). Bit order flips on CLZ availability so selection is one clz or ctz (sched.c:84-116).

  • sched_run: with no idle thread and an empty bitcache it loops sched_arch_idle() (sched.c:149-158); clears sched_context_switch_request, selects the highest-priority queue and its head sched_runqueues[nextrq].next->next (160-164), marks RUNNING, updates sched_active_pid and thread, TLS, the MPU stack guard (177-219).

  • Priority 0 is highest (“nice” values, sched.h:22-39); THREAD_PRIORITY_MIN = SCHED_PRIO_LEVELS-1 = IDLE (core/lib/include/thread_config.h:118-131). Equal priorities are semi-cooperative (sched.h:36-39); thread_yield does sched_runq_advance, which is clist_lpoprpush (core/thread.c:151-162, sched.h:318-321). Optional sched_round_robin: a ztimer (10 ms on ZTIMER_MSEC) advances the queue and yields, driven by sched_runq_callback on runqueue changes (sys/sched_round_robin/sched_round_robin.c:41-107; sys/include/sched_round_robin.h:42-71; hooks in sched.c:166-168, 237-242, 256-258).

  • States: STOPPED, ZOMBIE, SLEEPING, MUTEX, RECEIVE, SEND, REPLY_BLOCKED, FLAG_BLOCKED_ANY and ALL, MBOX_BLOCKED, COND_BLOCKED, RUNNING, PENDING (sched.h:159-172); STATUS_ON_RUNQUEUE == STATUS_RUNNING, so status >= RUNNING means runnable (sched.h:180). sched_set_status pushes or pops the runqueue when crossing that threshold and clears the bit when a queue empties (sched.c:227-276).

  • Switch request: sched_switch(other_prio) yields immediately via thread_yield_higher() in thread context, or sets sched_context_switch_request = 1 in ISR context (sched.c:278-302, sched.h:210-233). On Cortex-M the switch is PendSV (SCB_ICSR_PENDSVSET, cpu/cortexm_common/thread_arch.c:549) whose handler calls sched_run with IRQs disabled and skips the switch if the same thread wins (thread_arch.c:318-343).

  • IRQ API: irq_disable() returns the old PRIMASK, irq_restore(state) (core/lib/include/irq.h:46-72; cpu/cortexm_common/include/irq_arch.h:52-83); irq_is_in() is IPSR & 0xFF (irq_arch.h:119-121). Blocking from an ISR is refused, not asserted everywhere: thread_sleep returns early (thread.c:109-113), ztimer_sleep asserts (sys/ztimer/util.c:45), mutex _block asserts an active thread (core/mutex.c:62-64), msg_send redirects to msg_send_int (core/msg.c:65-67).

  • Idle: the core_idle_thread module runs pm_set_lowest() in a loop at THREAD_PRIORITY_IDLE (core/lib/init.c:92-98, 116-120); otherwise sched_arch_idle() (pm_set_lowest() or WFI, then briefly re-enable IRQs) runs inside the scheduler (thread_arch.c:553-566; sched.h:268-281).

  • MAXTHREADS defaults to 32 (sched.h:94-96); sched_threads[KERNEL_PID_LAST+1] is a static thread_t* array (sched.c:57), PIDs KERNEL_PID_FIRST=1 .. FIRST+MAXTHREADS-1 (sched.h:105-115); thread_create linearly scans for a free PID else -EOVERFLOW (thread.c:313-326), places the TCB at the top of the caller’s stack (259-272), paints the stack for thread_measure_stack_free (298-308, 207-234). thread_t fields: sp, status, priority, pid, flags, rq_entry, wait_data, msg_waiters, msg_queue (cib), msg_array (core/include/thread.h:167-208).

B2 IPC and synchronization

  • msg_t is sender_pid, u16 type, union {u32 value, pointer} (core/include/msg.h:194-200). _msg_send: if the target is not RECEIVE_BLOCKED, try to copy into the target’s per-thread cib ring (msg.c:43-61, 105-122); no queue and blocking leads to SEND_BLOCKED, priority-sorted into target->msg_waiters (134-154); msg_try_send returns 0 instead (124-129); if the target is receive-blocked, copy directly into target->wait_data and make it PENDING (159-170). _msg_receive pops the ring or blocks RECEIVE_BLOCKED; a blocked sender’s message is pulled in and the sender released (360-444). msg_send_receive and msg_reply rendezvous via REPLY_BLOCKED (260-326). msg_init_queue sets the per-thread array (504-510); docs: synchronous by default, asynchronous only with a queue, try_send drops (msg.h:87-110).

  • thread_flags_t is u16 in the TCB (core/include/thread_flags.h:118, thread.h:175). thread_flags_set ORs the mask and _thread_flags_wake compares against the mask parked in wait_data for ANY or ALL, sets PENDING and sched_context_switch_request (core/thread_flags.c:25-51, 143-157); wait_any, wait_all, wait_one block in STATUS_FLAG_BLOCKED_* and atomically clear what they return (53-62, 87-135). Predefined THREAD_FLAG_MSG_WAITING (1<<15) and THREAD_FLAG_TIMEOUT (1<<14) (thread_flags.h:93-99); message queueing and blocked sends set MSG_WAITING via thread_flags_set_internal so one wait covers messages, timeouts, and events (msg.c:57-59, 149-151). sys/event builds on this: event_post pushes a sender-allocated clist node and sets THREAD_FLAG_EVENT (sys/event/event.c:34-49), event_wait_multi loops pop then thread_flags_wait_any (85-110), timeouts via THREAD_FLAG_TIMEOUT (113-129, 162-177; sys/include/event.h:33, 45).

  • Mutex: mutex_t.queue is NULL, MUTEX_LOCKED ((list_node_t*)-1), or the head of a priority-sorted waiter list (core/include/mutex.h:36-72, 130; core/mutex.doc.md:17-24); waiters are inserted by thread_add_to_list (mutex.c:67-74, thread.c:187-205); unlock wakes the head, the highest priority (204-210). Priority inheritance is opt-in (core_mutex_priority_inheritance): raise the owner on block, restore on unlock (mutex.c:76-85, 216-224); the default is none (mutex.doc.md:5-9). mutex_cancel: mutex_cancel_t {mutex, thread, cancelled} (mutex.h:103-107, 167-169), mutex_lock_cancelable returns -ECANCELED (mutex.c:135-182), mutex_cancel removes the thread from the waiter list and wakes it, ISR-safe (263-293), used for ztimer_mutex_lock_timeout (util.c:221-243). mutex_unlock_and_sleep (233-261).

  • cond_wait unlocks, blocks COND_BLOCKED on a sorted list, relocks on return (core/cond.c:32-48); signal and broadcast then sched_switch(min_prio) (50-75). rmutex_t {mutex, refcount, owner} with an atomic owner read on relock (core/lib/include/rmutex.h:34-56, core/lib/rmutex.c:40-120). mbox: a cib ring of msg_t plus readers and writers lists; put hands directly to a waiting reader, else queues or blocks the writer (core/mbox.c:57-124).

B3 ztimer

  • Clock tree: ztimer_clock_t {list, ops, last, adjust_set, adjust_sleep, adjust_clock_start, users, max_value, lower_last, checkpoint, block_pm_mode} (sys/include/ztimer.h:365-388), ztimer_ops_t {set, now, cancel, start, stop} (335-360). A convert clock owns lower and a lower_entry timer whose callback is ztimer_handler(self) (sys/ztimer/convert.c:52-68); frac scales set and now with frac_scale (convert_frac.c:48-71, 108-116), shift uses >> and << (convert_shift.c:32-75), muldiv64 uses 64-bit ceiling division (convert_muldiv64.c:34-74). ZTIMER_USEC is the base timer directly at 1 MHz, shift for 250 or 500 kHz, else frac; ZTIMER_MSEC prefers an lptimer or RTT, else frac on the base timer (sys/ztimer/init.c:117-146, 212-247; design text ztimer.h:90-124).

  • Extension: a clock with max_value < UINT32_MAX arms at most max_value/2 and keeps intermediate callbacks even with no timers (core.c:398-422, 434-457); _ztimer_now_extend checkpoints checkpoint += (lower_now - lower_last) mod max_value (core.c:263-283, ztimer.h:161-177, 680-696); ztimer_extend is always pulled in (Makefile.dep:23-26).

  • Ondemand: ztimer_acquire and release count users and call ops->start and stop (core.c:53-113); ztimer_set acquires and subtracts adjust_clock_start (163-174); ztimer_handler releases per fired timer and stops re-arming when nobody needs the clock (474-492); periph_timer stops the hardware right after init (periph_timer.c:83-97, 131-135); converters forward start and stop downwards (convert.c:38-50). Without ondemand, pm_block and pm_unblock bracket the first and last timer (core.c:222-228, 361-367).

  • Timer list: entries store the delta to the predecessor, list.offset is the absolute base (ztimer.h:127-134). _add_entry_to_list walks summing deltas and rebases the successor (core.c:216-253); _ztimer_update_head_offset folds now - list.offset into the head entries (286-327); removal gives the offset back to the successor (329-370). ztimer_handler pops offset-0 entries, fires callbacks, re-checks for expiries that happened during callbacks, then re-arms and yields if not in an ISR (424-502).

  • Wrap: ztimer_now_t is u32 and only differences are meaningful (ztimer.h:304-308, 575-577); the backend set is always relative and clamped to a min tick count (periph_timer.c:44-48; CONFIG_ZTIMER_USEC_MIN 10, sys/include/ztimer/config.h:76-77) to avoid underflow (ztimer.h:180-188).

  • Periodic: ztimer_periodic_t.last; reset computes target = last + interval; offset = target - now; if (offset > interval) offset = 0 and last = target (periodic.c:29-41), drift-free with an immediate catch-up tick when late. ztimer_periodic_wakeup instead resets *last_wakeup = now on a miss, dropping phase (util.c:65-82).

  • Sleep and flags: ztimer_sleep uses a stack MUTEX_INIT_LOCKED unlocked by the callback and subtracts adjust_sleep (util.c:36-63); ztimer_set_timeout_flag sets THREAD_FLAG_TIMEOUT (177-191); ztimer_set_msg and ztimer_msg_receive_timeout (102-130); ztimer_mutex_lock_timeout via mutex_cancel (221-243).

  • Adjust: adjust_set subtracted in every ztimer_set (core.c:186-192), adjust_sleep in sleep, adjust_clock_start on cold start; ztimer_overhead_set and sleep measure after - pre - base (overhead.c:38-63); ztimer_auto_adjust takes the minimum over 100 samples at init (init.c:272-291, 386-405); defaults 0 (config.h:158-184). xtimer remains as a legacy API with ztimer_xtimer_compat (Makefile.dep:14-20); ztimer’s relative-offset list plus clock chain replaces xtimer’s single 64-bit clock.

RIOT: ideas to borrow

  • thread_flags: a 16-bit per-TCB word with ANY, ALL, ONE waits, and reserved bits so IPC and timers compose into one wait (thread_flags.c:25-51; msg.c:57-59; util.c:177-191).

  • The ztimer clock tree: a hardware clock plus converter layers, each a full clock (ztimer.h:90-124; convert.c:52-68); a relative-offset list that uses the full 32-bit range (core.c:216-253); max_value/2 extension with checkpointing (core.c:398-422, 263-283).

  • Overhead compensation adjust_set, adjust_sleep, adjust_clock_start with boot-time auto-calibration (core.c:186-192; init.c:279-291).

  • Ondemand clocks with user counting so timers themselves gate low power (core.c:53-113, 163-174).

  • mutex_cancel as the generic timeout primitive for any blocking lock (mutex.c:263-293; util.c:221-243).

  • A bitcache runqueue with CLZ-ordered bits, STATUS_ON_RUNQUEUE threshold encoding, TCB-in-stack placement, and stack painting (sched.c:84-116, 262-276; thread.c:259-308).

  • Sender-allocated event queues that never drop (event.c:34-49; event.h:33).

RIOT: weaknesses

  • No priority inheritance by default; an opt-in module only, and only for mutex_t (mutex.doc.md:5-9; mutex.c:76-85).

  • A static MAXTHREADS array and linear PID scan (sched.h:94-96; thread.c:313-326); a 32-level cap tied to a u32 bitcache (sched.c:60).

  • msg semantics are easy to misuse: blocking sends can deadlock without a receiver queue, msg_try_send silently drops, and queue sizing is per thread (msg.h:87-110; msg.c:124-129); msg_send_receive abuses the reply buffer for sending (msg.c:275-279).

  • Same-priority threads are cooperative unless sched_round_robin (itself a ztimer client) is enabled (sched.h:36-39; sched_round_robin.c:28-30).

  • 32-bit ztimer_now_t demands wrap discipline from users and continuous clock activity for comparisons (ztimer.h:575-600); ondemand forgets time while the clock is off.

  • Several ISR-misuse paths return silently instead of asserting (thread.c:109-113, 66-70).