# OBS - Observability

Group `OBS`. Design: `docs/specs/SPEC-015-observability-formats.md`; `docs/architecture/04-platform-architecture.md` §7. Related groups: FLT (crash record use), KRN-IRQ (latency metrics), PORT (descriptor contribution, trace clock), SIM (bridge), HW (trace metadata), TEST (benchmark baselines).

OBS-001 to 012 originate in `docs/architecture/04-platform-architecture.md` §7 (009 to 012 added by ADR-034 and R-003). All are restated here as the authoritative copy. New requirements start at 013.

---

## Requirements

### OBS-001  Deferred-format logging
**Statement.** Logging shall support deferred formatting with string interning in a non-loaded section, decodable from the ELF.
**Rationale.** ADR-009: logging cheap enough to leave on in production.
**Status.** Accepted 2026-10-07
**Verification.** Test: `obs/log_deferred_roundtrip` decodes a stream with `tools/logdec`.
**Trace.** SPEC-015 §2

### OBS-002  Compile-time and runtime filtering
**Statement.** Log severity shall be filterable at compile time per module and at runtime per module.
**Rationale.** Zero cost for excluded levels; control in the field for the rest.
**Status.** Accepted 2026-10-07
**Verification.** Footprint test (excluded calls absent); runtime filter test.
**Trace.** SPEC-015 §2.1

### OBS-003  ISR logging bounded, drops counted
**Statement.** ISR logging shall be non-blocking and bounded; drops shall be counted, not hidden.
**Rationale.** A log call must never change timing by blocking.
**Status.** Accepted 2026-10-07
**Verification.** Test: `obs/log_isr_drop_counter` under ring overflow.
**Trace.** SPEC-015 §2.3

### OBS-004  Transports
**Statement.** Log transports shall include UART, SWO/ITM, RTT-like memory channel, USB, retained RAM ring, and network, selected by configuration.
**Rationale.** Products differ in what is wired.
**Status.** Accepted 2026-10-07
**Verification.** Transport tests per reference board.
**Trace.** SPEC-015 §7

### OBS-005  Trace points compile to nothing when off
**Statement.** Trace points shall compile to nothing when disabled and to a bounded store when enabled.
**Rationale.** Zero-cost opt-out; no unbounded buffers.
**Status.** Accepted 2026-10-07
**Verification.** Footprint test with trace classes off; overflow test with `events_discarded`.
**Trace.** SPEC-015 §3.2

### OBS-006  CTF with generated metadata
**Statement.** Trace output shall be CTF with generated metadata; EmbDebug shall not be required to decode it.
**Rationale.** ADR-021.
**Status.** Accepted 2026-10-07
**Verification.** Babeltrace 2 reads a captured stream in CI.
**Trace.** SPEC-015 §3

### OBS-007  Streaming and snapshot
**Statement.** Trace shall support streaming (probe, UART, network) and snapshot (ring buffer dumped on fault or demand) modes.
**Rationale.** Live analysis and post-mortem need both.
**Status.** Accepted 2026-10-07
**Verification.** Test: streaming over the memory channel; snapshot dump through the descriptor.
**Trace.** SPEC-015 §3.3

### OBS-008  Debug descriptor
**Statement.** The image shall export a versioned debug descriptor sufficient to enumerate threads, states, priorities, stacks, owned locks, partitions, and devices without knowledge of private struct layout.
**Rationale.** ADR-011.
**Status.** Accepted 2026-10-07
**Verification.** `tools/descriptor` enumerates everything from a memory dump of the conformance run on each port.
**Trace.** SPEC-015 §5

### OBS-009  Worst-case monitors with opener address
**Statement.** Worst-case interrupt-masked and scheduler-locked intervals shall be recorded per CPU and per thread with the address of the opener, as a compile-time option.
**Rationale.** ADR-034.
**Status.** Accepted 2026-10-07
**Verification.** Test: `obs/monitor_records_pc` with a deliberately long critical section.
**Trace.** SPEC-015 §6

### OBS-010  Monitor thresholds
**Statement.** Thresholds per monitored category shall be configurable to trace, notify, or fault.
**Rationale.** Field-detectable latency violations.
**Status.** Accepted 2026-10-07
**Verification.** Test: each action triggered by threshold crossing.
**Trace.** SPEC-015 §6

### OBS-011  Monitor export
**Statement.** Monitor state shall be exported through the debug descriptor and the statistics API.
**Rationale.** Tools and applications read the same numbers.
**Status.** Accepted 2026-10-07
**Verification.** Descriptor consumer test; `emb_monitor_get` test.
**Trace.** SPEC-015 §5, §6

### OBS-012  Benchmark baseline regression
**Statement.** Benchmark results of a release shall be compared with the previous release's baseline on the same board and configuration; a regression beyond the configured threshold shall fail the release.
**Rationale.** R-003 §4: targets become baselines.
**Status.** Accepted 2026-10-07
**Verification.** Release pipeline test with an injected regression.
**Trace.** SPEC-015 §8; TEST-012

### OBS-013  Log record format
**Statement.** A log record shall be `id timestamp args` with the identifier being the address of the call site's metadata object in a `NOLOAD` dummy region (16 bits when it fits, else 32), typed arguments per the metadata signature, bounded string copies, an in-band `dropped(count)` record, and a periodic synchronization marker; the plain-text backend shall share the call sites.
**Rationale.** Smallest records, host-side formatting, mid-stream resynchronization.
**Status.** Accepted 2026-10-07
**Verification.** Decoder tests including mid-stream join and drop records; AVR plain-text test.
**Trace.** SPEC-015 §2.2 to §2.5

### OBS-014  Trace event catalogue and emission
**Statement.** The 1.0 trace event set shall be the catalogue of SPEC-015 §3.1 defined in `trace_events.yaml`; emitters, enum, and CTF metadata shall be generated from it; events shall be fixed-size with handles as index and generation and threads as index; rings shall be per-CPU CTF streams with 1 KB packets by default and `events_discarded` on overflow; classes shall be selectable at compile time and masked at run time.
**Rationale.** One catalogue keeps every specification's events decodable by one tool.
**Status.** Accepted 2026-10-07
**Verification.** Generator golden test; Babeltrace reads every event class.
**Trace.** SPEC-015 §3.1, §3.2

### OBS-015  Trace clock
**Statement.** Event timestamps shall be 32-bit trace clock ticks with the 64-bit value in packet headers and a CTF clock class declaring the frequency from the hardware description; the tiny profile may use the kernel tick.
**Rationale.** Tools convert to time without guesswork.
**Status.** Accepted 2026-10-07
**Verification.** Metadata check; Trace Compass displays absolute times.
**Trace.** SPEC-015 §3.4

### OBS-016  Crash record format
**Statement.** The crash record shall follow the v1 layout of SPEC-015 §4 (header, build id, uptime, reset reason, fault fields, status and general registers with counts, flight recorder tail, CRC-32), written before any recovery action to the retained section or `.noinit`; a v2 small variant shall serve AVR; partition mini-records shall go to a retained ring when enabled; formats shall be versioned and append-only; the kernel shall expose get and clear on the next boot.
**Rationale.** FLT-001 made concrete and decodable by `tools/crashdec`.
**Status.** Accepted 2026-10-07
**Verification.** Fault-injection test: record written, survives reset, decodes and symbolizes.
**Trace.** SPEC-015 §4

### OBS-017  Descriptor contents and discovery
**Statement.** The descriptor shall contain the sections of SPEC-015 §5 (header, types, thread, scheduler, objects, partitions, rings, monitors, arch), be exported at `emb_debug_descriptor`, carry the magic `"EMBD"`, and be versioned so that one generic plugin serves every kernel version.
**Rationale.** OBS-008 made exact.
**Status.** Accepted 2026-10-07
**Verification.** `tools/descriptor` test on every reference configuration; version compatibility test.
**Trace.** SPEC-015 §5

### OBS-018  Statistics API and monitor categories
**Statement.** The kernel shall offer `emb_stats_cpu_get`, `emb_stats_thread_get`, `emb_stats_object_get`, `emb_monitor_get`, `emb_monitor_set_threshold`, and `emb_stats_reset`, with monitor categories `IRQ_LOCK`, `SCHED_LOCK`, `ISR(vector)`, `THREAD_RUN`, `WORK_HANDLER`; all shall compile out with their options.
**Rationale.** 04 §7.5; ADR-034.
**Status.** Accepted 2026-10-07
**Verification.** API tests; footprint with statistics off.
**Trace.** SPEC-015 §6

### OBS-019  Framing and memory channel
**Statement.** Byte-stream transports shall carry COBS frames with a channel byte and CRC-16 (channels 0 log, 1 trace, 2 crash, 3 shell, 4 to 15 application); the memory channel shall export a control block at `emb_memchan` with an optional RTT-compatible layout.
**Rationale.** One framing for every stream; probe tooling reuse.
**Status.** Accepted 2026-10-07
**Verification.** Decoder resynchronization test; RTT-compatible read by an existing probe tool.
**Trace.** SPEC-015 §7

### OBS-020  Host tools and build-id checks
**Statement.** `tools/logdec`, `tools/tracedec`, `tools/crashdec`, and `tools/descriptor` shall ship with the kernel, refuse streams whose build id mismatches the ELF, and accept every format version they know.
**Rationale.** The decoders are part of the product.
**Status.** Accepted 2026-10-07
**Verification.** Tool tests including a mismatched build id.
**Trace.** SPEC-015 §10

### OBS-021  Profile defaults
**Statement.** Observability defaults per profile shall be those of SPEC-015 §9: plain-text log and small crash record on tiny, deferred log and snapshot trace on base, streaming trace and partition records on isolated and multicore, host pipe decoding on native; the descriptor shall be present in every profile.
**Rationale.** Footprint where it matters, evidence everywhere.
**Status.** Accepted 2026-10-07
**Verification.** Configuration review; footprint tests.
**Trace.** SPEC-015 §9