What is PangeaOS? — Detailed Architecture, Security & Availability Report v1.2.3

Executive Summary

PangeaOS is a research and engineering project designing a Rust-based Ring-0 Single Address Space Operating System (SASOS) for x86-64. Its defining characteristic is executing all OS components at the same hardware privilege level (CPL=0) within a shared virtual address space, while combining hardware memory protection keys (MPK/PKS), verified Software Fault Isolation (SFI), control-flow verification (CFG/CFI), generational Capabilities, IOMMU DMA isolation, and restricted single-instance recovery gates to suppress cross-component interference.

The central shift in v1.2.2 and v1.2.3 is refusing to express security as "halting the entire operating system whenever an anomaly is detected." Malformed inputs, service panics, runaway domain loops, memory quota overruns, IPC saturation, and single-device anomalies are localized in principle. Full-system fail-stop is restricted strictly to cases where the integrity of the kernel core (Domain 0), protection state, exception entries, page tables, or DMA isolation can no longer be maintained.

Because a Ring-0/SASOS places no hardware privilege-ring boundary between components, PangeaOS never claims that MPK or Rust types alone can contain arbitrary malicious Ring-0 native code. Every domain binary must pass SFI, positive instruction allowlist, and control-flow verification before execution—treating the verifier and gates as core components of the Trusted Computing Base (TCB) and strictly distinguishing design targets from hardware-verified guarantees.

---

1. Purpose & Core Philosophy of PangeaOS

1.1 Ring-0 (CPL=0)

Ring-0 is the highest privilege level on x86, granting direct access to page tables, interrupt control, MSRs, and device management. While this enables low-level control and short invocation paths, it means CPU privilege-level transitions cannot serve as the defensive boundary between OS components. PangeaOS v1.2.3 preserves this architectural choice: drivers, kernel services, and internal tasks execute at CPL=0 without introducing CPL=3 user processes, per-domain CR3 switches, or separate address spaces.

1.2 Single Address Space Operating System (SASOS)

In a SASOS, all components inhabit a common virtual address space. Rather than switching independent page tables on every task switch, PangeaOS partitions data regions within the single address space using hardware Protection Keys (K0–K15). A shared address space does not mean unrestricted access; rather, it requires strict runtime management of data region ownership (1TB per-domain virtual arenas), protection keys, gate entry points, mapping generations, and DMA ownership.

1.3 Hardware Consistency of PKU, U/S, and SMEP Finalized in v1.2.3

On x86-64 processors, the PKRU register (CR4.PKE=1) restricts access only on pages marked with U/S=1 (User attribute). However, when CR4.SMEP=1 (Supervisor Mode Execution Prevention) is enabled, fetching instructions at CPL=0 from a U/S=1 page immediately triggers a #PF page fault. In v1.2.3's Profile P-PKU, PangeaOS resolves this hardware constraint by mapping all executable code regions (Kernel .text, gate-asm, verified Domain .text) and Supervisor IST Bootstrap Stacks as U/S=0 (Supervisor, Read-Only W=0), while mapping all key-protected data regions (K0–K15 data, heap, stack, PT, IPC, CMEE) as U/S=1, NX=1 under CR0.WP=1, CR4.SMEP=1, CR4.SMAP=0. This allows PKRU to enforce hardware data isolation in Ring-0 while doubly preventing execution of data pages via NX=1 and SMEP=1.

---

2. Composition of Domain Isolation Layers

Isolation in PangeaOS relies on seven mutually reinforcing layers rather than any single mechanism:

Layer	Primary Role	Inherent Limitation
Rust Types & Ownership	Reduces invalid aliasing and ownership violations in safe code	Does not by itself prevent bugs in unsafe code, raw pointers, FFI, or logic errors
SFI Verification	Restricts domain memory accesses (within 1TB virtual windows) and control flow	Verifier defects or unverified opcodes result in TCB compromise
MPK / PKS (K0–K15)	Enforces per-domain data access on K1–K12, PT (K13), IPC (K14), and CMEE (K15)	Does not block instruction execution; requires sealing all register modification bypasses
Authenticated Gates	Centralizes dangerous updates to PKRU, page tables, and IOMMU state	Requires gate authenticity, golden hash verification, and strict entry validation
CFI / CET	Protects indirect branches and return addresses	Does not solve non-control data corruption or logical DoS
IOMMU & Interrupt Remapping	Restricts device DMA and hardware interrupts	Depends on platform firmware, chipset ACS topology, and physical hardware behavior
Watchdog & Quotas	Bounds infinite loops and resource exhaustion to finite time and volume	Not omnipotent if the underlying hardware or watchdog pathway itself fails

---

3. Why v1.2's "Panic-on-Anomaly" Fault Model Was Problematic

Overusing panic!() to halt the system whenever an anomaly is detected allows an attacker to satisfy the OS halt condition merely by sending crafted inputs. Even without bypassing memory protection, an adversary could deny availability via malformed values, queue saturation, restart loops, or interrupt storms.

A sound OS architecture does not treat all anomalies identically:

• Malformed requests are rejected (L0).
• A fault in a single domain suspends and quarantines only that domain (L1).
• CPU, memory, and IPC resources are capped by hard quotas (L2).
• A runaway device has its Bus Master disabled and is isolated (L3).
• Full-system fail-stop is executed only when kernel core integrity is lost (F1–F3).

---

4. Fault Classification & Concrete Runtime Behavior

4.1 Localized Faults (L0–L3)

Localized faults are anomalies whose origin and blast radius can be identified from trusted state (K0) and contained while preserving all other domains' protection invariants—such as service panics, invalid pointer dereferences inside a domain, verified SFI bounds rejections, CPU budget overruns, IPC index anomalies, memory quota exhaustion, and device interrupt storms. The kernel halts new scheduling for the domain, resets PKRU/PKRS to a safe absolute value, revokes Capabilities by bumping generation numbers, reclaims per-CPU state, disables device BME=0, awaits IOTLB/IR invalidation before touching memory, and either regenerates the domain from a clean image or permanently disables it.

4.2 System-Fatal Faults (F1–F2)

Fatal faults occur when OS-wide trust conditions—Domain 0, trusted gates, exception entries, the Capability ledger, page-table protection, or IOMMU state—are lost, such as suspected tampering with trusted page tables, corrupted recovery continuations, unconfirmed isolation of sensitive data, or uncorrected Machine Checks (#MC) corrupting shared memory.

4.3 Uncertain Cases (F3)

When fault attribution or revocation completion is uncertain, the kernel first isolates the affected CPU, domain, or device and pins all associated physical frames in Quarantine without reuse. If trusted protection mechanisms remain intact and cross-domain impact is contained, the OS continues in Restricted Mode; otherwise, it escalates to full fail-stop.

---

5. Defense Against Denial-of-Service (DoS)

5.1 CPU Runaway

Cooperative yielding cannot stop an infinite loop. PangeaOS uses forced APIC timer preemption and manages per-domain CPU budgets in Domain 0's trusted state. Domains cannot execute CLI, STI, POPFQ, or IRETQ. Hard lockups where maskable interrupts stall are caught by the NMI watchdog, which terminates only the attributed domain whenever trusted Per-CPU records remain valid.

5.2 Memory Exhaustion

Every domain is bounded by page count, heap byte, DMA pin, and concurrent allocation quotas. Exceeding a quota rejects the allocation (Err(QuotaExceeded)) without panicking the global allocator. Domain 0 reserves dedicated emergency memory, recovery stacks, and fixed-size log regions so an attacking domain cannot starve recovery execution.

5.3 IPC Floods & Double-Fetch Prevention

Ring capacity, pending request limits, per-second send rates, descriptor lengths, and per-sender strike counts are enforced. Full queues reject or drop requests rather than expanding without bound. Consumers snapshot untrusted head/tail indices once into local registers before validation, eliminating double-fetch TOCTOU vulnerabilities.

5.4 Restart Loops & Log Floods

Unbounded restarts of a failing domain consume CPU and memory continuously. PangeaOS records restart counts, consecutive failures, and recovery time, applying exponential backoff and permanently isolating domains that exceed thresholds. Logging is rate-limited via fixed-size per-CPU rings and token buckets without synchronous I/O.

5.5 Device DoS

Against MSI/MSI-X floods, IOMMU faults, and interrupt storms, the kernel disables device BME=0, awaits IOMMU/IR invalidation completion, and issues PCIe FLR where supported. Unconfirmed IOTLB flushes forbid frame erasure or reuse.

---

6. Why the Exception Entry Is Critical (Two-Stack IST Pivot)

In a CPL=0 to CPL=0 exception, the CPU does not automatically switch stacks via TSS.RSP0. PangeaOS therefore uses a Supervisor-mapped IST bootstrap stack (U/S=0, immune to PKRU blocking), switches to a bootstrap PKRU mask (PKRU_KERNEL_DEFAULT) before touching K0 state, and immediately pivots RSP onto the kernel private exception stack (k0_priv_rsp in K0) before invoking the Rust dispatcher—preventing nested exceptions from overwriting IST 1. Saved PKRU values and recovery continuations (SchedulerContinuation) are stored in K0 Per-CPU storage rather than domain-writable stacks. Furthermore, because panic = "abort" in #![no_std] Rust has signature -> !, localized panic recovery uses trusted assembly (gate_abort_to_scheduler) to perform a non-local jump to the scheduler continuation.

---

7. Consistency Between a Single Address Space and IPC

Under a single shared page table, PTE permission bits cannot simultaneously grant Read-Write to a sender and Read-Only to a receiver on the same virtual page. PangeaOS implements low-risk Class S (boundary verification and copy by a Domain 0 broker using K14 control words) first, and introduces Class B (Protection-Key re-keyed zero-copy) only with full cross-core PKRU and TLB shootdown synchronization, keeping buffers inaccessible to both parties during ownership transfer.

---

8. Memory Erasure, Page Tables & S-KARL

The contradiction between permanently read-only page tables and runtime PTE/Guard Page updates is resolved by the PT Update Gate: page tables reside in K13 (normally write-disabled) and are modified exclusively inside a single gate while keeping CR0.WP=1, followed by all-core TLB shootdown acknowledgement. Similarly, CMEE (cmee) manages software zeroization and PFN state transitions prior to frame reuse without falsely claiming that MOVNTI or CLFLUSHOPT alone defeat physical DRAM Cold Boot attacks (which require hardware TME/SME). S-KARL function shuffling provides layout diversity and is re-verified by bincheck-core after relocation.

---

9. Strengths & Architectural Trade-Offs

• Strengths: Eliminates address-space context switches between tasks; combines MPK data domains, Capabilities, SFI, CFI, and IOMMU to shrink bug blast radius; embeds forced preemption and quotas into the core architecture to prevent service DoS from causing global panics; pins unconfirmed frames rather than risking unsafe reuse.
• Trade-Offs: Because all domains run in Ring-0, the SFI verifier, instruction decoder, gates, and interrupt entries carry high TCB responsibility; SMP synchronization of shared page tables and Protection Keys is complex; SFI verification must inspect final machine code rather than relying on Rust source compilation alone; hardware features require physical validation beyond QEMU; and necessary fail-stop conditions remain when core TCB integrity is lost.

---

10. On "Completely Resolving Technical Contradictions"

Architectural contradictions are resolved by replacing mutually incompatible promises with explicit, implementable protocols—specifically separating local exceptions from global halts, replacing TSS.RSP0 assumptions with Two-Stack Supervisor IST pivots, storing PKRU/continuations in K0 Per-CPU state, masking XRSTOR PKRU bit 9, replacing raw byte searches with instruction-boundary decoding, separating Class S and Class B IPC, introducing the K13 PT Update Gate, finalizing Profile P-PKU (U/S=0 code vs. U/S=1, NX=1 data under SMEP=1, SMAP=0), pinning frames on TLB/IOTLB timeouts, and distinguishing MPK/SFI scope from arbitrary Ring-0 code execution. However, updating specification documents alone does not prove the absence of implementation bugs; verification is complete only when proven across actual machine code, concurrent SMP execution, and hardware fault injection.

---

11. Conclusion

The goal of PangeaOS v1.2.3 is to place localized fault containment and availability at the center of a Ring-0/SASOS architecture. Handling a fault means transitioning to a trusted stack, attributing the cause from K0 state, enforcing resource quotas, revoking capabilities, reclaiming stale access across all CPUs and DMA engines, and bounding restart rates. The governing principle is clear: never halt the entire operating system for a localizable fault, and never pretend it is safe to continue when the integrity of a trusted boundary cannot be confirmed.

---

References

• Intel Software Developer Manuals: https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html
• UEFI Specification 2.10 — ExitBootServices(): https://uefi.org/specs/UEFI/2.10/07_Services_Boot_Services.html
• The Embedded Rust Book — Panicking: https://doc.rust-lang.org/beta/embedded-book/start/panicking.html
• Linux kernel docs — Lockup watchdogs: https://cdn.kernel.org/doc/html/latest/admin-guide/lockup-watchdogs.html