← Back to Projects
Project 04 · Flagship AI Infrastructure Control Plane

DevOps Jarvis — Governed AI SRE Control Plane

An autonomous SRE control plane translating natural language intent into deterministic, policy-governed infrastructure mutations. Solves the existential risk of unconstrained AI execution by funneling all proposed actions through Open Policy Agent (OPA) validation, pre-execution dry-run simulation, PostgreSQL distributed lease fencing, and telemetry-aware canary progression.

ACTIVE IN LOCAL DOCKER COMPOSE STAGING
Runtime: 127.0.0.1:8000 · Python 3.14 / FastAPI

Safe Staging Environment & Execution Boundary

DevOps Jarvis is currently active and fully operational in local Docker Compose staging on 127.0.0.1:8000 with a multi-container stack: the FastAPI API control plane, an internal Open Policy Agent (OPA v0.68.0) container on http://opa:8181, and an internal PostgreSQL 17 cluster on postgres:5432.

Security Boundary: An autonomous infrastructure control plane must never be exposed publicly without enterprise SSO, cryptographic hardware tokens, and zero-trust perimeter controls. In staging, strict safety invariants are enforced: JARVIS_ALLOW_LIVE_HOST_ACTUATION="false" and JARVIS_ALLOW_LIVE_CANARY_ACTUATION="false". All production-grade capabilities, policy decisions, and verified test suites are documented below using authentic repository artifacts.

Genuine Application Interface & Control Plane Telemetry

Click any screenshot below to inspect the verified live interface, continuous health intelligence, and blast-radius dependency inspection captured from the running application:

1. Architecture & Governed Execution Pipeline

The foundational design axiom of DevOps Jarvis is Governed Bounded Autonomy. Large Language Models are strictly confined to intent understanding, telemetry summarization, and proposing typed action graphs. Zero raw LLM strings are ever executed directly on infrastructure.

[Operator Intent / Natural Language]
  │
  ▼
1. Intent Understanding & Typed Planner (src/jarvis/planning/)
    • Validates schemas against Pydantic domain models; rejects unknown verbs.
  │
  ▼
2. Engineering Context Graph & Dry-Run Simulation (src/jarvis/execution/simulation.py)
    • Traverses topological dependencies; evaluates blast radius across namespaces.
  │
  ▼
3. Authoritative Policy Engine (OPA Server) (deployments/opa/policies/m1.rego)
    • Evaluates Rego policy: ALLOW | REQUIRE_APPROVAL | BLOCK. Defaults to deny.
  │
  ▼
4. Cryptographic Human-in-the-Loop Gate (src/jarvis/approval/)
    • High-risk actions halt at AWAITING_APPROVAL until signed via operator API.
  │
  ▼
5. PostgreSQL Distributed Leased Execution (src/jarvis/execution/lease_locking.py)
    • Enforces monotonic fencing tokens; guarantees exactly-once execution safety.
  │
  ▼
6. Telemetry-Aware Verification & Rollback (src/jarvis/canary/)
    • Queries Prometheus for error rate / latency regression; aborts and resets on error.

2. Safety Workflow Demonstrations (A through G)

To prove system safety under production failure modes, 7 core workflows were formally implemented and verified against the test harness:

Workflow A: Low-Risk Allowed Operation (Autonomous) test_low_risk_is_allowed PASSED

Read-only inspections or log queries evaluate to RiskLevel.LOW. OPA rule allow if { input.risk_level == "low"; not input.block } triggers an immediate ALLOW, dispatching without human intervention.

Workflow B: High-Risk Approval Workflow (Human Gate) test_high_risk_requires_approval PASSED

Restarting a production service or scaling stateful replicas evaluates to RiskLevel.HIGH. OPA returns require_approval: true. State machine halts at AWAITING_APPROVAL. Execution controller hard-rejects dispatch until signed approval is submitted via POST /api/v1/remediation-plans/{id}/approve.

Workflow C: Destructive Action Blocked by Policy (Fail-Closed) test_critical_risk_requires_approval_via_block PASSED

Commands attempting destructive mutations (dropping tables, deleting namespaces, unvalidated image tags) evaluate to RiskLevel.CRITICAL. OPA rule block if { input.block == true } returns block: true. Execution controller blocks dispatch entirely. If OPA is unreachable, the engine fails closed with OpaConnectionError.

Workflow D: Pre-Execution Dry-Run Simulation test_simulation_with_dependency_context PASSED

Before live actuation, the simulator traverses the Engineering Context Graph to evaluate blast radius, upstream caller degradation, and manifest drift. If predicted blast-radius exceeds safety budgets, the task aborts with actionable diagnostics.

Workflow E: Distributed Lease & Stale Fencing Token Rejection test_m9_postgres_distributed_lease PASSED

Worker 1 acquires an execution lease with fencing_token = 1. Worker 1 pauses (GC pause / network partition). Lease expires after TTL. Worker 2 reclaims the lease with fencing_token = 2. When Worker 1 wakes up and attempts renewal or commit with stale token 1, SQL WHERE fencing_token = :token matches 0 rows. Execution aborts immediately, preventing split-brain corruption.

Workflow F: Crash Recovery & Interrupted Workflow Resume test_startup_recovers_expired_lease PASSED

If a control plane node crashes mid-execution, the periodic reconciliation loop detects tasks in EXECUTING with expired heartbeats. It reclaims the lease with an incremented fencing token, inspects target state to verify partial completion, and safely advances to verification or triggers compensation rollback.

Workflow G: Canary Progression and Telemetry Safety / Abort verify_m8_phase3_prometheus PASSED

Traffic weight shifts incrementally (10% $\to$ 25% $\to$ 50%). At each stage, Prometheus queries compute error rate delta ($\Delta_{err}$) and latency delta ($\Delta_{lat}$). Clean progression yields SAFE_TO_PROGRESS. An injected error surge ($\Delta_{err} \ge 5.0\%$, empirically verified at $+27.27\%$) triggers immediate ABORT_AND_RECOVER, resetting canary weight to 0% with zero downtime.

3. Capability Status Classification Matrix

To maintain absolute engineering credibility, the platform strictly categorizes all architectural capabilities across four verified status tiers:

Subsystem Status Tier Implementation Details & Verification Evidence
Intent & Plan Generation IMPLEMENTED & VERIFIED Pydantic typed domain models, intent parser, plan synthesis. Verified in tests/unit/planning/.
Multi-Stage Orchestration IMPLEMENTED & VERIFIED Deterministic state machine with dependency cascade handling (M10.1G engine, 8 REST endpoints).
Policy Guardrails (OPA) IMPLEMENTED & VERIFIED Authoritative Rego policy (m1.rego), fail-closed defaults. 28/28 unit tests green.
Risk & Approvals Engine IMPLEMENTED & VERIFIED Blast-radius scoring, operator approval API with cryptographic audit lineage.
Execution & Dry-Run IMPLEMENTED & VERIFIED Pre-execution simulation, idempotency key deduplication. 169 unit tests green.
Concurrency & Lease Fencing IMPLEMENTED & VERIFIED PostgreSQL distributed leases with monotonic fencing tokens. Alembic revision c5d6e7f8a9b0.
Telemetry & Canary Analysis IMPLEMENTED & VERIFIED Prometheus comparative telemetry evaluator, read-only host telemetry (M10.2A, 32/32 tests green).
Multi-Cloud Live Actuation IMPLEMENTED (SAFE STAGING) EKS, GKE, and AKS cloud cluster drivers implemented; live cloud mutation intentionally disabled in staging.
Local LLM Inference Node PLANNED On-premise GPU model serving node designed; currently utilizes structured local API heuristic provider.
BGP Anycast Routing Failover DOCUMENTED (UNVERIFIED) Multi-cloud cross-region BGP failover designed in Milestone 3 spec; physical multi-cloud routing hardware not provisioned.

4. Automated Test Verification Baseline

Test coverage is verified through an overall baseline of 2,301 automated tests (0 failures) across exhaustive unit, failure-injection, and integration test suites:

  • Milestone 9 Phase 3 Baseline: 1,419 passing tests (0 regressions), including 23 failure-injection tests and 14 concurrency tests.
  • Authoritative OPA Policy Suite: 28/28 unit tests passed green (0.39s), validating autonomous low-risk allowance, mandatory high-risk approvals, and fail-closed critical blocks.
  • Execution Controller Suite: 169 unit tests passed green, validating idempotency, tenant boundaries, state transitions, and rollback mechanisms.
  • M10.2A Read-Only Telemetry Suite: 32/32 tests passed green, validating zero-mutation host telemetry via Prometheus node_exporter with strict SSRF protection.