Ashish Labs / The Lab62 notes · 3 projects

The Lab

Notes written while building: decisions and what they cost, debugging sessions, experiments and post-mortems. Each one is filed against the system it came from.

20 notes · 16 July 2026 – PresentÆONA personal AI operating system (28 packages at genesis, 34 today) built to think alongside one person, running locally with cloud fallback.Open the file
21 notes · 16 June 2026 – 10 July 2026LEOAn ambitious research exploration into modeling an AI companion as an operating system kernel with 43 cognitive subsystems.Open the file
21 notes · 10 July 2026 – 16 July 2026VANIA 50-year personal AI operating system engineered with zero external infrastructure, strict AST certification, and complete local sovereignty.Open the file

Index

Search titles and summaries

62 of 62 notes

A001The Vacuous Success Bug: When Empty Plans Count as AchievementsA Python truth-value edge case caused every user commitment to be silently filed as a completed achievement within 400 milliseconds, permanently poisoning the proactive follow-up pipeline.ÆON · Debugging Discovery2026-08-11
A002Scoring a Running AI System 3/10: The Behavioral Integrity AuditA forensic audit of Ashi's running production instance — 22 live turns, real Gemini, real Postgres — scored it 3/10 as "TOOL-LIKE," with a 75% false-success rate on action requests and zero proactive contacts in the system's entire lifetime.ÆON · Post-Mortem2026-08-11
A003Benchmarking Six Sub-2B Language Models for On-Device Structured Intent InferenceSix GGUF models (268M–1.7B parameters) were benchmarked identically for structured JSON intent inference on a CPU-only VPS target; Qwen2.5-1.5B-Instruct was the only candidate genuinely doing the task rather than failing safely.ÆON · Experiment2026-08-06
A004Qwen2.5-1.5B's No-Action Ceiling: When Prompt Engineering Stops WorkingA systematic investigation proved that Qwen2.5-1.5B-Instruct's 18.2% no-action accuracy is a genuine capability ceiling, not a prompting or quantization problem: the failure rate is identical at Q4 and Q8, and categories with the most explicit prompt coverage fail at the same 100% rate as categories never mentioned in the prompt at all.ÆON · Experiment2026-08-08
A005Three Scorer Bugs in a Single Benchmark: A Methodology AuditA static analysis of a 123-generation benchmark run found three separate scorer bugs that were systematically biasing metrics in the lenient direction, including one that turned every complete call failure into a credited correct abstention.ÆON · Debugging Discovery2026-08-07
A006A Segfault in llama.cpp from an Abandoned Generation ThreadA cancellation design that signaled the event loop without waiting for the background thread to exit caused a process-level segmentation fault inside llama.cpp, because llama.cpp's native context is not safe for concurrent access.ÆON · Debugging Discovery2026-08-05
A007The 12-Hour Benchmark Hang That Wasn't: Monitoring vs. WorkA benchmark that completed in 22 minutes was mistaken for a 12-hour hang because a separate background notification watcher silently failed to deliver its completion signal — the benchmark itself was fine throughout.ÆON · Debugging Discovery2026-08-06
A008Rate Limit Is Not a Health Signal: Fixing a Provider Cascade FailureDaily quota exhaustion (HTTP 429) was being treated identically to infrastructure failure (service unavailable), which marked the only available provider as UNHEALTHY and caused 22 of 40 turns to fail — even as the provider successfully served 134 requests in the same run.ÆON · Debugging Discovery2026-07-26
A009Removing Ollama After 71 of 82 Soak Failures Were 'Timeout Connecting to Ollama'Ollama was removed entirely (no compatibility layer, no deprecated code path) after a 12-hour soak certification run revealed it was the cause of 86.6% of all turn failures — not by redesigning the embedding layer, but by routing it through the existing `LiteLLMProvider` abstraction to Gemini.ÆON · Architecture Decision2026-07-22
A010When pytest Truncates Your Live Database: The Destructive Test Interlock`uv run pytest packages/ashi-planning/tests` truncated the live production Postgres database mid-session, wiping 3 goals, 26 episodes, and 3 plans while `ashi serve` was running — because `PostgresConnection()` defaults to the developer's live store when `ASHI_DATABASE_URL` is unset.ÆON · Debugging Discovery2026-08-12
A011A Three-Package Dependency Cycle Undetected Until an Invariant AuditA live three-package circular dependency (`ashi-execution → ashi-planning → ashi-reflection → ashi-execution`) existed undetected for weeks because no automated check verified the package dependency graph's acyclicity — until a dedicated architectural invariant audit found it, and a `typing.Protocol` broke it.ÆON · Debugging Discovery2026-08-21
A012The Initiative Note That Was Computed Every Turn and Dropped Before Rendering`initiative_note` — the instruction that would make Ashi initiate a topic or check-in — was computed on every eligible turn by `presence/initiative.py` and then silently dropped one statement before it could reach the prompt, for the system's entire lifetime.ÆON · Debugging Discovery2026-08-11
A013Learning Made Judgment Worse: Strength Reinforcement Promoted a Benchmark FixtureAshi's learning system reinforced memory strength based on retrieval frequency, which promoted a benchmark test fixture called "Meridian" — accessed repeatedly during development testing — to the top of the system's project priority recommendations, above the user's real ongoing work.ÆON · Debugging Discovery2026-08-11
A014World Perception at 24 Microseconds: Designing an LLM-Free Ambient Awareness LayerAshi's world perception layer — which tracks filesystem, git state, and environment across 1,725 resources every 60 seconds — adds 24 µs to the turn-path (a snapshot read) while running the full 574 ms perception pass completely off the turn path in the background.ÆON · Technical Design2026-07-27
A015From 32 Seconds to 5.4 Seconds: Tracing and Eliminating Latency in a 28-Package LLM SystemPhase X.2 attribution tracing reduced first-token latency from 32,836 ms to 5,450 ms p50 by identifying that reasoning alone consumed 51% of total time — and addressing the compounding effects of retry loops, missing fallbacks, and uncapped contributor count.ÆON · Experiment2026-07-26
A016litellm's Undeclared fastapi Dependency: A Hidden Import That Broke Tool Calling`litellm.acompletion` with `tools=` failed for all providers as `APIConnectionError: No module named 'fastapi'`, because litellm conditionally imports fastapi and orjson when handling tool-calling responses — two packages litellm's own `pyproject.toml` does not declare as dependencies.ÆON · Debugging Discovery2026-08-11
A017SPACE BAR Does Nothing: Four Real Bugs Found Only by Actually Running the ProductThe M5 implementation was declared "implemented" with a browser UI showing `ONLINE` and `PRESS SPACE TO TALK` — but ONLINE was hardcoded JSX text that never checked the backend, and the spacebar had zero keydown handler anywhere in the codebase.ÆON · Post-Mortem2026-08-14
A018Architecture Freeze: When to Lock Down a Cognitive Substrate and WhyAfter implementing and certifying 17 packages over milestones M2.5–M4D, an architecture freeze was declared: no new packages, no new EventKind, no new ContributorPhase without a filed ADR — not because development stops, but because M5 (ambient/proactive) is the riskiest milestone and demands the most verified foundation.ÆON · Architecture Decision2026-07-25
A01928 Packages, One Acyclic Graph: Building a Personal AI as a MonorepoAshi is structured as 28 independent Python packages in a uv workspace, organized into a six-layer dependency hierarchy enforced at the build level — so architectural violations are visible at import time, not at runtime months later.ÆON · Technical Design2026-07-25
A020Managing a Free-Tier Gemini Quota Across 21 API KeysAshi operates on free-tier Gemini (500 req/day/project/model) using a credential pool across 21 independent Google Cloud projects, with a quota-reset strategy that accounts for the midnight Pacific reset time — not the developer's local time zone.ÆON · Technical Design2026-07-28
L001Designing a Cognitive Operating System: The LEO Architecture VisionLEO was designed as a Cognitive Operating System — an AI runtime modeled on OS primitives (kernel, scheduler, memory management, interrupt controller) rather than as an AI assistant or agent wrapper.LEO · Architecture Decisionundated
L002Building a Memory Management Unit for AI: The CMMU DesignLEO's CMMU applied OS-style memory management — address translation, protection, coherence, DMA, prefetch, and pressure monitoring — to AI cognitive memory, but 281 violations documented that the rest of the system bypassed it.LEO · Architecture Decisionundated
L003Runtime Architecture Enforcement: Building an Architecture GuardianLEO built a runtime Architecture Guardian that subscribes to system events and validates architectural constraints in real time — but the guardian could only detect violations it was notified about, not the 281 pre-existing static violations in the codebase.LEO · Architecture Decisionundated
L004281 Violations: What Happens When Architecture and Implementation DivergeA Phase 28 audit of the LEO codebase discovered 281 violations of the core CMMU architectural rule across the entire application layer, spanning every major module from API routers to cognition engines to proactive agents.LEO · Post-Mortemundated
L005Preventing AI Vendor Lock-In: The LEO Provider Platform DesignLEO built a Provider Platform that routes AI capability requests (chat, code, reasoning) to registered providers without any component outside the platform knowing which specific provider or model was used.LEO · Architecture Decisionundated
L006When Gemini Returns an Array Instead of an Object: A Semantic Router BugGemini occasionally returned a single-item JSON array instead of a JSON object when routing complex prompts, causing the semantic router to crash with a "list object has no attribute get" error.LEO · Debugging Discovery2026-06-22
L007Cross-Provider Failover Bug: Physical Model Names Lost in AbstractionLEO's multi-provider failover crashed with "model must be provided" errors because the ModelRouter passed abstract configuration parameters instead of concrete physical model names to fallback providers.LEO · Debugging Discoveryundated
L008Replacing Sequential Execution with a DAG: The Cognitive Execution GraphLEO replaced sequential cognitive task execution with a Directed Acyclic Graph (DAG) model to support speculative branching, parallel execution of independent capabilities, and warp engine hyper-scaling.LEO · Architecture Decisionundated
L009A Four-Tier Cognitive Memory Model: Working, Episodic, Semantic, and MoreLEO defined a four-tier cognitive memory model (Working, Recent, Episodic, Semantic) with lifecycle operations (promotion, demotion, consolidation, prefetch, garbage collection) managed by a dedicated Memory Intelligence subsystem.LEO · Architecture Decisionundated
L010Mock Proliferation: When 100+ Stub Implementations Signal Incomplete IntegrationThe LEO backend audit results revealed extensive use of stub and mock implementations throughout the application layer — not just in tests, but in production code — indicating that many components were architecturally shaped but not functionally integrated.LEO · Engineering Noteundated
L011297 Passing, 11 Failing: What the Test Suite Reveals About Architecture MaturityLEO's test suite ran 308 tests with 297 passing and 11 failing; the failures concentrated in event bus contract mismatches (wrong event topic names) and mock infrastructure gaps (add_mock_goal not implemented), revealing a disconnect between subsystem tests and integration-level event contracts.LEO · Engineering Noteundated
L012YAGNI Violations: 36 Single-Implementation Interfaces and the Cost of Over-EngineeringA static analysis audit identified 36 interface abstractions in LEO's codebase that had only one implementation — and 3 with zero implementations — representing YAGNI violations that added 754 lines of unnecessary abstraction code.LEO · Engineering Noteundated
L013Network-on-Chip for AI: Building an OS Interconnect Between Cognitive SubsystemsLEO implemented a Network-on-Chip inspired communication bus for its cognitive subsystems, providing P2P, multicast, and broadcast messaging with traffic management (flow control, congestion control, backpressure), QoS, reliability, and a dead-letter queue.LEO · Architecture Decisionundated
L014Atomic Boot Sequence with Rollback: Designing a Fail-Safe OS StartupLEO's Cognitive OS boot sequence starts services in topological dependency order and performs atomic rollback — stopping all previously started services in reverse order — if any critical service fails to start.LEO · Architecture Decisionundated
L01550 Regression Scenarios and a 9ms Fast Path: How LEO Validated its Routing PipelineLEO ran a 50-scenario regression suite covering intent routing, LangGraph node execution, multi-provider failover, memory pipelines, and edge cases — passing 49 with 1 expected validation error — and documented a 9ms fast path for common greeting patterns.LEO · Experiment2026-06-22
L016Writing an Architecture Constitution: LEO's Approach to Preventing DriftLEO maintained a formal Architecture Constitution (14 numbered documents) establishing immutable principles, layer rules, component ownership, engineering rules, and decision records — an attempt to encode architectural intent in a form that could be enforced both culturally and programmatically.LEO · Architecture Decisionundated
L01743 Cognition Subsystems: The Scope and Structure of a Cognitive AI LayerLEO's cognition layer contained 43 distinct subsystems covering adaptive intelligence, causal reasoning, metacognition, trust modeling, risk assessment, conflict resolution, self-modeling, identity, and work rhythm analysis.LEO · Architecture Decisionundated
L018Using LangGraph for Conversational State Machines in LEOLEO used LangGraph to implement conversational execution as a state machine with distinct nodes (ChatNode, CodingNode, FastChatNode, memory management nodes), routing user requests to the appropriate node based on intent classification.LEO · Engineering Noteundated
L019Silent Deprecation: Discovering a Dead SDK Through Startup WarningsLEO's startup log revealed a `FutureWarning` on every boot: `google.generativeai` (the Gemini Python SDK) was deprecated in favor of `google.genai`, with a migration required before the old package was removed.LEO · Debugging Discovery2026-06-28
L020The 225KB models.py: When a Single File Becomes the Data ModelLEO's `backend/app/models.py` grew to 225KB — a single-file SQLAlchemy model definition containing the entire data model for the system, indicating that the data layer evolved without decomposition into domain modules.LEO · Engineering Noteundated
L021Four Migration Workers: Iterative Attempts at Fixing 281 ViolationsThe LEO root directory contained four successive migration worker scripts (`scratch_migration_worker.py` through `scratch_migration_worker_4.py`), each representing a separate attempt to programmatically migrate the 281 CMMU violation instances.LEO · Build Logundated
V001Why VANI Is Not a Chatbot: Designing a Cognitive Operating SystemVANI is explicitly designed as a continuously running Cognitive Operating System (not a chatbot), governed by a frozen Vision document that prohibits chatbot-style features from becoming the architecture's identity.VANI · Architecture Decision2026-07
V002Architecture Before Implementation: How Phase 0 Produced No Features and Why That Was CorrectPhase 0 delivered a fully operational Cognitive Operating System foundation with zero user-facing cognitive features, deliberately enforcing the principle that infrastructure always precedes functionality.VANI · Engineering Note2026
V003Why I Rejected Every Graph Database and Used SQLite InsteadAfter evaluating Kuzu, Neo4j, Qdrant, and other specialized databases, VANI uses SQLite with property graph tables (nodes + edges) for its knowledge graph — because the graph will contain ~10K–100K nodes across decades, which SQLite handles trivially with zero operational overhead.VANI · Architecture Decision2026-07
V004AI Models Are Fuel, Not Architecture: The Model-Independent Cognitive DesignVANI's AI strategy separates cognitive identity (stored in data) from computational substrate (AI models), so that any model can be swapped without losing personality, memory, knowledge, or behavioral continuity.VANI · Architecture Decision2026-07
V005The Subsystem Pattern: Foundation → Context → State Machine → Engine → Executor → OrchestratorEvery VANI subsystem is internally decomposed into six layers (Foundation, Context, State Machine, Engine, Executor, Orchestrator), providing consistent ownership boundaries and eliminating the most common architectural coupling problems.VANI · Architecture Decision2026
V006Certification vs Testing: Building a System That Validates Its Own ArchitectureVANI uses "certification" (not unit testing) as its primary quality mechanism — an executable framework that validates architectural compliance, subsystem integration, and runtime behavior, including AST-based static analysis of the repository's own structure.VANI · Engineering Note2026
V007Zero Infrastructure: Designing a 50-Year Personal AI with 15 DependenciesVANI's entire persistent state lives in one directory of SQLite files and YAML configs, targeting 15 total external dependencies after 9 phases — because every additional dependency is a 50-year maintenance commitment.VANI · Architecture Decision2026-07
V008Eight ADRs in Phase 1: What Every Service Framework Decision RevealsPhase 1 produced eight formal Architectural Decision Records documenting why VANI chose a unified service framework, centralized configuration, structured logging, unified error hierarchy, kernel-controlled registration, kernel-orchestrated lifecycle, event-driven integration, and constructor-based dependency injection — each with explicit alternatives considered and rejected.VANI · Architecture Decision2026
V009The Guardian: Using AST Analysis to Make Architecture Self-ValidatingVANI includes an "Architecture Guardian" subsystem that continuously validates the repository's own structural correctness at runtime using static AST analysis — enforcing dependency rules, naming conventions, layer boundaries, and documentation requirements.VANI · Engineering Note2026
V010Building an AI That Lasts a Lifetime: The Case for Local-First Personal AIVANI is architected as local-first — data, reasoning, memory, and knowledge remain local by default, with cloud AI as an optional enhancement — because cloud dependency on a 50-year project means your lifelong partner can be shut down by a vendor.VANI · Architecture Decision2026-07
V011Phase 0 Known Limitations: What Was Deliberately Left Out and WhyVANI's Phase 0 documented 14 categories of intentional limitations — things explicitly postponed, not forgotten — establishing a clear boundary between the completed OS foundation and the cognitive capabilities that belong to later phases.VANI · Engineering Note2026
V012The Runtime Refactoring: How Three Design Improvements Were Discovered Mid-ImplementationDuring Phase 0 implementation of the Runtime subsystem, three architectural improvements were discovered in practice: the Context was simplified to passive storage, the State Machine became the sole lifecycle authority, and the Runtime Loop became lifecycle-independent — all mid-implementation corrections.VANI · Engineering Note2026
V013The Composition Root Pattern: Eliminating Startup Duplication Between Production and CertificationVANI resolved duplicated startup logic between production execution and certification by introducing a single Application Composition Root that both paths use — discovered as an improvement during Phase 0 implementation.VANI · Engineering Note2026
V014Rejecting LangChain: Why Direct AI Contracts Are More Maintainable Over DecadesVANI explicitly rejected LangChain (and similar AI orchestration frameworks) in favor of direct API calls through custom port contracts, citing "rapid abstraction churn, heavy dependencies, abstractions that leak."VANI · Architecture Decision2026-07
V015No Placeholder Modules: Engineering Discipline in a Long-Horizon ProjectVANI's README.md establishes three non-negotiable engineering principles that apply regardless of project phase: no placeholder modules, strict static typing enforced by MyPy, and production-grade code from the first commit.VANI · Engineering Note2026
V016The Governance Hierarchy: How Documentation Governs Engineering Decisions in VANIVANI uses a formal governance hierarchy in which every engineering decision has a documented authority level, and lower-authority decisions cannot override higher-authority ones — creating a document-driven engineering constitution.VANI · System Design2026
V017The Event Bus Architecture: Why VANI Services Never Call Each Other DirectlyVANI prohibits direct service-to-service calls — all inter-service communication must go through the Cognitive Interconnect (Event Bus), with publishers unaware of subscribers and subscribers unaware of publishers.VANI · Architecture Decision2026
V018The Runtime Cycle: The Fundamental Execution Unit of an AI OSVANI defines a "Runtime Cycle" as the fundamental execution unit of its cognitive OS — an infinite, deterministic sequence executing infrastructure subsystems in a guaranteed order, completely separated from domain logic.VANI · Architecture Decision2026
V019The Cognitive Scheduler: Decoupling Time from ExecutionVANI's Cognitive Scheduler strictly separates the determination of *when* a task should run from *what* the task does, acting purely as a gatekeeper of time and priority rather than an executor of logic.VANI · Architecture Decision2026
V020The Deterministic Boot Sequence: How an AI OS Starts UpVANI's Kernel orchestrates a strict, deterministic, 10-step boot sequence that validates architectural constraints and resolves dependencies before allowing the Cognitive OS to enter a running state.VANI · Architecture Decision2026
V021The Mission Control Dashboard: Designing UI for an AI Operating SystemVANI explicitly rejects the traditional chat-centric interface in favor of a "Mission Control dashboard" that exposes the system's cognitive status, active goals, memory activity, and resource usage.VANI · System Design2026