Skip to content

RFD 0001: Progressive Disclosure Architecture for AI System Instructions

Summary

This RFD documents the architectural decision to use a progressive disclosure pattern for AI system instructions in Samuel. Instead of a monolithic instruction file, the system uses a core CLAUDE.md (~280 lines of essential content) combined with on-demand context loading from the .claude/ directory.

Problem Statement

The original CLAUDE.md was 500 lines, consuming ~4,000 tokens per conversation. Despite preaching token efficiency, it violated its own principles by:

  • Front-loading comprehensive documentation
  • Using narrative paragraphs over bullets
  • Repeating concepts across multiple sections
  • Including theoretical structure (.claude/ folder didn't exist)

Core Tension: How to provide enough guidance for consistent AI behavior without overwhelming the context window?

Requirements

  1. Support both new projects and existing codebases
  2. Work across any tech stack (language-agnostic)
  3. Prioritize atomic tasks over large refactors
  4. Embed rules rather than reference external tools
  5. Keep under 500 lines
  6. Maintain enhanced 4D methodology
  7. Provide opinionated guardrails

Constraints

  • Token budget: ~2,000 tokens for system instructions (vs. 4,000)
  • Must work from Day 1 without .claude/ folder
  • Must scale to complex projects over time

Background

Research Findings

Token Efficiency:

  • Anthropic recommends: 150-200 lines for instruction files
  • Original system: 2.5-3x larger than optimal
  • Best practice: Progressive disclosure (minimal upfront, load context on-demand)

Industry Patterns:

  • Builder.io: Concise CLAUDE.md + @imports for depth
  • Agent-rules repos: Modular .mdc files
  • Most successful implementations: 100-250 lines

Core Insights:

  1. Automation > Documentation (ESLint config > prose about linting)
  2. Constraints > Aspirations ("No functions >50 lines" > "Write clean code")
  3. Right altitude matters (specific boundaries, not implementation steps)
  4. Sub-agent architecture for long tasks
  5. Protected boundaries prevent costly mistakes

Options Considered

Option A: Comprehensive Documentation (Status Quo)

Keep everything in one large CLAUDE.md file.

Pros:

  • Everything documented upfront
  • No need to manage multiple files
  • Single source of truth

Cons:

  • High token cost (~4,000 tokens)
  • Theoretical structure (referenced non-existent files)
  • Violates own efficiency principles

Effort: None (keep existing)

Option B: Radical Minimalism (50-100 lines)

Strip down to absolute minimum, rely on external tooling.

Pros:

  • Ultra-low token cost (~800 tokens)
  • Forces reliance on linters, formatters, CI

Cons:

  • Too little guidance for consistent AI behavior
  • Requires extensive external tooling setup
  • Poor experience for new projects

Effort: Low

Option C: Progressive Disclosure (Chosen)

Core guardrails always loaded, detailed context loaded on-demand from .claude/ directory.

Pros:

  • Token efficient (~1,800-2,000 tokens)
  • Practical from Day 1 (no setup required)
  • Scales naturally with project complexity
  • Self-documenting (AI knows when to create/update files)

Cons:

  • Requires discipline to maintain boundaries
  • More complex mental model

Effort: Medium

Proposal

Chosen: Option C - Progressive Disclosure Architecture

New CLAUDE.md Structure (~280 lines core content)

  1. Core Guardrails (60 lines): 30+ specific, testable rules
  2. Code quality: Function/file length, complexity, documentation
  3. Security: Input validation, parameterized queries, no secrets
  4. Testing: Coverage targets, test naming, edge cases
  5. Git: Conventional commits, atomic changes, branch naming

  6. Performance: No N+1, pagination, caching, bundle sizes

  7. Enhanced 4D Methodology (40 lines):

  8. ATOMIC mode (default): For single-file changes
  9. FEATURE mode: For multi-file features (3-5 subtasks)
  10. COMPLEX mode: For architecture changes (full decomposition)

  11. Clear escalation triggers

  12. SDLC Workflow (80 lines): Planning → Implementation → Validation → Documentation → Commit

  13. Each stage has concrete checkpoints

  14. Opinionated but flexible

  15. Context System (30 lines): How/when to use .claude/ files

  16. Progressive growth philosophy

  17. Loading protocol (session start/during/end)

  18. Language-Specific Overrides (30 lines): TypeScript, Python, Go, Rust

  19. Key tools and patterns per language

  20. Applied when tech stack identified

  21. Supporting Sections (40 lines): Protected boundaries, anti-patterns, emergency procedures

.claude/ Directory Structure

.claude/
├── README.md              # Philosophy and instructions
├── project.md             # Tech stack, architecture (created when decided)
├── patterns.md            # Coding patterns (emerges over time)
├── state.md               # Work tracking (for multi-session work)
├── language-guides/       # Language-specific guardrails (21 languages)
├── framework-guides/      # Framework-specific templates (33 frameworks)
├── workflows/             # Structured workflows (14 workflows)
├── rfd/                   # Private RFDs (early stage)
├── tasks/                 # PRDs and task lists
└── memory/                # Decision logs

Progressive Growth Philosophy

  • Day 1: Only CLAUDE.md + templates exist
  • Week 1: project.md created when architecture decided
  • Month 1: patterns.md emerges from repeated patterns
  • Ongoing: memory/ captures key decisions

Implementation Considerations

Token Budget

Component Lines Tokens (est.)
Core CLAUDE.md ~280 ~1,800
project.md (if loaded) ~50 ~300
Language guide (if loaded) ~150 ~900
Total (typical session) ~300-480 ~2,000-3,000

Loading Protocol

  1. Session Start: Load CLAUDE.md → Check for state.md → Read if exists
  2. During Work: Load language guide based on file extensions (automatic)
  3. Complex Features: Load workflows (PRD, task generation)
  4. Reference Needed: Load patterns.md, project.md, memory/ on-demand

Security Considerations

No direct security implications. Security guardrails are embedded in core CLAUDE.md and are always loaded.

Compatibility

  • Breaking changes: None for end users
  • Migration path: Existing projects can adopt incrementally
  • Backwards compatibility: Works without .claude/ directory

Consequences

Positive

  • Token savings: 55% reduction per conversation (4,000 → ~1,800)
  • Clarity: 30+ specific guardrails vs. general guidelines
  • Usability: Works immediately, no upfront setup required
  • Flexibility: Scales from 100 LOC scripts to 100k LOC systems

Negative

  • Less prescriptive process: Removed phase/milestone structure (too heavy for most projects)
  • More trust in AI: Relies on AI to know when to escalate/document
  • Template maintenance: Need to keep templates updated as best practices evolve

Patterns Discovered

  1. Documentation paradox: More docs ≠ better AI behavior (focused constraints > comprehensive guides)
  2. Context budgeting: Think of tokens like memory (spend wisely, load just-in-time)
  3. Self-referential problem: Instructions about efficiency must be efficient
  4. Progressive growth: Better to grow from minimal than trim from maximal

References

  • Original CLAUDE.md: 500 lines, ~4,000 tokens
  • New CLAUDE.md: ~280 lines core, ~1,800 tokens
  • Research: Anthropic docs, Builder.io patterns, agent-rules repos
  • Decision documented: .claude/memory/2025-01-15-claude-md-redesign.md