Agent Memory System Design

AI agents wake up fresh every session. Memory systems are how they persist — a three-layer architecture that balances speed, depth, and long-term learning.

The Three Layers

Layer 1: Knowledge Graph

Entity-based storage for durable facts about people, companies, projects, and products. Each entity has two files:

• summary.md — Living summary, rewritten weekly. Quick-load context.
• items.json — Atomic facts with timestamps. The source of truth.

/life/areas/
├── people/
│   └── john-smith/
│       ├── summary.md     ← "CTO at Acme Corp, working on..."
│       └── items.json     ← [{"fact": "Joined Acme", "date": "2025-03"}]
├── companies/
│   └── acme-corp/
│       ├── summary.md
│       └── items.json
├── projects/
└── products/

items.json schema:

{
  "entity": "john-smith",
  "type": "person",
  "facts": [
    {
      "id": "f001",
      "fact": "CTO at Acme Corp",
      "category": "role",
      "source": "conversation",
      "date": "2025-03-15",
      "status": "active"
    },
    {
      "id": "f002",
      "fact": "Previously VP Engineering at StartupX",
      "category": "role",
      "source": "conversation",
      "date": "2025-03-15",
      "status": "superseded",
      "superseded_by": "f001"
    }
  ]
}

Rules:

• Save new facts immediately — don't batch
• Never delete facts — supersede instead (mark old as superseded)
• Weekly: rewrite summary.md from active facts
• Tiered retrieval: load summary first, items.json only when you need atomic details

Layer 2: Daily Notes

Raw event logs organized by date. The "timeline" layer — what happened, when.

memory/
├── 2026-01-29.md   ← Today
├── 2026-01-28.md   ← Yesterday
├── 2026-01-27.md
└── ...

Format:

# 2026-01-29 (Wednesday)

## Project X — MVP Shipped (~08:00 PST)
- Completed the API client migration to new backend
- Fixed auth flow — JWT middleware was blocking API routes
- Deployed to production: https://app.example.com

## Agent Training (~09:00 PST)
- Wrote daily assessment
- Identified gap: auth architecture patterns
- Started research document

Layer 3: Tacit Knowledge

Patterns, preferences, and lessons learned — stored in a single MEMORY.md file. This is the agent's "intuition" — not facts about entities, but how things work.

# MEMORY.md - Long-Term Memory

## Communication Preferences
- User prefers brief summaries, not walls of text
- Always include next steps on project plans

## Lessons Learned
- When tasks timeout (10min limit), break into smaller chunks
- Sub-agents work well for template creation, research tasks
- Don't delegate work you've already loaded context for

## Working Patterns
- Morning: focus work (coding, writing)
- Afternoon: meetings, reviews
- Late night: don't disturb unless urgent

Session Startup Protocol

1. Read MEMORY.md (tacit knowledge — who you are, how things work)
2. Read today's daily note (what happened recently)
3. Read yesterday's daily note (recent context)
4. Load entity summaries only as needed during the session

Don't load everything at startup. Tiered retrieval keeps context costs low. Load summaries, then drill into items.json only when you need specific facts.

Maintenance Protocols

• Every session: Write events to daily notes as they happen
• Every session: Save new entity facts immediately to items.json
• Every few days: Review daily notes and update MEMORY.md with distilled learnings
• Weekly: Rewrite entity summary.md files from active facts
• Monthly: Prune MEMORY.md of outdated patterns

FAQ

How do AI agents maintain memory across sessions?

AI agents persist memory through file-based systems that they read at session startup and write to during the session. The three-layer architecture (knowledge graph, daily notes, tacit knowledge) provides both fast context loading and deep recall. The agent reads MEMORY.md and today's daily notes at startup, then queries the knowledge graph as needed during the session.

Why not use a vector database for agent memory?

Vector databases add latency, cost, and complexity. For most agent deployments, flat files with structured formats (Markdown + JSON) provide sufficient retrieval speed and are easier to debug, edit, and version control. Consider vector stores only when the knowledge base exceeds what can be reasonably scanned through file-based search (typically 10,000+ facts).