cairnquire/docs/adrs/adr-005-storage-strategy.md

ADR-005: Storage Strategy

Status

Accepted

Context

The system needs to store: structured data (users, documents, comments), document content, file attachments, and search indexes. Requirements include: zero external dependencies, backup simplicity, and resistance to data corruption.

Decision

libsql (SQLite) for structured data + content-addressed filesystem for files

Implementation

1. Database (libsql/SQLite):

   • Single file: data/db.sqlite
   • WAL mode for concurrent reads during writes
   • Embedded — no separate process needed
   • Backup: cp db.sqlite db.sqlite.backup (atomic with WAL checkpoint)

2. File Storage (Content-Addressed):

   • Directory: data/files/
   • Layout: data/files/ab/cd/abcdef1234567890abcdef1234567890abcdef12
   • SHA-256 hash as filename (hex encoded)
   • First two characters as subdirectory (for filesystem performance)
   • Immutable: write-once, never modify

3. Document Mapping:

   • Database stores: document_path → current_hash
   • History table stores all previous hashes
   • Attachment metadata stores original filename + content hash

Consequences

Positive

• SQLite is the most tested database in the world (billions of deployments)
• libsql adds replication option without changing the API
• Content-addressed files are naturally deduplicated
• Filesystem storage is trivial to backup (rsync, tar)
• No network dependencies, works offline
• Single directory contains all state — easy to migrate

Negative

• SQLite has write concurrency limits (mitigated by WAL + short transactions)
• Filesystem storage doesn't scale horizontally without shared storage
• No built-in file expiration/garbage collection (must implement orphan cleanup)
• Large attachments can bloat filesystem (mitigated by quotas)

Alternatives Considered

PostgreSQL

• Pros: Excellent concurrency, rich feature set, horizontal scaling
• Cons: Requires separate process, more complex backup, network dependency
• Rejected: Overkill for anticipated scale; SQLite is simpler and faster for read-heavy workloads

Object Storage (S3/MinIO)

• Pros: Unlimited scale, CDN-friendly
• Cons: Network dependency, vendor lock-in, complex local development
• Rejected: Violates zero-dependency and simple backup requirements

SQLite for Everything (BLOBs)

• Pros: Single file for everything
• Cons: Large BLOBs slow down database, harder to serve directly, backup size bloat
• Rejected: Separation of concerns — database for metadata, filesystem for content

References

• SQLite WAL mode: https://sqlite.org/wal.html
• Content-addressable storage: https://git-scm.com/book/en/v2/Git-Internals-Git-Objects
• libsql: https://github.com/tursodatabase/libsql