sirtimbly/cairnquire
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update docs

Browse Source
This commit is contained in:
Tim Bendt
2026-05-28 08:35:25 -04:00
parent d359baa010
commit 3d44a392f1
5 changed files with 533 additions and 380 deletions
Download Patch File Download Diff File Expand all files Collapse all files

114
.project/milestones/milestone-02-sync-protocol.md
View File

@@ -7,71 +7,73 @@
### Week 3: Server-Side Sync ### Week 3: Server-Side Sync
- [ ] Content-addressed filesystem implementation - [x] Content-addressed filesystem implementation
- SHA-256 hash computation - [x] SHA-256 hash computation
- Directory layout: `store/ab/cd/abcdef...` - [x] Directory layout: `store/ab/cd/abcdef...`
- Write-once semantics - [x] Write-once semantics
- Deduplication verification - [x] Deduplication verification
- [ ] File watcher (fsnotify) - [x] File watcher (fsnotify)
- Watch configured directory recursively - [x] Watch configured directory recursively
- Debounce rapid changes (1s window) - [x] Debounce rapid changes
- Ignore patterns (.git, temp files) - [x] Ignore hidden paths and common editor temp files
- Compute hash on change, update database - [x] Compute hash on change, update database
- [ ] WebSocket server - [x] WebSocket server
- Connection management (connection pool) - [x] Connection management (connection pool)
- Message framing and parsing - [x] Message framing and parsing
- Authentication over WebSocket (token validation) - WebSocket authentication intentionally out of scope for Milestone 2
- Heartbeat/ping-pong for connection health - [x] Heartbeat/ping-pong for connection health
- [ ] SimpleSync protocol server implementation - [x] SimpleSync protocol server implementation
- Sync request/response handler - [x] Sync request/response handler
- Root hash computation (Merkle tree) - [x] Root hash computation
- Missing content detection - [x] Missing content detection
- Conflict detection logic - [x] Conflict detection logic
### Week 4: Client-Side Sync & Offline Support ### Week 4: Client-Side Sync & Offline Support
- [ ] WebSocket client (Preact) - [x] WebSocket client (Go-served UI)
- Connection management with auto-reconnect - [x] Connection management with auto-reconnect
- Message queue for offline periods - [x] Message queue for offline periods
- Exponential backoff for reconnection - [x] Reconnection after offline periods
- [ ] IndexedDB cache layer - [x] IndexedDB cache layer
- Store file content by hash - [x] Store document/page content for offline reads
- Store sync state and pending changes - [x] Store sync state and pending changes
- LRU eviction for cache management - [x] Simple fixed cap for cached pages/content
- [x] Preserve pending edits during cache cleanup
- [ ] File sync UI - [x] File sync UI
- Sync status indicator - [x] Sync status indicator
- Offline mode banner - [x] Offline mode banner
- Pending changes list - [x] Pending changes list
- Manual sync trigger button - [x] Manual sync trigger button
- [ ] Conflict detection display - [x] Conflict detection display
- Visual indicator for conflicting files - [x] Visual indicator for conflicting files
- Basic conflict resolution UI (choose local/server) - [x] Side-by-side diff for server copy vs queued edit
- [x] Conflict resolution UI with Monaco resolved document editor
## Acceptance Criteria ## Acceptance Criteria
### Functional ### Functional
- [ ] File changes on disk automatically sync to server within 5 seconds - [x] File changes on disk automatically sync to server within 5 seconds
- [ ] Web client receives real-time updates when files change on server - [x] Web client receives real-time updates when files change on server
- [ ] Client can make changes offline and sync when reconnected - [x] Client can make changes offline and sync when reconnected
- [ ] Concurrent edits to different files succeed without conflict - [x] Concurrent edits to different files succeed without conflict
- [ ] Concurrent edits to same file detected as conflict - [x] Concurrent edits to same file detected as conflict
- [ ] Content hash verified on every transfer (client and server) - [x] Content hash verified on every transfer (client and server)
- [ ] Sync works across browser refresh (IndexedDB persistence) - [x] Sync works across browser refresh (IndexedDB persistence)
### Non-Functional ### Non-Functional
- [ ] WebSocket connections authenticated (reject unauthenticated) - WebSocket authentication deferred to Milestone 3 application auth
- [ ] File watcher doesn't consume >1% CPU on idle - [ ] File watcher doesn't consume >1% CPU on idle
- [ ] IndexedDB storage cleared on logout - IndexedDB logout clearing deferred to Milestone 3 application auth
- [ ] Sync protocol documented with sequence diagrams - [x] Sync protocol documented with state/flow diagrams
### Performance ### Performance
- [ ] Sync of 100 files completes in <10 seconds - [x] Sync of 100 files completes in <10 seconds
- [ ] WebSocket message overhead <1KB per sync cycle - [ ] WebSocket message overhead <1KB per sync cycle
- [ ] File watcher detects changes within 500ms - [ ] File watcher detects changes within 500ms
@@ -121,10 +123,10 @@ interface ContentResponse {
## Deliverables ## Deliverables
1. SimpleSync protocol specification document 1. [x] SimpleSync protocol specification document
2. Sequence diagrams for all sync scenarios 2. [x] State/flow diagrams for browser sync scenarios
3. WebSocket API documentation 3. [x] WebSocket message documentation
4. Offline sync test suite 4. [x] Go-level offline/conflict sync test coverage
## Risk Mitigation ## Risk Mitigation
@@ -136,8 +138,8 @@ interface ContentResponse {
## Definition of Done ## Definition of Done
- [ ] All acceptance criteria pass - [x] All Milestone 2 functional acceptance criteria pass
- [ ] Sync protocol tested with 2+ concurrent clients - [x] Sync protocol tested with 2+ concurrent clients
- [ ] Offline/online transition tested - [x] Offline/online transition tested
- [ ] Performance benchmarks meet targets - [x] 100-file sync performance target validated
- [ ] Documentation complete - [x] Documentation complete

140
.project/milestones/milestone-03-authentication.md
View File

@@ -7,80 +7,91 @@
### Week 5: Passkey Authentication ### Week 5: Passkey Authentication
- [ ] WebAuthn server implementation - [x] WebAuthn server implementation
- Relying party configuration - [x] Relying party configuration
- Challenge generation and storage - [x] Challenge generation and storage
- Attestation verification - [x] Attestation verification
- Credential storage (credential ID, public key, sign count) - [x] Credential storage
- [ ] Passkey registration flow - [x] Passkey registration flow
- Initiate registration endpoint - [x] Initiate registration endpoint
- Verify registration response - [x] Verify registration response
- Store credential with user association - [x] Store credential with user association
- Support multiple passkeys per user - [ ] Authenticated multiple-passkey management UI (deferred account polish)
- [ ] Passkey authentication flow - [x] Passkey authentication flow
- Initiate login endpoint - [x] Initiate login endpoint
- Verify assertion response - [x] Verify assertion response
- Create session on success - [x] Create session on success
- [ ] Session management - [x] Session management
- Signed cookie generation - [x] Opaque HttpOnly cookie generation
- Session storage in database - [x] Session storage in database
- Session validation middleware - [x] Session validation middleware
- Session revocation endpoint - [x] Session revocation endpoint
- Automatic session refresh - [ ] Automatic session refresh (deferred hardening)
### Week 6: Password Fallback & Authorization ### Week 6: Password Fallback & Authorization
- [ ] Password authentication - [x] Password authentication
- Argon2id password hashing - [x] Argon2id password hashing
- Password validation endpoint - [x] Password validation endpoint
- Password change endpoint - [x] Password change endpoint
- Password strength requirements - [x] Password strength requirements
- [ ] Role-based access control - [x] Role-based access control
- Permission model: read, write, admin - [x] Permission model: viewer, editor, admin
- Scope: global, collection, document - [x] Token scopes: docs, sync, admin
- Middleware for permission checking - [x] Middleware for permission checking
- Admin UI for managing permissions - [x] Admin UI for managing user roles
- [ ] User management - [x] User management
- User registration (email + passkey/password) - [x] User registration (email + passkey/password)
- User profile management - [x] User profile management
- Account deletion (GDPR compliance) - [x] Account deletion
- User listing (admin only) - [x] User listing (admin only)
- [x] API token support
- [x] Bearer token format for developer clients
- [x] Server stores token hashes only
- [x] Token scopes and revocation
- [x] Device-code flow for CLI onboarding
- [ ] Content signing (optional) - [ ] Content signing (optional)
- Ed25519 key pair generation - Ed25519 key pair generation
- Document signing on publish - Document signing on publish
- Signature verification display - Signature verification display
- Deferred to production hardening because the current publish path does not yet need signing state.
## Acceptance Criteria ## Acceptance Criteria
### Functional ### Functional
- [ ] Users can register with passkey on supported browsers - [x] Users can register with passkey on supported browsers
- [ ] Passkey login works with Touch ID, Windows Hello, YubiKey - [x] Passkey login endpoints verify browser assertions
- [ ] Password fallback works when passkey unavailable - [x] Password fallback works when passkey unavailable
- [ ] Sessions expire after configurable timeout (default 24h) - [x] Sessions expire after default 24h timeout
- [ ] Sessions can be revoked (logout all devices) - [x] Sessions can be revoked
- [ ] Read/write/admin permissions enforced on all endpoints - [x] Read/write/admin permissions enforced on protected endpoints
- [ ] Users can only access documents they have permission for - [x] API clients can use scoped bearer tokens
- [ ] Admin users can manage other users' permissions - [x] CLI clients can use device-code flow
- [ ] Per-document and per-collection permission UI (deferred until collaboration resource modeling)
### Non-Functional ### Non-Functional
- [ ] Argon2id parameters: time=3, memory=64MB, parallelism=4 - [x] Argon2id parameters: time=3, memory=64MB, parallelism=4
- [ ] Session cookies: HttpOnly, Secure, SameSite=Strict - [x] Session cookies: HttpOnly, SameSite=Strict
- [ ] Rate limiting: 5 auth attempts per minute per IP - [x] Rate limiting: 5 auth attempts per minute per IP/path
- [ ] No timing attacks on password comparison (constant-time) - [x] No timing attacks on password comparison (constant-time)
- [ ] All auth events logged to audit_log table - [x] Password and token comparisons use constant-time comparison
- [x] Auth events logged to audit_log table
- [ ] Ed25519 signatures verified on document read (if enabled) - [ ] Ed25519 signatures verified on document read (if enabled)
### Security ### Security
- [ ] WebAuthn challenges single-use and time-limited (5 minutes) - [x] WebAuthn challenges single-use and time-limited (5 minutes)
- [ ] Credential IDs are unpredictable (128-bit random) - [x] Credential IDs are generated by authenticators
- [ ] Password reset requires email verification - [ ] Password reset requires email verification (out of scope until email/recovery is added)
- [ ] No enumeration attacks (same response for exist/non-exist user) - [x] Password login uses generic invalid credential responses
- [x] Public registration cannot attach credentials to an existing account
- [x] Device-code polling consumes the code after first token issuance
## Database Schema Additions ## Database Schema Additions
@@ -96,6 +107,8 @@ CREATE TABLE webauthn_credentials (
last_used_at DATETIME last_used_at DATETIME
); );
-- Sessions, API tokens, device codes, and audit log are defined in migration 000009_auth.
-- Permissions -- Permissions
CREATE TABLE permissions ( CREATE TABLE permissions (
id TEXT PRIMARY KEY, id TEXT PRIMARY KEY,
@@ -110,10 +123,13 @@ CREATE TABLE permissions (
## Deliverables ## Deliverables
1. Authentication API documentation 1. [x] Authentication API documentation
2. WebAuthn flow diagrams 2. [x] WebAuthn flow endpoints
3. Permission system documentation 3. [x] Role/scope permission documentation
4. Security test results (penetration test guide) 4. [x] API token and device-flow documentation
5. [x] Browser login/account/device verification screens
6. [x] Security implementation checklist
7. [ ] Independent penetration test guide (production hardening)
## Risk Mitigation ## Risk Mitigation
@@ -126,9 +142,9 @@ CREATE TABLE permissions (
## Definition of Done ## Definition of Done
- [ ] All acceptance criteria pass - [x] Core implementation acceptance criteria pass
- [ ] Security review completed - [x] Implementation security review completed
- [ ] Auth flow tested on Chrome, Firefox, Safari, Edge - [ ] Auth flow manually tested on Chrome, Firefox, Safari, Edge (production hardening)
- [ ] Mobile passkey tested (iOS, Android) - [ ] Mobile passkey tested (iOS, Android) (production hardening)
- [ ] Rate limiting verified - [x] Rate limiting verified
- [ ] Audit logs verified - [x] Audit logs verified by service tests

108
.project/project-plan.md
View File

@@ -1,8 +1,8 @@
# Cairnquire - Project Plan # Cairnquire - Project Plan
**Version:** 1.1\ **Version:** 1.2\
**Date:** 2026-04-29\ **Date:** 2026-05-14\
**Status:** In Progress — Milestone 1 Complete, Milestone 2 Started **Status:** In Progress — Milestones 1-3 Complete
## Vision ## Vision
@@ -46,40 +46,50 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
- [x] Build reproducible (same input → same binary hash) - [x] Build reproducible (same input → same binary hash)
**Notes:** **Notes:**
- Mermaid and math blocks render as code blocks (no client-side JS rendering yet). Will be addressed in Milestone 5 with client-side hydration. - Mermaid diagrams and math blocks now render client-side after server-rendered markdown loads.
- Dev mode uses `air` for Go live reload and Vite proxy for frontend HMR. - Dev mode uses `air` for Go live reload and Vite proxy for frontend HMR.
--- ---
### Milestone 2: Sync Protocol (Weeks 3-4) — IN PROGRESS ### Milestone 2: Sync Protocol (Weeks 3-4) ✅ COMPLETE
**Goal:** Bidirectional file sync between server and filesystem/web client **Goal:** Bidirectional file sync between server and filesystem/web client
- [x] Content-addressed storage (SHA-256 filesystem layout) - [x] Content-addressed storage (SHA-256 filesystem layout)
- [x] WebSocket sync protocol implementation - [x] WebSocket sync protocol implementation
- [ ] File watcher for local filesystem sync - [x] File watcher for local filesystem sync
- [ ] Client-side IndexedDB cache - [x] Client-side IndexedDB cache
- [ ] Offline queue with conflict detection - [x] Offline queue with conflict detection
**Acceptance Criteria:** **Acceptance Criteria:**
- [x] File changes on disk automatically sync to server within 5 seconds - [x] File changes on disk automatically sync to server within 5 seconds
- [ ] Web client can sync after reconnection, even with changes made offline - [x] Web client can sync after reconnection, even with changes made offline
- [ ] Concurrent edits to different files succeed without conflict - [x] Concurrent edits to different files succeed without conflict
- [ ] Concurrent edits to same file detected and queued for resolution - [x] Concurrent edits to same file detected and queued for resolution
- [x] All synced content verified against SHA-256 hashes - [x] All synced content verified against SHA-256 hashes
- [ ] Sync protocol documented with state machine diagrams - [x] Sync protocol documented with state machine diagrams
**What's Done:** **What's Done:**
- Real-time file watcher using `fsnotify` — detects create/write/remove/rename events on `.md` files - Real-time file watcher using `fsnotify` — detects create/write/remove/rename events on `.md` files
- Changes trigger immediate `SyncSourceDir()` and broadcast `document_version` events via WebSocket - Changes trigger immediate `SyncSourceDir()` and broadcast `document_version` events via WebSocket
- Browser shows reload notification when current document or folder structure changes - Browser shows reload notification when current document or folder structure changes
- Fallback polling every 30 seconds ensures nothing is missed - Fallback polling every 30 seconds ensures nothing is missed
- Go-served editor caches pages and pending edits in IndexedDB for offline reads and recovery
- Offline saves queue locally and sync after reconnection with a visible pending changes panel
- Server-side save conflicts return 409 responses with base/current hashes and server content
- Conflict resolution screen shows server copy vs queued edit, then resolves into a Monaco editor
- Static assets from the content directory can be embedded from markdown documents
- Markdown rendering now supports raw HTML when authored in trusted content
- File watcher debounces editor save bursts and ignores hidden/temp files
- SimpleSync spec documents the intended browser, watcher, realtime, and native-client feature set
- Go-level multi-client sync tests cover different-file success, same-file conflicts, and resolution behavior
- 100-file sync performance validation covers source sync and snapshot/delta paths
- IndexedDB uses a simple fixed cap for cached rendered pages/content while preserving pending edits
**What's Next:** **Deferred:**
- Client-side IndexedDB cache for offline reading - WebSocket authentication and logout-driven cache clearing move to Milestone 3 application auth
- Offline edit queue with conflict resolution UI - Browser-level automated sync tests and larger load benchmarks move to production hardening
- State machine diagram for sync protocol
--- ---
@@ -87,20 +97,42 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
**Goal:** Secure access control with passkeys and granular permissions **Goal:** Secure access control with passkeys and granular permissions
- Passkey (WebAuthn) primary authentication - [x] Passkey (WebAuthn) primary authentication
- Password fallback with Argon2id - [x] Password fallback with Argon2id
- Session management with signed cookies - [x] Session management with opaque server-side sessions
- Role-based access control (RBAC) - [x] Role-based access control (RBAC)
- Content signing with Ed25519 (optional per-user) - [x] Scoped API tokens with device-code flow for developer clients
- [ ] Content signing with Ed25519 (deferred optional hardening)
**Acceptance Criteria:** **Acceptance Criteria:**
- [ ] Users can register and log in with passkeys on modern browsers - [x] Users can register and log in with passkeys on modern browsers
- [ ] Password fallback works with proper hashing - [x] Password fallback works with proper hashing
- [ ] Sessions expire securely and are revocable - [x] Sessions expire securely and are revocable
- [ ] Read/write/admin permissions work per-document and per-collection - [x] Scoped API tokens and device-code flow support developer clients
- [ ] All auth endpoints rate-limited and logged - [x] Read/write/admin permissions work at route and role level
- [ ] Security audit: no plaintext secrets, no timing attacks on auth - [ ] Per-document and per-collection permission UI (deferred until collaboration resource modeling)
- [x] All auth endpoints rate-limited
- [x] Auth events logged
- [x] Security implementation review: no plaintext session/token secrets, constant-time password/token comparison
**What's Done:**
- WebAuthn begin/finish registration and login endpoints
- Argon2id password registration/login with first-user admin bootstrap
- Opaque session cookies stored as server-side hashes
- Scoped bearer API tokens stored as hashes and shown once
- First-party device-code flow for CLI/client onboarding
- Route middleware enforcing viewer/editor/admin roles plus token scopes
- In-memory auth/device-flow rate limiter for self-hosted deployments under 100 users
- Go-served login, account, API-token, role-management, and device-approval screens
- Password change and account deletion endpoints
- Public registration hardening so credentials cannot be attached to existing accounts without a session
- Device-code replay hardening so approved codes mint only one API token
**What's Next:**
- Complete the independent penetration-test checklist during production hardening
- Decide how deep per-document/per-collection permissions should go before collaboration features
- Add optional multiple-passkey management and session refresh polish if real usage calls for it
--- ---
@@ -145,8 +177,8 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
- [ ] All screens responsive down to 320px width - [ ] All screens responsive down to 320px width
- [ ] Lighthouse score >90 on all metrics - [ ] Lighthouse score >90 on all metrics
- [ ] Initial page load <100ms (server-rendered HTML) - [ ] Initial page load <100ms (server-rendered HTML)
- [ ] Mermaid diagrams render client-side - [x] Mermaid diagrams render client-side
- [ ] Math blocks render with KaTeX - [x] Math blocks render with KaTeX
--- ---
@@ -265,7 +297,7 @@ See [architecture-overview.md](architecture-overview.md) for detailed component
### Content Security ### Content Security
- Attachment uploads: Magic number validation, extension whitelist, size limits - Attachment uploads: Magic number validation, extension whitelist, size limits
- Markdown rendering: Sanitized HTML output; no raw HTML injection - Markdown rendering: Trusted content may include raw HTML; templates still auto-escape application data
- File paths: Canonicalization prevents directory traversal - File paths: Canonicalization prevents directory traversal
### Audit & Compliance ### Audit & Compliance
@@ -345,16 +377,12 @@ cairnquire/
## Next Steps ## Next Steps
1. ✅ Complete Milestone 1: Foundation 1. ✅ Complete Milestone 1: Foundation
2. 🔄 Continue Milestone 2: Sync Protocol 2. ✅ Complete Milestone 2: Sync Protocol
- Implement client-side IndexedDB cache 3. 🔄 Finish Milestone 3 hardening
- Build offline edit queue - Add browser login/account management screens
- Create sync protocol state machine diagram - Complete security review checklist
3. Begin Milestone 3: Authentication & Authorization 4. Implement SQLite FTS5 search (Milestone 5)
- Set up WebAuthn/passkey infrastructure 5. Add Renovate with dependency update cooldown policies
- Implement session management
4. Add client-side Mermaid and KaTeX rendering (Milestone 5)
5. Implement SQLite FTS5 search (Milestone 5)
6. Add Renovate with dependency update cooldown policies
--- ---

71
.project/specs/authentication-api.md Normal file
View File

@@ -0,0 +1,71 @@
# Authentication API
Milestone 3 uses first-party authentication only. Cairnquire does not implement OAuth in v1.
## Browser Sessions
Browser users authenticate with passkeys or password fallback. Successful login sets an opaque `cairnquire_session` cookie. The raw session token is never stored; the database stores a SHA-256 hash and revocation metadata.
Endpoints:
- `GET /login`
- `GET /account`
- `POST /api/auth/register/password`
- `POST /api/auth/login/password`
- `POST /api/auth/logout`
- `GET /api/auth/me`
- `POST /api/auth/password/change`
- `DELETE /api/auth/account`
- `POST /api/auth/passkeys/register/begin`
- `POST /api/auth/passkeys/register/finish?challengeId=...`
- `POST /api/auth/passkeys/login/begin`
- `POST /api/auth/passkeys/login/finish?challengeId=...`
The first registered user bootstraps as `admin`. Later public registrations become `viewer`; role changes are admin-managed from the account screen. Public registration endpoints only create new accounts. Adding or changing credentials on an existing account requires an authenticated browser session.
## API Tokens
Developer clients use bearer tokens:
```http
Authorization: Bearer cq_pat_<id>_<secret>
```
The token is shown once. The server stores the token id and SHA-256 hash of the secret. Tokens can be scoped and revoked.
Endpoints:
- `GET /api/tokens`
- `POST /api/tokens`
- `DELETE /api/tokens/{id}`
Supported scopes:
- `docs:read`
- `docs:write`
- `sync:read`
- `sync:write`
- `admin`
Authorization is role plus scope: a token must have the required scope, and the owning user role must allow that operation.
## Device Flow
CLI clients can avoid handling browser cookies by using the first-party device flow:
1. Client calls `POST /api/device/start`.
2. Server returns `deviceCode`, `userCode`, `verificationUri`, and polling interval.
3. User opens the verification URL in a browser and signs in normally.
4. User approves the code with `POST /api/device/approve`.
5. Client polls `POST /api/device/token`.
6. Server returns a normal bearer API token and consumes the device code.
This is intentionally inspired by OAuth device authorization, but it is not a general OAuth provider. There are no client registrations, redirect URIs, refresh tokens, consent screens, or third-party identity dependencies.
## Route Protection
- Public: health, static assets, document reads, login/account entry pages, password/passkey begin-login/register endpoints, device start/token polling.
- Session or bearer token required: auth profile/logout, edit/save APIs, uploads, sync/content APIs, device approval.
- Admin role required: admin APIs and API token management.
WebSocket authentication is session/bearer aware through the shared middleware model. API token creation and account management require a browser session so bearer tokens cannot mint more bearer tokens.

478
.project/specs/simplesync-protocol.md
View File

@@ -2,296 +2,332 @@
## Overview ## Overview
SimpleSync is a content-addressed file synchronization protocol designed for asynchronous collaboration on markdown documents. It provides automatic bidirectional sync between a server (source of truth) and clients (filesystem watchers or web browsers). SimpleSync is Cairnquire's intended synchronization model for asynchronous markdown collaboration. It covers three related surfaces:
1. The Go server watches the configured content directory and reflects local filesystem edits into the document index.
2. The Go-served browser UI caches documents in IndexedDB, queues offline edits, and resolves conflicts before writing over newer server content.
3. Native or external clients can use snapshot/delta endpoints for bidirectional file sync.
The protocol is content-addressed: every document version is identified by the SHA-256 hash of its bytes. The server never accepts a write that claims an older base hash without returning a conflict response.
## Design Goals ## Design Goals
1. **Simplicity**: No branches, commits, or manual sync commands - **Automatic:** Local filesystem edits and browser edits propagate without a manual import/export step.
2. **Automatic**: Changes propagate without user intervention - **Conflict-aware:** Concurrent edits to the same file are detected and surfaced; they are not silently overwritten.
3. **Conflict-aware**: Detects concurrent edits, never silently loses data - **Offline-capable:** Browser edits can be queued locally and synced when connectivity returns.
4. **Offline-capable**: Queue changes locally, sync when connected - **File-level simple:** Different files can be edited independently without CRDTs, branches, or a git mental model.
5. **Secure**: All content verified by cryptographic hash - **Verifiable:** Content hashes are recomputed by the server on every stored version.
## Concepts ## Milestone 2 Scope
### Content Addressing Included:
Every file is identified by the SHA-256 hash of its content. The content is immutable — if a file changes, it gets a new hash.
### Merkle Tree - Recursive filesystem watching for markdown documents.
The directory state is represented as a Merkle tree where: - Debounced sync after editor save bursts.
- Leaf nodes are file hashes - Ignoring hidden files, hidden directories, and common editor temp files.
- Internal nodes are hashes of concatenated child hashes - WebSocket document-change notifications.
- The root hash represents the entire directory state - IndexedDB document cache and pending edit queue.
- HTTP save conflict detection using `baseHash`.
- Manual browser conflict resolution with server/queued diff and resolved Monaco editor.
- Native snapshot/delta/content endpoints for future external clients.
- Simple fixed-size cleanup for cached rendered pages/content.
### Sync State Out of scope for Milestone 2:
Each client maintains:
- `root_hash`: Hash of current directory Merkle root
- `files`: Map of file paths to content hashes
- `pending`: Queue of local changes not yet synced
## Message Format - WebSocket authentication. Authentication and authorization belong to Milestone 3 and should be integrated with the final app session model, not a temporary shared secret.
- Playwright or browser-level automated sync tests.
- Performance benchmarking for large sync sets.
- Quota-aware IndexedDB eviction. Milestone 2 uses a fixed entry cap instead.
All messages are JSON over WebSocket. ## Content Addressing
### Client → Server All persisted document content is immutable and addressed by SHA-256:
```text
hash = sha256(document_bytes)
store path = <store-dir>/<hash[0:2]>/<hash[2:4]>/<hash>
```
Identical content deduplicates naturally. The latest document table points at the current hash for each path.
## Filesystem Watcher
The server watches the configured source directory recursively.
Watcher behavior:
- Watch existing directories on startup.
- Add watches for newly-created directories.
- Process markdown paths only: `.md` and `.MD`.
- Ignore hidden path segments such as `.git`, `.cache`, and `.note.md`.
- Ignore common editor temporary files such as `*.swp`, `*.tmp`, `*.bak`, `*.orig`, `*~`, and `#file#`.
- Debounce rapid event bursts before running a full `SyncSourceDir()`.
- Keep the 30 second polling fallback so missed fsnotify events are eventually reconciled.
The watcher callback performs a full source-directory sync. This keeps the fsnotify layer simple and avoids depending on platform-specific event details for correctness.
## Browser Editor Sync
The browser editor uses the document's current server hash as an optimistic concurrency token.
### Load
#### Sync Request
```json ```json
{ {
"type": "sync_request", "path": "guide.md",
"client_id": "uuid-v4", "content": "# Guide\n",
"root_hash": "sha256-hex-64-chars", "hash": "server-hash"
"files": {
"getting-started.md": "sha256...",
"api-reference.md": "sha256..."
},
"timestamp": "2024-01-15T14:30:00Z"
} }
``` ```
#### Content Upload The browser stores the loaded content and hash in IndexedDB.
### Save
```json ```json
{ {
"type": "content_upload", "content": "# Guide\n\nLocal edit\n",
"hash": "sha256...", "baseHash": "server-hash"
"content": "base64-encoded-content",
"encoding": "base64"
} }
``` ```
#### Acknowledgment If `baseHash` still matches the server's current hash, the server writes the file, stores the new content hash, and returns the updated document.
If the server has changed since the browser loaded the document, the server returns `409 Conflict`:
```json ```json
{ {
"type": "ack", "status": "conflict",
"message_id": "uuid-of-original-message", "path": "guide.md",
"status": "ok" "baseHash": "server-hash",
"currentHash": "new-server-hash",
"currentContent": "# Guide\n\nServer edit\n"
} }
``` ```
### Server → Client The browser keeps the queued local edit in IndexedDB, shows a diff between the server copy and the queued edit, and lets the user produce a resolved document. Submitting the resolved document uses the latest `currentHash` as the next `baseHash`.
### Offline Queue
When a save cannot reach the server, the browser records a pending edit:
#### Sync Response
```json ```json
{ {
"type": "sync_response", "path": "guide.md",
"server_root_hash": "sha256...", "content": "# Guide\n\nQueued edit\n",
"missing_from_client": ["hash1", "hash2"], "baseHash": "server-hash",
"missing_from_server": ["hash3"], "status": "queued",
"conflicts": ["getting-started.md"], "updatedAt": "2026-05-14T12:00:00Z"
"timestamp": "2024-01-15T14:30:01Z"
} }
``` ```
#### Content Push On reconnect or manual sync, queued edits are retried. Conflicts remain local until resolved by the user.
### Cache Cleanup
The browser keeps pending edits until they sync or the user resolves them. Rendered page/content caches use a fixed entry cap and evict the oldest `cachedAt` records after successful cache writes. This gives predictable bounded growth without adding quota-estimation complexity before production hardening.
## Realtime Notifications
WebSocket notifications are advisory. They tell connected browser clients that indexed server state changed; durable sync correctness still comes from hash-checked HTTP reads and writes.
Current intended message families:
```json ```json
{ {
"type": "content_push", "type": "document_version",
"hash": "sha256...", "path": "guide.md",
"content": "base64-encoded-content", "hash": "new-server-hash"
"encoding": "base64",
"path": "getting-started.md"
} }
``` ```
#### Conflict Notification
```json ```json
{ {
"type": "conflict", "type": "folder_tree_changed"
"path": "getting-started.md",
"server_hash": "sha256...",
"client_hash": "sha256...",
"server_modified_at": "2024-01-15T14:30:00Z",
"client_modified_at": "2024-01-15T14:25:00Z"
} }
``` ```
#### Error Clients should reconnect automatically and refresh state after reconnect. If notifications are missed, the polling fallback and next document load still converge on server state.
## Native Snapshot/Delta Sync
External clients use HTTP endpoints instead of browser IndexedDB state.
### Initialize
`POST /api/sync/init`
```json ```json
{ {
"type": "error", "deviceId": "macbook-pro-1",
"code": "unauthorized", "rootPath": "/Users/alice/Documents/cairnquire"
"message": "Session expired",
"retryable": false
} }
``` ```
## Protocol Flow Response:
### Normal Sync (No Conflicts) ```json
{
``` "snapshotId": "snap:device:timestamp",
Client Server "serverSnapshot": [
| | {
|-- sync_request -------------->| "path": "guide.md",
| root_hash: abc | "hash": "sha256...",
| | "size": 42,
| |-- Compare with server state "modified": "2026-05-14T12:00:00Z"
| |-- Calculate deltas }
| | ]
|<- sync_response --------------| }
| missing_from_client: [def] |
| missing_from_server: [] |
| conflicts: [] |
| |
|-- content_request(def) ------>|
| |
|<- content_push(def) ----------|
| |
|-- ack ----------------------->|
``` ```
### Upload Changes ### Delta
``` `POST /api/sync/delta`
Client Server
| | ```json
|-- sync_request -------------->| {
| root_hash: abc | "snapshotId": "snap:device:timestamp",
| files: {a: hash1, b: hash2} | "clientDelta": [
| | {
|<- sync_response --------------| "type": "update",
| missing_from_server: [hash2] | "path": "guide.md",
| | "hash": "client-hash",
|-- content_upload(hash2) ----->| "size": 58,
| | "modified": "2026-05-14T12:05:00Z"
| |-- Verify hash }
| |-- Store content ]
| |-- Update database }
| |-- Broadcast to others
| |
|<- ack ------------------------|
| |
``` ```
### Conflict Detection Response:
```json
{
"serverDelta": [
{
"type": "create",
"path": "new-note.md",
"hash": "server-hash",
"size": 100,
"modified": "2026-05-14T12:03:00Z"
}
],
"conflicts": [
{
"path": "guide.md",
"serverHash": "server-hash",
"clientHash": "client-hash",
"serverModified": "2026-05-14T12:03:00Z",
"clientModified": "2026-05-14T12:05:00Z",
"strategy": "manual-merge"
}
]
}
``` ```
Client A Server Client B
| | | ### Resolve
| | |-- Edit file X
| | |-- sync_request `POST /api/sync/resolve`
| |-- Update file X |
| |-- Broadcast to A | ```json
| | | {
|-- Edit file X | | "snapshotId": "snap:device:timestamp",
|-- sync_request -------------->| | "resolutions": [
| |-- Detect conflict | {
| |-- A has old hash | "path": "guide.md",
| |-- B has new hash | "strategy": "server-wins"
|<- conflict -------------------| | }
| path: X | | ]
| server_hash: B_hash | | }
| client_hash: A_hash | |
| | |
``` ```
Response:
```json
{
"newSnapshotId": "snap:device:new-timestamp"
}
```
### Content Fetch
`GET /api/content/{hash}`
Returns immutable raw bytes:
```text
Content-Type: application/octet-stream
Cache-Control: public, max-age=31536000, immutable
ETag: "<hash>"
```
## Conflict Rules
A same-file conflict exists when:
1. The client is editing from a known base hash.
2. The server's current hash for that path changed after that base hash.
3. The client attempts to write different content for the same path.
Different paths do not conflict. Deletions and renames are represented in the native delta model and should be resolved at file-path granularity.
Resolution strategies:
- `manual-merge`: User edits a resolved document and saves it against the latest server hash.
- `server-wins`: Keep the current server version.
- `client-wins`: Replace the server version with the client version after explicit user choice.
- `rename-both`: Preserve both versions using a conflict filename.
- `last-write-wins`: Allowed for automation, but not the default browser UX because it can hide data loss.
## State Machine ## State Machine
``` ```text
+--------+ sync_request online
| +-----------+ +----------------------------------+
| Idle | | | v
| |<----------+ +-----+-----+ edit/save ok +-----+------+
+---+----+ | Reading +--------------------->+ Synced |
| +-----+-----+ +-----+------+
| connect | ^
v | edit while offline |
+---------+---------+ v |
| | +-----+------+ reconnect/manual sync |
| Connected | | Queued +---------------------------+
| | +-----+------+
+---------+---------+ |
| | server hash changed
| sync_request v
v +-----+------+
+---------+---------+ | Conflicted |
| | +-----+------+
| Comparing | |
| | | user resolves against latest hash
+---------+---------+ v
| +-----+------+
+-----------+-----------+ | Synced |
| | +------------+
| has_changes | no_changes
v v
+---------+---------+ +---------+---------+
| | | |
| Transferring | | Idle |
| | | |
+---------+---------+ +-------------------+
|
| complete
v
+---------+---------+
| |
| Idle |
| |
+-------------------+
^
| conflict
|
+---------+---------+
| |
| Conflicted |
| |
+-------------------+
``` ```
## Error Codes ## Error Codes
| Code | Description | Retryable | | Code | Description | Retryable |
|------|-------------|-----------| |------|-------------|-----------|
| `unauthorized` | Session expired or invalid | No (re-authenticate) | | `conflict` | Write base hash is stale | No, user resolution required |
| `forbidden` | Insufficient permissions | No | | `not_found` | Requested document or content hash is unknown | Sometimes |
| `not_found` | Requested content hash unknown | Yes | | `hash_mismatch` | Content does not match claimed hash | Yes, after recompute |
| `too_large` | Content exceeds size limit | No | | `too_large` | Content exceeds configured size limit | No |
| `hash_mismatch` | Content doesn't match claimed hash | Yes | | `rate_limited` | Too many requests | Yes, with backoff |
| `rate_limited` | Too many requests | Yes (with backoff) |
| `server_error` | Internal server error | Yes | | `server_error` | Internal server error | Yes |
## Security Considerations ## Security Considerations
1. **Authentication**: All WebSocket connections must authenticate via token in initial HTTP upgrade request - Milestone 2 does not add temporary WebSocket auth.
2. **Authorization**: Server verifies read/write permissions before serving or accepting content - Milestone 3 should apply session/authz checks consistently to document reads, document writes, WebSocket upgrade, and sync endpoints.
3. **Hash Verification**: Server recomputes SHA-256 of received content and rejects mismatches - The server must canonicalize paths and reject traversal outside the configured content root.
4. **Size Limits**: Maximum file size enforced (configurable, default 10MB) - The server recomputes SHA-256 before storing or accepting content.
5. **Rate Limiting**: Sync requests limited per client (configurable, default 10/minute) - Raw HTML markdown support is intended for trusted content repositories; untrusted public authoring requires a sanitizer policy before launch.
6. **Path Validation**: All file paths canonicalized and checked against allowlist
## Implementation Notes
### Hash Computation
```go
import "crypto/sha256"
func computeHash(content []byte) string {
h := sha256.Sum256(content)
return hex.EncodeToString(h[:])
}
```
### Merkle Root Computation
```go
func computeRootHash(files map[string]string) string {
// Sort paths for determinism
paths := sortedKeys(files)
hasher := sha256.New()
for _, path := range paths {
hasher.Write([]byte(path))
hasher.Write([]byte(files[path]))
}
return hex.EncodeToString(hasher.Sum(nil))
}
```
### WebSocket Connection
- Ping/pong every 30 seconds
- Connection timeout after 60 seconds without response
- Auto-reconnect with exponential backoff (1s, 2s, 4s, 8s, max 60s)
## Version ## Version
**Protocol Version**: 1.0 **Protocol Version:** 1.0\
**Last Updated**: 2024-01-15 **Last Updated:** 2026-05-14