sirtimbly/cairnquire
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update docs

Browse Source
This commit is contained in:
Tim Bendt
2026-05-28 08:35:25 -04:00
parent d359baa010
commit 3d44a392f1
5 changed files with 533 additions and 380 deletions
Download Patch File Download Diff File Expand all files Collapse all files

478
.project/specs/simplesync-protocol.md
View File

@@ -2,296 +2,332 @@
## Overview
SimpleSync is a content-addressed file synchronization protocol designed for asynchronous collaboration on markdown documents. It provides automatic bidirectional sync between a server (source of truth) and clients (filesystem watchers or web browsers).
SimpleSync is Cairnquire's intended synchronization model for asynchronous markdown collaboration. It covers three related surfaces:
1. The Go server watches the configured content directory and reflects local filesystem edits into the document index.
2. The Go-served browser UI caches documents in IndexedDB, queues offline edits, and resolves conflicts before writing over newer server content.
3. Native or external clients can use snapshot/delta endpoints for bidirectional file sync.
The protocol is content-addressed: every document version is identified by the SHA-256 hash of its bytes. The server never accepts a write that claims an older base hash without returning a conflict response.
## Design Goals
1. **Simplicity**: No branches, commits, or manual sync commands
2. **Automatic**: Changes propagate without user intervention
3. **Conflict-aware**: Detects concurrent edits, never silently loses data
4. **Offline-capable**: Queue changes locally, sync when connected
5. **Secure**: All content verified by cryptographic hash
- **Automatic:** Local filesystem edits and browser edits propagate without a manual import/export step.
- **Conflict-aware:** Concurrent edits to the same file are detected and surfaced; they are not silently overwritten.
- **Offline-capable:** Browser edits can be queued locally and synced when connectivity returns.
- **File-level simple:** Different files can be edited independently without CRDTs, branches, or a git mental model.
- **Verifiable:** Content hashes are recomputed by the server on every stored version.
## Concepts
## Milestone 2 Scope
### Content Addressing
Every file is identified by the SHA-256 hash of its content. The content is immutable — if a file changes, it gets a new hash.
Included:
### Merkle Tree
The directory state is represented as a Merkle tree where:
- Leaf nodes are file hashes
- Internal nodes are hashes of concatenated child hashes
- The root hash represents the entire directory state
- Recursive filesystem watching for markdown documents.
- Debounced sync after editor save bursts.
- Ignoring hidden files, hidden directories, and common editor temp files.
- WebSocket document-change notifications.
- IndexedDB document cache and pending edit queue.
- HTTP save conflict detection using `baseHash`.
- Manual browser conflict resolution with server/queued diff and resolved Monaco editor.
- Native snapshot/delta/content endpoints for future external clients.
- Simple fixed-size cleanup for cached rendered pages/content.
### Sync State
Each client maintains:
- `root_hash`: Hash of current directory Merkle root
- `files`: Map of file paths to content hashes
- `pending`: Queue of local changes not yet synced
Out of scope for Milestone 2:
## Message Format
- WebSocket authentication. Authentication and authorization belong to Milestone 3 and should be integrated with the final app session model, not a temporary shared secret.
- Playwright or browser-level automated sync tests.
- Performance benchmarking for large sync sets.
- Quota-aware IndexedDB eviction. Milestone 2 uses a fixed entry cap instead.
All messages are JSON over WebSocket.
## Content Addressing
### Client → Server
All persisted document content is immutable and addressed by SHA-256:
```text
hash = sha256(document_bytes)
store path = <store-dir>/<hash[0:2]>/<hash[2:4]>/<hash>
```
Identical content deduplicates naturally. The latest document table points at the current hash for each path.
## Filesystem Watcher
The server watches the configured source directory recursively.
Watcher behavior:
- Watch existing directories on startup.
- Add watches for newly-created directories.
- Process markdown paths only: `.md` and `.MD`.
- Ignore hidden path segments such as `.git`, `.cache`, and `.note.md`.
- Ignore common editor temporary files such as `*.swp`, `*.tmp`, `*.bak`, `*.orig`, `*~`, and `#file#`.
- Debounce rapid event bursts before running a full `SyncSourceDir()`.
- Keep the 30 second polling fallback so missed fsnotify events are eventually reconciled.
The watcher callback performs a full source-directory sync. This keeps the fsnotify layer simple and avoids depending on platform-specific event details for correctness.
## Browser Editor Sync
The browser editor uses the document's current server hash as an optimistic concurrency token.
### Load
#### Sync Request
```json
{
"type": "sync_request",
"client_id": "uuid-v4",
"root_hash": "sha256-hex-64-chars",
"files": {
"getting-started.md": "sha256...",
"api-reference.md": "sha256..."
},
"timestamp": "2024-01-15T14:30:00Z"
"path": "guide.md",
"content": "# Guide\n",
"hash": "server-hash"
}
```
#### Content Upload
The browser stores the loaded content and hash in IndexedDB.
### Save
```json
{
"type": "content_upload",
"hash": "sha256...",
"content": "base64-encoded-content",
"encoding": "base64"
"content": "# Guide\n\nLocal edit\n",
"baseHash": "server-hash"
}
```
#### Acknowledgment
If `baseHash` still matches the server's current hash, the server writes the file, stores the new content hash, and returns the updated document.
If the server has changed since the browser loaded the document, the server returns `409 Conflict`:
```json
{
"type": "ack",
"message_id": "uuid-of-original-message",
"status": "ok"
"status": "conflict",
"path": "guide.md",
"baseHash": "server-hash",
"currentHash": "new-server-hash",
"currentContent": "# Guide\n\nServer edit\n"
}
```
### Server → Client
The browser keeps the queued local edit in IndexedDB, shows a diff between the server copy and the queued edit, and lets the user produce a resolved document. Submitting the resolved document uses the latest `currentHash` as the next `baseHash`.
### Offline Queue
When a save cannot reach the server, the browser records a pending edit:
#### Sync Response
```json
{
"type": "sync_response",
"server_root_hash": "sha256...",
"missing_from_client": ["hash1", "hash2"],
"missing_from_server": ["hash3"],
"conflicts": ["getting-started.md"],
"timestamp": "2024-01-15T14:30:01Z"
"path": "guide.md",
"content": "# Guide\n\nQueued edit\n",
"baseHash": "server-hash",
"status": "queued",
"updatedAt": "2026-05-14T12:00:00Z"
}
```
#### Content Push
On reconnect or manual sync, queued edits are retried. Conflicts remain local until resolved by the user.
### Cache Cleanup
The browser keeps pending edits until they sync or the user resolves them. Rendered page/content caches use a fixed entry cap and evict the oldest `cachedAt` records after successful cache writes. This gives predictable bounded growth without adding quota-estimation complexity before production hardening.
## Realtime Notifications
WebSocket notifications are advisory. They tell connected browser clients that indexed server state changed; durable sync correctness still comes from hash-checked HTTP reads and writes.
Current intended message families:
```json
{
"type": "content_push",
"hash": "sha256...",
"content": "base64-encoded-content",
"encoding": "base64",
"path": "getting-started.md"
"type": "document_version",
"path": "guide.md",
"hash": "new-server-hash"
}
```
#### Conflict Notification
```json
{
"type": "conflict",
"path": "getting-started.md",
"server_hash": "sha256...",
"client_hash": "sha256...",
"server_modified_at": "2024-01-15T14:30:00Z",
"client_modified_at": "2024-01-15T14:25:00Z"
"type": "folder_tree_changed"
}
```
#### Error
Clients should reconnect automatically and refresh state after reconnect. If notifications are missed, the polling fallback and next document load still converge on server state.
## Native Snapshot/Delta Sync
External clients use HTTP endpoints instead of browser IndexedDB state.
### Initialize
`POST /api/sync/init`
```json
{
"type": "error",
"code": "unauthorized",
"message": "Session expired",
"retryable": false
"deviceId": "macbook-pro-1",
"rootPath": "/Users/alice/Documents/cairnquire"
}
```
## Protocol Flow
Response:
### Normal Sync (No Conflicts)
```
Client Server
| |
|-- sync_request -------------->|
| root_hash: abc |
| |
| |-- Compare with server state
| |-- Calculate deltas
| |
|<- sync_response --------------|
| missing_from_client: [def] |
| missing_from_server: [] |
| conflicts: [] |
| |
|-- content_request(def) ------>|
| |
|<- content_push(def) ----------|
| |
|-- ack ----------------------->|
```json
{
"snapshotId": "snap:device:timestamp",
"serverSnapshot": [
{
"path": "guide.md",
"hash": "sha256...",
"size": 42,
"modified": "2026-05-14T12:00:00Z"
}
]
}
```
### Upload Changes
### Delta
```
Client Server
| |
|-- sync_request -------------->|
| root_hash: abc |
| files: {a: hash1, b: hash2} |
| |
|<- sync_response --------------|
| missing_from_server: [hash2] |
| |
|-- content_upload(hash2) ----->|
| |
| |-- Verify hash
| |-- Store content
| |-- Update database
| |-- Broadcast to others
| |
|<- ack ------------------------|
| |
`POST /api/sync/delta`
```json
{
"snapshotId": "snap:device:timestamp",
"clientDelta": [
{
"type": "update",
"path": "guide.md",
"hash": "client-hash",
"size": 58,
"modified": "2026-05-14T12:05:00Z"
}
]
}
```
### Conflict Detection
Response:
```json
{
"serverDelta": [
{
"type": "create",
"path": "new-note.md",
"hash": "server-hash",
"size": 100,
"modified": "2026-05-14T12:03:00Z"
}
],
"conflicts": [
{
"path": "guide.md",
"serverHash": "server-hash",
"clientHash": "client-hash",
"serverModified": "2026-05-14T12:03:00Z",
"clientModified": "2026-05-14T12:05:00Z",
"strategy": "manual-merge"
}
]
}
```
Client A Server Client B
| | |
| | |-- Edit file X
| | |-- sync_request
| |-- Update file X |
| |-- Broadcast to A |
| | |
|-- Edit file X | |
|-- sync_request -------------->| |
| |-- Detect conflict |
| |-- A has old hash |
| |-- B has new hash |
|<- conflict -------------------| |
| path: X | |
| server_hash: B_hash | |
| client_hash: A_hash | |
| | |
### Resolve
`POST /api/sync/resolve`
```json
{
"snapshotId": "snap:device:timestamp",
"resolutions": [
{
"path": "guide.md",
"strategy": "server-wins"
}
]
}
```
Response:
```json
{
"newSnapshotId": "snap:device:new-timestamp"
}
```
### Content Fetch
`GET /api/content/{hash}`
Returns immutable raw bytes:
```text
Content-Type: application/octet-stream
Cache-Control: public, max-age=31536000, immutable
ETag: "<hash>"
```
## Conflict Rules
A same-file conflict exists when:
1. The client is editing from a known base hash.
2. The server's current hash for that path changed after that base hash.
3. The client attempts to write different content for the same path.
Different paths do not conflict. Deletions and renames are represented in the native delta model and should be resolved at file-path granularity.
Resolution strategies:
- `manual-merge`: User edits a resolved document and saves it against the latest server hash.
- `server-wins`: Keep the current server version.
- `client-wins`: Replace the server version with the client version after explicit user choice.
- `rename-both`: Preserve both versions using a conflict filename.
- `last-write-wins`: Allowed for automation, but not the default browser UX because it can hide data loss.
## State Machine
```
+--------+ sync_request
| +-----------+
| Idle | |
| |<----------+
+---+----+
|
| connect
v
+---------+---------+
| |
| Connected |
| |
+---------+---------+
|
| sync_request
v
+---------+---------+
| |
| Comparing |
| |
+---------+---------+
|
+-----------+-----------+
| |
| has_changes | no_changes
v v
+---------+---------+ +---------+---------+
| | | |
| Transferring | | Idle |
| | | |
+---------+---------+ +-------------------+
|
| complete
v
+---------+---------+
| |
| Idle |
| |
+-------------------+
^
| conflict
|
+---------+---------+
| |
| Conflicted |
| |
+-------------------+
```text
online
+----------------------------------+
| v
+-----+-----+ edit/save ok +-----+------+
| Reading +--------------------->+ Synced |
+-----+-----+ +-----+------+
| ^
| edit while offline |
v |
+-----+------+ reconnect/manual sync |
| Queued +---------------------------+
+-----+------+
|
| server hash changed
v
+-----+------+
| Conflicted |
+-----+------+
|
| user resolves against latest hash
v
+-----+------+
| Synced |
+------------+
```
## Error Codes
| Code | Description | Retryable |
|------|-------------|-----------|
| `unauthorized` | Session expired or invalid | No (re-authenticate) |
| `forbidden` | Insufficient permissions | No |
| `not_found` | Requested content hash unknown | Yes |
| `too_large` | Content exceeds size limit | No |
| `hash_mismatch` | Content doesn't match claimed hash | Yes |
| `rate_limited` | Too many requests | Yes (with backoff) |
| `conflict` | Write base hash is stale | No, user resolution required |
| `not_found` | Requested document or content hash is unknown | Sometimes |
| `hash_mismatch` | Content does not match claimed hash | Yes, after recompute |
| `too_large` | Content exceeds configured size limit | No |
| `rate_limited` | Too many requests | Yes, with backoff |
| `server_error` | Internal server error | Yes |
## Security Considerations
1. **Authentication**: All WebSocket connections must authenticate via token in initial HTTP upgrade request
2. **Authorization**: Server verifies read/write permissions before serving or accepting content
3. **Hash Verification**: Server recomputes SHA-256 of received content and rejects mismatches
4. **Size Limits**: Maximum file size enforced (configurable, default 10MB)
5. **Rate Limiting**: Sync requests limited per client (configurable, default 10/minute)
6. **Path Validation**: All file paths canonicalized and checked against allowlist
## Implementation Notes
### Hash Computation
```go
import "crypto/sha256"
func computeHash(content []byte) string {
h := sha256.Sum256(content)
return hex.EncodeToString(h[:])
}
```
### Merkle Root Computation
```go
func computeRootHash(files map[string]string) string {
// Sort paths for determinism
paths := sortedKeys(files)
hasher := sha256.New()
for _, path := range paths {
hasher.Write([]byte(path))
hasher.Write([]byte(files[path]))
}
return hex.EncodeToString(hasher.Sum(nil))
}
```
### WebSocket Connection
- Ping/pong every 30 seconds
- Connection timeout after 60 seconds without response
- Auto-reconnect with exponential backoff (1s, 2s, 4s, 8s, max 60s)
- Milestone 2 does not add temporary WebSocket auth.
- Milestone 3 should apply session/authz checks consistently to document reads, document writes, WebSocket upgrade, and sync endpoints.
- The server must canonicalize paths and reject traversal outside the configured content root.
- The server recomputes SHA-256 before storing or accepting content.
- Raw HTML markdown support is intended for trusted content repositories; untrusted public authoring requires a sanitizer policy before launch.
## Version
**Protocol Version**: 1.0
**Last Updated**: 2024-01-15
**Protocol Version:** 1.0\
**Last Updated:** 2026-05-14