cairnquire/.project/specs/sync-protocol-v1.md

Sync Protocol v1 Specification

Overview

The Native Sync Protocol enables bidirectional filesystem synchronization between a native macOS client and the cairnquire server. The browser client continues to operate via the existing WebSocket + REST API.

Data Model

Snapshot

A point-in-time tree of all files in the sync scope:

{
  "id": "snap:device-1:1234567890",
  "deviceId": "device-1",
  "createdAt": "2024-01-15T10:30:00Z",
  "files": [
    {
      "path": "hello.md",
      "hash": "a1b2c3...",
      "size": 42,
      "modified": "2024-01-15T10:00:00Z"
    }
  ]
}

Delta

A list of changes since a snapshot:

{
  "changes": [
    {
      "type": "create|update|delete|rename",
      "path": "new-file.md",
      "oldPath": "old-file.md",
      "hash": "a1b2c3...",
      "size": 42,
      "modified": "2024-01-15T10:00:00Z"
    }
  ]
}

Conflict

When server and client both changed the same file since the last common snapshot:

{
  "path": "hello.md",
  "serverHash": "a1b2c3...",
  "clientHash": "d4e5f6...",
  "serverModified": "2024-01-15T10:00:00Z",
  "clientModified": "2024-01-15T10:05:00Z",
  "strategy": "last-write-wins"
}

Resolution Strategies

• last-write-wins (default): Use the most recently modified version
• rename-both: Keep both versions, renaming one as file (conflict).md
• manual-merge: Flag for UI resolution
• server-wins: Always use server version
• client-wins: Always use client version

Endpoints

All endpoints require authentication via X-API-Key header.

POST /api/sync/init

Initialize a new sync session.

Request:

{
  "deviceId": "macbook-pro-1",
  "rootPath": "/Users/alice/Documents/md-hub"
}

Response:

{
  "snapshotId": "snap:macbook-pro-1:1234567890",
  "serverSnapshot": [
    {
      "path": "hello.md",
      "hash": "a1b2c3...",
      "size": 42,
      "modified": "2024-01-15T10:00:00Z"
    }
  ]
}

POST /api/sync/delta

Upload client changes and receive server changes + conflicts.

Request:

{
  "snapshotId": "snap:macbook-pro-1:1234567890",
  "clientDelta": [
    {
      "type": "update",
      "path": "hello.md",
      "hash": "d4e5f6...",
      "content": "# Hello\n\nUpdated content\n",
      "size": 50,
      "modified": "2024-01-15T10:05:00Z"
    }
  ]
}

Response:

{
  "serverDelta": [
    {
      "type": "create",
      "path": "new-guide.md",
      "hash": "g7h8i9...",
      "size": 100,
      "modified": "2024-01-15T10:02:00Z"
    }
  ],
  "newSnapshotId": "snap:macbook-pro-1:1234567999",
  "conflicts": [
    {
      "path": "hello.md",
      "serverHash": "a1b2c3...",
      "clientHash": "d4e5f6...",
      "serverModified": "2024-01-15T10:00:00Z",
      "clientModified": "2024-01-15T10:05:00Z",
      "strategy": "last-write-wins"
    }
  ]
}

For client create/update changes, the server accepts inline UTF-8 markdown in content and recomputes the SHA-256 hash before writing the file. If content is omitted, the supplied hash must already exist in the server content store.

POST /api/sync/resolve

Resolve conflicts and finalize the sync.

Request:

{
  "snapshotId": "snap:macbook-pro-1:1234567890",
  "resolutions": [
    {
      "path": "hello.md",
      "strategy": "server-wins"
    }
  ]
}

Response:

{
  "newSnapshotId": "snap:macbook-pro-1:1234567999"
}

GET /api/content/{hash}

Fetch raw file content by content hash (immutable, cache-friendly).

Response: Raw file bytes with headers:

Content-Type: application/octet-stream
Cache-Control: public, max-age=31536000, immutable
ETag: "a1b2c3..."

Protocol Flow

1. Client calls POST /api/sync/init
   → Receives server snapshot

2. Client computes local delta
   (fsevents + local state DB)

3. Client POSTs delta to /api/sync/delta
   → Receives serverDelta + conflicts

4. If conflicts exist:
   a. Client resolves conflicts (auto or UI)
   b. Client POSTs resolutions to /api/sync/resolve
   → Receives newSnapshotId

5. Client applies serverDelta to local filesystem
   (downloads content via /api/content/{hash})

6. Both sides now have the same snapshot

Authentication

Sync endpoints require API key authentication:

X-API-Key: <api-key>

API keys are long-lived tokens scoped to a user and device. They are stored in the api_keys table with SHA-256 hashes.

Implementation Details

Server Architecture

internal/sync/
├── protocol.go    # Types and constants
├── service.go     # Business logic (conflict resolution, snapshot management)
├── store.go       # Database operations

Database Schema

-- API keys for native client authentication
CREATE TABLE api_keys (
    id TEXT PRIMARY KEY,
    user_id TEXT NOT NULL,
    name TEXT NOT NULL,
    key_hash TEXT NOT NULL UNIQUE,
    scopes TEXT NOT NULL DEFAULT 'sync:read,sync:write',
    created_at TEXT NOT NULL,
    expires_at TEXT,
    last_used_at TEXT,
    FOREIGN KEY(user_id) REFERENCES users(id) ON DELETE CASCADE
);

-- Sync snapshots (point-in-time file trees)
CREATE TABLE sync_snapshots (
    id TEXT PRIMARY KEY,
    device_id TEXT NOT NULL,
    user_id TEXT NOT NULL,
    created_at TEXT NOT NULL,
    FOREIGN KEY(user_id) REFERENCES users(id) ON DELETE CASCADE
);

-- Files within a snapshot
CREATE TABLE sync_files (
    snapshot_id TEXT NOT NULL,
    path TEXT NOT NULL,
    hash TEXT NOT NULL,
    size_bytes INTEGER NOT NULL DEFAULT 0,
    modified_at TEXT NOT NULL,
    PRIMARY KEY (snapshot_id, path),
    FOREIGN KEY(snapshot_id) REFERENCES sync_snapshots(id) ON DELETE CASCADE
);

Conflict Detection

A conflict occurs when:

1. The server file existed in the snapshot
2. The server file has changed since the snapshot (hash changed)
3. The client also changed the same file (different hash in delta)

Content-Addressed Storage

All file content is stored in a content-addressable store using SHA-256 hashes:

• Storage path: <store-dir>/<hash[0:2]>/<hash[2:4]>/<hash>
• Immutable: once stored, content never changes
• Deduplication: identical content shares the same storage

Testing

The sync protocol includes table-driven tests that simulate client/server scenarios:

• TestInitSyncCreatesSnapshot: Validates snapshot generation
• TestApplyDeltaDetectsServerChanges: Detects server-side modifications
• TestApplyDeltaDetectsConflicts: Identifies concurrent modifications
• TestResolveConflictsCreatesNewSnapshot: Validates conflict resolution
• TestGetContentReturnsFileBytes: Tests content retrieval