Add conflict resolution diff UI and app branding

This commit is contained in:
2026-05-13 16:25:28 -04:00
parent 780ff3a02c
commit d359baa010
83 changed files with 4523 additions and 3735 deletions

View File

@@ -0,0 +1,272 @@
# Deployment Guide
## Requirements
- Docker 24.0+
- Docker Compose 2.20+
- 1GB RAM minimum (2GB recommended)
- 10GB disk space
- TLS certificate (Let's Encrypt or custom)
## Quick Start
```bash
# Clone repository
git clone https://github.com/your-org/cairnquire.git
cd cairnquire
# Copy and edit configuration
cp .env.example .env
# Edit .env with your domain, email, Postmark API key
# Start services
docker-compose up -d
# Verify health
curl https://your-domain.com/health
```
## Configuration
### Environment Variables
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `DOMAIN` | Yes | — | Public domain name |
| `DB_PATH` | No | `/data/db.sqlite` | Database file path |
| `FILESTORE_PATH` | No | `/data/files` | Content-addressed file storage |
| `NOTES_PATH` | No | `/notes` | Watched markdown directory |
| `POSTMARK_API_KEY` | Yes | — | Postmark server API token |
| `POSTMARK_FROM` | No | `notifications@DOMAIN` | From email address |
| `SESSION_SECRET` | Yes | — | 32-byte hex for session signing |
| `ENV` | No | `production` | `development` or `production` |
### Docker Compose
```yaml
version: "3.8"
services:
app:
build: .
ports:
- "8080:8080"
volumes:
- ./data:/data
- ./notes:/notes:ro
environment:
- DOMAIN=${DOMAIN}
- DB_PATH=/data/db.sqlite
- FILESTORE_PATH=/data/files
- NOTES_PATH=/notes
- POSTMARK_API_KEY=${POSTMARK_API_KEY}
- SESSION_SECRET=${SESSION_SECRET}
env_file:
- .env
restart: unless-stopped
healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://localhost:8080/health"]
interval: 30s
timeout: 3s
retries: 3
start_period: 10s
security_opt:
- no-new-privileges:true
read_only: true
tmpfs:
- /tmp:noexec,nosuid,size=100m
# Optional: Traefik for auto-HTTPS
traefik:
image: traefik:v3.0
command:
- "--api.insecure=false"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
- "--certificatesresolvers.letsencrypt.acme.email=${EMAIL}"
- "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./letsencrypt:/letsencrypt
labels:
- "traefik.enable=true"
restart: unless-stopped
```
## Production Checklist
### Pre-Deployment
- [ ] Change default `SESSION_SECRET` (generate: `openssl rand -hex 32`)
- [ ] Configure `POSTMARK_API_KEY` with valid token
- [ ] Set `DOMAIN` to public hostname
- [ ] Configure DNS A/AAAA records
- [ ] Set up SPF and DKIM for email domain
- [ ] Review `.env` file — no default secrets
### Security
- [ ] TLS 1.3 enforced (Traefik or reverse proxy)
- [ ] HSTS header configured
- [ ] Firewall rules: only 80/443 open
- [ ] Fail2ban or rate limiting on SSH
- [ ] Automatic security updates enabled
- [ ] Backup encryption key stored securely
### Monitoring
- [ ] Health check endpoint accessible
- [ ] Log rotation configured
- [ ] Disk space alerting (>80%)
- [ ] Memory usage alerting
- [ ] Uptime monitoring (external)
## Backup Procedures
### Automated Backup
```bash
#!/bin/bash
# backup.sh
BACKUP_DIR="/backups"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/md-hub-$TIMESTAMP.tar.gz"
# Create backup
tar czf "$BACKUP_FILE" \
-C /data \
db.sqlite \
files/
# Encrypt (optional)
gpg --symmetric --cipher-algo AES256 "$BACKUP_FILE"
rm "$BACKUP_FILE"
# Keep only last 30 days
find "$BACKUP_DIR" -name "md-hub-*.tar.gz.gpg" -mtime +30 -delete
```
**Cron**: `0 2 * * * /opt/cairnquire/backup.sh`
### Restore from Backup
```bash
# Stop application
docker-compose stop app
# Extract backup
tar xzf md-hub-20240115_020000.tar.gz -C /data
# Restart
docker-compose up -d app
# Verify
curl https://your-domain.com/health
```
## Upgrade Procedures
### Minor Update (patch version)
```bash
docker-compose pull
docker-compose up -d
```
### Major Update
1. Read release notes for breaking changes
2. Create backup: `./backup.sh`
3. Pull new image: `docker-compose pull`
4. Run database migrations (auto on startup)
5. Update configuration if needed
6. Deploy: `docker-compose up -d`
7. Verify health and functionality
8. Rollback if issues: restore from backup
## Troubleshooting
### Application Won't Start
```bash
# Check logs
docker-compose logs -f app
# Verify configuration
docker-compose exec app /server -check-config
# Test database
docker-compose exec app sqlite3 /data/db.sqlite ".tables"
```
### Sync Not Working
```bash
# Check file watcher
docker-compose exec app ls -la /notes
# Verify WebSocket
curl -i -N \
-H "Connection: Upgrade" \
-H "Upgrade: websocket" \
-H "Sec-WebSocket-Key: $(openssl rand -base64 16)" \
-H "Sec-WebSocket-Version: 13" \
https://your-domain.com/ws
```
### Email Not Sending
```bash
# Test Postmark connection
curl -X POST \
https://api.postmarkapp.com/server \
-H "X-Postmark-Server-Token: YOUR_TOKEN" \
-H "Accept: application/json"
# Check email queue
docker-compose exec app sqlite3 /data/db.sqlite \
"SELECT * FROM email_queue WHERE status='failed'"
```
## Scaling
### Vertical Scaling
- Increase container memory/CPU limits
- SQLite WAL mode handles concurrent reads well
- For heavy write loads, consider libsql replication
### Horizontal Scaling (Advanced)
```yaml
# Requires shared storage and load balancer
services:
app1:
# ... same config ...
environment:
- DB_PATH=/shared/db.sqlite
- FILESTORE_PATH=/shared/files
app2:
# ... same config ...
environment:
- DB_PATH=/shared/db.sqlite
- FILESTORE_PATH=/shared/files
nginx:
image: nginx:alpine
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
ports:
- "80:80"
```
**Note**: Horizontal scaling requires:
- Shared filesystem (NFS, EFS)
- libsql replication or external database
- Sticky sessions for WebSocket
## Support
- **Documentation**: `/docs` route (when authenticated)
- **Health Check**: `GET /health`
- **Metrics**: `GET /metrics` (Prometheus format)
- **Logs**: `docker-compose logs -f app`
- **Issues**: https://github.com/your-org/cairnquire/issues

View File

@@ -0,0 +1,456 @@
# Security Specification
## Threat Model
### Assets
1. **Document Content**: Markdown files and attachments
2. **User Credentials**: Passkeys, passwords, session tokens
3. **Collaboration Data**: Comments, notifications, history
4. **System State**: Database, configuration, logs
### Threat Actors
1. **External Attacker**: No system access, attempts remote exploitation
2. **Malicious User**: Authenticated, attempts unauthorized access
3. **Compromised Client**: Legitimate user's device infected
4. **Insider**: Administrator or developer with system access
### Threats
1. **Supply Chain Attack**: Malicious dependency injection
2. **Authentication Bypass**: Session hijacking, credential theft
3. **Authorization Bypass**: Horizontal/vertical privilege escalation
4. **Data Exfiltration**: Unauthorized document access
5. **Data Tampering**: Modification without authorization
6. **Denial of Service**: Resource exhaustion
7. **Information Disclosure**: Secrets in logs/responses
## Security Controls
### 1. Supply Chain Security
**Policy**:
- Only dependencies with >1M downloads/month or official Go team
- All versions pinned in go.sum
- No transitive dependencies without review
- Prefer standard library when possible
**Implementation**:
```
# go.mod
require (
github.com/go-chi/chi/v5 v5.0.12
github.com/tursodatabase/libsql-client-go v0.0.0-20240416075031-555ce55511f7
// All versions pinned
)
# Build
RUN go mod verify # Verify checksums
RUN go mod vendor # Optional: vendor for air-gapped
```
**Audit**:
- Weekly `govulncheck` scan
- Dependency review in CI
- No build-time network access (vendored or proxy with checksum)
### 2. Authentication Security
#### Passkey (WebAuthn)
```go
// Registration
func registerPasskey(user User) (*Credential, error) {
options, session, err := webauthn.BeginRegistration(user)
// Store session data server-side (never client-side)
// Return options to client
}
func verifyRegistration(sessionID string, response ProtocolCredentialCreation) (*Credential, error) {
session := getSessionData(sessionID)
credential, err := webauthn.FinishRegistration(user, session, response)
// Verify attestation
// Store credential: ID, public key, sign count
}
```
#### Password (Argon2id)
```go
import "golang.org/x/crypto/argon2"
const (
timeCost = 3
memoryCost = 64 * 1024 // 64MB
parallelism = 4
saltLength = 16
keyLength = 32
)
func hashPassword(password string) (string, error) {
salt := make([]byte, saltLength)
if _, err := rand.Read(salt); err != nil {
return "", err
}
hash := argon2.IDKey([]byte(password), salt, timeCost, memoryCost, parallelism, keyLength)
// Encode to string: $argon2id$v=19$m=65536,t=3,p=4$<salt>$<hash>
return encodeHash(salt, hash), nil
}
func verifyPassword(password, encodedHash string) (bool, error) {
salt, hash, params, err := decodeHash(encodedHash)
if err != nil {
return false, err
}
computedHash := argon2.IDKey([]byte(password), salt, params.time, params.memory, params.parallelism, uint32(len(hash)))
// Constant-time comparison
return subtle.ConstantTimeCompare(hash, computedHash) == 1, nil
}
```
#### Session Management
```go
func createSession(userID string, r *http.Request) (*Session, error) {
token := make([]byte, 32)
if _, err := rand.Read(token); err != nil {
return nil, err
}
session := &Session{
ID: uuid.New().String(),
UserID: userID,
TokenHash: sha256.Sum256(token),
CreatedAt: time.Now(),
ExpiresAt: time.Now().Add(24 * time.Hour),
IP: r.RemoteAddr,
UserAgent: r.UserAgent(),
}
// Store in database
if err := db.CreateSession(session); err != nil {
return nil, err
}
// Set cookie
http.SetCookie(w, &http.Cookie{
Name: "session",
Value: base64.URLEncoding.EncodeToString(token),
Expires: session.ExpiresAt,
HttpOnly: true,
Secure: true, // HTTPS only
SameSite: http.SameSiteStrictMode,
Path: "/",
})
return session, nil
}
```
### 3. Authorization
#### Permission Model
```go
type Permission string
const (
PermissionRead Permission = "read"
PermissionWrite Permission = "write"
PermissionAdmin Permission = "admin"
)
type ResourceType string
const (
ResourceGlobal ResourceType = "global"
ResourceCollection ResourceType = "collection"
ResourceDocument ResourceType = "document"
)
func checkPermission(userID string, resourceType ResourceType, resourceID string, required Permission) (bool, error) {
// Check explicit permission
perm, err := db.GetPermission(userID, resourceType, resourceID)
if err == nil && hasPermission(perm.Permission, required) {
return true, nil
}
// Check collection permission for documents
if resourceType == ResourceDocument {
collectionID := getDocumentCollection(resourceID)
perm, err = db.GetPermission(userID, ResourceCollection, collectionID)
if err == nil && hasPermission(perm.Permission, required) {
return true, nil
}
}
// Check global permission
perm, err = db.GetPermission(userID, ResourceGlobal, "")
if err == nil && hasPermission(perm.Permission, required) {
return true, nil
}
// Check document public status
if resourceType == ResourceDocument && required == PermissionRead {
doc, err := db.GetDocument(resourceID)
if err == nil && doc.PublicRead {
return true, nil
}
}
return false, nil
}
```
#### Middleware
```go
func RequirePermission(resourceType ResourceType, required Permission) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
userID := getUserID(r.Context())
resourceID := chi.URLParam(r, "id")
allowed, err := checkPermission(userID, resourceType, resourceID, required)
if err != nil || !allowed {
http.Error(w, "Forbidden", http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
})
}
}
// Usage:
r.With(RequirePermission(ResourceDocument, PermissionRead)).
Get("/docs/{id}", getDocumentHandler)
```
### 4. Input Validation
#### Markdown Sanitization
```go
import "github.com/yuin/goldmark"
import "github.com/yuin/goldmark/extension"
import "github.com/yuin/goldmark/renderer/html"
func renderMarkdown(source []byte) ([]byte, error) {
md := goldmark.New(
goldmark.WithExtensions(
extension.Table,
extension.Strikethrough,
extension.Linkify,
// No raw HTML extension — we control HTML output
),
goldmark.WithRendererOptions(
html.WithHardWraps(),
html.WithXHTML(),
html.WithUnsafe(), // DISABLED — no raw HTML passthrough
),
)
var buf bytes.Buffer
if err := md.Convert(source, &buf); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
```
#### File Upload Validation
```go
func validateUpload(file io.Reader, filename string, maxSize int64) ([]byte, string, error) {
// Size limit
limited := io.LimitReader(file, maxSize+1)
data, err := io.ReadAll(limited)
if err != nil {
return nil, "", err
}
if int64(len(data)) > maxSize {
return nil, "", fmt.Errorf("file too large")
}
// Magic number validation
contentType := detectContentType(data)
allowedTypes := map[string]bool{
"image/jpeg": true,
"image/png": true,
"image/gif": true,
"image/webp": true,
"application/pdf": true,
}
if !allowedTypes[contentType] {
return nil, "", fmt.Errorf("file type not allowed")
}
// Extension whitelist
ext := strings.ToLower(filepath.Ext(filename))
allowedExts := map[string]bool{
".jpg": true, ".jpeg": true, ".png": true,
".gif": true, ".webp": true, ".pdf": true,
}
if !allowedExts[ext] {
return nil, "", fmt.Errorf("file extension not allowed")
}
// Compute hash
hash := sha256.Sum256(data)
hashStr := hex.EncodeToString(hash[:])
return data, hashStr, nil
}
```
#### Path Sanitization
```go
func sanitizePath(userPath string) (string, error) {
// Clean the path
clean := filepath.Clean(userPath)
// Prevent traversal
if strings.Contains(clean, "..") {
return "", fmt.Errorf("path traversal detected")
}
// Ensure it's relative
if filepath.IsAbs(clean) {
return "", fmt.Errorf("absolute paths not allowed")
}
// Join with base and verify it's within base
fullPath := filepath.Join(baseDir, clean)
if !strings.HasPrefix(filepath.Clean(fullPath), filepath.Clean(baseDir)) {
return "", fmt.Errorf("path outside allowed directory")
}
return clean, nil
}
```
### 5. Transport Security
#### TLS Configuration
```go
tlsConfig := &tls.Config{
MinVersion: tls.VersionTLS13,
CurvePreferences: []tls.CurveID{
tls.X25519,
tls.CurveP256,
},
CipherSuites: []uint16{
tls.TLS_AES_256_GCM_SHA384,
tls.TLS_CHACHA20_POLY1305_SHA256,
tls.TLS_AES_128_GCM_SHA256,
},
PreferServerCipherSuites: true,
}
```
#### Security Headers
```go
func securityHeaders(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Security-Policy",
"default-src 'self'; "+
"script-src 'self' 'nonce-{nonce}'; "+
"style-src 'self' 'unsafe-inline'; "+
"img-src 'self' data:; "+
"connect-src 'self' wss:; "+
"frame-ancestors 'none'; "+
"base-uri 'self'; "+
"form-action 'self';")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("X-Frame-Options", "DENY")
w.Header().Set("X-XSS-Protection", "1; mode=block")
w.Header().Set("Referrer-Policy", "strict-origin-when-cross-origin")
w.Header().Set("Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload")
w.Header().Set("Permissions-Policy", "camera=(), microphone=(), geolocation=()")
next.ServeHTTP(w, r)
})
}
```
### 6. Operational Security
#### Logging
```go
func logAudit(event AuditEvent) {
entry := slog.With(
"event_type", event.Type,
"actor_id", redactIfSecret(event.ActorID),
"resource_type", event.ResourceType,
"resource_id", redactIfSecret(event.ResourceID),
"ip", event.IP,
"timestamp", event.Timestamp,
)
// Never log: passwords, tokens, session IDs, API keys
entry.Info("audit event")
}
func redactIfSecret(value string) string {
if looksLikeSecret(value) {
return "[REDACTED]"
}
return value
}
```
#### Container Security
```dockerfile
# Dockerfile
FROM golang:1.22-alpine AS builder
# ... build ...
FROM alpine:3.19
RUN addgroup -g 1000 appgroup && \
adduser -u 1000 -G appgroup -s /bin/sh -D appuser
# Read-only root filesystem
COPY --from=builder /app/server /server
COPY --from=builder /app/web/dist /web
# Data directory (writable)
RUN mkdir /data && chown appuser:appgroup /data
VOLUME /data
USER appuser
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=3s \
CMD wget -q --spider http://localhost:8080/health || exit 1
ENTRYPOINT ["/server"]
```
## Incident Response
### Detection
- Automated alerts on: failed auth spikes, permission errors, large file uploads
- Log analysis for anomalies
- Health check failures
### Response
1. **Isolate**: Block IP, revoke sessions, disable account if compromised
2. **Assess**: Determine scope of breach, affected data
3. **Contain**: Patch vulnerability, rotate secrets
4. **Recover**: Restore from clean backup if necessary
5. **Document**: Post-incident review, update security measures
## Compliance
### GDPR
- Data minimization: only email stored, no other PII
- Right to erasure: `/api/user/delete` endpoint
- Data portability: `/api/user/export` endpoint
- Consent: explicit opt-in for email notifications
### Audit Trail
All security-relevant events logged:
- Authentication attempts (success/failure)
- Permission changes
- Document access (if private)
- Admin actions
- Configuration changes
Retention: 90 days active, 1 year archive

View File

@@ -0,0 +1,297 @@
# SimpleSync Protocol Specification
## Overview
SimpleSync is a content-addressed file synchronization protocol designed for asynchronous collaboration on markdown documents. It provides automatic bidirectional sync between a server (source of truth) and clients (filesystem watchers or web browsers).
## Design Goals
1. **Simplicity**: No branches, commits, or manual sync commands
2. **Automatic**: Changes propagate without user intervention
3. **Conflict-aware**: Detects concurrent edits, never silently loses data
4. **Offline-capable**: Queue changes locally, sync when connected
5. **Secure**: All content verified by cryptographic hash
## Concepts
### Content Addressing
Every file is identified by the SHA-256 hash of its content. The content is immutable — if a file changes, it gets a new hash.
### Merkle Tree
The directory state is represented as a Merkle tree where:
- Leaf nodes are file hashes
- Internal nodes are hashes of concatenated child hashes
- The root hash represents the entire directory state
### Sync State
Each client maintains:
- `root_hash`: Hash of current directory Merkle root
- `files`: Map of file paths to content hashes
- `pending`: Queue of local changes not yet synced
## Message Format
All messages are JSON over WebSocket.
### Client → Server
#### Sync Request
```json
{
"type": "sync_request",
"client_id": "uuid-v4",
"root_hash": "sha256-hex-64-chars",
"files": {
"getting-started.md": "sha256...",
"api-reference.md": "sha256..."
},
"timestamp": "2024-01-15T14:30:00Z"
}
```
#### Content Upload
```json
{
"type": "content_upload",
"hash": "sha256...",
"content": "base64-encoded-content",
"encoding": "base64"
}
```
#### Acknowledgment
```json
{
"type": "ack",
"message_id": "uuid-of-original-message",
"status": "ok"
}
```
### Server → Client
#### Sync Response
```json
{
"type": "sync_response",
"server_root_hash": "sha256...",
"missing_from_client": ["hash1", "hash2"],
"missing_from_server": ["hash3"],
"conflicts": ["getting-started.md"],
"timestamp": "2024-01-15T14:30:01Z"
}
```
#### Content Push
```json
{
"type": "content_push",
"hash": "sha256...",
"content": "base64-encoded-content",
"encoding": "base64",
"path": "getting-started.md"
}
```
#### Conflict Notification
```json
{
"type": "conflict",
"path": "getting-started.md",
"server_hash": "sha256...",
"client_hash": "sha256...",
"server_modified_at": "2024-01-15T14:30:00Z",
"client_modified_at": "2024-01-15T14:25:00Z"
}
```
#### Error
```json
{
"type": "error",
"code": "unauthorized",
"message": "Session expired",
"retryable": false
}
```
## Protocol Flow
### Normal Sync (No Conflicts)
```
Client Server
| |
|-- sync_request -------------->|
| root_hash: abc |
| |
| |-- Compare with server state
| |-- Calculate deltas
| |
|<- sync_response --------------|
| missing_from_client: [def] |
| missing_from_server: [] |
| conflicts: [] |
| |
|-- content_request(def) ------>|
| |
|<- content_push(def) ----------|
| |
|-- ack ----------------------->|
```
### Upload Changes
```
Client Server
| |
|-- sync_request -------------->|
| root_hash: abc |
| files: {a: hash1, b: hash2} |
| |
|<- sync_response --------------|
| missing_from_server: [hash2] |
| |
|-- content_upload(hash2) ----->|
| |
| |-- Verify hash
| |-- Store content
| |-- Update database
| |-- Broadcast to others
| |
|<- ack ------------------------|
| |
```
### Conflict Detection
```
Client A Server Client B
| | |
| | |-- Edit file X
| | |-- sync_request
| |-- Update file X |
| |-- Broadcast to A |
| | |
|-- Edit file X | |
|-- sync_request -------------->| |
| |-- Detect conflict |
| |-- A has old hash |
| |-- B has new hash |
|<- conflict -------------------| |
| path: X | |
| server_hash: B_hash | |
| client_hash: A_hash | |
| | |
```
## State Machine
```
+--------+ sync_request
| +-----------+
| Idle | |
| |<----------+
+---+----+
|
| connect
v
+---------+---------+
| |
| Connected |
| |
+---------+---------+
|
| sync_request
v
+---------+---------+
| |
| Comparing |
| |
+---------+---------+
|
+-----------+-----------+
| |
| has_changes | no_changes
v v
+---------+---------+ +---------+---------+
| | | |
| Transferring | | Idle |
| | | |
+---------+---------+ +-------------------+
|
| complete
v
+---------+---------+
| |
| Idle |
| |
+-------------------+
^
| conflict
|
+---------+---------+
| |
| Conflicted |
| |
+-------------------+
```
## Error Codes
| Code | Description | Retryable |
|------|-------------|-----------|
| `unauthorized` | Session expired or invalid | No (re-authenticate) |
| `forbidden` | Insufficient permissions | No |
| `not_found` | Requested content hash unknown | Yes |
| `too_large` | Content exceeds size limit | No |
| `hash_mismatch` | Content doesn't match claimed hash | Yes |
| `rate_limited` | Too many requests | Yes (with backoff) |
| `server_error` | Internal server error | Yes |
## Security Considerations
1. **Authentication**: All WebSocket connections must authenticate via token in initial HTTP upgrade request
2. **Authorization**: Server verifies read/write permissions before serving or accepting content
3. **Hash Verification**: Server recomputes SHA-256 of received content and rejects mismatches
4. **Size Limits**: Maximum file size enforced (configurable, default 10MB)
5. **Rate Limiting**: Sync requests limited per client (configurable, default 10/minute)
6. **Path Validation**: All file paths canonicalized and checked against allowlist
## Implementation Notes
### Hash Computation
```go
import "crypto/sha256"
func computeHash(content []byte) string {
h := sha256.Sum256(content)
return hex.EncodeToString(h[:])
}
```
### Merkle Root Computation
```go
func computeRootHash(files map[string]string) string {
// Sort paths for determinism
paths := sortedKeys(files)
hasher := sha256.New()
for _, path := range paths {
hasher.Write([]byte(path))
hasher.Write([]byte(files[path]))
}
return hex.EncodeToString(hasher.Sum(nil))
}
```
### WebSocket Connection
- Ping/pong every 30 seconds
- Connection timeout after 60 seconds without response
- Auto-reconnect with exponential backoff (1s, 2s, 4s, 8s, max 60s)
## Version
**Protocol Version**: 1.0
**Last Updated**: 2024-01-15

View File

@@ -0,0 +1,285 @@
# Sync Protocol v1 Specification
## Overview
The Native Sync Protocol enables bidirectional filesystem synchronization between a native macOS client and the cairnquire server. The browser client continues to operate via the existing WebSocket + REST API.
## Data Model
### Snapshot
A point-in-time tree of all files in the sync scope:
```json
{
"id": "snap:device-1:1234567890",
"deviceId": "device-1",
"createdAt": "2024-01-15T10:30:00Z",
"files": [
{
"path": "hello.md",
"hash": "a1b2c3...",
"size": 42,
"modified": "2024-01-15T10:00:00Z"
}
]
}
```
### Delta
A list of changes since a snapshot:
```json
{
"changes": [
{
"type": "create|update|delete|rename",
"path": "new-file.md",
"oldPath": "old-file.md",
"hash": "a1b2c3...",
"size": 42,
"modified": "2024-01-15T10:00:00Z"
}
]
}
```
### Conflict
When server and client both changed the same file since the last common snapshot:
```json
{
"path": "hello.md",
"serverHash": "a1b2c3...",
"clientHash": "d4e5f6...",
"serverModified": "2024-01-15T10:00:00Z",
"clientModified": "2024-01-15T10:05:00Z",
"strategy": "last-write-wins"
}
```
### Resolution Strategies
- `last-write-wins` (default): Use the most recently modified version
- `rename-both`: Keep both versions, renaming one as `file (conflict).md`
- `manual-merge`: Flag for UI resolution
- `server-wins`: Always use server version
- `client-wins`: Always use client version
## Endpoints
All endpoints require authentication via `X-API-Key` header.
### POST /api/sync/init
Initialize a new sync session.
**Request:**
```json
{
"deviceId": "macbook-pro-1",
"rootPath": "/Users/alice/Documents/md-hub"
}
```
**Response:**
```json
{
"snapshotId": "snap:macbook-pro-1:1234567890",
"serverSnapshot": [
{
"path": "hello.md",
"hash": "a1b2c3...",
"size": 42,
"modified": "2024-01-15T10:00:00Z"
}
]
}
```
### POST /api/sync/delta
Upload client changes and receive server changes + conflicts.
**Request:**
```json
{
"snapshotId": "snap:macbook-pro-1:1234567890",
"clientDelta": [
{
"type": "update",
"path": "hello.md",
"hash": "d4e5f6...",
"size": 50,
"modified": "2024-01-15T10:05:00Z"
}
]
}
```
**Response:**
```json
{
"serverDelta": [
{
"type": "create",
"path": "new-guide.md",
"hash": "g7h8i9...",
"size": 100,
"modified": "2024-01-15T10:02:00Z"
}
],
"conflicts": [
{
"path": "hello.md",
"serverHash": "a1b2c3...",
"clientHash": "d4e5f6...",
"serverModified": "2024-01-15T10:00:00Z",
"clientModified": "2024-01-15T10:05:00Z",
"strategy": "last-write-wins"
}
]
}
```
### POST /api/sync/resolve
Resolve conflicts and finalize the sync.
**Request:**
```json
{
"snapshotId": "snap:macbook-pro-1:1234567890",
"resolutions": [
{
"path": "hello.md",
"strategy": "server-wins"
}
]
}
```
**Response:**
```json
{
"newSnapshotId": "snap:macbook-pro-1:1234567999"
}
```
### GET /api/content/{hash}
Fetch raw file content by content hash (immutable, cache-friendly).
**Response:** Raw file bytes with headers:
```
Content-Type: application/octet-stream
Cache-Control: public, max-age=31536000, immutable
ETag: "a1b2c3..."
```
## Protocol Flow
```
1. Client calls POST /api/sync/init
→ Receives server snapshot
2. Client computes local delta
(fsevents + local state DB)
3. Client POSTs delta to /api/sync/delta
→ Receives serverDelta + conflicts
4. If conflicts exist:
a. Client resolves conflicts (auto or UI)
b. Client POSTs resolutions to /api/sync/resolve
→ Receives newSnapshotId
5. Client applies serverDelta to local filesystem
(downloads content via /api/content/{hash})
6. Both sides now have the same snapshot
```
## Authentication
Sync endpoints require API key authentication:
```
X-API-Key: <api-key>
```
API keys are long-lived tokens scoped to a user and device. They are stored in the `api_keys` table with SHA-256 hashes.
## Implementation Details
### Server Architecture
```
internal/sync/
├── protocol.go # Types and constants
├── service.go # Business logic (conflict resolution, snapshot management)
├── store.go # Database operations
```
### Database Schema
```sql
-- API keys for native client authentication
CREATE TABLE api_keys (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
name TEXT NOT NULL,
key_hash TEXT NOT NULL UNIQUE,
scopes TEXT NOT NULL DEFAULT 'sync:read,sync:write',
created_at TEXT NOT NULL,
expires_at TEXT,
last_used_at TEXT,
FOREIGN KEY(user_id) REFERENCES users(id) ON DELETE CASCADE
);
-- Sync snapshots (point-in-time file trees)
CREATE TABLE sync_snapshots (
id TEXT PRIMARY KEY,
device_id TEXT NOT NULL,
user_id TEXT NOT NULL,
created_at TEXT NOT NULL,
FOREIGN KEY(user_id) REFERENCES users(id) ON DELETE CASCADE
);
-- Files within a snapshot
CREATE TABLE sync_files (
snapshot_id TEXT NOT NULL,
path TEXT NOT NULL,
hash TEXT NOT NULL,
size_bytes INTEGER NOT NULL DEFAULT 0,
modified_at TEXT NOT NULL,
PRIMARY KEY (snapshot_id, path),
FOREIGN KEY(snapshot_id) REFERENCES sync_snapshots(id) ON DELETE CASCADE
);
```
### Conflict Detection
A conflict occurs when:
1. The server file existed in the snapshot
2. The server file has changed since the snapshot (hash changed)
3. The client also changed the same file (different hash in delta)
### Content-Addressed Storage
All file content is stored in a content-addressable store using SHA-256 hashes:
- Storage path: `<store-dir>/<hash[0:2]>/<hash[2:4]>/<hash>`
- Immutable: once stored, content never changes
- Deduplication: identical content shares the same storage
## Testing
The sync protocol includes table-driven tests that simulate client/server scenarios:
- `TestInitSyncCreatesSnapshot`: Validates snapshot generation
- `TestApplyDeltaDetectsServerChanges`: Detects server-side modifications
- `TestApplyDeltaDetectsConflicts`: Identifies concurrent modifications
- `TestResolveConflictsCreatesNewSnapshot`: Validates conflict resolution
- `TestGetContentReturnsFileBytes`: Tests content retrieval