298 lines
8.6 KiB
Markdown
298 lines
8.6 KiB
Markdown
# SimpleSync Protocol Specification
|
|
|
|
## Overview
|
|
|
|
SimpleSync is a content-addressed file synchronization protocol designed for asynchronous collaboration on markdown documents. It provides automatic bidirectional sync between a server (source of truth) and clients (filesystem watchers or web browsers).
|
|
|
|
## Design Goals
|
|
|
|
1. **Simplicity**: No branches, commits, or manual sync commands
|
|
2. **Automatic**: Changes propagate without user intervention
|
|
3. **Conflict-aware**: Detects concurrent edits, never silently loses data
|
|
4. **Offline-capable**: Queue changes locally, sync when connected
|
|
5. **Secure**: All content verified by cryptographic hash
|
|
|
|
## Concepts
|
|
|
|
### Content Addressing
|
|
Every file is identified by the SHA-256 hash of its content. The content is immutable — if a file changes, it gets a new hash.
|
|
|
|
### Merkle Tree
|
|
The directory state is represented as a Merkle tree where:
|
|
- Leaf nodes are file hashes
|
|
- Internal nodes are hashes of concatenated child hashes
|
|
- The root hash represents the entire directory state
|
|
|
|
### Sync State
|
|
Each client maintains:
|
|
- `root_hash`: Hash of current directory Merkle root
|
|
- `files`: Map of file paths to content hashes
|
|
- `pending`: Queue of local changes not yet synced
|
|
|
|
## Message Format
|
|
|
|
All messages are JSON over WebSocket.
|
|
|
|
### Client → Server
|
|
|
|
#### Sync Request
|
|
```json
|
|
{
|
|
"type": "sync_request",
|
|
"client_id": "uuid-v4",
|
|
"root_hash": "sha256-hex-64-chars",
|
|
"files": {
|
|
"getting-started.md": "sha256...",
|
|
"api-reference.md": "sha256..."
|
|
},
|
|
"timestamp": "2024-01-15T14:30:00Z"
|
|
}
|
|
```
|
|
|
|
#### Content Upload
|
|
```json
|
|
{
|
|
"type": "content_upload",
|
|
"hash": "sha256...",
|
|
"content": "base64-encoded-content",
|
|
"encoding": "base64"
|
|
}
|
|
```
|
|
|
|
#### Acknowledgment
|
|
```json
|
|
{
|
|
"type": "ack",
|
|
"message_id": "uuid-of-original-message",
|
|
"status": "ok"
|
|
}
|
|
```
|
|
|
|
### Server → Client
|
|
|
|
#### Sync Response
|
|
```json
|
|
{
|
|
"type": "sync_response",
|
|
"server_root_hash": "sha256...",
|
|
"missing_from_client": ["hash1", "hash2"],
|
|
"missing_from_server": ["hash3"],
|
|
"conflicts": ["getting-started.md"],
|
|
"timestamp": "2024-01-15T14:30:01Z"
|
|
}
|
|
```
|
|
|
|
#### Content Push
|
|
```json
|
|
{
|
|
"type": "content_push",
|
|
"hash": "sha256...",
|
|
"content": "base64-encoded-content",
|
|
"encoding": "base64",
|
|
"path": "getting-started.md"
|
|
}
|
|
```
|
|
|
|
#### Conflict Notification
|
|
```json
|
|
{
|
|
"type": "conflict",
|
|
"path": "getting-started.md",
|
|
"server_hash": "sha256...",
|
|
"client_hash": "sha256...",
|
|
"server_modified_at": "2024-01-15T14:30:00Z",
|
|
"client_modified_at": "2024-01-15T14:25:00Z"
|
|
}
|
|
```
|
|
|
|
#### Error
|
|
```json
|
|
{
|
|
"type": "error",
|
|
"code": "unauthorized",
|
|
"message": "Session expired",
|
|
"retryable": false
|
|
}
|
|
```
|
|
|
|
## Protocol Flow
|
|
|
|
### Normal Sync (No Conflicts)
|
|
|
|
```
|
|
Client Server
|
|
| |
|
|
|-- sync_request -------------->|
|
|
| root_hash: abc |
|
|
| |
|
|
| |-- Compare with server state
|
|
| |-- Calculate deltas
|
|
| |
|
|
|<- sync_response --------------|
|
|
| missing_from_client: [def] |
|
|
| missing_from_server: [] |
|
|
| conflicts: [] |
|
|
| |
|
|
|-- content_request(def) ------>|
|
|
| |
|
|
|<- content_push(def) ----------|
|
|
| |
|
|
|-- ack ----------------------->|
|
|
```
|
|
|
|
### Upload Changes
|
|
|
|
```
|
|
Client Server
|
|
| |
|
|
|-- sync_request -------------->|
|
|
| root_hash: abc |
|
|
| files: {a: hash1, b: hash2} |
|
|
| |
|
|
|<- sync_response --------------|
|
|
| missing_from_server: [hash2] |
|
|
| |
|
|
|-- content_upload(hash2) ----->|
|
|
| |
|
|
| |-- Verify hash
|
|
| |-- Store content
|
|
| |-- Update database
|
|
| |-- Broadcast to others
|
|
| |
|
|
|<- ack ------------------------|
|
|
| |
|
|
```
|
|
|
|
### Conflict Detection
|
|
|
|
```
|
|
Client A Server Client B
|
|
| | |
|
|
| | |-- Edit file X
|
|
| | |-- sync_request
|
|
| |-- Update file X |
|
|
| |-- Broadcast to A |
|
|
| | |
|
|
|-- Edit file X | |
|
|
|-- sync_request -------------->| |
|
|
| |-- Detect conflict |
|
|
| |-- A has old hash |
|
|
| |-- B has new hash |
|
|
|<- conflict -------------------| |
|
|
| path: X | |
|
|
| server_hash: B_hash | |
|
|
| client_hash: A_hash | |
|
|
| | |
|
|
```
|
|
|
|
## State Machine
|
|
|
|
```
|
|
+--------+ sync_request
|
|
| +-----------+
|
|
| Idle | |
|
|
| |<----------+
|
|
+---+----+
|
|
|
|
|
| connect
|
|
v
|
|
+---------+---------+
|
|
| |
|
|
| Connected |
|
|
| |
|
|
+---------+---------+
|
|
|
|
|
| sync_request
|
|
v
|
|
+---------+---------+
|
|
| |
|
|
| Comparing |
|
|
| |
|
|
+---------+---------+
|
|
|
|
|
+-----------+-----------+
|
|
| |
|
|
| has_changes | no_changes
|
|
v v
|
|
+---------+---------+ +---------+---------+
|
|
| | | |
|
|
| Transferring | | Idle |
|
|
| | | |
|
|
+---------+---------+ +-------------------+
|
|
|
|
|
| complete
|
|
v
|
|
+---------+---------+
|
|
| |
|
|
| Idle |
|
|
| |
|
|
+-------------------+
|
|
^
|
|
| conflict
|
|
|
|
|
+---------+---------+
|
|
| |
|
|
| Conflicted |
|
|
| |
|
|
+-------------------+
|
|
```
|
|
|
|
## Error Codes
|
|
|
|
| Code | Description | Retryable |
|
|
|------|-------------|-----------|
|
|
| `unauthorized` | Session expired or invalid | No (re-authenticate) |
|
|
| `forbidden` | Insufficient permissions | No |
|
|
| `not_found` | Requested content hash unknown | Yes |
|
|
| `too_large` | Content exceeds size limit | No |
|
|
| `hash_mismatch` | Content doesn't match claimed hash | Yes |
|
|
| `rate_limited` | Too many requests | Yes (with backoff) |
|
|
| `server_error` | Internal server error | Yes |
|
|
|
|
## Security Considerations
|
|
|
|
1. **Authentication**: All WebSocket connections must authenticate via token in initial HTTP upgrade request
|
|
2. **Authorization**: Server verifies read/write permissions before serving or accepting content
|
|
3. **Hash Verification**: Server recomputes SHA-256 of received content and rejects mismatches
|
|
4. **Size Limits**: Maximum file size enforced (configurable, default 10MB)
|
|
5. **Rate Limiting**: Sync requests limited per client (configurable, default 10/minute)
|
|
6. **Path Validation**: All file paths canonicalized and checked against allowlist
|
|
|
|
## Implementation Notes
|
|
|
|
### Hash Computation
|
|
```go
|
|
import "crypto/sha256"
|
|
|
|
func computeHash(content []byte) string {
|
|
h := sha256.Sum256(content)
|
|
return hex.EncodeToString(h[:])
|
|
}
|
|
```
|
|
|
|
### Merkle Root Computation
|
|
```go
|
|
func computeRootHash(files map[string]string) string {
|
|
// Sort paths for determinism
|
|
paths := sortedKeys(files)
|
|
|
|
hasher := sha256.New()
|
|
for _, path := range paths {
|
|
hasher.Write([]byte(path))
|
|
hasher.Write([]byte(files[path]))
|
|
}
|
|
|
|
return hex.EncodeToString(hasher.Sum(nil))
|
|
}
|
|
```
|
|
|
|
### WebSocket Connection
|
|
- Ping/pong every 30 seconds
|
|
- Connection timeout after 60 seconds without response
|
|
- Auto-reconnect with exponential backoff (1s, 2s, 4s, 8s, max 60s)
|
|
|
|
## Version
|
|
|
|
**Protocol Version**: 1.0
|
|
**Last Updated**: 2024-01-15
|