317 lines
13 KiB
Markdown
317 lines
13 KiB
Markdown
# Architecture Overview
|
|
|
|
## System Context
|
|
|
|
MD Hub Secure is a self-hosted documentation platform. A single Go binary serves HTTP requests, manages a libsql database, watches a filesystem directory, and synchronizes content with connected web clients.
|
|
|
|
## Component Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ Client Layer │
|
|
│ │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────────┐ │
|
|
│ │ Browser │ │ File Editor │ │ Mobile / PWA │ │
|
|
│ │ (Preact SPA) │ │ (VS Code, │ │ (Responsive Web) │ │
|
|
│ │ │ │ Vim, etc.) │ │ │ │
|
|
│ └──────┬───────┘ └──────┬───────┘ └────────────┬─────────────┘ │
|
|
│ │ │ │ │
|
|
│ │ HTTP/WS │ File System │ HTTP/WS │
|
|
└─────────┼──────────────────┼────────────────────────┼────────────────┘
|
|
│ │ │
|
|
┌─────────┼──────────────────┼────────────────────────┼────────────────┐
|
|
│ │ │ Go Application Server │
|
|
│ │ │ │
|
|
│ ┌──────▼──────┐ ┌──────▼──────┐ │
|
|
│ │ HTTP Router │ │ File Watcher │ │
|
|
│ │ (chi) │ │ (fsnotify) │ │
|
|
│ └──────┬──────┘ └──────┬──────┘ │
|
|
│ │ │ │
|
|
│ ┌──────▼──────────────────▼──────┐ │
|
|
│ │ Request Processing │ │
|
|
│ │ ┌────────────┐ ┌───────────┐ │ │
|
|
│ │ │ Auth Layer │ │ Markdown │ │ │
|
|
│ │ │(WebAuthn) │ │ Renderer │ │ │
|
|
│ │ └────────────┘ └───────────┘ │ │
|
|
│ │ ┌────────────┐ ┌───────────┐ │ │
|
|
│ │ │ Sync Engine│ │ Search │ │ │
|
|
│ │ │(WebSocket) │ │ (FTS5) │ │ │
|
|
│ │ └────────────┘ └───────────┘ │ │
|
|
│ │ ┌────────────┐ ┌───────────┐ │ │
|
|
│ │ │ Email │ │ File │ │ │
|
|
│ │ │ (Postmark) │ │ Store │ │ │
|
|
│ │ └────────────┘ └───────────┘ │ │
|
|
│ └──────────┬─────────────────────┘ │
|
|
│ │ │
|
|
│ ┌──────────▼─────────────────────┐ │
|
|
│ │ Data Layer │ │
|
|
│ │ ┌────────────┐ ┌───────────┐ │ │
|
|
│ │ │ libsql │ │ Content- │ │ │
|
|
│ │ │ (SQLite) │ │ Addressed │ │ │
|
|
│ │ │ │ │ Files │ │ │
|
|
│ │ └────────────┘ └───────────┘ │ │
|
|
│ └────────────────────────────────┘ │
|
|
└────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Component Descriptions
|
|
|
|
### 1. HTTP Router (chi)
|
|
- Serves static assets (CSS, JS, fonts)
|
|
- Routes API requests to handlers
|
|
- Serves server-rendered HTML for document pages
|
|
- Enforces middleware chain: logging, recovery, CORS, CSP, auth
|
|
|
|
### 2. File Watcher (fsnotify)
|
|
- Watches configured directory for .md file changes
|
|
- Debounces rapid changes (1 second)
|
|
- Computes SHA-256 hash of new/changed files
|
|
- Triggers sync events to connected clients
|
|
|
|
### 3. Authentication Layer
|
|
- WebAuthn/passkey primary (FIDO2 compliant)
|
|
- Argon2id password fallback
|
|
- Session cookies: signed, HttpOnly, Secure, SameSite=Strict
|
|
- RBAC middleware checks permissions on every request
|
|
|
|
### 4. Markdown Renderer
|
|
- Goldmark parser with custom extensions:
|
|
- Wiki-links: `[[Page Name]]` and `[[Page Name|Display Text]]`
|
|
- Tags: `#tagname` parsed and linked
|
|
- Admonitions: `> [!NOTE]`, `> [!WARNING]`, etc.
|
|
- Mermaid diagrams: fenced code blocks with `mermaid` language
|
|
- Math: `$inline$` and `$$block$$` via KaTeX
|
|
- Server-side rendering produces semantic HTML
|
|
- Client-side hydration adds interactivity (folding, copy buttons)
|
|
|
|
### 5. Sync Engine (WebSocket)
|
|
- Implements SimpleSync protocol (see spec)
|
|
- Bidirectional: server pushes changes, clients request/pull
|
|
- Content-addressed: files identified by SHA-256 hash
|
|
- Conflict detection: three-way merge or manual resolution
|
|
- Offline queue: client stores pending changes in IndexedDB
|
|
|
|
### 6. Search (FTS5)
|
|
- SQLite FTS5 virtual table indexes document content
|
|
- Server provides search API for initial load
|
|
- Client syncs index to IndexedDB for offline search
|
|
- Index rebuilds incrementally on file changes
|
|
|
|
### 7. Email (Postmark)
|
|
- Go adapter using Postmark HTTP API
|
|
- Plain-text emails only (no HTML)
|
|
- Threading via Message-ID / In-Reply-To headers
|
|
- Webhook endpoint receives email replies, converts to comments
|
|
- Templates: notification, digest, conflict alert
|
|
|
|
### 8. File Store (Content-Addressed)
|
|
- Files stored by SHA-256 hash: `store/ab/cd/abcdef1234...`
|
|
- Immutable: once written, never modified
|
|
- Deduplication: identical content stored once
|
|
- Metadata in database maps filenames to current hash
|
|
- Attachments served with strict content-type validation
|
|
|
|
## Data Flow: Document Rendering
|
|
|
|
```
|
|
1. Request: GET /docs/getting-started
|
|
│
|
|
▼
|
|
2. Auth Middleware: Check session, verify read permission
|
|
│
|
|
▼
|
|
3. Document Lookup: Query DB for current hash of "getting-started.md"
|
|
│
|
|
▼
|
|
4. Cache Check: Is rendered HTML in cache? (TTL: 1 minute for dynamic)
|
|
│
|
|
├── Yes → Serve cached HTML
|
|
│
|
|
└── No → Continue
|
|
│
|
|
▼
|
|
5. File Read: Read content from `store/ab/cd/...hash...`
|
|
│
|
|
▼
|
|
6. Markdown Parse: Goldmark with extensions → AST
|
|
│
|
|
▼
|
|
7. Transform AST: Resolve wiki-links, extract tags, validate admonitions
|
|
│
|
|
▼
|
|
8. Render HTML: AST → html/template with layout
|
|
│
|
|
▼
|
|
9. Response: HTML with CSP nonce, Preact hydration marker
|
|
```
|
|
|
|
## Data Flow: File Sync
|
|
|
|
```
|
|
User saves file on laptop
|
|
│
|
|
▼
|
|
File Watcher detects change
|
|
│
|
|
▼
|
|
Compute SHA-256 hash
|
|
│
|
|
▼
|
|
Store in content-addressed filesystem
|
|
│
|
|
▼
|
|
Update database: filename → new hash
|
|
│
|
|
▼
|
|
Broadcast to WebSocket subscribers
|
|
│
|
|
▼
|
|
Connected clients receive update notification
|
|
│
|
|
▼
|
|
Client requests new content by hash (if not cached)
|
|
│
|
|
▼
|
|
Server sends file content
|
|
│
|
|
▼
|
|
Client verifies hash, updates UI
|
|
```
|
|
|
|
## Database Schema (Simplified)
|
|
|
|
```sql
|
|
-- Documents: current state and metadata
|
|
CREATE TABLE documents (
|
|
id TEXT PRIMARY KEY,
|
|
path TEXT UNIQUE NOT NULL, -- relative path like "getting-started.md"
|
|
current_hash TEXT NOT NULL, -- SHA-256 of current content
|
|
title TEXT NOT NULL,
|
|
created_at DATETIME NOT NULL,
|
|
updated_at DATETIME NOT NULL,
|
|
updated_by TEXT REFERENCES users(id),
|
|
permission_read TEXT NOT NULL DEFAULT 'public', -- public, authenticated, private
|
|
permission_write TEXT NOT NULL DEFAULT 'authenticated'
|
|
);
|
|
|
|
-- Document versions: immutable history
|
|
CREATE TABLE document_versions (
|
|
id TEXT PRIMARY KEY,
|
|
document_id TEXT NOT NULL REFERENCES documents(id),
|
|
hash TEXT NOT NULL, -- content hash
|
|
previous_hash TEXT, -- for diffing
|
|
created_at DATETIME NOT NULL,
|
|
created_by TEXT REFERENCES users(id),
|
|
change_summary TEXT, -- auto-generated or user-provided
|
|
signature TEXT -- optional Ed25519 signature
|
|
);
|
|
|
|
-- Users: minimal, no PII beyond email
|
|
CREATE TABLE users (
|
|
id TEXT PRIMARY KEY,
|
|
email TEXT UNIQUE NOT NULL,
|
|
display_name TEXT,
|
|
password_hash TEXT, -- NULL if passkey-only
|
|
passkey_credential_id TEXT, -- WebAuthn credential
|
|
created_at DATETIME NOT NULL,
|
|
last_seen_at DATETIME
|
|
);
|
|
|
|
-- Sessions: server-side state
|
|
CREATE TABLE sessions (
|
|
id TEXT PRIMARY KEY,
|
|
user_id TEXT NOT NULL REFERENCES users(id),
|
|
token_hash TEXT UNIQUE NOT NULL, -- SHA-256 of session token
|
|
created_at DATETIME NOT NULL,
|
|
expires_at DATETIME NOT NULL,
|
|
ip_address TEXT,
|
|
user_agent TEXT
|
|
);
|
|
|
|
-- Comments: linked to specific versions
|
|
CREATE TABLE comments (
|
|
id TEXT PRIMARY KEY,
|
|
document_id TEXT NOT NULL REFERENCES documents(id),
|
|
version_hash TEXT NOT NULL, -- which version this comments on
|
|
parent_id TEXT REFERENCES comments(id), -- threading
|
|
author_id TEXT NOT NULL REFERENCES users(id),
|
|
content TEXT NOT NULL,
|
|
created_at DATETIME NOT NULL,
|
|
resolved_at DATETIME,
|
|
resolved_by TEXT REFERENCES users(id)
|
|
);
|
|
|
|
-- Notifications: user preferences and queue
|
|
CREATE TABLE notification_settings (
|
|
user_id TEXT PRIMARY KEY REFERENCES users(id),
|
|
global_enabled BOOLEAN NOT NULL DEFAULT true,
|
|
digest_mode TEXT CHECK(digest_mode IN ('instant', 'hourly', 'daily')) DEFAULT 'instant',
|
|
email_enabled BOOLEAN NOT NULL DEFAULT true
|
|
);
|
|
|
|
CREATE TABLE document_watchers (
|
|
user_id TEXT REFERENCES users(id),
|
|
document_id TEXT REFERENCES documents(id),
|
|
folder_path TEXT, -- watch entire folder
|
|
created_at DATETIME NOT NULL,
|
|
PRIMARY KEY (user_id, document_id, folder_path)
|
|
);
|
|
|
|
-- FTS5 Search Index
|
|
CREATE VIRTUAL TABLE document_search USING fts5(
|
|
content,
|
|
document_id UNINDEXED,
|
|
tokenize='porter'
|
|
);
|
|
|
|
-- Audit Log: everything
|
|
CREATE TABLE audit_log (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
occurred_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
actor_id TEXT REFERENCES users(id),
|
|
action TEXT NOT NULL, -- read, write, delete, login, logout, sync
|
|
resource_type TEXT NOT NULL, -- document, user, comment, etc.
|
|
resource_id TEXT NOT NULL,
|
|
details TEXT, -- JSON metadata
|
|
ip_address TEXT
|
|
);
|
|
```
|
|
|
|
## Deployment Architecture
|
|
|
|
```yaml
|
|
# docker-compose.yml (simplified)
|
|
version: "3.8"
|
|
services:
|
|
app:
|
|
build: .
|
|
ports:
|
|
- "8080:8080"
|
|
volumes:
|
|
- ./data:/data # Database + attachments
|
|
- ./notes:/notes:ro # Source markdown files (optional)
|
|
environment:
|
|
- DB_PATH=/data/db.sqlite
|
|
- FILESTORE_PATH=/data/files
|
|
- NOTES_PATH=/notes
|
|
- POSTMARK_API_KEY=${POSTMARK_API_KEY}
|
|
- DOMAIN=${DOMAIN}
|
|
healthcheck:
|
|
test: ["CMD", "wget", "-q", "--spider", "http://localhost:8080/health"]
|
|
```
|
|
|
|
## Scaling Considerations
|
|
|
|
This architecture is intentionally single-node:
|
|
|
|
- **SQLite** handles read-heavy workloads excellently; write concurrency managed via WAL mode
|
|
- **Content-addressed files** are immutable and cache-friendly
|
|
- **Server-rendered HTML** reduces client compute
|
|
|
|
If horizontal scaling becomes necessary:
|
|
1. libsql replication (read replicas)
|
|
2. NFS or object storage for content-addressed files
|
|
3. Load balancer with sticky sessions for WebSocket
|
|
|
|
These are documented as future enhancements, not current requirements.
|