Update project docs and problem notes
This commit is contained in:
1
.monitor.pid
Normal file
1
.monitor.pid
Normal file
@@ -0,0 +1 @@
|
|||||||
|
96510
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# Getting Started
|
# MD Docs Secure
|
||||||
|
|
||||||
Welcome to **MD Hub Secure**.
|
Publish technical notes and documentation to a secure web application for collaboration with teams.
|
||||||
|
|
||||||
See [[Architecture Overview]] for the system shape and [[Project Plan]] for the milestone roadmap.
|
See [[Architecture Overview]] for the system shape and [[Project Plan]] for the milestone roadmap.
|
||||||
|
|
||||||
@@ -30,4 +30,3 @@ Block math:
|
|||||||
$$
|
$$
|
||||||
\int_0^1 x^2 dx = \frac{1}{3}
|
\int_0^1 x^2 dx = \frac{1}{3}
|
||||||
$$
|
$$
|
||||||
|
|
||||||
|
|||||||
1
docs/.last_state_hash
Normal file
1
docs/.last_state_hash
Normal file
@@ -0,0 +1 @@
|
|||||||
|
c3ce54e264212fa808c1110a4d276409
|
||||||
94
docs/README.md
Normal file
94
docs/README.md
Normal file
@@ -0,0 +1,94 @@
|
|||||||
|
# MD Hub Secure - Documentation Index
|
||||||
|
|
||||||
|
## Project Planning
|
||||||
|
|
||||||
|
| Document | Description |
|
||||||
|
|----------|-------------|
|
||||||
|
| [project-plan.md](project-plan.md) | Master project plan with milestones, timelines, and acceptance criteria |
|
||||||
|
| [architecture-overview.md](architecture-overview.md) | System architecture, component diagrams, and data flow |
|
||||||
|
|
||||||
|
## Architecture Decision Records (ADRs)
|
||||||
|
|
||||||
|
| Document | Decision |
|
||||||
|
|----------|----------|
|
||||||
|
| [ADR-001: Tech Stack](adrs/adr-001-tech-stack.md) | Go backend + Preact frontend |
|
||||||
|
| [ADR-002: Sync Protocol](adrs/adr-002-sync-protocol.md) | SimpleSync content-addressed protocol |
|
||||||
|
| [ADR-003: SSR Strategy](adrs/adr-003-ssr-strategy.md) | Go templates with Preact hydration |
|
||||||
|
| [ADR-004: Auth Strategy](adrs/adr-004-auth-strategy.md) | Passkeys primary, passwords fallback |
|
||||||
|
| [ADR-005: Storage Strategy](adrs/adr-005-storage-strategy.md) | libsql + content-addressed filesystem |
|
||||||
|
| [ADR-006: Email Strategy](adrs/adr-006-email-strategy.md) | Postmark gateway, plain-text only |
|
||||||
|
| [ADR-007: Security Model](adrs/adr-007-security-model.md) | Defense in depth, minimal dependencies |
|
||||||
|
|
||||||
|
## Milestones
|
||||||
|
|
||||||
|
| Document | Duration | Focus |
|
||||||
|
|----------|----------|-------|
|
||||||
|
| [Milestone 1: Foundation](milestones/milestone-01-foundation.md) | Weeks 1-2 | Server, database, markdown rendering |
|
||||||
|
| [Milestone 2: Sync Protocol](milestones/milestone-02-sync-protocol.md) | Weeks 3-4 | File sync, offline support, WebSocket |
|
||||||
|
| [Milestone 3: Authentication](milestones/milestone-03-authentication.md) | Weeks 5-6 | Passkeys, sessions, RBAC |
|
||||||
|
| [Milestone 4: Collaboration](milestones/milestone-04-collaboration.md) | Weeks 7-8 | Comments, email, conflict resolution |
|
||||||
|
| [Milestone 5: Search & UI](milestones/milestone-05-search-ui.md) | Weeks 9-10 | Full-text search, design system, responsive |
|
||||||
|
| [Milestone 6: Production](milestones/milestone-06-production.md) | Weeks 11-12 | Testing, security audit, ops docs |
|
||||||
|
|
||||||
|
## Technical Specifications
|
||||||
|
|
||||||
|
| Document | Topic |
|
||||||
|
|----------|-------|
|
||||||
|
| [SimpleSync Protocol](specs/simplesync-protocol.md) | Message formats, state machine, security |
|
||||||
|
| [Security Specification](specs/security-specification.md) | Threat model, controls, implementation |
|
||||||
|
| [Deployment Guide](specs/deployment-guide.md) | Docker setup, configuration, operations |
|
||||||
|
|
||||||
|
## Key Metrics
|
||||||
|
|
||||||
|
- **Timeline**: 12 weeks (3 months)
|
||||||
|
- **Test Coverage Target**: >80%
|
||||||
|
- **Performance Targets**:
|
||||||
|
- Initial page load: <100ms
|
||||||
|
- Search query: <50ms
|
||||||
|
- File sync: <5s
|
||||||
|
- **Security**: Zero critical/high vulnerabilities
|
||||||
|
- **Dependencies**: Minimal, all pinned and audited
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
1. Review all documentation
|
||||||
|
2. Approve project plan
|
||||||
|
3. Set up repository and CI pipeline
|
||||||
|
4. Begin Milestone 1: Foundation
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
md-hub-secure/
|
||||||
|
├── docs/
|
||||||
|
│ ├── README.md (this file)
|
||||||
|
│ ├── project-plan.md
|
||||||
|
│ ├── architecture-overview.md
|
||||||
|
│ ├── adrs/
|
||||||
|
│ │ ├── adr-001-tech-stack.md
|
||||||
|
│ │ ├── adr-002-sync-protocol.md
|
||||||
|
│ │ ├── adr-003-ssr-strategy.md
|
||||||
|
│ │ ├── adr-004-auth-strategy.md
|
||||||
|
│ │ ├── adr-005-storage-strategy.md
|
||||||
|
│ │ ├── adr-006-email-strategy.md
|
||||||
|
│ │ └── adr-007-security-model.md
|
||||||
|
│ ├── milestones/
|
||||||
|
│ │ ├── milestone-01-foundation.md
|
||||||
|
│ │ ├── milestone-02-sync-protocol.md
|
||||||
|
│ │ ├── milestone-03-authentication.md
|
||||||
|
│ │ ├── milestone-04-collaboration.md
|
||||||
|
│ │ ├── milestone-05-search-ui.md
|
||||||
|
│ │ └── milestone-06-production.md
|
||||||
|
│ └── specs/
|
||||||
|
│ ├── simplesync-protocol.md
|
||||||
|
│ ├── security-specification.md
|
||||||
|
│ └── deployment-guide.md
|
||||||
|
├── apps/
|
||||||
|
│ ├── server/ # Go application
|
||||||
|
│ └── web/ # Preact frontend
|
||||||
|
├── packages/
|
||||||
|
│ └── protocol/ # Shared types
|
||||||
|
├── docker-compose.yml
|
||||||
|
├── Dockerfile
|
||||||
|
└── Makefile
|
||||||
|
```
|
||||||
51
docs/STATUS.md
Normal file
51
docs/STATUS.md
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
# Project Status Report
|
||||||
|
|
||||||
|
**Generated:** 2026-04-29 09:35:52 UTC
|
||||||
|
**Monitoring Duration:** 9h 19m
|
||||||
|
**Iteration:** 56
|
||||||
|
**Branch:** codex/foundation-bootstrap
|
||||||
|
|
||||||
|
## Quick Stats
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
|--------|-------|
|
||||||
|
| Total Files | 95 |
|
||||||
|
| Go Files | 20 ( 2024 lines) |
|
||||||
|
| TypeScript Files | 3 ( 52 lines) |
|
||||||
|
| Markdown Files | 133 ( 27636 lines) |
|
||||||
|
| Docker Config | yes |
|
||||||
|
| Uncommitted Changes | 17 |
|
||||||
|
| Milestones Complete | 0/6 (0%) |
|
||||||
|
|
||||||
|
## Recent Activity
|
||||||
|
|
||||||
|
**Last Commit:** 8c454de - feat: add miller document layout (9 minutes ago)
|
||||||
|
|
||||||
|
## Milestone Progress
|
||||||
|
|
||||||
|
- [ ] Milestone 1: Foundation
|
||||||
|
- [ ] Milestone 2: Sync Protocol
|
||||||
|
- [ ] Milestone 3: Authentication
|
||||||
|
- [ ] Milestone 4: Collaboration
|
||||||
|
- [ ] Milestone 5: Search & UI
|
||||||
|
- [ ] Milestone 6: Production
|
||||||
|
|
||||||
|
**Current Focus:** Not Started
|
||||||
|
|
||||||
|
### Error Handling (10 ignored errors)
|
||||||
|
|
||||||
|
Several errors are being silently ignored. Consider handling or logging these.
|
||||||
|
|
||||||
|
## Notes for Human Review
|
||||||
|
|
||||||
|
>> **Active development.** 20 Go files with 2024 lines of code.
|
||||||
|
>> **17 uncommitted changes** detected. Codex may be mid-work.
|
||||||
|
>> **Code review items found.** See Action Items above.
|
||||||
|
|
||||||
|
## Files Changed This Session
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*This report auto-generated by MD Hub Secure progress monitor.*
|
||||||
|
*Next update in ~10 minutes or when filesystem changes detected.*
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
# MD Hub Secure - Project Plan
|
# MD Hub Secure - Project Plan
|
||||||
|
|
||||||
**Version:** 1.0
|
**Version:** 1.0\
|
||||||
**Date:** 2026-04-28
|
**Date:** 2026-04-28\
|
||||||
**Status:** Draft
|
**Status:** Draft
|
||||||
|
|
||||||
## Vision
|
## Vision
|
||||||
@@ -21,15 +21,17 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
|
|||||||
## Milestones
|
## Milestones
|
||||||
|
|
||||||
### Milestone 1: Foundation (Weeks 1-2)
|
### Milestone 1: Foundation (Weeks 1-2)
|
||||||
|
|
||||||
**Goal:** Working server that can serve markdown files as HTML
|
**Goal:** Working server that can serve markdown files as HTML
|
||||||
|
|
||||||
- Go HTTP server with chi router
|
- Go HTTP server with chi router
|
||||||
- libsql database with schema migration system
|
- libsql database with schema migration system
|
||||||
- Markdown-to-HTML pipeline with all extensions
|
- Markdown-to-HTML pipeline with most common extensions enabled
|
||||||
- Static file serving for attachments
|
- Static file serving for attachments
|
||||||
- Basic Dockerfile and docker-compose.yml
|
- Basic Dockerfile and docker-compose.yml
|
||||||
|
|
||||||
**Acceptance Criteria:**
|
**Acceptance Criteria:**
|
||||||
|
|
||||||
- [ ] Server starts and listens on configurable port
|
- [ ] Server starts and listens on configurable port
|
||||||
- [ ] Database auto-migrates on startup
|
- [ ] Database auto-migrates on startup
|
||||||
- [ ] Markdown files with wiki-links, tags, callouts, mermaid, math render correctly
|
- [ ] Markdown files with wiki-links, tags, callouts, mermaid, math render correctly
|
||||||
@@ -41,6 +43,7 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
|
|||||||
---
|
---
|
||||||
|
|
||||||
### Milestone 2: Sync Protocol (Weeks 3-4)
|
### Milestone 2: Sync Protocol (Weeks 3-4)
|
||||||
|
|
||||||
**Goal:** Bidirectional file sync between server and filesystem/web client
|
**Goal:** Bidirectional file sync between server and filesystem/web client
|
||||||
|
|
||||||
- Content-addressed storage (SHA-256 filesystem layout)
|
- Content-addressed storage (SHA-256 filesystem layout)
|
||||||
@@ -50,6 +53,7 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
|
|||||||
- Offline queue with conflict detection
|
- Offline queue with conflict detection
|
||||||
|
|
||||||
**Acceptance Criteria:**
|
**Acceptance Criteria:**
|
||||||
|
|
||||||
- [ ] File changes on disk automatically sync to server within 5 seconds
|
- [ ] File changes on disk automatically sync to server within 5 seconds
|
||||||
- [ ] Web client can sync after reconnection, even with changes made offline
|
- [ ] Web client can sync after reconnection, even with changes made offline
|
||||||
- [ ] Concurrent edits to different files succeed without conflict
|
- [ ] Concurrent edits to different files succeed without conflict
|
||||||
@@ -60,6 +64,7 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
|
|||||||
---
|
---
|
||||||
|
|
||||||
### Milestone 3: Authentication & Authorization (Weeks 5-6)
|
### Milestone 3: Authentication & Authorization (Weeks 5-6)
|
||||||
|
|
||||||
**Goal:** Secure access control with passkeys and granular permissions
|
**Goal:** Secure access control with passkeys and granular permissions
|
||||||
|
|
||||||
- Passkey (WebAuthn) primary authentication
|
- Passkey (WebAuthn) primary authentication
|
||||||
@@ -69,6 +74,7 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
|
|||||||
- Content signing with Ed25519 (optional per-user)
|
- Content signing with Ed25519 (optional per-user)
|
||||||
|
|
||||||
**Acceptance Criteria:**
|
**Acceptance Criteria:**
|
||||||
|
|
||||||
- [ ] Users can register and log in with passkeys on modern browsers
|
- [ ] Users can register and log in with passkeys on modern browsers
|
||||||
- [ ] Password fallback works with proper hashing
|
- [ ] Password fallback works with proper hashing
|
||||||
- [ ] Sessions expire securely and are revocable
|
- [ ] Sessions expire securely and are revocable
|
||||||
@@ -79,6 +85,7 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
|
|||||||
---
|
---
|
||||||
|
|
||||||
### Milestone 4: Collaboration Features (Weeks 7-8)
|
### Milestone 4: Collaboration Features (Weeks 7-8)
|
||||||
|
|
||||||
**Goal:** Comments, notifications, and conflict resolution
|
**Goal:** Comments, notifications, and conflict resolution
|
||||||
|
|
||||||
- Comment system linked to file versions (by hash)
|
- Comment system linked to file versions (by hash)
|
||||||
@@ -88,6 +95,7 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
|
|||||||
- Comment threading and email replies
|
- Comment threading and email replies
|
||||||
|
|
||||||
**Acceptance Criteria:**
|
**Acceptance Criteria:**
|
||||||
|
|
||||||
- [ ] Users can comment on specific lines/sections of a document
|
- [ ] Users can comment on specific lines/sections of a document
|
||||||
- [ ] Comments persist and are linked to document versions
|
- [ ] Comments persist and are linked to document versions
|
||||||
- [ ] Email notifications sent for watched file changes
|
- [ ] Email notifications sent for watched file changes
|
||||||
@@ -98,6 +106,7 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
|
|||||||
---
|
---
|
||||||
|
|
||||||
### Milestone 5: Search & UI Polish (Weeks 9-10)
|
### Milestone 5: Search & UI Polish (Weeks 9-10)
|
||||||
|
|
||||||
**Goal:** Fast search, design system, and production readiness
|
**Goal:** Fast search, design system, and production readiness
|
||||||
|
|
||||||
- Full-text search with SQLite FTS5
|
- Full-text search with SQLite FTS5
|
||||||
@@ -108,6 +117,7 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
|
|||||||
- Performance benchmarking
|
- Performance benchmarking
|
||||||
|
|
||||||
**Acceptance Criteria:**
|
**Acceptance Criteria:**
|
||||||
|
|
||||||
- [ ] Search returns results in <50ms on client
|
- [ ] Search returns results in <50ms on client
|
||||||
- [ ] Search works offline after initial sync
|
- [ ] Search works offline after initial sync
|
||||||
- [ ] Design system documented at `/design` route
|
- [ ] Design system documented at `/design` route
|
||||||
@@ -118,6 +128,7 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
|
|||||||
---
|
---
|
||||||
|
|
||||||
### Milestone 6: Production Hardening (Weeks 11-12)
|
### Milestone 6: Production Hardening (Weeks 11-12)
|
||||||
|
|
||||||
**Goal:** Deploy with confidence
|
**Goal:** Deploy with confidence
|
||||||
|
|
||||||
- Comprehensive test suite (unit, integration, e2e)
|
- Comprehensive test suite (unit, integration, e2e)
|
||||||
@@ -127,7 +138,8 @@ A minimal-dependency, ultra-fast documentation platform that publishes markdown
|
|||||||
- Documentation for operators
|
- Documentation for operators
|
||||||
|
|
||||||
**Acceptance Criteria:**
|
**Acceptance Criteria:**
|
||||||
- [ ] >80% test coverage
|
|
||||||
|
- [ ] 80% test coverage
|
||||||
- [ ] No critical or high-severity security findings
|
- [ ] No critical or high-severity security findings
|
||||||
- [ ] Backup script exports database + attachments as tar.gz
|
- [ ] Backup script exports database + attachments as tar.gz
|
||||||
- [ ] Health check endpoint returns status of all dependencies
|
- [ ] Health check endpoint returns status of all dependencies
|
||||||
@@ -177,44 +189,48 @@ See [architecture-overview.md](architecture-overview.md) for detailed component
|
|||||||
## Technology Stack
|
## Technology Stack
|
||||||
|
|
||||||
### Backend (Go 1.22+)
|
### Backend (Go 1.22+)
|
||||||
| Component | Library | Justification |
|
|
||||||
|-----------|---------|---------------|
|
| Component | Library | Justification |
|
||||||
| HTTP Router | `github.com/go-chi/chi` | Minimal, idiomatic, stdlib-compatible |
|
| ------------- | ------------------------------------------- | ---------------------------------------- |
|
||||||
| Database | `github.com/tursodatabase/libsql-client-go` | Embedded SQLite with replication option |
|
| HTTP Router | `github.com/go-chi/chi` | Minimal, idiomatic, stdlib-compatible |
|
||||||
| Migrations | `github.com/golang-migrate/migrate` | Battle-tested, version-controlled schema |
|
| Database | `github.com/tursodatabase/libsql-client-go` | Embedded SQLite with replication option |
|
||||||
| Auth/Passkeys | `github.com/go-webauthn/webauthn` | Reference implementation, well-audited |
|
| Migrations | `github.com/golang-migrate/migrate` | Battle-tested, version-controlled schema |
|
||||||
| Password Hash | `golang.org/x/crypto/argon2` | Winner of Password Hashing Competition |
|
| Auth/Passkeys | `github.com/go-webauthn/webauthn` | Reference implementation, well-audited |
|
||||||
| Markdown | `github.com/yuin/goldmark` + extensions | Fast, extensible, CommonMark compliant |
|
| Password Hash | `golang.org/x/crypto/argon2` | Winner of Password Hashing Competition |
|
||||||
| WebSocket | `github.com/gorilla/websocket` | Most widely used Go WS library |
|
| Markdown | `github.com/yuin/goldmark` + extensions | Fast, extensible, CommonMark compliant |
|
||||||
| File Watch | `github.com/fsnotify/fsnotify` | Cross-platform, stdlib-adjacent |
|
| WebSocket | `github.com/gorilla/websocket` | Most widely used Go WS library |
|
||||||
| Email | `github.com/keighl/postmark` + `net/smtp` | Postmark adapter + SMTP fallback |
|
| File Watch | `github.com/fsnotify/fsnotify` | Cross-platform, stdlib-adjacent |
|
||||||
| Crypto | `crypto/ed25519`, `crypto/sha256` | Standard library, no external deps |
|
| Email | `github.com/keighl/postmark` + `net/smtp` | Postmark adapter + SMTP fallback |
|
||||||
| Templates | `html/template` | Stdlib, auto-escaping, SSR-safe |
|
| Crypto | `crypto/ed25519`, `crypto/sha256` | Standard library, no external deps |
|
||||||
|
| Templates | `html/template` | Stdlib, auto-escaping, SSR-safe |
|
||||||
|
|
||||||
### Frontend (Preact + TypeScript)
|
### Frontend (Preact + TypeScript)
|
||||||
| Component | Library | Justification |
|
|
||||||
|-----------|---------|---------------|
|
| Component | Library | Justification |
|
||||||
| Framework | `preact` (10KB) | React API, tiny bundle, fast hydration |
|
| --------- | ------------------------------ | --------------------------------------- |
|
||||||
| Routing | `preact-iso` | Lightweight isomorphic routing |
|
| Framework | `preact` (10KB) | React API, tiny bundle, fast hydration |
|
||||||
| Markdown | `marked` + `micromark` | Server+client compatible |
|
| Routing | `preact-iso` | Lightweight isomorphic routing |
|
||||||
| Search | `flexsearch` | Client-side full-text, zero server load |
|
| Markdown | `marked` + `micromark` | Server+client compatible |
|
||||||
| Styling | Native CSS + custom properties | No build-time CSS processing needed |
|
| Search | `flexsearch` | Client-side full-text, zero server load |
|
||||||
| Icons | Inline SVG | No icon font dependency |
|
| Styling | Native CSS + custom properties | No build-time CSS processing needed |
|
||||||
| Build | `vite` | Fast HMR, modern output, minimal config |
|
| Icons | Inline SVG | No icon font dependency |
|
||||||
|
| Build | `vite` | Fast HMR, modern output, minimal config |
|
||||||
|
|
||||||
### Infrastructure
|
### Infrastructure
|
||||||
| Component | Technology | Justification |
|
|
||||||
|-----------|-----------|---------------|
|
| Component | Technology | Justification |
|
||||||
| Container | Docker + Compose | Universal deployment target |
|
| ------------- | ------------------ | --------------------------------------------- |
|
||||||
| Reverse Proxy | Traefik (optional) | Auto-HTTPS, included in compose |
|
| Container | Docker + Compose | Universal deployment target |
|
||||||
| Database | libsql/SQLite | Single file, zero external dependency |
|
| Reverse Proxy | Traefik (optional) | Auto-HTTPS, included in compose |
|
||||||
| Storage | Local filesystem | Content-addressed, immutable, backup-friendly |
|
| Database | libsql/SQLite | Single file, zero external dependency |
|
||||||
|
| Storage | Local filesystem | Content-addressed, immutable, backup-friendly |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Security Model
|
## Security Model
|
||||||
|
|
||||||
### Threats Addressed
|
### Threats Addressed
|
||||||
|
|
||||||
1. **Supply Chain Attacks**: Minimal external dependencies; all pinned with go.sum; vendored if needed
|
1. **Supply Chain Attacks**: Minimal external dependencies; all pinned with go.sum; vendored if needed
|
||||||
2. **Data Tampering**: All document versions SHA-256 hashed; optional Ed25519 signatures
|
2. **Data Tampering**: All document versions SHA-256 hashed; optional Ed25519 signatures
|
||||||
3. **Unauthorized Access**: RBAC with principle of least privilege; content never served without auth check
|
3. **Unauthorized Access**: RBAC with principle of least privilege; content never served without auth check
|
||||||
@@ -224,11 +240,13 @@ See [architecture-overview.md](architecture-overview.md) for detailed component
|
|||||||
7. **DoS**: Rate limiting on auth endpoints; file upload size limits; query timeouts
|
7. **DoS**: Rate limiting on auth endpoints; file upload size limits; query timeouts
|
||||||
|
|
||||||
### Content Security
|
### Content Security
|
||||||
|
|
||||||
- Attachment uploads: Magic number validation, extension whitelist, size limits
|
- Attachment uploads: Magic number validation, extension whitelist, size limits
|
||||||
- Markdown rendering: Sanitized HTML output; no raw HTML injection
|
- Markdown rendering: Sanitized HTML output; no raw HTML injection
|
||||||
- File paths: Canonicalization prevents directory traversal
|
- File paths: Canonicalization prevents directory traversal
|
||||||
|
|
||||||
### Audit & Compliance
|
### Audit & Compliance
|
||||||
|
|
||||||
- Every data modification logged with user, timestamp, IP, and content hash
|
- Every data modification logged with user, timestamp, IP, and content hash
|
||||||
- Immutable history table — updates are inserts
|
- Immutable history table — updates are inserts
|
||||||
- GDPR-compliant: full data export and deletion capability
|
- GDPR-compliant: full data export and deletion capability
|
||||||
@@ -237,14 +255,14 @@ See [architecture-overview.md](architecture-overview.md) for detailed component
|
|||||||
|
|
||||||
## Performance Targets
|
## Performance Targets
|
||||||
|
|
||||||
| Metric | Target | Measurement |
|
| Metric | Target | Measurement |
|
||||||
|--------|--------|-------------|
|
| ------------------- | ------------------------- | -------------------------------- |
|
||||||
| Initial Page Load | <100ms | TTFB + First Contentful Paint |
|
| Initial Page Load | <100ms | TTFB + First Contentful Paint |
|
||||||
| Time to Interactive | <200ms | Hydration complete |
|
| Time to Interactive | <200ms | Hydration complete |
|
||||||
| Search Query | <50ms | Client-side after initial sync |
|
| Search Query | <50ms | Client-side after initial sync |
|
||||||
| File Sync | <5s | From disk save to server persist |
|
| File Sync | <5s | From disk save to server persist |
|
||||||
| Bundle Size | <100KB | Gzipped JS + CSS |
|
| Bundle Size | <100KB | Gzipped JS + CSS |
|
||||||
| Concurrent Users | 1000 readers, 100 editors | Load tested |
|
| Concurrent Users | 1000 readers, 100 editors | Load tested |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -295,14 +313,14 @@ md-hub-secure/
|
|||||||
|
|
||||||
## Risk Register
|
## Risk Register
|
||||||
|
|
||||||
| Risk | Likelihood | Impact | Mitigation |
|
| Risk | Likelihood | Impact | Mitigation |
|
||||||
|------|-----------|--------|------------|
|
| -------------------------------- | ---------- | ------ | ----------------------------------------------------------------------------------- |
|
||||||
| WebAuthn/passkey adoption issues | Medium | Medium | Password fallback always available; extensive browser testing |
|
| WebAuthn/passkey adoption issues | Medium | Medium | Password fallback always available; extensive browser testing |
|
||||||
| SQLite performance at scale | Low | High | libsql replication path; FTS5 for search; benchmark early |
|
| SQLite performance at scale | Low | High | libsql replication path; FTS5 for search; benchmark early |
|
||||||
| Markdown extension parser bugs | Medium | Medium | Fuzz testing; strict CommonMark base; gradual feature rollout |
|
| Markdown extension parser bugs | Medium | Medium | Fuzz testing; strict CommonMark base; gradual feature rollout |
|
||||||
| Email deliverability | Medium | Medium | Postmark reputation monitoring; SPF/DKIM setup; SMTP fallback |
|
| Email deliverability | Medium | Medium | Postmark reputation monitoring; SPF/DKIM setup; SMTP fallback |
|
||||||
| Frontend bundle bloat | Medium | Medium | Bundle analysis in CI; tree-shaking enforced; lazy loading |
|
| Frontend bundle bloat | Medium | Medium | Bundle analysis in CI; tree-shaking enforced; lazy loading |
|
||||||
| File sync conflicts in teams | High | Medium | Clear UX for conflict resolution; email notification; automatic merge when possible |
|
| File sync conflicts in teams | High | Medium | Clear UX for conflict resolution; email notification; automatic merge when possible |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
272
docs/specs/deployment-guide.md
Normal file
272
docs/specs/deployment-guide.md
Normal file
@@ -0,0 +1,272 @@
|
|||||||
|
# Deployment Guide
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Docker 24.0+
|
||||||
|
- Docker Compose 2.20+
|
||||||
|
- 1GB RAM minimum (2GB recommended)
|
||||||
|
- 10GB disk space
|
||||||
|
- TLS certificate (Let's Encrypt or custom)
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone repository
|
||||||
|
git clone https://github.com/your-org/md-hub-secure.git
|
||||||
|
cd md-hub-secure
|
||||||
|
|
||||||
|
# Copy and edit configuration
|
||||||
|
cp .env.example .env
|
||||||
|
# Edit .env with your domain, email, Postmark API key
|
||||||
|
|
||||||
|
# Start services
|
||||||
|
docker-compose up -d
|
||||||
|
|
||||||
|
# Verify health
|
||||||
|
curl https://your-domain.com/health
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
| Variable | Required | Default | Description |
|
||||||
|
|----------|----------|---------|-------------|
|
||||||
|
| `DOMAIN` | Yes | — | Public domain name |
|
||||||
|
| `DB_PATH` | No | `/data/db.sqlite` | Database file path |
|
||||||
|
| `FILESTORE_PATH` | No | `/data/files` | Content-addressed file storage |
|
||||||
|
| `NOTES_PATH` | No | `/notes` | Watched markdown directory |
|
||||||
|
| `POSTMARK_API_KEY` | Yes | — | Postmark server API token |
|
||||||
|
| `POSTMARK_FROM` | No | `notifications@DOMAIN` | From email address |
|
||||||
|
| `SESSION_SECRET` | Yes | — | 32-byte hex for session signing |
|
||||||
|
| `ENV` | No | `production` | `development` or `production` |
|
||||||
|
|
||||||
|
### Docker Compose
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
version: "3.8"
|
||||||
|
|
||||||
|
services:
|
||||||
|
app:
|
||||||
|
build: .
|
||||||
|
ports:
|
||||||
|
- "8080:8080"
|
||||||
|
volumes:
|
||||||
|
- ./data:/data
|
||||||
|
- ./notes:/notes:ro
|
||||||
|
environment:
|
||||||
|
- DOMAIN=${DOMAIN}
|
||||||
|
- DB_PATH=/data/db.sqlite
|
||||||
|
- FILESTORE_PATH=/data/files
|
||||||
|
- NOTES_PATH=/notes
|
||||||
|
- POSTMARK_API_KEY=${POSTMARK_API_KEY}
|
||||||
|
- SESSION_SECRET=${SESSION_SECRET}
|
||||||
|
env_file:
|
||||||
|
- .env
|
||||||
|
restart: unless-stopped
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "wget", "-q", "--spider", "http://localhost:8080/health"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 3
|
||||||
|
start_period: 10s
|
||||||
|
security_opt:
|
||||||
|
- no-new-privileges:true
|
||||||
|
read_only: true
|
||||||
|
tmpfs:
|
||||||
|
- /tmp:noexec,nosuid,size=100m
|
||||||
|
|
||||||
|
# Optional: Traefik for auto-HTTPS
|
||||||
|
traefik:
|
||||||
|
image: traefik:v3.0
|
||||||
|
command:
|
||||||
|
- "--api.insecure=false"
|
||||||
|
- "--providers.docker=true"
|
||||||
|
- "--providers.docker.exposedbydefault=false"
|
||||||
|
- "--entrypoints.web.address=:80"
|
||||||
|
- "--entrypoints.websecure.address=:443"
|
||||||
|
- "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
|
||||||
|
- "--certificatesresolvers.letsencrypt.acme.email=${EMAIL}"
|
||||||
|
- "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
|
||||||
|
ports:
|
||||||
|
- "80:80"
|
||||||
|
- "443:443"
|
||||||
|
volumes:
|
||||||
|
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||||
|
- ./letsencrypt:/letsencrypt
|
||||||
|
labels:
|
||||||
|
- "traefik.enable=true"
|
||||||
|
restart: unless-stopped
|
||||||
|
```
|
||||||
|
|
||||||
|
## Production Checklist
|
||||||
|
|
||||||
|
### Pre-Deployment
|
||||||
|
- [ ] Change default `SESSION_SECRET` (generate: `openssl rand -hex 32`)
|
||||||
|
- [ ] Configure `POSTMARK_API_KEY` with valid token
|
||||||
|
- [ ] Set `DOMAIN` to public hostname
|
||||||
|
- [ ] Configure DNS A/AAAA records
|
||||||
|
- [ ] Set up SPF and DKIM for email domain
|
||||||
|
- [ ] Review `.env` file — no default secrets
|
||||||
|
|
||||||
|
### Security
|
||||||
|
- [ ] TLS 1.3 enforced (Traefik or reverse proxy)
|
||||||
|
- [ ] HSTS header configured
|
||||||
|
- [ ] Firewall rules: only 80/443 open
|
||||||
|
- [ ] Fail2ban or rate limiting on SSH
|
||||||
|
- [ ] Automatic security updates enabled
|
||||||
|
- [ ] Backup encryption key stored securely
|
||||||
|
|
||||||
|
### Monitoring
|
||||||
|
- [ ] Health check endpoint accessible
|
||||||
|
- [ ] Log rotation configured
|
||||||
|
- [ ] Disk space alerting (>80%)
|
||||||
|
- [ ] Memory usage alerting
|
||||||
|
- [ ] Uptime monitoring (external)
|
||||||
|
|
||||||
|
## Backup Procedures
|
||||||
|
|
||||||
|
### Automated Backup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# backup.sh
|
||||||
|
|
||||||
|
BACKUP_DIR="/backups"
|
||||||
|
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||||
|
BACKUP_FILE="$BACKUP_DIR/md-hub-$TIMESTAMP.tar.gz"
|
||||||
|
|
||||||
|
# Create backup
|
||||||
|
tar czf "$BACKUP_FILE" \
|
||||||
|
-C /data \
|
||||||
|
db.sqlite \
|
||||||
|
files/
|
||||||
|
|
||||||
|
# Encrypt (optional)
|
||||||
|
gpg --symmetric --cipher-algo AES256 "$BACKUP_FILE"
|
||||||
|
rm "$BACKUP_FILE"
|
||||||
|
|
||||||
|
# Keep only last 30 days
|
||||||
|
find "$BACKUP_DIR" -name "md-hub-*.tar.gz.gpg" -mtime +30 -delete
|
||||||
|
```
|
||||||
|
|
||||||
|
**Cron**: `0 2 * * * /opt/md-hub-secure/backup.sh`
|
||||||
|
|
||||||
|
### Restore from Backup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Stop application
|
||||||
|
docker-compose stop app
|
||||||
|
|
||||||
|
# Extract backup
|
||||||
|
tar xzf md-hub-20240115_020000.tar.gz -C /data
|
||||||
|
|
||||||
|
# Restart
|
||||||
|
docker-compose up -d app
|
||||||
|
|
||||||
|
# Verify
|
||||||
|
curl https://your-domain.com/health
|
||||||
|
```
|
||||||
|
|
||||||
|
## Upgrade Procedures
|
||||||
|
|
||||||
|
### Minor Update (patch version)
|
||||||
|
```bash
|
||||||
|
docker-compose pull
|
||||||
|
docker-compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
### Major Update
|
||||||
|
1. Read release notes for breaking changes
|
||||||
|
2. Create backup: `./backup.sh`
|
||||||
|
3. Pull new image: `docker-compose pull`
|
||||||
|
4. Run database migrations (auto on startup)
|
||||||
|
5. Update configuration if needed
|
||||||
|
6. Deploy: `docker-compose up -d`
|
||||||
|
7. Verify health and functionality
|
||||||
|
8. Rollback if issues: restore from backup
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Application Won't Start
|
||||||
|
```bash
|
||||||
|
# Check logs
|
||||||
|
docker-compose logs -f app
|
||||||
|
|
||||||
|
# Verify configuration
|
||||||
|
docker-compose exec app /server -check-config
|
||||||
|
|
||||||
|
# Test database
|
||||||
|
docker-compose exec app sqlite3 /data/db.sqlite ".tables"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Sync Not Working
|
||||||
|
```bash
|
||||||
|
# Check file watcher
|
||||||
|
docker-compose exec app ls -la /notes
|
||||||
|
|
||||||
|
# Verify WebSocket
|
||||||
|
curl -i -N \
|
||||||
|
-H "Connection: Upgrade" \
|
||||||
|
-H "Upgrade: websocket" \
|
||||||
|
-H "Sec-WebSocket-Key: $(openssl rand -base64 16)" \
|
||||||
|
-H "Sec-WebSocket-Version: 13" \
|
||||||
|
https://your-domain.com/ws
|
||||||
|
```
|
||||||
|
|
||||||
|
### Email Not Sending
|
||||||
|
```bash
|
||||||
|
# Test Postmark connection
|
||||||
|
curl -X POST \
|
||||||
|
https://api.postmarkapp.com/server \
|
||||||
|
-H "X-Postmark-Server-Token: YOUR_TOKEN" \
|
||||||
|
-H "Accept: application/json"
|
||||||
|
|
||||||
|
# Check email queue
|
||||||
|
docker-compose exec app sqlite3 /data/db.sqlite \
|
||||||
|
"SELECT * FROM email_queue WHERE status='failed'"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Scaling
|
||||||
|
|
||||||
|
### Vertical Scaling
|
||||||
|
- Increase container memory/CPU limits
|
||||||
|
- SQLite WAL mode handles concurrent reads well
|
||||||
|
- For heavy write loads, consider libsql replication
|
||||||
|
|
||||||
|
### Horizontal Scaling (Advanced)
|
||||||
|
```yaml
|
||||||
|
# Requires shared storage and load balancer
|
||||||
|
services:
|
||||||
|
app1:
|
||||||
|
# ... same config ...
|
||||||
|
environment:
|
||||||
|
- DB_PATH=/shared/db.sqlite
|
||||||
|
- FILESTORE_PATH=/shared/files
|
||||||
|
|
||||||
|
app2:
|
||||||
|
# ... same config ...
|
||||||
|
environment:
|
||||||
|
- DB_PATH=/shared/db.sqlite
|
||||||
|
- FILESTORE_PATH=/shared/files
|
||||||
|
|
||||||
|
nginx:
|
||||||
|
image: nginx:alpine
|
||||||
|
volumes:
|
||||||
|
- ./nginx.conf:/etc/nginx/nginx.conf
|
||||||
|
ports:
|
||||||
|
- "80:80"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note**: Horizontal scaling requires:
|
||||||
|
- Shared filesystem (NFS, EFS)
|
||||||
|
- libsql replication or external database
|
||||||
|
- Sticky sessions for WebSocket
|
||||||
|
|
||||||
|
## Support
|
||||||
|
|
||||||
|
- **Documentation**: `/docs` route (when authenticated)
|
||||||
|
- **Health Check**: `GET /health`
|
||||||
|
- **Metrics**: `GET /metrics` (Prometheus format)
|
||||||
|
- **Logs**: `docker-compose logs -f app`
|
||||||
|
- **Issues**: https://github.com/your-org/md-hub-secure/issues
|
||||||
456
docs/specs/security-specification.md
Normal file
456
docs/specs/security-specification.md
Normal file
@@ -0,0 +1,456 @@
|
|||||||
|
# Security Specification
|
||||||
|
|
||||||
|
## Threat Model
|
||||||
|
|
||||||
|
### Assets
|
||||||
|
1. **Document Content**: Markdown files and attachments
|
||||||
|
2. **User Credentials**: Passkeys, passwords, session tokens
|
||||||
|
3. **Collaboration Data**: Comments, notifications, history
|
||||||
|
4. **System State**: Database, configuration, logs
|
||||||
|
|
||||||
|
### Threat Actors
|
||||||
|
1. **External Attacker**: No system access, attempts remote exploitation
|
||||||
|
2. **Malicious User**: Authenticated, attempts unauthorized access
|
||||||
|
3. **Compromised Client**: Legitimate user's device infected
|
||||||
|
4. **Insider**: Administrator or developer with system access
|
||||||
|
|
||||||
|
### Threats
|
||||||
|
1. **Supply Chain Attack**: Malicious dependency injection
|
||||||
|
2. **Authentication Bypass**: Session hijacking, credential theft
|
||||||
|
3. **Authorization Bypass**: Horizontal/vertical privilege escalation
|
||||||
|
4. **Data Exfiltration**: Unauthorized document access
|
||||||
|
5. **Data Tampering**: Modification without authorization
|
||||||
|
6. **Denial of Service**: Resource exhaustion
|
||||||
|
7. **Information Disclosure**: Secrets in logs/responses
|
||||||
|
|
||||||
|
## Security Controls
|
||||||
|
|
||||||
|
### 1. Supply Chain Security
|
||||||
|
|
||||||
|
**Policy**:
|
||||||
|
- Only dependencies with >1M downloads/month or official Go team
|
||||||
|
- All versions pinned in go.sum
|
||||||
|
- No transitive dependencies without review
|
||||||
|
- Prefer standard library when possible
|
||||||
|
|
||||||
|
**Implementation**:
|
||||||
|
```
|
||||||
|
# go.mod
|
||||||
|
require (
|
||||||
|
github.com/go-chi/chi/v5 v5.0.12
|
||||||
|
github.com/tursodatabase/libsql-client-go v0.0.0-20240416075031-555ce55511f7
|
||||||
|
// All versions pinned
|
||||||
|
)
|
||||||
|
|
||||||
|
# Build
|
||||||
|
RUN go mod verify # Verify checksums
|
||||||
|
RUN go mod vendor # Optional: vendor for air-gapped
|
||||||
|
```
|
||||||
|
|
||||||
|
**Audit**:
|
||||||
|
- Weekly `govulncheck` scan
|
||||||
|
- Dependency review in CI
|
||||||
|
- No build-time network access (vendored or proxy with checksum)
|
||||||
|
|
||||||
|
### 2. Authentication Security
|
||||||
|
|
||||||
|
#### Passkey (WebAuthn)
|
||||||
|
```go
|
||||||
|
// Registration
|
||||||
|
func registerPasskey(user User) (*Credential, error) {
|
||||||
|
options, session, err := webauthn.BeginRegistration(user)
|
||||||
|
// Store session data server-side (never client-side)
|
||||||
|
// Return options to client
|
||||||
|
}
|
||||||
|
|
||||||
|
func verifyRegistration(sessionID string, response ProtocolCredentialCreation) (*Credential, error) {
|
||||||
|
session := getSessionData(sessionID)
|
||||||
|
credential, err := webauthn.FinishRegistration(user, session, response)
|
||||||
|
// Verify attestation
|
||||||
|
// Store credential: ID, public key, sign count
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Password (Argon2id)
|
||||||
|
```go
|
||||||
|
import "golang.org/x/crypto/argon2"
|
||||||
|
|
||||||
|
const (
|
||||||
|
timeCost = 3
|
||||||
|
memoryCost = 64 * 1024 // 64MB
|
||||||
|
parallelism = 4
|
||||||
|
saltLength = 16
|
||||||
|
keyLength = 32
|
||||||
|
)
|
||||||
|
|
||||||
|
func hashPassword(password string) (string, error) {
|
||||||
|
salt := make([]byte, saltLength)
|
||||||
|
if _, err := rand.Read(salt); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
|
||||||
|
hash := argon2.IDKey([]byte(password), salt, timeCost, memoryCost, parallelism, keyLength)
|
||||||
|
|
||||||
|
// Encode to string: $argon2id$v=19$m=65536,t=3,p=4$<salt>$<hash>
|
||||||
|
return encodeHash(salt, hash), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func verifyPassword(password, encodedHash string) (bool, error) {
|
||||||
|
salt, hash, params, err := decodeHash(encodedHash)
|
||||||
|
if err != nil {
|
||||||
|
return false, err
|
||||||
|
}
|
||||||
|
|
||||||
|
computedHash := argon2.IDKey([]byte(password), salt, params.time, params.memory, params.parallelism, uint32(len(hash)))
|
||||||
|
|
||||||
|
// Constant-time comparison
|
||||||
|
return subtle.ConstantTimeCompare(hash, computedHash) == 1, nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Session Management
|
||||||
|
```go
|
||||||
|
func createSession(userID string, r *http.Request) (*Session, error) {
|
||||||
|
token := make([]byte, 32)
|
||||||
|
if _, err := rand.Read(token); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
session := &Session{
|
||||||
|
ID: uuid.New().String(),
|
||||||
|
UserID: userID,
|
||||||
|
TokenHash: sha256.Sum256(token),
|
||||||
|
CreatedAt: time.Now(),
|
||||||
|
ExpiresAt: time.Now().Add(24 * time.Hour),
|
||||||
|
IP: r.RemoteAddr,
|
||||||
|
UserAgent: r.UserAgent(),
|
||||||
|
}
|
||||||
|
|
||||||
|
// Store in database
|
||||||
|
if err := db.CreateSession(session); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Set cookie
|
||||||
|
http.SetCookie(w, &http.Cookie{
|
||||||
|
Name: "session",
|
||||||
|
Value: base64.URLEncoding.EncodeToString(token),
|
||||||
|
Expires: session.ExpiresAt,
|
||||||
|
HttpOnly: true,
|
||||||
|
Secure: true, // HTTPS only
|
||||||
|
SameSite: http.SameSiteStrictMode,
|
||||||
|
Path: "/",
|
||||||
|
})
|
||||||
|
|
||||||
|
return session, nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Authorization
|
||||||
|
|
||||||
|
#### Permission Model
|
||||||
|
```go
|
||||||
|
type Permission string
|
||||||
|
|
||||||
|
const (
|
||||||
|
PermissionRead Permission = "read"
|
||||||
|
PermissionWrite Permission = "write"
|
||||||
|
PermissionAdmin Permission = "admin"
|
||||||
|
)
|
||||||
|
|
||||||
|
type ResourceType string
|
||||||
|
|
||||||
|
const (
|
||||||
|
ResourceGlobal ResourceType = "global"
|
||||||
|
ResourceCollection ResourceType = "collection"
|
||||||
|
ResourceDocument ResourceType = "document"
|
||||||
|
)
|
||||||
|
|
||||||
|
func checkPermission(userID string, resourceType ResourceType, resourceID string, required Permission) (bool, error) {
|
||||||
|
// Check explicit permission
|
||||||
|
perm, err := db.GetPermission(userID, resourceType, resourceID)
|
||||||
|
if err == nil && hasPermission(perm.Permission, required) {
|
||||||
|
return true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check collection permission for documents
|
||||||
|
if resourceType == ResourceDocument {
|
||||||
|
collectionID := getDocumentCollection(resourceID)
|
||||||
|
perm, err = db.GetPermission(userID, ResourceCollection, collectionID)
|
||||||
|
if err == nil && hasPermission(perm.Permission, required) {
|
||||||
|
return true, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check global permission
|
||||||
|
perm, err = db.GetPermission(userID, ResourceGlobal, "")
|
||||||
|
if err == nil && hasPermission(perm.Permission, required) {
|
||||||
|
return true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check document public status
|
||||||
|
if resourceType == ResourceDocument && required == PermissionRead {
|
||||||
|
doc, err := db.GetDocument(resourceID)
|
||||||
|
if err == nil && doc.PublicRead {
|
||||||
|
return true, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false, nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Middleware
|
||||||
|
```go
|
||||||
|
func RequirePermission(resourceType ResourceType, required Permission) func(http.Handler) http.Handler {
|
||||||
|
return func(next http.Handler) http.Handler {
|
||||||
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
userID := getUserID(r.Context())
|
||||||
|
resourceID := chi.URLParam(r, "id")
|
||||||
|
|
||||||
|
allowed, err := checkPermission(userID, resourceType, resourceID, required)
|
||||||
|
if err != nil || !allowed {
|
||||||
|
http.Error(w, "Forbidden", http.StatusForbidden)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
next.ServeHTTP(w, r)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Usage:
|
||||||
|
r.With(RequirePermission(ResourceDocument, PermissionRead)).
|
||||||
|
Get("/docs/{id}", getDocumentHandler)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Input Validation
|
||||||
|
|
||||||
|
#### Markdown Sanitization
|
||||||
|
```go
|
||||||
|
import "github.com/yuin/goldmark"
|
||||||
|
import "github.com/yuin/goldmark/extension"
|
||||||
|
import "github.com/yuin/goldmark/renderer/html"
|
||||||
|
|
||||||
|
func renderMarkdown(source []byte) ([]byte, error) {
|
||||||
|
md := goldmark.New(
|
||||||
|
goldmark.WithExtensions(
|
||||||
|
extension.Table,
|
||||||
|
extension.Strikethrough,
|
||||||
|
extension.Linkify,
|
||||||
|
// No raw HTML extension — we control HTML output
|
||||||
|
),
|
||||||
|
goldmark.WithRendererOptions(
|
||||||
|
html.WithHardWraps(),
|
||||||
|
html.WithXHTML(),
|
||||||
|
html.WithUnsafe(), // DISABLED — no raw HTML passthrough
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
if err := md.Convert(source, &buf); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return buf.Bytes(), nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### File Upload Validation
|
||||||
|
```go
|
||||||
|
func validateUpload(file io.Reader, filename string, maxSize int64) ([]byte, string, error) {
|
||||||
|
// Size limit
|
||||||
|
limited := io.LimitReader(file, maxSize+1)
|
||||||
|
data, err := io.ReadAll(limited)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", err
|
||||||
|
}
|
||||||
|
if int64(len(data)) > maxSize {
|
||||||
|
return nil, "", fmt.Errorf("file too large")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Magic number validation
|
||||||
|
contentType := detectContentType(data)
|
||||||
|
allowedTypes := map[string]bool{
|
||||||
|
"image/jpeg": true,
|
||||||
|
"image/png": true,
|
||||||
|
"image/gif": true,
|
||||||
|
"image/webp": true,
|
||||||
|
"application/pdf": true,
|
||||||
|
}
|
||||||
|
if !allowedTypes[contentType] {
|
||||||
|
return nil, "", fmt.Errorf("file type not allowed")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extension whitelist
|
||||||
|
ext := strings.ToLower(filepath.Ext(filename))
|
||||||
|
allowedExts := map[string]bool{
|
||||||
|
".jpg": true, ".jpeg": true, ".png": true,
|
||||||
|
".gif": true, ".webp": true, ".pdf": true,
|
||||||
|
}
|
||||||
|
if !allowedExts[ext] {
|
||||||
|
return nil, "", fmt.Errorf("file extension not allowed")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compute hash
|
||||||
|
hash := sha256.Sum256(data)
|
||||||
|
hashStr := hex.EncodeToString(hash[:])
|
||||||
|
|
||||||
|
return data, hashStr, nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Path Sanitization
|
||||||
|
```go
|
||||||
|
func sanitizePath(userPath string) (string, error) {
|
||||||
|
// Clean the path
|
||||||
|
clean := filepath.Clean(userPath)
|
||||||
|
|
||||||
|
// Prevent traversal
|
||||||
|
if strings.Contains(clean, "..") {
|
||||||
|
return "", fmt.Errorf("path traversal detected")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ensure it's relative
|
||||||
|
if filepath.IsAbs(clean) {
|
||||||
|
return "", fmt.Errorf("absolute paths not allowed")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Join with base and verify it's within base
|
||||||
|
fullPath := filepath.Join(baseDir, clean)
|
||||||
|
if !strings.HasPrefix(filepath.Clean(fullPath), filepath.Clean(baseDir)) {
|
||||||
|
return "", fmt.Errorf("path outside allowed directory")
|
||||||
|
}
|
||||||
|
|
||||||
|
return clean, nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Transport Security
|
||||||
|
|
||||||
|
#### TLS Configuration
|
||||||
|
```go
|
||||||
|
tlsConfig := &tls.Config{
|
||||||
|
MinVersion: tls.VersionTLS13,
|
||||||
|
CurvePreferences: []tls.CurveID{
|
||||||
|
tls.X25519,
|
||||||
|
tls.CurveP256,
|
||||||
|
},
|
||||||
|
CipherSuites: []uint16{
|
||||||
|
tls.TLS_AES_256_GCM_SHA384,
|
||||||
|
tls.TLS_CHACHA20_POLY1305_SHA256,
|
||||||
|
tls.TLS_AES_128_GCM_SHA256,
|
||||||
|
},
|
||||||
|
PreferServerCipherSuites: true,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Security Headers
|
||||||
|
```go
|
||||||
|
func securityHeaders(next http.Handler) http.Handler {
|
||||||
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Security-Policy",
|
||||||
|
"default-src 'self'; "+
|
||||||
|
"script-src 'self' 'nonce-{nonce}'; "+
|
||||||
|
"style-src 'self' 'unsafe-inline'; "+
|
||||||
|
"img-src 'self' data:; "+
|
||||||
|
"connect-src 'self' wss:; "+
|
||||||
|
"frame-ancestors 'none'; "+
|
||||||
|
"base-uri 'self'; "+
|
||||||
|
"form-action 'self';")
|
||||||
|
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||||
|
w.Header().Set("X-Frame-Options", "DENY")
|
||||||
|
w.Header().Set("X-XSS-Protection", "1; mode=block")
|
||||||
|
w.Header().Set("Referrer-Policy", "strict-origin-when-cross-origin")
|
||||||
|
w.Header().Set("Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload")
|
||||||
|
w.Header().Set("Permissions-Policy", "camera=(), microphone=(), geolocation=()")
|
||||||
|
|
||||||
|
next.ServeHTTP(w, r)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Operational Security
|
||||||
|
|
||||||
|
#### Logging
|
||||||
|
```go
|
||||||
|
func logAudit(event AuditEvent) {
|
||||||
|
entry := slog.With(
|
||||||
|
"event_type", event.Type,
|
||||||
|
"actor_id", redactIfSecret(event.ActorID),
|
||||||
|
"resource_type", event.ResourceType,
|
||||||
|
"resource_id", redactIfSecret(event.ResourceID),
|
||||||
|
"ip", event.IP,
|
||||||
|
"timestamp", event.Timestamp,
|
||||||
|
)
|
||||||
|
|
||||||
|
// Never log: passwords, tokens, session IDs, API keys
|
||||||
|
entry.Info("audit event")
|
||||||
|
}
|
||||||
|
|
||||||
|
func redactIfSecret(value string) string {
|
||||||
|
if looksLikeSecret(value) {
|
||||||
|
return "[REDACTED]"
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Container Security
|
||||||
|
```dockerfile
|
||||||
|
# Dockerfile
|
||||||
|
FROM golang:1.22-alpine AS builder
|
||||||
|
# ... build ...
|
||||||
|
|
||||||
|
FROM alpine:3.19
|
||||||
|
RUN addgroup -g 1000 appgroup && \
|
||||||
|
adduser -u 1000 -G appgroup -s /bin/sh -D appuser
|
||||||
|
|
||||||
|
# Read-only root filesystem
|
||||||
|
COPY --from=builder /app/server /server
|
||||||
|
COPY --from=builder /app/web/dist /web
|
||||||
|
|
||||||
|
# Data directory (writable)
|
||||||
|
RUN mkdir /data && chown appuser:appgroup /data
|
||||||
|
VOLUME /data
|
||||||
|
|
||||||
|
USER appuser
|
||||||
|
EXPOSE 8080
|
||||||
|
|
||||||
|
HEALTHCHECK --interval=30s --timeout=3s \
|
||||||
|
CMD wget -q --spider http://localhost:8080/health || exit 1
|
||||||
|
|
||||||
|
ENTRYPOINT ["/server"]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Incident Response
|
||||||
|
|
||||||
|
### Detection
|
||||||
|
- Automated alerts on: failed auth spikes, permission errors, large file uploads
|
||||||
|
- Log analysis for anomalies
|
||||||
|
- Health check failures
|
||||||
|
|
||||||
|
### Response
|
||||||
|
1. **Isolate**: Block IP, revoke sessions, disable account if compromised
|
||||||
|
2. **Assess**: Determine scope of breach, affected data
|
||||||
|
3. **Contain**: Patch vulnerability, rotate secrets
|
||||||
|
4. **Recover**: Restore from clean backup if necessary
|
||||||
|
5. **Document**: Post-incident review, update security measures
|
||||||
|
|
||||||
|
## Compliance
|
||||||
|
|
||||||
|
### GDPR
|
||||||
|
- Data minimization: only email stored, no other PII
|
||||||
|
- Right to erasure: `/api/user/delete` endpoint
|
||||||
|
- Data portability: `/api/user/export` endpoint
|
||||||
|
- Consent: explicit opt-in for email notifications
|
||||||
|
|
||||||
|
### Audit Trail
|
||||||
|
All security-relevant events logged:
|
||||||
|
- Authentication attempts (success/failure)
|
||||||
|
- Permission changes
|
||||||
|
- Document access (if private)
|
||||||
|
- Admin actions
|
||||||
|
- Configuration changes
|
||||||
|
|
||||||
|
Retention: 90 days active, 1 year archive
|
||||||
@@ -5,5 +5,3 @@
|
|||||||
- `go-libsql` is the active official Go binding, but it requires CGO and precompiled native libraries. That weakens the "same source -> same binary hash" goal until the exact build environment is locked down.
|
- `go-libsql` is the active official Go binding, but it requires CGO and precompiled native libraries. That weakens the "same source -> same binary hash" goal until the exact build environment is locked down.
|
||||||
- The embedded-replica connector in `go-libsql` rejects an empty primary URL. The foundation therefore uses the local `file:` libsql driver path when no primary is configured, and reserves embedded-replica setup for later sync work.
|
- The embedded-replica connector in `go-libsql` rejects an empty primary URL. The foundation therefore uses the local `file:` libsql driver path when no primary is configured, and reserves embedded-replica setup for later sync work.
|
||||||
- Docker verification is currently blocked by the local environment because the daemon socket at `/Users/tim/.orbstack/run/docker.sock` was not reachable during `docker build`.
|
- Docker verification is currently blocked by the local environment because the daemon socket at `/Users/tim/.orbstack/run/docker.sock` was not reachable during `docker build`.
|
||||||
- The docs do not define a config file format. This implementation uses JSON to keep the bootstrap dependency-free.
|
|
||||||
- The docs require "all Markdown extensions" but do not define a canonical sample corpus. A representative content sample is included and should be expanded as renderer coverage grows.
|
|
||||||
|
|||||||
67
scripts/README.md
Normal file
67
scripts/README.md
Normal file
@@ -0,0 +1,67 @@
|
|||||||
|
# MD Hub Secure - Development Scripts
|
||||||
|
|
||||||
|
This directory contains helper scripts for monitoring development progress.
|
||||||
|
|
||||||
|
## Scripts
|
||||||
|
|
||||||
|
### `install-monitor.sh`
|
||||||
|
Installs and starts the progress monitor as a background process.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scripts/install-monitor.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
The monitor will:
|
||||||
|
- Watch for filesystem changes every 10 minutes
|
||||||
|
- Update `docs/STATUS.md` with current progress
|
||||||
|
- Auto-exit if no changes detected for 10 minutes (Codex idle/finished)
|
||||||
|
- Provide code review notes and action items
|
||||||
|
|
||||||
|
### `monitor.sh`
|
||||||
|
The actual monitoring script. Usually called via `install-monitor.sh`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scripts/monitor.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Runs in foreground. Press Ctrl+C to stop.
|
||||||
|
|
||||||
|
### `status.sh`
|
||||||
|
Quick status check without starting the monitor.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scripts/status.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Shows:
|
||||||
|
- Whether monitor is running
|
||||||
|
- Current progress report from `docs/STATUS.md`
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
1. Start monitoring (run once):
|
||||||
|
```bash
|
||||||
|
./scripts/install-monitor.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Check status anytime:
|
||||||
|
```bash
|
||||||
|
./scripts/status.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
3. View live logs:
|
||||||
|
```bash
|
||||||
|
tail -f monitor.log
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Stop monitor:
|
||||||
|
```bash
|
||||||
|
kill $(cat .monitor.pid)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Files Generated
|
||||||
|
|
||||||
|
- `docs/STATUS.md` — Current project status and code review notes
|
||||||
|
- `docs/.last_state_hash` — Internal file for tracking changes
|
||||||
|
- `monitor.log` — Log output from monitor
|
||||||
|
- `.monitor.pid` — Process ID of running monitor
|
||||||
115
scripts/install-monitor.sh
Executable file
115
scripts/install-monitor.sh
Executable file
@@ -0,0 +1,115 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# MD Hub Secure - Install Monitor
|
||||||
|
# Sets up the progress monitor to run in the background
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_DIR="/Users/tim/developer/md-hub-secure"
|
||||||
|
MONITOR_SCRIPT="$SCRIPT_DIR/monitor.sh"
|
||||||
|
PID_FILE="$PROJECT_DIR/.monitor.pid"
|
||||||
|
LOG_FILE="$PROJECT_DIR/monitor.log"
|
||||||
|
|
||||||
|
echo "MD Hub Secure - Monitor Installation"
|
||||||
|
echo "===================================="
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check if monitor is already running
|
||||||
|
if [[ -f "$PID_FILE" ]]; then
|
||||||
|
OLD_PID=$(cat "$PID_FILE")
|
||||||
|
if kill -0 "$OLD_PID" 2>/dev/null; then
|
||||||
|
echo "Monitor is already running (PID: $OLD_PID)"
|
||||||
|
echo ""
|
||||||
|
echo "Options:"
|
||||||
|
echo " 1) Stop existing monitor and start fresh"
|
||||||
|
echo " 2) View current status"
|
||||||
|
echo " 3) Exit"
|
||||||
|
echo ""
|
||||||
|
read -p "Choose option [1-3]: " choice
|
||||||
|
|
||||||
|
case $choice in
|
||||||
|
1)
|
||||||
|
echo "Stopping existing monitor..."
|
||||||
|
kill "$OLD_PID" 2>/dev/null || true
|
||||||
|
rm -f "$PID_FILE"
|
||||||
|
;;
|
||||||
|
2)
|
||||||
|
echo ""
|
||||||
|
cat "$PROJECT_DIR/docs/STATUS.md" 2>/dev/null || echo "No status file yet."
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
3)
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Invalid option"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
else
|
||||||
|
# Stale PID file
|
||||||
|
rm -f "$PID_FILE"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Make sure monitor script exists
|
||||||
|
if [[ ! -f "$MONITOR_SCRIPT" ]]; then
|
||||||
|
echo "ERROR: Monitor script not found at $MONITOR_SCRIPT"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Ensure it's executable
|
||||||
|
chmod +x "$MONITOR_SCRIPT"
|
||||||
|
|
||||||
|
# Create necessary directories
|
||||||
|
mkdir -p "$PROJECT_DIR/docs"
|
||||||
|
mkdir -p "$PROJECT_DIR/scripts"
|
||||||
|
|
||||||
|
echo "Starting monitor in background..."
|
||||||
|
echo " Script: $MONITOR_SCRIPT"
|
||||||
|
echo " Log: $LOG_FILE"
|
||||||
|
echo " PID: $PID_FILE"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Start monitor in background
|
||||||
|
nohup "$MONITOR_SCRIPT" > "$LOG_FILE" 2>&1 &
|
||||||
|
MONITOR_PID=$!
|
||||||
|
|
||||||
|
# Save PID
|
||||||
|
echo $MONITOR_PID > "$PID_FILE"
|
||||||
|
|
||||||
|
sleep 2
|
||||||
|
|
||||||
|
# Check if it's actually running
|
||||||
|
if kill -0 "$MONITOR_PID" 2>/dev/null; then
|
||||||
|
echo "Monitor started successfully!"
|
||||||
|
echo " PID: $MONITOR_PID"
|
||||||
|
echo ""
|
||||||
|
echo "Commands:"
|
||||||
|
echo " View status: cat docs/STATUS.md"
|
||||||
|
echo " View log: tail -f monitor.log"
|
||||||
|
echo " Stop monitor: kill $(cat "$PID_FILE")"
|
||||||
|
echo " Restart: ./scripts/install-monitor.sh"
|
||||||
|
echo ""
|
||||||
|
echo "The monitor will:"
|
||||||
|
echo " - Check for filesystem changes every 10 minutes"
|
||||||
|
echo " - Update docs/STATUS.md with current progress"
|
||||||
|
echo " - Auto-exit if no changes detected (Codex idle/finished)"
|
||||||
|
echo " - Provide code review notes for Codex"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Generate initial status
|
||||||
|
echo "Generating initial status report..."
|
||||||
|
"$MONITOR_SCRIPT" > /dev/null 2>&1 || true
|
||||||
|
|
||||||
|
if [[ -f "$PROJECT_DIR/docs/STATUS.md" ]]; then
|
||||||
|
echo ""
|
||||||
|
echo "Current Status:"
|
||||||
|
echo "---------------"
|
||||||
|
head -30 "$PROJECT_DIR/docs/STATUS.md"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "ERROR: Monitor failed to start. Check $LOG_FILE"
|
||||||
|
rm -f "$PID_FILE"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
357
scripts/monitor.sh
Executable file
357
scripts/monitor.sh
Executable file
@@ -0,0 +1,357 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# MD Hub Secure - Progress Monitor
|
||||||
|
# Watches the project directory and updates docs/STATUS.md with progress
|
||||||
|
# Exits if no filesystem changes detected between iterations
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_DIR="/Users/tim/developer/md-hub-secure"
|
||||||
|
DOCS_DIR="$PROJECT_DIR/docs"
|
||||||
|
STATUS_FILE="$DOCS_DIR/STATUS.md"
|
||||||
|
HASH_FILE="$DOCS_DIR/.last_state_hash"
|
||||||
|
ITERATION=0
|
||||||
|
MAX_ITERATIONS=1000 # Safety limit
|
||||||
|
START_TIME=$(date +%s)
|
||||||
|
|
||||||
|
# Colors for terminal output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
log() {
|
||||||
|
echo -e "${BLUE}[$(date '+%Y-%m-%d %H:%M:%S')]${NC} $1" >&2
|
||||||
|
}
|
||||||
|
|
||||||
|
warn() {
|
||||||
|
echo -e "${YELLOW}[$(date '+%Y-%m-%d %H:%M:%S')] WARNING:${NC} $1" >&2
|
||||||
|
}
|
||||||
|
|
||||||
|
error() {
|
||||||
|
echo -e "${RED}[$(date '+%Y-%m-%d %H:%M:%S')] ERROR:${NC} $1" >&2
|
||||||
|
}
|
||||||
|
|
||||||
|
success() {
|
||||||
|
echo -e "${GREEN}[$(date '+%Y-%m-%d %H:%M:%S')]${NC} $1" >&2
|
||||||
|
}
|
||||||
|
|
||||||
|
# Compute hash of all tracked files (excluding node_modules, .git, etc.)
|
||||||
|
compute_state_hash() {
|
||||||
|
find "$PROJECT_DIR" \
|
||||||
|
-type f \
|
||||||
|
-not -path '*/node_modules/*' \
|
||||||
|
-not -path '*/.git/*' \
|
||||||
|
-not -path '*/data/*' \
|
||||||
|
-not -path '*/vendor/*' \
|
||||||
|
-not -name '.last_state_hash' \
|
||||||
|
-not -name 'STATUS.md' \
|
||||||
|
-not -name '*.lock' \
|
||||||
|
| sort \
|
||||||
|
| xargs stat -f '%m %N' 2>/dev/null \
|
||||||
|
| md5 -q
|
||||||
|
}
|
||||||
|
|
||||||
|
# Check if filesystem changed
|
||||||
|
has_changes() {
|
||||||
|
local current_hash
|
||||||
|
current_hash=$(compute_state_hash)
|
||||||
|
|
||||||
|
if [[ -f "$HASH_FILE" ]]; then
|
||||||
|
local last_hash
|
||||||
|
last_hash=$(cat "$HASH_FILE")
|
||||||
|
if [[ "$current_hash" == "$last_hash" ]]; then
|
||||||
|
return 1 # No changes
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "$current_hash" > "$HASH_FILE"
|
||||||
|
return 0 # Changes detected (or first run)
|
||||||
|
}
|
||||||
|
|
||||||
|
# Count lines of code by language
|
||||||
|
analyze_codebase() {
|
||||||
|
log "Analyzing codebase..."
|
||||||
|
|
||||||
|
local go_files=0
|
||||||
|
local go_lines=0
|
||||||
|
local ts_files=0
|
||||||
|
local ts_lines=0
|
||||||
|
local md_files=0
|
||||||
|
local md_lines=0
|
||||||
|
local total_files=0
|
||||||
|
local docker_files=0
|
||||||
|
|
||||||
|
# Count Go files
|
||||||
|
if command -v find &> /dev/null; then
|
||||||
|
go_files=$(find "$PROJECT_DIR" -name '*.go' -not -path '*/vendor/*' -not -path '*/node_modules/*' | wc -l)
|
||||||
|
go_lines=$(find "$PROJECT_DIR" -name '*.go' -not -path '*/vendor/*' -not -path '*/node_modules/*' -exec cat {} + 2>/dev/null | wc -l)
|
||||||
|
|
||||||
|
ts_files=$(find "$PROJECT_DIR" -name '*.ts' -o -name '*.tsx' | grep -v node_modules | wc -l)
|
||||||
|
ts_lines=$(find "$PROJECT_DIR" -name '*.ts' -o -name '*.tsx' | grep -v node_modules | xargs cat 2>/dev/null | wc -l)
|
||||||
|
|
||||||
|
md_files=$(find "$PROJECT_DIR" -name '*.md' | wc -l)
|
||||||
|
md_lines=$(find "$PROJECT_DIR" -name '*.md' -exec cat {} + 2>/dev/null | wc -l)
|
||||||
|
|
||||||
|
docker_files=$(find "$PROJECT_DIR" -name 'Dockerfile*' -o -name 'docker-compose*' | wc -l)
|
||||||
|
|
||||||
|
total_files=$(find "$PROJECT_DIR" -type f -not -path '*/node_modules/*' -not -path '*/.git/*' -not -path '*/data/*' | wc -l)
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "$go_files|$go_lines|$ts_files|$ts_lines|$md_files|$md_lines|$total_files|$docker_files"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Check git status if available
|
||||||
|
check_git_status() {
|
||||||
|
if [[ -d "$PROJECT_DIR/.git" ]]; then
|
||||||
|
cd "$PROJECT_DIR"
|
||||||
|
local branch=$(git branch --show-current 2>/dev/null || echo "unknown")
|
||||||
|
local uncommitted=$(git status --short 2>/dev/null | wc -l | tr -d ' ')
|
||||||
|
local last_commit=$(git log -1 --format="%h - %s (%cr)" 2>/dev/null || echo "No commits yet")
|
||||||
|
echo "$branch|$uncommitted|$last_commit"
|
||||||
|
else
|
||||||
|
echo "no-git|0|N/A"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Check if Docker is configured
|
||||||
|
check_docker() {
|
||||||
|
if [[ -f "$PROJECT_DIR/docker-compose.yml" ]]; then
|
||||||
|
echo "yes"
|
||||||
|
else
|
||||||
|
echo "no"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Check milestone progress
|
||||||
|
check_milestones() {
|
||||||
|
local completed=0
|
||||||
|
local in_progress=0
|
||||||
|
local total=6
|
||||||
|
|
||||||
|
if [[ -f "$DOCS_DIR/milestones/milestone-01-foundation.md" ]]; then
|
||||||
|
# Check for markers in milestone files
|
||||||
|
for i in $(seq 1 6); do
|
||||||
|
local file="$DOCS_DIR/milestones/milestone-0${i}-*.md"
|
||||||
|
if ls $file 1> /dev/null 2>&1; then
|
||||||
|
if grep -q "COMPLETED" $file 2>/dev/null; then
|
||||||
|
completed=$((completed + 1))
|
||||||
|
elif grep -q "IN_PROGRESS" $file 2>/dev/null; then
|
||||||
|
in_progress=$((in_progress + 1))
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "$completed|$in_progress|$total"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Generate code review comments
|
||||||
|
generate_code_review() {
|
||||||
|
log "Scanning for code review items..."
|
||||||
|
|
||||||
|
local review=""
|
||||||
|
|
||||||
|
# Check for TODO/FIXME/XXX in code
|
||||||
|
if find "$PROJECT_DIR" -name '*.go' -o -name '*.ts' -o -name '*.tsx' | grep -v node_modules | xargs grep -n 'TODO\|FIXME\|XXX\|HACK' 2>/dev/null | head -20 > /tmp/todos.txt; then
|
||||||
|
local todo_count=$(wc -l < /tmp/todos.txt | tr -d ' ')
|
||||||
|
if [[ $todo_count -gt 0 ]]; then
|
||||||
|
review+="### Action Items Found ($todo_count)\n\n"
|
||||||
|
review+="| File | Line | Comment |\n"
|
||||||
|
review+="|------|------|---------|\n"
|
||||||
|
while IFS=: read -r file line comment; do
|
||||||
|
# Extract just the filename
|
||||||
|
local fname=$(basename "$file")
|
||||||
|
review+="| $fname | $line | $comment |\n"
|
||||||
|
done < /tmp/todos.txt
|
||||||
|
review+="\n"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check for security issues
|
||||||
|
if find "$PROJECT_DIR" -name '*.go' | xargs grep -l 'http.Get\|http.Post\|ioutil.ReadFile' 2>/dev/null | head -5 > /tmp/security.txt; then
|
||||||
|
local sec_count=$(wc -l < /tmp/security.txt | tr -d ' ')
|
||||||
|
if [[ $sec_count -gt 0 ]]; then
|
||||||
|
review+="### Security Review Notes\n\n"
|
||||||
|
review+="- Found $sec_count files using potentially unsafe operations\n"
|
||||||
|
review+="- Ensure all HTTP requests have timeouts\n"
|
||||||
|
review+="- Validate all file paths before reading\n\n"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check for error handling
|
||||||
|
if find "$PROJECT_DIR" -name '*.go' | xargs grep -n '_ = ' 2>/dev/null | head -10 > /tmp/errors.txt; then
|
||||||
|
local err_count=$(wc -l < /tmp/errors.txt | tr -d ' ')
|
||||||
|
if [[ $err_count -gt 0 ]]; then
|
||||||
|
review+="### Error Handling ($err_count ignored errors)\n\n"
|
||||||
|
review+="Several errors are being silently ignored. Consider handling or logging these.\n\n"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo -e "$review"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Generate the status report
|
||||||
|
generate_status() {
|
||||||
|
local iteration=$1
|
||||||
|
local now=$(date '+%Y-%m-%d %H:%M:%S UTC')
|
||||||
|
local elapsed=$(( $(date +%s) - START_TIME ))
|
||||||
|
local elapsed_str="$((elapsed / 3600))h $(((elapsed % 3600) / 60))m"
|
||||||
|
|
||||||
|
# Gather metrics
|
||||||
|
IFS='|' read -r go_files go_lines ts_files ts_lines md_files md_lines total_files docker_files <<< "$(analyze_codebase)"
|
||||||
|
IFS='|' read -r git_branch uncommitted last_commit <<< "$(check_git_status)"
|
||||||
|
IFS='|' read -r completed in_progress total_milestones <<< "$(check_milestones)"
|
||||||
|
local has_docker=$(check_docker)
|
||||||
|
|
||||||
|
# Generate code review
|
||||||
|
local review=$(generate_code_review)
|
||||||
|
|
||||||
|
# Calculate progress percentage
|
||||||
|
local progress=$((completed * 100 / total_milestones))
|
||||||
|
|
||||||
|
cat > "$STATUS_FILE" << EOF
|
||||||
|
# Project Status Report
|
||||||
|
|
||||||
|
**Generated:** $now
|
||||||
|
**Monitoring Duration:** $elapsed_str
|
||||||
|
**Iteration:** $iteration
|
||||||
|
**Branch:** $git_branch
|
||||||
|
|
||||||
|
## Quick Stats
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
|--------|-------|
|
||||||
|
| Total Files | $total_files |
|
||||||
|
| Go Files | $go_files (${go_lines} lines) |
|
||||||
|
| TypeScript Files | $ts_files (${ts_lines} lines) |
|
||||||
|
| Markdown Files | $md_files (${md_lines} lines) |
|
||||||
|
| Docker Config | $has_docker |
|
||||||
|
| Uncommitted Changes | $uncommitted |
|
||||||
|
| Milestones Complete | $completed/$total_milestones (${progress}%) |
|
||||||
|
|
||||||
|
## Recent Activity
|
||||||
|
|
||||||
|
**Last Commit:** $last_commit
|
||||||
|
|
||||||
|
## Milestone Progress
|
||||||
|
|
||||||
|
- [ ] Milestone 1: Foundation
|
||||||
|
- [ ] Milestone 2: Sync Protocol
|
||||||
|
- [ ] Milestone 3: Authentication
|
||||||
|
- [ ] Milestone 4: Collaboration
|
||||||
|
- [ ] Milestone 5: Search & UI
|
||||||
|
- [ ] Milestone 6: Production
|
||||||
|
|
||||||
|
**Current Focus:** $([[ $in_progress -gt 0 ]] && echo "In Progress" || echo "Not Started")
|
||||||
|
|
||||||
|
$review
|
||||||
|
|
||||||
|
## Notes for Human Review
|
||||||
|
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Add contextual notes based on what we see
|
||||||
|
if [[ $go_files -eq 0 ]]; then
|
||||||
|
echo ">> **No Go files found.** Codex may be working on other aspects or hasn't started yet." >> "$STATUS_FILE"
|
||||||
|
elif [[ $go_files -lt 5 ]]; then
|
||||||
|
echo ">> **Early stage.** Only $go_files Go files exist. Foundation work likely in progress." >> "$STATUS_FILE"
|
||||||
|
else
|
||||||
|
echo ">> **Active development.** $go_files Go files with ${go_lines} lines of code." >> "$STATUS_FILE"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ $uncommitted -gt 0 ]]; then
|
||||||
|
echo ">> **$uncommitted uncommitted changes** detected. Codex may be mid-work." >> "$STATUS_FILE"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -n "$review" ]]; then
|
||||||
|
echo ">> **Code review items found.** See Action Items above." >> "$STATUS_FILE"
|
||||||
|
fi
|
||||||
|
|
||||||
|
cat >> "$STATUS_FILE" << EOF
|
||||||
|
|
||||||
|
## Files Changed This Session
|
||||||
|
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# List recently modified files
|
||||||
|
if [[ -f "$HASH_FILE" ]]; then
|
||||||
|
find "$PROJECT_DIR" \
|
||||||
|
-type f \
|
||||||
|
-mtime -0.007 \
|
||||||
|
-not -path '*/node_modules/*' \
|
||||||
|
-not -path '*/.git/*' \
|
||||||
|
-not -path '*/data/*' \
|
||||||
|
-not -name '.last_state_hash' \
|
||||||
|
-not -name 'STATUS.md' \
|
||||||
|
| sort | head -20 \
|
||||||
|
| sed 's|^| - |' >> "$STATUS_FILE" || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
cat >> "$STATUS_FILE" << EOF
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*This report auto-generated by MD Hub Secure progress monitor.*
|
||||||
|
*Next update in ~10 minutes or when filesystem changes detected.*
|
||||||
|
EOF
|
||||||
|
|
||||||
|
success "Status report updated: $STATUS_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main monitoring loop
|
||||||
|
main() {
|
||||||
|
log "Starting MD Hub Secure progress monitor..."
|
||||||
|
log "Project: $PROJECT_DIR"
|
||||||
|
log "Watching for filesystem changes every 10 minutes"
|
||||||
|
log "Press Ctrl+C to stop, or wait for auto-exit on inactivity"
|
||||||
|
|
||||||
|
# Ensure docs directory exists
|
||||||
|
mkdir -p "$DOCS_DIR"
|
||||||
|
|
||||||
|
while true; do
|
||||||
|
ITERATION=$((ITERATION + 1))
|
||||||
|
|
||||||
|
if [[ $ITERATION -gt $MAX_ITERATIONS ]]; then
|
||||||
|
warn "Maximum iterations ($MAX_ITERATIONS) reached. Stopping."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Iteration $ITERATION - checking for changes..."
|
||||||
|
|
||||||
|
if [[ $ITERATION -gt 1 ]] && ! has_changes; then
|
||||||
|
warn "No filesystem changes detected since last check."
|
||||||
|
warn "Stopping monitor (Codex may have finished or is inactive)."
|
||||||
|
|
||||||
|
# Add final note to status
|
||||||
|
cat >> "$STATUS_FILE" << EOF
|
||||||
|
|
||||||
|
## Monitor Stopped
|
||||||
|
|
||||||
|
**Stopped:** $(date '+%Y-%m-%d %H:%M:%S UTC')
|
||||||
|
**Reason:** No filesystem activity detected
|
||||||
|
|
||||||
|
No changes were observed between this check and the previous one.
|
||||||
|
This could mean:
|
||||||
|
- Codex has completed work
|
||||||
|
- Codex is paused or idle
|
||||||
|
- No files have been modified recently
|
||||||
|
|
||||||
|
To restart monitoring, run: `./scripts/monitor.sh`
|
||||||
|
EOF
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
generate_status $ITERATION
|
||||||
|
|
||||||
|
success "Iteration $ITERATION complete. Sleeping for 10 minutes..."
|
||||||
|
sleep 600 # 10 minutes
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# Handle signals
|
||||||
|
trap 'error "Monitor interrupted"; exit 1' INT TERM
|
||||||
|
|
||||||
|
# Run main loop
|
||||||
|
main
|
||||||
25
scripts/status.sh
Executable file
25
scripts/status.sh
Executable file
@@ -0,0 +1,25 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# MD Hub Secure - Quick status check
|
||||||
|
|
||||||
|
STATUS_FILE="/Users/tim/developer/md-hub-secure/docs/STATUS.md"
|
||||||
|
PID_FILE="/Users/tim/developer/md-hub-secure/.monitor.pid"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
if [[ -f "$PID_FILE" ]]; then
|
||||||
|
PID=$(cat "$PID_FILE")
|
||||||
|
if kill -0 "$PID" 2>/dev/null; then
|
||||||
|
echo "Monitor: RUNNING (PID: $PID)"
|
||||||
|
else
|
||||||
|
echo "Monitor: STOPPED (stale PID file)"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "Monitor: NOT RUNNING"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
if [[ -f "$STATUS_FILE" ]]; then
|
||||||
|
cat "$STATUS_FILE"
|
||||||
|
else
|
||||||
|
echo "No status file found. Run ./scripts/install-monitor.sh to start monitoring."
|
||||||
|
fi
|
||||||
Reference in New Issue
Block a user