Compare commits
7 Commits
040f0d2d15
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
559ca40238 | ||
|
|
772f2199b7 | ||
|
|
3d9cb7507a | ||
|
|
003b467a21 | ||
|
|
1be98c62bc | ||
|
|
73e80f6533 | ||
|
|
88cf1003a8 |
322
agents.md
Normal file
322
agents.md
Normal file
@@ -0,0 +1,322 @@
|
|||||||
|
# Homelab Infrastructure & Deployment Notes
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
This document contains comprehensive notes about the homelab Docker Swarm cluster setup, troubleshooting procedures, and deployment configurations.
|
||||||
|
|
||||||
|
## Infrastructure Architecture
|
||||||
|
|
||||||
|
### Host Configuration
|
||||||
|
- **Main Node**: `tpi-n1` (192.168.2.130) - Docker Swarm Manager
|
||||||
|
- **Worker Node**: `tpi-n2` - Docker Swarm Worker
|
||||||
|
- **Storage**:
|
||||||
|
- Main drive: `/dev/mmcblk0p2` (29GB) - System and applications
|
||||||
|
- NVMe drive: `/mnt/nvme` (916GB) - Docker data storage
|
||||||
|
|
||||||
|
### Docker Configuration
|
||||||
|
- **Data Directory**: `/mnt/nvme/docker` (moved from `/var/lib/docker`)
|
||||||
|
- **DNS Configuration**: Local dnsmasq forwarder at 127.0.0.1
|
||||||
|
- **External DNS**: 8.8.8.8, 8.8.4.4, 1.1.1.1
|
||||||
|
- **Docker Daemon Config**: `/etc/docker/daemon.json`
|
||||||
|
|
||||||
|
### Services Running
|
||||||
|
- **Traefik**: Load balancer and SSL termination
|
||||||
|
- **Dokploy**: Deployment management (port 3000)
|
||||||
|
- **Gitea**: Git server (port 2222 for SSH)
|
||||||
|
- **Swarmpit**: Docker Swarm management UI (port 888)
|
||||||
|
- **Bendtstudio**: Main web application (5 replicas)
|
||||||
|
- **MariaDB**: Database for Pancake application
|
||||||
|
|
||||||
|
## Major Maintenance Tasks
|
||||||
|
|
||||||
|
### 1. Docker Data Migration (Completed ✅)
|
||||||
|
**Problem**: Main drive 100% full (28G/29G used)
|
||||||
|
**Solution**: Moved 19GB Docker data to NVMe drive
|
||||||
|
|
||||||
|
**Commands Used**:
|
||||||
|
```bash
|
||||||
|
# Stop Docker services
|
||||||
|
sudo systemctl stop docker docker.socket
|
||||||
|
|
||||||
|
# Move data to NVMe
|
||||||
|
sudo cp -a /var/lib/docker /mnt/nvme/
|
||||||
|
|
||||||
|
# Update Docker config
|
||||||
|
echo '{"data-root": "/mnt/nvme/docker"}' | sudo tee /etc/docker/daemon.json
|
||||||
|
|
||||||
|
# Restart Docker
|
||||||
|
sudo systemctl start docker
|
||||||
|
```
|
||||||
|
|
||||||
|
**Result**:
|
||||||
|
- Freed 15GB on main drive (100% → 46% usage)
|
||||||
|
- Docker data on fast NVMe storage
|
||||||
|
- All services maintained without downtime
|
||||||
|
|
||||||
|
### 2. Nginx Configuration Fix (Completed ✅)
|
||||||
|
**Problem**: Pancake static assets returning 404, pretty URLs not working
|
||||||
|
**Root Cause**: Apache .htaccess rules not translated to nginx properly
|
||||||
|
|
||||||
|
**Files Modified**:
|
||||||
|
- `nginx.template.conf` - Main configuration template
|
||||||
|
- All running containers - Updated nginx configuration
|
||||||
|
|
||||||
|
**Key Changes**:
|
||||||
|
```nginx
|
||||||
|
# Fixed static asset paths
|
||||||
|
location /pancake/third_party {
|
||||||
|
alias ${NIXPACKS_PHP_ROOT_DIR}/pancake/third_party;
|
||||||
|
# ... caching headers
|
||||||
|
}
|
||||||
|
|
||||||
|
# Added pretty URL support
|
||||||
|
location /pancake {
|
||||||
|
try_files $uri $uri/ @pancake_fallback;
|
||||||
|
}
|
||||||
|
|
||||||
|
location @pancake_fallback {
|
||||||
|
rewrite ^.*$ /pancake/index.php last;
|
||||||
|
}
|
||||||
|
|
||||||
|
# Fixed PHP handling
|
||||||
|
location ~ \.php$ {
|
||||||
|
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
|
||||||
|
# ... other fastcgi params
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Result**:
|
||||||
|
- ✅ Static assets serving correctly
|
||||||
|
- ✅ Pretty URLs working (bendtstudio.com/pancake/admin)
|
||||||
|
- ✅ Apache .htaccess functionality replicated in nginx
|
||||||
|
|
||||||
|
### 3. DNS Resolution Fix (Completed ✅)
|
||||||
|
**Problem**: Docker containers couldn't resolve `ghcr.io` causing build failures
|
||||||
|
**Root Cause**: No proper DNS forwarding for containers
|
||||||
|
|
||||||
|
**Solution Implemented**:
|
||||||
|
```bash
|
||||||
|
# Install dnsmasq
|
||||||
|
sudo apt install -y dnsmasq
|
||||||
|
|
||||||
|
# Configure DNS forwarding
|
||||||
|
cat > /etc/dnsmasq.conf << EOF
|
||||||
|
server=8.8.8.8
|
||||||
|
server=8.8.4.4
|
||||||
|
server=1.1.1.1
|
||||||
|
listen-address=127.0.0.1
|
||||||
|
bind-interfaces
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Start dnsmasq
|
||||||
|
sudo systemctl enable dnsmasq && sudo systemctl start dnsmasq
|
||||||
|
|
||||||
|
# Update system DNS
|
||||||
|
echo 'nameserver 127.0.0.1' | sudo tee /etc/resolv.conf
|
||||||
|
|
||||||
|
# Update Docker to use local DNS
|
||||||
|
echo '{"data-root": "/mnt/nvme/docker", "dns": ["127.0.0.1"]}' | sudo tee /etc/docker/daemon.json
|
||||||
|
sudo systemctl restart docker
|
||||||
|
```
|
||||||
|
|
||||||
|
**Result**:
|
||||||
|
- ✅ ghcr.io resolution working
|
||||||
|
- ✅ Docker builds successful
|
||||||
|
- ✅ Deployments working via Dokploy UI
|
||||||
|
|
||||||
|
## Service-Specific Notes
|
||||||
|
|
||||||
|
### Gitea (Git Server)
|
||||||
|
- **SSH Access**: `git@gitea.bendtstudio.com:2222` or `git@gitea.bendtstudio.com:username/repo.git`
|
||||||
|
- **Web UI**: https://gitea.bendtstudio.com
|
||||||
|
- **Status**: Working correctly, SSH authentication successful
|
||||||
|
- **Note**: "No shell access" message is normal for Gitea
|
||||||
|
|
||||||
|
### Dokploy (Deployment Management)
|
||||||
|
- **Web UI**: https://dokploy.bendtstudio.com (port 3000)
|
||||||
|
- **Usage**:
|
||||||
|
1. Push code to Gitea repository
|
||||||
|
2. Dokploy automatically detects new commits
|
||||||
|
3. Trigger manual redeployment via web UI
|
||||||
|
4. Monitor build logs in real-time
|
||||||
|
- **Build Process**: Uses Nixpacks for containerization
|
||||||
|
- **Current Status**: ✅ Working with DNS fix
|
||||||
|
|
||||||
|
### Bendtstudio Web Application
|
||||||
|
- **Domain**: https://bendtstudio.com
|
||||||
|
- **Pancake App**: https://bendtstudio.com/pancake
|
||||||
|
- **Replicas**: 5 containers for load balancing
|
||||||
|
- **Static Assets**: All serving correctly from `/pancake/third_party/`
|
||||||
|
- **Database**: MariaDB container for Pancake data
|
||||||
|
|
||||||
|
## Troubleshooting Procedures
|
||||||
|
|
||||||
|
### Docker Issues
|
||||||
|
```bash
|
||||||
|
# Check Docker status
|
||||||
|
sudo systemctl status docker
|
||||||
|
|
||||||
|
# Check container logs
|
||||||
|
docker logs <container_name>
|
||||||
|
|
||||||
|
# Check service status
|
||||||
|
docker service ls
|
||||||
|
|
||||||
|
# Restart Docker daemon
|
||||||
|
sudo systemctl restart docker
|
||||||
|
```
|
||||||
|
|
||||||
|
### DNS Issues
|
||||||
|
```bash
|
||||||
|
# Check DNS resolution
|
||||||
|
nslookup ghcr.io
|
||||||
|
|
||||||
|
# Test from container
|
||||||
|
docker exec <container> curl -I https://ghcr.io
|
||||||
|
|
||||||
|
# Restart dnsmasq
|
||||||
|
sudo systemctl restart dnsmasq
|
||||||
|
|
||||||
|
# Check Docker DNS config
|
||||||
|
cat /etc/docker/daemon.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Network Issues
|
||||||
|
```bash
|
||||||
|
# Check port mapping
|
||||||
|
docker port <container_name>
|
||||||
|
|
||||||
|
# Test external access
|
||||||
|
nc -v <host_ip> <port>
|
||||||
|
|
||||||
|
# Check Traefik routes
|
||||||
|
curl -s http://localhost:8080/api/http/routers
|
||||||
|
|
||||||
|
# Check container networks
|
||||||
|
docker inspect <container> --format '{{json .NetworkSettings.Networks}}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Application Deployment Issues
|
||||||
|
```bash
|
||||||
|
# Check deployment logs
|
||||||
|
docker service logs <service_name> --tail 50
|
||||||
|
|
||||||
|
# Force redeployment
|
||||||
|
docker service update --force <service_name>
|
||||||
|
|
||||||
|
# Check service configuration
|
||||||
|
docker service inspect <service_name>
|
||||||
|
|
||||||
|
# Scale services
|
||||||
|
docker service scale <service_name>=<replicas>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Monitoring Commands
|
||||||
|
|
||||||
|
### System Resources
|
||||||
|
```bash
|
||||||
|
# Disk usage
|
||||||
|
df -h
|
||||||
|
|
||||||
|
# Memory usage
|
||||||
|
free -h
|
||||||
|
|
||||||
|
# Docker space usage
|
||||||
|
docker system df
|
||||||
|
|
||||||
|
# Container resource usage
|
||||||
|
docker stats
|
||||||
|
```
|
||||||
|
|
||||||
|
### Docker Swarm Health
|
||||||
|
```bash
|
||||||
|
# Check swarm status
|
||||||
|
docker node ls
|
||||||
|
|
||||||
|
# Check service health
|
||||||
|
docker service ls
|
||||||
|
|
||||||
|
# Check individual services
|
||||||
|
docker service ps <service_name>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration Files
|
||||||
|
|
||||||
|
### Docker Daemon Configuration
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"data-root": "/mnt/nvme/docker",
|
||||||
|
"dns": ["127.0.0.1"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Nginx Template Key Sections
|
||||||
|
```nginx
|
||||||
|
# Static assets for pancake/third_party
|
||||||
|
location /pancake/third_party {
|
||||||
|
alias ${NIXPACKS_PHP_ROOT_DIR}/pancake/third_party;
|
||||||
|
expires 1y;
|
||||||
|
add_header Cache-Control "public, immutable";
|
||||||
|
}
|
||||||
|
|
||||||
|
# Pretty URLs for Pancake
|
||||||
|
location /pancake {
|
||||||
|
try_files $uri $uri/ @pancake_fallback;
|
||||||
|
}
|
||||||
|
|
||||||
|
location @pancake_fallback {
|
||||||
|
rewrite ^.*$ /pancake/index.php last;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Future Improvements
|
||||||
|
|
||||||
|
### DNS Enhancement
|
||||||
|
- Configure dnsmasq to forward internal domains to local DNS server
|
||||||
|
- Set up conditional forwarding for homelab services
|
||||||
|
- Add DNS caching for better performance
|
||||||
|
|
||||||
|
### Backup Strategy
|
||||||
|
- Regular backups of Docker volumes to NVMe
|
||||||
|
- Automated snapshots of configuration files
|
||||||
|
- Git repository tracking of all changes
|
||||||
|
|
||||||
|
### Monitoring
|
||||||
|
- Set up Prometheus/Grafana for system monitoring
|
||||||
|
- Log aggregation for better troubleshooting
|
||||||
|
- Alert configuration for critical services
|
||||||
|
|
||||||
|
## Emergency Procedures
|
||||||
|
|
||||||
|
### Full System Recovery
|
||||||
|
```bash
|
||||||
|
# 1. Check all services
|
||||||
|
docker service ls
|
||||||
|
|
||||||
|
# 2. Restart critical services
|
||||||
|
docker service update --force dokploy
|
||||||
|
docker service update --force traefik
|
||||||
|
|
||||||
|
# 3. Check DNS resolution
|
||||||
|
curl -I https://ghcr.io
|
||||||
|
|
||||||
|
# 4. Verify storage
|
||||||
|
df -h
|
||||||
|
docker system df
|
||||||
|
```
|
||||||
|
|
||||||
|
### Service Restoration
|
||||||
|
```bash
|
||||||
|
# Restore from backup if needed
|
||||||
|
docker volume ls
|
||||||
|
docker volume restore <volume_name> <backup_file>
|
||||||
|
|
||||||
|
# Re-deploy from last known good state
|
||||||
|
git log --oneline -10
|
||||||
|
git checkout <commit_hash>
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Last Updated**: 2025-11-29
|
||||||
|
**Maintainer**: sirtimbly
|
||||||
|
**Environment**: Production Docker Swarm Cluster
|
||||||
@@ -1,168 +1,70 @@
|
|||||||
# upstream php {
|
|
||||||
# server unix:/var/run/php/php8.3-fpm.sock;
|
|
||||||
# }
|
|
||||||
worker_processes 5;
|
worker_processes 5;
|
||||||
daemon off;
|
daemon off;
|
||||||
|
|
||||||
worker_rlimit_nofile 8192;
|
|
||||||
|
|
||||||
events {
|
events {
|
||||||
worker_connections 4096; # Default: 1024
|
worker_connections 4096;
|
||||||
}
|
}
|
||||||
|
|
||||||
http {
|
http {
|
||||||
|
# Include standard MIME types
|
||||||
include $!{nginx}/conf/mime.types;
|
include $!{nginx}/conf/mime.types;
|
||||||
index index.html index.htm index.php;
|
|
||||||
|
# Set default type
|
||||||
default_type application/octet-stream;
|
default_type application/octet-stream;
|
||||||
log_format main '$remote_addr - $remote_user [$time_local] $status '
|
|
||||||
'"$request" $body_bytes_sent "$http_referer" '
|
# Logging
|
||||||
'"$http_user_agent" "$http_x_forwarded_for"';
|
|
||||||
access_log /dev/stdout;
|
access_log /dev/stdout;
|
||||||
error_log /dev/stdout;
|
error_log /dev/stdout;
|
||||||
sendfile on;
|
|
||||||
tcp_nopush on;
|
# Optimization
|
||||||
server_names_hash_bucket_size 128; # this seems to be required for some vhosts
|
sendfile on;
|
||||||
|
tcp_nopush on;
|
||||||
|
server_names_hash_bucket_size 128;
|
||||||
|
|
||||||
server {
|
server {
|
||||||
listen ${PORT};
|
listen ${PORT};
|
||||||
listen [::]:${PORT};
|
listen [::]:${PORT};
|
||||||
server_name localhost;
|
server_name localhost;
|
||||||
|
|
||||||
|
# Set the root to the app directory
|
||||||
|
root /app;
|
||||||
|
|
||||||
|
# KEY FIX: Add index.html to the index directive so the root loads
|
||||||
|
index index.html index.php;
|
||||||
|
|
||||||
|
charset utf-8;
|
||||||
|
|
||||||
$if(NIXPACKS_PHP_ROOT_DIR) (
|
# Root location for the static landing page
|
||||||
root ${NIXPACKS_PHP_ROOT_DIR};
|
|
||||||
) else (
|
|
||||||
root /app;
|
|
||||||
)
|
|
||||||
# Block access to hidden files and directories
|
|
||||||
location ~ /\. {
|
|
||||||
deny all;
|
|
||||||
}
|
|
||||||
|
|
||||||
add_header X-Frame-Options "SAMEORIGIN";
|
|
||||||
add_header X-Content-Type-Options "nosniff";
|
|
||||||
|
|
||||||
index index.php;
|
|
||||||
|
|
||||||
|
|
||||||
# Static files for root directory
|
|
||||||
location / {
|
location / {
|
||||||
|
# Try to serve the file directly, then the directory, then 404
|
||||||
try_files $uri $uri/ =404;
|
try_files $uri $uri/ =404;
|
||||||
|
|
||||||
# Expires headers for static assets
|
|
||||||
location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot|bmp|webp|cur)$ {
|
|
||||||
expires 1y;
|
|
||||||
add_header Cache-Control "public, immutable";
|
|
||||||
}
|
|
||||||
|
|
||||||
# No cache for HTML
|
|
||||||
location ~* \.(html)$ {
|
|
||||||
expires 0;
|
|
||||||
add_header Cache-Control "no-cache";
|
|
||||||
}
|
|
||||||
|
|
||||||
# No cache for data interchange
|
|
||||||
location ~* \.(json|xml|jsonld|rdf|rss|atom|geojson|topojson|vtt|webmanifest|appcache)$ {
|
|
||||||
expires 0;
|
|
||||||
add_header Cache-Control "no-cache";
|
|
||||||
}
|
|
||||||
|
|
||||||
# No cache for PDFs
|
|
||||||
location ~* \.(pdf)$ {
|
|
||||||
expires 0;
|
|
||||||
add_header Cache-Control "no-cache";
|
|
||||||
}
|
|
||||||
|
|
||||||
# 1 hour for web feeds
|
|
||||||
location ~* \.(rss|atom)$ {
|
|
||||||
expires 1h;
|
|
||||||
add_header Cache-Control "public";
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# Static assets for pancake/third_party
|
# Pancake App Logic
|
||||||
location /pancake/third_party {
|
# No 'alias' needed for static assets because they live in /app/pancake/third_party
|
||||||
alias ${NIXPACKS_PHP_ROOT_DIR}/pancake/third_party;
|
# which matches the URI structure relative to root /app
|
||||||
expires 1y;
|
|
||||||
add_header Cache-Control "public, immutable";
|
|
||||||
|
|
||||||
# MIME types
|
|
||||||
location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot|bmp|webp|cur|flv|mp4|ogv|webm|swf)$ {
|
|
||||||
expires 1y;
|
|
||||||
add_header Cache-Control "public, immutable";
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
# PHP application for /pancake with pretty URLs
|
|
||||||
location /pancake {
|
location /pancake {
|
||||||
# First try to serve the requested file/directory, then fallback to index.php
|
|
||||||
try_files $uri $uri/ @pancake_fallback;
|
try_files $uri $uri/ @pancake_fallback;
|
||||||
}
|
}
|
||||||
|
|
||||||
# Fallback location for Pancake pretty URLs
|
|
||||||
location @pancake_fallback {
|
location @pancake_fallback {
|
||||||
rewrite ^.*$ /pancake/index.php last;
|
rewrite ^ /pancake/index.php last;
|
||||||
}
|
}
|
||||||
|
|
||||||
# Handle PHP files
|
# PHP Processing
|
||||||
|
|
||||||
location ~ \.php$ {
|
location ~ \.php$ {
|
||||||
|
try_files $uri =404;
|
||||||
fastcgi_pass 127.0.0.1:9000;
|
fastcgi_pass 127.0.0.1:9000;
|
||||||
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
|
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
|
||||||
include $!{nginx}/conf/fastcgi_params;
|
include $!{nginx}/conf/fastcgi_params;
|
||||||
|
# fastcgi.conf often duplicates params but is safer to include if available
|
||||||
include $!{nginx}/conf/fastcgi.conf;
|
include $!{nginx}/conf/fastcgi.conf;
|
||||||
}
|
}
|
||||||
|
|
||||||
# Gzip compression
|
# Security: Deny hidden files
|
||||||
gzip on;
|
location ~ /\. {
|
||||||
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript application/x-javascript application/atom+xml application/rss+xml application/ld+json application/manifest+json application/vnd.geo+json font/opentype image/svg+xml;
|
deny all;
|
||||||
|
|
||||||
# Security headers
|
|
||||||
add_header X-Content-Type-Options nosniff;
|
|
||||||
add_header X-UA-Compatible "IE=edge";
|
|
||||||
|
|
||||||
# UTF-8 encoding
|
|
||||||
charset utf-8;
|
|
||||||
|
|
||||||
# MIME types
|
|
||||||
types {
|
|
||||||
application/atom+xml atom;
|
|
||||||
application/json json map topojson;
|
|
||||||
application/ld+json jsonld;
|
|
||||||
application/rss+xml rss;
|
|
||||||
application/vnd.geo+json geojson;
|
|
||||||
application/xml rdf xml;
|
|
||||||
application/javascript js;
|
|
||||||
application/manifest+json webmanifest;
|
|
||||||
application/x-web-app-manifest+json webapp;
|
|
||||||
text/cache-manifest appcache;
|
|
||||||
audio/mp4 f4a f4b m4a;
|
|
||||||
audio/ogg oga ogg opus;
|
|
||||||
image/bmp bmp;
|
|
||||||
image/svg+xml svg svgz;
|
|
||||||
image/webp webp;
|
|
||||||
video/mp4 f4v f4p m4v mp4;
|
|
||||||
video/ogg ogv;
|
|
||||||
video/webm webm;
|
|
||||||
video/x-flv flv;
|
|
||||||
image/x-icon cur ico;
|
|
||||||
application/font-woff woff;
|
|
||||||
application/font-woff2 woff2;
|
|
||||||
application/vnd.ms-fontobject eot;
|
|
||||||
application/x-font-ttf ttc ttf;
|
|
||||||
font/opentype otf;
|
|
||||||
application/octet-stream safariextz;
|
|
||||||
application/x-bb-appworld bbaw;
|
|
||||||
application/x-chrome-extension crx;
|
|
||||||
application/x-opera-extension oex;
|
|
||||||
application/x-xpinstall xpi;
|
|
||||||
text/vcard vcard vcf;
|
|
||||||
text/vnd.rim.location.xloc xloc;
|
|
||||||
text/vtt vtt;
|
|
||||||
text/x-component htc;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# Error pages
|
|
||||||
error_page 404 /404.html;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
2
nixpacks.toml
Normal file
2
nixpacks.toml
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
[start]
|
||||||
|
cmd = "supervisord -c /app/supervisord.conf"
|
||||||
Reference in New Issue
Block a user