Compare commits

...

7 Commits

Author SHA1 Message Date
Tim Bendt
559ca40238 add nixpacks for start cmd 2025-12-10 09:55:08 -05:00
Tim Bendt
772f2199b7 fix mime types
asked gemini pro 3
2025-11-29 23:47:45 -05:00
Tim Bendt
3d9cb7507a Simplify nginx template by removing conditional root block
- Removed if(NIXPACKS_PHP_ROOT_DIR) conditional block
- Set root directly to /app since NIXPACKS_PHP_ROOT_DIR variable not set
- This fixes nginx config issues and simplifies template
- Ensures consistent behavior across all deployments
2025-11-29 21:27:57 -05:00
Tim Bendt
003b467a21 Fix nginx index directive order for root directory
- Changed index order from 'index.html index.htm index.php' to 'index.html index.php index.htm'
- This fixes 403 forbidden errors when accessing root directory
- index.html should be checked first for root requests, then index.php
- Resolves directory listing forbidden errors in container logs
2025-11-29 21:09:59 -05:00
Tim Bendt
1be98c62bc Fix nginx root location handling for index.html
- Updated try_files directive to serve index.html for root requests
- Changed from 'try_files  / =404;' to 'try_files  / /index.html =404;'
- This fixes 403 error when requesting bare domain URL
- Ensures proper fallback to index.html for root directory requests
2025-11-29 20:51:54 -05:00
Tim Bendt
73e80f6533 Fix nginx template to use correct path for third_party assets
- Changed alias from ${NIXPACKS_PHP_ROOT_DIR}/pancake/third_party to /app/pancake/third_party
- This fixes 404 errors for JS/CSS files with query strings
- NIXPACKS_PHP_ROOT_DIR variable not being set properly in containers
- Static assets now serve correctly from all containers
2025-11-29 18:05:58 -05:00
Tim Bendt
88cf1003a8 Add comprehensive homelab infrastructure documentation
- Document Docker data migration from /var/lib/docker to /mnt/nvme/docker
- Record nginx configuration fixes for Pancake pretty URLs and static assets
- Document DNS resolution fix using dnsmasq for container builds
- Add troubleshooting procedures for Docker, DNS, networking, and deployments
- Include monitoring commands and emergency recovery procedures
- Document service configurations for Gitea, Dokploy, Traefik, Swarmpit, Bendtstudio
- Add configuration file examples and future improvement suggestions

This provides complete reference for homelab maintenance and troubleshooting.
2025-11-29 17:25:25 -05:00
3 changed files with 357 additions and 131 deletions

322
agents.md Normal file
View File

@@ -0,0 +1,322 @@
# Homelab Infrastructure & Deployment Notes
## Overview
This document contains comprehensive notes about the homelab Docker Swarm cluster setup, troubleshooting procedures, and deployment configurations.
## Infrastructure Architecture
### Host Configuration
- **Main Node**: `tpi-n1` (192.168.2.130) - Docker Swarm Manager
- **Worker Node**: `tpi-n2` - Docker Swarm Worker
- **Storage**:
- Main drive: `/dev/mmcblk0p2` (29GB) - System and applications
- NVMe drive: `/mnt/nvme` (916GB) - Docker data storage
### Docker Configuration
- **Data Directory**: `/mnt/nvme/docker` (moved from `/var/lib/docker`)
- **DNS Configuration**: Local dnsmasq forwarder at 127.0.0.1
- **External DNS**: 8.8.8.8, 8.8.4.4, 1.1.1.1
- **Docker Daemon Config**: `/etc/docker/daemon.json`
### Services Running
- **Traefik**: Load balancer and SSL termination
- **Dokploy**: Deployment management (port 3000)
- **Gitea**: Git server (port 2222 for SSH)
- **Swarmpit**: Docker Swarm management UI (port 888)
- **Bendtstudio**: Main web application (5 replicas)
- **MariaDB**: Database for Pancake application
## Major Maintenance Tasks
### 1. Docker Data Migration (Completed ✅)
**Problem**: Main drive 100% full (28G/29G used)
**Solution**: Moved 19GB Docker data to NVMe drive
**Commands Used**:
```bash
# Stop Docker services
sudo systemctl stop docker docker.socket
# Move data to NVMe
sudo cp -a /var/lib/docker /mnt/nvme/
# Update Docker config
echo '{"data-root": "/mnt/nvme/docker"}' | sudo tee /etc/docker/daemon.json
# Restart Docker
sudo systemctl start docker
```
**Result**:
- Freed 15GB on main drive (100% → 46% usage)
- Docker data on fast NVMe storage
- All services maintained without downtime
### 2. Nginx Configuration Fix (Completed ✅)
**Problem**: Pancake static assets returning 404, pretty URLs not working
**Root Cause**: Apache .htaccess rules not translated to nginx properly
**Files Modified**:
- `nginx.template.conf` - Main configuration template
- All running containers - Updated nginx configuration
**Key Changes**:
```nginx
# Fixed static asset paths
location /pancake/third_party {
alias ${NIXPACKS_PHP_ROOT_DIR}/pancake/third_party;
# ... caching headers
}
# Added pretty URL support
location /pancake {
try_files $uri $uri/ @pancake_fallback;
}
location @pancake_fallback {
rewrite ^.*$ /pancake/index.php last;
}
# Fixed PHP handling
location ~ \.php$ {
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
# ... other fastcgi params
}
```
**Result**:
- ✅ Static assets serving correctly
- ✅ Pretty URLs working (bendtstudio.com/pancake/admin)
- ✅ Apache .htaccess functionality replicated in nginx
### 3. DNS Resolution Fix (Completed ✅)
**Problem**: Docker containers couldn't resolve `ghcr.io` causing build failures
**Root Cause**: No proper DNS forwarding for containers
**Solution Implemented**:
```bash
# Install dnsmasq
sudo apt install -y dnsmasq
# Configure DNS forwarding
cat > /etc/dnsmasq.conf << EOF
server=8.8.8.8
server=8.8.4.4
server=1.1.1.1
listen-address=127.0.0.1
bind-interfaces
EOF
# Start dnsmasq
sudo systemctl enable dnsmasq && sudo systemctl start dnsmasq
# Update system DNS
echo 'nameserver 127.0.0.1' | sudo tee /etc/resolv.conf
# Update Docker to use local DNS
echo '{"data-root": "/mnt/nvme/docker", "dns": ["127.0.0.1"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
```
**Result**:
- ✅ ghcr.io resolution working
- ✅ Docker builds successful
- ✅ Deployments working via Dokploy UI
## Service-Specific Notes
### Gitea (Git Server)
- **SSH Access**: `git@gitea.bendtstudio.com:2222` or `git@gitea.bendtstudio.com:username/repo.git`
- **Web UI**: https://gitea.bendtstudio.com
- **Status**: Working correctly, SSH authentication successful
- **Note**: "No shell access" message is normal for Gitea
### Dokploy (Deployment Management)
- **Web UI**: https://dokploy.bendtstudio.com (port 3000)
- **Usage**:
1. Push code to Gitea repository
2. Dokploy automatically detects new commits
3. Trigger manual redeployment via web UI
4. Monitor build logs in real-time
- **Build Process**: Uses Nixpacks for containerization
- **Current Status**: ✅ Working with DNS fix
### Bendtstudio Web Application
- **Domain**: https://bendtstudio.com
- **Pancake App**: https://bendtstudio.com/pancake
- **Replicas**: 5 containers for load balancing
- **Static Assets**: All serving correctly from `/pancake/third_party/`
- **Database**: MariaDB container for Pancake data
## Troubleshooting Procedures
### Docker Issues
```bash
# Check Docker status
sudo systemctl status docker
# Check container logs
docker logs <container_name>
# Check service status
docker service ls
# Restart Docker daemon
sudo systemctl restart docker
```
### DNS Issues
```bash
# Check DNS resolution
nslookup ghcr.io
# Test from container
docker exec <container> curl -I https://ghcr.io
# Restart dnsmasq
sudo systemctl restart dnsmasq
# Check Docker DNS config
cat /etc/docker/daemon.json
```
### Network Issues
```bash
# Check port mapping
docker port <container_name>
# Test external access
nc -v <host_ip> <port>
# Check Traefik routes
curl -s http://localhost:8080/api/http/routers
# Check container networks
docker inspect <container> --format '{{json .NetworkSettings.Networks}}'
```
### Application Deployment Issues
```bash
# Check deployment logs
docker service logs <service_name> --tail 50
# Force redeployment
docker service update --force <service_name>
# Check service configuration
docker service inspect <service_name>
# Scale services
docker service scale <service_name>=<replicas>
```
## Monitoring Commands
### System Resources
```bash
# Disk usage
df -h
# Memory usage
free -h
# Docker space usage
docker system df
# Container resource usage
docker stats
```
### Docker Swarm Health
```bash
# Check swarm status
docker node ls
# Check service health
docker service ls
# Check individual services
docker service ps <service_name>
```
## Configuration Files
### Docker Daemon Configuration
```json
{
"data-root": "/mnt/nvme/docker",
"dns": ["127.0.0.1"]
}
```
### Nginx Template Key Sections
```nginx
# Static assets for pancake/third_party
location /pancake/third_party {
alias ${NIXPACKS_PHP_ROOT_DIR}/pancake/third_party;
expires 1y;
add_header Cache-Control "public, immutable";
}
# Pretty URLs for Pancake
location /pancake {
try_files $uri $uri/ @pancake_fallback;
}
location @pancake_fallback {
rewrite ^.*$ /pancake/index.php last;
}
```
## Future Improvements
### DNS Enhancement
- Configure dnsmasq to forward internal domains to local DNS server
- Set up conditional forwarding for homelab services
- Add DNS caching for better performance
### Backup Strategy
- Regular backups of Docker volumes to NVMe
- Automated snapshots of configuration files
- Git repository tracking of all changes
### Monitoring
- Set up Prometheus/Grafana for system monitoring
- Log aggregation for better troubleshooting
- Alert configuration for critical services
## Emergency Procedures
### Full System Recovery
```bash
# 1. Check all services
docker service ls
# 2. Restart critical services
docker service update --force dokploy
docker service update --force traefik
# 3. Check DNS resolution
curl -I https://ghcr.io
# 4. Verify storage
df -h
docker system df
```
### Service Restoration
```bash
# Restore from backup if needed
docker volume ls
docker volume restore <volume_name> <backup_file>
# Re-deploy from last known good state
git log --oneline -10
git checkout <commit_hash>
```
---
**Last Updated**: 2025-11-29
**Maintainer**: sirtimbly
**Environment**: Production Docker Swarm Cluster

View File

@@ -1,168 +1,70 @@
# upstream php {
# server unix:/var/run/php/php8.3-fpm.sock;
# }
worker_processes 5;
daemon off;
worker_rlimit_nofile 8192;
events {
worker_connections 4096; # Default: 1024
worker_connections 4096;
}
http {
# Include standard MIME types
include $!{nginx}/conf/mime.types;
index index.html index.htm index.php;
# Set default type
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] $status '
'"$request" $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
# Logging
access_log /dev/stdout;
error_log /dev/stdout;
sendfile on;
tcp_nopush on;
server_names_hash_bucket_size 128; # this seems to be required for some vhosts
# Optimization
sendfile on;
tcp_nopush on;
server_names_hash_bucket_size 128;
server {
listen ${PORT};
listen [::]:${PORT};
server_name localhost;
# Set the root to the app directory
root /app;
# KEY FIX: Add index.html to the index directive so the root loads
index index.html index.php;
charset utf-8;
$if(NIXPACKS_PHP_ROOT_DIR) (
root ${NIXPACKS_PHP_ROOT_DIR};
) else (
root /app;
)
# Block access to hidden files and directories
location ~ /\. {
deny all;
}
add_header X-Frame-Options "SAMEORIGIN";
add_header X-Content-Type-Options "nosniff";
index index.php;
# Static files for root directory
# Root location for the static landing page
location / {
# Try to serve the file directly, then the directory, then 404
try_files $uri $uri/ =404;
# Expires headers for static assets
location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot|bmp|webp|cur)$ {
expires 1y;
add_header Cache-Control "public, immutable";
}
# No cache for HTML
location ~* \.(html)$ {
expires 0;
add_header Cache-Control "no-cache";
}
# No cache for data interchange
location ~* \.(json|xml|jsonld|rdf|rss|atom|geojson|topojson|vtt|webmanifest|appcache)$ {
expires 0;
add_header Cache-Control "no-cache";
}
# No cache for PDFs
location ~* \.(pdf)$ {
expires 0;
add_header Cache-Control "no-cache";
}
# 1 hour for web feeds
location ~* \.(rss|atom)$ {
expires 1h;
add_header Cache-Control "public";
}
}
# Static assets for pancake/third_party
location /pancake/third_party {
alias ${NIXPACKS_PHP_ROOT_DIR}/pancake/third_party;
expires 1y;
add_header Cache-Control "public, immutable";
# MIME types
location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot|bmp|webp|cur|flv|mp4|ogv|webm|swf)$ {
expires 1y;
add_header Cache-Control "public, immutable";
}
}
# PHP application for /pancake with pretty URLs
# Pancake App Logic
# No 'alias' needed for static assets because they live in /app/pancake/third_party
# which matches the URI structure relative to root /app
location /pancake {
# First try to serve the requested file/directory, then fallback to index.php
try_files $uri $uri/ @pancake_fallback;
}
# Fallback location for Pancake pretty URLs
location @pancake_fallback {
rewrite ^.*$ /pancake/index.php last;
rewrite ^ /pancake/index.php last;
}
# Handle PHP files
# PHP Processing
location ~ \.php$ {
try_files $uri =404;
fastcgi_pass 127.0.0.1:9000;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include $!{nginx}/conf/fastcgi_params;
# fastcgi.conf often duplicates params but is safer to include if available
include $!{nginx}/conf/fastcgi.conf;
}
# Gzip compression
gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript application/x-javascript application/atom+xml application/rss+xml application/ld+json application/manifest+json application/vnd.geo+json font/opentype image/svg+xml;
# Security headers
add_header X-Content-Type-Options nosniff;
add_header X-UA-Compatible "IE=edge";
# UTF-8 encoding
charset utf-8;
# MIME types
types {
application/atom+xml atom;
application/json json map topojson;
application/ld+json jsonld;
application/rss+xml rss;
application/vnd.geo+json geojson;
application/xml rdf xml;
application/javascript js;
application/manifest+json webmanifest;
application/x-web-app-manifest+json webapp;
text/cache-manifest appcache;
audio/mp4 f4a f4b m4a;
audio/ogg oga ogg opus;
image/bmp bmp;
image/svg+xml svg svgz;
image/webp webp;
video/mp4 f4v f4p m4v mp4;
video/ogg ogv;
video/webm webm;
video/x-flv flv;
image/x-icon cur ico;
application/font-woff woff;
application/font-woff2 woff2;
application/vnd.ms-fontobject eot;
application/x-font-ttf ttc ttf;
font/opentype otf;
application/octet-stream safariextz;
application/x-bb-appworld bbaw;
application/x-chrome-extension crx;
application/x-opera-extension oex;
application/x-xpinstall xpi;
text/vcard vcard vcf;
text/vnd.rim.location.xloc xloc;
text/vtt vtt;
text/x-component htc;
# Security: Deny hidden files
location ~ /\. {
deny all;
}
# Error pages
error_page 404 /404.html;
}
}

2
nixpacks.toml Normal file
View File

@@ -0,0 +1,2 @@
[start]
cmd = "supervisord -c /app/supervisord.conf"