maddox 3dc0f4dacf Initial SilverBullet sync

2026-01-25 00:20:24 +00:00

22 KiB

Raw Blame History

Hetzner Server (192.168.12.3) - Custom Applications

Last updated: 2026-01-05

This document provides detailed information about custom applications and specialized configurations running on the Hetzner server.

hetzner/)](#traefik-configuration-manager-traefik-mod)
Node-RED Financial Automation
Common Operations

Traefik Configuration Manager (traefik-mod)

Overview

A custom Flask web application that provides a database-backed configuration management system for Traefik reverse proxy. Replaces manual YAML file editing with a web interface while maintaining Git version control.

Container: traefik-mod Image: traefik-mod-traefik-mod (custom build) Source Location: /volume1/docker/traefik-mod/ Access URLs:

Architecture

Technology Stack:

Backend: Python Flask
Database: MariaDB (mariadb-secondary container)
Version Control: Git (local repository)
Config Format: YAML (auto-generated from database)
Authentication: Authentik SSO via Traefik middleware

Data Flow:

User → Web Interface (Flask)
           ↓
      MariaDB Database
           ↓
      YAML Generator
           ↓
      /matrix/traefik/config/dyno.yml
           ↓
      Git Commit (local)
           ↓
      Change History (database)

Database Schema

Database: traefik_config (on mariadb-secondary) User: traefik_user

Tables:

routers: HTTP/TCP router configurations
services: Backend service definitions
middlewares: Middleware configurations (auth, headers, rate limiting, etc.)
config_history: Audit trail of all configuration changes

Key Features:

Primary keys: Auto-incrementing IDs
Unique constraints: Router/service/middleware names
JSON fields: Complex configurations (servers, headers, etc.)
Timestamps: created_at, updated_at for all entries
Audit trail: Full history of changes with user and timestamp

File Structure

/volume1/docker/traefik-mod/
├── docker-compose.yml          # Container definition
├── Dockerfile                   # Custom image build
├── app.py                       # Flask application
├── requirements.txt             # Python dependencies
├── .env                         # Environment variables
├── maria.md                     # phpMyAdmin setup instructions
├── backups/                     # Configuration backups (30-day retention)
├── templates/                   # HTML templates
├── static/                      # CSS, JavaScript
├── utils/
│   ├── db.py                    # Database connection
│   ├── yaml_generator.py        # YAML generation from DB
│   └── git_manager.py           # Git operations
├── scripts/
│   └── migrate_yaml_to_db.py   # Migration script (YAML → DB)
└── docs/
    ├── MIGRATION_GUIDE.md       # Complete migration documentation
    ├── database-schema.md       # Database schema details
    └── IMPLEMENTATION_COMPLETE.md  # Testing checklist

Configuration

Environment Variables:

TZ: America/New_York
FLASK_ENV: production
FLASK_SECRET_KEY: <secret>
DYNO_FILE_PATH: /config/dyno.yml
BACKUP_DIR: /backups
BACKUP_RETENTION_DAYS: 30
BACKUP_SCHEDULE: 0 0 * * *
TRAEFIK_CONTAINER_NAME: traefik
UID: 1000
GID: 1000
DB_HOST: mariadb-secondary
DB_PORT: 3306
DB_NAME: traefik_config
DB_USER: traefik_user
DB_PASSWORD: <password>
GIT_ENABLED: true
SQL_ECHO: false

Volume Mounts:

/matrix/traefik/config/dyno.yml:/config/dyno.yml:rw - Generated Traefik config
./backups:/backups:rw - Backup storage
/var/run/docker.sock:/var/run/docker.sock:ro - Docker API access (read-only)

Resource Limits:

CPU: 0.5
Memory Limit: 256M
Memory Reservation: 64M

Web Interface Features

Configuration Management:

Create/Edit/Delete routers, services, middlewares
Form view for guided configuration
YAML edit mode for advanced users
Real-time YAML validation
Search across all configuration items

Router Management:

HTTP and TCP router support
Domain-based routing rules
TLS configuration with cert resolvers
Middleware assignment
Service assignment
Priority configuration

Service Management:

Load balancer configuration
Server definitions (host:port)
Health check configuration
Sticky sessions
Pass host header settings

Middleware Management:

Authentication (BasicAuth, Authentik forward auth)
Headers (secure headers, CORS, custom headers)
Rate limiting
IP whitelisting
Redirects (scheme, regex)
Strip prefix
Retry configuration

Version Control:

Automatic Git commits on every change
Commit messages with change details
Browse Git history
View diffs between versions
Rollback capability (via Git)

Audit Trail:

Complete history in config_history table
User identification
Timestamp tracking
Change type (create, update, delete)
Before/after values

Operations:

Restart Traefik button (with countdown timer)
Backup before every change
Export configuration as YAML
Import from existing YAML (migration)

Database Setup

Manual Setup via phpMyAdmin:

User prefers phpMyAdmin for database management. Complete setup instructions are in /volume1/docker/traefik-mod/maria.md.

Steps:

Access phpMyAdmin on primary server
Connect to mariadb-secondary (192.168.12.3:3306)
Create database: traefik_config
Create user: traefik_user
Grant privileges: ALL on traefik_config.*
Import schema (if provided) or let application create tables

Important:

Uses --skip-db-creation flag in initialization
No MariaDB root password required in .env
Database/user created manually by user via phpMyAdmin

API Endpoints

Web Interface:

GET / - Dashboard/home page
GET /routers - List all routers
GET /routers/new - Create new router
GET /routers/<id>/edit - Edit router
POST /routers/<id>/delete - Delete router
Similar patterns for /services and /middlewares

System Operations:

GET /health - Healthcheck endpoint
POST /restart-traefik - Restart Traefik container
GET /search - Global search
GET /history - View change history
GET /export - Export configuration as YAML

YAML Operations:

POST /generate-yaml - Regenerate YAML from database
GET /view-yaml - View current generated YAML
GET /backup - Manual backup

Common Operations

Access Web Interface:

# Browser access (requires Authentik login)
https://tm.3ddbrewery.com
https://tm.fails.me

View Logs:

cd /volume1/docker/traefik-mod
docker compose logs -f

Rebuild Container:

cd /volume1/docker/traefik-mod
docker compose down
docker compose build
docker compose up -d

Database Access (via phpMyAdmin):

Open phpMyAdmin on primary server
Connect to 192.168.12.3:3306
Select traefik_config database
Browse tables: routers, services, middlewares, config_history

Manual YAML Generation:

docker exec -it traefik-mod python -c "
from utils.yaml_generator import YAMLGenerator
generator = YAMLGenerator()
generator.write_yaml_file(backup=True)
"

View Git History:

# Last 20 commits
docker exec -it traefik-mod git -C /config log --oneline -20

# Detailed commit
docker exec -it traefik-mod git -C /config show <commit-hash>

# View all commits
docker exec -it traefik-mod git -C /config log --oneline

Migration (YAML → Database):

# Dry run (preview changes)
docker exec -it traefik-mod python scripts/migrate_yaml_to_db.py --dry-run

# Execute migration
docker exec -it traefik-mod python scripts/migrate_yaml_to_db.py

Restart Traefik:

# Via web interface (recommended)
# Click "Restart Traefik" button on dashboard

# Manual restart
docker restart matrix-traefik

# Check Traefik status
docker ps | grep traefik
docker logs matrix-traefik --tail 50

Backup Configuration:

# Backups stored in ./backups/
ls -lh /volume1/docker/traefik-mod/backups/

# Manual backup
docker exec -it traefik-mod python -c "
from utils.yaml_generator import YAMLGenerator
generator = YAMLGenerator()
generator.write_yaml_file(backup=True)
"

Troubleshooting

Check Application Health:

# Healthcheck endpoint
curl http://localhost:5000/health

# Container status
docker ps | grep traefik-mod
docker inspect traefik-mod | grep -A 5 Health

Database Connection Issues:

# Test database connection
docker exec -it traefik-mod python -c "
from utils.db import get_db
db = get_db()
print('Database connected successfully')
"

# Check MariaDB is accessible
docker exec -it traefik-mod ping mariadb-secondary
docker exec -it traefik-mod nc -zv mariadb-secondary 3306

YAML Generation Issues:

# Check YAML file exists and is writable
docker exec -it traefik-mod ls -lh /config/dyno.yml

# Validate YAML syntax
docker exec -it traefik-mod python -c "
import yaml
with open('/config/dyno.yml', 'r') as f:
    config = yaml.safe_load(f)
    print('YAML is valid')
"

Git Issues:

# Check Git status
docker exec -it traefik-mod git -C /config status

# View Git config
docker exec -it traefik-mod git -C /config config --list

Development Setup

Build Custom Image:

cd /volume1/docker/traefik-mod
docker compose build

Run in Development Mode:

# Edit .env
FLASK_ENV=development
SQL_ECHO=true

# Restart container
docker compose down && docker compose up -d

# View detailed logs
docker compose logs -f

Database Schema Updates:

Modify database schema via phpMyAdmin
Update docs/database-schema.md
Update migration scripts if needed
Test YAML generation
Rebuild container

Security

Authentication:

Protected by Authentik forward authentication
Traefik middleware: authentik@file
Requires valid Authentik login

Authorization:

All authenticated users have full access
No role-based access control (all-or-nothing)
Trust model: users with Authentik access are trusted

Database Security:

Dedicated database user (traefik_user)
Limited to traefik_config database only
Credentials in .env (not committed to Git)

Docker Socket:

Read-only access to Docker socket
Used only for restarting Traefik container
No other Docker operations permitted

Audit Trail:

All changes logged to config_history table
Git commits provide secondary audit trail
Timestamps and user identification

Backup and Recovery

Automatic Backups:

Triggered before every configuration change
Stored in /volume1/docker/traefik-mod/backups/
Retention: 30 days
File format: dyno.yml.backup-YYYYMMDD-HHMMSS

Git History:

Local Git repository in /config/ (inside container)
Every change committed automatically
Commit messages include change type and details
Full history preserved indefinitely

Manual Backup:

# Copy current YAML
cp /matrix/traefik/config/dyno.yml /volume1/docker/backup/dyno.yml.$(date +%Y%m%d)

# Export database
docker exec mariadb-secondary mysqldump -u traefik_user -p traefik_config > /volume1/docker/backup/traefik_config_$(date +%Y%m%d).sql

Recovery:

# Restore from backup file
cp /volume1/docker/traefik-mod/backups/dyno.yml.backup-YYYYMMDD-HHMMSS /matrix/traefik/config/dyno.yml
docker restart matrix-traefik

# Restore from Git history
docker exec -it traefik-mod git -C /config log --oneline
docker exec -it traefik-mod git -C /config checkout <commit-hash> dyno.yml
docker restart matrix-traefik

# Restore database from dump
docker exec -i mariadb-secondary mysql -u traefik_user -p traefik_config < backup_file.sql

Node-RED Financial Automation

Overview

Node-RED instance dedicated to financial transaction automation, including automated imports, categorization, and synchronization with Firefly III.

Container: node-red Image: nodered/node-red:latest Source Location: /volume1/docker/node-red/ Data Location: /volume1/docker/node-red/ (flows, settings, node_modules) Access URL: https://node-het.3ddbrewery.com

Architecture

Technology Stack:

Platform: Node-RED (Node.js-based flow programming)
Runtime: Node.js
Authentication: Username/password (NODE_RED_USERNAME/PASSWORD)
Data Storage: JSON files in /data
Credential Encryption: NODE_RED_CREDENTIAL_SECRET

Integration Points:

Firefly III: API integration for transaction management
node-staging Database: MariaDB database for transaction staging
ntfy: Notification service for alerts
External Services: Banks, financial APIs (via flows)

Configuration

Environment Variables:

TZ: America/New_York
NODE_RED_CREDENTIAL_SECRET: 396572538122375581201991
NODE_RED_USERNAME: maddox
NODE_RED_PASSWORD: ./sk8nh8
NPM_CONFIG_CACHE: /data/.npm
NPM_CONFIG_PREFIX: /data/.npm
NODE_PATH: /data/node_modules

Volume Mounts:

/volume1/docker/node-red:/data:rw - Node-RED data (flows, settings, modules)
/home/maddox:/media:rw - User directory access for file operations

Network:

Connected to traefik network for external access
No isolated network (direct access to other services)

Resource Limits:

No explicit CPU/memory limits configured
Runs as user 1000:1000

Custom Healthcheck

Healthcheck Configuration:

test: ["CMD-SHELL", "node /healthcheck.js || (curl -H 'Title: Node-RED Down' -H 'Priority: high' -H 'Tags: alert,node-red,down' -d 'Node-RED health check failed on Hetzner-fin' http://192.168.1.70:6741/hetzner_alerts && exit 1)"]
interval: 120s
timeout: 60s
retries: 4
start_period: 120s

Features:

Primary check: Custom Node.js healthcheck script
Fallback: Sends notification to ntfy if health check fails
Notification server: http://192.168.1.70:6741/hetzner_alerts
Alert topic: hetzner_alerts
Tags: alert, node-red, down
High priority notification

Notification Details:

Title: "Node-RED Down"
Message: "Node-RED health check failed on Hetzner-fin"
Ensures administrators are alerted to Node-RED failures

Flows and Functionality

Purpose: Node-RED on this server runs automated financial transaction flows:

Transaction imports from external sources
Data transformation and categorization
Staging in node-staging database
Synchronization with Firefly III
Error handling and notifications

Flow Structure: Flows are stored in /volume1/docker/node-red/flows.json (accessible in /data inside container)

Common Flow Patterns:

Scheduled Triggers: Cron-based automation
API Polling: Check external sources for new transactions
Data Transformation: Parse and normalize transaction data
Database Operations: Stage transactions in node-staging
Firefly API Calls: Create/update transactions in Firefly III
Error Handling: Catch errors, log, send notifications

Database Integration

node-staging Database:

Host: mariadb-secondary (192.168.12.3:3306)
Database: node-staging
Purpose: Staging area for financial transactions
Replication: Does NOT replicate (isolated for testing)
Access: Node-RED flows connect directly to MariaDB

Connection Details: Node-RED flows include MySQL nodes configured to connect to:

Host: 192.168.12.3 (or mariadb-secondary if on same Docker network)
Port: 3306
Database: node-staging
User: (configured in flow credentials)

Installed NPM Packages

Package Management:

NPM cache: /data/.npm
NPM prefix: /data/.npm
Node modules: /data/node_modules

Common Packages (likely installed):

node-red-node-mysql - MySQL database integration
node-red-contrib-cron-plus - Advanced scheduling
HTTP request nodes (built-in) - API calls to Firefly III
Dashboard nodes - UI for monitoring (if used)

Install Additional Packages:

docker exec -it node-red npm install <package-name>

View Installed Packages:

docker exec -it node-red npm list --depth=0

Common Operations

Access Web Interface:

# Browser access
https://node-het.3ddbrewery.com

# Login credentials
Username: maddox
Password: ./sk8nh8

View Logs:

docker logs node-red -f

Restart Node-RED:

docker restart node-red

Access Container Shell:

docker exec -it node-red /bin/bash

Install NPM Packages:

docker exec -it node-red npm install <package-name>

# Example: Install MySQL node
docker exec -it node-red npm install node-red-node-mysql

Backup Flows:

# Flows stored in /volume1/docker/node-red/flows.json
cp /volume1/docker/node-red/flows.json /volume1/docker/backup/node-red-flows-$(date +%Y%m%d).json

# Backup entire Node-RED data directory
tar -czf /volume1/docker/backup/node-red-data-$(date +%Y%m%d).tar.gz /volume1/docker/node-red/

Restore Flows:

# Copy backup to flows.json
cp /volume1/docker/backup/node-red-flows-YYYYMMDD.json /volume1/docker/node-red/flows.json

# Restart Node-RED
docker restart node-red

Check Database Connection:

# Test connection from Node-RED container
docker exec -it node-red sh -c "nc -zv 192.168.12.3 3306"

# Or test MySQL connection
docker exec -it mariadb-secondary mysql -u <user> -p node-staging -e "SHOW TABLES;"

Troubleshooting

Node-RED Won't Start:

# Check logs
docker logs node-red

# Check file permissions
ls -lh /volume1/docker/node-red/

# Ensure ownership is 1000:1000
sudo chown -R 1000:1000 /volume1/docker/node-red/

Flow Execution Issues:

# Check Node-RED debug panel (in web interface)
# Enable debug nodes in flows
# View logs in real-time
docker logs node-red -f

Database Connection Failures:

# Verify MariaDB is running
docker ps | grep mariadb

# Test network connectivity
docker exec -it node-red ping mariadb-secondary

# Check database exists
docker exec -it mariadb-secondary mysql -u root -p -e "SHOW DATABASES;"

Credential Decryption Issues:

# Verify NODE_RED_CREDENTIAL_SECRET hasn't changed
docker inspect node-red | grep CREDENTIAL_SECRET

# If secret changed, flows_cred.json needs re-encryption
# (This requires manual intervention)

Security

Authentication:

Username/password authentication enabled
Credentials: maddox / ./sk8nh8
Protected by Traefik reverse proxy
No additional Authentik protection (direct auth only)

Credential Storage:

Flow credentials encrypted with NODE_RED_CREDENTIAL_SECRET
Stored in /data/flows_cred.json
Secret must remain constant for credential decryption

File Access:

Has access to /home/maddox directory (mounted as /media)
Can read/write files on host filesystem
Runs as user 1000:1000 (limited privileges)

Network Access:

Can access any service on traefik network
Can access external internet (for API calls)
Direct access to MariaDB on port 3306

Monitoring and Alerts

Healthcheck:

Custom healthcheck script (/healthcheck.js)
Runs every 120 seconds
Timeout: 60 seconds
Retries: 4 before considered unhealthy

Failure Notifications:

Automatic ntfy notification on health check failure
Notification URL: http://192.168.1.70:6741/hetzner_alerts
Topic: hetzner_alerts
Priority: high

Autoheal Integration:

Container labeled with autoheal=true
Autoheal monitors health status
Automatic restart if unhealthy

Watchtower Integration:

Container labeled with watchtower enable
Automatic image updates
Rolling restart on update

Development and Debugging

Enable Debug Logging:

# In Node-RED web interface:
# 1. Add Debug nodes to flows
# 2. View Debug panel (right sidebar)
# 3. Enable verbose logging in settings

# Container logs show Node-RED startup and errors
docker logs node-red -f

Edit Settings:

# Settings file
nano /volume1/docker/node-red/settings.js

# Restart after changes
docker restart node-red

Install Development Tools:

# Access container
docker exec -it node-red /bin/bash

# Install tools
npm install -g <package>

# View environment
env | grep NODE

Common Operations

Service Restart

Restart All Services:

# Restart everything in /volume1/docker/
cd /volume1/docker
for dir in */; do
    if [ -f "$dir/docker-compose.yml" ]; then
        echo "Restarting $dir"
        cd "$dir"
        docker compose restart
        cd ..
    fi
done

Restart Specific Service:

cd /volume1/docker/<service-name>
docker compose restart

Log Viewing

View Logs for Service:

cd /volume1/docker/<service-name>
docker compose logs -f

View Logs for Specific Container:

docker logs <container-name> -f

# Examples:
docker logs traefik-mod -f
docker logs node-red -f
docker logs mariadb-secondary -f

Container Rebuilding

Rebuild Custom Images:

# Traefik-mod
cd /volume1/docker/traefik-mod
docker compose down
docker compose build
docker compose up -d

# Check status
docker ps | grep traefik-mod

Health Monitoring

Check All Container Health:

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.State}}"

# Filter only unhealthy containers
docker ps --filter health=unhealthy

Check Autoheal Logs:

docker logs autoheal -f

Upgrade matrix implementation

just update
 - or -
git pull

 - then - 
just roles

 - then -
just install-all
 - or -
just setup-all

# If you receive an error about a missing role
just roles

# If needed, add files to commit and stash
git add -A
git stash
git pull

# Check services
just run-tags self-check
 - or -
ansible-playbook -i inventory/hosts setup.yml --tags=self-check

Summary

Custom Applications:

traefik-mod: Database-backed Traefik configuration manager with web UI
node-red: Financial automation platform with custom flows and database integration

Key Features:

Both applications use web interfaces for management
Database integration with mariadb-secondary
Automatic backups and version control (traefik-mod)
Health monitoring and notifications (node-red)
Authentication via Authentik (traefik-mod) or built-in (node-red)

Management:

Web-based administration preferred
phpMyAdmin for database management
Git version control for configuration tracking
Automated monitoring via autoheal and watchtower

22 KiB Raw Blame History

Hetzner Server (192.168.12.3) - Custom Applications

Table of Contents

Traefik Configuration Manager (traefik-mod)

Overview

Architecture

Database Schema

File Structure

Configuration

Web Interface Features

Database Setup

API Endpoints

Common Operations

Troubleshooting

Development Setup

Security

Backup and Recovery

Node-RED Financial Automation

Overview

Architecture

Configuration

Custom Healthcheck

Flows and Functionality

Database Integration

Installed NPM Packages

Common Operations

Troubleshooting

Security

Monitoring and Alerts

Development and Debugging

Common Operations

Service Restart

Log Viewing

Container Rebuilding

Health Monitoring

Upgrade matrix implementation

Summary

22 KiB

Raw Blame History