maddox 56df1ec12c Update docs/control-server-guide.md

2026-01-24 01:35:56 +00:00

7.9 KiB

Raw Blame History

Control Server Operations Guide

Host: control (CT 127)
IP: 192.168.1.127
Location: pve2
User: maddox
Last Updated: January 24, 2026

Quick Start

# Launch interactive menu
~/scripts/control-menu.sh

# Or jump straight to multi-host SSH
~/scripts/ssh-manager.sh

# Or run Ansible directly
cd ~/clustered-fucks
ansible all -m ping

Directory Structure

/home/maddox/
├── .ssh/
│   ├── config              # SSH host definitions
│   ├── id_ed25519          # SSH private key
│   └── id_ed25519.pub      # Public key (add to new hosts)
│
├── clustered-fucks/        # Git repo (synced to Forgejo)
│   ├── ansible.cfg         # Ansible configuration
│   ├── inventory/
│   │   ├── hosts.yml       # Host inventory
│   │   └── group_vars/
│   │       └── all.yml     # Global variables
│   ├── playbooks/
│   │   ├── check-status.yml
│   │   ├── docker-prune.yml
│   │   ├── restart-utils.yml
│   │   ├── update-all.yml
│   │   ├── deploy-utils.yml
│   │   └── deploy-mealie.yml   # NEW
│   └── compose-files/      # Docker compose files for services
│       └── databases/
│           └── mealie/
│               ├── docker-compose.yml
│               └── .env    # NOT in git (secrets)
│
└── scripts/
    ├── ssh-manager.sh      # tmux multi-host launcher
    ├── control-menu.sh     # Interactive Ansible menu
    └── add-host.sh         # New host onboarding

Managed Hosts

Host	IP	User	Type	Ansible Group
pve2	.3	root	Proxmox	proxmox_nodes
pve-dell	.4	root	Proxmox	proxmox_nodes
replicant	.80	maddox	VM	docker_hosts
databases	.81	root	VM	docker_hosts
immich	.82	root	VM	docker_hosts
media-transcode	.120	root	LXC	docker_hosts
network-services	.121	root	LXC	docker_hosts
download-stack	.122	root	LXC	docker_hosts
docker666	.123	root	LXC	docker_hosts
tailscale-home	.124	root	LXC	docker_hosts
dns-lxc	.125	root	LXC	infrastructure
nas	.251	maddox	NAS	legacy
alien	.252	maddox	Docker	legacy

Ansible Playbooks

Playbook	Target	Description
`check-status.yml`	all_managed	Disk, memory, container counts
`update-all.yml`	docker_hosts	apt upgrade all hosts
`docker-prune.yml`	docker_hosts	Clean unused Docker resources
`restart-utils.yml`	docker_hosts	Restart utils stack
`deploy-utils.yml`	docker_hosts	Deploy utils to new host
`deploy-mealie.yml`	databases	Deploy Mealie stack

Running Playbooks

cd ~/clustered-fucks

# Check all hosts
ansible-playbook playbooks/check-status.yml

# Update specific host
ansible-playbook playbooks/update-all.yml --limit databases

# Deploy service
ansible-playbook playbooks/deploy-mealie.yml

Git Workflow

Repository Info

Local: ~/clustered-fucks/
Remote: ssh://git@192.168.1.81:2222/maddox/clustered-fucks.git
Web: https://git.3ddbrewery.com/maddox/clustered-fucks

Daily Workflow

cd ~/clustered-fucks

# 1. Before making changes, get latest
git pull

# 2. Make your changes (edit files, create playbooks, etc.)
vim playbooks/new-playbook.yml

# 3. See what changed
git status

# 4. Stage all changes
git add -A

# 5. Commit with a message
git commit -m "Add deployment playbook for service X"

# 6. Push to Forgejo
git push

After Creating/Editing Files

Always commit your changes! Otherwise they only exist on the control server and could be lost.

cd ~/clustered-fucks
git add -A
git commit -m "Description of what you changed"
git push

Quick Reference

Command	What it does
`git status`	Show what files have changed
`git pull`	Get latest changes from Forgejo
`git add -A`	Stage all changes for commit
`git commit -m "msg"`	Save changes with a message
`git push`	Upload commits to Forgejo
`git log --oneline -5`	Show last 5 commits
`git diff`	Show what changed (before staging)

What Goes in Git vs What Doesn't

✅ Commit these:

Ansible playbooks (playbooks/*.yml)
Inventory files (inventory/hosts.yml)
Docker compose files (compose-files/**/docker-compose.yml)
Documentation and scripts

❌ Never commit these:

.env files (contain passwords/secrets)
Private keys
Temporary files

The .gitignore file already excludes .env files.

Service Migration Workflow

Standard Process

cd ~/clustered-fucks

# 1. Create compose file directory
mkdir -p compose-files/<target_host>/<service>

# 2. Create docker-compose.yml
vim compose-files/<target_host>/<service>/docker-compose.yml

# 3. Create .env for secrets (not committed to git)
vim compose-files/<target_host>/<service>/.env

# 4. Create Ansible playbook
vim playbooks/deploy-<service>.yml

# 5. Deploy via Ansible
ansible-playbook playbooks/deploy-<service>.yml

# 6. Rsync data from alien (if needed)
ssh alien "docker stop <container>"
rsync -avP maddox@alien:/path/to/data/ root@<target>:/home/docker/appdata/<service>/

# 7. Start the service
ansible <target> -m shell -a "cd /home/docker/appdata/<service> && docker compose up -d"

# 8. Update Traefik route (on Hetzner)

# 9. Test via domain

# 10. Stop old container on alien
ssh alien "docker stop <container>"

# 11. Commit playbook to git
git add -A
git commit -m "Add <service> deployment"
git push

Playbook Template

---
- name: Deploy <Service> to <host>
  hosts: <target_host>
  become: yes
  vars:
    service_name: <service>
    service_dir: /home/docker/appdata/{{ service_name }}

  tasks:
    - name: Create directories
      file:
        path: "{{ item }}"
        state: directory
        mode: '0755'
      loop:
        - "{{ service_dir }}"

    - name: Ensure proxy network exists
      community.docker.docker_network:
        name: proxy
        state: present

    - name: Copy docker-compose.yml
      copy:
        src: ../compose-files/<host>/{{ service_name }}/docker-compose.yml
        dest: "{{ service_dir }}/docker-compose.yml"
        mode: '0644'

    - name: Copy .env file
      copy:
        src: ../compose-files/<host>/{{ service_name }}/.env
        dest: "{{ service_dir }}/.env"
        mode: '0600'

    - name: Start stack
      community.docker.docker_compose_v2:
        project_src: "{{ service_dir }}"
        state: present
        recreate: always

Common Operations

SSH to Specific Host

ssh databases
ssh alien
ssh nas  # Uses port 44822 automatically

Run Command on All Docker Hosts

ansible docker_hosts -m shell -a "docker ps -q | wc -l"

Check Container Logs

ansible databases -m shell -a "docker logs mealie 2>&1 | tail -30"

Copy File to Host

ansible databases -m copy -a "src=./file.txt dest=/tmp/file.txt"

Troubleshooting

SSH Connection Refused

# Access via Proxmox console:
# For LXC: pct enter <CT_ID>
# For VM: qm terminal <VM_ID>

# Then inside:
apt install openssh-server
systemctl enable ssh
systemctl start ssh

SSH Permission Denied

# Copy your key to the host
ssh-copy-id <hostname>

# Or fix permissions on target:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Git Push Fails

# Check remote is correct
git remote -v

# Should show:
# origin  ssh://git@192.168.1.81:2222/maddox/clustered-fucks.git

# If wrong, fix it:
git remote set-url origin ssh://git@192.168.1.81:2222/maddox/clustered-fucks.git

Changelog

Date	Change
2026-01-23	Initial deployment
2026-01-24	Added deploy-mealie.yml, enhanced git workflow docs

7.9 KiB Raw Blame History