Add control server operations guide
This commit is contained in:
Maddox
parent
14f6348bf4
commit
4cb3a41f1c
1 changed files with 560 additions and 0 deletions
560
docs/control-server-guide.md
Normal file
560
docs/control-server-guide.md
Normal file
|
|
@ -0,0 +1,560 @@
|
|||
# Control Server Operations Guide
|
||||
|
||||
**Host:** control (CT 127)
|
||||
**IP:** 192.168.1.127
|
||||
**Location:** pve2
|
||||
**User:** maddox
|
||||
**Last Updated:** January 23, 2026
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The control server is the centralized command center for managing the Proxmox cluster infrastructure. It provides:
|
||||
|
||||
- **Passwordless SSH** to all 13 managed hosts
|
||||
- **Ansible automation** for cluster-wide operations
|
||||
- **tmux sessions** for multi-host management
|
||||
- **Git-based configuration** synced to Forgejo
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Launch Interactive Menu
|
||||
```bash
|
||||
~/scripts/control-menu.sh
|
||||
```
|
||||
|
||||
### Launch Multi-Host SSH Session
|
||||
```bash
|
||||
~/scripts/ssh-manager.sh
|
||||
```
|
||||
|
||||
### Run Ansible Ad-Hoc Command
|
||||
```bash
|
||||
cd ~/clustered-fucks
|
||||
ansible all -m ping
|
||||
ansible docker_hosts -m shell -a "docker ps --format 'table {{.Names}}\t{{.Status}}'"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
/home/maddox/
|
||||
├── .ssh/
|
||||
│ ├── config # SSH host definitions
|
||||
│ ├── tmux-hosts.conf # tmux session configuration
|
||||
│ ├── id_ed25519 # SSH private key
|
||||
│ └── id_ed25519.pub # SSH public key (add to new hosts)
|
||||
│
|
||||
├── clustered-fucks/ # Git repo (synced to Forgejo)
|
||||
│ ├── ansible.cfg # Ansible configuration
|
||||
│ ├── inventory/
|
||||
│ │ ├── hosts.yml # Host inventory
|
||||
│ │ └── group_vars/
|
||||
│ │ └── all.yml # Global variables
|
||||
│ └── playbooks/
|
||||
│ ├── check-status.yml
|
||||
│ ├── docker-prune.yml
|
||||
│ ├── restart-utils.yml
|
||||
│ ├── update-all.yml
|
||||
│ └── deploy-utils.yml
|
||||
│
|
||||
└── scripts/
|
||||
├── ssh-manager.sh # tmux multi-host launcher
|
||||
├── control-menu.sh # Interactive Ansible menu
|
||||
└── add-host.sh # New host onboarding
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Managed Hosts
|
||||
|
||||
| Host | IP | User | Port | Type | Group |
|
||||
|------|-----|------|------|------|-------|
|
||||
| pve2 | .3 | root | 22 | Proxmox | proxmox_nodes |
|
||||
| pve-dell | .4 | root | 22 | Proxmox | proxmox_nodes |
|
||||
| replicant | .80 | maddox | 22 | VM | docker_hosts |
|
||||
| databases | .81 | root | 22 | VM | docker_hosts |
|
||||
| immich | .82 | root | 22 | VM | docker_hosts |
|
||||
| media-transcode | .120 | root | 22 | LXC | docker_hosts |
|
||||
| network-services | .121 | root | 22 | LXC | docker_hosts |
|
||||
| download-stack | .122 | root | 22 | LXC | docker_hosts |
|
||||
| docker666 | .123 | root | 22 | LXC | docker_hosts |
|
||||
| tailscale-home | .124 | root | 22 | LXC | docker_hosts |
|
||||
| dns-lxc | .125 | root | 22 | LXC | infrastructure |
|
||||
| nas | .251 | maddox | 44822 | NAS | legacy |
|
||||
| alien | .252 | maddox | 22 | Docker | legacy |
|
||||
|
||||
---
|
||||
|
||||
## Ansible Host Groups
|
||||
|
||||
| Group | Members | Use Case |
|
||||
|-------|---------|----------|
|
||||
| `all` | All 13 hosts | Connectivity tests |
|
||||
| `docker_hosts` | 8 hosts | Docker operations |
|
||||
| `all_managed` | 11 hosts | System updates |
|
||||
| `proxmox_nodes` | pve2, pve-dell | Node-level ops |
|
||||
| `infrastructure` | dns-lxc | Non-Docker infra |
|
||||
| `legacy` | nas, alien | Manual operations |
|
||||
| `vms` | replicant, databases, immich | VM-specific |
|
||||
| `lxcs` | 6 LXC containers | LXC-specific |
|
||||
|
||||
---
|
||||
|
||||
## Playbooks Reference
|
||||
|
||||
### check-status.yml
|
||||
Reports disk usage, memory usage, and container counts.
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/check-status.yml
|
||||
```
|
||||
|
||||
**Target:** all_managed
|
||||
**Output:** Per-host status line (Disk=X% Mem=X% Containers=X)
|
||||
|
||||
---
|
||||
|
||||
### update-all.yml
|
||||
Runs apt update and upgrade on all Docker hosts.
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/update-all.yml
|
||||
|
||||
# With reboot if required:
|
||||
ansible-playbook playbooks/update-all.yml -e "reboot=true"
|
||||
```
|
||||
|
||||
**Target:** docker_hosts
|
||||
**Note:** Checks for reboot requirement, notifies but doesn't auto-reboot unless `-e "reboot=true"`
|
||||
|
||||
---
|
||||
|
||||
### docker-prune.yml
|
||||
Cleans unused Docker resources (images, networks, build cache).
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/docker-prune.yml
|
||||
```
|
||||
|
||||
**Target:** docker_hosts
|
||||
**Note:** dns-lxc will fail (no Docker) - this is expected
|
||||
|
||||
---
|
||||
|
||||
### restart-utils.yml
|
||||
Restarts the utils stack (watchtower, autoheal, docker-proxy) on all hosts.
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/restart-utils.yml
|
||||
```
|
||||
|
||||
**Target:** docker_hosts
|
||||
**Note:** Uses host-specific `docker_appdata` variable for non-standard paths
|
||||
|
||||
---
|
||||
|
||||
### deploy-utils.yml
|
||||
Deploys standardized utils stack to a new host.
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/deploy-utils.yml --limit new-host
|
||||
```
|
||||
|
||||
**Target:** docker_hosts
|
||||
**Note:** Creates directory structure and .env file only; compose file must be added separately
|
||||
|
||||
---
|
||||
|
||||
## Scripts Reference
|
||||
|
||||
### ssh-manager.sh
|
||||
|
||||
Launches a tmux session with SSH connections to all hosts.
|
||||
|
||||
```bash
|
||||
~/scripts/ssh-manager.sh
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Window 0: Control (local shell)
|
||||
- Windows 1-13: Individual host SSH sessions
|
||||
- Final window: Multi-View (all hosts in split panes)
|
||||
|
||||
**Navigation:**
|
||||
- `Ctrl+b` then window number to switch
|
||||
- `Ctrl+b d` to detach (keeps session running)
|
||||
- `tmux attach -t cluster` to reattach
|
||||
|
||||
---
|
||||
|
||||
### control-menu.sh
|
||||
|
||||
Interactive menu for common operations.
|
||||
|
||||
```bash
|
||||
~/scripts/control-menu.sh
|
||||
```
|
||||
|
||||
**Menu Options:**
|
||||
```
|
||||
[1] Ping All - Test connectivity
|
||||
[2] Check Status - Disk/memory/containers
|
||||
[3] Update All - apt upgrade docker hosts
|
||||
[4] Docker Prune - Clean unused resources
|
||||
[5] Restart Utils - Restart utils stack everywhere
|
||||
|
||||
[A] Ad-hoc Command - Run custom command
|
||||
[I] Inventory - Show host list
|
||||
[S] SSH Manager - Launch tmux session
|
||||
|
||||
[Q] Quit
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### add-host.sh
|
||||
|
||||
Wizard for onboarding new hosts.
|
||||
|
||||
```bash
|
||||
~/scripts/add-host.sh
|
||||
```
|
||||
|
||||
**Steps:**
|
||||
1. Prompts for hostname, IP, user, port, description
|
||||
2. Tests SSH connectivity
|
||||
3. Copies SSH key if needed
|
||||
4. Adds to `~/.ssh/config`
|
||||
5. Adds to `~/.ssh/tmux-hosts.conf`
|
||||
|
||||
**Note:** Ansible inventory must be edited manually.
|
||||
|
||||
---
|
||||
|
||||
## Common Operations
|
||||
|
||||
### SSH to a Specific Host
|
||||
```bash
|
||||
ssh replicant
|
||||
ssh databases
|
||||
ssh nas # Uses port 44822 automatically
|
||||
```
|
||||
|
||||
### Run Command on All Docker Hosts
|
||||
```bash
|
||||
cd ~/clustered-fucks
|
||||
ansible docker_hosts -m shell -a "docker ps -q | wc -l"
|
||||
```
|
||||
|
||||
### Run Command on Specific Host
|
||||
```bash
|
||||
ansible replicant -m shell -a "df -h"
|
||||
```
|
||||
|
||||
### Copy File to All Hosts
|
||||
```bash
|
||||
ansible docker_hosts -m copy -a "src=/path/to/file dest=/path/to/dest"
|
||||
```
|
||||
|
||||
### Check Specific Service
|
||||
```bash
|
||||
ansible docker_hosts -m shell -a "docker ps --filter name=watchtower --format '{{.Status}}'"
|
||||
```
|
||||
|
||||
### View Ansible Inventory
|
||||
```bash
|
||||
ansible-inventory --graph
|
||||
ansible-inventory --list
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Git Workflow
|
||||
|
||||
### Repository Location
|
||||
- **Local:** `~/clustered-fucks/`
|
||||
- **Remote:** `ssh://git@192.168.1.81:2222/maddox/clustered-fucks.git`
|
||||
- **Web:** https://git.3ddbrewery.com/maddox/clustered-fucks
|
||||
|
||||
### Standard Workflow
|
||||
```bash
|
||||
cd ~/clustered-fucks
|
||||
|
||||
# Make changes to playbooks/inventory
|
||||
vim playbooks/new-playbook.yml
|
||||
|
||||
# Commit and push
|
||||
git add -A
|
||||
git commit -m "Add new playbook"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
### Pull Latest Changes
|
||||
```bash
|
||||
cd ~/clustered-fucks
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Host
|
||||
|
||||
### 1. Run Onboarding Script
|
||||
```bash
|
||||
~/scripts/add-host.sh
|
||||
```
|
||||
|
||||
### 2. Edit Ansible Inventory
|
||||
```bash
|
||||
vim ~/clustered-fucks/inventory/hosts.yml
|
||||
```
|
||||
|
||||
Add under appropriate group:
|
||||
```yaml
|
||||
new-host:
|
||||
ansible_host: 192.168.1.XXX
|
||||
ansible_user: root
|
||||
```
|
||||
|
||||
If non-standard appdata path:
|
||||
```yaml
|
||||
new-host:
|
||||
ansible_host: 192.168.1.XXX
|
||||
ansible_user: root
|
||||
docker_appdata: /custom/path/appdata
|
||||
```
|
||||
|
||||
### 3. Test Connection
|
||||
```bash
|
||||
ansible new-host -m ping
|
||||
```
|
||||
|
||||
### 4. Commit Changes
|
||||
```bash
|
||||
cd ~/clustered-fucks
|
||||
git add -A
|
||||
git commit -m "Add new-host to inventory"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### SSH Connection Refused
|
||||
```bash
|
||||
# Check if SSH is running on target
|
||||
ssh -v hostname
|
||||
|
||||
# If connection refused, access via Proxmox console:
|
||||
# For LXC: pct enter <CT_ID>
|
||||
# For VM: qm terminal <VM_ID>
|
||||
|
||||
# Inside container/VM:
|
||||
apt install openssh-server
|
||||
systemctl enable ssh
|
||||
systemctl start ssh
|
||||
```
|
||||
|
||||
### SSH Permission Denied
|
||||
```bash
|
||||
# Check key is in authorized_keys on target
|
||||
ssh-copy-id hostname
|
||||
|
||||
# If still failing, check permissions on target:
|
||||
# (via Proxmox console)
|
||||
chmod 700 ~
|
||||
chmod 700 ~/.ssh
|
||||
chmod 600 ~/.ssh/authorized_keys
|
||||
chown -R root:root ~/.ssh # or appropriate user
|
||||
```
|
||||
|
||||
### Ansible "Missing sudo password"
|
||||
The host is configured with `ansible_become: yes` but no password is set.
|
||||
|
||||
Fix: Either remove `ansible_become: yes` from inventory, or set up passwordless sudo on target:
|
||||
```bash
|
||||
echo "username ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/username
|
||||
```
|
||||
|
||||
### Playbook Skips Host
|
||||
Check if host is in the correct group:
|
||||
```bash
|
||||
ansible-inventory --graph
|
||||
```
|
||||
|
||||
Check host variables:
|
||||
```bash
|
||||
ansible-inventory --host hostname
|
||||
```
|
||||
|
||||
### Docker Command Not Found
|
||||
Host is in `docker_hosts` but doesn't have Docker. Move to `infrastructure` group:
|
||||
```yaml
|
||||
infrastructure:
|
||||
hosts:
|
||||
hostname:
|
||||
ansible_host: 192.168.1.XXX
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Non-Standard Configurations
|
||||
|
||||
### Hosts with Different Appdata Paths
|
||||
|
||||
| Host | Path |
|
||||
|------|------|
|
||||
| replicant | `/home/maddox/docker/appdata` |
|
||||
| docker666 | `/root/docker/appdata` |
|
||||
| All others | `/home/docker/appdata` |
|
||||
|
||||
These are handled via `docker_appdata` variable in inventory.
|
||||
|
||||
### Hosts with Non-Standard SSH
|
||||
|
||||
| Host | Port | User |
|
||||
|------|------|------|
|
||||
| nas | 44822 | maddox |
|
||||
|
||||
Configured in both `~/.ssh/config` and `inventory/hosts.yml`.
|
||||
|
||||
### Hosts Without Utils Stack
|
||||
|
||||
| Host | Reason |
|
||||
|------|--------|
|
||||
| tailscale-home | Only runs Headscale, no utils needed |
|
||||
| dns-lxc | No Docker installed |
|
||||
|
||||
---
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Update Ansible
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt upgrade ansible
|
||||
```
|
||||
|
||||
### Regenerate SSH Keys (if compromised)
|
||||
```bash
|
||||
# Generate new key
|
||||
ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519
|
||||
|
||||
# Distribute to all hosts (will prompt for passwords)
|
||||
for host in pve2 pve-dell replicant databases immich media-transcode network-services download-stack docker666 tailscale-home dns-lxc alien; do
|
||||
ssh-copy-id $host
|
||||
done
|
||||
|
||||
# NAS requires special handling
|
||||
ssh-copy-id -p 44822 maddox@192.168.1.251
|
||||
```
|
||||
|
||||
### Backup Configuration
|
||||
```bash
|
||||
cd ~/clustered-fucks
|
||||
git add -A
|
||||
git commit -m "Backup: $(date +%Y-%m-%d)"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Reference Files
|
||||
|
||||
### ~/.ssh/config
|
||||
```
|
||||
Host *
|
||||
StrictHostKeyChecking accept-new
|
||||
ServerAliveInterval 60
|
||||
ServerAliveCountMax 3
|
||||
|
||||
Host pve2
|
||||
HostName 192.168.1.3
|
||||
User root
|
||||
|
||||
Host pve-dell
|
||||
HostName 192.168.1.4
|
||||
User root
|
||||
|
||||
Host replicant
|
||||
HostName 192.168.1.80
|
||||
User maddox
|
||||
|
||||
Host databases
|
||||
HostName 192.168.1.81
|
||||
User root
|
||||
|
||||
Host immich
|
||||
HostName 192.168.1.82
|
||||
User root
|
||||
|
||||
Host media-transcode
|
||||
HostName 192.168.1.120
|
||||
User root
|
||||
|
||||
Host network-services
|
||||
HostName 192.168.1.121
|
||||
User root
|
||||
|
||||
Host download-stack
|
||||
HostName 192.168.1.122
|
||||
User root
|
||||
|
||||
Host docker666
|
||||
HostName 192.168.1.123
|
||||
User root
|
||||
|
||||
Host tailscale-home
|
||||
HostName 192.168.1.124
|
||||
User root
|
||||
|
||||
Host dns-lxc
|
||||
HostName 192.168.1.125
|
||||
User root
|
||||
|
||||
Host nas
|
||||
HostName 192.168.1.251
|
||||
User maddox
|
||||
Port 44822
|
||||
|
||||
Host alien
|
||||
HostName 192.168.1.252
|
||||
User maddox
|
||||
```
|
||||
|
||||
### ~/clustered-fucks/ansible.cfg
|
||||
```ini
|
||||
[defaults]
|
||||
inventory = inventory/hosts.yml
|
||||
remote_user = root
|
||||
host_key_checking = False
|
||||
retry_files_enabled = False
|
||||
gathering = smart
|
||||
fact_caching = jsonfile
|
||||
fact_caching_connection = /tmp/ansible_facts
|
||||
fact_caching_timeout = 86400
|
||||
stdout_callback = yaml
|
||||
forks = 10
|
||||
|
||||
[privilege_escalation]
|
||||
become = False
|
||||
|
||||
[ssh_connection]
|
||||
pipelining = True
|
||||
ssh_args = -o ControlMaster=auto -o ControlPersist=60s
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
| Date | Change |
|
||||
|------|--------|
|
||||
| 2026-01-23 | Initial deployment, all hosts connected, playbooks tested |
|
||||
Loading…
Reference in a new issue