podbox

Installation

My proof of concept server running this container stack is built on AlmaLinux 9.4. podman and systemd with quadlet support is required if you are using another distro.

[!WARNING] Perform dnf update immediately

Repositories

These may not really be necessary to set up, but you should absolutely review them and decide for yourself.

• AlmaLinux
• CentOS SIGs
• Extra
  • EPEL and CRB
    • dnf install epel-release
    • dnf config-manager --set-enabled crb
  • ELRepo
    • dnf install elrepo-release
  • RPM Fusion

Disks

Partitions

Repeat the following steps for all disks that you want to join together into one single logical volume.

# Find /dev/sdX paths for disks
# WARNING: Make sure you confirm the disk is correct
lsblk -f
# Clear the partition table
dd if=/dev/zero of=/dev/sdX bs=512 count=1 conv=notrunc
# Create LVM partition
parted --fix --align optimal --script /dev/sdX \
    mklabel gpt \
    mkpart primary ext4 1MiB -2048s \
    set 1 lvm on

LVM

# Create physical volume
pvcreate /dev/sdX1
# Create volume group for disks
vgcreate library /dev/sdX1
# Add more disks to volume group
vgextend library /dev/sdY1
# Create logical volume across all disks in volume group
lvcreate -l100%FREE -n books library
# Add filesystem to logical volume
mke2fs -t ext4 /dev/library/books
# Check it
e2fsck -f /dev/library/books

/etc/systemd/system/volumes-books.mount

[Mount]
What=/dev/library/books
Where=/volumes/books
Type=ext4

[Install]
WantedBy=default.target

mkdir -p /volumes/books
chown -R $ctuser:$ctuser /volumes

SSH

SSH is optional, but highly encouraged. Cockpit gives you a terminal too, but that's nowhere near as good as what you can do with a real terminal emulator and ssh clients.

dnf install openssh-server

## Generate strong key on your laptop or workstation/desktop
ssh-keygen -t ed25519 -a 32 -f ~/.ssh/"$localhost-to-$remotehost"

## Copy key to AlmaLinux
ssh-copy-id -i ~/.ssh/"$localhost-to-$remotehost" "$user@$remotehost"

Override sshd config

We don't want to allow anyone to login as root remotely ever. You must be a sudoer with public key auth to elevate to root.

printf '%s\n' 'PermitRootLogin no' > /etc/ssh/sshd_config.d/01-root.conf
printf '%s\n' \
    'PubkeyAuthentication yes' \
    'PasswordAuthentication no' > /etc/ssh/sshd_config.d/01-pubkey.conf

Cockpit -> https://ip-addr:9090

[!WARNING] Disable the firewall if you are lazy Exposing ports for other services can be exhausting and I have not learned how to do this for containers properly. Each container may need a new rule for something, not sure.

systemctl disable --now firewalld

[!TODO] Should be able to set up good firewall with only 80/443 open.

Enable the socket-activated cockpit service and allow it through the firewall.

systemctl enable --now cockpit.socket

# FIXME: Unnecessary? Default works?
firewall-cmd --permanent --zone=public --add-service=cockpit
firewall-cmd --reload

Add SSH keys

[!TIP] Skip if you copied your keys with ssh-copy-id above.

Accounts -> Your account -> Authorized public SSH keys -> Add Key

Install SELinux troubleshoot tool

This is a component for Cockpit.

dnf install setroubleshoot-server

Podman

Podman is a daemonless container hypervisor. This document prepares a fully rootless environment for our containers to run in.

Install

dnf install podman
systemctl enable --now podman

[!NOTE] Read the docs. man podman-systemd.unit

slirp4netns

[!TODO] This may not be necessary but my system is currently using it.

dnf install slirp4netns

Install DNS server for podman

[!TODO] Not sure how to resolve these correctly yet but the journal logs it so it's running for something.

dnf install aardvark-dns

Enable unprivileged port binding

[!NOTE] This is only necessary if you are setting up the reverse proxy.

printf '%s\n' 'net.ipv4.ip_unprivileged_port_start=80' > /etc/sysctl.d/99-unprivileged-port-binding.conf
sysctl 'net.ipv4.ip_unprivileged_port_start=80'

Prepare container user

This user will be the owner of all containers with no login shell or root privileges.

# Prepare a group id outside of the normal range
groupadd --gid 2000 $ctuser
# Create user with restrictions
# We need the $HOME to live in
useradd --create-home \
    --shell /usr/bin/false \
    --password $ctuser_pw \
    --no-user-group \
    --gid $ctuser \
    --groups systemd-journal \
    --uid 2000 \
    $ctuser
# Lock user from password login
usermod --lock $ctuser
# Add container sub-ids
usermod --add-subuids 200000-299999 --add-subgids 200000-299999 $ctuser
# Start $ctuser session at boot without login
loginctl enable-linger $ctuser

[!TIP] Optionally setup ssh keys to directly login to $ctuser.

[!NOTE] The login shell doesn't exist. Launch bash -l manually to get a shell or else your ssh will exit with a status of 1.

Setup $ctuser env

# Switch to user (`-i` doesn't work without a login shell)
sudo -u $ctuser bash -l
# Create dirs
mkdir -p ~/.config/{containers/systemd,environment.d} ~/containers/storage
# Prepare `systemd --user` env
echo 'XDG_RUNTIME_DIR=/run/user/$UID' >> ~/.config/environment.d/10-xdg.conf
# Enable container auto-update
podman system migrate
# WARNING: Set strict versions for all containers or risk catastrophe
systemctl --user enable --now podman-auto-update
exit

[!WARNING] I disabled SELinux to not deal with this for every container. /etc/selinux/config -> SELINUX=disabled

[!TODO] Set up the correct policies permanently instead of disabling SELinux

Temporarily set SELinux policy to allow containers to use devices.

setsebool -P container_use_devices 1