Flatcar Linux

Flatcar Linux is the ideal guest operating system to run containerized applications on your homelab hypervisor.

The system is tiny like Alpine, but doesn't come with the same maintenance headaches as Alpine. The system boots very fast. The system is immutable making it very hard to break. The system installs its own updates, reducing the amount of ongoing maintenance needed.

Prerequisites

This article assumes that you have a working Proxmox server already installed, and that you are familiar with the use of Docker and SSH Keys.

Writing a Butane Config

Unlike traditional server distros like Ubuntu Server that get configured by logging into the guest during and after installation, Flatcar is configured through config files that live on the host machine, in this case Proxmox.

Instead of writing the configuration directly on the Proxmox host, Flatcar uses a local configuration checker and compiler called "Butane" that turns an easy-to-edit YAML file into the "Ignition" JSON file that you will ultimately upload to your Proxmox server.

The exact configuration you choose will depend on your needs, but I recommend at least setting up SSH keys and networking.

# Indicate the Butane version to be used
variant: flatcar
version: 1.0.0

# Set up the user account. The "core" user is the default, but you can provide
# your own name if you prefer. Put the public portion of your SSH key here so
# that you can access the server immediately after installation.
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - sk-ssh-ed25519@openssh.com ....

storage:
  links:
    # Enable the docker service so that it can be auto-started
    - path: /etc/systemd/system/multi-user.target.wants/docker.service
      target: /usr/lib/systemd/system/docker.service
      hard: false
      overwrite: true
  files:
    # Set the networking information. Use the details from your own network.
    # Be aware that flatcar is IPv6-capable so you may want to set IPv6 details
    # as well.
    - path: /etc/systemd/network/static.network
      contents:
        inline: |
          [Match]
          Name=eth0

          [Network]
          Address=192.168.0.1/24
          Gateway=192.168.0.1
          DNS=192.168.0.1

systemd:
  units:
    # Start docker automatically instead of waiting for the user to run docker
    # commands
    - name: docker.service
      enabled: true

    # Change the SSH server port. This is necessary if you intend to run a
    # service like Gitea that wants the SSH port for itself. If you aren't
    # planning to run any services like that, this is unnecessary.
    - name: sshd.socket
      dropins:
      - name: 10-sshd-port.conf
        contents: |
          [Socket]
          ListenStream=
          ListenStream=2222
          FreeBind=true

Once you've created your butane file, you can turn it into the ignition JSON using a command like this from your local machine. You may need to tweak the command slightly if you're running Windows. You could also download the butane command directly on your proxmox host, if you don't want to run it locally.

# If you have Docker installed locally and don't want to install Butane directly
docker run --rm -i quay.io/coreos/butane:latest < your_config.yaml
# Then copy the output so you can paste it into Proxmox later

# If you don't have Docker installed locally or want to install Butane directly
# either on your computer or on your proxmox host. Check below for the latest
# release URLs.
# https://github.com/coreos/butane/releases
wget https://github.com/coreos/butane/releases/download/v0.28.0/butane-x86_64-unknown-linux-gnu -O butane
chmod +x butane
./butane < your_config.yaml
# then copy the output so you can paste it into Proxmox later

# If you install Butane onto your Proxmox host, you can write your YAML file
# from the shell:
nano your_config.yaml
# Then you can write the Butane output directly to where it belongs:
mkdir -p /var/lib/vz/snippets # Make sure the folder exists
./butane < your_config.yaml > /var/lib/vz/snippets/your_config

Installing Flatcar on Proxmox

Flatcar has their own instructions for installing a Flatcar Linux VM on your Proxmox host, but those instructions assume you want to manage your system through the shell. It is also possible to manage it through the GUI, for which I'll provide instructions here. You can't do the whole installation through the GUI, but it makes configuring things like disk space, memory, storage pools, and so on just a bit easier.

It's important to note that Flatcar provides instructions for setting up the VM using just the regular CloudInit UI and no Butane config. This is a TRAP! Doing that leaves you with a mostly useless system, and it is painful to fix it afterwards. It's fine for just poking around, but when you actually want to deploy a service on the VM you'll regret doing it.

Go to the Flatcar Releases page and choose the release channel you're interested in. I chose "Stable". Click the architecture you'll be running on, most likely amd64 to be brought to their release file server. Search for "proxmox" and download the proxmox installation image flatcar_production_proxmoxve_image.img (it should be about 500MB).

Browse to your Proxmox server's storage where you keep ISOs. By default, that's the "local" storage listed at the bottom of the left-hand tree menu. Open the "ISO Images" page and upload the .img file to your proxmox server.

Create your VM, assigning it all the core counts and memory that you want, but do not start it. Go to the Hardware settings and add a "CloudInit" drive on whatever storage you prefer. Remove any disk you may have created. Before starting it, we will need to go to the shell so we can import the Flatcar disk and apply the butane config.

Click on your Proxmox node in the left-hand tree menu (the item just below the top-level "Datacenter" item). Assuming you're signed in as the default root user, go to the "Shell" tab and run these commands:

export VM_ID=123 # Put your VM ID in here

# If you did not already do it from the GUI, create the CloudInit drive that
# will deliver your config files to the VM 
qm set $VM_ID --ide2 local-lvm:cloudinit 

# Import the flatcar disk. If you need your VM to have more than minimal disk
# space, you should return to the Hardware tab later and expand this disk.
qm disk import $VM_ID flatcar_production_proxmoxve_image.img local-lvm

# Make sure the folder for Ignition configs exists
mkdir -p /var/lib/vz/snippets 

# Put the Ignition json into the file with whatever tool you prefer. If you
# ran Butane directly on your Proxmox host and wrote the output directly here,
# you can skip this step.
cat > /var/lib/vz/snippets/your_config <<'EOF'
{"ignition":{"version":"3.3.0"},... (PASTE YOUR JSON HERE)
EOF

# Set your VM to use the config. Make sure the last part "your_config" matches
# the file name you chose.
qm set $VM_ID --cicustom "user=local:snippets/your_config"

If you desire, you can also add a Serial device to the VM. This will disable the NoVNC console and give you an xterm.js console instead. The advantage of this is that you'll be able to paste directly into the VM without using SSH. However, once SSH is set up, there's not much of a practical difference.

Now you can start your Flatcar VM, and it should install itself entirely on its own after a minute or so.

Notes

If you create the VM from the shell, the documentation says to create the VM using the --ipconfig0 "ip=dhcp" flag. However, if you do this it will override the Butane networking settings you put into your YAML config, so be careful of that.

If you download the Flatcar image and upload it through the GUI, then want to create a Flatcar VM through the shell afterward, the "local" storage is at the path /var/lib/vz/template/iso/flatcar_production_proxmoxve_image.img.

Deploying Services

Once your Flatcar VM is installed, you'll want to access it and deploy your services onto it.

First, SSH into flatcar at the static IP address you assigned it, using the username you assigned it.

ssh core@192.168.0.2
# or if you set up Flatcar with an alternate SSH port
ssh -p 2222 core@192.168.0.2

Once you have access to the shell, you will likely want to install docker compose. I recommend following the official instructions to install the plugin manually. Because Flatcar is built on a immutable base, most software can't be installed directly to Flatcar like this. However, because compose is a plugin instead of a standalone package, it can be installed to the writable home folder just fine.

Flatcar ships with vim by default, rather than something like nano, so you can either use vim to write your docker-compose.yml file or you can write it locally and upload it using a tool like SFTP or SCP. On Windows, you can use WinSCP, or an SSH client with a built-in file transfer tool like MobaXTerm. If you are running Windows 11 or later, the standard scp command should also be available.

scp docker-compose.yml core@192.168.0.2:
# or if you set up Flatcar with an alternate SSH port
scp -P 2222 docker-compose.yml core@192.168.0.2:

Changing Your Flatcar Configuration

If you make changes to your Butane config after installing Flatcar, the "official" method of updating your VM is to delete the VM and provision a new one. That might be easy if you're using deployment tools like terraform and ansible, and have persistent network storage instead of using storage local to the VM, but for a budding homelab it can be a real pain. So instead, we'll use the less-official method of "resetting" the Flatcar VM to a state where it will re-run its initialization scripts.

First, make the appropriate changes to your Butane configuration then refer back to the instructions for converting it to Ignition JSON and uploading it to the Proxmox server.

Next, after updating the flatcar config on the proxmox host, regenerate the VM's CloudInit ISO by running this command on the Proxmox host shell (putting your own VM's ID in place of 123)

qm cloudinit update 123

Next, log into your Flatcar VM's shell and run this command to reset it:

sudo flatcar-reset --keep-machine-id --keep-paths /home/core /var/lib/docker /etc/ssh

/home/core is the user folder, the default working directory you get when you access the shell. This is where I keep all my compose files, so I wouldn't want to lose it.
/var/lib/docker is all the data associated with your configured docker instances, volumes, etc
/etc/ssh contains the SSH host keys. Keeping this folder prevents "remote host key changed" errors from appearing when you try to SSH into the system after changing its configuration.

Now you can reboot your Flatcar VM and it should accept the new Butane/Ignition configuration.

Why Not Lightwhale Linux?

A substantially similar Distro is available called Lightwhale Linux, that comes pre-configured for running Docker with no up-front configuration. However, Lightwhale linux does not have the kind of support behind it that I would want from a long-term deployment platform. Lightwhale is the independent project of just one guy and a community, Stephan Henningsen of Denamrk.

Flatcar Linux, on the other hand, is managed by LF Projects, LLC (a subsidiary of the Linux Foundation) and is widely deployed to the tune of tens or hundreds of thousands of instances by companies like Adobe and AT&T, DeepL, and Qualys. So despite the increased up-front costs of configuring Flatcar, the long-term viability appears to be much higher than Lightwhale.