I use VMs for OS isolated and hardware isolation. I run containers on top of OSs for apps.

That said I use Harvester so I can run containers on bare metal AND VMs as containers.

Even on one bare metal I do this, so I can spin up a test VM of harvester to confirm configs. If you one node is super beefy a neat trick is to have VMs running a cluster so you can have a kind of high availability even if your infra is a single point of failure.

Security is a big reason to use VMs. Containers share the kernel with the host. That means that any of the kernel vulnerabilities from this year (like DirtyFrag and CopyFail) could have been used to compromise your host. Once the host is compromised, the only way to be safe is to literally buy a new machine. Seriously. Viruses can bury deep and even infect the motherboard firmware to persist indefinitely.

Kernel vulnerabilities are frequent enough that I find VMs worth it. I still use containers in my VMs though.

It’s not an either-or scenario. Running services in Docker/Podman is great and makes a lot of sense, as you’ve found. But there’s no reason the OS running those Docker containers can’t be a VM on a hypervisor like Proxmox. Then you get the simplicity of Docker, in addition to the isolation and segmentation (network and process) provided by VMs, and snapshot-based incremental backups from PBS. It’s the best of both worlds. You wouldn’t have a VM per service like you ran before, instead you’d have a VM per group of related services with common networking and security requirements. For example, all of your publicly exposed services can run in Docker in their own isolated VM that’s walled off from the rest of your network, while your internal-only services also run in Docker, but on a separate VM on your internal network.

Proxmox running LXC’s and VM’s, all of them also have docker installed

I’ll die on this hill.

Containers run on “bare metal” in the same way that other processes on your system run.

Proxmox exclusively with unprivileged LXCs, which themselves host other docker containers/stacks managed with Dockhand (a more modern alternative to both Portainer and Dockge). I have a very tiny low power machine so I have to be kind with my resources, therefore no VMs. But LXCs are simpler anyway. I can easily pass devices like /dev/tun for tailscale and i forget the name of the iGPU for accelerated workloads (for Immich, Nextcloud).

Reasoning: with containers it’s easy to make mistakes without dire consequences (just docker compose down -v and start over). The LXCs are easy to back up and restore. They hold internal running state of whatever docker containers I run in them.

Pro tip: don’t map host paths using the UI (mp0 etc), use lxc.mount instead. This will let you continue to have snapshots of your LXCs whereas the other approach makes your LXC incompatible with snapshots.

Good luck!

I use Proxmox as a VDI server. I understand that this isn’t homelab territory, but selfhosted VDIs have proven exponentially more reliable and easier to manage than cloud based ones (after the initial PitA setup). Flawless copy/paste, screen resize, and most importantly, file transfers. When you connect to 12 different clients, each with a different set of security requirements, system hardening, monitoring, and VPN access, being able to console amd fully interact with a sandboxed desktop becomes priceless.

I moved away from proxmox about 3 years ago to incus, primarily because Proxmox has a lot going on that I don’t need, but also incus is much more amenable to configuration with ansible. Also, and maybe this has changed recently, but passing through PCI devices to incus is much easier.

There are different reasons to run different things. Some apps are not able to be run as containers, or don’t make sense to run containerized.

VMs offer better isolation between systems, which is important for high-security applications.

Traditional Hypervisors also allow you to easily run fleets of completely different OSes on the same hardware, which you can’t do with containers.

On the other hand, if you want/need high scalability, extremely efficient resource usage, and you don’t need to run radically different OS environments simultaneously, containers, especially Docker containers or Podman containers are a great choice.

Incus containers are a great middle ground that I’m a fan of more than Docker/podman containers or traditional VMs. They are system containers, vs application containers like Docker/Podman.

That means they are actually full-fledged Linux distros by default, but are still much smaller than a traditional VM.

Also, for any of these solutions, if you’re constantly doing admin tasks manually, you’re doing it wrong. All modern platforms have APIs or other ways to interact with them via code.

For my own setup, I have everything. My main server runs XCP-ng for the hypervisor, and I have several traditional VMs on it, including my primary NAS. I also run a virtual Incus host that I have my Minecraft server running in, and some random other system containers. I have a Docker host VM, but I don’t really use it.

I also have my home media server which is a standalone system running TrueNAS on bare metal with Tailscale and Jellyfin running as Docker containers within TrueNAS.

My setup is messy because it has been built slowly over several years, and I don’t have enough time or energy to rip it all down and put it back together in a more optimal way.

Been using proxmox for 10+years. It is rock solid. Cluster of 3 automatically handles VM movement. Plenty of options for snaps, backups, etc. Tons of support out there too

Having to manually start and stop your VMs is very atypical, proxmox can absolutely handle that itself

Proxmox with LXC’s and a few VM’s.
Often with Portainer as well.

It’s just nice to use and easy to automate backups.

I used to run barebone on Debian, but I accepted that I’m not good enough in the terminal and to used to graphical solutions to go back.

I don’t care much for proxmox but that’s because I live in the terminal anyway and have a lot of experience administering Linux systems. So for me it’s just an extra layer of complexity, the benefits don’t really apply when my setup has IaC, most of my containers are in Podman rather than LXCs, my backups are automated with my own scripts and cronjobs, and I only run a couple of VMs. Also it makes encrypting the base system more difficult than it is with a standard distro

because sometimes i need a vm (home assisstant has more features thsn home assisstant container), sometimes a container (i mostly use lxc).

its not really an extra lauer because proxmox is an os.you use nixos i use proxmox. everything is out of the box. proxmox has comminntiu scripts to run everything and set them up. there is an i stall script for about everything https://community-scripts.org/

sure podman has similar setup but i also need vms and they are all mamaged in one place

EDIT: you can read the script, its text

Docker has its uses, but it has gone to far.

I already have a webserver with Apache, mariadb and everything. Why does your PHP based website only come as a docker solution? Fuck you.

When you go all in with docker, it quickly becomes a mess with rando NICs and containers named “random 32 character string” and dependencies up the wazoo. Ending up running ancient packages because the developer refuses to apt upgrade their shit.

Proxmox or any other hypervisor platform gives you control. Whereas you have to take it for docker and youre still at the mercy or the developers to keep your shit secure.