Recently, a friend of mine showed me a project that lets people run a private World of Warcraft server. I saw a great opportunity to learn how to use Podman. What an adventure this has been, and I’m not talking about playing WoW.

First things first, what is AzerothCore?

AzerothCore is an open-source implementation of a World of Warcraft 3.3.5 server. If you connect an unmodified client (not included in AzerothCore) to it, then you can play on your very own server with your own rules.

Interestingly, AzerothCore does not provide pre-built binaries. Instead, users are instructed to build it from source themselves. This makes the barrier to entry quite high, especially for non-technical people. However, I have to give a huge shout-out to the documentation AzerothCore has created. I must say that I have never seen such a well-documented project of this caliber. Every little step is described, and no gaps are left open. That way, we were able to get the server running on Windows on my friend’s PC quite quickly.

After seeing the pain of setting everything up on Windows, though, I was motivated to do it right on Linux: installing random executables just to get libraries, dealing with files spread all over the place, and copying .dlls from OpenSSL, among other things.

Preamble

The hardware and OS

I’m mainly including this section because I wanted to highlight how usable old hardware can be. My rather old PC, with an AMD Ryzen 1700X and Nvidia GTX 970, was collecting dust, and I never dared to use it because of its power consumption. Sure enough, even while idle and doing nothing, it consumed around 80W. At full CPU load, it consumed up to 160W. With some undervolting, I got this down by around 10W.

But the big change was getting rid of the GPU. Mind you, the Ryzen 1700X does not include integrated graphics, so I always assumed it would not work. But apart from the motherboard speaker beeping at boot, it is actually fully usable. After removing the GPU, idle power consumption was down to 25W and reached 100W under load. That’s actually quite usable for what you get, in my opinion!

On to the operating system. Since I’ve used it before, I installed openSUSE MicroOS. I like having an immutable OS as a base because it lets me upgrade the OS without much hassle or fear of breaking things. But this also leads me to…

Why Podman?

For years now, Docker has become the de facto standard for the entire container lifecycle. From building to running, everything knows Docker. So why diverge?

First of all, Docker runs containers as root by default. Personally, I don’t like running code I didn’t write through something that has root privileges, even if the container itself does not run as root. The daemon still adds to the attack surface. Rootless Docker does exist, but it has edge cases. I’ve personally run into permission issues with volumes and networks. So far, everything has been fixable, but you definitely feel that rootless Docker is not the “happy path” everyone uses. Podman, on the other hand, is primarily rootless and does not use a daemon process. Podman describes this as “one less single point of failure,” but honestly, I have never heard of a Docker daemon causing availability issues. To me, a leaner approach to managing containers is the main goal here.

Then, Podman has first-class support from both openSUSE and Fedora. Fedora I use daily on my workstation, openSUSE MicroOS is currently my toy-server OS of choice. So I thought there must be some backing to it from very smart people.

Finally, I just like learning things. Betting all your money on a single horse is never a good idea. Diversifying basically always pays off. So maybe I’m about to strike gold?

Journey to production

AzerothCore essentially consists of two services and one database:

  • worldserver is responsible for the game logic
  • authserver handles login
  • mysql is shared between worldserver and authserver

Both worldserver and authserver interact with a properly initialized database. Thus, they include migration scripts, which are executed on boot. For Docker deployments, however, initializing the database is done via a separate container. The project calls this container db-import, so I’ll use the same name here. Furthermore, worldserver requires the “client data.” This data contains information about the map for tasks such as line-of-sight checks and pathfinding. The client data is not included in AzerothCore, but can be freely downloaded from another GitHub repo.

In total, we end up with this dependency graph:

  block-beta
  columns 5
  world["worldserver"] space userdata["user-data"] space:2
  space:5
  space:2 dbinit["db-import"] space db[("DB")]
  space:5
  auth["authserver"]

  world --> userdata
  world --> dbinit
  auth --> dbinit
  dbinit --> db
  world --> db
  auth --> db

  block-beta
  columns 5
  world["worldserver"] space userdata["user-data"] space:2
  space:5
  space:2 dbinit["db-import"] space db[("DB")]
  space:5
  auth["authserver"]

  world --> userdata
  world --> dbinit
  auth --> dbinit
  dbinit --> db
  world --> db
  auth --> db

The important part here is that worldserver does not commit changes to the database all the time. Instead, they get aggregated and written out every now and then. This is important because latency is king for a game server1. However, this introduces unique consistency challenges. Most importantly, if we do a controlled shutdown of the worldserver, we have to wait until the latest data is written out.

However, this adds an interesting challenge: We must introduce a fixed shutdown order of our services. If the database halts before or while the worldserver is still writing changes, data might be lost or, worse, corrupted.

systemd to the rescue

For those unaware, systemd is a really powerful and by now the de facto standard init system for Linux. It is responsible for orchestrating all the different services you might need. Most importantly for us, you can define dependencies between those services and those dependencies are respected during boot, but also during shutdown. Since triggering a shutdown or a reboot is always done via systemd, we therefore have quite good guarantees for the order of execution as well as the order of termination.

Fortunately, Podman has excellent support for systemd services in the form of “quadlets”. The idea here is simple: You define what containers you need. Podman generates systemd services from them. The level of abstraction is very thin here, though, as the quadlet definitions themselves are written like regular systemd services. Most importantly, they use the same syntax.

Here is a simple example of a quadlet container:

[Unit]
Description=Minimal Demo

[Container]
# Podman-specific instructions - those are converted
# to systemd ExecStart instructions
Image=docker.io/library/ubuntu:26.04
Exec=echo "Hello from container" && sleep 60

[Install]
# Start this systemd unit when the server boots
WantedBy=multi-user.target

If you run podman quadlet install <file>, then this quadlet will be copied over to $HOME/.config/containers/systemd/ (by default) and make the container available as a systemd service. You can get the actual content of the systemd service by inspecting what the podman-system-generator does:

# /usr/lib/systemd/system-generators/podman-system-generator --user --dryrun
---00-minimal.service---
[Unit]
Wants=podman-user-wait-network-online.service
After=podman-user-wait-network-online.service
Description=Minimal Demo
SourcePath=/home/shayden/.config/containers/systemd/00-minimal.container
RequiresMountsFor=%t/containers

[X-Container]
# Podman-specific instructions - those are converted to systemd ExecStart instructions
Image=docker.io/library/ubuntu:26.04
Exec=echo "Hello from container" && sleep 60

[Install]
# Start this systemd unit when the server boots
WantedBy=multi-user.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/bin/podman rm -v -f -i systemd-%N
ExecStopPost=-/usr/bin/podman rm -v -f -i systemd-%N
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name systemd-%N --replace --rm --cgroups=split --sdnotify=conmon -d docker.io/library/ubuntu:26.04 echo "Hello\x20from\x20container" && sleep 60

Creating quadlets from a Docker Compose file

Since AzerothCore already provides a very well-written docker-compose.yml, the first and most sensible approach seemed to be converting it to quadlets. A third-party tool called podlet promises to do most of the work.

However, the provided docker-compose.yml is quite advanced (for example, it uses environment variable interpolation such as USER_ID: ${DOCKER_USER_ID:-1000}), so podlet fails to convert it automatically. After trying to remove everything that kept failing, I pivoted to using AI instead. After all, AI should be great at systemd services and, by extension, at Podman quadlets too.

So I asked Claude Opus to whip something up for me, and whip something up it did. It created .container and .volume quadlets for everything, as well as shell scripts to help me push the quadlets to my server from my work PC, install them, create database backups via a systemd timer, initialize secrets, and more. This was very valuable. Without it as a starting point, I think I would have given up before even starting to write the first quadlet.

One reason is that the syntax of quadlets isn’t very nice. With a Docker Compose file, I’m used to having everything in one easy-to-follow place. You can read the file from top to bottom and know everything there is to know. With quadlets, all the dependencies are loosely linked via their service names.

For example, this is the final quadlet I ended up with for my database (comments mostly written by AI):

# MySQL 8.4 for AzerothCore.
#
# Notify=healthy makes systemd consider this unit "started" only once the
# healthcheck passes. Everything that declares After=ac-database.service
# therefore starts against a database that actually accepts queries -- this is
# the quadlet equivalent of Compose's `condition: service_healthy`.
#
# Installed to: ~/.config/containers/systemd/ac-database.container

[Unit]
Description=AzerothCore MySQL database
# Stop propagation and start-up come from the pod: quadlet gives this unit
# BindsTo=/After=azerothcore-pod.service automatically.

[Container]
ContainerName=ac-database
Image=docker.io/library/mysql:8.4
Pod=azerothcore.pod

# Non-secret settings only (TZ etc). The root password arrives as a mounted
# secret, never as an environment variable.
EnvironmentFile=%h/.config/azerothcore/ac.env

# Mounted at /run/secrets/mysql-root-password, mode 0444 so the in-container
# MySQL user (UID 999) can read it. Using the _FILE variant keeps the password
# out of the container environment entirely -- it will not show up in
# `podman inspect ac-database`, nor in /proc/<pid>/environ.
Secret=ac-mysql-root-password,type=mount,target=mysql-root-password,mode=0444
Environment=MYSQL_ROOT_PASSWORD_FILE=/run/secrets/mysql-root-password

Volume=ac-database.volume:/var/lib/mysql
Volume=%h/azerothcore/db-backup:/backup:Z
Volume=%h/.config/azerothcore/mysql/00-azerothcore.cnf:/etc/mysql/conf.d/00-azerothcore.cnf:ro,Z
Volume=%h/.config/azerothcore/mysql/healthcheck.sh:/usr/local/bin/ac-healthcheck.sh:ro,Z

# The healthcheck is a script rather than an inline command so that the secret
# is read inside the container, not expanded by systemd on the host.
HealthCmd=/usr/local/bin/ac-healthcheck.sh
HealthInterval=10s
HealthTimeout=5s
HealthRetries=30
HealthStartPeriod=20s
Notify=healthy

# Give InnoDB time to flush dirty pages on shutdown before podman escalates
# to SIGKILL. Raise this if you run a very large buffer pool.
# Deliberately NO AutoUpdate= here. A database restart is the heaviest and
# riskiest operation in this stack, and `local` policy would fire on any
# stray `podman pull mysql:8.4`. Bounce MySQL by hand, deliberately:
#     systemctl --user restart ac-database.service
StopTimeout=300

User="%U:%G"

[Service]
Restart=on-failure
RestartSec=10
# First boot has to initialize the data directory; be patient.
TimeoutStartSec=900
TimeoutStopSec=330

[Quadlet]
DefaultDependencies=false

First of all, you may notice how verbose this configuration is compared with a Docker Compose file, even though it achieves basically the same thing. Secondly, notice how the container references a volume mount for ac-database.volume, but there is no definition for it. That is because you need to define it in its own quadlet file:

# MySQL data directory.
#
# Kept as a named volume so Podman owns the UID mapping in rootless mode.
# Inspect the real path with:  podman volume inspect ac-database
[Volume]
VolumeName=ac-database

Compared with a Docker Compose file, this creates a very loose coupling between the two. You, as a system administrator, have to ensure that everything is in place in the correct order.

Now let’s move on to the next step. We want to start something that depends on the database, like the db-import step from above:

# One-shot: creates the databases if missing and applies pending SQL updates.
#
# Runs to completion before authserver/worldserver are allowed to start, which
# is the quadlet equivalent of Compose's `condition: service_completed_successfully`.
# Both game servers are built with AC_UPDATES_ENABLE_DATABASES=0, so this
# container is the single writer of schema migrations -- no race between them.
#
# Installed to: ~/.config/containers/systemd/ac-db-import.container

[Unit]
Description=AzerothCore database import and migrations
Requires=ac-database.service
After=ac-database.service

[Container]
ContainerName=ac-db-import
Image=localhost/acore/ac-wotlk-db-import:prod
Pod=azerothcore.pod

# Non-secret settings only (TZ etc).
EnvironmentFile=%h/.config/azerothcore/ac.env

# AzerothCore has no *_FILE convention, so the DSNs must be environment
# variables. type=env still keeps them out of `podman inspect` output, unlike
# the plain EnvironmentFile values which are expanded into Config.Env.
Secret=ac-dsn-login,type=env,target=AC_LOGIN_DATABASE_INFO
Secret=ac-dsn-world,type=env,target=AC_WORLD_DATABASE_INFO
Secret=ac-dsn-characters,type=env,target=AC_CHARACTER_DATABASE_INFO
Secret=ac-dsn-playerbots,type=env,target=AC_PLAYERBOTS_DATABASE_INFO

Environment=AC_DATA_DIR=/azerothcore/env/dist/data
Environment=AC_LOGS_DIR=/azerothcore/env/dist/logs
# Create the schemas on a virgin database instead of bailing out.
Environment=AC_FORCE_CREATE_DB=1

Volume=%h/azerothcore/etc:/azerothcore/env/dist/etc:Z
Volume=%h/azerothcore/logs:/azerothcore/env/dist/logs:Z

# A new db-import image means new migrations, so re-run it on update. The game
# servers declare PartOf=ac-db-import.service, so they are stopped around it.
AutoUpdate=local

[Service]
Type=oneshot
RemainAfterExit=yes
# A first-time world DB import is long.
TimeoutStartSec=3600
Restart=no

[Quadlet]
DefaultDependencies=false

I need to specify that this container requires another service and must run after it via the Requires and After directives respectively. Since these directives are copied over to the final systemd service, they must reference the actual systemd service name. We can’t write ac-database.container, the name of the quadlet. This is just a minor annoyance, but to me it shows how inconsistent the experience is within Podman quadlets.

Starting quadlets with systemctl

But the most annoying part is actually running the quadlets. And this, I have to highlight, is not purely Podman’s fault; it’s systemd’s way of working.

Let’s assume that you have all your quadlets in place. For AzerothCore, this means something like this:

# podman quadlet list --format '{{.Name}}'
ac-authserver.container
ac-client-data-init.container
ac-client-data.volume
ac-database.container
ac-database.volume
ac-db-import.container
ac-worldserver.container
azerothcore.pod

Now you want to start them. What do you do? Since everything in my setup is linked to a Podman “pod,” I should only have to run systemctl --user start azerothcore-pod.service. And then… nothing. systemctl, while powerful, does not give you any output when starting a service. Instead, it waits for the service to become healthy. On the initial start, Podman has to:

  • download the MySQL image & client data
  • start the MySQL container
  • run the db-import step to execute all migrations
  • start the worldserver
  • start the authserver

This all takes time, and you don’t see any of it when starting azerothcore-pod.service. Instead, you’re supposed to use journalctl --user -u <unit-to-inspect>. But there is another catch: journalctl does not distinguish between unit executions. It’s more like a flat aggregation of logs with no knowledge of when a unit started or stopped. This means that every time a container runs, you have to figure out for yourself where the latest execution’s logs begin.

But as I mentioned, I’m not starting just one service; I’m starting quite a few at the same time. How can I monitor all of them? How can I be sure they’re starting? And how can I debug them when they fail to start? I ended up installing systemd-manager-tui to get at least an overview over all the quadlets. But it wasn’t a silver bullet either. For example, while the database is still starting, all the other services will be marked as loaded, not as starting or waiting for dependencies. I therefore have no feedback about whether (1) a container is waiting for its dependencies, (2) it won’t start because some dependencies are misconfigured (again, they’re loosely coupled by references to each other in the quadlet files, which are easy to get wrong), or (3) there’s some other error.

Compared with docker compose up, I can see which services are starting, along with their logs (if I don’t pass the -d flag). I can even see at a glance which services are healthy. Fetching the logs of a single container is as easy as running docker compose logs <service>.

Debugging nightmare with quadlets

Why all of this matters became very clear when I ran into permission issues with my quadlets. The root cause was that my volume mounts did not have the correct owner set. While there are multiple solutions you could use here, I wanted to be able to edit the files inside and outside the container without having to revert to using sudo nano on my host. For this, there is a neat Podman feature called userns. It allows you to preserve the UID and GID from the host inside the container. So user 1000 inside the container is user 1000 outside the container.

However, no documentation clearly explained whether I needed to set this option on the pod or on each container individually. Initially, AI set the flag for each container, which, as it turned out, was wrong.

But finding this out was far from easy. Normally, when I run into permission issues with Docker containers, I can simply run docker compose exec -it <service> /bin/bash or docker compose run -it --entrypoint=/bin/bash <service> if the service isn’t starting properly. Then I can investigate the situation and usually find a solution to my issues quite easily.

However, Podman does not provide any such niceties for quadlets. Instead, what I had to do was:

  1. Detect that a container didn’t start via a combination of systemctl --user status <unit> and journalctl --user -u <unit>
  2. Investigate logs for this container via journalctl --user -u <unit>. As mentioned, this means manually finding out which logs correspond to the latest run. I actually fell into the trap multiple times of viewing logs I thought were the latest run even though they weren’t.
  3. Find a plan on what to change. Usually, this meant that I had to find some way of figuring out how Podman was starting the container. I found the easiest way to understand this was to use systemd-manager-tui to “edit” the systemd unit specification. What I actually needed was copying the ExecStart instruction of the service, so that I could run it manually - after cleaning up all the systemd-specific syntax like %h resolving to $HOME.

I had to run this loop countless times because I kept running into issues that weren’t straightforward to solve. In the end, I did get it working, and I’m happy to share my quadlet setup in a ZIP file here.

This setup is highly specific to my environment. Most notably, monolith.local, which is mentioned in the files, is the server that runs AzerothCore. You most likely won’t be able to use it as-is for your own Podman setup. However, it should be a very solid starting point. For example, you could ask AI to adapt the setup to your own needs and servers.

Final verdict

I have been running this setup for about two months now, and so far everything is running smoothly. Even though the path was bumpy, the final destination is clean and stable. But would I do it again? Probably not. Now I would probably try Podman kube instead. Sadly, I discovered it too late, and I don’t feel like migrating just for fun yet. But it does look promising: everything is defined in a single file. And with the backlog of documentation available for Kubernetes, I might no longer have to rely on Podman’s documentation alone.

For now, I am content. I’m happy that I took the chance to learn something new, but I’m still frustrated by how much more complicated it was than I expected. In the end, I think the much smaller community behind Podman stood out most. If more people used it, the documentation and tooling would probably be much better. But without good documentation and tooling, the community won’t grow either.

  1. As it turns out, this might not actually be true. Nevertheless, it’s always a good idea to stop your database service last so as not to interrupt any pending writes. ↩︎