View source for Portainer to Docker Compose Migration Guide

= Migrating from Portainer to Docker Compose =

== A Field-Tested Guide with Real Mistakes and How to Fix Them ==

'''Context:''' This tutorial was built from a real migration of a home server (Dell Optiplex 3070 Micros) running multiple services in Portainer, moved to pure Docker Compose under <code>/opt/stacks/</code>. It includes every mistake made along the way and how to recover.

'''Target audience:''' Someone who knows basic Linux and Docker but may forget steps, mix up machines, or make assumptions that get them in trouble.

== The Big Picture ==

=== What we are doing ===
Moving services managed by Portainer (which has its own hidden config layer) into clean, self-contained Docker Compose stacks stored under <code>/opt/stacks/</code>. Each service gets its own folder with a <code>compose.yml</code> and <code>.env</code> file.

=== Why this matters for backup ===
Everything lives under <code>/opt/stacks/</code> — one folder to back up, one folder to restore. No hunting for Portainer's hidden volumes or config files.

=== The 3-2-1 backup goal ===
* '''Copy 1:''' Live data on PC03 (primary server)
* '''Copy 2:''' Warm backup on PC02 (sleeps, wakes via WoL 3x/day and on failure)
* '''Copy 3:''' Synology NAS (cold archive, always on)

== Before You Start: Know Your Environment ==

=== Check which machine you are on ===
<syntaxhighlight lang="bash">
hostname
# Should print something like PC03dell3050
</syntaxhighlight>

'''Common mistake:''' Running commands on PC02 when you think you are on PC03, or vice versa. Always check <code>hostname</code> before doing anything destructive.

=== Know your folder structure ===
<pre>
/opt/stacks/
├── wordpress/
│   ├── compose.yml
│   ├── .env
│   ├── wordpress_data/    ← webroot files
│   └── db_data/           ← database files
├── mediawiki/
├── zulip/
├── openwebui/
└── ...
</pre>

All data lives INSIDE the stack folder as bind mounts. No named Docker volumes. This makes backup and restore straightforward.

=== Know your user situation ===
Most of these operations require root. If you are logged in as a regular user:
<syntaxhighlight lang="bash">
sudo -i
# or
su -
# enter root password when prompted
</syntaxhighlight>

'''If you cannot write to /opt/stacks/,''' you are not root. Do not proceed until you are.

== Step 0: Preparation Checklist ==

Before migrating any service, do these once:

=== Create the shared Docker network ===
All services and NPM (your reverse proxy) share this network:
<syntaxhighlight lang="bash">
docker network create npm_proxy
</syntaxhighlight>
Safe to run even if it already exists — it will just say it already exists.

=== Create the base stacks directory ===
<syntaxhighlight lang="bash">
mkdir -p /opt/stacks
</syntaxhighlight>

== Step 1: Read the Portainer Stack ==

In Portainer, go to '''Stacks → [stack name] → Editor''' and copy the entire compose content.

'''What to look for:'''

{| class="wikitable"
! Field !! What it tells you
|-
| <code>image:</code> || The Docker image and version
|-
| <code>ports:</code> || Host port → container port mapping
|-
| <code>volumes:</code> || Where data lives — see below
|-
| <code>environment:</code> || Credentials and config — '''do not lose these'''
|-
| <code>container_name:</code> || The exact name Docker uses — you will need this
|}

=== Understanding volumes — the most important thing ===

Portainer stacks can use two types of volumes:

'''Type 1 — Named volumes''' (Portainer manages them, hidden location):
<syntaxhighlight lang="yaml">
volumes:
  - mydata:/var/lib/data
volumes:
  mydata:
</syntaxhighlight>
The actual files are at <code>/var/lib/docker/volumes/STACKNAME_mydata/_data/</code>

'''Type 2 — Bind mounts''' (explicit path on the host):
<syntaxhighlight lang="yaml">
volumes:
  - /home/user/myapp/data:/var/lib/data
</syntaxhighlight>
The files are exactly where the path says.

'''How to tell which type you have:''' If the volume line has a full path starting with <code>/</code>, it is a bind mount. If it is just a name with no <code>/</code>, it is a named volume.

== Step 2: Find Where the Data Actually Is ==

=== For named volumes — Portainer renames them ===
'''Critical:''' Portainer adds the stack name as a prefix. If your stack is named <code>wp240927</code> and your volume is named <code>db</code>, the actual Docker volume name is <code>wp240927_db</code>.

Find the real names:
<syntaxhighlight lang="bash">
docker volume ls | grep -i KEYWORD
# Example:
docker volume ls | grep -i wordpress
</syntaxhighlight>

Find the actual path on disk:
<syntaxhighlight lang="bash">
docker volume inspect VOLUMENAME
# Look for "Mountpoint" in the output
# Example output:
# "Mountpoint": "/var/lib/docker/volumes/wp240927_db/_data"
</syntaxhighlight>

=== For bind mounts ===
Just check if the path exists:
<syntaxhighlight lang="bash">
ls /home/user/wp250223-8070/
</syntaxhighlight>

== Step 3: Create the New Stack Files ==

=== Directory structure ===
<syntaxhighlight lang="bash">
mkdir -p /opt/stacks/SERVICENAME/data_folder_name
</syntaxhighlight>

=== Create compose.yml ===
Use <code>cat ></code> to write the file directly in the terminal. This avoids file transfer issues:

<syntaxhighlight lang="bash">
cat > /opt/stacks/SERVICENAME/compose.yml << 'EOF'
services:
  app:
    image: someimage:latest
    restart: unless-stopped
    ports:
      - "PORT:PORT"
    environment:
      SOME_VAR: ${SOME_VAR}
    volumes:
      - ./data:/app/data
    networks:
      - proxy

networks:
  proxy:
    external: true
    name: npm_proxy
EOF
</syntaxhighlight>

'''Key changes from the Portainer version:'''
* Remove <code>version:</code> line (obsolete in modern Compose)
* Change <code>restart: always</code> to <code>restart: unless-stopped</code>
* Move credentials out of <code>environment:</code> into <code>.env</code> using <code>${VAR_NAME}</code> syntax
* Change named volume paths to <code>./foldername</code> (relative bind mounts)
* Add the <code>npm_proxy</code> network so NPM can reach the service

=== Create .env file ===
<syntaxhighlight lang="bash">
cat > /opt/stacks/SERVICENAME/.env << 'EOF'
DB_NAME=mydb
DB_USER=myuser
DB_PASSWORD=change_this_password
EOF
</syntaxhighlight>

'''Critical:''' The values in <code>.env</code> must match what is already inside your database files. If you copy the DB files from the old stack, use the same credentials the old stack had. Changing them will break the connection.

'''How to find the original credentials:''' Before stopping the old stack, go to Portainer → Stack → Editor and read the environment variables.

== Step 4: Copy the Data ==

=== Stop the old Portainer stack first ===
Go to Portainer → Stacks → [stack name] → '''Stop'''.

'''Do NOT delete it yet.''' It is your rollback. Only delete after you confirm the new stack works.

=== For bind mount data (path already known) ===
<syntaxhighlight lang="bash">
rsync -av --progress \
  /old/path/data/ \
  /opt/stacks/SERVICENAME/data/
</syntaxhighlight>

'''The trailing slash on the source matters.''' With a trailing slash, rsync copies the contents. Without it, rsync copies the folder itself (creating an extra nesting level).

=== For named volume data (path from docker volume inspect) ===
<syntaxhighlight lang="bash">
rsync -av --progress \
  /var/lib/docker/volumes/STACKNAME_VOLUMENAME/_data/ \
  /opt/stacks/SERVICENAME/data/
</syntaxhighlight>

=== For data on another machine (SSH) ===
If rsync is available on both machines:
<syntaxhighlight lang="bash">
# Run on the destination machine, pulling from source
rsync -av --progress -e ssh \
  root@SOURCE_IP:/var/lib/docker/volumes/STACKNAME_VOLUMENAME/_data/ \
  /opt/stacks/SERVICENAME/data/
</syntaxhighlight>

If rsync is NOT installed on the source machine:
<syntaxhighlight lang="bash">
# On source machine: create a tar archive
tar -czvf /home/user/service-backup.tar.gz \
  /var/lib/docker/volumes/STACKNAME_vol1/_data \
  /var/lib/docker/volumes/STACKNAME_vol2/_data

# Transfer via SCP from destination
scp root@SOURCE_IP:/home/user/service-backup.tar.gz /opt/stacks/SERVICENAME/
</syntaxhighlight>

Then extract directly into target folders:
<syntaxhighlight lang="bash">
cd /opt/stacks/SERVICENAME

tar -xzvf service-backup.tar.gz \
  --strip-components=6 \
  -C data/subfolder \
  var/lib/docker/volumes/STACKNAME_vol1/_data
</syntaxhighlight>

'''The <code>--strip-components</code> number''' = how many path levels to strip. Count the slashes in <code>var/lib/docker/volumes/STACKNAME_vol1/_data</code> — that is 6 components (var, lib, docker, volumes, STACKNAME_vol1, _data).

'''If you cannot write to /opt/stacks/ via SFTP:''' Upload to <code>/home/user/</code> instead, then move:
<syntaxhighlight lang="bash">
mv /home/user/service-backup.tar.gz /opt/stacks/SERVICENAME/
</syntaxhighlight>

=== Fix file ownership after copying ===
Different services run as different users inside their containers:

{| class="wikitable"
! Service !! uid:gid !! Why
|-
| WordPress (webroot) || 33:33 || www-data
|-
| MariaDB / MySQL || 999:999 || mysql
|-
| PostgreSQL || 70:70 || postgres
|-
| Most Node apps || 1000:1000 || node
|-
| RabbitMQ || 999:999 || rabbitmq
|-
| Redis || 999:999 || redis
|}

<syntaxhighlight lang="bash">
chown -R 33:33 /opt/stacks/wordpress/wordpress_data/
chown -R 999:999 /opt/stacks/wordpress/db_data/
</syntaxhighlight>

== Step 5: Start the New Stack ==

<syntaxhighlight lang="bash">
cd /opt/stacks/SERVICENAME
docker compose up -d
</syntaxhighlight>

Watch the logs:
<syntaxhighlight lang="bash">
docker compose logs -f
</syntaxhighlight>

Press <code>Ctrl+C</code> to stop watching logs. '''This does NOT stop the containers.'''

Test locally before checking the public URL:
<syntaxhighlight lang="bash">
curl -I http://localhost:PORT
</syntaxhighlight>

Expected responses:
* <code>200 OK</code> — service is up and working
* <code>301 Moved Permanently</code> — service is up, redirecting (usually HTTP→HTTPS, this is fine)
* <code>500 Internal Server Error</code> — service is up but something is wrong (usually credentials)
* <code>Connection refused</code> — container is not running or wrong port

== Common Mistakes and How to Fix Them ==

=== Mistake 1: Old container still running, new one cannot start ===

'''Symptom:'''
<pre>
Error response from daemon: Conflict. The container name "/myapp" is already in use
</pre>

'''Cause:''' Stopping a Portainer stack does not always stop the containers. Portainer stopped managing them but left them running.

'''Fix:'''
<syntaxhighlight lang="bash">
docker rm -f CONTAINERNAME
# Then retry:
docker compose up -d
</syntaxhighlight>

=== Mistake 2: Wrong credentials in .env — database refuses connection ===

'''Symptom:''' <code>500 Internal Server Error</code> or "Error establishing database connection"

'''Cause:''' The <code>.env</code> credentials do not match what is stored inside the database files. The database files remember what credentials they were initialized with.

'''Diagnosis — check what the container actually sees:'''
<syntaxhighlight lang="bash">
docker compose exec SERVICENAME env | grep DB_
# or for WordPress specifically:
docker compose exec wordpress env | grep WORDPRESS_DB
</syntaxhighlight>

'''Diagnosis — test the credentials directly:'''
<syntaxhighlight lang="bash">
docker compose exec db mariadb -u USERNAME -p'PASSWORD' DBNAME -e "SELECT 1;"
# If it returns "1", those credentials work.
</syntaxhighlight>

'''Fix:''' Edit <code>.env</code> to match the credentials that actually work, then do a FULL restart:
<syntaxhighlight lang="bash">
docker compose down
docker compose up -d
</syntaxhighlight>

'''Critical:''' <code>docker compose restart</code> does NOT re-read <code>.env</code>. You must use <code>down</code> + <code>up</code>.

=== Mistake 3: Portainer volume names are different from what the compose file says ===

'''Symptom:''' <code>docker volume inspect VOLUMENAME</code> returns <code>no such volume</code>

'''Cause:''' Portainer prefixes volumes with the stack name. If your stack is <code>wp240927</code> and your volume is <code>db</code>, the real name is <code>wp240927_db</code>.

'''Fix:'''
<syntaxhighlight lang="bash">
docker volume ls | grep -i KEYWORD
# Use the full name shown in the output
</syntaxhighlight>

=== Mistake 4: Data lands in the wrong place after extraction ===

'''Symptom:''' <code>ls data/postgresql/</code> shows <code>_data</code> instead of actual database files

'''Cause:''' <code>--strip-components</code> count was wrong, or the tar had an unexpected nesting level.

'''Fix — check what is actually there:'''
<syntaxhighlight lang="bash">
find /opt/stacks/SERVICENAME -name "PG_VERSION" 2>/dev/null
find /opt/stacks/SERVICENAME -name "*.conf" 2>/dev/null | head -5
</syntaxhighlight>

Then move the contents up:
<syntaxhighlight lang="bash">
mv data/postgresql/_data/* data/postgresql/
rmdir data/postgresql/_data
</syntaxhighlight>

=== Mistake 5: .env changes not taking effect ===

'''Symptom:''' You edited <code>.env</code> and restarted, but the container still uses old values.

'''Cause:''' <code>docker compose restart</code> only restarts the process, it does not rebuild the environment.

'''Fix:'''
<syntaxhighlight lang="bash">
docker compose down
docker compose up -d
</syntaxhighlight>

Verify the new values are actually injected:
<syntaxhighlight lang="bash">
docker compose exec SERVICENAME env | grep VARNAME
</syntaxhighlight>

=== Mistake 6: NPM proxy still pointing to old IP/port after migration ===

'''Symptom:''' Service is running locally (<code>curl -I http://localhost:PORT</code> returns 200) but the public domain gives 502 Bad Gateway.

'''Cause:''' NPM's proxy host was pointing to the old machine's IP or old port.

'''Fix:''' In NPM → Proxy Hosts → Edit the domain → update Forward Hostname to the new IP or <code>localhost</code>.

=== Mistake 7: Zulip/service doing HTTP→HTTPS redirect loop through NPM ===

'''Symptom:''' "This page isn't redirecting properly" error in browser.

'''Cause:''' Service internally redirects HTTP to HTTPS, but NPM is sending HTTP. The service sends back a 301 to HTTPS, which loops.

'''Fix:''' In NPM, change the scheme to <code>https</code> and expose the service's HTTPS port instead of the HTTP port. Also disable SSL verification in NPM since services often use self-signed certs.

== Verifying Everything Works ==

After bringing up each service:

<syntaxhighlight lang="bash">
# 1. Check containers are running
docker compose ps

# 2. Check logs for errors
docker compose logs --tail=20

# 3. Test local connectivity
curl -I http://localhost:PORT

# 4. Check the public domain loads
# (Do this in your browser)

# 5. Verify data survived (service-specific)
# For WordPress: log into wp-admin and check posts
# For Wiki: browse to a page you know exists
# For Zulip: log in and check message history
</syntaxhighlight>

== After Confirming a Service Works ==

Wait 24–48 hours of stable operation, then:

# In Portainer: '''Delete''' the old stack
# Clean up old data directories if they were bind mounts:
<syntaxhighlight lang="bash">
rm -rf /home/user/old-stack-data/
</syntaxhighlight>
# Clean up orphaned named volumes:
<syntaxhighlight lang="bash">
docker volume ls | grep OLDSTACKNAME
docker volume rm OLDSTACKNAME_volumename
</syntaxhighlight>

== Backup Strategy (What This All Feeds Into) ==

Once all services are under <code>/opt/stacks/</code>, the backup is simple:

'''PC03 → Synology (3x daily via Restic):'''
<syntaxhighlight lang="bash">
restic -r sftp:synology:/backups/pc03 backup /opt/stacks/
</syntaxhighlight>

'''PC02 warm backup:'''
* PC02 sleeps most of the time
* Synology watchdog pings PC03 every 60s
* If PC03 fails 3 checks → Synology sends WoL magic packet → PC02 wakes
* PC06 relay (iptables DNAT) switches traffic to PC02
* When PC03 comes back → Restic syncs PC02→PC03 → manual confirm → traffic switches back

'''PC00 desktop → Synology (daily):'''
<syntaxhighlight lang="bash">
restic -r sftp:synology:/backups/pc00 backup /home/USERNAME/
</syntaxhighlight>

== Quick Reference: Ports Used ==

{| class="wikitable"
! Service !! Port !! Domain
|-
| WordPress || 8070 || blog.gi7b.org
|-
| MediaWiki || 8191 || wiki.gi7b.org
|-
| Open WebUI || 3000 || —
|-
| SearXNG || 8290 || —
|-
| Zulip || 8010 (HTTP), 8011 (HTTPS) || chat.gi7b.org
|-
| Forgejo || 3001 || —
|-
| Gemini sites || 27081 || —
|-
| Jitsi || TBD || meet.gi7b.org
|-
| Grafana || TBD || —
|-
| Frappe || TBD || —
|}

== Giving This Context to a New AI Chat ==

Paste this block at the start of a new conversation:

<pre>
I am continuing a home server migration project. Here is the context:

Setup:
- PC03 (192.168.196.76) = primary server, Dell Optiplex 3070, Ubuntu, Docker Compose
- PC02 (192.168.196.189) = warm backup, same hardware, sleeps until needed
- PC06 = Contabo Japan VPS, pure iptables DNAT relay (no NGINX), forwards traffic to PC03
- Synology NAS = always on, cold backup archive, runs watchdog container
- ZeroTier connects all machines

All stacks live under /opt/stacks/SERVICENAME/ on PC03
Each stack has: compose.yml, .env, and data folders as bind mounts
Shared Docker network: npm_proxy (external)
Reverse proxy: NGINX Proxy Manager (NPM) running as its own stack on PC03

Completed migrations (Portainer → Docker Compose):
- WordPress (blog.gi7b.org) → port 8070 ✓
- MediaWiki (wiki.gi7b.org) → port 8191 ✓ (check status)
- Open WebUI → port 3000 ✓
- SearXNG → port 8290 ✓
- Zulip (chat.gi7b.org) → port 8010/8011 ✓ (troubleshooting NPM HTTPS)
- Forgejo → port 3001 ✓ (fresh install, no data migration)
- Gemini-sites → port 27081 ✓

Still to do:
- Jitsi (meet.gi7b.org)
- Grafana + Prometheus
- Frappe / ERPNext
- PC06 iptables watchdog script
- Synology WoL watchdog container
- Restic backup setup (PC03→Synology, PC02 agent, PC00→Synology)
- Clean up abandoned Mastodon volumes on PC03
</pre>

[[Category:Tutorials]]
[[Category:Docker]]
[[Category:System Administration]]
[[Category:Self-Hosting]]