Upgrade

An upgrade applies the files of a newer build, by the method you installed with, and keeps what is yours: your values.yaml or your .env, and your Secrets. Every build has new files, since they name its images by digest: there is no upgrade of the images alone, and the engine, Command Center, the executor images the engine starts and the permissions they need always move together. The commands use your namespace in $NS on Kubernetes, and your stack name in $STACK on Docker Swarm (ai-2501, or the CLI’s for an installation moved from it). Each Release says until when its version is supported, 3 months after its release. Fixes go into the newest version only: upgrade before that date.

​

Before you start

1. Read the Upgrade actions. The release notes of each release list, under Upgrade actions, what changed in what you apply, such as a permission, a new required secret or value, or a change that interrupts running tasks. Read them for every release you cross, not only the last one.
2. Back up the database, schema and data. Restoring this backup is what makes going back possible. The commands take the role and database from $U and $DB: 2501 and 2501, the CLI’s, which the chart keeps on Kubernetes; on Docker Swarm, your database.user and database.name (2501 by default, postgres and 2501 for an installation moved from the CLI).
   U=2501 DB=2501   # on Docker Swarm, your database.user and database.name
   # The PostgreSQL 2501 provides, on Kubernetes
   kubectl -n "$NS" exec deploy/postgres -- pg_dump -U "$U" -d "$DB" --format=custom > 2501-preupgrade.dump
   # The PostgreSQL 2501 provides, on Docker Swarm, on the node labelled 2501-data
   docker exec "$(docker ps -q -f "label=com.docker.swarm.service.name=${STACK}_postgres")" pg_dump -U "$U" -d "$DB" --format=custom > 2501-preupgrade.dump
   # Your own database
   pg_dump "$DIRECT_URL" --format=custom --file=2501-preupgrade.dump
   
   Keep automated backups for disaster recovery too, not only one dump before an upgrade.
3. Download and verify the new build’s files from its Release on github.com/2501-ai/install, as for the install: see Verify the installation files. On Kubernetes and OpenShift, https://github.com/2501-ai/install/releases/latest/download/2501.tgz is always the newest build’s chart; helm show chart 2501.tgz prints its build, and the commands below take it in place of 2501-<build>.tgz. If you copy the images into your registry, copy the new ones first.

​

Review the change

Look at everything the new build changes before you apply it, permissions included:

​

Apply

With the new build’s files, run the same commands as for the install:

• kubectl or oc: render the new 2501-<build>.tgz with the same values.yaml, then the same kubectl apply --prune of the result, as in Apply. If the Upgrade actions name a new value, add it to your values.yaml first: the render refuses one without it. Your Secrets stay as they are, and the diff shows only what the build changes.
• Helm: helm upgrade --install with the new 2501-<build>.tgz and the same values.yaml, as in With Helm or ArgoCD.
• ArgoCD: replace the unpacked chart in your Git repository with the new one (rm -rf <path>/2501 && tar -xzf 2501-<build>.tgz -C <path>), commit, and sync.
• Docker Swarm: add any new variable to your .env and create any new secret, then the same docker stack deploy --with-registry-auth --prune with the new compose.yaml.

The engine stops before the new one starts, so tasks in flight get up to 110 seconds to finish and new tasks wait until it is back. Command Center keeps serving until its new version is ready. Then check the installation as after an install: Kubernetes or Docker Swarm.

​

Go back

If the new build misbehaves, go back to the last one that worked, with the data that went with it:

1. Stop the engine, Command Center and the bootstrap of the new build (with ArgoCD, turn off automated sync first, or it starts them again), which retries its migrations, and wait until their pods or containers are gone, so nothing writes to the database while it is restored. scale returns at once, and the engine gives running tasks up to 110 seconds. The wait fails after five minutes; do not go on until it succeeds.
   kubectl -n "$NS" scale deploy/engine deploy/command-center --replicas=0
   kubectl -n "$NS" delete job -l app.kubernetes.io/instance=ai2501,app.kubernetes.io/component=bootstrap
   kubectl -n "$NS" wait --for=delete pod -l 'app.kubernetes.io/instance=ai2501,app.kubernetes.io/component in (engine,command-center,bootstrap)' --timeout=300s
   
   On Docker Swarm, the next deploy creates the bootstrap again. Run gone on every node:
   docker service scale "${STACK}_engine=0" "${STACK}_command-center=0"
   docker service rm "${STACK}_bootstrap"
   gone() {
     for i in $(seq 60); do
       c=$(for s in engine command-center bootstrap; do docker ps -q -f "label=com.docker.swarm.service.name=${STACK}_$s" || exit 1; done) || return 1
       [ -z "$c" ] && return 0
       sleep 5
     done
     echo "engine, Command Center or bootstrap still running after 5 minutes" >&2; return 1
   }
   gone
   
2. Restore the backup into an empty database. The names are quoted: 2501 is not a valid name without quotes. template0 keeps the new database from inheriting template1’s collation version, which a newer PostgreSQL image refuses to copy. For the PostgreSQL 2501 provides on Kubernetes:
   kubectl -n "$NS" exec deploy/postgres -- psql -U "$U" -d postgres -c "DROP DATABASE \"$DB\" WITH (FORCE)" -c "CREATE DATABASE \"$DB\" OWNER \"$U\" TEMPLATE template0"
   kubectl -n "$NS" exec -i deploy/postgres -- pg_restore -U "$U" -d "$DB" --exit-on-error < 2501-preupgrade.dump
   
   On Docker Swarm, on the node labelled 2501-data:
   pg=$(docker ps -q -f "label=com.docker.swarm.service.name=${STACK}_postgres")
   docker exec "$pg" psql -U "$U" -d postgres -c "DROP DATABASE \"$DB\" WITH (FORCE)" -c "CREATE DATABASE \"$DB\" OWNER \"$U\" TEMPLATE template0"
   docker exec -i "$pg" pg_restore -U "$U" -d "$DB" --exit-on-error < 2501-preupgrade.dump
   
   For your own database, recreate it empty and run pg_restore -d "$DIRECT_URL" --exit-on-error 2501-preupgrade.dump.
3. Apply the previous build by the method you use: its chart, from its Release, with the values.yaml you ran it with (from your Git history), or its Swarm files. A Release never changes, so they are still there. With ArgoCD, revert the commit that brought in the new chart.
4. If the two builds run different PostgreSQL images, PostgreSQL warns of a “collation version mismatch”: rebuild the indexes, as in step 4 of Move from the CLI.
5. Check the installation before handing it back to users, and tell 2501 what went wrong, with the details you still have.

A restore rewinds the data to the moment of the backup. Always restore it when you go back: an upgrade can change the database in a way the previous build cannot read, and the previous engine then fails every task. On Docker Swarm, an engine that fails to start after an update is rolled back to the previous image on its own, which puts the previous engine on the upgraded database: restore the backup then too.