> ## Documentation Index
> Fetch the complete documentation index at: https://docs.2501.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade

> Move an installation to a newer build of the chart, and back

An upgrade applies the files of a newer build, by the method you installed with, and keeps what is yours: your `values.yaml` or your `.env`, and your Secrets. Every build has new files, since they name its images by digest: there is no upgrade of the images alone, and the engine, Command Center, the executor images the engine starts and the permissions they need always move together. The commands use your namespace in `$NS` on Kubernetes, and your stack name in `$STACK` on Docker Swarm (`ai-2501`, or the CLI's for an installation moved from it).

Each Release says until when its version is supported, 3 months after its release. Fixes go into the newest version only: upgrade before that date.

## Before you start

1. **Read the Upgrade actions.** The [release notes](https://www.2501.ai/changelog) of each release list, under Upgrade actions, what changed in what you apply, such as a permission, a new required secret or value, or a change that interrupts running tasks. Read them for every release you cross, not only the last one.
2. **Back up the database**, schema and data. Restoring this backup is what makes going back possible. The commands take the role and database from `$U` and `$DB`: `2501` and `2501`, the CLI's, which the chart keeps on Kubernetes; on Docker Swarm, your `database.user` and `database.name` (`2501` by default, `postgres` and `2501` for an installation [moved from the CLI](https://docs.2501.ai/0.19/deployment/migrate-from-cli.md)).

   ```bash theme={null}
   U=2501 DB=2501   # on Docker Swarm, your database.user and database.name
   # The PostgreSQL 2501 provides, on Kubernetes
   kubectl -n "$NS" exec deploy/postgres -- pg_dump -U "$U" -d "$DB" --format=custom > 2501-preupgrade.dump
   # The PostgreSQL 2501 provides, on Docker Swarm, on the node labelled 2501-data
   docker exec "$(docker ps -q -f "label=com.docker.swarm.service.name=${STACK}_postgres")" pg_dump -U "$U" -d "$DB" --format=custom > 2501-preupgrade.dump
   # Your own database
   pg_dump "$DIRECT_URL" --format=custom --file=2501-preupgrade.dump
   ```

   Keep automated backups for disaster recovery too, not only one dump before an upgrade.
3. **Download and verify the new build's files** from its Release on [github.com/2501-ai/install](https://github.com/2501-ai/install), as for the install: see [Verify the installation files](https://docs.2501.ai/0.19/deployment/verify-downloads.md#verify-the-installation-files). On Kubernetes and OpenShift, `https://github.com/2501-ai/install/releases/latest/download/2501.tgz` is always the newest build's chart; `helm show chart 2501.tgz` prints its build, and the commands below take it in place of `2501-<build>.tgz`. If you copy the images into your registry, copy the new ones first.

## Review the change

Look at everything the new build changes before you apply it, permissions included:

| Method | Review with |
| - | - |
| Helm | `helm diff upgrade ai2501 ./2501-<build>.tgz -n "$NS" -f values.yaml` (the helm-diff plugin) |
| ArgoCD | The application's diff, after committing the new chart and before syncing |
| kubectl or oc | `kubectl -n "$NS" diff -f manifests.yaml` on the new build's render, `helm template ai2501 ./2501-<build>.tgz -n "$NS" -f values.yaml > manifests.yaml`, with the same `values.yaml`, and with the same `--prune`, `-l` and `--prune-allowlist` flags as the apply, so it also shows what the apply deletes: see [Apply](https://docs.2501.ai/0.19/deployment/kubernetes.md#apply) |
| Docker Swarm | `diff` between your current `compose.yaml` and the new one, and between `env.example` and the new one: a new variable goes in your `.env`, a new secret appears in the first lines of `compose.yaml` |

## Apply

With the new build's files, run the same commands as for the install:

* **kubectl or oc**: render the new `2501-<build>.tgz` with the same `values.yaml`, then the same `kubectl apply --prune` of the result, as in [Apply](https://docs.2501.ai/0.19/deployment/kubernetes.md#apply). If the Upgrade actions name a new value, add it to your `values.yaml` first: the render refuses one without it. Your Secrets stay as they are, and the diff shows only what the build changes.
* **Helm**: `helm upgrade --install` with the new `2501-<build>.tgz` and the same `values.yaml`, as in [With Helm or ArgoCD](https://docs.2501.ai/0.19/deployment/kubernetes.md#with-helm-or-argocd).
* **ArgoCD**: replace the unpacked chart in your Git repository with the new one (`rm -rf <path>/2501 && tar -xzf 2501-<build>.tgz -C <path>`), commit, and sync.
* **Docker Swarm**: add any new variable to your `.env` and create any new secret, then the same `docker stack deploy --with-registry-auth --prune` with the new `compose.yaml`.

The engine stops before the new one starts, so tasks in flight get up to 110 seconds to finish and new tasks wait until it is back. Command Center keeps serving until its new version is ready. Then check the installation as after an install: [Kubernetes](https://docs.2501.ai/0.19/deployment/kubernetes.md#check) or [Docker Swarm](https://docs.2501.ai/0.19/deployment/docker-swarm.md#check).

## Go back

If the new build misbehaves, go back to the last one that worked, with the data that went with it:

1. Stop the engine, Command Center and the bootstrap of the new build (with ArgoCD, turn off automated sync first, or it starts them again), which retries its migrations, and wait until their pods or containers are gone, so nothing writes to the database while it is restored. `scale` returns at once, and the engine gives running tasks up to 110 seconds. The wait fails after five minutes; do not go on until it succeeds.

   ```bash theme={null}
   kubectl -n "$NS" scale deploy/engine deploy/command-center --replicas=0
   kubectl -n "$NS" delete job -l app.kubernetes.io/instance=ai2501,app.kubernetes.io/component=bootstrap
   kubectl -n "$NS" wait --for=delete pod -l 'app.kubernetes.io/instance=ai2501,app.kubernetes.io/component in (engine,command-center,bootstrap)' --timeout=300s
   ```

   On Docker Swarm, the next deploy creates the bootstrap again. Run `gone` on every node:

   ```bash theme={null}
   docker service scale "${STACK}_engine=0" "${STACK}_command-center=0"
   docker service rm "${STACK}_bootstrap"
   gone() {
     for i in $(seq 60); do
       c=$(for s in engine command-center bootstrap; do docker ps -q -f "label=com.docker.swarm.service.name=${STACK}_$s" || exit 1; done) || return 1
       [ -z "$c" ] && return 0
       sleep 5
     done
     echo "engine, Command Center or bootstrap still running after 5 minutes" >&2; return 1
   }
   gone
   ```

2. Restore the backup into an empty database. The names are quoted: `2501` is not a valid name without quotes. `template0` keeps the new database from inheriting `template1`'s collation version, which a newer PostgreSQL image refuses to copy. For the PostgreSQL 2501 provides on Kubernetes:

   ```bash theme={null}
   kubectl -n "$NS" exec deploy/postgres -- psql -U "$U" -d postgres -c "DROP DATABASE \"$DB\" WITH (FORCE)" -c "CREATE DATABASE \"$DB\" OWNER \"$U\" TEMPLATE template0"
   kubectl -n "$NS" exec -i deploy/postgres -- pg_restore -U "$U" -d "$DB" --exit-on-error < 2501-preupgrade.dump
   ```

   On Docker Swarm, on the node labelled 2501-data:

   ```bash theme={null}
   pg=$(docker ps -q -f "label=com.docker.swarm.service.name=${STACK}_postgres")
   docker exec "$pg" psql -U "$U" -d postgres -c "DROP DATABASE \"$DB\" WITH (FORCE)" -c "CREATE DATABASE \"$DB\" OWNER \"$U\" TEMPLATE template0"
   docker exec -i "$pg" pg_restore -U "$U" -d "$DB" --exit-on-error < 2501-preupgrade.dump
   ```

   For your own database, recreate it empty and run `pg_restore -d "$DIRECT_URL" --exit-on-error 2501-preupgrade.dump`.

3. Apply the previous build by the method you use: its chart, from its Release, with the `values.yaml` you ran it with (from your Git history), or its Swarm files. A Release never changes, so they are still there. With ArgoCD, revert the commit that brought in the new chart.

4. If the two builds run different PostgreSQL images, PostgreSQL warns of a "collation version mismatch": rebuild the indexes, as in step 4 of [Move from the CLI](https://docs.2501.ai/0.19/deployment/migrate-from-cli.md).

5. Check the installation before handing it back to users, and tell 2501 what went wrong, with the details you still have.

A restore rewinds the data to the moment of the backup. Always restore it when you go back: an upgrade can change the database in a way the previous build cannot read, and the previous engine then fails every task. On Docker Swarm, an engine that fails to start after an update is rolled back to the previous image on its own, which puts the previous engine on the upgraded database: restore the backup then too.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.