You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This updates the current upgrade documentation to reflect the
changes made to the Operator upgrade procedure and removes
the instructions related to the previous upgrade procedures
that are no longer valid.
Various Operator releases will require action by the Operator administrator of your organization in order to upgrade to the next release of the Operator. Some upgrade steps are automated within the Operator but not all are possible at this time.
8
+
# Upgrading the Crunchy PostgreSQL Operator
10
9
11
-
This section of the documentation shows specific steps required to upgrade different versions of the Postgres Operator depending on your current environment.
10
+
Below are two methods for upgrading your existing deployment of the PostgreSQL Operator.
12
11
13
-
[Upgrade Postgres Operator to 3.5] ( {{< relref "upgrade/upgradeto35.md" >}})
12
+
If you are upgrading from PostgreSQL Operator 4.1.0 or later, you can use the [Automated Upgrade Procedure](/upgrade#automated-upgrade-procedure).
14
13
15
-
[Postgres Operator 3.5 Minor Version Upgrade] ( {{< relref "upgrade/upgrade35.md" >}})
14
+
For versions before 4.1.0, please see the appropriate [manual procedure](/upgrade#manually-upgrading-the-operator-and-postgresql-clusters).
16
15
17
-
[Upgrade Postgres Operator from 3.5 to 4.1] ( {{< relref "upgrade/upgrade35to4.md" >}})
16
+
## Automated Upgrade Procedure
18
17
19
-
[Upgrade Postgres Operator from 4.X to 4.3.0 (Bash)] ( {{< relref "upgrade/upgrade4xto42_bash.md" >}})
18
+
The automated upgrade to a new release of the PostgreSQL Operator comprises two main steps:
20
19
21
-
[Upgrade Postgres Operator from 4.X to 4.3.0 (Ansible)] ( {{< relref "upgrade/upgrade4xto42_ansible.md" >}})
20
+
* Upgrading the PostgreSQL Operator itself
21
+
* Upgrading the existing PostgreSQL Clusters to the new release
22
22
23
-
[Upgrade Postgres Operator from 4.1.0 to a patch release] ( {{< relref "upgrade/upgrade41.md" >}})
23
+
The first step will result in an upgraded PostgreSQL Operator that is able to create and manage new clusters as expected, but will be unable to manage existing clusters until they have been upgraded. The second step upgrades the clusters to the current Operator version, allowing them to once again be fully managed by the Operator.
24
24
25
-
## Upgrading A Postgres Cluster
25
+
The automated upgrade procedure is designed to facilate the quickest and most efficient method to the current release of the PostgreSQL Operator. However, as with any upgrade, there are several considerations before beginning.
26
26
27
-
Using the operator, it is possible to upgrade a postgres cluster in place. When a pgo upgrade command is issued, and a --CCPImageTag is specified, the operator will upgrade each replica and the primary to the new CCPImageTag version. It is important to note that the postgres version of the new container should be compatible with the current running version. There is currently no version check done to ensure compatibility.
27
+
### Considerations
28
28
29
-
The upgrade is accomplished by updating the CCPImageTag version in the deployment, which causes the old pod to be terminated and a new pod created with the updated deployment specification.
29
+
1. Cluster Downtime - The re-creation of clusters will take some time, generally on the order of minutes but potentially longer depending on the operating environment. As such, the timing of the upgrade will be an important consideration. It should be noted that the upgrade of the PostgreSQL Operator itself will leave any existing cluster resources in place until individual pgcluster upgrades are performed.
30
30
31
-
When the upgrade starts, and if _autofail_ is enabled for the cluster, each replica is upgraded seqentially, waiting for the previous replica to go ready before updating the next. After the replicas complete, the primary is then upgraded to the new image. Please note that the upgrade process respects the _autofail_ setting as currently definied for the cluster being upgraded. Therefore, if autofail is enabled when the primary deployment is updated, the cluster behaves as though the primary had failed and begins the failover process. See _Automatic Failover_ in the _Overview_ section for more details about the PostgreSQL Operator failover process and expected behavior.
31
+
2. Destruction and Re-creation of Certain Resources - As this upgrade process does destroy and recreate most elements of the cluster, unhealthy Kubernetes or Openshift environments may have difficulty recreating the necessary elements. Node availability, necessary PVC storage allocations and processing requirements are a few of the resource considerations to make before proceeding.
32
32
33
-
When the cluster is not in _autofail_ mode (i.e. autofail is disabled), the primary and all replicas are updated at the same time, after which they will remain in an "unready" status. This is because when autofail is disabled, no attempt will be made to start the PostgreSQL databases contained within the primary or replica pods once the containers have been started following the update. It is therefore necessary to re-enable autofail following a minor upgrade during which autofail was disabled in order to fully bring the cluster back online.
33
+
3. Compatibility with Custom Configurations - Given the nearly endless potential for custom configuration settings, it is important to consider any resource or implemenation that might be uniquely tied to the current PostgreSQL Operator version.
34
34
35
-
At this time, the backrest-repo container is not upgraded during this upgrade as it is part of the postgres operator release and is updated with the operator.
35
+
4. Versions Supported - This upgrade currently supports cluster upgrades from PostgreSQL Operator version 4.1.0 and later.
36
36
37
-
## Minor Upgrade Example
37
+
5. PostgreSQL Major Version Requirements - The underlying PostgreSQL major version must match between the old and new clusters. For example, if you are upgrading a 4.1.0 version of the PostgreSQL Operator and the cluster is using PostgreSQL 11.5, your upgraded clusters will need to use container images with a later minor version of PostgreSQL 11. Note that this is not a requirement for new clusters, which may use any currently supported version. For more information, please see the [Compatibility Requirements]({{< relref "configuration/compatibility.md" >}}).
38
38
39
-
In this example, we are upgrading a cluster from PostgreSQL 11.6 to 11.7 using the `crunchy-postgres:centos7-11.7-4.3.0` container:
39
+
6. Storage Requirements - An essential part of both the automated and manual upgrade procedures is the reuse of existing PVCs. As such, it is essential that the existing storage settings are maintained for any upgraded clusters.
7. As opposed to the manual upgrade procedures, the automated upgrade is designed to leave existing resources (such as CRDs, config maps, secrets, etc) in place whenever possible to minimize the need for resource re-creation.
42
42
43
-
For more information, please see the `pgo upgrade` documentation [here.] ( {{< relref "pgo-client/reference/pgo_upgrade.md" >}})
43
+
##### NOTE: As with any upgrade procedure, it is strongly recommended that a full logical backup is taken before any upgrade procedure is started. Please see the [Logical Backups](/pgo-client/common-tasks#logical-backups-pg_dump--pg_dumpall) section of the Common Tasks page for more information.
44
+
45
+
### Automated Upgrade when using an Ansible installation of the PostgreSQL Operator
46
+
47
+
For existing PostgreSQL Operator deployments that were installed using Ansible, the upgrade process is straightforward.
48
+
49
+
First, you will copy your existing inventory file as a backup for your existing settings. You will reference these settings, but you will need to use the updated version of the inventory file for the current version of PostgreSQL Operator.
50
+
51
+
Once you've checked out the appropriate release tag, please follow the [Update Instructions]({{< relref "installation/install-with-ansible/updating-operator.md" >}}), being sure to update the new inventory file with your required settings. Please keep the above [Considerations](/upgrade#considerations) in mind, particularly with regard to the version and storage requirements listed.
52
+
53
+
Once the update is complete, you should now see the PostgreSQL Operator pods are up and ready. It is strongly recommended that you create a test cluster to validate proper functionality before moving on to the [Automated Cluster Upgrade](/upgrade#postgres-operator-automated-cluster-upgrade) section below.
54
+
55
+
### Automated Upgrade when using a Bash installation of the PostgreSQL Operator
56
+
57
+
Like the Ansible procedure given above, the Bash upgrade procedure for upgrading the PostgreSQL Operator will require some manual configuration steps before the upgrade can take place. These updates will be made to your user's environment variables and the pgo.yaml configuration file.
58
+
59
+
#### PostgreSQL Operator Configuration Updates
60
+
61
+
To begin, you will need to make the following updates to your existing configuration.
62
+
63
+
##### Bashrc File Updates
64
+
65
+
First, you will make the following updates to your $HOME/.bashrc file.
66
+
67
+
When upgrading from version 4.1.X, in `$HOME/.bashrc`
68
+
69
+
Add the following variables:
70
+
71
+
```
72
+
export TLS_CA_TRUST=""
73
+
export ADD_OS_TRUSTSTORE=false
74
+
export NOAUTH_ROUTES=""
75
+
76
+
# Disable default inclusion of OS trust in PGO clients
Next, you will and save a copy of your existing pgo.yaml file (`$PGOROOT/conf/postgres-operator/pgo.yaml`) as pgo_old.yaml or similar.
92
+
93
+
Once this is saved, you will checkout the current release of the PostgreSQL Operator and update the pgo.yaml for the current version, making sure to make updates to the CCPImageTag and storage settings in line with the [Considerations](/upgrade#considerations) given above.
94
+
95
+
#### Upgrading the Operator
96
+
97
+
Once the above configuration updates are completed, the PostgreSQL Operator can be upgraded.
98
+
To help ensure that needed resources are not inadvertently deleted during an upgrade of the PostgreSQL Operator, a helper script is provided. This script provides a similar function to the Ansible installation method's 'update' tag, where the Operator is undeployed, and the designated namespaces, RBAC rules, pods, etc are redeployed or recreated as appropriate, but required CRDs and other resources are left in place.
99
+
100
+
To use the script, execute:
101
+
```
102
+
$PGOROOT/deploy/upgrade-pgo.sh
103
+
```
104
+
This script will undeploy the current PostgreSQL Operator, configure the desired namespaces, install the RBAC configuration, deploy the new Operator, and, attempt to install a new PGO client, assuming default location settings are being used.
105
+
106
+
After this script completes, it is strongly recommended that you create a test cluster to validate the Operator is functioning as expected before moving on to the individual cluster upgrades.
107
+
108
+
## PostgreSQL Operator Automated Cluster Upgrade
109
+
110
+
Previously, the existing cluster upgrade focused on updating a cluster's underlying container images. However, due to the various changes in the PostgreSQL Operator's operation between the various versions (including numerous updates to the relevant CRDs, integration of Patroni for HA and other significant changes), updates between PostgreSQL Operator releases required the manual deletion of the existing clusters while preserving the underlying PVC storage. After installing the new PostgreSQL Operator version, the clusters could be recreated manually with the name of the new cluster matching the existing PVC's name.
111
+
112
+
The automated upgrade process provides a mechanism where, instead of being deleted, the existing PostgreSQL clusters will be left in place during the PostgreSQL Operator upgrade. While normal Operator functionality will be restricted on these existing clusters until they are upgraded to the currently installed PostgreSQL Operator version, the pods, services, etc will still be in place and accessible via other methods (e.g. kubectl, service IP, etc).
113
+
114
+
To upgrade a particular cluster, use
115
+
```
116
+
pgo upgrade mycluster
117
+
```
118
+
This will follow a similar process to the documented manual process, where the pods, deployments, replicasets, pgtasks and jobs are deleted, the cluster's replicas are scaled down and replica PVCs deleted, but the primary PVC and backrest-repo PVC are left in place. Existing services for the primary, replica and backrest-shared-repo are also kept and will be updated to the requirements of the current version. Configmaps and secrets are kept except where deletion is required. For a cluster 'mycluster', the following configmaps will be deleted (if they exist) and recreated:
119
+
```
120
+
mycluster-leader
121
+
mycluster-config
122
+
mycluster-pgha-config
123
+
```
124
+
along with the following secret:
125
+
```
126
+
mycluster-backrest-repo-config
127
+
```
128
+
129
+
The pgcluster CRD will be read, updated automatically and replaced, at which point the normal cluster creation process will take over. The end result of the upgrade should be an identical numer of pods, deployments, replicas, etc with a new pgbackrest backup taken, but existing backups left in place.
130
+
131
+
Finally, to disable PostgreSQL version checking during the upgrade, such as for when container images are re-tagged and no longer follow the standard version tagging format, use the "disable-version-check" flag:
132
+
133
+
```
134
+
pgo upgrade mycluster --disable-version-check
135
+
```
136
+
137
+
That will allow the upgrade to proceed, regardless of the tag values. Please note, the underlying image must still be chosen in accordance with the [considerations](/upgrade#considerations) listed above.
138
+
139
+
140
+
## Manually Upgrading the Operator and PostgreSQL Clusters
141
+
142
+
In the event that the automated upgrade cannot be used, below are manual upgrade procedures for both PostgreSQL Operator 3.5 and 4 releases. These procedures will require action by the Operator administrator of your organization in order to upgrade to the current release of the Operator. Some upgrade steps are still automated within the Operator, but not all are possible with this upgrade method. As such, the pages below show the specific steps required to upgrade different versions of the PostgreSQL Operator depending on your current environment.
143
+
144
+
In the event that you are performing a manual upgrade, it is recommended to upgrade to the latest PostgreSQL Operator available.
0 commit comments