spdocker/source/server/upgrade.md

subtitle	header-img	date	description	related
Upgrades	img/header_img/server.jpg	2018-03-07 09:50:12	Upgrading Spectrum Protect Server in a Docker Container	page title index Containerising SP page title firstrun Initial Container Setup page title normalrun Normal Run of Container page title recover Recovering the SP Database

Upgrading Spectrum Protect Server (in a container)

Updated for Spectrum Protect 8.1.7!

NOTES:

• The storage used in the example below is from the host file systems, passed to the container with Docker's -v option.
• Upgrading needs special thought. I have found that upgrading within a major release, EG: 7.1.0 to 7.1.1, 7.1.2 to 7.1.3, etc works transparently. If upgrading that results in a version of DB2 changing (like v7 to v8) additional consideration needs to be made that has not yet been captured in my upgrade option below. (Upgrading from v7 to v8 failed for me because of DB2 issues that I wasn't able to resolve - DB2 changed versions between 7 and 8.) After the upgrade Spectrum Protect is smart enough to realise that it has been upgraded and will do a one time schema update.
• As always, make sure you have a back-out plan, in case things go pair shaped. Docker itself provides a great back-out option, especially if you use it with persistent Docker Container storage that can exploit snapshots and reverting (or directly accessing) a snapshot. In the example below, I'm using Linux LVM snapshots to provide my backup option. Spectrum Scale would also be a good choice.

Configuration

Our configuration will use these defaults

TSM_DB=/srv/sp/database
TSM_USER=/srv/sp/user
TSM_DATA=/srv/sp/data

• TSM_DB is where the TSM Database will be stored on the Docker Host, and will be passed to the container with -v $TSM_DB:/database
• TSM_USER is where the TSM Instance user's home directory is on the Docker Host, and will be passed to the container with -v $TSM_USER:/tsm
• TSM_DATA is where the TSM Disk based storage pools are stored on the Docker Host, and will be passed to the container with -v $TSM_DATA:/data If data is stored in a cloud pool, then this would be where the acceleration pool is holding data before it gets sent to the cloud.

Upgrade Steps

Backup Spectrum Protect's Database

backup db type=FULL devclass=<YOUR BACKUP DEVCLASS> protectk=y pass=<PASSWORD> wait=yes

Protect: SERVER1>backup db t=f devclass=dbb protectk=y pass=password wait=y
ANR0984I Process 143 for Database Backup started in the FOREGROUND at 14:48:02.
ANR4559I Backup DB is in progress.
ANR2280I Full database backup started as process 143.
ANR4626I Database backup will use 1 streams for processing with the number originally requested 1.
ANR4550I Full database backup (process 143) completed. Total bytes backed up: 2,215,641,088.
ANR0985I Process 143 for Database Backup running in the FOREGROUND completed with completion state SUCCESS at 14:52:45.
ANR2394I BACKUP DB: Server device configuration information was written to all device configuration files.

Protect: SERVER1>q volh type=dbb
...
                                 Date/Time: 14-03-2018 14:48:04
                               Volume Type: BACKUPFULL
                             Backup Series: 253
                          Backup Operation: 0
                                Volume Seq: 100,001
                              Device Class: DBB
                               Volume Name: /data/dbb/20999284.DBV

Make sure your backup is successful and make a note of where it is, in case you need it.

[root@host]# ls -aln $TSM_DATA/dbb/|grep 20999284
-rw------- 1 201 201 2215642514 Mar 14 14:51 20999284.dbv

Stop Spectrum Protect

Stop your SP server.

Protect: SERVER1>halt
ANR2234W This command will halt the server; if the command is issued from a remote client, it may not be possible to restart the server from the remote location.

Do you wish to proceed? (Yes (Y)/No (N)) y

ANS8002I Highest return code was 0.

[root@host]#

NOTE: Don't use docker stop to stop your SP server. By default docker sends a SIGTERM signal to the containers process 1 and then waits 10 seconds for it to die. If it hasn't died, it will then send it a SIGKILL. While you can control the timeout between the SIGTERM and SIGKILL signal and you can even control which kill signal is sent, Spectrum Protect does correctly handle SIGTERM and will start to shutdown, so if you do want to use docker stop to shutdown Specturm Protect, make sure you start it with the appropriate --stop-timeout, since 10s is likely not enough time. As a recommendation, I always use SP HALT command to stop an SP server.

If you can, snapshot your persistent storage.

If you can snapshot your persistent storage that you provide to the SP Docker Container, that will save you doing a "restore db" if something goes pair shaped.

[root@host]# df /srv/sp
Filesystem                   1K-blocks     Used Available Use% Mounted on
/dev/mapper/vgroot-lvprotect  20511356 10031500   9431280  52% /srv/sp

[root@host]# lvcreate -s -L5G -n lvspsnapshot vgroot/lvprotect
  Using default stripesize 64.00 KiB.
  Logical volume "lvspsnapshot" created.

NOTE: If you are using IBM Spectrum Scale, you can use mmcrsnapshot

Delete your container

docker rm -v spectrumprotect

Start the container with the new version of SP

You need to do a one-time start of the container with the option upgrade.

docker run --detach=true --hostname=tsm --interactive=false -v $TSM_USER:/tsm -v $TSM_DB:/database --privileged=true --rm=false --tty=false --name=spectrumprotect ibm/spectrumprotect:8.1.7 upgrade

NOTE: You can see the console by running docker logs -f spectrumprotect to make sure the upgrade works as expected. Once you see success, you can delete the container with docker rm -v spectrumprotect.

[root@host]# docker logs -f spectrumprotect
DB2 installation is being initialized.

 Total number of tasks to be performed: 4 
Total estimated time for all tasks to be performed: 309 second(s) 

Task #1 start
Description: Setting default global profile registry variables 
Estimated time 1 second(s) 
Task #1 end 
...
The execution completed successfully.

For more information see the DB2 installation log at "/tmp/db2iupdt.log.5".
DBI1446I  The db2iupdt command is running.


DBI1070I  Program db2iupdt completed successfully.

Normal Run

Spectrum Protect is ready to run on the new version as normal. The first time it runs at a later release, you should see some ANR0276I messages.

docker run --detach=true --hostname=tsm --interactive=false -v $TSM_USER:/tsm -v $TSM_DB:/database -v $TSM_DATA:/data --memory=16g -p 1500:1500 -p 1543:1543 --privileged=true --rm=false --restart=no --tty=false --name=spectrumprotect ibm/spectrumprotect:8.1.7

ANR7800I DSMSERV generated at 00:18:30 on Feb  5 2019.

IBM Spectrum Protect for Linux/x86_64
Version 8, Release 1, Level 7.000

Licensed Materials - Property of IBM

(C) Copyright IBM Corporation 1990, 2017.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.

ANR7801I Subsystem process ID is 7.
...
ANR0275I Detected schema change for table Activity.Log, attempting to
reconcile.
ANR0276I Schema reconciliation for table Activity.Log succeeded.
ANR0275I Detected schema change for table Server.Connect, attempting to
reconcile.
ANR0276I Schema reconciliation for table Server.Connect succeeded.
...
IBM Spectrum Protect:SERVER1> 

I did this on the lab server, and while admittedly it doesn't have much data - the whole "upgrade" process took 12 minutes after the DB backup was complete - and that was with me documenting it for this wiki. So it could have been quicker!

Happy days :)

If something did go pear shaped, then revert your snapshot (or if it is read only, copy the files back to the original path) and then start your container with the original image (the old version) of SP. It should start up as normal as if it was only shut down temporarily. If you really have problems, you have that backup to recover from! Start the container with the restore <PASSWORD> option - you'll need a DBDIRS.TXT file as well in the instance home directory. More info on recovering here.