Monday, September 21, 2009

Monthly backups with Tivoli Storage Manager (TSM)

This is a general overview of how monthly backups (or long time retention) can be accommodated in TSM. I tried all of them on different configurations I did over the years. The only one I find to work easily (backups are happening, and you can do restores) is Additional Node. Even that the administration overhead is present, at least you can restore from it easily, where the other options, restore can be complicated, or full restore not even possible. Backupsets are the cleanest solution. But they were dropped in TSM 6.1.

- Generate Backupsets monthly
- advantages:
No additional backup required from nodes
It can be generated any time of the day/month as it supports generation on data in the past.
In conjunction with Active Storage Pool you don’t read data from tapes when you generate them, you read from disk.
Tapes can be kicked out of library.
You can prolong the expiration of a certain backup if required (legal or other matters).
- disadvantages:
Dropped in 6.1 (no future),IBM will extend support in 6.1 starting with 6.1.2
Data is only on one set of tapes.
You need something else for TDPs. 
You need to monitor which tape expire when.

- Archive for each node when needed
- advantages :
You can kick data out of library,
Easy to setup,
- disadvantages:
Full data transfer from node required each time you run it.
Different include, exclude statements to be maintained.  
You can’t restore system state unless you use Windows backup,
Increase to db size.  

- Additional node in different domain with long retention
- advantages:
It's an incremental every month
You can do longer retention periods, 
You can offload the library with the tapes.
- disadvantages:
administration overhead, additional domains, additional nodes,
Additional failing schedulers,
Doubles the active data on TSM,
Increase data transfer in TSM,
Increase TSM database,
if tapes are offloaded from library, reclamation will begin to fail and will need to be run manually.

The above ones are the “easy ones”. The following are more complex configurations:

- Export node 
- advantages:
tapes can be taken out of the library.
No Retention period, backup can be kept “forever”.
- disadvantages:
If restore required you need to import the node
Export time takes quite a long time, as you export the whole node each month,
You use quite a lot of tapes.
Configuration is complex, and if you take the tapes out, you need a way of managing the tapes on offshelf.

- Additional server for long time retention.
- advantages:
Does not impact the daily database in case of DR,
Don’t need collocation because when you need to restore a monthly backup, performance is not need it,
It's an incremental every month
You can do longer retention periods, 
You can offload the library with the tapes
          - disadvantages:
administration overhead, Additional TSM server, additional domains, additional nodes,
Additional failing schedulers, 

Friday, September 18, 2009

TSM server upgrade procedure on Solaris

I hope this posts will help people using TSM to go by with some of more complex activities. One of them is upgrade of TSM server when you are running on Solaris. 


Software versions used for during this procedure:


Solaris 10
TSM Server Solaris 5.5.1.4 to TSM Server Solaris 5.5.3.0


Depending on how big your TSM server is, you can expect that an upgrade procedure will take between an hour and a couple of hours if you run into problems. 
This procedure will present the steps required to install the TSM license package as an optional step. This is only required if you are upgrading between major versions of TSM (5.3 to 5.4 or 5.5) 


Preparation before upgrade


- Command lines to delete and create all the paths and drives. This is required because we are upgrading the Device Manager (TIVsmSdev package), and this might remap the OS paths to the library drives. It will be handy to have the scripts in hand so you don't spend time creating the commands during the upgrade. If you are doing the upgrade on a library client / library manager configuration, and you are upgrading the client then you only need to delete the paths for the server you are upgrading. If you are upgrading a library manager, then all the paths, and all the drives might need be deleted, even that the path from library clients to drives do not change. The reason to delete them is to remap the drives IDs. 
- If you have drives that are offline before the upgrade, or they are left for "history" purpose, make sure they are still available at OS level. If the drive remap happens at OS level, you might have problems mapping the right TSM drives to the right devices. 
- Also because of the above, all drives should be clear of any process or session, as the drives will not delete if they are in use. 
- If you delete the drives and you are using ACSLS, make note of the DriveId format that is needed to recreate the drives. 

- Command lines to run full TSM database backups. If you have special ways of doing the backups (I know I do) then make sure you have them at hand.

Upgrade steps


1. Login on the TSM server as root.
2. Add to dsmserv.opt :

Disablescheds yes (this stops client and server schedule to be run)
NOMIGRRECL  (will prevent reclamation and migration start during upgrade)

3. Ensure that no TSM processes are running. Stop processes if running.
4. Ensure that no tapes are mounted. Use the DISMOUNT VOLUME volume_name to dismount.Dismount will work only if the tape is in status IDLE.
5. Cancel all remaining sessions. issue
DISAble SESSions CLIent
DISAble SESSions SERVer (optional only if you have other servers connecting)

6. Perform a TSM Database full backup from TSM console:
backup db type=full devclass=tape_or_disk_dev_class
7. Perform a Device configuration backup from TSM console:
backup devconfig filenames=path\devconfig.bkp
8. Perform a Volume history backup from TSM console:
backup volhistory filenames=path\volhistory.bkp
9. Make note of the tape the backup was done to. If TSM does not come back, and you dont know the tape used, check the volhostiry.bkp file, the last entry should be the backup tape.
10. (Optional) If you are doing an OS upgrade as well, then it is a good time to move the following files to a safe (different box) the following files: devconfig.bkp, volhistory.bkp, dsmserv.opt (from /opt/tivoli/tsm/server/bin). A DRM plan would be helpfull as well if you get in the situation where your box does not start any more, and you need to build your TSM server on a new box.
11. Ensure that no TSM processes are running. Stop processes if running. Use cancel process process_number
12. Ensure that no tapes are mounted. Use the DISMOUNT VOLUME to dismount.
13. Halt the TSM server. Issue HALT on the server console. If you use svcadm to start your TSM server issue " svcadm disable dsmserv " . This should stop the TSM server.
14. Once TSM server is down, remove privious version from server:
pkgrm TIVsmS
pkgrm TIVsmSdev
15. (Optional) Remove the license package if you are doing an upgrade between major TSM version.
pkgrm TIVsmlic
16. Extract TSM version 5.5.3.0 from package:
gunzip 5.5.3.0-TIV-TSMSRV-Solaris.tar.Z
tar -xvf 5.5.3.0-TIV-TSMSRV-Solaris.tar
17. (Optional) Extract TSM version 5.5.0.0 for the license package.
18. Install binaries for new version 5.5.3.0
pkgadd TIVsmlic
pkgadd TIVsmSdev
pkgadd TIVsmS
19. Start TSM server in console mode. Go to /opt/tivoli/tsm/server/bin and run ./dsmserv . This will start the console, and you can see all the messages. Pay attention to any error.
20. On the server or on library manager, delete all the paths from the server you are upgrading to the drives it has access to. If you upgrade the library manager, delete all the paths and all the drives.
21. Once all the necessary paths are deleted, login as root on the server you are upgrading and execute: /opt/tivoli/tsm/devices/bin/autoconf (this will recreate the devices and make sure everything is mapped). Make sure that the steps autoconf does finished with return code 0.
22. Run ls -l /dev/rmt/*mt. This script will print you the WWN of the tape drives OS sees. At this point you should know based on WWN, which tape drive is which. Modify your drive and path define scripts from the current server to the library. If you do more than one server at the time, execute the ls command on each server, and create the define path scripts based on each server output. If the define path scripts execute without error, then the server was able to communicate with each drive.
23. Ensure that the server is started. Run command from TSM console:
q path (to see if the paths are online)
q drive f=d (to see if drives are online, and if tapes are mounted)
q mount (to see status of mounted tapes)
q pro (to see TSM processes are running)
24. Remove from dsmserv.opt :
NOMIGRRECL
DISABLESCHEDS
25. HALT TSM and exit console mode.
26. if you use svcadm , issue svcadm enable dsmserv
26. Issue command :
Enable SESSions CLIent
Enable SESSions SERVer






Post upgrade steps:


1. Execute a TSM server backup. It is a good way to see if server to library communication is working. Check if the drive and path remain online. 
2. Start migration, or other processes that use tape drives. Note if the tape are getting mounted in the right drive, and that no errors are present in the library manager or library client regarding mounts. 
3. Copy the installation packages to somewhere safe on the server. You might need them in the future. 
4. Monitor the server for errors and issues. Also monitor administrative and client schedules (check again if you removed the dsmserv options, if you didnt, restart the server). 


Hope the above helps people that want to upgrade the TSM server on Solaris. 
Come again, I will post more procedures like this in the future. 


Regards, 


Liviu Dunaev 











Wednesday, September 16, 2009

Ulpia Software was born

Hi all,

This is it. It is born. It was quite some time in the making, but it is finally here. Why today ? Why not two weeks ago when the domains were purchased? Why not 5 months ago when the three owners shake hands? Why not in a month or so when we will actually have the first beta released?

But too many questions. It is today because I finally have the feeling we are going in the right direction, and there are only details from this point. It is true there is a lot of challenges in out future, but we are far away from nothing.

I hope many years from now to look back at this post and say this was the moment when it all begun. The moment when we can finally say we started our history.

So what is the whole idea behind Ulpia Software?

Ulpia Software is a project in the making for the last year and a half. It all started with identifying a need on the marked around managing enterprise backup server about the same time. Since then we been actively worked towards the software that is called Backup Live Console (BLC). The first version of Backup Live Console is still under development. We are far from over, but the software is beginning to look very promising, and I hope by the end of this year we will have it available.

But more about BLC at a later time, at the moment a lot of developing is happening, and a lot of testing as well. Having something running on 4 operating systems and 3+ browsers is a major endeavor, even that you use open software and you pay attention to standards. But what is a rule (standard) if it is not broken?

It is probably important to say that this project had some team changes along the way, but at the moment I have the feeling we are in the best equation possible. The team has vast knowledge in different areas like IBM Tivoli Storage Manager, IBM Lotus Notes and IBM Lotus Domino, also huge knowledge in software development on enterprise environments and software. In time I will present more of the members of the current team. But for those that were with us, but left the project along the way for different reasons, I would like to thank them for the effort and knowledge provided. Backup Live Console would not have been where it is today without this effort.

Regarding this blog, it will be used as an open door into our company. Our idea is to have a close relationship with our users, and give back as much as possible to the community. I believe the only way of succeeding in this project will be with the help of the vast backup administrators community that is out there.

So this is it, the time has come to present this to the world, good luck to us all.

Regards,

Liviu Dunaev