Re: Daily TSM maintenance schedules
Roger, If you find a solution to that last problem, i.e. only 24 hours in a day, please post it to the list. :-) I am trying to understand the schedule you present. Are you saying you run expiration at the same time your backups, backup stgpools, and migration are running? Or do you run expiration as soon as each of those finishes? If it is the former, that certainly isn't ideal. It would seem to me that expiration would severely impact the performance of those, especially backup stgpools. Have you recently timed running an 'expire inv' during a time that nothing else is running, to see how long it takes when it isn't contending with anything else? Do you have an enormous database? Or are the disks behind the database extremely slow? Is bufpoolsize set too low? Those are the main things that affect 'expire inv' speed. Expire inventory isn't something that should take over your system. If everything is tuned right it should only take a couple hours. If your database is an out-of-control size, then maybe it is time to consider splitting into two TSM servers. You can reply back to me alone if you want to discuss further, but not trouble the whole listserv with the particulars. Best Regards, John D. Schneider The Computer Coaching Community, LLC john.schnei...@computercoachingcommunity.com Office: (314) 635-5424 / Toll Free: (866) 796-9226 Cell: (314) 750-8721 Original Message Subject: Re: [ADSM-L] Daily TSM maintenance schedules From: Roger Deschner rog...@uic.edu Date: Sat, August 29, 2009 11:21 am To: ADSM-L@VM.MARIST.EDU We're big. (2tb/day backup data Mon-Fri nights) Expiration is the 20-ton elephant in the room. If it doesn't run to completion every once in a while, we're in trouble and we have the dreaded database bloat. It has happened, and it wasn't pretty. You can call this schedule crazy, but it's what works for a big TSM system: CLIENT BACKUP + EXPIRATION + RECLAMATION (5PM) BACKUP STGPOOLS + EXPIRATION MIGRATION + EXPIRATION BACKUP DEVCONFIG + BACKUP VOLHIST *** BACKUP DB (incr on weekdays, full on Sun) + RECLAMATION DELETE VOLHIST (triggered by end of BACKUP DB) + RECLAMATION RECLAMATION + EXPIRATION (1-5PM - a relatively slow time. This is my maintenance window.) back to top *** These are fast. Runs while the tape lib is dismounting the migration tapes, and mounting the DB BACKUP tape. Saturday morning only: DELETE FILESPACE (we queue them up all week) You've got to work DELETE FILESPACE into your schedule, because it involves very heavy database I/O. You can't EXPIRE INVENTORY or BACKUP DB during DELETE FILESPACE. MIGRATION won't even start if DELETE FILESPACE is running. We do allow users to do it themselves, but in actual practice they never do. We've got to nag them about ancient, abandoned filespaces, and then we do it for them Sat AM. This is also when we remove old nodes. We use a tape reuse delay of 2 days, as disaster protection for doing things in the wrong order. The idea here is to keep the CPU and the 15,000rpm database disks busy 24/7, and to use as many tape drives as possible for as much of the time as possible. I am constantly tuning this schedule. The basic problem is that there are only 24 hours in a day. Roger Deschner University of Illinois at Chicago rog...@uic.edu Academic Computing Communications Center ==I have not lost my mind -- it is backed up on tape somewhere.= On Fri, 28 Aug 2009, Howard Coles wrote: Correction, it's marked for expiration and it is still recoverable, until the Expire process runs and removes it from the Database. I know this from experience, as we disable our expiration process for a few days due to a server failure, and once due to a legal request. The Expire Process actually removes the DB entry for that version of the file. See Ya' Howard -Original Message- From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of Wanda Prather Sent: Friday, August 28, 2009 11:21 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Daily TSM maintenance schedules Agreed. expire inventory is actually something of a misnomer. If you have your retention set to 14 versions, and someone takes the 15th backup, the oldest version expires right then, you can't get it back. That type of expiration takes place, no matter whether EXPIRE INVENTORY runs or not. The EXPIRE INVENTORY is what updates the %utilization on your tapes, based on the files that have expired. I think it also does some cleanup to make space in the DB for the expired files reusable. So you want to run EXPIRE INVENTORY before reclaim. But I don't think it affects your migration in any way. W On Fri, Aug 28, 2009 at 11:23 AM, Thomas Denier thomas.den...@jeffersonhospital.org wrote: -Sergio O. Fuentes wrote: - I'm revising my TSM administrative schedules and wanted to take an informal poll on how many of you lay out your daily TSM maintenance routines
Re: Daily TSM maintenance schedules
We're big. (2tb/day backup data Mon-Fri nights) Expiration is the 20-ton elephant in the room. If it doesn't run to completion every once in a while, we're in trouble and we have the dreaded database bloat. It has happened, and it wasn't pretty. You can call this schedule crazy, but it's what works for a big TSM system: CLIENT BACKUP + EXPIRATION + RECLAMATION (5PM) BACKUP STGPOOLS + EXPIRATION MIGRATION + EXPIRATION BACKUP DEVCONFIG + BACKUP VOLHIST *** BACKUP DB (incr on weekdays, full on Sun) + RECLAMATION DELETE VOLHIST (triggered by end of BACKUP DB) + RECLAMATION RECLAMATION + EXPIRATION (1-5PM - a relatively slow time. This is my maintenance window.) back to top *** These are fast. Runs while the tape lib is dismounting the migration tapes, and mounting the DB BACKUP tape. Saturday morning only: DELETE FILESPACE (we queue them up all week) You've got to work DELETE FILESPACE into your schedule, because it involves very heavy database I/O. You can't EXPIRE INVENTORY or BACKUP DB during DELETE FILESPACE. MIGRATION won't even start if DELETE FILESPACE is running. We do allow users to do it themselves, but in actual practice they never do. We've got to nag them about ancient, abandoned filespaces, and then we do it for them Sat AM. This is also when we remove old nodes. We use a tape reuse delay of 2 days, as disaster protection for doing things in the wrong order. The idea here is to keep the CPU and the 15,000rpm database disks busy 24/7, and to use as many tape drives as possible for as much of the time as possible. I am constantly tuning this schedule. The basic problem is that there are only 24 hours in a day. Roger Deschner University of Illinois at Chicago rog...@uic.edu Academic Computing Communications Center ==I have not lost my mind -- it is backed up on tape somewhere.= On Fri, 28 Aug 2009, Howard Coles wrote: Correction, it's marked for expiration and it is still recoverable, until the Expire process runs and removes it from the Database. I know this from experience, as we disable our expiration process for a few days due to a server failure, and once due to a legal request. The Expire Process actually removes the DB entry for that version of the file. See Ya' Howard -Original Message- From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of Wanda Prather Sent: Friday, August 28, 2009 11:21 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Daily TSM maintenance schedules Agreed. expire inventory is actually something of a misnomer. If you have your retention set to 14 versions, and someone takes the 15th backup, the oldest version expires right then, you can't get it back. That type of expiration takes place, no matter whether EXPIRE INVENTORY runs or not. The EXPIRE INVENTORY is what updates the %utilization on your tapes, based on the files that have expired. I think it also does some cleanup to make space in the DB for the expired files reusable. So you want to run EXPIRE INVENTORY before reclaim. But I don't think it affects your migration in any way. W On Fri, Aug 28, 2009 at 11:23 AM, Thomas Denier thomas.den...@jeffersonhospital.org wrote: -Sergio O. Fuentes wrote: - I'm revising my TSM administrative schedules and wanted to take an informal poll on how many of you lay out your daily TSM maintenance routines. Functions I'm talking about here include: BACKUP DISK STGPOOLS BACKUP TAPE STGPOOLS BACKUP DEVCONFIG BACKUP VOLHIST BACKUP DB TYPE=FULL PREPARE DELETE VOLHIST MIGRATE STG EXPIRE INV RECLAIM TAPE RECLAIM OFFSITES CLIENT BACKUP WINDOW STARTS (back to top) The above sequence is roughly how I handle our maintenance and is based off of the IBM Redbook (sg247379) TSM Deployment Guide for 5.5 (page 300). I'm seriously considering altering it in this manner: BACKUP STGPOOLS BACKUP DEVCONFIG BACKUP VOLHIST BACKUP DB PREPARE DELETE VOLHIST EXPIRE INV RECLAIM MIGRATE STG CLIENT BACKUP WINDOW STARTS (back to top) The key difference here, is that I'd be expiring right after the DB Backups, and reclaiming space before migration. I feel that this would be more efficient in terms of processing actual unexpired data and data storage (since reclamation would have freed up storage space). I would be concerned that migration would run in perpetuity in cases where the migration window runs into the client backup window. Therefore, I might have migrations run before reclamations. Does anyone else expire data right after your DB backups on a daily basis? Suggestions from anyone? Thank you kindly. I don't understand what benefit to hope for in terms of 'processing actual unexpired data'. You seem to be described a system in which disk storage pools are short-term buffers for incoming backups. With this usage pattern, disk storage pools will contain few
Daily TSM maintenance schedules
Hi all! I'm revising my TSM administrative schedules and wanted to take an informal poll on how many of you lay out your daily TSM maintenance routines. Functions I'm talking about here include: BACKUP DISK STGPOOLS BACKUP TAPE STGPOOLS BACKUP DEVCONFIG BACKUP VOLHIST BACKUP DB TYPE=FULL PREPARE DELETE VOLHIST MIGRATE STG EXPIRE INV RECLAIM TAPE RECLAIM OFFSITES CLIENT BACKUP WINDOW STARTS (back to top) The above sequence is roughly how I handle our maintenance and is based off of the IBM Redbook (sg247379) TSM Deployment Guide for 5.5 (page 300). I'm seriously considering altering it in this manner: BACKUP STGPOOLS BACKUP DEVCONFIG BACKUP VOLHIST BACKUP DB PREPARE DELETE VOLHIST EXPIRE INV RECLAIM MIGRATE STG CLIENT BACKUP WINDOW STARTS (back to top) The key difference here, is that I'd be expiring right after the DB Backups, and reclaiming space before migration. I feel that this would be more efficient in terms of processing actual unexpired data and data storage (since reclamation would have freed up storage space). I would be concerned that migration would run in perpetuity in cases where the migration window runs into the client backup window. Therefore, I might have migrations run before reclamations. Does anyone else expire data right after your DB backups on a daily basis? Suggestions from anyone? Thank you kindly. Sergio U. of Maryland
Re: Daily TSM maintenance schedules
Hmm.. seems like I'm not explaining the my whole story here. The purpose for this tinkering of admin schedules is to be as efficient as possible so I'm examining the repercussions of adjusting the expiration schedule. We do in fact have a larger SATA pool we use for storing about a week's worth of backup data. In addition to that, a few of our retention policies limit backup versions to only 2 or 7 versions. Given these parameters, there may be instances where there is expired data on the SATA pools (devclass file). I'm trying to avoid data movement (SATA to SATA or SATA to TAPE) of expired data which may still be processed if expire inventory has not yet run. Seems like there would be little benefit in doing so. However, we may be running into a situation from noon to 5pm where the only thing I will be allowed to do is reclamations or expirations. Which is why I wanted to do migrations last. Well, back to the drawing board. SF -Original Message From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of Thomas Denier Sent: Friday, August 28, 2009 11:24 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Daily TSM maintenance schedules I don't understand what benefit to hope for in terms of 'processing actual unexpired data'. You seem to be described a system in which disk storage pools are short-term buffers for incoming backups. With this usage pattern, disk storage pools will contain few inactive backup files of any sort, and no backup files that have been inactive long enough to be candidates for removal by an expiration process. The 'expire-reclaim-migrate' version of the proposed change will cause reclaimed volumes to revert to scratch or 'Empty' status a few hours earlier than they would under your current policy. The 'expire-migrate-reclaim' version of the proposed change would eliminate even this marginal advantage.
Re: Daily TSM maintenance schedules
We run Expire and migrate at the same time. While both run slower, the total time is less. Andy Huebner -Original Message- From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of Sergio O. Fuentes Sent: Friday, August 28, 2009 9:29 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Daily TSM maintenance schedules Hi all! I'm revising my TSM administrative schedules and wanted to take an informal poll on how many of you lay out your daily TSM maintenance routines. Functions I'm talking about here include: BACKUP DISK STGPOOLS BACKUP TAPE STGPOOLS BACKUP DEVCONFIG BACKUP VOLHIST BACKUP DB TYPE=FULL PREPARE DELETE VOLHIST MIGRATE STG EXPIRE INV RECLAIM TAPE RECLAIM OFFSITES CLIENT BACKUP WINDOW STARTS (back to top) The above sequence is roughly how I handle our maintenance and is based off of the IBM Redbook (sg247379) TSM Deployment Guide for 5.5 (page 300). I'm seriously considering altering it in this manner: BACKUP STGPOOLS BACKUP DEVCONFIG BACKUP VOLHIST BACKUP DB PREPARE DELETE VOLHIST EXPIRE INV RECLAIM MIGRATE STG CLIENT BACKUP WINDOW STARTS (back to top) The key difference here, is that I'd be expiring right after the DB Backups, and reclaiming space before migration. I feel that this would be more efficient in terms of processing actual unexpired data and data storage (since reclamation would have freed up storage space). I would be concerned that migration would run in perpetuity in cases where the migration window runs into the client backup window. Therefore, I might have migrations run before reclamations. Does anyone else expire data right after your DB backups on a daily basis? Suggestions from anyone? Thank you kindly. Sergio U. of Maryland This e-mail (including any attachments) is confidential and may be legally privileged. If you are not an intended recipient or an authorized representative of an intended recipient, you are prohibited from using, copying or distributing the information in this e-mail or its attachments. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete all copies of this message and any attachments. Thank you.
Re: Daily TSM maintenance schedules
Correction, it's marked for expiration and it is still recoverable, until the Expire process runs and removes it from the Database. I know this from experience, as we disable our expiration process for a few days due to a server failure, and once due to a legal request. The Expire Process actually removes the DB entry for that version of the file. See Ya' Howard -Original Message- From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of Wanda Prather Sent: Friday, August 28, 2009 11:21 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Daily TSM maintenance schedules Agreed. expire inventory is actually something of a misnomer. If you have your retention set to 14 versions, and someone takes the 15th backup, the oldest version expires right then, you can't get it back. That type of expiration takes place, no matter whether EXPIRE INVENTORY runs or not. The EXPIRE INVENTORY is what updates the %utilization on your tapes, based on the files that have expired. I think it also does some cleanup to make space in the DB for the expired files reusable. So you want to run EXPIRE INVENTORY before reclaim. But I don't think it affects your migration in any way. W On Fri, Aug 28, 2009 at 11:23 AM, Thomas Denier thomas.den...@jeffersonhospital.org wrote: -Sergio O. Fuentes wrote: - I'm revising my TSM administrative schedules and wanted to take an informal poll on how many of you lay out your daily TSM maintenance routines. Functions I'm talking about here include: BACKUP DISK STGPOOLS BACKUP TAPE STGPOOLS BACKUP DEVCONFIG BACKUP VOLHIST BACKUP DB TYPE=FULL PREPARE DELETE VOLHIST MIGRATE STG EXPIRE INV RECLAIM TAPE RECLAIM OFFSITES CLIENT BACKUP WINDOW STARTS (back to top) The above sequence is roughly how I handle our maintenance and is based off of the IBM Redbook (sg247379) TSM Deployment Guide for 5.5 (page 300). I'm seriously considering altering it in this manner: BACKUP STGPOOLS BACKUP DEVCONFIG BACKUP VOLHIST BACKUP DB PREPARE DELETE VOLHIST EXPIRE INV RECLAIM MIGRATE STG CLIENT BACKUP WINDOW STARTS (back to top) The key difference here, is that I'd be expiring right after the DB Backups, and reclaiming space before migration. I feel that this would be more efficient in terms of processing actual unexpired data and data storage (since reclamation would have freed up storage space). I would be concerned that migration would run in perpetuity in cases where the migration window runs into the client backup window. Therefore, I might have migrations run before reclamations. Does anyone else expire data right after your DB backups on a daily basis? Suggestions from anyone? Thank you kindly. I don't understand what benefit to hope for in terms of 'processing actual unexpired data'. You seem to be described a system in which disk storage pools are short-term buffers for incoming backups. With this usage pattern, disk storage pools will contain few inactive backup files of any sort, and no backup files that have been inactive long enough to be candidates for removal by an expiration process. The 'expire-reclaim-migrate' version of the proposed change will cause reclaimed volumes to revert to scratch or 'Empty' status a few hours earlier than they would under your current policy. The 'expire-migrate-reclaim' version of the proposed change would eliminate even this marginal advantage.
