Re: Lots of newbie questions
On Tue, 15 Aug 2006 14:34:49 -0500, Troy Frank [EMAIL PROTECTED] said: Where would you draw the line with this? For monthly snapshots, you're saying it works better than archives/backupsets. Would you also extend this to replacing yearly archives that need to be around 7 years? I don't see why not, but I've not spent as much time mulling it over as you probably have. Yes; in fact we're setting about doing just such a thing. In this environment, I expect a (to us) totally alien concern to be the most important: Tape reliability. You wrote EOT of tape number one; You will now not touch that tape for seven years, or it's copy volumes. How do you assure yourself that the data is legible? For live data, we usually have churn in our tape pools; expiration and reclamation usually cycle through the entire corpus of tapes in a reasonable timeframe. Long-term storage of static data blows that model. Once framed that way the solution is obvious: when possible, take the oldest tape you've got and MOVE DATA on it. If you can do this a few times a month, You will sharply curtail the possibility of a write-only volume. - Allen S. Rout
Re: Lots of newbie questions
We don't have 7 year retentions but we do have a process that does a 'move data' on any tape that was last written over a year ago. David Allen S. Rout [EMAIL PROTECTED] 8/16/2006 10:20:10 AM On Tue, 15 Aug 2006 14:34:49 -0500, Troy Frank [EMAIL PROTECTED] said: Where would you draw the line with this? For monthly snapshots, you're saying it works better than archives/backupsets. Would you also extend this to replacing yearly archives that need to be around 7 years? I don't see why not, but I've not spent as much time mulling it over as you probably have. Yes; in fact we're setting about doing just such a thing. In this environment, I expect a (to us) totally alien concern to be the most important: Tape reliability. You wrote EOT of tape number one; You will now not touch that tape for seven years, or it's copy volumes. How do you assure yourself that the data is legible? For live data, we usually have churn in our tape pools; expiration and reclamation usually cycle through the entire corpus of tapes in a reasonable timeframe. Long-term storage of static data blows that model. Once framed that way the solution is obvious: when possible, take the oldest tape you've got and MOVE DATA on it. If you can do this a few times a month, You will sharply curtail the possibility of a write-only volume. - Allen S. Rout
Re: Lots of newbie questions
We have a script and it does a move data on any tape (primary or copy) that is older then (not read or written in) x days. We are currently using 6 months as our x. This makes sure that offsite tapes do not sit for 7 years also. -- Phillip (901)320-4462 (901)320-4856 FAX -Original Message- From: Allen S. Rout [mailto:[EMAIL PROTECTED] Sent: Wednesday, August 16, 2006 9:20 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Lots of newbie questions On Tue, 15 Aug 2006 14:34:49 -0500, Troy Frank [EMAIL PROTECTED] said: Where would you draw the line with this? For monthly snapshots, you're saying it works better than archives/backupsets. Would you also extend this to replacing yearly archives that need to be around 7 years? I don't see why not, but I've not spent as much time mulling it over as you probably have. Yes; in fact we're setting about doing just such a thing. In this environment, I expect a (to us) totally alien concern to be the most important: Tape reliability. You wrote EOT of tape number one; You will now not touch that tape for seven years, or it's copy volumes. How do you assure yourself that the data is legible? For live data, we usually have churn in our tape pools; expiration and reclamation usually cycle through the entire corpus of tapes in a reasonable timeframe. Long-term storage of static data blows that model. Once framed that way the solution is obvious: when possible, take the oldest tape you've got and MOVE DATA on it. If you can do this a few times a month, You will sharply curtail the possibility of a write-only volume. - Allen S. Rout * This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.
Re: Lots of newbie questions
On Fri, Aug 11, 2006 at 09:29:26AM -0400, Allen S. Rout wrote: Said from another direction, TSM doesn't worry about selecting only backed-up data from a stgpool when it's migrating, it takes whatever is there. Thanks for the clarification. Best practice-wise, do you try to define different private groups and tape labels for onsite, archive and offsite storage? Or do Your recordkeeping problem is too complex to do by casual inspection, ... You will inevitably end up needing more Primary tapes when all you've got on hand will be labeled for Reverse thudpucker use or some such; Then you'll cross your own lines, and after the first time it gets easier and easier I've been running purposedly labeled tapes/pools for quite a while in the leveled backup world of Networker. Jumping to a 'just the system pick it' mentality is one of the hurdles (and appeals) of TSM. I fully understand your point on the media labels, and will learn to live with it. :) Having an extra drive outside a library isn't exactly a _bad_ idea, but unless you're trying to write a different format I'd expect you to get more value out of adding it to the library. I don't think I can convert an external drive to run in the library. You make good arguments for having internal drives, and I don't disagree. But given that I have an external tape drive, it shouldn't be an either/or for me to use it or the library. If you really, really want to do this, I'd suggest: - Define all of the drives to be in the library. set the one which is physically outside to usually be offline. - When you want to use the external drive, set the interior drives offline, the exterior online. Run the job, mount, dismount, etc. - When you're done, re-set to normal operations. I suspect that by itself wouldn't work because the SCSI library would want to do something to checkin/mount the drive, and I wouldn't want to reconfigure switching the library to manual or not. I'm thinking more along the lines of defining the SCSI library and manual library, and switching the devclass between them as needed. That's just a stopgap - it doesn't really make it usable, unless I define an entirely new devclass and use it for relatively few things (like backupsets or db backups to tape.) No one else has dealt with transitioning between both manual and automatic libraries? Personally, I'd much prefer checkin and checkout of desired volumes to this. And get a quote on how much the next bigger step of library is, and count the amount of time you spend screwing around with inadequate library space. That way you can demonstrate to the folks who are hoarding the beans when they start losing money because they didn't cough up a library at the outset. TSM is tremendously economical with tape and drive resources compared to other backup products. Feed it well; feed it hardware instead of your time. That point is muddied when a drive configuration that worked just fine with other backup products is either unusable or inadequate for TSM. I'm sticking with my current hardware/specs for this first year and have already warned the bean counters that odds are I'll need something near the start of year two -- be it more drives, more library space, more staging disk, beefier or multiple servers, etc. Now, if you were me, you'd try to develop a theory of copystg utilization workflow, and solve it for a minimum. But I suggest you just twirl the knob to the other end, and see if you like that tradeoff better. Good point. I'll try some variations and see how it goes. You should consider data which has expired to be Gone, except for heroic measures. If you Really Need data which expired out of the database in the last week or so (a common period for retention of DB backups) then yes, you can do a complete DR scenario, and consult the as-yet unreused tape volumes for the data. Icky squicky. Thanks, that's the definition I was looking for. As for Really Need, it depends who asks... :) It's usually an answer like 'You messed this up last month, overwrote it every week and didn't notice until the first of this month, and have waited until NOW to ask me to get it back? No, it's gone.. My backups will probably be augmented with regular archives (or backupsets) for the important systems, so the provision of 'losing' anything (at least for the people who Really Need it) should be pretty low. Any opinion on archives vs. backupsets for a monthly snapshot kept for 6 months? Thanks for humoring me, Dave
Re: Lots of newbie questions
On Tue, 15 Aug 2006 11:04:11 -0500, Dave Mussulman [EMAIL PROTECTED] said: Any opinion on archives vs. backupsets for a monthly snapshot kept for 6 months? Yes; E) None of the above. If you want a monthly snapshot, define a new devclass, with different retention parameters; perhaps: verexists: 8 verdeleted: 8 retextra: 300 retonly:300 This is slightly paranoid, with a little extra in each category. Then define a FOO-MONTHLY node as an echo of each node FOO in your normal management class. Run the incrs for the -MONTHLY nodes, monthly. ;) You'll accumulate 8 versions with these settings, and since you're only cutting new versions (at a maximum) monthly, you'll have 8 months. I used 8 instead of 6 Just In Case someone runs an incr out of school, so you don't the oldest version off the end of the queue. For a more detailed level of snapshot, you could set FREQUENCY=30, and run incrementals against the -MONTHLY nodes every day. This way you still keep 95% of the advantages of incremental-forever processing, and still leverage the TSM database. Resort to backupsets if your needs include restore operations independant of a running TSM server, or independant of good network. - Allen S. Rout
Re: Lots of newbie questions
Where would you draw the line with this? For monthly snapshots, you're saying it works better than archives/backupsets. Would you also extend this to replacing yearly archives that need to be around 7 years? I don't see why not, but I've not spent as much time mulling it over as you probably have. [EMAIL PROTECTED] 8/15/2006 12:26 PM On Tue, 15 Aug 2006 11:04:11 -0500, Dave Mussulman [EMAIL PROTECTED] said: Any opinion on archives vs. backupsets for a monthly snapshot kept for 6 months? Yes; E) None of the above. If you want a monthly snapshot, define a new devclass, with different retention parameters; perhaps: verexists: 8 verdeleted: 8 retextra: 300 retonly:300 This is slightly paranoid, with a little extra in each category. Then define a FOO-MONTHLY node as an echo of each node FOO in your normal management class. Run the incrs for the -MONTHLY nodes, monthly. ;) You'll accumulate 8 versions with these settings, and since you're only cutting new versions (at a maximum) monthly, you'll have 8 months. I used 8 instead of 6 Just In Case someone runs an incr out of school, so you don't the oldest version off the end of the queue. For a more detailed level of snapshot, you could set FREQUENCY=30, and run incrementals against the -MONTHLY nodes every day. This way you still keep 95% of the advantages of incremental-forever processing, and still leverage the TSM database. Resort to backupsets if your needs include restore operations independant of a running TSM server, or independant of good network. - Allen S. Rout Confidentiality Notice follows: The information in this message (and the documents attached to it, if any) is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken, or omitted to be taken in reliance on it is prohibited and may be unlawful. If you have received this message in error, please delete all electronic copies of this message (and the documents attached to it, if any), destroy any hard copies you may have created and notify me immediately by replying to this email. Thank you.
Re: Lots of newbie questions
On Thu, 10 Aug 2006 12:10:09 -0500, Dave Mussulman [EMAIL PROTECTED] said: I do my backups to a DISK pool that has a 85/40 migrate percentage and a tape next storage pool. If everything (my disk and tape pool) is synced up to my copypool before a backup runs, and the backup only goes to disk, the 'backup stg' for the tape has nothing to do. I understand that. If I backup the disk pool and manually migrate the data from disk to tape, and then backup the tape pool, it has nothing to do. (Since that data was already in the copypool.) I understand that. But, if during a backup the tape pool starts an automatic migration, the next time I do a 'backup stg' for the tape pool, it has data to copy to the copypool. So, what's happening? Migration begins for a particular node or collocgroup, and then completes for that node or collocgroup (node-ish) before moving to the next one. If that node-ish has backed up more stuff since the last backup stgpool completed, then the uncopied data will get migrated too. Said from another direction, TSM doesn't worry about selecting only backed-up data from a stgpool when it's migrating, it takes whatever is there. Defining a copystgpool for the destination of the migration is advertised to help with this: It's claimed that data will be simultaneously written to the target primary pool and the copy pool. I have not been able to elicit this behavior myself, and am somewhat grumpy therefore. Best practice-wise, do you try to define different private groups and tape labels for onsite, archive and offsite storage? Or do people really just make one large 'pool' of tapes and not care if tape 0004 has archive data on it and stays for years, 0005 goes offsite, 0006 has a two week DB backup on it, and 0007 is a normal tape pool tape? Your recordkeeping problem is too complex to do by casual inspection, even if you separate the classes as you suggest. Work out queries against the database that satisfy your auditing notions, and trust them. Oh, and run them. ;) You will inevitably end up needing more Primary tapes when all you've got on hand will be labeled for Reverse thudpucker use or some such; Then you'll cross your own lines, and after the first time it gets easier and easier Reminds me of the first time I checked all of the volumes in a copystgpool out of my library. Just this once, I thought. But once you're there in the back seat Uh. Digression. I have a LTO3 tape library and an external LTO3 drive. In our Networker environment, we found it a pretty good practice to have a drive outside of the jukebox for one-off operations (old restores, etc.) as well as some sort of fault tolerance if the jukebox or that SCSI bus went south. How do I setup that environment in TSM? Having an extra drive outside a library isn't exactly a _bad_ idea, but unless you're trying to write a different format I'd expect you to get more value out of adding it to the library. The purposes for which we'd like external drives are mostly what are called backupsets, intended to be used independant of the TSM Server, written in media formats accessible to drives directly attached to the clients. If you really, really want to do this, I'd suggest: - Define all of the drives to be in the library. set the one which is physically outside to usually be offline. - When you want to use the external drive, set the interior drives offline, the exterior online. Run the job, mount, dismount, etc. - When you're done, re-set to normal operations. Personally, I'd much prefer checkin and checkout of desired volumes to this. And get a quote on how much the next bigger step of library is, and count the amount of time you spend screwing around with inadequate library space. That way you can demonstrate to the folks who are hoarding the beans when they start losing money because they didn't cough up a library at the outset. TSM is tremendously economical with tape and drive resources compared to other backup products. Feed it well; feed it hardware instead of your time. Consolidating copypool tapes for offsite use.I had my reclaimation threshold for my copypool at 40%. I used 'backup stg' with maxpr=[# of drives] to minimize the amount of time the backup took. In proper consultant fashion, I'll tell you the time with your watch. You minimized the time the backup took, and then found that this minimization squoze out, maximizing another part of your workflow. Now, if you were me, you'd try to develop a theory of copystg utilization workflow, and solve it for a minimum. But I suggest you just twirl the knob to the other end, and see if you like that tradeoff better. So 'backup stg' with only one process, accept that it'll take longer, and see if that's quick enough. If you can get a little bit of offsite reclamation accomplished in most of your rotation cycles, then you're doing pretty well. I'm believe I understand the way a DR scenario should
[OT] Re: Lots of newbie questions
Allen Rout wrote, in part: Reminds me of the first time I checked all of the volumes in a copystgpool out of my library. Just this once, I thought. But once you're there in the back seat Uh. Digression. Why do you always leave out the *good* parts of your stories! :-) Andy Raibeck IBM Software Group Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew Raibeck/Tucson/[EMAIL PROTECTED] Internet e-mail: [EMAIL PROTECTED] IBM Tivoli Storage Manager support web page: http://www-306.ibm.com/software/sysmgmt/products/support/IBMTivoliStorageManager.html The only dumb question is the one that goes unasked. The command line is your friend. Good enough is the enemy of excellence.
Re: [OT] Re: Lots of newbie questions
On Fri, 11 Aug 2006 08:07:17 -0600, Andrew Raibeck [EMAIL PROTECTED] said: Allen Rout wrote, in part: Reminds me of the first time I checked all of the volumes in a copystgpool out of my library. Just this once, I thought. But once you're there in the back seat Uh. Digression. Why do you always leave out the *good* parts of your stories! :-) Oh, all right. Once the boundary I had considered sacred had been crossed, I found it harder and harder to refuse successive requests. Eventually I was doing it for everyone; it was automatic. I even wrote a script for it. It wasn't until a major conversion experience (deploying the remote library in Atlanta) that I regained the capability to say no with conviction, to stick to my guns. Now I've got 81 empty cells and 44 scratch volumes. I can reasonably hope that such sordid behavior is permanently in my past. - Allen S. Rout - The script is named '/var/tsm/bin/stupid-checkouts'
Lots of newbie questions
Hello, Forgive the laundry list of questions, but here's a few things as a newbie I don't quite understand. Each paragraph is a different question/topic, so feel free to chime in on just a few or any that you're comfortable answering. Thanks! I'm using Operational Reporting with the automatic notification turned on for failed or missed schedules. I have a node associated with a schedule that no longer exists (not powered on,) just to test failures and notifications. However, I never get notifications about failed or missed schedules from it (not the email or mentioned in the daily report.) In the client schedules part of the report, it's always in a Pending status. At what point does pending turn into failed or missed? How can I configure that so I get notifications about systems that missed their scheduled backup? I'm using an administrative schedule to backup my DB to a FILE class twice a day, and then I do full backups of the DB to tape right before my offsite rotation. I read somewhere that since I'm using DRM, I shouldn't use 'del volhist' to remove old db backups. However, I don't think the DRMDBBACKUPEXPIREDAYS configuration setting is applying to my FILE backups. Is that normal? Should I be running both drm and 'del volhist'? I do my backups to a DISK pool that has a 85/40 migrate percentage and a tape next storage pool. If everything (my disk and tape pool) is synced up to my copypool before a backup runs, and the backup only goes to disk, the 'backup stg' for the tape has nothing to do. I understand that. If I backup the disk pool and manually migrate the data from disk to tape, and then backup the tape pool, it has nothing to do. (Since that data was already in the copypool.) I understand that. But, if during a backup the tape pool starts an automatic migration, the next time I do a 'backup stg' for the tape pool, it has data to copy to the copypool. So, what's happening? Since the migration is going on, does TSM automatically route data from the node directly to the tape? (My maxsize parameter for the disk pool is No Limit, so I would guess no.) Or is TSM migrating from the disk pool the newest data that wasn't already copied to the copypool? In that case, why doesn't it migrate older data that's already copied? What's the selection criteria for what gets migrated? Or would best practice say to manually migrate the disk pool daily to minimize the chance of this condition? Best practice-wise, do you try to define different private groups and tape labels for onsite, archive and offsite storage? Or do people really just make one large 'pool' of tapes and not care if tape 0004 has archive data on it and stays for years, 0005 goes offsite, 0006 has a two week DB backup on it, and 0007 is a normal tape pool tape? Since there's not one (standard) unified view for volumes (DB backups, offsite backups, checked in scratch volumes, not checked in scratch volumes) I worry a little about keeping track of and 'losing something' if they're all in one group. How do sites handle that issue? I have a LTO3 tape library and an external LTO3 drive. In our Networker environment, we found it a pretty good practice to have a drive outside of the jukebox for one-off operations (old restores, etc.) as well as some sort of fault tolerance if the jukebox or that SCSI bus went south. How do I setup that environment in TSM? It looks like I cannot use the same device class across two libraries. Doesn't that hinder me if I want to use the external drive in the same way as the jukebox drives, sharing storage pools, etc.? My jukebox isn't very large and I anticipate having to use overflow storage pools, which is where being able to mix the manual library (external drive) and SCSI library would be nice. Consolidating copypool tapes for offsite use. I had my reclaimation threshold for my copypool at 40%. I used 'backup stg' with maxpr=[# of drives] to minimize the amount of time the backup took. However, it left me with many under-utilized offsite drives, that as soon as I moved offsite, were then reclaimed (which then sat until the reusedelay expired.) That seems inefficient - that I move a larger than necessary number of tapes each time I do an offsite rotation (right now, weekly) and as soon as they're offsite, they're reclaimed. To fix this, I put the reclaimation threshold back to 100%, and set it down just before my offsite rotation. I've also taken a look at the to-be-offsited tapes and do some 'move data's as required to try to minimize the amount of offsite tapes. Is that standard practice? I feel like I'm fighting the natural way TSM works, given that it makes so many other decisions just fine without my direct intervention. (And that's a compliment - I can't say that for Networker.) Is there something I'm missing to make offsite tape usage more streamlined? So my offsite rotation procedure is starting to look like: - expire inventory - backup all the local storage
