Spontaneous reclamation
TSM Server 5.1.9.0 on AIX 5.2 ML-4 TSM Client mix of 5.1.6.0 and 5.1.7.0 mostly on Windows This morning I found a primary storage pool tape pending as the result of reclamation that launched spontaneously just after midnight. The logs show no evidence of anyone issuing any commands to launch that operation. I'm trying to figure out how that happened since we maintain pretty tight control over the TSM maintenance processes. Specifically, on our system, maintenance occurs weekly. On Monday at 9:00 am we run expiration which of course triggers reclamation of the primary pools. Expiration completes late Monday afternoon with all primary reclamation completing sometime late Monday night to early Tuesday morning. We leave our copypool reclaim point set at 100% throughout this process. On Tuesday at 6:00 pm we launch copypool reclamation, after which the reclaim point is returned to 100%. That is our entire maintenance schedule. No other expirations or reclamations are scheduled to occur at any time. So what could have provoked a primary tape reclamation at midnight, approximately nine hours before the start of the regular maintenance cycle? Just curious. Tab Trepagnier TSM Administrator Laitram, L.L.C.
Re: Spontaneous reclamation
I've seen reclamations kick off after deleteing nodes. [EMAIL PROTECTED] 03/21/2005 10:17:04 AM TSM Server 5.1.9.0 on AIX 5.2 ML-4 TSM Client mix of 5.1.6.0 and 5.1.7.0 mostly on Windows This morning I found a primary storage pool tape pending as the result of reclamation that launched spontaneously just after midnight. The logs show no evidence of anyone issuing any commands to launch that operation. I'm trying to figure out how that happened since we maintain pretty tight control over the TSM maintenance processes. Specifically, on our system, maintenance occurs weekly. On Monday at 9:00 am we run expiration which of course triggers reclamation of the primary pools. Expiration completes late Monday afternoon with all primary reclamation completing sometime late Monday night to early Tuesday morning. We leave our copypool reclaim point set at 100% throughout this process. On Tuesday at 6:00 pm we launch copypool reclamation, after which the reclaim point is returned to 100%. That is our entire maintenance schedule. No other expirations or reclamations are scheduled to occur at any time. So what could have provoked a primary tape reclamation at midnight, approximately nine hours before the start of the regular maintenance cycle? Just curious. Tab Trepagnier TSM Administrator Laitram, L.L.C.
Re: Spontaneous reclamation
Lawrence, So have I, but there was no such activity last night. Only normal backups, with the occasional migration since some of our data paths have small disk pools that function only as mount point multipliers and migrate to tape at 10% filled. Thanks for the tip. Tab ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU wrote on 03/21/2005 09:37:24 AM: I've seen reclamations kick off after deleteing nodes. [EMAIL PROTECTED] 03/21/2005 10:17:04 AM TSM Server 5.1.9.0 on AIX 5.2 ML-4 TSM Client mix of 5.1.6.0 and 5.1.7.0 mostly on Windows This morning I found a primary storage pool tape pending as the result of reclamation that launched spontaneously just after midnight. The logs show no evidence of anyone issuing any commands to launch that operation. I'm trying to figure out how that happened since we maintain pretty tight control over the TSM maintenance processes. Specifically, on our system, maintenance occurs weekly. On Monday at 9:00 am we run expiration which of course triggers reclamation of the primary pools. Expiration completes late Monday afternoon with all primary reclamation completing sometime late Monday night to early Tuesday morning. We leave our copypool reclaim point set at 100% throughout this process. On Tuesday at 6:00 pm we launch copypool reclamation, after which the reclaim point is returned to 100%. That is our entire maintenance schedule. No other expirations or reclamations are scheduled to occur at any time. So what could have provoked a primary tape reclamation at midnight, approximately nine hours before the start of the regular maintenance cycle? Just curious. Tab Trepagnier TSM Administrator Laitram, L.L.C.
Re: Spontaneous reclamation
During backups TSM will expire files that roll over the vere setting. Can someone confirm that this is done immediately and not when expiration runs? Maybe a client expired THE file that held a tape at 50% and thus that tape went to 51% reclaimable. Guillaume Gilbert Systems Specialist 514.866.8876 Office 514.866.0901 Fax 514.290.6526 BlackBerry [EMAIL PROTECTED] StorageTek Canada -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Tab Trepagnier Sent: March 21, 2005 11:02 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: Spontaneous reclamation David, I'll ask the obvious. Any chance you copypool reached 100% triggering a reclaim? The reclamation ran on a primary storage pool. There is nothing *obvious* that could have pushed ONE tape over the pool's 50% reclamation threshold during the midnight to 1:00 am period when reclamation launched. Reclamation ran on that one volume only. During our maintenance cycle, which is running now, that particular pool can reclaim as many as 30 volumes. Thanks. Tab
Re: Spontaneous reclamation
I'll ask the obvious. Any chance you copypool reached 100% triggering a reclaim? [EMAIL PROTECTED] 03/21/05 10:17 AM TSM Server 5.1.9.0 on AIX 5.2 ML-4 TSM Client mix of 5.1.6.0 and 5.1.7.0 mostly on Windows This morning I found a primary storage pool tape pending as the result of reclamation that launched spontaneously just after midnight. The logs show no evidence of anyone issuing any commands to launch that operation. I'm trying to figure out how that happened since we maintain pretty tight control over the TSM maintenance processes. Specifically, on our system, maintenance occurs weekly. On Monday at 9:00 am we run expiration which of course triggers reclamation of the primary pools. Expiration completes late Monday afternoon with all primary reclamation completing sometime late Monday night to early Tuesday morning. We leave our copypool reclaim point set at 100% throughout this process. On Tuesday at 6:00 pm we launch copypool reclamation, after which the reclaim point is returned to 100%. That is our entire maintenance schedule. No other expirations or reclamations are scheduled to occur at any time. So what could have provoked a primary tape reclamation at midnight, approximately nine hours before the start of the regular maintenance cycle? Just curious. Tab Trepagnier TSM Administrator Laitram, L.L.C.
Re: Spontaneous reclamation
David, I'll ask the obvious. Any chance you copypool reached 100% triggering a reclaim? The reclamation ran on a primary storage pool. There is nothing *obvious* that could have pushed ONE tape over the pool's 50% reclamation threshold during the midnight to 1:00 am period when reclamation launched. Reclamation ran on that one volume only. During our maintenance cycle, which is running now, that particular pool can reclaim as many as 30 volumes. Thanks. Tab
Re: Spontaneous reclamation
Gilbert, Guillaume wrote: During backups TSM will expire files that roll over the vere setting. Can someone confirm that this is done immediately and not when expiration runs? Maybe a client expired THE file that held a tape at 50% and thus that tape went to 51% reclaimable. very well possible. Alternatively, you could have a volume that was being reclaimed, but reclaim failed for some reason, and after a few days of (re-)trying, TSM finally succeded in reclaiming that tape. Guillaume Gilbert Systems Specialist 514.866.8876 Office 514.866.0901 Fax 514.290.6526 BlackBerry [EMAIL PROTECTED] StorageTek Canada -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Tab Trepagnier Sent: March 21, 2005 11:02 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: Spontaneous reclamation David, I'll ask the obvious. Any chance you copypool reached 100% triggering a reclaim? The reclamation ran on a primary storage pool. There is nothing *obvious* that could have pushed ONE tape over the pool's 50% reclamation threshold during the midnight to 1:00 am period when reclamation launched. Reclamation ran on that one volume only. During our maintenance cycle, which is running now, that particular pool can reclaim as many as 30 volumes. Thanks. Tab -- Met vriendelijke groeten, Remco Post SARA - Reken- en Netwerkdiensten http://www.sara.nl High Performance Computing Tel. +31 20 592 3000Fax. +31 20 668 3167 I really didn't foresee the Internet. But then, neither did the computer industry. Not that that tells us very much of course - the computer industry didn't even foresee that the century was going to end. -- Douglas Adams
Re: Spontaneous reclamation
Richard, The only explanation I can think of for this would be the influence of the EXPINterval server option, if that hasn't been deactivated; but I'd expect that you'd have seen evidence in the Activity Log. I just verified that ExpInterval is set to zero. I was pretty sure that it was. We've been controlling the expiration via cron scripts since ADSM 2 was first installed. And in reference to Remco's message, we check for failed processes daily. There were no failed reclamations at the end of the last maintenance cycle, so there was nothing lingering for TSM to resume. I think Guillaume might have figured it out. Let's say we keep five versions of a file. If, during the night, the system picks up a sixth version, does the oldest version scroll off immediately? Or does it wait for reclamation? If immediately, that is almost certainly what happened. But what is odd is that I've never noticed that occurrence before despite administering the system for the last seven years! Thanks. Tab
Re: Spontaneous reclamation
Do you have any 5.3 clients? That client level adds a 'delete backup' command. The documentation states that this command causes the TSM server to remove a backup from server storage immediately. If the documentation is accurate, a client using this command could reduce the occupancy of a volume at any time, independent of inventory expiration.
Re: Spontaneous reclamation
From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Tab Trepagnier I think Guillaume might have figured it out. Let's say we keep five versions of a file. If, during the night, the system picks up a sixth version, does the oldest version scroll off immediately? Or does it wait for reclamation? If immediately, that is almost certainly what happened. But what is odd is that I've never noticed that occurrence before despite administering the system for the last seven years! No, the oldest version of the file doesn't just roll off. You have to run an inventory expiration to delete the pointer to the old file; once the pointer is deleted, and you meet your reclamation threshhold, the file indicated by the expired pointer will vanish when the volume it resides in is reclaimed and reused. -- Mark Stapleton ([EMAIL PROTECTED]) Office 262.521.5627
Re: Spontaneous reclamation
Mark, Thanks for that clarification. That matches what I've seen on the archive side of the system. Tab Stapleton, Mark [EMAIL PROTECTED] Sent by: ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU 03/21/2005 12:39 PM Please respond to ADSM: Dist Stor Manager ADSM-L@VM.MARIST.EDU To ADSM-L@VM.MARIST.EDU cc Subject Re: Spontaneous reclamation From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Tab Trepagnier I think Guillaume might have figured it out. Let's say we keep five versions of a file. If, during the night, the system picks up a sixth version, does the oldest version scroll off immediately? Or does it wait for reclamation? If immediately, that is almost certainly what happened. But what is odd is that I've never noticed that occurrence before despite administering the system for the last seven years! No, the oldest version of the file doesn't just roll off. You have to run an inventory expiration to delete the pointer to the old file; once the pointer is deleted, and you meet your reclamation threshhold, the file indicated by the expired pointer will vanish when the volume it resides in is reclaimed and reused. -- Mark Stapleton ([EMAIL PROTECTED]) Office 262.521.5627
Re: Spontaneous reclamation
Thanks for the clarification Mark. This is something I had never really tested and had always wondered about. Guillaume Gilbert Systems Specialist 514.866.8876 Office 514.866.0901 Fax 514.290.6526 BlackBerry [EMAIL PROTECTED] StorageTek Canada -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Stapleton, Mark Sent: March 21, 2005 13:39 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: Spontaneous reclamation From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Tab Trepagnier I think Guillaume might have figured it out. Let's say we keep five versions of a file. If, during the night, the system picks up a sixth version, does the oldest version scroll off immediately? Or does it wait for reclamation? If immediately, that is almost certainly what happened. But what is odd is that I've never noticed that occurrence before despite administering the system for the last seven years! No, the oldest version of the file doesn't just roll off. You have to run an inventory expiration to delete the pointer to the old file; once the pointer is deleted, and you meet your reclamation threshhold, the file indicated by the expired pointer will vanish when the volume it resides in is reclaimed and reused. -- Mark Stapleton ([EMAIL PROTECTED]) Office 262.521.5627
Re: Spontaneous reclamation
Any objects fail - big ones, like database backups? If you had a filling tape, went full, session goes on to another tape, then fails, you might get a tape below your reclamation threshold. Another possibility would be if a client deleted backups or archives themselves. For things like this, I like to keep my reclamation threshold at 100 when I don't want it to run. Nick Cassimatis Senior IT Specialist, Tivoli Storage Software Office: 877-462-6709 T/L 930-1720 email: [EMAIL PROTECTED]
