Re: Another VE mystery - restoring from tape - but shouldn't be

[Ray Carlson]

Out of curiosity, had the data been DeDuped when it was on the Disk, before it 
was migrated to tape, and then moved back to disk?
We saw many problems with data corruption and damaged files when this was done 
with a TSM server prior to 6.3.3.
Ray

On Jun 19, 2013, at 11:20 AM, Andrew Raibeck  wrote:

> Hi Wanda,
> 
> Identify one of the tapes being mounted from TAPEPOOL2, then do a QUERY
> CONTENT on it. See if there is anything for DC1. There has to be
> *something* even if we haven't yet put our finger on it.
> 
> Best regards,
> 
> - Andy
> 
> 
> 
> Andrew Raibeck | Tivoli Storage Manager Level 3 Technical Lead |
> stor...@us.ibm.com
> 
> IBM Tivoli Storage Manager links:
> Product support:
> http://www.ibm.com/support/entry/portal/Overview/Software/Tivoli/Tivoli_Storage_Manager
> 
> Online documentation:
> https://www.ibm.com/developerworks/mydeveloperworks/wikis/home/wiki/Tivoli
> Documentation Central/page/Tivoli Storage Manager
> Product Wiki:
> https://www.ibm.com/developerworks/mydeveloperworks/wikis/home/wiki/Tivoli
> +Storage+Manager/page/Home
> 
> "ADSM: Dist Stor Manager"  wrote on 2013-06-19
> 11:53:07:
> 
>> From: "Prather, Wanda" 
>> To: ADSM-L@vm.marist.edu,
>> Date: 2013-06-19 11:55
>> Subject: Re: Another VE mystery - restoring from tape - but shouldn't be
>> Sent by: "ADSM: Dist Stor Manager" 
>> 
>> Richard,
>> Here is the Q OCC result.
>> The Q OCC shows all the data for FSID 86 is in VMCTLMC (seq disk),
>> SLOWDEDUP (seq disk) or the COPYPOOL (tape COPY pool).
>> 
>> But when the customer tries a restore, the tapes getting mounted are
>> from primary pool TAPEPOOL2.
>> How can a restore be calling for tapes in a pool where the filespace
>> has no data according to Q OCC?
>> I suggested they open a sev 1 with Tivoli.  This can't be right.
>> 
>> 
>> tsm: HCG-TSM-SERVER>q occ dc1 86 nametype=fsid
>> 
>> Node Name  Type Filespace   FSID Storage
>> Number of  Physical   Logical
>>Name Pool Name
>> Files Space Space
>> 
>> Occupied  Occupied
>> 
>> (MB)  (MB)
>> --  -- - --
>> - - -
>> DC1Bkup \VMFULL-H-86 COPYPOOL
>> 8,928 99,057.72 99,062.73
>> WDDMZOCU-
>> LARX1
>> DC1Bkup \VMFULL-H-86 SLOWDEDUP
>> 4,464 - 98,733.95
>> WDDMZOCU-
>> LARX1
>> DC1Bkup \VMFULL-H-86 VMCTLMC
>> 4,464315.14315.14
>> WDDMZOCU-
>> LARX1
>> 
>> -Original Message-
>> From: Prather, Wanda
>> Sent: Wednesday, June 19, 2013 11:03 AM
>> To: Richard Cowen
>> Subject: RE: Another VE mystery - restoring from tape - but shouldn't be
>> 
 Did the MOVE NODEDATA result in a zillion new volumes (.BFS's) ?
>> No, we don't use scratch volumes in the seq disk pool
>> 
>> 
 Can you get the activity log for the time the process(es) ran ?
>> Any chance you have query occupancy node=dcname filespace=victim1
>> stgpool= before and after move?
>> If not, does the query for tape pool now show zero?
>> 
>> Don't have them and right now I don't have access, but those are
>> great ideas, will ask the customer for them.
>> 
>> Thanks!
>> 
>> Wanda
>> 
>> -Original Message-
>> From: Prather, Wanda [mailto:wanda.prat...@icfi.com]
>> Sent: Tuesday, June 18, 2013 11:35 AM
>> To: Richard Cowen
>> Subject: RE: Another VE mystery - restoring from tape - but shouldn't be
>> 
>> Hi Richard,
>> 
>> 
 Did you get a "zillion" tape mounts during the MOVE NODEDATA?
>> Yes
>> 
>> 
 Do you know it finished without errors?
>> Yes.  And we ran another MOVE NODEDATA to verify there was no more
>> data to move for that filespace.
>> 
 How, exactly, did the data go from sequential fast disk ->
>> sequential slow disk - > tape ?
>> Ordinary migration, at different times, as the pools hit migration
> thresholds.
>> 
 I didn't think TSM would "migrate" more than one level, so maybe
>> the last step was using a MOVE command?
>> Yes, your migration hierarchy can have as many levels as you want,
>> as long as you don't try to go from a sequential pool back to a disk
>> (random) pool.
>> 
 What does a QUERY NODEDATA show for primary pools and copy pools?
>> Would not be informative, as we only moved some of the filespaces,
>> not all of them.
>> 
 Are you running aggressive reclamation on the tape pool?
>> No, and it's not collocated.  Which is why we decided to move the
>> filespace back to the seq disk pool.
>> I don't think it's odd the data was too spread out to make the
>> restore from tape not work well.
>> What is odd is that all the data from one VM is supposedly in one
> filespace.
>> We moved that

Re: hypothetical situation with dedup turned on

[Ray Carlson]

FYI - We had all the normal settings and still lost our source dedupe data 
while running 6.3.0 to 6.3.1, and no TSM was not smart enough to figure out it 
was gone and back up a fresh copy.  We are currently at 6.3.2.7.  It was one of 
the 6.3.2.x updates, released around 10/1/2012, that finally fixed the problem, 
so that we don't appear to have lost any more source data since then.  So make 
sure you are running one of the latest releases of TSM.
Ray

On Nov 16, 2012, at 9:51 AM, "Prather, Wanda"  wrote:

> I can confirm that if you do audit volume fix=yes , then a MOVE DATA, that it 
> LOOKS like it works.  
> The issue you mention below hadn't occurred to me. 
> Ick.  Could turn ME into a vodka drinker..
> But believe me, I"ve got "deduprequiresbackup yes" in place, and back up to a 
> non-deduped copy pool.
> 
> (And, by the way, the term "full incremental" makes me twitch, even without 
> the vodka!)
> 
> 
> 
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Alex 
> Paschal
> Sent: Friday, November 16, 2012 1:09 AM
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: [ADSM-L] hypothetical situation with dedup turned on
> 
> Hi, David.  You can still do as you're already doing:  "audit volume fix=yes" 
> to find the damaged blocks, then do a "move data" against the good data.  
> That would leave the unreadable data on the volume.
> 
> If the copypool volume is unavailable for a "restore volume", then the only 
> thing you could do is "delete volume discarddata=yes" and take the 
> concomitant loss of data that refers to the bad blocks. TSM should then 
> re-back up that data during the next full incremental backup.  (Full 
> incremental?  Oxymoron!  Also, maybe too much vodka. Stoli's Orange, tonight. 
>  ;-)
> 
> Question for the IBMers:  Is TSM smart enough to delete all of the file 
> objects that refer to the deduped/damaged/discarded blocks?  I would expect 
> so, especially with the ~new DB2 referential integrity enforcement, but I 
> think that's exactly what David's question is getting at.  Could we get an 
> authoritative answer on that?
> 
> And a more egg-head question from me:  if a few damaged blocks are inside an 
> aggregate, my understanding is that the entire aggregate would be marked bad 
> during the audit, which means TSM wouldn't be able to move data 
> reconstruct=yes, which would cause a larger footprint of data loss.  Is my 
> hypothesis correct?
> 
> Hmm.  Now that I think about it, CRC would have to be enabled on the stgpool 
> to detect those few bad blocks within an aggregate, otherwise the 
> headers/magic numbers for the aggregate/blocks would still be readable/good 
> and the aggregate would audit as intact. Thoughts?
> 
> Another question:  do file volumes get magic numbers?  Haha  (Sorry, I blame 
> the vodka.)
> 
> 
> On 11/15/2012 12:58 PM, Tyree, David wrote:
>> This a hypothetical situation.
>> 
>> In this situation the needed tape from the copy pool is not available.
>> I realize that the data would be lost but how what you do next?
>> 
>> if we were still running v5 of TSM we would do a move data (MOVE VOL XXX) to 
>> save what we could then delete the volume (DEL VOL XXX). We would lose some 
>> data but the next backup cycle would rebackup any missing active data.
>> 
>> Since we are now running v6 with dedup it seems like the process would be 
>> different. Each volume no longer contains a complete set of files. They now 
>> contain parts of files.
>> 
>> 
>> David Tyree
>> Interface Analyst
>> South Georgia Medical Center
>> 229.333.1155
>> 
>> -Original Message-
>> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf 
>> Of Grigori Solonovitch
>> Sent: Thursday, November 15, 2012 2:58 PM
>> To: ADSM-L@VM.MARIST.EDU
>> Subject: Re: [ADSM-L] hypothetical situation with dedup turned on
>> 
>> Have you tried to use standard copy pol to recover any problems in primary 
>> pool?
>> 
>> Grigori G. Solonovitch
>> Senior Technical Architect  Ahli United Bank Kuwait  
>> www.ahliunited.com.kw
>> 
>> Please consider the environment before printing this E-mail
>> 
>> 
>> -Original Message-
>> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf 
>> Of Tyree, David
>> Sent: Wednesday, November 14, 2012 10:36 PM
>> To: ADSM-L@VM.MARIST.EDU
>> Subject: [ADSM-L] hypothetical situation with dedup turned on
>> 
>> I've had some sys admins ask me about a possible situation 
>> with using dedup on our primary storage pool. We are currently using dedup 
>> and I can't come up with a good answer.
>> Ok, our primary storage pool is using dedup. Something 
>> (corruption, whatever) happens to one of the files in the primary pool and 
>> the data needed to recover the file in the primary pool is not available.
>> I attempt to do a restore of the corrupt  file and the 
>> needed tape is not available.
>> How would I go about fi

Re: Re : Missing source data for Deduplication in 6.3.x

[Ray Carlson]

Nothing new.  I'm having various "Generate backupset" jobs fail.  Sometime my 
expire will fail, and even some of my reclamations.  IBM is attributing it all 
to the same missing data, and say that all my problem fall under IC85825.  
Supposedly, TSM 6.3.2.1 was supposed to fix this problem, and keep it from 
happening in the future.  I don't know it it didn't work or if the cleanup for 
what had previously happening didn't work.  Regardless, I am still having 
failures.  The bad part is that people may not even know they have the problem 
until they run a "Generate Backupset" or attempt to do a restore of data that 
is missing the source data from the identify dedupe.
Good luck with getting your problem fixed.  If enough people realize they have 
the problem and complain, maybe it will get fixed.
On Sep 17, 2012, at 7:08 AM, Tristan Kelkermans wrote:

> Hi,
> 
> We have the same issue here, anything new regarding this issue ?
> 
> Regards
> 
> Tristan

Re: Missing source data for Deduplication in 6.3.x

[Ray Carlson]

Unfortunately, this is being handled through a third company, so I don't have 
the number.

Sent from my iPhone


On Aug 31, 2012, at 10:09 AM, "Prather, Wanda"  wrote:

> Can you share the APAR number so I can read up on this issue?
> thanks
> W
> 
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Ray 
> Carlson
> Sent: Wednesday, August 29, 2012 10:05 AM
> To: ADSM-L@VM.MARIST.EDU
> Subject: [ADSM-L] Missing source data for Deduplication in 6.3.x
> 
> Is anyone else having the problem of TSM saying that it is missing the source 
> data for Deduplicated data?  This is causing some of our "Generate Backupset" 
> and some of our restore jobs to fail.  Unfortunately, unless you run some job 
> that actually attempts to use or restore from the missing data, you don't 
> even know that it is missing.  We have been working with IBM for several 
> weeks/months now and they say that version 6.3.2.01 will stop the loss of 
> future data, but we haven't been able to verify that as of yet.  We have 4 
> TSM servers, but this source data loss has only shown up on one of them so 
> far.  Thanks for any additional information anyone can provide.  Ray

Missing source data for Deduplication in 6.3.x

[Ray Carlson]

Is anyone else having the problem of TSM saying that it is missing the source 
data for Deduplicated data?  This is causing some of our "Generate Backupset" 
and some of our restore jobs to fail.  Unfortunately, unless you run some job 
that actually attempts to use or restore from the missing data, you don't even 
know that it is missing.  We have been working with IBM for several 
weeks/months now and they say that version 6.3.2.01 will stop the loss of 
future data, but we haven't been able to verify that as of yet.  We have 4 TSM 
servers, but this source data loss has only shown up on one of them so far.  
Thanks for any additional information anyone can provide.  Ray

Re: Damaged files in stgpool

[Ray Carlson]

You can try audit of the various volumes.
Also, if you are using Deduplication, there is a know problem of TSM losing the 
source data on versions prior to 6.3.2.01.
Ray

On Aug 27, 2012, at 9:56 AM, Dan Olson wrote:

> Hi all,
> 
> After a mass restore attempt pulling from 800+ volumes I've got a bunch of 
> files from various volumes showing up with a show damaged, 452 bad bitfiles, 
> of which a whole lot more files impacted.
> 
> Is there a way to tell TSM to retry the reads of the bitfiles?  When I try to 
> move data its not even trying to mount tapes anymore, I assume because there 
> is a bitfile marked bad on some dependent volume.
> 
> 
> Daniel Murphy-Olson
> Systems Administrator
> Mathematics & Computer Science Division
> Argonne National Laboratory
> 630-252-0055

Re: DATA Corruption using Deduplication in TSM 6.3.1.1 WARNING

[Ray Carlson]

Can anyone tell me what is in PMR 62476,033,000?  Apparently IBM believes that 
someone following the steps in that PMR caused the lose.  
Thanks,
Ray
On May 30, 2012, at 7:39 AM, Ray Carlson wrote:

> It was a brand new 2008 Server with a 6.3.0 install, which has been upgraded 
> to 6.3.1.1 because of other problems. Then the database was converted from a 
> 5.5 version.  
> It started with completely empty and new disk storage pools.  Then data was 
> moved from tape back to the disks.  Finally, Deduplication and Replication 
> were implemented.  It had been running for several months before the problem 
> arose.  I don't know if it is old data or fresh data since I haven't been 
> able to identify exactly which file is involved.  Unfortunately I do not have 
> the call reference number since that part is being handled by an outside 
> company.  I have also not had the opportunity attempt a restore using the DR 
> tapes since that would involve recalling multiple tapes from an offsite 
> company.  Basically, this is stopping some of my Generate backupsets and it 
> required that I do a disk to disk copy of the data I was trying to restore.  
> I still had the original data on the server and was simply using restore to 
> move it to a replacement server.  Thankfully, I haven't lost my original data 
> on the servers, just the ability to restore it from tsm if it was lost.
> Ray
> On May 30, 2012, at 4:09 AM, DeGroat, Steve wrote:
> 
>> Same here, we are about to go "live" on a new 6.3.1 server expecting client 
>> and server side deduplication.  Any additional information about the 
>> environment would be helpful. 
>> 
>> Steve DeGroat
>> Storage Manager
>> ITS Production Services
>> Yale University
>> 203.436.4540
>> 
>> "If you build it, they will come."
>> 
>> 
>> -Original Message-
>> From: ADSM: Dist Stor Manager [mailto:ADSM-L@vm.marist.edu] On Behalf Of 
>> Steven Langdale
>> Sent: Wednesday, May 30, 2012 3:24 AM
>> To: ADSM-L@vm.marist.edu
>> Subject: Re: [ADSM-L] DATA Corruption using Deduplication in TSM 6.3.1.1 
>> WARNING
>> 
>> Ray,  on top of what Remco posted, woudl you also post your call ref?
>> 
>> I'm about to implement a new 6.3 environ using dedupe so would like to press 
>> our IBM technical rep on the issue.  You never know, with a few people 
>> asking about it, it mya get fixed quicker.
>> 
>> Best of luck with your issue though!
>> 
>> Steven
>> 
>> On 30 May 2012 07:08, Remco Post  wrote:
>> 
>>> Hi Ray,
>>> 
>>> Thanks for the warning.
>>> 
>>> I was wondering if you could tell us a bit more about this TSM server. 
>>> Is it a converted 5.5 server? At which 6.x level did you start using 
>>> TSM version 6 for this server, 6.1, 6.2 or 6.3? Do the errors occur 
>>> with any particular kind of data? Is it 'old' data, or fresh data 
>>> recently written to TSM? Are you able to restore the files from 
>>> copypool if this error occurs?
>>> 
>>> On 29 mei 2012, at 20:46, Ray Carlson wrote:
>>> 
>>>> Or as IBM called it, "Orphaned deduplicate references".
>>>> 
>>>> We are running TSM 6.3.1.1 on a Windows 2008 Server, and using the
>>> Identify command to do deduplication on the Server, not the client.
>>>> 
>>>> Interestingly, everything seemed to be mostly working.  We had a few
>>> volumes that would not be reclaimed or moved because it said the 
>>> deduplicated data had not been backed up to the copy pool, but that 
>>> was jut an annoyance.
>>>> 
>>>> Then we discovered that we could not do restores of various servers.
>>> The error we got was:
>>>> "05/21/2012 20:52:45 ANRD_2547000324 bfRtrv(bfrtrv.c:1161)
>>> Thread<129>: Error  obtaining deduplication information for object
>>> 254560532 in super bitfile 664355697 in pool 7 (SESSION: 8235, PROCESS:
>>> 375)".
>>>> 
>>>> A Severity 1 trouble ticket was opened with IBM back on 5/21 and 
>>>> various
>>> information was gathered and provided to IBM.  So far IBM has not been 
>>> able to identify the root cause or provide a fix.  They have 
>>> transferred the ticket to the Development team.
>>>> 
>>>> So here I sit, not knowing which servers, if any, I could restore if
>>> needed.  Unfortunately, most operations appear to be fine and report 
>>> Success.  Only when I try to do a Generate Backupset, or do a Restore, 
>>> do I discover that there is a problem and the job fails.  Also, it 
>>> doesn't just skip the file/files that it can't restore and restore 
>>> everything else, it simply stops the restore and says it failed.
>>>> 
>>>> I'm wondering how many other people are in the same situation, but 
>>>> do
>>> not realize it.
>>>> 
>>>> BEWARE Deduplication
>>>> 
>>>> Ray Carlson
>>> 
>>> --
>>> Met vriendelijke groeten/Kind Regards,
>>> 
>>> Remco Post
>>> r.p...@plcs.nl
>>> +31 6 248 21 622
>>> 
> 

Re: DATA Corruption using Deduplication in TSM 6.3.1.1 WARNING

[Ray Carlson]

It was a brand new 2008 Server with a 6.3.0 install, which has been upgraded to 
6.3.1.1 because of other problems. Then the database was converted from a 5.5 
version.  
It started with completely empty and new disk storage pools.  Then data was 
moved from tape back to the disks.  Finally, Deduplication and Replication were 
implemented.  It had been running for several months before the problem arose.  
I don't know if it is old data or fresh data since I haven't been able to 
identify exactly which file is involved.  Unfortunately I do not have the call 
reference number since that part is being handled by an outside company.  I 
have also not had the opportunity attempt a restore using the DR tapes since 
that would involve recalling multiple tapes from an offsite company.  
Basically, this is stopping some of my Generate backupsets and it required that 
I do a disk to disk copy of the data I was trying to restore.  I still had the 
original data on the server and was simply using restore to move it to a 
replacement server.  Thankfully, I haven't lost my original data on the 
servers, just the ability to restore it from tsm if it was lost.
Ray
On May 30, 2012, at 4:09 AM, DeGroat, Steve wrote:

> Same here, we are about to go "live" on a new 6.3.1 server expecting client 
> and server side deduplication.  Any additional information about the 
> environment would be helpful. 
> 
> Steve DeGroat
> Storage Manager
> ITS Production Services
> Yale University
> 203.436.4540
> 
> "If you build it, they will come."
> 
> 
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@vm.marist.edu] On Behalf Of 
> Steven Langdale
> Sent: Wednesday, May 30, 2012 3:24 AM
> To: ADSM-L@vm.marist.edu
> Subject: Re: [ADSM-L] DATA Corruption using Deduplication in TSM 6.3.1.1 
> WARNING
> 
> Ray,  on top of what Remco posted, woudl you also post your call ref?
> 
> I'm about to implement a new 6.3 environ using dedupe so would like to press 
> our IBM technical rep on the issue.  You never know, with a few people asking 
> about it, it mya get fixed quicker.
> 
> Best of luck with your issue though!
> 
> Steven
> 
> On 30 May 2012 07:08, Remco Post  wrote:
> 
>> Hi Ray,
>> 
>> Thanks for the warning.
>> 
>> I was wondering if you could tell us a bit more about this TSM server. 
>> Is it a converted 5.5 server? At which 6.x level did you start using 
>> TSM version 6 for this server, 6.1, 6.2 or 6.3? Do the errors occur 
>> with any particular kind of data? Is it 'old' data, or fresh data 
>> recently written to TSM? Are you able to restore the files from 
>> copypool if this error occurs?
>> 
>> On 29 mei 2012, at 20:46, Ray Carlson wrote:
>> 
>>> Or as IBM called it, "Orphaned deduplicate references".
>>> 
>>> We are running TSM 6.3.1.1 on a Windows 2008 Server, and using the
>> Identify command to do deduplication on the Server, not the client.
>>> 
>>> Interestingly, everything seemed to be mostly working.  We had a few
>> volumes that would not be reclaimed or moved because it said the 
>> deduplicated data had not been backed up to the copy pool, but that 
>> was jut an annoyance.
>>> 
>>> Then we discovered that we could not do restores of various servers.
>> The error we got was:
>>> "05/21/2012 20:52:45 ANRD_2547000324 bfRtrv(bfrtrv.c:1161)
>> Thread<129>: Error  obtaining deduplication information for object
>> 254560532 in super bitfile 664355697 in pool 7 (SESSION: 8235, PROCESS:
>> 375)".
>>> 
>>> A Severity 1 trouble ticket was opened with IBM back on 5/21 and 
>>> various
>> information was gathered and provided to IBM.  So far IBM has not been 
>> able to identify the root cause or provide a fix.  They have 
>> transferred the ticket to the Development team.
>>> 
>>> So here I sit, not knowing which servers, if any, I could restore if
>> needed.  Unfortunately, most operations appear to be fine and report 
>> Success.  Only when I try to do a Generate Backupset, or do a Restore, 
>> do I discover that there is a problem and the job fails.  Also, it 
>> doesn't just skip the file/files that it can't restore and restore 
>> everything else, it simply stops the restore and says it failed.
>>> 
>>> I'm wondering how many other people are in the same situation, but 
>>> do
>> not realize it.
>>> 
>>> BEWARE Deduplication
>>> 
>>> Ray Carlson
>> 
>> --
>> Met vriendelijke groeten/Kind Regards,
>> 
>> Remco Post
>> r.p...@plcs.nl
>> +31 6 248 21 622
>> 

DATA Corruption using Deduplication in TSM 6.3.1.1 WARNING

[Ray Carlson]

Or as IBM called it, "Orphaned deduplicate references".  

We are running TSM 6.3.1.1 on a Windows 2008 Server, and using the Identify 
command to do deduplication on the Server, not the client.

Interestingly, everything seemed to be mostly working.  We had a few volumes 
that would not be reclaimed or moved because it said the deduplicated data had 
not been backed up to the copy pool, but that was jut an annoyance.  

Then we discovered that we could not do restores of various servers.  The error 
we got was: 
"05/21/2012 20:52:45 ANRD_2547000324 bfRtrv(bfrtrv.c:1161) Thread<129>: 
Error  obtaining deduplication information for object 254560532 in super 
bitfile 664355697 in pool 7 (SESSION: 8235, PROCESS: 375)".

A Severity 1 trouble ticket was opened with IBM back on 5/21 and various 
information was gathered and provided to IBM.  So far IBM has not been able to 
identify the root cause or provide a fix.  They have transferred the ticket to 
the Development team.

So here I sit, not knowing which servers, if any, I could restore if needed.  
Unfortunately, most operations appear to be fine and report Success.  Only when 
I try to do a Generate Backupset, or do a Restore, do I discover that there is 
a problem and the job fails.  Also, it doesn't just skip the file/files that it 
can't restore and restore everything else, it simply stops the restore and says 
it failed.

I'm wondering how many other people are in the same situation, but do not 
realize it.  

BEWARE Deduplication 

Ray Carlson

Re: replication issue tsm 6.3.0

[Ray Carlson]

Tim,

I have been fighting this problem for several months now.  IBM is looking at 
it, but haven't come up with a solution.

Ray

On May 2, 2012, at 10:39 AM, Tim Brown wrote:

> Error during nodegroup replication, but I cant see any errors on the target 
> server
> 
> 
> 
>   ANR1815W Check target server for storage problems during
> 
>  replication of node MINERVA, PERSEUS, SILVANUS, THESEUS,
> 
>  POKUCMS2, POKXPR1, POKSESAP1, POKOSVMS1, POKXPRSAT2,
> 
>  CRONUS, POKVERINTDB, POKVERINTREC, POKVERINTARC.
> 
>  (SESSION: 51623, PROCESS: 709)
> 
> 
> 
> Rerun of replicate works
> 
> 
> 
> Thanks,
> 
> 
> 
> Tim Brown
> Supervisor Computer Operations
> 
> Central Hudson Gas & Electric
> 284 South Ave
> Poughkeepsie, NY 12601
> Email:   tbr...@cenhud.com << 
>  mailto:tbr...@cenhud.com>>
> Phone: 845-486-5643
> Fax: 845-486-5921
> Cell: 845-235-4255
> 
> 
> 
> 
> "This message contains confidential information and is only for the intended 
> recipient. If the reader of this message is not the intended recipient, or an 
> employee or agent responsible for delivering this message to the intended 
> recipient, please notify the sender immediately by replying to this note and 
> deleting all copies and attachments."

Re: TSM Export issue unavailable tapes causing the exports to suspend themselves

[Ray Carlson]

I had a similar problem, tape changing to unavailable, a few months back.  In 
my case, it was a problem with how the tape was identified.  The Library, an 
XLS, had it as "123456".  TMS has "123456" as a scratch tape and also had tape 
"123456L4" as the tape I wanted.  I had to check "123456" out of TMS and then 
check it back in as "123456L4".  This may not be the cause of your problem, but 
it is something to check.
Ray

On Mar 7, 2012, at 8:32 AM, Hughes, Timothy wrote:

> Hi 
> 
> Thanks Daniel..I will also check these a and/or b could be a issue
> 
> Regards
> 
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of 
> Daniel Sparrman
> Sent: Tuesday, March 06, 2012 5:06 PM
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: TSM Export issue unavailable tapes causing the exports to 
> suspend themselves
> 
> Hi
> 
> I'm guessing you've already checked 
> 
> a) Element numbers (they CAN change)
> 
> b) That your drives hasnt gone over-due on cleaning
> 
> c) That the tapes has been usable before
> 
> d) That your library device is available (if not, it's gonna put your volumes 
> in an unavailable state)
> 
> Not questioning your competence, but have you tried running a audit library?
> 
> Most of the issues with TSM & tape volumes usually adhere from configuration 
> issues. You either had it before, or you got it now. If you got it now, a 
> configuration change has happened. If you had it before, I'm sure it's one of 
> the above.
> 
> If it's nothing of the above, describe your problem more in detail and I 
> might be able to help you.
> 
> Regards
> 
> Daniel Sparrman
> 
> 
> Daniel Sparrman
> Exist i Stockholm AB
> Växel: 08-754 98 00
> Fax: 08-754 97 30
> daniel.sparr...@exist.se
> http://www.existgruppen.se
> Posthusgatan 1 761 30 NORRTÄLJE
> 
> 
> -"ADSM: Dist Stor Manager"  skrev: - 
> Till: ADSM-L@VM.MARIST.EDU
> Från: "Hughes, Timothy" 
> Sänt av: "ADSM: Dist Stor Manager" 
> Datum: 03/06/2012 21:03
> Ärende: Re: TSM Export issue unavailable tapes causing the exports to suspend 
> themselves
> 
> 
> Richard thanks,
> 
> When the volumes are updated to access=readwrite the commands seems to work 
> and in the error it doesn't show much. It seems the tapes go offline fairly 
> quickly, As soon as the request for a mount of the tape is made, At least 
> that's the way it seems to me.
> 
> Thanks again
> 
> 
> 03/06/12   14:35:23  ANR1402W Mount request denied for volume C07498L3 - 
> volume
>  unavailable. (SESSION: 113956, PROCESS: 5603)
> 03/06/12   14:35:23  ANR1420W Read access denied for volume C07498L3 - 
> volume
>  access mode = "unavailable". (SESSION: 113956, 
> PROCESS:
>  5603)
> 03/06/12   14:35:38  ANR1402W Mount request denied for volume C23919L3 - 
> volume
>  unavailable. (SESSION: 113956, PROCESS: 5604)
> 03/06/12   14:35:38  ANR1420W Read access denied for volume C23919L3 - 
> volume
>  access mode = "unavailable". (SESSION: 113956, 
> PROCESS:
>  5604)
> 03/06/12   14:35:46  ANR1402W Mount request denied for volume C23727L3 - 
> volume
>  unavailable. (SESSION: 113956, PROCESS: 5605)
> 03/06/12   14:35:46  ANR1420W Read access denied for volume C23727L3 - 
> volume
>  access mode = "unavailable". (SESSION: 113956, 
> PROCESS:
>  5605)
> 03/06/12   14:36:43  ANR2017I Administrator ADMIN1 issued command: QUERY
>  ACTLOG search=denied  (SESSION: 113956)
> 
> tsm >q actlog search=update
> 
> Date/TimeMessage
>  
> --
> 03/06/12   14:27:22  ANR2017I Administrator ADMIN1 issued command: UPDATE
>  VOLUME c23727l3 access=readwrite  (SESSION: 113956)
> 03/06/12   14:27:22  ANR2207I Volume C23727L3 updated. (SESSION: 113956)
> 03/06/12   14:33:23  ANR2017I Administrator ADMIN1 issued command: UPDATE
>  VOLUME c23919l3 access=readwrite  (SESSION: 113956)
> 03/06/12   14:33:23  ANR2207I Volume C23919L3 updated. (SESSION: 113956)
> 03/06/12   14:33:40  ANR2017I Administrator ADMIN1 issued command: UPDATE
>  VOLUME c07498l3 access=readwrite  (SESSION: 113956)
> 03/06/12   14:33:40  ANR2207I Volume C07498L3 updated. (SESSION: 113956)
> 03/06/12   14:33:42  ANR2017I Administrator ADMIN1 issued command: UPDATE
>  VOLUME c07498l3 access=readwrite  (SESSION: 113956)
> 03/06/12   14:33:42  ANR2207I Volume C07498L3 updated. (SESSION: 113956)
> 03/06/12   14:35:01  ANR2017I Administrator ADMIN1 issued command: UPDATE
>  VOLUME c23727l3 access=readwrite  (SESSION: 113956)
> 03/06/12   14:35:01  ANR2207I Volume C23727L3 updated. (SESSION

Re: Deployment Engine Failed to initialize

[Ray Carlson]

I have a few Solaris 9 5.5.3 clients and some Linux 5.5.1 clients working with 
a 6.3.0 server.  their just not supported.
Be aware that NODE REPLICATION of a 6.3.0 Client to a 6.3.0 Server will 
sometimes fail because of wanting data from your offsite DR tapes.  IBM is 
supposedly working on a fix, but does not have an time estimate.  Meanwhile, 
there is no known workaround.
Ray

On Feb 28, 2012, at 2:55 PM, Zoltan Forray/AC/VCU wrote:

> I keep wondering if this is FACT or RECOMMENDATION.
> 
> Based on this document:
> https://www-304.ibm.com/support/docview.wss?uid=swg21053218  it
> says you can not use the 6.3 CLIENT with a 5.5 SERVER.   I documented this
> and some user didn't pay attention and installed the 6.3 client on a node
> still on a 5.5 server.  So far, there haven't been any issues and it works
> just fine.
> 
> I am about to convert a 5.5 server to 6.x and was planning to jump
> straight to 6.3.  However, there a few nodes still running 5.4 and LOWER
> level clients (down to 5.1 for an IRIX system).
> 
> Anyone try a 5.x client with a 6.3 server?
> 
> 
> Zoltan Forray
> TSM Software & Hardware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> zfor...@vcu.edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit http://infosecurity.vcu.edu/phishing.html
> 
> 
> 
> From:   "Stackwick, Stephen" 
> To: ADSM-L@VM.MARIST.EDU
> Date:   02/28/2012 03:28 PM
> Subject:Re: [ADSM-L] Deployment Engine Failed to initialize
> Sent by:"ADSM: Dist Stor Manager" 
> 
> 
> 
> The problem with 6.3 is all your clients need to be at V6. If that's not
> an issue, I might even wait a little bit longer for the first fixes which
> Tivoli claims will be this quarter.
> 
> Steve
> 
> STEPHEN STACKWICK | Senior Consultant | 301.518.6352 (m) |
> sstackw...@icfi.com | icfi.com
> ICF INTERNATIONAL | 410 E. Pratt Street Suite 2214, Baltimore, MD 21202 |
> 410.539.1135 (o)
> 
> 
> 
> From: ADSM: Dist Stor Manager [ADSM-L@VM.MARIST.EDU] on behalf of
> Sheridan, Peter T. [peter.sheri...@cunamutual.com]
> Sent: Tuesday, February 28, 2012 11:27
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: [ADSM-L] Deployment Engine Failed to initialize
> 
> Would people recommend going with 6.2.3 or 6.3 ?
> 
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@vm.marist.edu] On Behalf Of
> Colwell, William F.
> Sent: Tuesday, February 28, 2012 10:22 AM
> To: ADSM-L@vm.marist.edu
> Subject: Re: [ADSM-L] Deployment Engine Failed to initialize
> 
> I agree with Zoltan.  I have 2 very large instances at 6.1.5.10 in
> production
> 
> doing large amounts of dedup processing.  I am aware of the reorg issues
> but it
> 
> doesn't bother me, I am not interested in reorging the tables.  In any
> case
> 
> 6.3 doesn't solve all the reorg issues, see apar ic81261 and flash
> 1580639.
> 
> 
> 
> Thanks,
> 
> 
> 
> Bill Colwell
> 
> Draper Lab
> 
> 
> 
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of
> Zoltan Forray/AC/VCU
> Sent: Tuesday, February 28, 2012 9:39 AM
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: Deployment Engine Failed to initialize
> 
> 
> 
> WOW - such harsh words about 6.1 !   I don't agree..my main
> production
> 
> 6.x system is 6.1.5.10 with no issues.  At least it hasn't had this
> wacky,
> 
> problem my other 6.2.x servers have had with a DB backup randomly,
> 
> intermittently failing with no discernible reason(note, there are
> docs
> 
> that say you really need to be at least at 6.1.4.1 to resolve some big
> 
> problems, especially with reorgs)
> 
> 
> 
> 
> 
> Zoltan Forray
> 
> TSM Software & Hardware Administrator
> 
> Virginia Commonwealth University
> 
> UCC/Office of Technology Services
> 
> zfor...@vcu.edu - 804-828-4807
> 
> Don't be a phishing victim - VCU and other reputable organizations will
> 
> never use email to request that you reply with your password, social
> 
> security number or confidential personal information. For more details
> 
> visit http://infosecurity.vcu.edu/phishing.html
> 
> 
> 
> 
> 
> 
> 
> From:   "Prather, Wanda" 
> 
> To: ADSM-L@VM.MARIST.EDU
> 
> Date:   02/28/2012 05:57 AM
> 
> Subject:Re: [ADSM-L] Deployment Engine Failed to initialize
> 
> Sent by:"ADSM: Dist Stor Manager" 
> 
> 
> 
> 
> 
> 
> 
> What Remco said.
> 
> Nothing Good will Happen on 6.1.
> 
> I finally got a production system stable on 6.1.3 by disabling reorgs,
> but
> 
> that was Windows.
> 
> I wouldn't even think of doing it on Linux.
> 
> 
> 
> W
> 
> 
> 
> -Original Message-
> 
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of
> 
> Remco Post
> 
> Sent: Monday, February 27, 2012 5:10 PM
> 
> To: ADSM-L@VM.

Re: Mixed LTO3 and LTO5 tape drives

[Ray Carlson]

Not sure if it is related, but we could not use LTO5 media in our LTO5 drives 
until we updated to version 5.5.5.0.
Ray

On Oct 26, 2010, at 7:14 PM, Rodney Luk wrote:

> We replaced 4 LTO3 tape drives with new LTO5 drives in our 3584 library. The 
> library has 4 LTO3 and 4 LTO5 tape drives now. A new device class has created 
> for the LTO5 drives. A new primary storage pool has created and bind to the 
> new LTO5 device class. I am able to  migrate the data from the disk pools to 
> the new LTO5 storage pool but I couldn't move data from the LTO3 primary tape 
> storage pool to the new LTO5 tape storage.
> 
> The TSM server is at 5.5.4.0 on a Windows Server 2003 machine. The tape 
> driver has update to the latest version 6212. I have recreated all the drives 
> and paths.
> 
> Appreciated for any help,
> Rodney
> 
> Below is the activity logs
> 
> 10/26/2010 3:28:23 PM ANR1420W Read access denied for volume T8L5 - 
> volume access mode = "unavailable".
> 10/26/2010 3:28:23 PM ANR0565W Retrieve or restore failed for session 17470 
> for node Sever01 (WinNT). The storage volume T8L5 is inaccessible.
> 10/26/2010 4:37:38 PM ANR1258W Files on volume A01504L3 needed for move data 
> cannot be accessed - access mode is "unavailable", "offsite", or "destroyed".
> 10/26/2010 4:40:59 PM ANR8311E An I/O error occurred while accessing drive 
> MT1.0.0.7 (\\.\Tape6) for GET_TAPE_DRIVE_INFORMATION operation, errno = 121, 
> rc = 1.
> 10/26/2010 4:41:03 PM ANR8311E An I/O error occurred while accessing drive 
> MT0.0.0.7 (\\.\Tape7) for GET_TAPE_DRIVE_INFORMATION operation, errno = 121, 
> rc = 1.
> 10/26/2010 4:41:18 PM ANR1401W Mount request denied for volume A01141L3 - 
> mount failed.
> 10/26/2010 4:41:21 PM ANR1404W Scratch volume mount request denied - mount 
> failed.
> 10/26/2010 4:41:21 PM ANR8300E I/O error on library LB0.1.0.4 (OP=8401C058, 
> CC=314, KEY=FF, ASC=FF, ASCQ=FF,~SENSE=**NONE**,~Description=The source slot 
> or drive was empty in an attempt to move a volume).  Refer to Appendix C in 
> the 'Messages' manual for recommended action.
> 10/26/2010 4:41:21 PM ANR8312E Volume T8L5 could not be located in 
> library LB0.1.0.4.
> 10/26/2010 4:41:21 PM ANR8358E Audit operation is required for library 
> LB0.1.0.4.
> 10/26/2010 4:41:21 PM ANR8381E LTO volume T8L5 could not be mounted in 
> drive MT1.0.0.6 (\\.\Tape5).
> 10/26/2010 4:41:21 PM ANR1402W Mount request denied for volume T8L5 - 
> volume unavailable.
> 10/26/2010 4:41:21 PM ANR1410W Access mode for volume T8L5 now set to 
> "unavailable".
> 10/26/2010 4:44:37 PM ANR8311E An I/O error occurred while accessing drive 
> MT0.0.0.7 (\\.\Tape7) for GET_TAPE_DRIVE_INFORMATION operation, errno = 121, 
> rc = 1.
> 10/26/2010 4:45:08 PM ANR1404W Scratch volume mount request denied - mount 
> failed.
> 10/26/2010 4:48:25 PM ANR8311E An I/O error occurred while accessing drive 
> MT1.0.0.7 (\\.\Tape6) for GET_TAPE_DRIVE_INFORMATION operation, errno = 121, 
> rc = 1.
> 10/26/2010 4:48:43 PM ANR1404W Scratch volume mount request denied - mount 
> failed.
> 10/26/2010 4:52:02 PM ANR8311E An I/O error occurred while accessing drive 
> MT0.0.0.7 (\\.\Tape7) for GET_TAPE_DRIVE_INFORMATION operation, errno = 121, 
> rc = 1.
> 10/26/2010 4:52:20 PM ANR1401W Mount request denied for volume A01716L3 - 
> mount failed.
> 10/26/2010 4:52:20 PM ANR1144W Move data process terminated for volume 
> A01716L3 - storage media inaccessible.
> 
> Below is the information about the drives and path
> 
> 5:05:33 PM   SERVER01 : q drive f=d
>Library Name: LB0.1.0.4
>  Drive Name: MT0.0.0.3
> Device Type: LTO
> On-Line: Yes
>Read Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULT-
>   RIUM2,ULTRIUMC,ULTRIUM
>   Write Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULT-
>   RIUM2
> Element: 257
> Drive State: EMPTY
> Volume Name:
>Allocated to:
> WWN:
>   Serial Number: 1210011646
>  Last Update by (administrator): USER
>   Last Update Date/Time: 10/25/2010 10:00:11
> Cleaning Frequency (Gigabytes/ASNEEDED/NONE): NONE
>Library Name: LB0.1.0.4
>  Drive Name: MT0.0.0.4
> Device Type: LTO
> On-Line: Yes
>Read Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULT-
>   RIUM2,ULTRIUMC,ULTRIUM
>   Write Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULT-
>

Re: Disabling automatic mount processes

[Ray Carlson]

For a Library to do its own audit, then yes you want to take the drives 
offline.  However, you will need one drive online and empty when you want TSM 
to synchronize its Library audit with the physical library audit.

Ray C

On Sep 26, 2010, at 10:23 AM, Huebner,Andy,FORT WORTH,IT wrote:

> We take the drives off-line during maintenance.  We have restarted real and 
> imagined libraries with no ill effects to TSM by simply taking the drives 
> off-line.  Off-line drives should not affect an audit.
> 
> Andy Huebner
> 
> 
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ads...@vm.marist.edu] On Behalf Of 
> Skylar Thompson
> Sent: Saturday, September 25, 2010 6:42 PM
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: [ADSM-L] Disabling automatic mount processes
> 
> Does disabling the library prevent library audits as well? Basically I'm
> trying to keep stuff from getting mounted and interfering with the audit.
> 
> On 09/25/2010 02:34 PM, Richard Sims wrote:
>> On Sep 25, 2010, at 5:06 PM, Skylar Thompson wrote:
>> 
>>> Does anyone know of a way to temporarily keep TSM from starting
>>> automated processes that involve library mounts? I'd like to be able to
>>> keep migrations/reclamations/database backups/whatever from occurring
>>> when trying to diagnose tape library problems.
>> 
>> Depending upon the library, taking it offline will certainly prevent TSM 
>> access.  Within TSM you can adjust thresholds and change Type Admin 
>> schedules to Active=No for the duration, but there will be urgent needs such 
>> as DBBackuptrigger which require some kind of attention.
>> 
>>   Richard Sims
> 
> --
> -- Skylar Thompson (skyl...@u.washington.edu)
> -- Genome Sciences Department, System Administrator
> -- Foege Building S048, (206)-685-7354
> -- University of Washington School of Medicine
> 
> This e-mail (including any attachments) is confidential and may be legally 
> privileged. If you are not an intended recipient or an authorized 
> representative of an intended recipient, you are prohibited from using, 
> copying or distributing the information in this e-mail or its attachments. If 
> you have received this e-mail in error, please notify the sender immediately 
> by return e-mail and delete all copies of this message and any attachments.
> 
> Thank you.