Re: Another VE mystery - restoring from tape - but shouldn't be
Richard, Here is the Q OCC result. The Q OCC shows all the data for FSID 86 is in VMCTLMC (seq disk), SLOWDEDUP (seq disk) or the COPYPOOL (tape COPY pool). But when the customer tries a restore, the tapes getting mounted are from primary pool TAPEPOOL2. How can a restore be calling for tapes in a pool where the filespace has no data according to Q OCC? I suggested they open a sev 1 with Tivoli. This can't be right. tsm: HCG-TSM-SERVERq occ dc1 86 nametype=fsid Node Name Type Filespace FSID StorageNumber of Physical Logical Name Pool Name Files Space Space Occupied Occupied (MB) (MB) -- -- - -- - - - DC1Bkup \VMFULL-H-86 COPYPOOL 8,928 99,057.72 99,062.73 WDDMZOCU- LARX1 DC1Bkup \VMFULL-H-86 SLOWDEDUP 4,464 - 98,733.95 WDDMZOCU- LARX1 DC1Bkup \VMFULL-H-86 VMCTLMC4,464 315.14315.14 WDDMZOCU- LARX1 -Original Message- From: Prather, Wanda Sent: Wednesday, June 19, 2013 11:03 AM To: Richard Cowen Subject: RE: Another VE mystery - restoring from tape - but shouldn't be Did the MOVE NODEDATA result in a zillion new volumes (.BFS's) ? No, we don't use scratch volumes in the seq disk pool Can you get the activity log for the time the process(es) ran ? Any chance you have query occupancy node=dcname filespace=victim1 stgpool=tape,disk before and after move? If not, does the query for tape pool now show zero? Don't have them and right now I don't have access, but those are great ideas, will ask the customer for them. Thanks! Wanda -Original Message- From: Prather, Wanda [mailto:wanda.prat...@icfi.com] Sent: Tuesday, June 18, 2013 11:35 AM To: Richard Cowen Subject: RE: Another VE mystery - restoring from tape - but shouldn't be Hi Richard, Did you get a zillion tape mounts during the MOVE NODEDATA? Yes Do you know it finished without errors? Yes. And we ran another MOVE NODEDATA to verify there was no more data to move for that filespace. How, exactly, did the data go from sequential fast disk - sequential slow disk - tape ? Ordinary migration, at different times, as the pools hit migration thresholds. I didn't think TSM would migrate more than one level, so maybe the last step was using a MOVE command? Yes, your migration hierarchy can have as many levels as you want, as long as you don't try to go from a sequential pool back to a disk(random) pool. What does a QUERY NODEDATA show for primary pools and copy pools? Would not be informative, as we only moved some of the filespaces, not all of them. Are you running aggressive reclamation on the tape pool? No, and it's not collocated. Which is why we decided to move the filespace back to the seq disk pool. I don't think it's odd the data was too spread out to make the restore from tape not work well. What is odd is that all the data from one VM is supposedly in one filespace. We moved that filespace back to disk with move nodedata fsid= But still getting tape mounts on the restore. Thanks for your interest! Wanda -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Monday, June 17, 2013 9:45 PM To: ADSM-L@VM.MARIST.EDUmailto:ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Another VE mystery - restoring from tape - but shouldn't be TSM 6.2.3 Backup done with TSM VE 6.4.0.0 and TSM client 6.4.0.0. VMCTLMC points to a dedicated sequential pool on fast disk. That pool has no NEXTSTGPOOL defined. VMMC points backup data to a deduplicated sequential pool on fast disk storage. After the server dedups it there, it migrates to a slower NAS-based deduplicated storage pool on disk. When that slower pool fills, data migrates out to tape. DEDUPREQUIRESBACKUP set to YES. (No client-side dedup used.) We have done full VM restores through the plug-in and file-level restores through the recovery agent in the past, including testing restores from tape, with no problems. Now one of the customer's VM datastores has met with an unfortunate accident in a dark alley. We need to restore 7 full VM's. From the dsmc command line, restore vm victim1 datastore=newhealthyone starts up OK, but was calling for a zillion tape mounts, and therefore was restoring at the rate of about 4GB per 24 hours. So, we did MOVE NODEDATA DCNAME FILESPACE=victim1 to bring the data from the tape back to the sequential
Re: Another VE mystery - restoring from tape - but shouldn't be
Hi Wanda, Identify one of the tapes being mounted from TAPEPOOL2, then do a QUERY CONTENT on it. See if there is anything for DC1. There has to be *something* even if we haven't yet put our finger on it. Best regards, - Andy Andrew Raibeck | Tivoli Storage Manager Level 3 Technical Lead | stor...@us.ibm.com IBM Tivoli Storage Manager links: Product support: http://www.ibm.com/support/entry/portal/Overview/Software/Tivoli/Tivoli_Storage_Manager Online documentation: https://www.ibm.com/developerworks/mydeveloperworks/wikis/home/wiki/Tivoli Documentation Central/page/Tivoli Storage Manager Product Wiki: https://www.ibm.com/developerworks/mydeveloperworks/wikis/home/wiki/Tivoli +Storage+Manager/page/Home ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 2013-06-19 11:53:07: From: Prather, Wanda wanda.prat...@icfi.com To: ADSM-L@vm.marist.edu, Date: 2013-06-19 11:55 Subject: Re: Another VE mystery - restoring from tape - but shouldn't be Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu Richard, Here is the Q OCC result. The Q OCC shows all the data for FSID 86 is in VMCTLMC (seq disk), SLOWDEDUP (seq disk) or the COPYPOOL (tape COPY pool). But when the customer tries a restore, the tapes getting mounted are from primary pool TAPEPOOL2. How can a restore be calling for tapes in a pool where the filespace has no data according to Q OCC? I suggested they open a sev 1 with Tivoli. This can't be right. tsm: HCG-TSM-SERVERq occ dc1 86 nametype=fsid Node Name Type Filespace FSID Storage Number of Physical Logical Name Pool Name Files Space Space Occupied Occupied (MB) (MB) -- -- - -- - - - DC1Bkup \VMFULL-H-86 COPYPOOL 8,928 99,057.72 99,062.73 WDDMZOCU- LARX1 DC1Bkup \VMFULL-H-86 SLOWDEDUP 4,464 - 98,733.95 WDDMZOCU- LARX1 DC1Bkup \VMFULL-H-86 VMCTLMC 4,464315.14315.14 WDDMZOCU- LARX1 -Original Message- From: Prather, Wanda Sent: Wednesday, June 19, 2013 11:03 AM To: Richard Cowen Subject: RE: Another VE mystery - restoring from tape - but shouldn't be Did the MOVE NODEDATA result in a zillion new volumes (.BFS's) ? No, we don't use scratch volumes in the seq disk pool Can you get the activity log for the time the process(es) ran ? Any chance you have query occupancy node=dcname filespace=victim1 stgpool=tape,disk before and after move? If not, does the query for tape pool now show zero? Don't have them and right now I don't have access, but those are great ideas, will ask the customer for them. Thanks! Wanda -Original Message- From: Prather, Wanda [mailto:wanda.prat...@icfi.com] Sent: Tuesday, June 18, 2013 11:35 AM To: Richard Cowen Subject: RE: Another VE mystery - restoring from tape - but shouldn't be Hi Richard, Did you get a zillion tape mounts during the MOVE NODEDATA? Yes Do you know it finished without errors? Yes. And we ran another MOVE NODEDATA to verify there was no more data to move for that filespace. How, exactly, did the data go from sequential fast disk - sequential slow disk - tape ? Ordinary migration, at different times, as the pools hit migration thresholds. I didn't think TSM would migrate more than one level, so maybe the last step was using a MOVE command? Yes, your migration hierarchy can have as many levels as you want, as long as you don't try to go from a sequential pool back to a disk (random) pool. What does a QUERY NODEDATA show for primary pools and copy pools? Would not be informative, as we only moved some of the filespaces, not all of them. Are you running aggressive reclamation on the tape pool? No, and it's not collocated. Which is why we decided to move the filespace back to the seq disk pool. I don't think it's odd the data was too spread out to make the restore from tape not work well. What is odd is that all the data from one VM is supposedly in one filespace. We moved that filespace back to disk with move nodedata fsid= But still getting tape mounts on the restore. Thanks for your interest! Wanda -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Monday, June 17, 2013 9:45 PM To: ADSM-L@VM.MARIST.EDUmailto:ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Another VE mystery - restoring from tape - but shouldn't be TSM 6.2.3 Backup done with TSM VE 6.4.0.0 and TSM client 6.4.0.0. VMCTLMC points to a dedicated sequential pool on fast
Re: Another VE mystery - restoring from tape - but shouldn't be
To go along with Andy's recommendations I've seen in TSM server 6.2.3.0 where a query occupancy and a select from occupancy don't match up. Might be worthwhile to do a: select * from occupancy where node_name='DC1' and filespace='\VMFULL-HWDDMZOCULARX1' See how they compare -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Andrew Raibeck Sent: Wednesday, June 19, 2013 12:20 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Another VE mystery - restoring from tape - but shouldn't be Hi Wanda, Identify one of the tapes being mounted from TAPEPOOL2, then do a QUERY CONTENT on it. See if there is anything for DC1. There has to be *something* even if we haven't yet put our finger on it. Best regards, - Andy Andrew Raibeck | Tivoli Storage Manager Level 3 Technical Lead | stor...@us.ibm.com IBM Tivoli Storage Manager links: Product support: http://www.ibm.com/support/entry/portal/Overview/Software/Tivoli/Tivoli_Stor age_Manager Online documentation: https://www.ibm.com/developerworks/mydeveloperworks/wikis/home/wiki/Tivoli Documentation Central/page/Tivoli Storage Manager Product Wiki: https://www.ibm.com/developerworks/mydeveloperworks/wikis/home/wiki/Tivoli +Storage+Manager/page/Home ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 2013-06-19 11:53:07: From: Prather, Wanda wanda.prat...@icfi.com To: ADSM-L@vm.marist.edu, Date: 2013-06-19 11:55 Subject: Re: Another VE mystery - restoring from tape - but shouldn't be Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu Richard, Here is the Q OCC result. The Q OCC shows all the data for FSID 86 is in VMCTLMC (seq disk), SLOWDEDUP (seq disk) or the COPYPOOL (tape COPY pool). But when the customer tries a restore, the tapes getting mounted are from primary pool TAPEPOOL2. How can a restore be calling for tapes in a pool where the filespace has no data according to Q OCC? I suggested they open a sev 1 with Tivoli. This can't be right. tsm: HCG-TSM-SERVERq occ dc1 86 nametype=fsid Node Name Type Filespace FSID Storage Number of Physical Logical Name Pool Name Files Space Space Occupied Occupied (MB) (MB) -- -- - -- - - - DC1Bkup \VMFULL-H-86 COPYPOOL 8,928 99,057.72 99,062.73 WDDMZOCU- LARX1 DC1Bkup \VMFULL-H-86 SLOWDEDUP 4,464 - 98,733.95 WDDMZOCU- LARX1 DC1Bkup \VMFULL-H-86 VMCTLMC 4,464315.14315.14 WDDMZOCU- LARX1 -Original Message- From: Prather, Wanda Sent: Wednesday, June 19, 2013 11:03 AM To: Richard Cowen Subject: RE: Another VE mystery - restoring from tape - but shouldn't be Did the MOVE NODEDATA result in a zillion new volumes (.BFS's) ? No, we don't use scratch volumes in the seq disk pool Can you get the activity log for the time the process(es) ran ? Any chance you have query occupancy node=dcname filespace=victim1 stgpool=tape,disk before and after move? If not, does the query for tape pool now show zero? Don't have them and right now I don't have access, but those are great ideas, will ask the customer for them. Thanks! Wanda -Original Message- From: Prather, Wanda [mailto:wanda.prat...@icfi.com] Sent: Tuesday, June 18, 2013 11:35 AM To: Richard Cowen Subject: RE: Another VE mystery - restoring from tape - but shouldn't be Hi Richard, Did you get a zillion tape mounts during the MOVE NODEDATA? Yes Do you know it finished without errors? Yes. And we ran another MOVE NODEDATA to verify there was no more data to move for that filespace. How, exactly, did the data go from sequential fast disk - sequential slow disk - tape ? Ordinary migration, at different times, as the pools hit migration thresholds. I didn't think TSM would migrate more than one level, so maybe the last step was using a MOVE command? Yes, your migration hierarchy can have as many levels as you want, as long as you don't try to go from a sequential pool back to a disk (random) pool. What does a QUERY NODEDATA show for primary pools and copy pools? Would not be informative, as we only moved some of the filespaces, not all of them. Are you running aggressive reclamation on the tape pool? No, and it's not collocated. Which is why we decided to move the filespace back to the seq disk pool. I don't think it's odd the data was too spread out to make the restore from tape not work well. What is odd is that all the data from one VM is supposedly in one filespace. We moved that
Re: Another VE mystery - restoring from tape - but shouldn't be
Out of curiosity, had the data been DeDuped when it was on the Disk, before it was migrated to tape, and then moved back to disk? We saw many problems with data corruption and damaged files when this was done with a TSM server prior to 6.3.3. Ray On Jun 19, 2013, at 11:20 AM, Andrew Raibeck stor...@us.ibm.com wrote: Hi Wanda, Identify one of the tapes being mounted from TAPEPOOL2, then do a QUERY CONTENT on it. See if there is anything for DC1. There has to be *something* even if we haven't yet put our finger on it. Best regards, - Andy Andrew Raibeck | Tivoli Storage Manager Level 3 Technical Lead | stor...@us.ibm.com IBM Tivoli Storage Manager links: Product support: http://www.ibm.com/support/entry/portal/Overview/Software/Tivoli/Tivoli_Storage_Manager Online documentation: https://www.ibm.com/developerworks/mydeveloperworks/wikis/home/wiki/Tivoli Documentation Central/page/Tivoli Storage Manager Product Wiki: https://www.ibm.com/developerworks/mydeveloperworks/wikis/home/wiki/Tivoli +Storage+Manager/page/Home ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 2013-06-19 11:53:07: From: Prather, Wanda wanda.prat...@icfi.com To: ADSM-L@vm.marist.edu, Date: 2013-06-19 11:55 Subject: Re: Another VE mystery - restoring from tape - but shouldn't be Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu Richard, Here is the Q OCC result. The Q OCC shows all the data for FSID 86 is in VMCTLMC (seq disk), SLOWDEDUP (seq disk) or the COPYPOOL (tape COPY pool). But when the customer tries a restore, the tapes getting mounted are from primary pool TAPEPOOL2. How can a restore be calling for tapes in a pool where the filespace has no data according to Q OCC? I suggested they open a sev 1 with Tivoli. This can't be right. tsm: HCG-TSM-SERVERq occ dc1 86 nametype=fsid Node Name Type Filespace FSID Storage Number of Physical Logical Name Pool Name Files Space Space Occupied Occupied (MB) (MB) -- -- - -- - - - DC1Bkup \VMFULL-H-86 COPYPOOL 8,928 99,057.72 99,062.73 WDDMZOCU- LARX1 DC1Bkup \VMFULL-H-86 SLOWDEDUP 4,464 - 98,733.95 WDDMZOCU- LARX1 DC1Bkup \VMFULL-H-86 VMCTLMC 4,464315.14315.14 WDDMZOCU- LARX1 -Original Message- From: Prather, Wanda Sent: Wednesday, June 19, 2013 11:03 AM To: Richard Cowen Subject: RE: Another VE mystery - restoring from tape - but shouldn't be Did the MOVE NODEDATA result in a zillion new volumes (.BFS's) ? No, we don't use scratch volumes in the seq disk pool Can you get the activity log for the time the process(es) ran ? Any chance you have query occupancy node=dcname filespace=victim1 stgpool=tape,disk before and after move? If not, does the query for tape pool now show zero? Don't have them and right now I don't have access, but those are great ideas, will ask the customer for them. Thanks! Wanda -Original Message- From: Prather, Wanda [mailto:wanda.prat...@icfi.com] Sent: Tuesday, June 18, 2013 11:35 AM To: Richard Cowen Subject: RE: Another VE mystery - restoring from tape - but shouldn't be Hi Richard, Did you get a zillion tape mounts during the MOVE NODEDATA? Yes Do you know it finished without errors? Yes. And we ran another MOVE NODEDATA to verify there was no more data to move for that filespace. How, exactly, did the data go from sequential fast disk - sequential slow disk - tape ? Ordinary migration, at different times, as the pools hit migration thresholds. I didn't think TSM would migrate more than one level, so maybe the last step was using a MOVE command? Yes, your migration hierarchy can have as many levels as you want, as long as you don't try to go from a sequential pool back to a disk (random) pool. What does a QUERY NODEDATA show for primary pools and copy pools? Would not be informative, as we only moved some of the filespaces, not all of them. Are you running aggressive reclamation on the tape pool? No, and it's not collocated. Which is why we decided to move the filespace back to the seq disk pool. I don't think it's odd the data was too spread out to make the restore from tape not work well. What is odd is that all the data from one VM is supposedly in one filespace. We moved that filespace back to disk with move nodedata fsid= But still getting tape mounts on the restore. Thanks for your interest! Wanda -Original Message- From: ADSM:
Poor LAN Free performance VIOS
All Got a new LAN Free client, nothing out of the ordinary, but LAN Free perf is slower than I'd hope. I'm going to a ProtecTier VTL and have 20 sessions going. Not started doing any fault finding yet, but the one big difference with this client is that the tape HBA's are virtual ones (NPIV). Has anyone seen less than stellar performance with this kind of setup, or should the virtual HBA's be lower down the list of things to check? For info, the VIO's do have seperate real HBA's for tape disk. Thoughts/comments welcome Thanks Steven
Re: Poor LAN Free performance VIOS
What speed are you getting? What OS? (AIX?) Poor performance vs a real tape drive or vs other streams to the VTL? In my DataDomain world the DD is always slower than a 3592 or LTO-4 drive when looking at a single stream. Together the streams do make a river in the 1,400 MB/sec range. Andy Huebner -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Steven Langdale Sent: Wednesday, June 19, 2013 4:46 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Poor LAN Free performance VIOS All Got a new LAN Free client, nothing out of the ordinary, but LAN Free perf is slower than I'd hope. I'm going to a ProtecTier VTL and have 20 sessions going. Not started doing any fault finding yet, but the one big difference with this client is that the tape HBA's are virtual ones (NPIV). Has anyone seen less than stellar performance with this kind of setup, or should the virtual HBA's be lower down the list of things to check? For info, the VIO's do have seperate real HBA's for tape disk. Thoughts/comments welcome Thanks Steven
