SV: TSM NDMP separate data and tape server
Hi Grant, What do you mean? You can backup a NetApp direct to Tape but still keep track in TSM. /Christian -Ursprungligt meddelande- Från: Grant Street [mailto:gra...@al.com.au] Skickat: den 5 juni 2013 08:59 Till: ADSM-L@VM.MARIST.EDU Ämne: TSM NDMP separate data and tape server Hi I just want to confirm, my research Can you have the NDMP Data Server on a Windows server backing up to a the Tape server on a Netapp using TSM? I know a Netapp can be a DATA and TAPE server in one but according to the NDMP Definitions you should be able to separate them. Thanks Grant
Planning for NDMP backup
Hi everyone, we're planning on implementing a TSM server that also does backup NetApp Filers and we've run into a few questions. Hope that you can help me with these. 1. Is it possible to store the NDMP backups on disk or is only tape possible? 2. If only tape, how many drives are recommended for NDMP backup? One per Filer if they backup at the same time? Thanks in advance for any hints. Regards, Michael
Re: Sequential dedup pool doesn't seem to reclaim as it should
Hi Wanda From personal experience my reclaims show a similar behavior, and from what I understand that used tape count will keep on increasing. The amount of time it takes to dehydrate data from disk to copy when doing a reclaim will impact the reclaim operation and of course the amount of volumes that it will leave behind with data it needs. The only time I find reclaim operations completing is during the weekends. Does anyone have any recommendations? I've personally come to the conclusion that having smaller stg pools is key to more efficient reclaims on deduped pools -Nick -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Thursday, June 06, 2013 7:49 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Sequential dedup pool doesn't seem to reclaim as it should TSM 6.3.3 on Win2K8-64 I have a sequential pool on disk with DEDUP=yes. (Happens to be for TSM-VE data, but I don't think that's relevant.) Settings are below. There is 1 identify duplicates process always active. Reclaim threshold is set to 20. Every night the clients back up. At 4am we start the backup stgpool to a tape copy pool. When that is in process, several reclaims kick in on their own. But once those are finished, they don't ever crank up again later in the day. Every day it leaves several volumes above the reclaim threshold. Right now there are 5. *Identify Duplicates is finished and idle. *No client activity. *Backup stgpool file-ve copypool returns no data to be copied. It has been that way for the last 9 hours. If I start the reclaim myself with reclaim stgpool file-ve threshold=20, it runs just fine. But it won't reclaim (and therefore dedup) on its own. Shouldn't it? I'd like to have those volumes empty (and deduped) before the next backup cycle. I can force it by scheduling the extra reclaim command, but I don't understand why it doesn't kick off on its own more than once a day? tsm: LFTSMq stgpool file-ve f=d Storage Pool Name: FILE-VE Storage Pool Type: Primary Device Class Name: ONLINEFILE Estimated Capacity: 20,447 G Space Trigger Util: 68.3 Pct Util: 68.3 Pct Migr: 68.3 Pct Logical: 89.3 High Mig Pct: 98 Low Mig Pct: 70 Migration Delay: 0 Migration Continue: Yes Migration Processes: 1 Reclamation Processes: 2 Next Storage Pool: Reclaim Storage Pool: Maximum Size Threshold: No Limit Access: Read/Write Description: Dedup VE pool Overflow Location: Cache Migrated Files?: Collocate?: No Reclamation Threshold: 20 Offsite Reclamation Limit: Maximum Scratch Volumes Allowed: 0 Number of Scratch Volumes Used: 0 Delay Period for Volume Reuse: 0 Day(s) Migration in Progress?: No Amount Migrated (MB): 0.00 Elapsed Migration Time (seconds): 0 Reclamation in Progress?: Yes Last Update by (administrator): WANDA Last Update Date/Time: 05/29/2013 10:25:01 Storage Pool Data Format: Native Copy Storage Pool(s): Active Data Pool(s): Continue Copy on Error?: Yes CRC Data: No Reclamation Type: Threshold Overwrite Data when Deleted: Deduplicate Data?: Yes Processes For Identifying Duplicates: 1 more... (ENTER to continue, 'C' to cancel) Duplicate Data Not Stored: 20,588 G (60%) Auto-copy Mode: Client Contains Data Deduplicated by Client?: No Wanda Prather | Senior Technical Specialist | wanda.prat...@icfi.com | www.icfi.com ICF International | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410.539.1135 (o)
Re: When will there be support for Exchange 2013?
Hello, Thank you for posting your thoughts about the timing of the product delivery. While I am an IBM employee, the opinions I express here are mine and do not reflect on IBM. Know that I, and the people I work with, are working on delivering quality software that exceeds expectations. We want to work with customers. As Del posted earlier this week, there is a beta program. If you can participate, here's more information: https://www.ibm.com/developerworks/mydeveloperworks/blogs/869bac74-5fc2-4b94-81a2-6153890e029a/entry/current_flashcopy_manager_early_access_program_and_beta_progarm_for_future_release_looking_for_participants5?lang=en You can also contact me directly and I can connect you with the beta program leader. Angela Angela Robertson IBM Software Group Durham, NC 27703 aprob...@us.ibm.com ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 06/05/2013 05:32:42 PM: Remco Post r.p...@plcs.nl Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu 06/05/2013 05:32 PM Please respond to ADSM: Dist Stor Manager ADSM-L@vm.marist.edu To ADSM-L@vm.marist.edu, cc Subject Re: [ADSM-L] When will there be support for Exchange 2013? What I keep on wondering about... I'm quite sure that Microsoft releases developer previews of all of their products. I'm also quite sure that IBM would get access to such developer previews if it wanted to. Why the hell is IBM always at least half a year late in supporting new versions of the number one client platform? Do you really not care about the portion of your customers that are on the leading edge? And... are you still wondering why you're loosing customers? On 5 jun. 2013, at 16:40, Del Hoobler hoob...@us.ibm.com wrote: As you probably understand, we are not at liberty to give specific dates, however our target for adding Exchange Server 2013 toleration support for both FlashCopy Manager and Data Protection for Exchange is early 3Q13(in the next PTF for both.) As usual these targets do not represent commitment and can change at IBM's discretion. If anyone is interested in working with IBM earlier than that, there is a beta program available. Contact your IBM/Tivoli representative for details. Thanks, Del ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 06/05/2013 10:08:04 AM: From: Bill Boyer bjdbo...@comcast.net To: ADSM-L@vm.marist.edu, Date: 06/05/2013 10:12 AM Subject: When will there be support for Exchange 2013? Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu Anyone know? Bill Boyer DSS, Inc. (610) 927-4407 Enjoy life. It has an expiration date. - ?? -- Met vriendelijke groeten/Kind Regards, Remco Post r.p...@plcs.nl +31 6 248 21 622
Re: Planning for NDMP backup
Hi Michael I believe the backups can be stored on either disk or tape, but i only ever use tape. This is mainly due to the size of the backups, ours range between 1tb - 25tb. If on disk, that's a lot of disk! For the server that ran the NDMP backups, we always had at least 2 drives defined to the data movers. Other activities such as normal B/A backups took place which required reclaim and migration tasks and had to share the drives with the NDMP backups. If i only defined one drive to NDMP and the drive was busy, the backup would fail. So at least 2, and 4 drives if the NAS filer had the fibre to allow it. One problem i had: because of the size of the dumps, we never had the time to make offsite copies of the tapes. Right or wrong (and i know it's wrong), i had limited resources in terms of drives and time. Lot's of different things with NDMP backups. These are a brief summary of my own notes. Define a domain for the NAS backups: define dom nas desc='NAS Domain' define pol nas nasp Now register a node: reg node node-name password type=nas dom=nas Obtain the IP address of the NAS filer: (From the NAS filer) ifconfig sm_vif Note the IP address Now define the datamover: define datamover datamover-name type=nas hla=ip-address lla=port userid=node-name password=password-of-NAS-filer dataf=netappdump (Note the format must be netappdump) (Note2: I kept the datamover-name the same as the filer-name - no point confusing things and keeps it simple) From the filer: storage show tape Obtain the Alias name. Normally something like st0 and st1 Define the paths in TSM: def path filer-name drive0 srct=datamover desttype=drive library= device=rst0a def path filer-name drive0 srct=datamover desttype=drive library= device=rst1a (Note the device of rst0a and rst1a. This is the st0 and st1 taken from the filer and you need to suffix with an add 'a') q path Source Name Source Type Destination Destination On-Line NameType --- --- --- --- --- tsm-server SERVER AUTOLIB0LIBRARY Yes tsm-server SERVER DRIVE0 DRIVE Yes tsm-server SERVER DRIVE1 DRIVE Yes filer1 DATAMOVER DRIVE0 DRIVE Yes filer1 DATAMOVER DRIVE1 DRIVE Yes (So now i see the 'normal' tape drive paths, plus the file ones) Define my devc: define devclass NASCLASS devtype=nas library= mountretention=0 mountlimit=drives estcapacity=1600g (this was an LTO4 library) And a stgpool: def stgpool stgpool-name NASCLASS poolt=primary acc=readw col=no reuse=3 dataf=netappd maxscr=2000 (Note the devc was the tape devc i just created. You can do this to disk, of course) And the mgmtclass and copygroup: DEF MGMT NAS STANDARD 1M SPACEMGTECH=NONE AUTOMIGNO=0 MIGREQUIRESBK=YES MIGDEST=SPACEMGPOOL DESC=NAS MC DEF COPY NAS STANDARD 1M T=BACKUP DEST=stgpool-name FREQ=0 VERE=28 VERD=28 RETE=28 RETONLY=28 MODE=MODIFIED SER=DYNAMIC And finally: VALIDATE POL NAS NASP ACTIVATE POL NAS NASP I recommend you use a TOC (table of contents). See the admin guide or admin ref about this Set up a Virtual Filespce: DEFINE VIRTUALFS nodename virtual-fs-name fs-name path namet=server (Note: The fs-name and path must match that of the Netapp filer) (Note the virtual-fs-name is what TSM server knows the filer storage area as) This will backup the filer: BACKUP NODE node-name virtual fs name MGMT=nnn MODE=FULL TOC=YES THE BACKUP RUNS AS A PROCESS (not as a session) SO CHECK THE PROCESS TO CONFIRM THAT THE BACKUP HAS BEEN SUCCESSFUL! Notes: Q NODE will not show NAS nodes. Need to use Q NODE TYPE=NAS The Admin Guide states that Reclamation and Migration are not supported for these storage pools I am sure there may be better ways, perhaps dedupe etc, not sure. But this is what i setup Best of luck On 7 June 2013 08:51, Michael Roesch michael.roe...@gmail.com wrote: Hi everyone, we're planning on implementing a TSM server that also does backup NetApp Filers and we've run into a few questions. Hope that you can help me with these. 1. Is it possible to store the NDMP backups on disk or is only tape possible? 2. If only tape, how many drives are recommended for NDMP backup? One per Filer if they backup at the same time? Thanks in advance for any hints. Regards, Michael
Re: TSM NDMP separate data and tape server
He is talking about a 3-way NDMP backup introduced with NDMP v2. It lets you use a second file server as a remote storage agent. File Server 1 -- LAN -- File Server 2 -- Local/SAN attached tape drive. The following thread discusses it, but I never got it working and I don't think there was a definite resolution or documentation on it. I was able to get the Remote NDMP over IP variant working that is described in the TSM 5.4 5.5 Technical guide: File Server 1 -- LAN -- TSM Server -- TSM Native storage pool (not Netappdump) but not the classic3-way backup. http://adsm.org/lists/html/ADSM-L/2009-10/msg00255.html Regards, Shawn Shawn Drew -Original Message- From: ADSM-L@VM.MARIST.EDU [mailto:ADSM-L@VM.MARIST.EDU] Sent: Friday, June 07, 2013 3:18 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] SV: TSM NDMP separate data and tape server Hi Grant, What do you mean? You can backup a NetApp direct to Tape but still keep track in TSM. /Christian -Ursprungligt meddelande- Från: Grant Street [mailto:gra...@al.com.au] Skickat: den 5 juni 2013 08:59 Till: ADSM-L@VM.MARIST.EDU Ämne: TSM NDMP separate data and tape server Hi I just want to confirm, my research Can you have the NDMP Data Server on a Windows server backing up to a the Tape server on a Netapp using TSM? I know a Netapp can be a DATA and TAPE server in one but according to the NDMP Definitions you should be able to separate them. Thanks Grant This message and any attachments (the message) is intended solely for the addressees and is confidential. If you receive this message in error, please delete it and immediately notify the sender. Any use not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. The internet can not guarantee the integrity of this message. BNP PARIBAS (and its subsidiaries) shall (will) not therefore be liable for the message if modified. Please note that certain functions and services for BNP Paribas may be performed by BNP Paribas RCC, Inc.
Re: Planning for NDMP backup
You can use disk or tape. Disk can be used through a VTL or as a normal file device class. If you want to use file device classes, look in the manual for backup up a NAS file server to native pools Regards, Shawn Shawn Drew -Original Message- From: ADSM-L@VM.MARIST.EDU [mailto:ADSM-L@VM.MARIST.EDU] Sent: Friday, June 07, 2013 3:51 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Planning for NDMP backup Hi everyone, we're planning on implementing a TSM server that also does backup NetApp Filers and we've run into a few questions. Hope that you can help me with these. 1. Is it possible to store the NDMP backups on disk or is only tape possible? 2. If only tape, how many drives are recommended for NDMP backup? One per Filer if they backup at the same time? Thanks in advance for any hints. Regards, Michael This message and any attachments (the message) is intended solely for the addressees and is confidential. If you receive this message in error, please delete it and immediately notify the sender. Any use not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. The internet can not guarantee the integrity of this message. BNP PARIBAS (and its subsidiaries) shall (will) not therefore be liable for the message if modified. Please note that certain functions and services for BNP Paribas may be performed by BNP Paribas RCC, Inc.
Re: Planning for NDMP backup
If you have a NetApp, is there any particular reason you're not using snapdiff? My experiences with NDMP are all bad. Troubles with reclamation, troubles with creating copy pools, having to cancel backups, reclamation and backup stg because the recovery log was getting way too full. - Cameron Hanover chano...@umich.edu Reminds me of my safari in Africa. Somebody forgot the corkscrew and for several days we had to live on nothing but food and water. --W. C. Fields On Jun 7, 2013, at 3:51 AM, Michael Roesch michael.roe...@gmail.com wrote: Hi everyone, we're planning on implementing a TSM server that also does backup NetApp Filers and we've run into a few questions. Hope that you can help me with these. 1. Is it possible to store the NDMP backups on disk or is only tape possible? 2. If only tape, how many drives are recommended for NDMP backup? One per Filer if they backup at the same time? Thanks in advance for any hints. Regards, Michael
Re: Planning for NDMP backup
Your backup command is: BACKUP NODE node-name virtual fs name MGMT=nnn MODE=FULL TOC=YES I use 'TOC=P'. We have one file system where we cannot generate a TOC because of the number of files. TOC=PREFERRED will prevent the backup from failing when TOC creation fails. Jim -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of white jeff Sent: Friday, June 07, 2013 10:05 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Planning for NDMP backup Hi Michael I believe the backups can be stored on either disk or tape, but i only ever use tape. This is mainly due to the size of the backups, ours range between 1tb - 25tb. If on disk, that's a lot of disk! For the server that ran the NDMP backups, we always had at least 2 drives defined to the data movers. Other activities such as normal B/A backups took place which required reclaim and migration tasks and had to share the drives with the NDMP backups. If i only defined one drive to NDMP and the drive was busy, the backup would fail. So at least 2, and 4 drives if the NAS filer had the fibre to allow it. One problem i had: because of the size of the dumps, we never had the time to make offsite copies of the tapes. Right or wrong (and i know it's wrong), i had limited resources in terms of drives and time. Lot's of different things with NDMP backups. These are a brief summary of my own notes. Define a domain for the NAS backups: define dom nas desc='NAS Domain' define pol nas nasp Now register a node: reg node node-name password type=nas dom=nas Obtain the IP address of the NAS filer: (From the NAS filer) ifconfig sm_vif Note the IP address Now define the datamover: define datamover datamover-name type=nas hla=ip-address lla=port userid=node-name password=password-of-NAS-filer dataf=netappdump (Note the format must be netappdump) (Note2: I kept the datamover-name the same as the filer-name - no point confusing things and keeps it simple) From the filer: storage show tape Obtain the Alias name. Normally something like st0 and st1 Define the paths in TSM: def path filer-name drive0 srct=datamover desttype=drive library= device=rst0a def path filer-name drive0 srct=datamover desttype=drive library= device=rst1a (Note the device of rst0a and rst1a. This is the st0 and st1 taken from the filer and you need to suffix with an add 'a') q path Source Name Source Type Destination Destination On-Line NameType --- --- --- --- --- tsm-server SERVER AUTOLIB0LIBRARY Yes tsm-server SERVER DRIVE0 DRIVE Yes tsm-server SERVER DRIVE1 DRIVE Yes filer1 DATAMOVER DRIVE0 DRIVE Yes filer1 DATAMOVER DRIVE1 DRIVE Yes (So now i see the 'normal' tape drive paths, plus the file ones) Define my devc: define devclass NASCLASS devtype=nas library= mountretention=0 mountlimit=drives estcapacity=1600g (this was an LTO4 library) And a stgpool: def stgpool stgpool-name NASCLASS poolt=primary acc=readw col=no reuse=3 dataf=netappd maxscr=2000 (Note the devc was the tape devc i just created. You can do this to disk, of course) And the mgmtclass and copygroup: DEF MGMT NAS STANDARD 1M SPACEMGTECH=NONE AUTOMIGNO=0 MIGREQUIRESBK=YES MIGDEST=SPACEMGPOOL DESC=NAS MC DEF COPY NAS STANDARD 1M T=BACKUP DEST=stgpool-name FREQ=0 VERE=28 VERD=28 RETE=28 RETONLY=28 MODE=MODIFIED SER=DYNAMIC And finally: VALIDATE POL NAS NASP ACTIVATE POL NAS NASP I recommend you use a TOC (table of contents). See the admin guide or admin ref about this Set up a Virtual Filespce: DEFINE VIRTUALFS nodename virtual-fs-name fs-name path namet=server (Note: The fs-name and path must match that of the Netapp filer) (Note the virtual-fs-name is what TSM server knows the filer storage area as) This will backup the filer: BACKUP NODE node-name virtual fs name MGMT=nnn MODE=FULL TOC=YES THE BACKUP RUNS AS A PROCESS (not as a session) SO CHECK THE PROCESS TO CONFIRM THAT THE BACKUP HAS BEEN SUCCESSFUL! Notes: Q NODE will not show NAS nodes. Need to use Q NODE TYPE=NAS The Admin Guide states that Reclamation and Migration are not supported for these storage pools I am sure there may be better ways, perhaps dedupe etc, not sure. But this is what i setup Best of luck On 7 June 2013 08:51, Michael Roesch michael.roe...@gmail.com wrote: Hi everyone, we're planning on implementing a TSM server that also does backup NetApp Filers and we've run into a few questions. Hope that you can help me with these. 1. Is it possible to store the NDMP backups on disk or is only tape possible? 2. If only tape, how many drives are recommended for NDMP backup? One per Filer if they backup at the same time? Thanks in
Re: Planning for NDMP backup
To paraphrase Churchill, NDMP is the worst form of NAS backups, except for all the others.. I keep testing snapdiff with every new TSM client and/or ONTAP update. It's just not as reliable as NDMP for me. Bottom line is that if you can't handle the periodic full-scan backups (like journaling) then don't bother. At least for our (250TB) environment. You can make NDMP on TSM bearable with lots of scripting and classic backup thinking (i.e. regular fulls and no copypools) Also, separate NDMP storage pools and group backups with the same retention. i.e NDMP_35day, NDMP_1year, etc. This lets the tapes expire regularly without reclamation. As far as drives per filer, it seems like a standard to not go over 4 drives per filer concurrently but, as always, it depends. Some file servers are more powerful and can handle more, some less. Some volumes are just plain slower than others and you need to plan around that. Someone also mentioned something about not being to create a TOC because of too many files. During an NDMP backup, the TOC is temporarily stored in the DB temp space then dumped to the actual TOCDestination after the backup finishes. (if your DB is 80% utilized, then the 20% unused space is the temp space). I ran our NDMP TSM Server/library manager with a 150GB db that was .05 pct utilized. This solved all of our TOC issues, so may help. (This was TSM 5.5, so I'm not sure how that is handled in TSM 6) That said, we scrapped TSM NDMP here. We have a small dedicated Netbackup environment just for NAS now. It uses a heck of a lot of tapes, but it finishes reliably. It was like magic when I was able to preview the tape I would need for a restore without having to run selects against the backups/contents tables and not having to wait a couple hours to load a particularly large TOC. Regards, Shawn Shawn Drew -Original Message- From: ADSM-L@VM.MARIST.EDU [mailto:ADSM-L@VM.MARIST.EDU] Sent: Friday, June 07, 2013 11:30 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Planning for NDMP backup If you have a NetApp, is there any particular reason you're not using snapdiff? My experiences with NDMP are all bad. Troubles with reclamation, troubles with creating copy pools, having to cancel backups, reclamation and backup stg because the recovery log was getting way too full. - Cameron Hanover chano...@umich.edu Reminds me of my safari in Africa. Somebody forgot the corkscrew and for several days we had to live on nothing but food and water. --W. C. Fields On Jun 7, 2013, at 3:51 AM, Michael Roesch michael.roe...@gmail.com wrote: Hi everyone, we're planning on implementing a TSM server that also does backup NetApp Filers and we've run into a few questions. Hope that you can help me with these. 1. Is it possible to store the NDMP backups on disk or is only tape possible? 2. If only tape, how many drives are recommended for NDMP backup? One per Filer if they backup at the same time? Thanks in advance for any hints. Regards, Michael This message and any attachments (the message) is intended solely for the addressees and is confidential. If you receive this message in error, please delete it and immediately notify the sender. Any use not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. The internet can not guarantee the integrity of this message. BNP PARIBAS (and its subsidiaries) shall (will) not therefore be liable for the message if modified. Please note that certain functions and services for BNP Paribas may be performed by BNP Paribas RCC, Inc.
Re: Planning for NDMP backup
We went the other route and backup our filers (Isilon and BlueARC, not NetApp) over NFS using a pool of TSM proxy nodes and schedules that float between the nodes. We have a small staff relative to the size of our environment, and decided that while we could support one backup environment well, we couldn't do two. So far, it's scaled better than can be expected. At the extreme, we've pumped over 50TB/day through the proxy nodes. -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine On 06/07/13 09:16, Shawn DREW wrote: To paraphrase Churchill, NDMP is the worst form of NAS backups, except for all the others.. I keep testing snapdiff with every new TSM client and/or ONTAP update. It's just not as reliable as NDMP for me. Bottom line is that if you can't handle the periodic full-scan backups (like journaling) then don't bother. At least for our (250TB) environment. You can make NDMP on TSM bearable with lots of scripting and classic backup thinking (i.e. regular fulls and no copypools) Also, separate NDMP storage pools and group backups with the same retention. i.e NDMP_35day, NDMP_1year, etc. This lets the tapes expire regularly without reclamation. As far as drives per filer, it seems like a standard to not go over 4 drives per filer concurrently but, as always, it depends. Some file servers are more powerful and can handle more, some less. Some volumes are just plain slower than others and you need to plan around that. Someone also mentioned something about not being to create a TOC because of too many files. During an NDMP backup, the TOC is temporarily stored in the DB temp space then dumped to the actual TOCDestination after the backup finishes. (if your DB is 80% utilized, then the 20% unused space is the temp space). I ran our NDMP TSM Server/library manager with a 150GB db that was .05 pct utilized. This solved all of our TOC issues, so may help. (This was TSM 5.5, so I'm not sure how that is handled in TSM 6) That said, we scrapped TSM NDMP here. We have a small dedicated Netbackup environment just for NAS now. It uses a heck of a lot of tapes, but it finishes reliably. It was like magic when I was able to preview the tape I would need for a restore without having to run selects against the backups/contents tables and not having to wait a couple hours to load a particularly large TOC. Regards, Shawn Shawn Drew -Original Message- From: ADSM-L@VM.MARIST.EDU [mailto:ADSM-L@VM.MARIST.EDU] Sent: Friday, June 07, 2013 11:30 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Planning for NDMP backup If you have a NetApp, is there any particular reason you're not using snapdiff? My experiences with NDMP are all bad. Troubles with reclamation, troubles with creating copy pools, having to cancel backups, reclamation and backup stg because the recovery log was getting way too full. - Cameron Hanover chano...@umich.edu Reminds me of my safari in Africa. Somebody forgot the corkscrew and for several days we had to live on nothing but food and water. --W. C. Fields On Jun 7, 2013, at 3:51 AM, Michael Roesch michael.roe...@gmail.com wrote: Hi everyone, we're planning on implementing a TSM server that also does backup NetApp Filers and we've run into a few questions. Hope that you can help me with these. 1. Is it possible to store the NDMP backups on disk or is only tape possible? 2. If only tape, how many drives are recommended for NDMP backup? One per Filer if they backup at the same time? Thanks in advance for any hints. Regards, Michael This message and any attachments (the message) is intended solely for the addressees and is confidential. If you receive this message in error, please delete it and immediately notify the sender. Any use not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. The internet can not guarantee the integrity of this message. BNP PARIBAS (and its subsidiaries) shall (will) not therefore be liable for the message if modified. Please note that certain functions and services for BNP Paribas may be performed by BNP Paribas RCC, Inc.
Re: Sequential dedup pool doesn't seem to reclaim as it should
I have noticed reclamation does not always start when volumes hit the threshold and I do not use the de-dup option. I have found that if I re-set the reclaim threshold that reclaim kicks off. So for the example below run: upd stgpool file-ve reclaim=20 No net change, but it seems to trigger the reclaim process. I think this may be a Benign Undocumented Gottcha. Andy Huebner -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Marouf, Nick Sent: Friday, June 07, 2013 5:50 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Sequential dedup pool doesn't seem to reclaim as it should Hi Wanda From personal experience my reclaims show a similar behavior, and from what I understand that used tape count will keep on increasing. The amount of time it takes to dehydrate data from disk to copy when doing a reclaim will impact the reclaim operation and of course the amount of volumes that it will leave behind with data it needs. The only time I find reclaim operations completing is during the weekends. Does anyone have any recommendations? I've personally come to the conclusion that having smaller stg pools is key to more efficient reclaims on deduped pools -Nick -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Thursday, June 06, 2013 7:49 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Sequential dedup pool doesn't seem to reclaim as it should TSM 6.3.3 on Win2K8-64 I have a sequential pool on disk with DEDUP=yes. (Happens to be for TSM-VE data, but I don't think that's relevant.) Settings are below. There is 1 identify duplicates process always active. Reclaim threshold is set to 20. Every night the clients back up. At 4am we start the backup stgpool to a tape copy pool. When that is in process, several reclaims kick in on their own. But once those are finished, they don't ever crank up again later in the day. Every day it leaves several volumes above the reclaim threshold. Right now there are 5. *Identify Duplicates is finished and idle. *No client activity. *Backup stgpool file-ve copypool returns no data to be copied. It has been that way for the last 9 hours. If I start the reclaim myself with reclaim stgpool file-ve threshold=20, it runs just fine. But it won't reclaim (and therefore dedup) on its own. Shouldn't it? I'd like to have those volumes empty (and deduped) before the next backup cycle. I can force it by scheduling the extra reclaim command, but I don't understand why it doesn't kick off on its own more than once a day? tsm: LFTSMq stgpool file-ve f=d Storage Pool Name: FILE-VE Storage Pool Type: Primary Device Class Name: ONLINEFILE Estimated Capacity: 20,447 G Space Trigger Util: 68.3 Pct Util: 68.3 Pct Migr: 68.3 Pct Logical: 89.3 High Mig Pct: 98 Low Mig Pct: 70 Migration Delay: 0 Migration Continue: Yes Migration Processes: 1 Reclamation Processes: 2 Next Storage Pool: Reclaim Storage Pool: Maximum Size Threshold: No Limit Access: Read/Write Description: Dedup VE pool Overflow Location: Cache Migrated Files?: Collocate?: No Reclamation Threshold: 20 Offsite Reclamation Limit: Maximum Scratch Volumes Allowed: 0 Number of Scratch Volumes Used: 0 Delay Period for Volume Reuse: 0 Day(s) Migration in Progress?: No Amount Migrated (MB): 0.00 Elapsed Migration Time (seconds): 0 Reclamation in Progress?: Yes Last Update by (administrator): WANDA Last Update Date/Time: 05/29/2013 10:25:01 Storage Pool Data Format: Native Copy Storage Pool(s): Active Data Pool(s): Continue Copy on Error?: Yes CRC Data: No Reclamation Type: Threshold Overwrite Data when Deleted: Deduplicate Data?: Yes Processes For Identifying Duplicates: 1 more... (ENTER to continue, 'C' to cancel) Duplicate Data Not Stored: 20,588 G (60%) Auto-copy Mode: Client Contains Data Deduplicated by Client?: No Wanda Prather | Senior Technical Specialist | wanda.prat...@icfi.com | www.icfi.com ICF International | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410.539.1135 (o)
Re: Re: Planning for NDMP backup
We finally gave up on NDMP entirely. We replaced it with NetApp's SnapVault solution. Friendlier for the service owner to use, and managing it all became Not My Problem. When we did use NDMP with TSM, I zoned all the drives to the filers (shared the drives with TSM and two arrays), but set a mount limit on the device class. That way, I didn't need to worry about an NDMP operation consuming too many drives, nor need to dedicate one or two drives to just NDMP operations. I used NETAPPDUMP format tape pools, scheduled the NDMP backups to one at a time per filer (more than that not only consumed more tape drives, but also slowed the backups down considerably), and scheduled the storage pool backup for about 10 days later (took around a week to complete one set of full backups). I did use ToCs (stored in a native format pool). We only ran one set of full backups, monthly, with no differentials. We relied on filer snapshots for restores from later than the last full, and replication to another filer for DR recovery. I configured the web client on a Windows server for the service owner to have a more useful interface to browse or restore backups. I started playing with SnapDiff, but didn't get far with it for CIFS shares, and had utter failure for NFS shares. Abandoned the testing once they started talking about switching to SnapVault. I'm in Wanda's camp as far as NDMP goes... =Dave Skylar Thompson wrote: We went the other route and backup our filers (Isilon and BlueARC, not NetApp) over NFS using a pool of TSM proxy nodes and schedules that float between the nodes. We have a small staff relative to the size of our environment, and decided that while we could support one backup environment well, we couldn't do two. So far, it's scaled better than can be expected. At the extreme, we've pumped over 50TB/day through the proxy nodes. On 06/07/13 09:16, Shawn DREW wrote: To paraphrase Churchill, NDMP is the worst form of NAS backups, except for all the others.. I keep testing snapdiff with every new TSM client and/or ONTAP update. It's just not as reliable as NDMP for me. Bottom line is that if you can't handle the periodic full-scan backups (like journaling) then don't bother. At least for our (250TB) environment. You can make NDMP on TSM bearable with lots of scripting and classic backup thinking (i.e. regular fulls and no copypools) Also, separate NDMP storage pools and group backups with the same retention. i.e NDMP_35day, NDMP_1year, etc. This lets the tapes expire regularly without reclamation. As far as drives per filer, it seems like a standard to not go over 4 drives per filer concurrently but, as always, it depends. Some file servers are more powerful and can handle more, some less. Some volumes are just plain slower than others and you need to plan around that. Someone also mentioned something about not being to create a TOC because of too many files. During an NDMP backup, the TOC is temporarily stored in the DB temp space then dumped to the actual TOCDestination after the backup finishes. (if your DB is 80% utilized, then the 20% unused space is the temp space). I ran our NDMP TSM Server/library manager with a 150GB db that was .05 pct utilized. This solved all of our TOC issues, so may help. (This was TSM 5.5, so I'm not sure how that is handled in TSM 6) That said, we scrapped TSM NDMP here. We have a small dedicated Netbackup environment just for NAS now. It uses a heck of a lot of tapes, but it finishes reliably. It was like magic when I was able to preview the tape I would need for a restore without having to run selects against the backups/contents tables and not having to wait a couple hours to load a particularly large TOC. Regards, Shawn Shawn Drew -Original Message- From: ADSM-L@VM.MARIST.EDU [mailto:ADSM-L@VM.MARIST.EDU] Sent: Friday, June 07, 2013 11:30 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Planning for NDMP backup If you have a NetApp, is there any particular reason you're not using snapdiff? My experiences with NDMP are all bad. Troubles with reclamation, troubles with creating copy pools, having to cancel backups, reclamation and backup stg because the recovery log was getting way too full. - Cameron Hanover chano...@umich.edu Reminds me of my safari in Africa. Somebody forgot the corkscrew and for several days we had to live on nothing but food and water. --W. C. Fields On Jun 7, 2013, at 3:51 AM, Michael Roesch michael.roe...@gmail.com wrote: Hi everyone, we're planning on implementing a TSM server that also does backup NetApp Filers and we've run into a few questions. Hope that you can help me with these. 1. Is it possible to store the NDMP backups on disk or is only tape possible? 2. If only tape, how many drives are recommended for NDMP
