Re: Windows servers with a kazillion files and Win2K8...
The issue is that with 75M files, it's paramount to keep the metadata in something faster than spinning disk. Windows 2003 32-bit uses 36-bit memory addressing. You can use 32G of RAM. That might help, but what you WANT in cache never seems to STAY in cache. Bulk data pushes your MFT out of cache. What I'd recommend is using something like IBM's Easy Tier, though most SAN manufacturers have something similar. Basically, you have a bunch of spinning disk for your bulk data, and a small number of enterprise grade SSDs installed. The array code will figure out where your IOPS bottlenecks are, and will move just the offending blocks into the faster media. This amounts to having your MFT and a few small hotspots on flash. It would probably take one or two filesystem scans for optimal performance to be reached. While Enterprise SSD is not as fast as being in RAM, it's much faster than being on spinning disk. Don't confuse with consumer grade SSD which just won't have the IOPS performance for what you're doing. For IBM, the lowest cost way to get into EasyTier is the StorWize V7000. It's a combination storage drawer and SVC. I'm not sure about EMC, Hitachi and NetApp's comparable products. With friendly Regards, Josh-Daniel S. Davis On Mon, Feb 28, 2011 at 9:51 AM, Strand, Neil B. nbstr...@leggmason.com wrote: Wanda, If it is a 32 bit system, the most memory that can be addressed is 4G. 2^32 = 4,294,967,296 bytes 4,294,967,296 / (1,024 x 1,024) = 4,096 MB = 4GB Moving to a 64bit system would allow additional memory to be fed to the beast. If the windows servers are not running the application but simply providing filespace to the application that is running on another server, see if the following is possible: - Implement DFS and provide a virtual tree that is composed of multiple physical data repositories. Each repository could be backed up using a proxy - recovery may be a bit convoluted, but possible. Identify the problem not as a backup problem but a data management problem that requires some level of granularity to be introduced to the environment. Cheers, Neil Strand Storage Engineer - Legg Mason Baltimore, MD. (410) 580-7491 Whatever you can do or believe you can, begin it. Boldness has genius, power and magic. -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Friday, February 25, 2011 2:35 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Windows servers with a kazillion files and Win2K8... Thanks for the reply and the reference; I'll read that. It's a 32 bit system. Do you think adding RAM will help with the issues navigating the file tree? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Storer, Raymond Sent: Friday, February 25, 2011 2:22 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Windows servers with a kazillion files and Win2K8... Wanda, is this a 32 or 64 bit system? An NTFS file system will support about 4 Billion files on a single volume http://technet.microsoft.com/en-us/library/cc781134(WS.10).aspx . If you are having performance issues with this and you can switch it to a 64bit platform and add loads of RAM, I would do it. Ray -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Friday, February 25, 2011 2:03 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Windows servers with a kazillion files and Win2K8... I have a site with an application that generates kazillions of tiny files that are stored forever. I've already yelled about it, but it's a purchased, customer-facing black-box app that they really can't change. (Naturally, when it was bought umpty years ago, nobody thought about the problem reaching this size or what the ramifications would be.) Every day the app creates more files. They have multiple Win2K3 servers that already have multiple luns containing over 35M files each, one is over 75M files. We are using journaling to back them up successfully (most days). But it's a struggle just to expand the file tree with Windows explorer, and there are exposures on the days when the journal gets overrun (takes 72 hours for TSM to scan the filesystem and revalidate the journal). Looking for anything that might help save our bacon. Has anybody had experience with this issue and Win2K8? Does Win2K8 do any better than Win2K3 at handling huge numbers of files in 1 NTFS directory? Upgrading the OS is something application-independent we might be able to do. Thanks for any insight! W Wanda Prather | Senior Technical Specialist | wprat...@icfi.commailto:wprat...@icfi.com | www.jasi.comwww.jasi.com%20 ICF Jacob Sundstrom | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410.539.1135 CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of
Re: Windows servers with a kazillion files and Win2K8...
Wanda, If it is a 32 bit system, the most memory that can be addressed is 4G. 2^32 = 4,294,967,296 bytes 4,294,967,296 / (1,024 x 1,024) = 4,096 MB = 4GB Moving to a 64bit system would allow additional memory to be fed to the beast. If the windows servers are not running the application but simply providing filespace to the application that is running on another server, see if the following is possible: - Implement DFS and provide a virtual tree that is composed of multiple physical data repositories. Each repository could be backed up using a proxy - recovery may be a bit convoluted, but possible. Identify the problem not as a backup problem but a data management problem that requires some level of granularity to be introduced to the environment. Cheers, Neil Strand Storage Engineer - Legg Mason Baltimore, MD. (410) 580-7491 Whatever you can do or believe you can, begin it. Boldness has genius, power and magic. -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Friday, February 25, 2011 2:35 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Windows servers with a kazillion files and Win2K8... Thanks for the reply and the reference; I'll read that. It's a 32 bit system. Do you think adding RAM will help with the issues navigating the file tree? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Storer, Raymond Sent: Friday, February 25, 2011 2:22 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Windows servers with a kazillion files and Win2K8... Wanda, is this a 32 or 64 bit system? An NTFS file system will support about 4 Billion files on a single volume http://technet.microsoft.com/en-us/library/cc781134(WS.10).aspx . If you are having performance issues with this and you can switch it to a 64bit platform and add loads of RAM, I would do it. Ray -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Friday, February 25, 2011 2:03 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Windows servers with a kazillion files and Win2K8... I have a site with an application that generates kazillions of tiny files that are stored forever. I've already yelled about it, but it's a purchased, customer-facing black-box app that they really can't change. (Naturally, when it was bought umpty years ago, nobody thought about the problem reaching this size or what the ramifications would be.) Every day the app creates more files. They have multiple Win2K3 servers that already have multiple luns containing over 35M files each, one is over 75M files. We are using journaling to back them up successfully (most days). But it's a struggle just to expand the file tree with Windows explorer, and there are exposures on the days when the journal gets overrun (takes 72 hours for TSM to scan the filesystem and revalidate the journal). Looking for anything that might help save our bacon. Has anybody had experience with this issue and Win2K8? Does Win2K8 do any better than Win2K3 at handling huge numbers of files in 1 NTFS directory? Upgrading the OS is something application-independent we might be able to do. Thanks for any insight! W Wanda Prather | Senior Technical Specialist | wprat...@icfi.commailto:wprat...@icfi.com | www.jasi.comwww.jasi.com%20 ICF Jacob Sundstrom | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410.539.1135 CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message. IMPORTANT: E-mail sent through the Internet is not secure. Legg Mason therefore recommends that you do not send any confidential or sensitive information to us via electronic mail, including social security numbers, account numbers, or personal identification numbers. Delivery, and or timely delivery of Internet mail is not guaranteed. Legg Mason therefore recommends that you do not send time sensitive or action-oriented messages to us via electronic mail. This message is intended for the addressee only and may contain privileged or confidential information. Unless you are the intended recipient, you may not use, copy or disclose to anyone any information contained in this message. If you have received this message in error, please notify the author by replying to this message and then kindly delete the message. Thank you.
Re: Windows servers with a kazillion files and Win2K8...
Image backups an option? Might get the fs back-up under control, but what about a FS restore using file based back-ups? Doing a multi million object restore will take ages. Best regards, Met vriendelijke groet, Karel Bos MS Global Practice IT Infrastructure Services Commvault/IBM Tivoli Storage Manager specialist Operations Central Atos Origin Nederland B.V. Naritaweg 52, 1R.10 1043 BZ Amsterdam The Netherlands karel@atosorigin.com +31(0)88 2659543 www.nl.atosorigin.com -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Strand, Neil B. Sent: maandag 28 februari 2011 16:51 To: ADSM-L@VM.MARIST.EDU Subject: Re: Windows servers with a kazillion files and Win2K8... Wanda, If it is a 32 bit system, the most memory that can be addressed is 4G. 2^32 = 4,294,967,296 bytes 4,294,967,296 / (1,024 x 1,024) = 4,096 MB = 4GB Moving to a 64bit system would allow additional memory to be fed to the beast. If the windows servers are not running the application but simply providing filespace to the application that is running on another server, see if the following is possible: - Implement DFS and provide a virtual tree that is composed of multiple physical data repositories. Each repository could be backed up using a proxy - recovery may be a bit convoluted, but possible. Identify the problem not as a backup problem but a data management problem that requires some level of granularity to be introduced to the environment. Cheers, Neil Strand Storage Engineer - Legg Mason Baltimore, MD. (410) 580-7491 Whatever you can do or believe you can, begin it. Boldness has genius, power and magic. -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Friday, February 25, 2011 2:35 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Windows servers with a kazillion files and Win2K8... Thanks for the reply and the reference; I'll read that. It's a 32 bit system. Do you think adding RAM will help with the issues navigating the file tree? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Storer, Raymond Sent: Friday, February 25, 2011 2:22 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Windows servers with a kazillion files and Win2K8... Wanda, is this a 32 or 64 bit system? An NTFS file system will support about 4 Billion files on a single volume http://technet.microsoft.com/en-us/library/cc781134(WS.10).aspx . If you are having performance issues with this and you can switch it to a 64bit platform and add loads of RAM, I would do it. Ray -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Friday, February 25, 2011 2:03 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Windows servers with a kazillion files and Win2K8... I have a site with an application that generates kazillions of tiny files that are stored forever. I've already yelled about it, but it's a purchased, customer-facing black-box app that they really can't change. (Naturally, when it was bought umpty years ago, nobody thought about the problem reaching this size or what the ramifications would be.) Every day the app creates more files. They have multiple Win2K3 servers that already have multiple luns containing over 35M files each, one is over 75M files. We are using journaling to back them up successfully (most days). But it's a struggle just to expand the file tree with Windows explorer, and there are exposures on the days when the journal gets overrun (takes 72 hours for TSM to scan the filesystem and revalidate the journal). Looking for anything that might help save our bacon. Has anybody had experience with this issue and Win2K8? Does Win2K8 do any better than Win2K3 at handling huge numbers of files in 1 NTFS directory? Upgrading the OS is something application-independent we might be able to do. Thanks for any insight! W Wanda Prather | Senior Technical Specialist | wprat...@icfi.commailto:wprat...@icfi.com | www.jasi.comwww.jasi.com%20 ICF Jacob Sundstrom | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410.539.1135 CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message. IMPORTANT: E-mail sent through the Internet is not secure. Legg Mason therefore recommends that you do not send any confidential or sensitive information to us via electronic mail, including social security numbers, account numbers, or personal identification numbers. Delivery
Re: Windows servers with a kazillion files and Win2K8...
Wanda, is this a 32 or 64 bit system? An NTFS file system will support about 4 Billion files on a single volume http://technet.microsoft.com/en-us/library/cc781134(WS.10).aspx . If you are having performance issues with this and you can switch it to a 64bit platform and add loads of RAM, I would do it. Ray -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Friday, February 25, 2011 2:03 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Windows servers with a kazillion files and Win2K8... I have a site with an application that generates kazillions of tiny files that are stored forever. I've already yelled about it, but it's a purchased, customer-facing black-box app that they really can't change. (Naturally, when it was bought umpty years ago, nobody thought about the problem reaching this size or what the ramifications would be.) Every day the app creates more files. They have multiple Win2K3 servers that already have multiple luns containing over 35M files each, one is over 75M files. We are using journaling to back them up successfully (most days). But it's a struggle just to expand the file tree with Windows explorer, and there are exposures on the days when the journal gets overrun (takes 72 hours for TSM to scan the filesystem and revalidate the journal). Looking for anything that might help save our bacon. Has anybody had experience with this issue and Win2K8? Does Win2K8 do any better than Win2K3 at handling huge numbers of files in 1 NTFS directory? Upgrading the OS is something application-independent we might be able to do. Thanks for any insight! W Wanda Prather | Senior Technical Specialist | wprat...@icfi.commailto:wprat...@icfi.com | www.jasi.comwww.jasi.com%20 ICF Jacob Sundstrom | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410.539.1135 CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message.
Re: Windows servers with a kazillion files and Win2K8...
Thanks for the reply and the reference; I'll read that. It's a 32 bit system. Do you think adding RAM will help with the issues navigating the file tree? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Storer, Raymond Sent: Friday, February 25, 2011 2:22 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Windows servers with a kazillion files and Win2K8... Wanda, is this a 32 or 64 bit system? An NTFS file system will support about 4 Billion files on a single volume http://technet.microsoft.com/en-us/library/cc781134(WS.10).aspx . If you are having performance issues with this and you can switch it to a 64bit platform and add loads of RAM, I would do it. Ray -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Friday, February 25, 2011 2:03 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Windows servers with a kazillion files and Win2K8... I have a site with an application that generates kazillions of tiny files that are stored forever. I've already yelled about it, but it's a purchased, customer-facing black-box app that they really can't change. (Naturally, when it was bought umpty years ago, nobody thought about the problem reaching this size or what the ramifications would be.) Every day the app creates more files. They have multiple Win2K3 servers that already have multiple luns containing over 35M files each, one is over 75M files. We are using journaling to back them up successfully (most days). But it's a struggle just to expand the file tree with Windows explorer, and there are exposures on the days when the journal gets overrun (takes 72 hours for TSM to scan the filesystem and revalidate the journal). Looking for anything that might help save our bacon. Has anybody had experience with this issue and Win2K8? Does Win2K8 do any better than Win2K3 at handling huge numbers of files in 1 NTFS directory? Upgrading the OS is something application-independent we might be able to do. Thanks for any insight! W Wanda Prather | Senior Technical Specialist | wprat...@icfi.commailto:wprat...@icfi.com | www.jasi.comwww.jasi.com%20 ICF Jacob Sundstrom | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410.539.1135 CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message.
Re: Windows servers with a kazillion files and Win2K8...
Hi Wanda, This is a tough one, probably no one fell swoop approach to this. Some things to consider: - What is the nature of these files that are created every day? Are they of a temporary nature so they can be deleted after a period of time? If not, once created, how often are they referenced again? Maybe they are a candidate for archive (and delete after successful archive)? * Is a standard TSM file backup sufficient to meet this applications needs should the files need to be restored? * For that matter, what are the restore requirements? Even if backup was not an issue, can they all be restored in the event of a disk failure or other event that requires a full restore? * One approach could go something like this: - Use two nodes: one node to back up the rest of the system, the second node to back up this application directory. - For the rest of the system, use standard incremental backup.Use EXCLUDE.DIR to skip the application directory. - For the application directory, exclude all directories and files except for the application directory. Then use incremental by date (-incrbydate option) to back up the application directory. I'm sure you are well aware of the caveats associated with incremental by date, but if these files are created once, then never updated again, it might be a reasonable approach. Caveat: if the files are constantly created even during the backup operation, then -incrbydate might not be viable since files created while the backup is running might not get picked up by that backup, and they won't get picked up by the next backup either. * Another approach is a variation on the above: - Use two nodes as above, and back up the rest of the system as above. - Create a script that scans the application directory for files created during the last 24-hour period. For each file that was created in the last 24 hours, write it out to a file that can be fed to a command line client file list backup. For example, let's say the application directory backup begins at 23:00. So the script will search for files created between yesterday at 23:00 and today at 23:00, generate a file list, then kick off the dsmc selective -filelist=listfile. * Image backup? Best regards, Andy Raibeck IBM Software Group Tivoli Storage Manager Client Product Development Level 3 Team Lead Internal Notes e-mail: Andrew Raibeck/Hartford/IBM@IBMUS Internet e-mail: stor...@us.ibm.com IBM Tivoli Storage Manager support web page: http://www.ibm.com/support/entry/portal/Overview/Software/Tivoli/Tivoli_Storage_Manager ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 2011-02-25 14:03:24: From: Prather, Wanda wprat...@icfi.com To: ADSM-L@vm.marist.edu Date: 2011-02-25 14:04 Subject: Windows servers with a kazillion files and Win2K8... Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu I have a site with an application that generates kazillions of tiny files that are stored forever. I've already yelled about it, but it's a purchased, customer-facing black-box app that they really can't change. (Naturally, when it was bought umpty years ago, nobody thought about the problem reaching this size or what the ramifications would be.) Every day the app creates more files. They have multiple Win2K3 servers that already have multiple luns containing over 35M files each, one is over 75M files. We are using journaling to back them up successfully (most days). But it's a struggle just to expand the file tree with Windows explorer, and there are exposures on the days when the journal gets overrun (takes 72 hours for TSM to scan the filesystem and revalidate the journal). Looking for anything that might help save our bacon. Has anybody had experience with this issue and Win2K8? Does Win2K8 do any better than Win2K3 at handling huge numbers of files in 1 NTFS directory? Upgrading the OS is something application-independent we might be able to do. Thanks for any insight! W Wanda Prather | Senior Technical Specialist | wprat...@icfi.com mailto:wprat...@icfi.com | www.jasi.comwww.jasi.com%20 ICF Jacob Sundstrom | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410.539.1135
Re: Windows servers with a kazillion files and Win2K8...
I really don't know what Windows 2008 or more h/w would bring to the table, but for the file system performance itself, it might be worth running a defrag -- at least the analysis portion -- to see what the file system and MFT fragmentation looks like. Advanced defrag tools can also defrag the MFT, that might help speed up performance (by how much, I couldn't say... 75 million files is a lot no matter how you slice it). Best regards, Andy Raibeck IBM Software Group Tivoli Storage Manager Client Product Development Level 3 Team Lead Internal Notes e-mail: Andrew Raibeck/Hartford/IBM@IBMUS Internet e-mail: stor...@us.ibm.com IBM Tivoli Storage Manager support web page: http://www.ibm.com/support/entry/portal/Overview/Software/Tivoli/Tivoli_Storage_Manager ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 2011-02-25 14:34:46: From: Prather, Wanda wprat...@icfi.com To: ADSM-L@vm.marist.edu Date: 2011-02-25 14:38 Subject: Re: Windows servers with a kazillion files and Win2K8... Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu Thanks for the reply and the reference; I'll read that. It's a 32 bit system. Do you think adding RAM will help with the issues navigating the file tree? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Storer, Raymond Sent: Friday, February 25, 2011 2:22 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Windows servers with a kazillion files and Win2K8... Wanda, is this a 32 or 64 bit system? An NTFS file system will support about 4 Billion files on a single volume http:// technet.microsoft.com/en-us/library/cc781134(WS.10).aspx . If you are having performance issues with this and you can switch it to a 64bit platform and add loads of RAM, I would do it. Ray -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Friday, February 25, 2011 2:03 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Windows servers with a kazillion files and Win2K8... I have a site with an application that generates kazillions of tiny files that are stored forever. I've already yelled about it, but it's a purchased, customer-facing black-box app that they really can't change. (Naturally, when it was bought umpty years ago, nobody thought about the problem reaching this size or what the ramifications would be.) Every day the app creates more files. They have multiple Win2K3 servers that already have multiple luns containing over 35M files each, one is over 75M files. We are using journaling to back them up successfully (most days). But it's a struggle just to expand the file tree with Windows explorer, and there are exposures on the days when the journal gets overrun (takes 72 hours for TSM to scan the filesystem and revalidate the journal). Looking for anything that might help save our bacon. Has anybody had experience with this issue and Win2K8? Does Win2K8 do any better than Win2K3 at handling huge numbers of files in 1 NTFS directory? Upgrading the OS is something application-independent we might be able to do. Thanks for any insight! W Wanda Prather | Senior Technical Specialist | wprat...@icfi.com mailto:wprat...@icfi.com | www.jasi.comwww.jasi.com%20 ICF Jacob Sundstrom | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410.539.1135 CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message.
Re: Windows servers with a kazillion files and Win2K8...
Wanda, Yuck. The short answer is that, yes, moving from a 32-bit OS to a 64-bit OS and piling as much memory as you can afford will give you some relief - especially during full scan backups (when the journal gets invalidated). Depending on the nature of the file structure (many directories vs kazillions of files in a single directory), with a powerful enough server and fast disk, you can play games such as creating multiple nodes within each server, each of which is responsible for backing up a portion of the file tree. Having said that, I sure wouldn't want to be the one who has to administer the thing. I know you said they cant change the app, but I have found it is surprising how that can change when higher mgmt finds out that the system has designed itself to be unrecoverable in the event of a disaster. Good luck, Steve Schaub Systems Engineer II, Windows Backup/Recovery BlueCross BlueShield of Tennessee -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Friday, February 25, 2011 2:03 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Windows servers with a kazillion files and Win2K8... I have a site with an application that generates kazillions of tiny files that are stored forever. I've already yelled about it, but it's a purchased, customer-facing black-box app that they really can't change. (Naturally, when it was bought umpty years ago, nobody thought about the problem reaching this size or what the ramifications would be.) Every day the app creates more files. They have multiple Win2K3 servers that already have multiple luns containing over 35M files each, one is over 75M files. We are using journaling to back them up successfully (most days). But it's a struggle just to expand the file tree with Windows explorer, and there are exposures on the days when the journal gets overrun (takes 72 hours for TSM to scan the filesystem and revalidate the journal). Looking for anything that might help save our bacon. Has anybody had experience with this issue and Win2K8? Does Win2K8 do any better than Win2K3 at handling huge numbers of files in 1 NTFS directory? Upgrading the OS is something application-independent we might be able to do. Thanks for any insight! W Wanda Prather | Senior Technical Specialist | wprat...@icfi.commailto:wprat...@icfi.com | www.jasi.comwww.jasi.com%20 ICF Jacob Sundstrom | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410.539.1135 - Please see the following link for the BlueCross BlueShield of Tennessee E-mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm
Re: Windows servers with a kazillion files and Win2K8...
Wanda, please have your server admins perform some performance troubleshooting to see what might gain you the most improvements. Here is a link to help: http://technet.microsoft.com/en-us/library/cc779038(WS.10).aspx If you can't make heads or tails of this I'll try to help. Also, there are some simple things that you'll want to check--e.g. indexing service(off), anti-virus(exclude directory with all those files). If possible, I would consider moving to a 64bit platform. While it will not solve any potential spindle problems, it will allow you to put more usable RAM on the system and hold more data in RAM for caching. Ray -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Friday, February 25, 2011 2:35 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Windows servers with a kazillion files and Win2K8... Thanks for the reply and the reference; I'll read that. It's a 32 bit system. Do you think adding RAM will help with the issues navigating the file tree? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Storer, Raymond Sent: Friday, February 25, 2011 2:22 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Windows servers with a kazillion files and Win2K8... Wanda, is this a 32 or 64 bit system? An NTFS file system will support about 4 Billion files on a single volume http://technet.microsoft.com/en-us/library/cc781134(WS.10).aspx . If you are having performance issues with this and you can switch it to a 64bit platform and add loads of RAM, I would do it. Ray -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Friday, February 25, 2011 2:03 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Windows servers with a kazillion files and Win2K8... I have a site with an application that generates kazillions of tiny files that are stored forever. I've already yelled about it, but it's a purchased, customer-facing black-box app that they really can't change. (Naturally, when it was bought umpty years ago, nobody thought about the problem reaching this size or what the ramifications would be.) Every day the app creates more files. They have multiple Win2K3 servers that already have multiple luns containing over 35M files each, one is over 75M files. We are using journaling to back them up successfully (most days). But it's a struggle just to expand the file tree with Windows explorer, and there are exposures on the days when the journal gets overrun (takes 72 hours for TSM to scan the filesystem and revalidate the journal). Looking for anything that might help save our bacon. Has anybody had experience with this issue and Win2K8? Does Win2K8 do any better than Win2K3 at handling huge numbers of files in 1 NTFS directory? Upgrading the OS is something application-independent we might be able to do. Thanks for any insight! W Wanda Prather | Senior Technical Specialist | wprat...@icfi.commailto:wprat...@icfi.com | www.jasi.comwww.jasi.com%20 ICF Jacob Sundstrom | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410.539.1135 CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message. CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message.
Re: Windows servers with a kazillion files and Win2K8...
Some application systems (e.g., medical imaging, research data) accumulate files in forward-marching manner, as in creating new timestamp directories for the chronological storage of data. Those provide the opportunity to formulate Excludes to avoid rescanning directories previously treated for incremental backup, thus limiting the view to new areas of the file system so as to minimize work. I don't know if Tivoli Continuous Data Protection for Files might be applicable to this environment: worth a review, as it works with the OS to treat new data. If no ready solution, the overall issue may need to be brought to the fore with the vendor and application owner as untenable as it has been proceeding. In cases like this, a solid definition of backup/recovery needs is called for, both in terms of application needs and any legal requirements. Sometimes, just a solid mirroring of data is all that is needed. The vendor contract may have to be reviewed to see what remedies may be available, if there no good solutions are apparent, and the vendor is unhelpful. Richard Sims
Re: Windows servers with a kazillion files and Win2K8...
Good call. And, defrag is much improved on Server 2008. Ray -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Andrew Raibeck Sent: Friday, February 25, 2011 3:05 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Windows servers with a kazillion files and Win2K8... I really don't know what Windows 2008 or more h/w would bring to the table, but for the file system performance itself, it might be worth running a defrag -- at least the analysis portion -- to see what the file system and MFT fragmentation looks like. Advanced defrag tools can also defrag the MFT, that might help speed up performance (by how much, I couldn't say... 75 million files is a lot no matter how you slice it). Best regards, Andy Raibeck IBM Software Group Tivoli Storage Manager Client Product Development Level 3 Team Lead Internal Notes e-mail: Andrew Raibeck/Hartford/IBM@IBMUS Internet e-mail: stor...@us.ibm.com IBM Tivoli Storage Manager support web page: http://www.ibm.com/support/entry/portal/Overview/Software/Tivoli/Tivoli_Storage_Manager ADSM: Dist Stor Manager ADSM-L@vm.marist.edu wrote on 2011-02-25 14:34:46: From: Prather, Wanda wprat...@icfi.com To: ADSM-L@vm.marist.edu Date: 2011-02-25 14:38 Subject: Re: Windows servers with a kazillion files and Win2K8... Sent by: ADSM: Dist Stor Manager ADSM-L@vm.marist.edu Thanks for the reply and the reference; I'll read that. It's a 32 bit system. Do you think adding RAM will help with the issues navigating the file tree? -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Storer, Raymond Sent: Friday, February 25, 2011 2:22 PM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] Windows servers with a kazillion files and Win2K8... Wanda, is this a 32 or 64 bit system? An NTFS file system will support about 4 Billion files on a single volume http:// technet.microsoft.com/en-us/library/cc781134(WS.10).aspx . If you are having performance issues with this and you can switch it to a 64bit platform and add loads of RAM, I would do it. Ray -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Prather, Wanda Sent: Friday, February 25, 2011 2:03 PM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] Windows servers with a kazillion files and Win2K8... I have a site with an application that generates kazillions of tiny files that are stored forever. I've already yelled about it, but it's a purchased, customer-facing black-box app that they really can't change. (Naturally, when it was bought umpty years ago, nobody thought about the problem reaching this size or what the ramifications would be.) Every day the app creates more files. They have multiple Win2K3 servers that already have multiple luns containing over 35M files each, one is over 75M files. We are using journaling to back them up successfully (most days). But it's a struggle just to expand the file tree with Windows explorer, and there are exposures on the days when the journal gets overrun (takes 72 hours for TSM to scan the filesystem and revalidate the journal). Looking for anything that might help save our bacon. Has anybody had experience with this issue and Win2K8? Does Win2K8 do any better than Win2K3 at handling huge numbers of files in 1 NTFS directory? Upgrading the OS is something application-independent we might be able to do. Thanks for any insight! W Wanda Prather | Senior Technical Specialist | wprat...@icfi.com mailto:wprat...@icfi.com | www.jasi.comwww.jasi.com%20 ICF Jacob Sundstrom | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410.539.1135 CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message. CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message.
