Re: JBB Question(s)
I've said this before, but I'll repeat. Based on painful experience, it makes a HUGE difference getting TSM to backup a Windows fileserver with 6TB and millions of files located on 2 monster volumes versus getting it to backup the same amount of data spread across 6 or 8 (merely) large volumes. Journal databases are a case in point. My suggestions are: 1. Limit volumes to no more than 600GB each (use DFS if necessary) 2. set memoryefficientbackup yes 3. Turn on journaling (backup a volume at a time if needed to get the journals to initialize) 4. If there are a lot of .pst files stored, consider using subfile 5. Upgrade to x64 Windows when possible Steve Schaub Systems Engineer, WNI BlueCross BlueShield of Tennessee 423-535-6574 (desk) 423-785-7347 (cell) -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Bos, Karel Sent: Tuesday, December 12, 2006 10:15 AM To: ADSM-L@VM.MARIST.EDU Subject: Re: [ADSM-L] JBB Question(s) Hi, I partly agree with the OS statement. Partly because I find it difficult to explain why an OS is able to contain one disk with +6 million files but the back-up application isn't able to get the back-up stable (journaling) or working at all (normal incremental) without using time consuming options like memory efficient. Splitting a disk over multiple nodes means hard coding the subdirs under root in the opt files. If a system administrator puts new data in a different folder, this data will be mist. Work around, adding a extra node which has a exclude.dir for all dirs already being managed by the other nodes. But what if the root is de container of all data? Meaning, the profile disk of a windows box with all profiles (6000+) in the root of the disk? Do I really want to be force to configure multiple nodes + one extra to get the backup of this monster running? MemEff runs for over 36 hours and Journal db is +2 GB within 24 hours. Regards, Karel -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Mark Stapleton Sent: dinsdag 12 december 2006 15:46 To: ADSM-L@VM.MARIST.EDU Subject: Re: JBB Question(s) From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Otto Chvosta >After that dbviewb reports that the journal state is 'not valid'. So we >tried a further inremental backup (scheduled) to get a valid state of >the journal database. >This incremental was stopped with > >ANS1999E Incremental processing of '\\fileserver\q$' stopped. >ANS1030E The operating system refused a TSM request for memory >allocation. > >We tried it again and again ... same result :-((( From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Schaub, Steve >Add this to the dsm.opt file and run the incremental again: >*== * > >* Reduce memory usage by processing a directory at a time (slower) * > >*== * >memoryefficientbackup yes > >Large windows fileservers with deep directory structures often exhaust >memory trying to traverse the entire filesystem during the initial scan. >This option scans the filesystem in chunks. To add a bit of detail: All modern Windows versions (except possibly Vista) have a hard-set limit of total memory that can be dedicated to a single process thread. (I believe it's 192MB, but don't quote me on that.) It is a hard limit that cannot be gotten around. Steve's workaround is an option. The other option is to use two nodenames for the same machine, with two option files/sets of TSM services/etc. One node backs up half the machine (by using include/exclude lines in the option files), and the other node backs up the other half. The real fix? Use a real server OS. -- Mark Stapleton ([EMAIL PROTECTED]) Senior TSM consultant Please see the following link for the BlueCross BlueShield of Tennessee E-mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm
Re: JBB Question(s)
From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Bos, Karel >I partly agree with the OS statement. Partly because I find it difficult >to explain why an OS is able to contain one disk with +6 million files >but the back-up application isn't able to get the back-up stable >(journaling) or working at all (normal incremental) without using time >consuming options like memory efficient. Disk accessibility and (proper) use of RAM for processes are two entirely different issues. Windows has (IMO) never been a real server operating system, for this and many other reasons. >Splitting a disk over multiple nodes means hard coding the subdirs under >root in the opt files. If a system administrator puts new data in a >different folder, this data will be mist. Work around, adding a extra >node which has a exclude.dir for all dirs already being managed by the >other nodes. ...or the judicious use of DOMAIN statements could do the same this, as could regular expressions. (Does anyone know if Windows clients recognize regular expressions in include/exclude statements?) >But what if the root is de container of all data? Meaning, the profile >disk of a windows box with all profiles (6000+) in the root of the disk? >Do I really want to be force to configure multiple nodes + one extra to >get the backup of this monster running? MemEff runs for over 36 hours >and Journal db is +2 GB within 24 hours. Such a system (all data in the root filesystem/drive) is very poorly designed. If you want highly robust storage accessibility, Windows/Intel are *not* the way to go for either software or hardware. The only reason many data shops try to cram 2-4TB of storage onto a single Windows server is because there is no desire to lay out the money or skills to do the work properly (i.e., UNIX). -- Mark Stapleton ([EMAIL PROTECTED]) Senior TSM consultant
Re: JBB Question(s)
Hi, I partly agree with the OS statement. Partly because I find it difficult to explain why an OS is able to contain one disk with +6 million files but the back-up application isn't able to get the back-up stable (journaling) or working at all (normal incremental) without using time consuming options like memory efficient. Splitting a disk over multiple nodes means hard coding the subdirs under root in the opt files. If a system administrator puts new data in a different folder, this data will be mist. Work around, adding a extra node which has a exclude.dir for all dirs already being managed by the other nodes. But what if the root is de container of all data? Meaning, the profile disk of a windows box with all profiles (6000+) in the root of the disk? Do I really want to be force to configure multiple nodes + one extra to get the backup of this monster running? MemEff runs for over 36 hours and Journal db is +2 GB within 24 hours. Regards, Karel -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Mark Stapleton Sent: dinsdag 12 december 2006 15:46 To: ADSM-L@VM.MARIST.EDU Subject: Re: JBB Question(s) From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Otto Chvosta >After that dbviewb reports that the journal state is 'not valid'. So we >tried a further inremental backup (scheduled) to get a valid state of >the journal database. >This incremental was stopped with > >ANS1999E Incremental processing of '\\fileserver\q$' stopped. >ANS1030E The operating system refused a TSM request for memory >allocation. > >We tried it again and again ... same result :-((( From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Schaub, Steve >Add this to the dsm.opt file and run the incremental again: >*== * > >* Reduce memory usage by processing a directory at a time (slower) * > >*== * >memoryefficientbackup yes > >Large windows fileservers with deep directory structures often exhaust >memory trying to traverse the entire filesystem during the initial scan. >This option scans the filesystem in chunks. To add a bit of detail: All modern Windows versions (except possibly Vista) have a hard-set limit of total memory that can be dedicated to a single process thread. (I believe it's 192MB, but don't quote me on that.) It is a hard limit that cannot be gotten around. Steve's workaround is an option. The other option is to use two nodenames for the same machine, with two option files/sets of TSM services/etc. One node backs up half the machine (by using include/exclude lines in the option files), and the other node backs up the other half. The real fix? Use a real server OS. -- Mark Stapleton ([EMAIL PROTECTED]) Senior TSM consultant ÿþD i t b e r i c h t i s v e r t r o u w e l i j k e n k a n g e h e i m e i n f o r m a t i e b e v a t t e n e n k e l b e s t e m d v o o r d e g e a d r e s s e e r d e . I n d i e n d i t b e r i c h t n i e t v o o r u i s b e s t e m d , v e r z o e k e n w i j u d i t o n m i d d e l l i j k a a n o n s t e m e l d e n e n h e t b e r i c h t t e v e r n i e t i g e n . A a n g e z i e n d e i n t e g r i t e i t v a n h e t b e r i c h t n i e t v e i l i g g e s t e l d i s m i d d e l s v e r z e n d i n g v i a i n t e r n e t , k a n A t o s O r i g i n n i e t a a n s p r a k e l i j k w o r d e n g e h o u d e n v o o r d e i n h o u d d a a r v a n . H o e w e l w i j o n s i n s p a n n e n e e n v i r u s v r i j n e t w e r k t e h a n t e r e n , g e v e n w i j g e e n e n k e l e g a r a n t i e d a t d i t b e r i c h t v i r u s v r i j i s , n o c h a a n v a a r d e n w i j e n i g e a a n s p r a k e l i j k h e i d v o o r d e m o g e l i j k e a a n w e z i g h e i d v a n e e n v i r u s i n d i t b e r i c h t . O p a l o n z e r e c h t s v e r h o u d i n g e n , a a n b i e d i n g e n e n o v e r e e n k o m s t e n w a a r o n d e r A t o s O r i g i n g o e d e r e n e n / o f d i e n s t e n l e v e r t z i j n m e t u i t s l u i t i n g v a n a l l e a n d e r e v o o r w a a r d e n d e L e v e r i n g s v o o r w a a r d e n v a n A t o s O r i g i n v a n t o e p a s s i n g . D e z e w o r d e n u o p a a n v r a a g d i r e c t k o s t e l o o s t o e g e z o n d e n . T h i s e - m a i l a n d t h e d o c u m e n t s a t t a c h e d a r e c o n f i d e n t i a l a n d i n t e n d e d s
Re: JBB Question(s)
From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Otto Chvosta >After that dbviewb reports that the journal state is 'not valid'. So we >tried a further inremental backup (scheduled) to get a valid state of >the journal database. >This incremental was stopped with > >ANS1999E Incremental processing of '\\fileserver\q$' stopped. >ANS1030E The operating system refused a TSM request for memory >allocation. > >We tried it again and again ... same result :-((( From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Schaub, Steve >Add this to the dsm.opt file and run the incremental again: >*== * > >* Reduce memory usage by processing a directory at a time (slower) * > >*== * >memoryefficientbackup yes > >Large windows fileservers with deep directory structures often exhaust >memory trying to traverse the entire filesystem during the initial scan. >This option scans the filesystem in chunks. To add a bit of detail: All modern Windows versions (except possibly Vista) have a hard-set limit of total memory that can be dedicated to a single process thread. (I believe it's 192MB, but don't quote me on that.) It is a hard limit that cannot be gotten around. Steve's workaround is an option. The other option is to use two nodenames for the same machine, with two option files/sets of TSM services/etc. One node backs up half the machine (by using include/exclude lines in the option files), and the other node backs up the other half. The real fix? Use a real server OS. -- Mark Stapleton ([EMAIL PROTECTED]) Senior TSM consultant
Re: JBB Question(s)
Add this to the dsm.opt file and run the incremental again: *==* * Reduce memory usage by processing a directory at a time (slower) * *==* memoryefficientbackup yes Large windows fileservers with deep directory structures often exhaust memory trying to traverse the entire filesystem during the initial scan. This option scans the filesystem in chunks. Steve Schaub Systems Engineer, WNI BlueCross BlueShield of Tennessee 423-535-6574 (desk) 423-785-7347 (cell) -Original Message- From: ADSM: Dist Stor Manager [mailto:[EMAIL PROTECTED] On Behalf Of Otto Chvosta Sent: Tuesday, December 12, 2006 8:40 AM To: ADSM-L@VM.MARIST.EDU Subject: [ADSM-L] JBB Question(s) Hi all, We'll try to backup a filespace of a fileserver (Windows Storage Server 2003 R2, Memory=2GB, Filespacedata: 650GB, 6.759.000 Files, complex directory structure, TSM Client Level 5.3.4 - TSM Server Level 5.3.4.0) using journal based backup. We set up the journal service and made a full incremental backup (processing time ~ 50 hours) for this filespace. After that dbviewb reports that the journal state is 'not valid'. So we tried a further inremental backup (scheduled) to get a valid state of the journal database. This incremental was stopped with ANS1999E Incremental processing of '\\fileserver\q$' stopped. ANS1030E The operating system refused a TSM request for memory allocation. We tried it again and again ... same result :-((( our modifications in tsmjbbd.ini before restarting the journal service: . . . NotifyBufferSize=0x0040 DirNotifyBufferSize=0x0020 JournaledFileSystems=Q: ; . . . [JournaledFileSystemSettings.Q:\] ; JournalDBSize=0x ; NotifyBufferSize=0x0140 ; DirNotifyBufferSize=0x00A0 ; PreserveDBOnExit=1 ; DeferFsMonStart=1 ; DeferRetryInterval=30 ; logFsErrors=0 Any suggestions how we can get a valid journal state ? ... and what we can do to prevent the ANS1030E ? Thank you in advance Otto Medical University of Vienna P.S: Does anyone of you know where I can download the actual dbviewb.exe ? Please see the following link for the BlueCross BlueShield of Tennessee E-mail disclaimer: http://www.bcbst.com/email_disclaimer.shtm