Dan, I appreciate the info. I am understanding this little by little. I wonder if you could please clarify something for me.
Like I mentioned. We were running a DG/UX, Universe system for about 10 years. All files were dynamic and I never had to really get into learning the nity gritty of the file structures. Everything performed very well no matter what. I never had to do any kind of file sizing. So, forgive my ignorance. So, Im reading your article in Spectrum about Dynamic files. Im confused about what overflow really is in a dynamic file ? Is overflow synonymous with the OVER.30 physical file ? The article states that the data portion consists of all the record id's and offsets. The OVER.30 consists of everything else. Where is the overflow ? If a record is too large and has to be semi located in overflow, is it some kind of chunk at the end of the physical file ? Is it the 'internals' class that discusses this topic ? Has anyone ever written a "Universe File Internals For Dummies" ? -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dan Fitzgerald Sent: Tuesday, February 15, 2005 1:32 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Universe/Linux Performance Issue Oops! Chuck's right about the output of analyze.shm, but not about dynamic files ;) [In case there's any question, Chuck is in my personal pantheon, up there with Aung San Suu Kyi, Nelson Mandela, and... hmm... ok, it's a short list]. I was mixing that column with the base mod, which I output on a report that parses ANALYZE.FILE. When you have a file whose current mod exceeds the base mod, a high percentage of writes will involve splitting & a small re-org. This is enormously expensive. You have to lock the existing data, read it, go grab an available chunk of disk, parse & divide the data according to the algorithm, write both frames out, unlock them, and then read, lock, update, and unlock the header. The update lock of the header occurs earlier in the process than this, so you're holding the lock for a few disk accesses, creating a bottleneck. The only time I like dynamic files is when I find myself tempted to give a file a seperation of 32 or more. Even then, because I deal with a lot of systems that depend on others to maintain them, I take a look at a GROUP.STAT.DETAIL to see if there's just a few massively oversized items skewing the results, and try to determine if they're heavily used. If FILE.USAGE shows a lot of oversize reads & writes, dynamic might -MIGHT- be a better choice, although I've heard from a usually authoritative source that static files don't do as bad a job as I imagined at storing & retrieving oversized items. Fact is, I just don't like 'em. It's empirical, not rational. A few times, I've done nothing else to a slow process than convert a dynamic to a static & see an immediate increase in throughput. Admittedly, one big problem with them is that they have been erroneously presented as an alternative to tuning files, so I'm always running into ones that have gone to seed. I should be more libertarian about it, but repeated encounters with poor maintenance habits have brought out the inner autocrat. "Our greatest duty in this life is to help others. And please, if you can't help them, could you at least not hurt them?" - H.H. the Dalai Lama "When buying & selling are controlled by legislation, the first thing to be bought & sold are the legislators" - P.J. O'Rourke Dan Fitzgerald >From: "Stevenson, Charles" <[EMAIL PROTECTED]> >Reply-To: u2-users@listserver.u2ug.org >To: <u2-users@listserver.u2ug.org> >Subject: RE: [U2] Universe/Linux Performance Issue >Date: Mon, 14 Feb 2005 17:41:38 -0500 > >-----Original Message----- > >I may soon become a fan of static files if this kind of thing keeps up. >Here is the output that you are referring to. > >Dynamic Files: >Slot # Inode Device Ref Count Htype Split Merge Curmod Basemod >Largerec Filesp Selects Nextsplit > 0 1900545 59425 1 20 80 50 2 2 >1628 2344 0 1 > 1 1835009 59425 110 20 80 50 1943 1024 >1628 3223064 0 920 > >[snip] > >Based on the above it looks like a fair amount of file sizing is in >order. > >Anthony > >-------------------------- > >For starters, I disagree with Dan, whom I respect greatly, about >dynamic files. He admits to hating them, I'm partial to them, but I do >not use them indiscriminately. (in MY opinion; Dan would say I overuse >them.) That said: > >Curmod being different from Basemod (analyze.shm -d columns) does not >imply needing a resize. The Basemod is just the next power of 2 below >the current modulus. It is used by the split/merge algorithm to figure >out what group to split or merge next. If Basemod is different from >Curmod, that simply means that the current modulus is not a power of 2. > >If you do resize a dynamic file, you should think about the >minimum.modulus parameter. For example, resizing them with the >minimum.modulus set to what the current modulus is would pre-allocate >enough _contiguous_ space to house the current data in DATA.30. There >is no way for resize to pre-allocate contiguous space in OVER.30. If >you let minimum.modulus default to 1, then you will get repeated splits >as the file grows, and that space could be all over the disk. In a >modern RAID environment, I'm not so sure that matters. > >Another approach would be to use a unix cp or mv, or dos copy. I >*think* (someone correct me, please!) that the OS looks for enough >contiguous space to hold the entire file when deciding where to put it. > >cds >------- >u2-users mailing list >u2-users@listserver.u2ug.org >To unsubscribe please visit http://listserver.u2ug.org/ ------- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ ------- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
