Most disks and disk systems cache huge amounts of information these days, and, depending on 20 factors or so, one solution will be better than another for a given file.
For the wholesale, SELECT F WITH...., The fewest disk records will almost always win. For files that have ~10 records/group and have ~10% of the groups overflowed, then perhaps 1% of record reads will do a second read for the overflow buffer because the target key was not in the primary group. Writing a new record would possibly hit the 10% mark for reading overflow buffers. But lowering the split.load will increase the number of splits slightly, and increase the total number of groups considerably. What you have shown is that you need to increase the the modulus (and select time) of a large file more than 10% in order to decrease the read and update times for you records 0.5% of the time (assuming, that you have only reduced the number of overflow groups by ~50%.) As Charles suggests, this is an interesting exercise, but your actual results will rapidly change if you actually add /remove records from your file, change the load or number of files on your system, put in a new drive, cpu, memory board, or install a new release of Universe, move to raid, etc. -Rick -----Original Message----- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Thursday, July 05, 2012 2:38 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] RESIZE - dynamic files The hardward "look ahead" of the disk drive reader will grab consecutive "frames" into memory, since it assumes you'll want the "next" frame next. So the less overflow you have, the faster a full file scan will become. At least that's my theory ;) -----Original Message----- From: Rick Nuckolls <r...@lynden.com> To: 'U2 Users List' <u2-users@listserver.u2ug.org> Sent: Thu, Jul 5, 2012 2:29 pm Subject: Re: [U2] RESIZE - dynamic files Chris, For the type of use that you described earlier; BASIC selects and reads, educing overflow will have negligible performance benefit, especially compared o changing the GROUP.SIZE back to 1 (2048) bytes. If you purge the file in elatively small percentages, then it will never merge anyway (because you will eed to delete 20-30% of the file for that to happen with the mergeload at 50%, o your optimum minimum modulus solution will probably be "how ever large it rows" The overhead for a group split is not as bad as it sounds unless your pdates/sec count is extremely high, such as during a copy. If you do regular SELECT and SCANS of the entire file, then your goal should be o reduce the total disk size of the file, and not worry much about common verflow. The important thing is that the file is dynamic, so you will never ncounter the issues that undersized statically hashed files develop. We have thousands of dynamically hashed files on our (Solaris) systems, with an xtremely low problem rate. Rick -----Original Message----- rom: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] n Behalf Of Chris Austin ent: Thursday, July 05, 2012 11:21 AM o: u2-users@listserver.u2ug.org ubject: Re: [U2] RESIZE - dynamic files ick, You are correct, I should be using the smaller size (I just haven't changed it et). Based on the reading I have done you should nly use the larger group size when the average record size is greater than 1000 ytes. As far as being better off with the defaults that's basically what I'm trying to est (as well as learn how linear hashing works). I was able o reduce my overflow by 18% and I only increased my empty groups by a very mall amount as well as only increased my file size y 8%. This in theory should be better for reads/writes than what I had before. To test the performance I need to write a ton of records and then capture the utput and compare the output using timestamps. Chris From: r...@lynden.com To: u2-users@listserver.u2ug.org Date: Thu, 5 Jul 2012 09:22:02 -0700 Subject: Re: [U2] RESIZE - dynamic files Chis, I still am wondering what is prompting you to continue using the larger group ize. I think that Martin, and the UV documentation is correct in this case; you ould be as well or better off with the defaults. -Rick On Jul 5, 2012, at 9:13 AM, "Martin Phillips" <martinphill...@ladybridge.com> rote: coming > Hi, > > The various suggestions about setting the minimum modulus to reduce overflow re all very well but effectively you are turning a > dynamic file into a static one, complete with all the continual maintenance ork needed to keep the parameters in step with the > data. > > In most cases, the only parameter that is worth tuning is the group size to ry to pack things nicely. Even this is often fine left > alone though getting it to match the underlying o/s page size is helpful. > > I missed the start of this thread but, unless you have a performance problem r are seriously short of space, my recommendation > would be to leave the dynamic files to look after themselves. > > A file without overflow is not necessarily the best solution. Winding the plit load down to 70% means that at least 30% of the file > is dead space. The implication of this is that the file is larger and will ake more disk reads to process sequentially from one end > to the other. > > > Martin Phillips > Ladybridge Systems Ltd > 17b Coldstream Lane, Hardingstone, Northampton NN4 6DB, England > +44 (0)1604-709200 > > > > -----Original Message----- > From: u2-users-boun...@listserver.u2ug.org > [mailto:u2-users-boun...@listserver.u2ug.org] n Behalf Of Chris Austin > Sent: 05 July 2012 15:19 > To: u2-users@listserver.u2ug.org > Subject: Re: [U2] RESIZE - dynamic files > > > I was able to drop from 30% overflow to 12% by making 2 changes: > > 1) changed the split from 80% to 70% (that alone reduce 10% overflow) > 2) changed the MINIMUM.MODULUS to 118,681 (calculated this way -> [ (record ata + id) * 1.1 * 1.42857 (70% split load)] / 4096 ) > > My disk size only went up 8%.. > > My file looks like this now: > > File name .................. GENACCTRN_POSTED > Pathname ................... GENACCTRN_POSTED > File type .................. DYNAMIC > File style and revision .... 32BIT Revision 12 > Hashing Algorithm .......... GENERAL > No. of groups (modulus) .... 118681 current ( minimum 118681, 140 empty, > 14431 overflowed, 778 badly ) > Number of records .......... 1292377 > Large record size .......... 3267 bytes > Number of large records .... 180 > Group size ................. 4096 bytes > Load factors ............... 70% (split), 50% (merge) and 63% (actual) > Total size ................. 546869248 bytes > Total size of record data .. 287789178 bytes > Total size of record IDs ... 21539538 bytes > Unused space ............... 237532340 bytes > Total space for records .... 546861056 bytes > > Chris > > > >> From: keith.john...@datacom.co.nz >> To: u2-users@listserver.u2ug.org >> Date: Wed, 4 Jul 2012 14:05:02 +1200 >> Subject: Re: [U2] RESIZE - dynamic files >> >> Doug may have had a key bounce in his input >> >>> Let's do the math: >>> >>> 258687736 (Record Size) >>> 192283300 (Key Size) >>> ======== >> >> The key size is actually 19283300 in Chris' figures >> >> Regarding 68,063 being less than the current modulus of 82,850. I think he answer may lie in the splitting process. >> >> As I understand it, the first time a split occurs group 1 is split and its ontents are split between new group 1 and new group 2. > All the other groups effectively get 1 added to their number. The next split s group 3 (which was 2) into 3 and 4 and so forth. A > pointer is kept to say where the next split will take place and also to help ort out how to adjust the algorithm to identify which > group matches a given key. >> >> Based on this, if you started with 1000 groups, by the time you have split he 500th time you will have 1500 groups. The first > 1000 will be relatively empty, the last 500 will probably be overflowed, but ot terribly badly. By the time you get to the 1000th > split, you will have 2000 groups and they will, one hopes, be quite easonably spread with very little overflow. >> >> So I expect the average access times would drift up and down in a cycle. he cycle time would get longer as the file gets bigger > but the worst time would be roughly the the same each cycle. >> >> Given the power of two introduced into the algorithm by the before/after he split thing, I wonder if there is such a need to > start off with a prime? >> >> Regards, Keith >> >> PS I'm getting a bit Tony^H^H^H^Hverbose nowadays. >> >> _______________________________________________ >> U2-Users mailing list >> U2-Users@listserver.u2ug.org >> http://listserver.u2ug.org/mailman/listinfo/u2-users > > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ______________________________________________ 2-Users mailing list 2-us...@listserver.u2ug.org ttp://listserver.u2ug.org/mailman/listinfo/u2-users ______________________________________________ 2-Users mailing list 2-us...@listserver.u2ug.org ttp://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
