Thanks Stefan, > I'll leave the further interpretation of the above results to > the interested recipient / reader.
I will give it a try, then. > S08-64: > SR QR SU QU[1] QU[2] > mh 659m34s 1m09s 81m06s ERROR - > mH - - 383m17s ERROR - > Mh - - 77m03s 344m32s 1m34s > MH 644m01s 0m49s 390m42s 342m47s 1m36s These h/SU results are strange, because a 5 x faster time due to a hash function with more collisions is.. puzzling. In any case, with 64-bits oids on a 8GB machine, the shredding generates very intense swapping, and Linux performance under swapping is known to be highly variable. Difficult to draw conclusions. I am still investigating the behavior of MT_mmap under load and it may well be that shredding performance can be improved by better memory management (less swapping). From the result below we know that it can be done in 25m. The not-yet checked in improvements in index construction will strongly reduce the QU[1] time by taking less memory; in the order of the 100m seen below. > S16-32: > SR QR SU QU[1] QU[2] > mh 127m59s 0m17s 43m14s ERROR - > mH 110m33s 0m16s 26m26s ERROR - > Mh 128m11s 0m18s 44m00s 100m50s 1m15s > MH 191m42s 0m17s 25m43s 101m37s 1m21s These numbers show that MonetDB prefers to operate on RAM resident data. I am glad you reproduced my 25 minutes h/SU result. Must be a coicidence that the mini hash self-join benchmark on 50M string numbers ran in 26s with the new hash and 43s with the old hash (we see the same numbers in minutes here repeated for shredding). In my current version, QU[2] with the ordered indices is back at 17s. Reason for 1m15s is that you tend to loose the hash-table on the index bats due to swapping. So that includes hash table creation. Whether QU[1] is acceptable is highly questionable. Starting the database takes 1.5 hours!! I am seriously considering to switch to persistent ordered indices for the updatable case. For that to happen, the worksing set needs to be extended with persistent delta-bats on those indices, changes to these deltas need to be logged (in order to make the indices recoverable). > -----Original Message----- > From: Stefan Manegold [mailto:[EMAIL PROTECTED] > Sent: Wednesday, October 17, 2007 1:54 PM > To: [email protected] > Cc: Peter Boncz; Peter Boncz > Subject: Re: [Monetdb-checkins] MonetDB/src/gdk gdk_atoms.mx, > MonetDB_1-20, 1.134, 1.134.6.1 gdk_posix.mx, MonetDB_1-20, > 1.143, 1.143.2.1 > > > Just for the records: > > I finally managed to finsh my experiments regarding > [ 1811229 ] [ADT] Adding large document, with update support > http://sourceforge.net/tracker/index.php?func=detail&aid=18112 > 29&group_id=56967&atid=482468 > and the related code changes. For those interested, here's > the detailed > story: > > > "S08-64" System (beo-24): > - 2x 64-bit Dual-Core Opteron270 @ 2 Ghz > - 8 GB memory > - MonetDB/XQuery 0.20, 64-bit, 64-bit OIDs, --enable-optimize > (gcc 4.1.2) > > "S16-32" System (core-1): > - 4x 64-bit Dual-Core Opteron870 @ 2 Ghz > - 16 GB memory > - MonetDB/XQuery 0.20, 64-bit, 32-bit OIDs, --enable-optimize > (gcc 4.1.2) > > Document: > http://mirror.openstreetmap.nl/planet/planet-071003.osm.bz2 > (extracted: 19 GB XML file) > > "SR" Shredding read-only: > pf:add-doc(".../planet-071003.osm","planet-071003.osm") > > "SU" Shredding updateable: > pf:add-doc(".../planet-071003.osm","planet-071003.osm","planet > -071003.osm",5) > > "QR"/"QU" Count query: > count(doc("planet-071003.osm")//*) > > Configurations: > m: without Peter's mmap fix in gdk_posix.mx > (i.e., using rev. 1.143 of gdk_posix.mx) > M: with Peter's mmap fix in gdk_posix.mx > (i.e., using rev. 1.143.2.1 of gdk_posix.mx) > h: without Peter's new string hash function in gdk_atoms.mx > (i.e., using rev. 1.134 of gdk_atoms.mx) > H: with Peter's new string hash function in gdk_atoms.mx > (i.e., using rev. 1.134.6.1 of gdk_atoms.mx) > > > Results (wall-clock times): > > S08-64: > SR QR SU QU[1] QU[2] > mh 659m34s 1m09s 81m06s ERROR - > mH - - 383m17s ERROR - > Mh - - 77m03s 344m32s 1m34s > MH 644m01s 0m49s 390m42s 342m47s 1m36s > > S16-32: > SR QR SU QU[1] QU[2] > mh 127m59s 0m17s 43m14s ERROR - > mH 110m33s 0m16s 26m26s ERROR - > Mh 128m11s 0m18s 44m00s 100m50s 1m15s > MH 191m42s 0m17s 25m43s 101m37s 1m21s > > (NB: "SR" includes building of indices, while "SU" does not; > consequently, "QR" can exploit the indices built during > "SR", while "QU[1]" > has to build the indices first, and only "QU[2]" can exploit them.) > > > Apparently, the mmap fix in gdk_posix.mx seems to be > sufficient to prevent > the remap-ERROR reported (for a system & configuration > similar to "S08-64") > in > [ 1811229 ] [ADT] Adding large document, with update support > http://sourceforge.net/tracker/index.php?func=detail&aid=18112 > 29&group_id=56967&atid=482468 > > I'll leave the further interpretation of the above results to > the interested > recipient / reader. > > > Stefan > > ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Monetdb-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/monetdb-developers
