Hi, The mmap method is very costly on Windows as it does not use a virtual file with holes, but really creates it first. This may also be the case on macos. However, the pert degradation on Linux is unexpected. Maybe newer kernels have introduced different behavior.
I think the principle of falling back on mmap is sound, however the threshold needs to be increased. Maybe at a fixed perc (20%?) of available ram. Peter On Feb 19, 2009, at 10:49 PM, "Stefan Manegold" <[email protected]> wrote: > On Tue, Feb 10, 2009 at 09:13:21AM +0000, Stefan Manegold wrote: >> Update of /cvsroot/monetdb/MonetDB/src/gdk >> In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv18180/MonetDB/ >> src/gdk >> >> Modified Files: >> Tag: Feb2009 >> gdk_heap.mx >> Log Message: >> >> [on behalf of Peter] re-doing >> >> 2009/02/05 - boncz: MonetDB/src/gdk/gdk_heap.mx,1.108 >> - make large allocs fall back again on mmap > > Peter, > > (how) can we make the limit (more) dependend on / adaptive to the > actual > memory size? > > Shouldn't this have an effect only with blocks > gdk_vm_minsize == > 128 GB ? > > I have sample scenarios (on Linux) where, e.g., queries produce > intermediate > results of ~ 5GB --- on my 64-bit 8GB machine those queries run in > 40 secs > before this change (i.e., with h->storage = STORE_MMAP;), but take > 200 secs > (i.e., 5 times as long) after this change (i.e., with h->newstorage = > STORE_MMAP;) ... > > Jan R. has seen similar effects with XMark on his Mac ... > > Thanks, > > Stefan > >> now only the actually intended changes >> and as bug fix, i.e., in the Feb2009 release branch >> >> >> Index: gdk_heap.mx >> =================================================================== >> RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_heap.mx,v >> retrieving revision 1.107 >> retrieving revision 1.107.2.1 >> diff -u -d -r1.107 -r1.107.2.1 >> --- gdk_heap.mx 22 Jan 2009 14:47:53 -0000 1.107 >> +++ gdk_heap.mx 10 Feb 2009 09:13:19 -0000 1.107.2.1 >> @@ -115,7 +115,7 @@ >> fp = GDKfilelocate(nme, "wb", ext); >> if (fp != NULL) { >> fclose(fp); >> - h->storage = STORE_MMAP; >> + h->newstorage = STORE_MMAP; >> HEAPload(h, nme, ext, FALSE); >> } >> GDKfree(of); >> >> >> --- >> --- >> --- >> --------------------------------------------------------------------- >> Create and Deploy Rich Internet Apps outside the browser with >> Adobe(R)AIR(TM) >> software. With Adobe AIR, Ajax developers can use existing skills >> and code to >> build responsive, highly engaging applications that combine the >> power of local >> resources and data with the reach of the web. Download the Adobe >> AIR SDK and >> Ajax docs to start building applications today-http://p.sf.net/sfu/ >> adobe-com >> _______________________________________________ >> Monetdb-checkins mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > -- > | Dr. Stefan Manegold | mailto:[email protected] | > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > | The Netherlands | Fax : +31 (20) 592-4312 | > > No virus found in this incoming message. > Checked by AVG - www.avg.com > Version: 8.0.237 / Virus Database: 270.10.24/1954 - Release Date: 02/17/09 07 > :07:00 ------------------------------------------------------------------------------ Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise -Strategies to boost innovation and cut costs with open source participation -Receive a $600 discount off the registration fee with the source code: SFAD http://p.sf.net/sfu/XcvMzF8H _______________________________________________ Monetdb-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/monetdb-developers
