On 2008-02-11 19:05, Peter Boncz wrote:
> Hi Stefan
>
> Thanks for your response. However, as I have said, but let me say it as clear
> as possble: I am *sure* there is a bug still unfound in the way re-allocations
> are done when mem_bigsize redirects it into virtual memory.
>
> For this reason, at this stage, with the goal of a stable release, it is just
> my recommendation *not* to re-enable it.
>
> In M5 nor in M5.
I think Stefan meant reenable access to the variable in M5 that Martin
turned off, not reenable the GDKvmalloc switch that you turned off.
> Peter
>
>
>
>
> Dear all,
>
> On Mon, Feb 11, 2008 at 02:54:59PM +0100, Peter Boncz wrote:
>> Hi Stefan,
>>
>> The pftijah bug was unrelated to vmalloc(), a relationship that I never
>> assumed.
>
> Indeed, that was (only) my mis-interpretation. Sorry.
>
>> The vmalloc() troubles occured, as you may recall, during the data
>> preparation of the sorted 100GB TPC-H.
>>
>> As it is entirely foreseeable that we at some point again will re-encounter
>> an
>> OS with a malloc implementation that is prone to fragmentation, I'd keep the
>> vmalloc() code in for the moment (though in disabled state). Linux and
>> Windows
>> do not need it.
>>
>> So, my advise is to for the monet let vmalloc() be disabled, until the issue
>> is properly investigated and a bugfix is found. Let's call it option (0).
>
> Ok.
>
> Just to sum up the current state.
> Please check, whether this is OK and consistent:
>
>
> a)
> Documentation in MonetDB4/conf/MonetDB.conf.in &
> MonetDB5/conf/monetdb5.conf.in says:
> "
> # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to
> # 1/2 of the physically available amount of main-memory
> # during start-up in src/tools/Mserver.mx
> # memory chunks of size >= gdk_mem_bigsize (in bytes) will be mmaped
> anonymously
> #gdk_mem_bigsize=262144
> # memory chunks of size >= gdk_vm_minsize (in bytes) will be mmaped;
> #gdk_vm_minsize=137438953472
> "
>
> b)
> Documentation in ./monetweb/Docs/XQuery/MonetDB.conf.texi says
> "
> @item @code{gdk_mem_bigsize}: minimum size for memory-mapped columns, e.g.
> @code{262144}.
> @item @code{gdk_vm_minsize}: column size above which memory mapped files are
> used always, e.g. @code{1749291171}
> "
>
> c)
> MonetDB/src/common/monet_options.mx sets
> "
> set[i].kind = opt_builtin;
> set[i].name = strdup("gdk_mem_bigsize");
> set[i].value = strdup("262144");
> "
>
> d)
> MonetDB/src/common/monet_options.py.in sets
> "
> # gdk_mem_bigsize & gdk_vm_minsize will be set/limited to
> # 1/2 of the physically available amount of main-memory
> # during start-up in src/tools/Mserver.mx
> gdk_mem_bigsize = '256K'
> gdk_vm_minsize = '128G'
> "
>
> e)
> GDKinit() in MonetDB/src/gdk/gdk_utils.mx sets
> "
> if ((p = GDKgetenv("gdk_mem_bigsize"))) {
> /* when allocating >6% of all RAM; do so using vmalloc() iso
> malloc() */
> lng max_mem_bigsize = GDK_mem_maxsize/16;
>
> /* sanity check to avoid memory fragmentation */
> GDK_mem_bigsize = (size_t) MIN(max_mem_bigsize, strtol(p,
> NULL, 10));
> }
> "
>
> f)
> GDKmallocmax() in MonetDB/src/gdk/gdk_utils.mx ignores GDK_mem_bigsize
>
> g)
> HEAPalloc() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows:
> "
> /* when using anonymous vm we malloc we need 64K chunks, also we
> * 20% extra malloc */
> if (h->size > GDK_mem_bigsize) {
> h->maxsize = (size_t) ((double) h->maxsize * BATMARGIN) - 1;
> h->maxsize = (1 + (h->maxsize >> 16)) << 16;
> }
> "
>
> h)
> HEAPextend() in MonetDB/src/gdk/gdk_heap.mx uses GDK_mem_bigsize as follows:
> "
> /* extend a malloced heap, possibly switching over to
> file-mapped storage */
> Heap bak = *h;
> int can_mmap = (h->filename && size >= GDK_mem_bigsize);
> int must_mmap = can_mmap && (size >= GDK_vm_minsize ||
> (h->newstorage != STORE_MEM));
> [...]
> if (can_mmap) {
> /* in anonymous vm, if have to realloc anyway, we
> reserve some extra space */
> if (size > h->maxsize) {
> h->maxsize = (size_t) ((double) size *
> BATMARGIN);
> }
> /* when using anonymous vm we malloc we need 64K
> chunks */
> h->maxsize = (1 + ((h->maxsize - 1) >> 16)) << 16;
> [...]
> /* too big: convert it to a disk-based temporary heap */
> if (can_mmap) {
> char privext[PATHLENGTH], *of = h->filename;
> FILE *fp;
>
> h->filename = NULL;
> sprintf(privext, "%s.priv", ext);
> fp = GDKfilelocate(nme, "wb", privext);
> if (fp != NULL) {
> fclose(fp);
> /* a non-persistent heap: we create a .priv
> but *not* MMAP_PRIV !!! */
> h->storage = STORE_MMAP;
> h->base = NULL;
> if (HEAPload(h, nme, privext, FALSE) >= 0) {
> memcpy(h->base, bak.base, bak.free);
> HEAPfree(&bak);
> return 0;
> }
> }
> GDKfree(of);
> }
> "
>
> i)
> MIL offers mem_bigsize() & mem_bigsize(lng) to get / set mem_bigsize
> in MonetDB4/src/modules/plain/sys.mx
>
> j)
> both MIL & MAL offer commands to set mem_maxsize and vm_minsize, however,
> only the MIL variants (still) related mem_maxsize & vm_minsize with
> mem_bigsize,
> while the MAL variants no longer do so:
>
> MonetDB4/src/modules/plain/sys.mx:
> "
> int
> set_mem_maxsize(lng *num)
> {
> @:num2sze(mem_maxsize)@
> if (sze < GDK_mem_bigsize)
> set_mem_bigsize(num);
> GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze);
> return GDK_SUCCEED;
> }
>
> int
> set_vm_minsize(lng *num)
> {
> @:num2sze(vm_minsize)@
> if (sze < GDK_mem_bigsize)
> set_mem_bigsize(num);
> GDK_vm_minsize = MAX(GDK_mem_bigsize, sze);
> return GDK_SUCCEED;
> }
> "
>
> MonetDB5/src/modules/kernel/status.mx
> "
> int
> set_mem_maxsize(lng *num)
> {
> @:num2sze(mem_maxsize)@
> #if 0
> if (sze < GDK_mem_bigsize)
> set_mem_bigsize(num);
> GDK_mem_maxsize = MAX(GDK_mem_bigsize, sze);
> #endif
> GDK_mem_maxsize = sze;
> return GDK_SUCCEED;
> }
>
> int
> set_vm_minsize(lng *num)
> {
> @:num2sze(vm_minsize)@
> #if 0
> if (sze < GDK_mem_bigsize)
> set_mem_bigsize(num);
> GDK_vm_minsize = MAX(GDK_mem_bigsize, sze);
> #endif
> GDK_vm_minsize = sze;
> return GDK_SUCCEED;
> }
> "
>
>
> If all this is considered OK and consistent, we're done.
> Otherwise, we need to do what ever is required to get the
> code and documentation consistent, preferably before the release.
>
> Stefan
>
>
>> Peter
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Dear all,
>>
>> since Roberto's bug appears to be fixed by Peter's ".priv" (vm_minsize
>> related) fixes (Thanks!), and disabling of vmalloc() (mem_bigsize related)
>> does not appear have any impact on this bug (just tested successfully both
>> with vmalloc() disabled and vmalloc() enabled), there are two ways to finish
>> this release (more or less) quickly:
>>
>> (1) finish the disabling of vmalloc() in all parts of the code as indicated
>> below (or better: remove all remains completely) --- this might need
>> some extra testing to ensure that it does not have any (significant)
>> impact on performance ...
>>
>> (2) for now (i.e., in the release branch) re-enable vmalloc(), and then
>> finish the removal of it (incl. extensive testing) in the development
>> trunk for th next release.
>>
>> Since I'm not completely aware of the impact of vmalloc(), I cannot really
>> make this decision --- the "save" back-up would IMHO be (2).
>>
>> Any other comments, opinions, suggestions, expertise?
>>
>> Thanks in advance!
>>
>> Stefan
>>
>>
>> On Sun, Feb 10, 2008 at 01:33:04PM +0100, Stefan Manegold wrote:
>>> Peter: Thanks for disabling mem_bigsize / vmalloc().
>>>
>>> All:
>>> (a)
>>> Unfortunately, this does not (yet) solve
>>> [ 1872685 ] PF(tijah): HEAPsetmmap: Assertion `p' failed.
>>> http://sourceforge.net/tracker/index.php?func=detail&aid=1872685&group_id=56967&atid=482468
>>>
>>> But this appears to be more vm_minsize related than mem_bigsize related.
>>>
>>>
>>> (b)
>>> We should check all occurances of mem_bigsize and remove / "hide" those that
>>> have become irrelevant with the deactivation of mem_bigsize / vmalloc():
>>> ========
>>> $ cvsfiles * | egrep -v ' |/Tests/' | xargs grep -n --color 'mem_bigsize'
>>> --------
>>> MonetDB4/conf/MonetDB.conf.in:29:# gdk_mem_bigsize & gdk_vm_minsize will be
>> set/limited to
>>> MonetDB4/conf/MonetDB.conf.in:32:# memory chunks of size >= gdk_mem_bigsize
>> (in bytes) will be mmaped anonymously
>>> MonetDB4/conf/MonetDB.conf.in:33:#gdk_mem_bigsize=262144
>>> MonetDB5/conf/monetdb5.conf.in:33:# gdk_mem_bigsize & gdk_vm_minsize will be
>> set/limited to
>>> MonetDB5/conf/monetdb5.conf.in:36:# memory chunks of size >= gdk_mem_bigsize
>> (in bytes) will be mmaped anonymously
>>> MonetDB5/conf/monetdb5.conf.in:37:#gdk_mem_bigsize=262144
>>> MonetDB/src/common/monet_options.mx:410: set[i].name =
>> strdup("gdk_mem_bigsize");
>>> MonetDB/src/common/monet_options.py.in:39: # gdk_mem_bigsize &
>> gdk_vm_minsize will be set/limited to
>>> MonetDB/src/common/monet_options.py.in:42: gdk_mem_bigsize = '256K'
>>> MonetDB/src/gdk/gdk_heap.mx:121: if (h->size > GDK_mem_bigsize) {
>>> MonetDB/src/gdk/gdk_heap.mx:189: int can_mmap = (h->filename
>> && size >= GDK_mem_bigsize);
>>> MonetDB/src/gdk/gdk.mx:2210:gdk_export size_t GDK_mem_bigsize; /* size
>> after which we use VM rather than heap */
>>> MonetDB/src/gdk/gdk_utils.mx:396:size_t GDK_mem_bigsize = 1 << 20;
>>> MonetDB/src/gdk/gdk_utils.mx:964:larger than GDK_mem_bigsize) to anonymous
>> virtual memory. The seamless
>>> MonetDB/src/gdk/gdk_utils.mx:1054: if (size > GDK_mem_bigsize) {
>>> MonetDB/src/gdk/gdk_utils.mx:1191: if (size <= GDK_mem_bigsize) {
>>> MonetDB/src/gdk/gdk_utils.mx:1529: if ((p =
>> GDKgetenv("gdk_mem_bigsize"))) {
>>> MonetDB/src/gdk/gdk_utils.mx:1531: lng max_mem_bigsize =
>> GDK_mem_maxsize/16;
>>> MonetDB/src/gdk/gdk_utils.mx:1534: GDK_mem_bigsize = (size_t)
>> MIN(max_mem_bigsize, strtol(p, NULL, 10));
>>> monetweb/Docs/XQuery/MonetDB.conf.texi:115:@item @code{gdk_mem_bigsize}:
>> minimum size for memory-mapped columns, e.g. @code{262144}.
>>> MonetDB4/src/modules/plain/sys.mx:96:.COMMAND mem_bigsize() : lng =
>> get_mem_bigsize;
>>> MonetDB4/src/modules/plain/sys.mx:98:.COMMAND mem_bigsize(lng) : void =
>> set_mem_bigsize;
>>> MonetDB4/src/modules/plain/sys.mx:1430:get_mem_bigsize(lng *num)
>>> MonetDB4/src/modules/plain/sys.mx:1432: *num = GDK_mem_bigsize;
>>> MonetDB4/src/modules/plain/sys.mx:1437:set_mem_bigsize(lng *num)
>>> MonetDB4/src/modules/plain/sys.mx:1439: @:num2sze(mem_bigsize)@
>>> MonetDB4/src/modules/plain/sys.mx:1440: GDK_mem_bigsize = MAX(32768, sze);
>>> MonetDB4/src/modules/plain/sys.mx:1462: if (sze < GDK_mem_bigsize)
>>> MonetDB4/src/modules/plain/sys.mx:1463: set_mem_bigsize(num);
>>> MonetDB4/src/modules/plain/sys.mx:1464: GDK_mem_maxsize =
>> MAX(GDK_mem_bigsize, sze);
>>> MonetDB4/src/modules/plain/sys.mx:1486: if (sze < GDK_mem_bigsize)
>>> MonetDB4/src/modules/plain/sys.mx:1487: set_mem_bigsize(num);
>>> MonetDB4/src/modules/plain/sys.mx:1488: GDK_vm_minsize =
>> MAX(GDK_mem_bigsize, sze);
>>> MonetDB5/src/modules/kernel/status.mx:405:get_mem_bigsize(lng *num)
>>> MonetDB5/src/modules/kernel/status.mx:407: *num = GDK_mem_bigsize;
>>> MonetDB5/src/modules/kernel/status.mx:412:set_mem_bigsize(lng *num)
>>> MonetDB5/src/modules/kernel/status.mx:414: @:num2sze(mem_bigsize)@
>>> MonetDB5/src/modules/kernel/status.mx:415: GDK_mem_bigsize = MAX(32768,
>> sze);
>>> MonetDB5/src/modules/kernel/status.mx:439: if (sze < GDK_mem_bigsize)
>>> MonetDB5/src/modules/kernel/status.mx:440:
>>> set_mem_bigsize(num);
>>> MonetDB5/src/modules/kernel/status.mx:441: GDK_mem_maxsize =
>> MAX(GDK_mem_bigsize, sze);
>>> MonetDB5/src/modules/kernel/status.mx:466: if (sze < GDK_mem_bigsize)
>>> MonetDB5/src/modules/kernel/status.mx:467:
>>> set_mem_bigsize(num);
>>> MonetDB5/src/modules/kernel/status.mx:468: GDK_vm_minsize =
>> MAX(GDK_mem_bigsize, sze);
>>> monetweb/MonetDB/QuickTour/HelloWorld/content.shtml:42:[ "gdk_mem_bigsize",
>> "262144"
>>> ========
>>>
>>> more tomorrow during MADAM ...
>>>
>>> Stefan
>>>
>>>
>>> On Sun, Feb 10, 2008 at 10:37:54AM +0100, Peter Boncz wrote:
>>>> no consequences.
>>>>
>>>>
>>>>
>>>>
>>>> Peter,
>>>>
>>>> What about the consequences in:
>>>> build/MonetDB/src/gdk/gdk_heap.c
>>>> build/MonetDB5/src/modules/kernel/status.c
>>>> build/MonetDB/src/gdk/gdk.h
>>>>
>>>> Peter Boncz wrote:
>>>>> Update of /cvsroot/monetdb/MonetDB/src/gdk
>>>>> In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29651
>>>>>
>>>>> Modified Files:
>>>>> Tag: MonetDB_1-22
>>>>> gdk_utils.mx
>>>>> Log Message:
>>>>> disable use of vmalloc()
>>>>>
>>>>>
>>>>>
>>>>> Index: gdk_utils.mx
>>>>> ===================================================================
>>>>> RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_utils.mx,v
>>>>> retrieving revision 1.206
>>>>> retrieving revision 1.206.2.1
>>>>> diff -u -d -r1.206 -r1.206.2.1
>>>>> --- gdk_utils.mx 14 Jan 2008 10:48:58 -0000 1.206
>>>>> +++ gdk_utils.mx 9 Feb 2008 15:38:11 -0000 1.206.2.1
>>>>> @@ -1050,6 +1050,7 @@
>>>>> #endif
>>>>> }
>>>>> size = (size + 7) & ~7; /* round up to a multiple of eight */
>>>>> +#if 0
>>>>> if (size > GDK_mem_bigsize) {
>>>>> size_t newsize = size + sizeof(size_t) + sizeof(size_t);
>>>>> size_t newmax = MAX(*maxsize, newsize);
>>>>> @@ -1064,6 +1065,7 @@
>>>>> *maxsize = newmax - (sizeof(size_t) + sizeof(size_t));
>>>>> return (void *) s;
>>>>> }
>>>>> +#endif
>>>>> CHKMEM(size, 0);
>>>>> GDKmalloc_prefixsize(s, size);
>>>>> if (s == NULL) {
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------------
>>>>> This SF.net email is sponsored by: Microsoft
>>>>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>>>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>>>>> _______________________________________________
>>>>> Monetdb-checkins mailing list
>>>>> [EMAIL PROTECTED]
>>>>> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
>>>>>
>>>>
>>>> -------------------------------------------------------------------------
>>>> This SF.net email is sponsored by: Microsoft
>>>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>>>> _______________________________________________
>>>> Monetdb-developers mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> | Dr. Stefan Manegold | mailto:[EMAIL PROTECTED] |
>>> | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ |
>>> | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 |
>>> | The Netherlands | Fax : +31 (20) 592-4312 |
>> --
>> | Dr. Stefan Manegold | mailto:[EMAIL PROTECTED] |
>> | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ |
>> | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 |
>> | The Netherlands | Fax : +31 (20) 592-4312 |
>>
>>
>
--
Sjoerd Mullender
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Monetdb-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-developers
