Re: operator new no longer routes through rtl_AllocMemory in libsalcpprt under gbuild link rules

[Stephan Bergmann]

On 03/27/2012 12:58 PM, Caolán McNamara wrote:

On Thu, 2012-03-22 at 14:24 +0100, Stephan Bergmann wrote:

I unfortunately lost track long ago which platforms we use our own
memory allocator on, which platforms we additionally use libsalcpprt on,
and what the rationale was for the exceptions.


I'm now inclined to leave things as they are. Given that MacOSX and
Windows didn't route operator new through the rtl_Alloc family it
doesn't seem to make a massive pile of sense to me to have the Linux one
do so on its own without an evidence-based reason to do so again.


 
"[API CHANGE] Remove salcpprt static library"


___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: operator new no longer routes through rtl_AllocMemory in libsalcpprt under gbuild link rules

[Stephan Bergmann]

On 05/18/2012 04:23 PM, Michael Stahl wrote:

checking "a massive allocation" doesn't really tell us all that much,
because allocators typically operate in 2 modes:
- for large objects, directly request zero-backed memory via mmap
- for small objects, maintain elaborate data structures to allocate
these from arenas or whatever


I had hoped to address this in the test code in my other post with 
100,000 times allocating 10,000 bytes each.  However, further inspection 
reveals that allocation is apparently brk-based there, so that it can 
only ever shrink memory back to the highest allocation that has not yet 
been freed.  (That is, changing the second, deleting, loop in my test 
code to run short one item prevents any shrinkage.)  Contrast that with 
the always-mmap approach of rtl/alloc.h, which allows for different 
shrinkage patterns.  (Whether or not that is relevant in practice.)


Stephan
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: operator new no longer routes through rtl_AllocMemory in libsalcpprt under gbuild link rules

[Michael Stahl]

On 18/05/12 15:38, Arnaud Versini wrote:
> Hi,
> 
> I checked on Linux and Windows, the memory after a massive allocation
> and a massive deallocation is really freed, so currently la raison
> d'être of our internal allocator is gone (need to be checked on Mac Os
> X). About performance I don't know.

checking "a massive allocation" doesn't really tell us all that much,
because allocators typically operate in 2 modes:
- for large objects, directly request zero-backed memory via mmap
- for small objects, maintain elaborate data structures to allocate
these from arenas or whatever

in the case where a single object is backed by its own mmap it is
trivial for the allocator to unmap the memory on free, and i would
expect every system allocator to do that (even 10 years ago); the far
more interesting case is what happens when a large number of small
objects of different sizes get allocated: is the allocator smart enough
to release the memory when all, or a substantial number of them, are
freed again?

> Currently there is a mix of system allocator (or jemalloc) and our
> internal allocator. I don't think it is optimal.
> 
> I propose to delete all our allocation code and use small stubs for ABI
> compatibility until LibreOffice 4.0, and we can use the system allocator
> or jemalloc if there is performance issues, perhaps on Windows.
> 
> I ask you if I could see if it is a good idea to go on this way or not,
> and how (branch or trunk).

i don't think there is any pressing need to do this right now, but of
course you can experiment a bit, i'm also not happy that we maintain our
own allocator, but i'd like to wait for LO4 with actually doing a change.

___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: operator new no longer routes through rtl_AllocMemory in libsalcpprt under gbuild link rules

[Stephan Bergmann]

On 05/18/2012 03:38 PM, Arnaud Versini wrote:

I checked on Linux and Windows, the memory after a massive allocation
and a massive deallocation is really freed, so currently la raison
d'être of our internal allocator is gone (need to be checked on Mac Os
X). About performance I don't know.


Indeed, on Fedora 16 at least,

$ cat test.cc
#include 
#include 
int main() {
char * p[10];
for (int i = 0; i != 10; ++i) p[i] = new char[1];
std::cout << "...";
std::cin.ignore();
for (int i = 0; i != 10; ++i) delete p[i];
std::cout << "...";
std::cin.ignore();
}
$ g++ test.cc
$ ./a.out

and "pmap ... | grep total" at the two checkpoints reveals a 
shrink-again allocation mechanism (from 990752K back to 12736K here).


Stephan
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: operator new no longer routes through rtl_AllocMemory in libsalcpprt under gbuild link rules

[Arnaud Versini]

Hi,

I checked on Linux and Windows, the memory after a massive allocation and a
massive deallocation is really freed, so currently la raison d'être of our
internal allocator is gone (need to be checked on Mac Os X). About
performance I don't know.

Currently there is a mix of system allocator (or jemalloc) and our internal
allocator. I don't think it is optimal.

I propose to delete all our allocation code and use small stubs for ABI
compatibility until LibreOffice 4.0, and we can use the system allocator or
jemalloc if there is performance issues, perhaps on Windows.

I ask you if I could see if it is a good idea to go on this way or not, and
how (branch or trunk).

Thanks in advance


2012/3/27 Stephan Bergmann 

>
> As discussed offline, and for the record:  The original raison d'être for
> our home-grown memory allocation machinery, IIRC, was loading certain Calc
> documents, which temporarily required large amounts of heap memory that
> could be released again later on (either after loading was complete or when
> closing the specific document; I do not remember).  As traditional malloc
> implementations only ever grow a process's heap and never shrink it again,
> this meant that an soffice process's memory requirements as seen by the OS
> could be excessively large, needlessly degrading overall system behaviour.
>
> At least to me, it always felt kind of wrong to include memory management
> facilities in our code base. -- It increases maintenance cost and cuts us
> off advancements in the OSs' stock memory management facilities.
>
> I have no idea if there are still any scenarios where our home-grown
> memory management is beneficial, let alone if such scenarios would depend
> on our global new/delete handlers (i.e., if such scenarios obtain the
> relevant memory via stock C++ new, instead of directly via
> rtl_allocateMemory or explicitly defined operator new instances that in
> turn call rtl_allocateMemory).




-- 
Arnaud Versini
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: operator new no longer routes through rtl_AllocMemory in libsalcpprt under gbuild link rules

[Stephan Bergmann]

On 03/27/2012 12:58 PM, Caolán McNamara wrote:

On Thu, 2012-03-22 at 14:24 +0100, Stephan Bergmann wrote:

I unfortunately lost track long ago which platforms we use our own
memory allocator on, which platforms we additionally use libsalcpprt on,
and what the rationale was for the exceptions.


I'm now inclined to leave things as they are. Given that MacOSX and
Windows didn't route operator new through the rtl_Alloc family it
doesn't seem to make a massive pile of sense to me to have the Linux one
do so on its own without an evidence-based reason to do so again.


As discussed offline, and for the record:  The original raison d'être 
for our home-grown memory allocation machinery, IIRC, was loading 
certain Calc documents, which temporarily required large amounts of heap 
memory that could be released again later on (either after loading was 
complete or when closing the specific document; I do not remember).  As 
traditional malloc implementations only ever grow a process's heap and 
never shrink it again, this meant that an soffice process's memory 
requirements as seen by the OS could be excessively large, needlessly 
degrading overall system behaviour.


At least to me, it always felt kind of wrong to include memory 
management facilities in our code base. -- It increases maintenance cost 
and cuts us off advancements in the OSs' stock memory management facilities.


I have no idea if there are still any scenarios where our home-grown 
memory management is beneficial, let alone if such scenarios would 
depend on our global new/delete handlers (i.e., if such scenarios obtain 
the relevant memory via stock C++ new, instead of directly via 
rtl_allocateMemory or explicitly defined operator new instances that in 
turn call rtl_allocateMemory).


Stephan
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: operator new no longer routes through rtl_AllocMemory in libsalcpprt under gbuild link rules

[Caolán McNamara]

On Thu, 2012-03-22 at 14:24 +0100, Stephan Bergmann wrote:
> I unfortunately lost track long ago which platforms we use our own 
> memory allocator on, which platforms we additionally use libsalcpprt on, 
> and what the rationale was for the exceptions.

I'm now inclined to leave things as they are. Given that MacOSX and
Windows didn't route operator new through the rtl_Alloc family it
doesn't seem to make a massive pile of sense to me to have the Linux one
do so on its own without an evidence-based reason to do so again.

C.

___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: operator new no longer routes through rtl_AllocMemory in libsalcpprt under gbuild link rules

[Stephan Bergmann]

On 03/22/2012 02:13 PM, Caolán McNamara wrote:

On Thu, 2012-03-22 at 13:43 +0100, Michael Stahl wrote:

i don't care that much about which memory allocator we use, but well the
options are 1) link salcpprt into every executable (i.e. do it in
solenv) 2) link it just into soffice.bin 3) don't use it at all.


If we do it at all I guess just for soffice.bin is sufficient really.


The uno and unopkg executables come to mind.


iirc the custom allocator was only used on Linux anyway because given
the way Windows DLLs work it's not possible to override operator new for
the whole process


aha! I guess from the old solenv files then that libsalcpprt was linked
in for all the unix-gcc using platforms, except macosx. I wonder if
macosx was always a deliberate exclusion for some reason, or just
accidental.


I unfortunately lost track long ago which platforms we use our own 
memory allocator on, which platforms we additionally use libsalcpprt on, 
and what the rationale was for the exceptions.


Stephan
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: operator new no longer routes through rtl_AllocMemory in libsalcpprt under gbuild link rules

[Caolán McNamara]

On Thu, 2012-03-22 at 13:43 +0100, Michael Stahl wrote:
> i don't care that much about which memory allocator we use, but well the
> options are 1) link salcpprt into every executable (i.e. do it in
> solenv) 2) link it just into soffice.bin 3) don't use it at all.

If we do it at all I guess just for soffice.bin is sufficient really.

> iirc the custom allocator was only used on Linux anyway because given
> the way Windows DLLs work it's not possible to override operator new for
> the whole process

aha! I guess from the old solenv files then that libsalcpprt was linked
in for all the unix-gcc using platforms, except macosx. I wonder if
macosx was always a deliberate exclusion for some reason, or just
accidental.

> also, do we have a configure option to switch allocators?  that should
> be respected in any case if it's set.

We do indeed, but that basically affects what
rtl_allocateMemory/rtl_freeMemory points to, either custom impls or
malloc/free. So, when linking to salcpprt, new/delete ends up backing
onto whatever is requested by --with-alloc via
rtl_allocateMemory/rtl_freeMemory, so no particular extra action
required there.

C.

___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: operator new no longer routes through rtl_AllocMemory in libsalcpprt under gbuild link rules

[Michael Stahl]

On 22/03/12 13:14, Caolán McNamara wrote:
> Traditionally we linked our binaries with libsalcpprt which provides
> global operator new/delete etc to reroute those allocations through our
> sal rtl_allocateMemory and rtl_freeMemory
> 
> Somewhere along the line, perhaps gbuild conversions, this has gotten
> lost. I assume this is an oversight, and not deliberate ? I see very
> little mention of salcpprt in solenv/gbuild but you can see all the
> remnants in the older build system in solenv. i.e. by default we linked
> to it when creating executables with some executables excluded from the
> scheme.

ah that was hacked into solenv directly? didn't know that.

> e.g. attached is a hack to link soffice.bin under Linux to salcpprt to
> make it work again for me locally for some hacking.

looks fine for a local fix except perhaps it would be better style to
use gb_Executable_add_libs instead of gb_Executable_add_ldflags.

> Should we restore linking to libsalcpprt by default again ? I'm unsure
> what the windows link lines should be.

i don't care that much about which memory allocator we use, but well the
options are 1) link salcpprt into every executable (i.e. do it in
solenv) 2) link it just into soffice.bin 3) don't use it at all.

iirc the custom allocator was only used on Linux anyway because given
the way Windows DLLs work it's not possible to override operator new for
the whole process, you'd have to link it into every DLL. (also, iirc on
windows in general it's necessary that objects are freed by the DLL that
allocated them because every DLL could have different allocator... well
at least all the ones built by our build system probably use the same
allocator in practice).

also, do we have a configure option to switch allocators?  that should
be respected in any case if it's set.

___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice

operator new no longer routes through rtl_AllocMemory in libsalcpprt under gbuild link rules

[Caolán McNamara]

Traditionally we linked our binaries with libsalcpprt which provides
global operator new/delete etc to reroute those allocations through our
sal rtl_allocateMemory and rtl_freeMemory

Somewhere along the line, perhaps gbuild conversions, this has gotten
lost. I assume this is an oversight, and not deliberate ? I see very
little mention of salcpprt in solenv/gbuild but you can see all the
remnants in the older build system in solenv. i.e. by default we linked
to it when creating executables with some executables excluded from the
scheme.

e.g. attached is a hack to link soffice.bin under Linux to salcpprt to
make it work again for me locally for some hacking.

Should we restore linking to libsalcpprt by default again ? I'm unsure
what the windows link lines should be.

C.
diff --git a/desktop/Executable_soffice.bin.mk b/desktop/Executable_soffice.bin.mk
index e0c5891..5c86c92 100644
--- a/desktop/Executable_soffice.bin.mk
+++ b/desktop/Executable_soffice.bin.mk
@@ -100,6 +100,18 @@ $(eval $(call gb_Executable_set_ldflags,\
 
 endif
 
+#TO-DO fixme, pre-gbuild, executables, unless explicitly
+#marked otherwise, default linked to salcpprt which
+#contains the global new stuff which reroutes new
+#to our internal allocators
+ifeq ($(COM)-$(OS),GCC-LINUX)
+
+$(eval $(call gb_Executable_add_ldflags,$(sofficebin),\
+-Wl$(COMMA)--whole-archive -lsalcpprt -Wl$(COMMA)--no-whole-archive \
+))
+
+endif
+
 $(eval $(call gb_Executable_add_nativeres,$(sofficebin),sofficebin/src))
 
 # vim: set ts=4 sw=4 et:
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice