Re: Build with jemalloc
Thanks for you reply. On Sat, 31 Aug 2013 12:20:04 +0200 Michael Stahl mst...@redhat.com wrote: although as far as i vaguely remember the really bad system memory allocators are in Windows XP and earlier; the Vista and newer allocators are said to be much better. It is a interesting things. I tried to watch memory usage on XP and 7, And the output is a little different. and i would suspect if LibreOffice requires more memory than MS Office it is most likely not because of the memory allocator but because of sub-optimal data structures in the application cores (plus some constant overhead because LibreOffice is portable and thus needs more platform abstraction layers). From my inspection, VCL calls quite many malloc/free pair (in the face, new() call of some objects). UI compornent on LibreOffice supported many platform, so it woud be unavoidable. i don't know why this does not work for the specific case of soffice.bin, because i don't know what symbols are in the extra libraries you link. perhaps there are different calling conventions in the jemalloc library and the soffice.bin objects or something like that, so it still picks up the allocator symbols from msvcrt.lib. Thank you your comment. I tested building avery small program with jemalloc, and it seems fine. So I check build related things again. this does not work on Windows because DLLs do not have a global symbol namespace, their namespace is local to each DLL. and every DLL knows not only the name of an imported symbol, but also which other DLL it is imported from. so i think what you would need to do to replace the memory allocator on Windows is to link _every_ DLL and executable against your custom allocator. Actually, I also tried to modify another link commands, almost passed the build but cli_maker was not. oh, and for quite a lot of classes in the code base a custom memory allocator is used anyway due to overloading operator new for the particular class, see include/tools/mempool.hxx and the underlying rtl_cache API in include/rtl/alloc.h. this implements a SLAB like allocator which is used for the most-used classes and should be quite efficient and reduce the fragmentation considerably. Sounds good. So I understand that LibreOffice supports jemalloc in many Unix-like systems but Windows is not. And many classes are used internal memory allocator on Unicos and Windows. Is this right? ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Build with jemalloc
hello Takatsugu,
On 30/08/13 07:12, NOKUBI Takatsugu wrote:
Does anyone tried to build LibreOffice with jemallc?
I tried to use LibreOffice on Windows environment, and it required
too much memory rather than Microsoft Office.
So I tryied to inspect memory usage with Microsoft Performance Analyzer.
http://msdn.microsoft.com/en-us/library/windows/desktop/hh448170.aspx
I can find many malloc/free and new/delete pair, so I suspect
possibility of heap memory fragmantation.
it is quite possible that there could be issues with the memory
allocator, because LibreOffice is a very demanding application.
although as far as i vaguely remember the really bad system memory
allocators are in Windows XP and earlier; the Vista and newer allocators
are said to be much better.
and i would suspect if LibreOffice requires more memory than MS Office
it is most likely not because of the memory allocator but because of
sub-optimal data structures in the application cores (plus some constant
overhead because LibreOffice is portable and thus needs more platform
abstraction layers).
Then I tried to build with jemalloc, a famous memory allocator.
I modified desktop/Executable_soffice_bin.mk like the following:
$(eval $(call gb_Executable_add_ldflags,soffice_bin,\
/STACK:1000 \
/NODEFAULTLIB:msvcrt /NODEFAULTLIB:libcmt \
'c:\cygwin\libo\jemalloc\jemalloc_s.lib' \
'c:\cygwin\libo\jemalloc\libgcc.a' \
'C:\Program Files\Microsoft Visual Studio 11.0\VC\lib\msvcrt.lib' \
'C:\Program Files\Microsoft Visual Studio 11.0\VC\lib\libcmt.lib' \
))
(it’s a dirty hack)
So gbuild invoked the following command (a little bit long):
S=C:/cygwin/libo O=C:/cygwin/opt/lo/build2/solver/wntmsci14.pro
W=C:/cygwin/opt/lo/build2/workdir/wntmsci14.pro
mkdir -p $W/LinkTarget/Executable/
rm -f $W/LinkTarget/Executable/soffice_bin.exe
RESPONSEFILE=C:/cygwin/tmp/gbuild.ThJJlZ
unset INCLUDE
link -release -opt:noref -incremental:no -debug -nxcompat -dynamicbase
-manifest -SUBSYSTEM:WINDOWS,5.01-LIBPATH:$O/lib
-LIBPATH:C:/PROGRA~1/Java/JDK17~1.0_2/lib
-LIBPATH:C:/PROGRA~1/MICROS~1.0/VC/lib -LIBPATH:C:/PROGRA~1/WI3CF2~1/8.0/lib
-LIBPATH:C:/PROGRA~1/WI3CF2~1/8.0/lib/win8/um/x86
-LIBPATH:C:/PROGRA~1/MICROS~1.NET/SDK/v2.0//lib
-LIBPATH:C:/PROGRA~1/MI5E29~1/lib/x86/STACK:1000
/NODEFAULTLIB:msvcrt
/NODEFAULTLIB:libcmt
'c:\cygwin\libo\jemalloc\jemalloc_s.lib' 'c:\cygwin\libo\jemalloc\libgcc.a'
'C:\Program Files\Microsoft Visual Studio 11.0\VC\lib\msvcrt.lib'
'C:\Program Files\Microsoft Visual Studio 11.0\VC\lib\libcmt.lib'
@${RESPONSEFILE} isal.lib isofficeapp.lib iuwinapi.lib ooopathutils.lib
winextendloaderenv.lib advapi32.lib user32.lib
-out:$W/LinkTarget/Executable/soffice_bin.exe; RC=$?; rm ${RESPONSEFILE}
if [ -f $W/LinkTarget/Executable/soffice_bin.exe.manifest ]; then mt.exe
-nologo -manifest $W/LinkTarget/Executable/soffice_bin.exe.manifest
outputresource:$W/LinkTarget/Executable/soffice_bin.exe\;1
touch -r $W/LinkTarget/Executable/soffice_bin.exe
W/LinkTarget/Executable/soffice_bin.exe.manifest; fi
if [ -f $S/solenv/inc/DeclareDPIAware.manifest ]; then
mt.exe -nologo -manifest $S/solenv/inc/DeclareDPIAware.manifest
-updateresource:$W/LinkTarget/Executable/soffice_bin.exe\;1 ;
fi ; exit $RC
I thknk it seems fine, but the output binary hadn’t have static jemalloc code.
i don't know why this does not work for the specific case of
soffice.bin, because i don't know what symbols are in the extra
libraries you link. perhaps there are different calling conventions in
the jemalloc library and the soffice.bin objects or something like that,
so it still picks up the allocator symbols from msvcrt.lib.
Does anyone tried same thing?
Or how can I build LibreOffice with jemalloc static library?
but in any case just adapting soffice.bin is not sufficient on Windows,
because dynamic libraries on Windows work differently than on ELF based
platforms. with ELF you can add global symbols for malloc, operator new
etc. to the executable and they will be visible to every DSO in the
process, effectively overriding the same symbols from libc.so.
(incidentally this is how we replace the system memory allocator on
Linux with the one in our sal library.)
this does not work on Windows because DLLs do not have a global symbol
namespace, their namespace is local to each DLL. and every DLL knows
not only the name of an imported symbol, but also which other DLL it is
imported from. so i think what you would need to do to replace the
memory allocator on Windows is to link _every_ DLL and executable
against your custom allocator.
(there is already the --with-alloc=jemalloc configure option but it will
work only on ELF platforms due to this reason.)
oh, and for quite a lot of classes in the code base a custom memory
allocator is used anyway due to overloading operator new for the
particular class, see
Build with jemalloc
Does anyone tried to build LibreOffice with jemallc?
I tried to use LibreOffice on Windows environment, and it required too much
memory rather than Microsoft Office.
So I tryied to inspect memory usage with Microsoft Performance Analyzer.
http://msdn.microsoft.com/en-us/library/windows/desktop/hh448170.aspx
I can find many malloc/free and new/delete pair, so I suspect possibility of
heap memory fragmantation.
Then I tried to build with jemalloc, a famous memory allocator.
I modified desktop/Executable_soffice_bin.mk like the following:
$(eval $(call gb_Executable_add_ldflags,soffice_bin,\
/STACK:1000 \
/NODEFAULTLIB:msvcrt /NODEFAULTLIB:libcmt \
'c:\cygwin\libo\jemalloc\jemalloc_s.lib' \
'c:\cygwin\libo\jemalloc\libgcc.a' \
'C:\Program Files\Microsoft Visual Studio 11.0\VC\lib\msvcrt.lib' \
'C:\Program Files\Microsoft Visual Studio 11.0\VC\lib\libcmt.lib' \
))
(it’s a dirty hack)
So gbuild invoked the following command (a little bit long):
S=C:/cygwin/libo O=C:/cygwin/opt/lo/build2/solver/wntmsci14.pro
W=C:/cygwin/opt/lo/build2/workdir/wntmsci14.pro
mkdir -p $W/LinkTarget/Executable/
rm -f $W/LinkTarget/Executable/soffice_bin.exe
RESPONSEFILE=C:/cygwin/tmp/gbuild.ThJJlZ
unset INCLUDE
link -release -opt:noref -incremental:no -debug -nxcompat -dynamicbase
-manifest -SUBSYSTEM:WINDOWS,5.01-LIBPATH:$O/lib
-LIBPATH:C:/PROGRA~1/Java/JDK17~1.0_2/lib
-LIBPATH:C:/PROGRA~1/MICROS~1.0/VC/lib -LIBPATH:C:/PROGRA~1/WI3CF2~1/8.0/lib
-LIBPATH:C:/PROGRA~1/WI3CF2~1/8.0/lib/win8/um/x86
-LIBPATH:C:/PROGRA~1/MICROS~1.NET/SDK/v2.0//lib
-LIBPATH:C:/PROGRA~1/MI5E29~1/lib/x86/STACK:1000
/NODEFAULTLIB:msvcrt
/NODEFAULTLIB:libcmt
'c:\cygwin\libo\jemalloc\jemalloc_s.lib' 'c:\cygwin\libo\jemalloc\libgcc.a'
'C:\Program Files\Microsoft Visual Studio 11.0\VC\lib\msvcrt.lib'
'C:\Program Files\Microsoft Visual Studio 11.0\VC\lib\libcmt.lib'
@${RESPONSEFILE} isal.lib isofficeapp.lib iuwinapi.lib ooopathutils.lib
winextendloaderenv.lib advapi32.lib user32.lib
-out:$W/LinkTarget/Executable/soffice_bin.exe; RC=$?; rm ${RESPONSEFILE}
if [ -f $W/LinkTarget/Executable/soffice_bin.exe.manifest ]; then mt.exe
-nologo -manifest $W/LinkTarget/Executable/soffice_bin.exe.manifest
outputresource:$W/LinkTarget/Executable/soffice_bin.exe\;1
touch -r $W/LinkTarget/Executable/soffice_bin.exe
W/LinkTarget/Executable/soffice_bin.exe.manifest; fi
if [ -f $S/solenv/inc/DeclareDPIAware.manifest ]; then
mt.exe -nologo -manifest $S/solenv/inc/DeclareDPIAware.manifest
-updateresource:$W/LinkTarget/Executable/soffice_bin.exe\;1 ;
fi ; exit $RC
I thknk it seems fine, but the output binary hadn’t have static jemalloc code.
Does anyone tried same thing?
Or how can I build LibreOffice with jemalloc static library?
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice
