Re: Android port problems
On 17 May 2012 14:56, David Tardon wrote: >> >> On Thu, 2012-05-17 at 12:22 +0300, Tor Lillqvist wrote: >> > libraries into fewer. There is an option for that already, >> > --enable-mergelibs, that merges quite a large number of them into one, >> > libmergedlo.so. This works only for libraries in gbuildified modules, >> > built in tail_build. >> ... >> > Anyway, this indeed has helped a lot. But not enough. I am not sure if >> > the number of libraries merged this way can be easily increased much >> > at this stage of gbuildification. I don't know what is much but it can be easily increased I think. At least conditionally for Android. Originally I understood we want to merge libraries which are "always used". I have a year old list of libraries (attached) where I also mark the status of modules. It was created by starting ./soffice --calc ./soffice --writer ... 5 different runs. '5ucpgvfs1.uno.so ->ucb <- GNU <- TAIL' says that the library was loaded 5 times (always) and is from tail_build's module ucb. But currently we are merging also fwl and fwm which are not used at the start of Libreoffice. I think we can merge these only for Android and also add more small component libs like fileacc. Attached patch. The simple rules for adding: - library needs to be from tail_build - It must not create circular dependencies because of linking chains > All these modules are candidates for adding into tail_build (adding of > connectivity immediately allows adding of other modules: svl, sot, > unixODBC). And I think there will be a couple of other candidates after > these three branches are all merged into master. shell/, unotools/. (Well, we need to move crashrep/ behind tail_build) Next blocker would be pyuno, blocking sax. Actually I have created patch based on feature/gbuild_conversions which adds fsstorage, svl, sot and vcl to libmerged. I am waiting for the merge. But vcl is big, if there are problems with linking.. Best, Matus 5ucpgvfs1.uno.so ->ucb <- GNU <- TAIL 5stocservices.uno.so ->stoc <- GNU 5localebe1.uno.so->shell<- GNU <- TAIL 5libxolo.so ->xmloff <- GNU <- TAIL 5libxmlreader.so ->xmlreader<- GNU 5libxml2.so.2->libxml2 <- EXT 5libxcrlo.so ->xmlscript<- GNU <- TAIL 5libvclplug_gtklo.so ->vcl <- GNU <- TAIL 5libvclplug_genlo.so ->vcl <- GNU <- TAIL 5libvcllo.so ->vcl <- GNU <- TAIL 5libuuilo.so ->uui <- GNU <- TAIL 5libutllo.so ->unotools <- GNU 5libuno_sal.so.3 ->sal <- GNU 5libuno_salhelpergcc3.so.3 ->salhelper<- GNU 5libuno_cppu.so.3->cppu <- GNU 5libuno_cppuhelpergcc3.so.3 ->cppuhelper <- GNU 5libucpfile1.so ->ucb <- GNU <- TAIL 5libucbhelper4gcc3.so->ucbhelper<- GNU 5libucb1.so ->ucb <- GNU <- TAIL 5libtllo.so ->tools<- GNU 5libtklo.so ->toolkit <- GNU <- TAIL 5libsvxlo.so ->svx <- GNU <- TAIL 5libsvxcorelo.so ->svx <- GNU <- TAIL 5libsvtlo.so ->svtools <- GNU <- TAIL 5libsvllo.so ->svl <- GNU <- TAIL 5libstore.so.3 ->store<- GNU 5libstdc++.so.6 ->external <- EXT 5libspllo.so ->desktop <- GNU <- TAIL 5libsotlo.so ->sot <- GNU <- TAIL 5libsofficeapp.so->desktop <- GNU <- TAIL 5libsfxlo.so ->sfx2 <- GNU <- TAIL 5libsblo.so ->basic<- GNU <- TAIL 5libsaxlo.so ->sax <- GNU 5libreg.so.3 ->registry <- GNU 5libpixman-1.so.0->cairo<- EXT 5libpackage2.so ->package <- GNU <- TAIL 5liblocaledata_en.so ->i18npool <- GNU 5liblnglo.so ->linguistic <- GNU <- TAIL 5libjvmfwk.so.3 ->jvmfwk <- GNU 5libjvmaccessgcc3.so.3 ->jvmaccess<- GNU 5libicuuc.so.44 ->icu <- EXT 5lib
Re: Android port problems
On Thu, May 17, 2012 at 12:37:48PM +0100, Michael Meeks wrote: > Hi Tor, > > Thanks for the nice summary :-) > > On Thu, 2012-05-17 at 12:22 +0300, Tor Lillqvist wrote: > > libraries into fewer. There is an option for that already, > > --enable-mergelibs, that merges quite a large number of them into one, > > libmergedlo.so. This works only for libraries in gbuildified modules, > > built in tail_build. > ... > > Anyway, this indeed has helped a lot. But not enough. I am not sure if > > the number of libraries merged this way can be easily increased much > > at this stage of gbuildification. > > IIRC the gbuild-ification work is continuing apace, and my hope was > that the two davids, matus & others' work on the large number of > gbuild_* git branches would get merged - and unblock tail_build so that > it can incorporate many more of the libraries that are already > gnu-makeized which in turn would help us getting much more into > libmerged ... > > That at least is my hope: David ? what's the plan there :-) hopefully > that good stuff is all going in somewhat before the 3.6 feature-freeze ? Right. What I have finished converting is connectivity & reportbuilder (on feature/gbuild_conversions, based on Peter's work), shell (feature/gbuild_shell) and scp2 (feature/gbuild_scp2; finished, but not announced yet, because I am working on a script for comparing the install scripts produced by the old and new build). I plan to merge all three of them before the feature freeze. I do not think I will manage to convert anything else in the remaining three weeks. David Ostrovsky is currently working on testtools, but I am not sure it will be ready in time for 3.6. All these modules are candidates for adding into tail_build (adding of connectivity immediately allows adding of other modules: svl, sot, unixODBC). And I think there will be a couple of other candidates after these three branches are all merged into master. I must admit I do not know much about that "merged libs" stuff, because running libreoffice on android and similar devices is outside of my interest, so I cannot tell if these additions to tail_build are going to help Tor or not... D. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Android port problems
> How do Mozilla get away without any text relocations ? They do some hack to avoid the problematic crtbegin.o object when linking their shared objects, if I understand correctly. See https://bugzilla.mozilla.org/show_bug.cgi?id=720621 . And they apparently don't use the shared GNU C++ library (which as present in the NDK, libgnustl_shared.so, also contains a text relocation), but either the static GNU C++ library from the NDK, or manage with the system C++ library. (Check the .so files included in the fennec .apk, http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-r7/ ) --tml ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Android port problems
On 2012-05-17 11:22, Tor Lillqvist wrote: recall. Unfortunately the GNU linker isn't exactly known for its blazing speed, so especially if you are building with debugging Perhaps we should be using the GOLD linker? Much faster (6x) for some people Looks like Mozilla has had some success https://wiki.mozilla.org/Mobile/Fennec/Android#Increase_linking_speed_.28AKA_using_gold.29 And at least one other person http://stackoverflow.com/questions/4955070/does-binutils-gold-has-support-for-arm-android Disclaimer: http://www.peralex.com/disclaimer.html ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: Android port problems
Hi Tor, Thanks for the nice summary :-) On Thu, 2012-05-17 at 12:22 +0300, Tor Lillqvist wrote: > libraries into fewer. There is an option for that already, > --enable-mergelibs, that merges quite a large number of them into one, > libmergedlo.so. This works only for libraries in gbuildified modules, > built in tail_build. ... > Anyway, this indeed has helped a lot. But not enough. I am not sure if > the number of libraries merged this way can be easily increased much > at this stage of gbuildification. IIRC the gbuild-ification work is continuing apace, and my hope was that the two davids, matus & others' work on the large number of gbuild_* git branches would get merged - and unblock tail_build so that it can incorporate many more of the libraries that are already gnu-makeized which in turn would help us getting much more into libmerged ... That at least is my hope: David ? what's the plan there :-) hopefully that good stuff is all going in somewhat before the 3.6 feature-freeze ? Then again - the linking speedup from merging all that lot together has an upper bound of ~7% CPU time in 3.5.4 (at least for me), though there'd be potentially useful size savings, improved optimisations etc. > 1) One way would be to identify libraries that are used only by one > other library, and then merge those. Unfortunately doing this > uncoditionally (on all platforms) encounters quite some stop energy. Hopefully not :-) if we can show that there is some pointless split, I don't see why we can't merge things. Of course, there are a number of existing divisions that are perhaps worthwhile / necessary to keep: URE vs. non-URE, GUI & non-GUI, but beyond that - IMHO merging makes sense [ though I'd be happier to see it done with static libraries, rather than wholesale code movement perhaps ;-]. > So it would have to be done conditionally just for Android Clearly we want to share as much as possible with libmerged I guess, and that work sounds generally useful. > 2) Another way would be to start using another dynamic linker, namely > the one Mozilla uses on Android, known as "faulty.lib" for some > reason. See https://github.com/glandium/faulty.lib . It doesn't have > the silly limitation on the number of shared libraries. > > Unfortunately, it brings in another problem: It doesn't handle text > relocations. How do Mozilla get away without any text relocations ? and/or - why are we re-locating .text - it sounds profoundly crazy :-) surely that is readonly. Is there a better toolchain to use ? > Of course, linking all required code into one shared object means that > this linking will take quite some time, especially if debugging > information has been included. Nasty indeed; unclear what we can do here, beyond profiling and/or begging the linker guys :-) Of course, continuing to have a non-merged option for rapid development where it is not necessary seems important to me; and perhaps CPU's or L3 cache-sizes will save us in the end ;-) HTH, Michael. -- michael.me...@suse.com <><, Pseudo Engineer, itinerant idiot ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Android port problems
As some of you might know, the system dynamic linker on Android (which, like all the Android user level runtime, is not related at all to the one on desktop (GNU/)Linux) has a silly limitation on the number of shared object (dynamic libraries) that can be in use: 128. Given that the number of system libraries used by a typical process running LO code is already a few dozens, there is definitely not enough "slots" available for the dynamic libraries used during running of typical LO code that loads documents. OK, the above is nothing new, has been known for some time already. So something has to be done. One way forward is to merge many of the LO libraries into fewer. There is an option for that already, --enable-mergelibs, that merges quite a large number of them into one, libmergedlo.so. This works only for libraries in gbuildified modules, built in tail_build. There might also some other limitations, I don't recall. Unfortunately the GNU linker isn't exactly known for its blazing speed, so especially if you are building with debugging information, linking libmerged can take quite long (depending on your build machine). At one point, when my Linux laptop for some reason had decided to keep its CPU speed throttled at the lowest, it took hours;) Anyway, this indeed has helped a lot. But not enough. I am not sure if the number of libraries merged this way can be easily increased much at this stage of gbuildification. So what other ways forward? 1) One way would be to identify libraries that are used only by one other library, and then merge those. Unfortunately doing this uncoditionally (on all platforms) encounters quite some stop energy. So it would have to be done conditionally just for Android (or using the same condition that currently is used for libmerged). Will make makefiles messy. Still, could be the easiest solution, although how much such merging can be realistically done, whether enough, is unclear. 2) Another way would be to start using another dynamic linker, namely the one Mozilla uses on Android, known as "faulty.lib" for some reason. See https://github.com/glandium/faulty.lib . It doesn't have the silly limitation on the number of shared libraries. Unfortunately, it brings in another problem: It doesn't handle text relocations. The Android NDK produces shared libraries that contain text relocations. Apparently there is just one such relocation in each, and it isn't the fault of the toolchain as such, but one of the C runtime objects linked in to shared libraries contains non-PIC code. See http://code.google.com/p/android/issues/detail?id=23203 . 3) A third way would be to just stop using such a large number of shared libraries, and build the LO code as static archives, do static linking of UNO components, as we have to do for iOS anyway, so there is some code to take care of the related complications already. (See ifdefs for DISABLE_DYNLOADING.) On iOS the code of an app is a single executable program (that links only to system libraries dynamically), on Android a LO code -using app would have, in this case then, a single shared library (that also links only to system libraries dynamically). Of course, linking all required code into one shared object means that this linking will take quite some time, especially if debugging information has been included. (This is true for an iOS build, too, even if it doesn't use the GNU linker. There it is the "dsymutil" phase that extracts debugging information from the objects and collects it into file(s) in a separate .dSYM directory that takes the most time. Others have noticed, too, this might be interesting at some stage: http://src.chromium.org/svn/trunk/src/build/mac/strip_save_dsym ) --tml ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
