Re: Debian shared libs use far more memory than required
Stephane Chauveau wrote: Thiemo Seufer wrote: Andreas Barth wrote: [snip] DEBIAN PACKAGE FROM REPOSITORY: 11 .rodata 000840cb 0021a180 0021a180 ... 21 .data 000233c0 003f1d60 003f1d60 ... MY OWN RECOMPILED DEBIAN PACKAGE: 11 .rodata 000a43ad 001f3180 001f3180 ... 21 .data 0748 003f3460 003f3460 ... That's 0x0233c0-0x748 = 140KB moved from shared to non-shared 140KB of non shared memory per GTK application is HUGE!!! Fortunately it is not as bad as it sounds iff the constant data is collated together in larger chunks. The kernel does copy-on-write, if a .data page is never written, the memory usage is effectively the same. I checked the content of the .data section in libgtk and the unexpected data appears to be composed of all exported symbols aligned to a multiple of 16. Obviously a symbol table of some kind. The whole thing sounds like the result of hiding _GLOBAL_OFFSET_TABLE_ and _PROCEDURE_LINKAGE_TABLE_ in newer binutils. Thiemo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Re: Debian shared libs use far more memory than required
Thiemo Seufer wrote: The whole thing sounds like the result of hiding _GLOBAL_OFFSET_TABLE_ and _PROCEDURE_LINKAGE_TABLE_ in newer binutils. I assume that you mean that the problem the problem is solved by using the latest binutils (and not that it was introduceed by them). Is there an easy way to ugrapde the binutils used by the build system? -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian shared libs use far more memory than required
On Fri, Aug 26, 2005 at 10:25:15AM +0200, Thiemo Seufer wrote: I checked the content of the .data section in libgtk and the unexpected data appears to be composed of all exported symbols aligned to a multiple of 16. Obviously a symbol table of some kind. The whole thing sounds like the result of hiding _GLOBAL_OFFSET_TABLE_ and _PROCEDURE_LINKAGE_TABLE_ in newer binutils. Just for reference, from the buildd log of the package in question: Toolchain package versions: libc6-dev_2.3.2.ds1-22 linux-kernel-headers_2.6.13+0rc3-1 gcc-4.0_4.0.1-3 g++-4.0_4.0.1-3 binutils_2.16.1-2 libstdc++6-4.0-dev_4.0.1-3 libstdc++6_4.0.1-3 And binutils_2.16.1-2 is still the latest version. Kurt -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian shared libs use far more memory than required
Stephane Chauveau wrote: Thiemo Seufer wrote: The whole thing sounds like the result of hiding _GLOBAL_OFFSET_TABLE_ and _PROCEDURE_LINKAGE_TABLE_ in newer binutils. I assume that you mean that the problem the problem is solved by using the latest binutils (and not that it was introduceed by them). That was my guess, but Kurt says the binutils version was the same in both cases. Thiemo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian shared libs use far more memory than required
Andreas Barth wrote: [snip] DEBIAN PACKAGE FROM REPOSITORY: 11 .rodata 000840cb 0021a180 0021a180 ... 21 .data 000233c0 003f1d60 003f1d60 ... MY OWN RECOMPILED DEBIAN PACKAGE: 11 .rodata 000a43ad 001f3180 001f3180 ... 21 .data 0748 003f3460 003f3460 ... That's 0x0233c0-0x748 = 140KB moved from shared to non-shared 140KB of non shared memory per GTK application is HUGE!!! Fortunately it is not as bad as it sounds iff the constant data is collated together in larger chunks. The kernel does copy-on-write, if a .data page is never written, the memory usage is effectively the same. [snip] My first though was that the official debian package was probably compiled with a different version of gcc. However, I could not reproduce the official package size with gcc-3.3, gcc-3.4 and gcc-4.0 so the problem is probably elsewhere: libtool? binutils ? The immediate suspect is binutils, particularily ld. It might be interesting to do test compiles with an older binutils version (2.15 vs. 2.16?) and see if the problem is reproducible. Thiemo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian shared libs use far more memory than required
On Thu, Aug 25, 2005 at 07:47:17PM +0200, Thiemo Seufer wrote: The immediate suspect is binutils, particularily ld. It might be interesting to do test compiles with an older binutils version (2.15 vs. 2.16?) and see if the problem is reproducible. The package in question was already build using binutils 2.16 according to the buildd log. Kurt -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian shared libs use far more memory than required
On Thu, Aug 25, 2005 at 08:07:42PM +0200, Kurt Roeckx wrote: On Thu, Aug 25, 2005 at 07:47:17PM +0200, Thiemo Seufer wrote: The immediate suspect is binutils, particularily ld. It might be interesting to do test compiles with an older binutils version (2.15 vs. 2.16?) and see if the problem is reproducible. The package in question was already build using binutils 2.16 according to the buildd log. For each ever avalable version? Bastian -- Captain's Log, star date 21:34.5... signature.asc Description: Digital signature
Re: Debian shared libs use far more memory than required
Hi, [full cite left for d-ds usage] * Stephane Chauveau ([EMAIL PROTECTED]) [050825 01:16]: I was profiling the memory usage of libgtk when discovered something very strange. As you may know, shared memory are usually mapped in 2 different memory segments. The first segment containing constant data code and is shared by all applications using the library. The second segment contains non-constant data and each application gets its own copy. What I discovered is that in the official Debian package for libgtk, a large part of the constant data is moved from the shared segment to the non-shared one. That is easy to see in the section table dumped with 'objdump' where the section .rodata is shared while the section .data is not. $ objdump -h -w libgtk-x11-2.0.so The section size is in the 3rd column DEBIAN PACKAGE FROM REPOSITORY: 11 .rodata 000840cb 0021a180 0021a180 ... 21 .data 000233c0 003f1d60 003f1d60 ... MY OWN RECOMPILED DEBIAN PACKAGE: 11 .rodata 000a43ad 001f3180 001f3180 ... 21 .data 0748 003f3460 003f3460 ... That's 0x0233c0-0x748 = 140KB moved from shared to non-shared 140KB of non shared memory per GTK application is HUGE!!! The problem does not seem to be limited to libgtk. I picked a library randomly: libasound Here are the results: DEBIAN PACKAGE FROM REPOSITORY: 12 .rodata 8a78 0009f240 0009f240 ... 14 .data 00010e20 001ad000 001ad000 ... MY OWN RECOMPILED DEBIAN PACKAGE: 12 .rodata 000140f8 0009eda0 0009eda0 ... 21 .data 5740 001cbd20 001cbd20 ... For libasound 'only' 0x10e20-0x5740 = 47KB are not shared. My first though was that the official debian package was probably compiled with a different version of gcc. However, I could not reproduce the official package size with gcc-3.3, gcc-3.4 and gcc-4.0 so the problem is probably elsewhere: libtool? binutils ? The problem is also not specified to AMD64. The x86 version of libgtk also has more non-shared data than it should. I would like to file a bug I do not in which package. Thanks for this valuable hint. However, debian-devel is the better auditorium, so I added them to the Cc-list. Perhaps one of the people there can throw some light at this issue. Cheers, Andi -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian shared libs use far more memory than required
Kurt Roeckx wrote: On Thu, Aug 25, 2005 at 07:47:17PM +0200, Thiemo Seufer wrote: The immediate suspect is binutils, particularily ld. It might be interesting to do test compiles with an older binutils version (2.15 vs. 2.16?) and see if the problem is reproducible. The package in question was already build using binutils 2.16 according to the buildd log. I am not really surprised because I just compared the linker scripts from 2.15 and 2.16. They have a different section ordering and the official debian package clearly follows the 2.16 ordering. Also, there was no differences between the linker scripts in 2.16 and in 2.16.1 My main suspect is still libtool. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian shared libs use far more memory than required
On Fri, Aug 26, 2005 at 12:21:07AM +0200, Stephane Chauveau wrote: I am not really surprised because I just compared the linker scripts from 2.15 and 2.16. They have a different section ordering and the official debian package clearly follows the 2.16 ordering. Also, there was no differences between the linker scripts in 2.16 and in 2.16.1 My main suspect is still libtool. The libtool version in debian didn't change for a while so I doubt this. And since the source itself didn't change either, I don't see how libtool can cause this at all. Do you only see this difference with libraries, or also with binaries? This seems to have been a temporary problem that has now gone away, and all the obvious candidates don't seem to be the cause. Building the same source (from sid) on sarge gives: .rodata 689389 2101632 .data4 3907584 Which is comparable to the current version in sarge: .rodata 684173 2084992 .data 4 3883168 Kurt -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Debian shared libs use far more memory than required
Thiemo Seufer wrote: Andreas Barth wrote: [snip] DEBIAN PACKAGE FROM REPOSITORY: 11 .rodata 000840cb 0021a180 0021a180 ... 21 .data 000233c0 003f1d60 003f1d60 ... MY OWN RECOMPILED DEBIAN PACKAGE: 11 .rodata 000a43ad 001f3180 001f3180 ... 21 .data 0748 003f3460 003f3460 ... That's 0x0233c0-0x748 = 140KB moved from shared to non-shared 140KB of non shared memory per GTK application is HUGE!!! Fortunately it is not as bad as it sounds iff the constant data is collated together in larger chunks. The kernel does copy-on-write, if a .data page is never written, the memory usage is effectively the same. I checked the content of the .data section in libgtk and the unexpected data appears to be composed of all exported symbols aligned to a multiple of 16. Obviously a symbol table of some kind. Unfortunately, they only seem to be grouped per object file. Some small non-constant data can be found every few KB. I made a quick estimation and it seems that writing to those non-constant data would access about 15 pages or 60K with a 4K page size. Not as bad as the initial 140KB but still impressive. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
