Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On Mon, 2011-06-27 at 16:37 +0800, Dongxiao Xu wrote: +do_install_locale () { + dest=${D}/${includedir}/glibc-locale-internal-${MULTIMACH_TARGET_SYS} + install -d ${dest} ${dest}${bindir} + cp -fpPR ${D}${base_libdir} ${dest}${base_prefix} + cp -fpPR ${D}${libdir} ${dest}${exec_prefix} + cp -fpPR ${D}${datadir} ${dest}${exec_prefix} + cp -fpPR ${D}${bindir}/localedef ${dest}${bindir} + cp -fpPR ${WORKDIR}/SUPPORTED ${dest} +} This turns out to lose if you don't have libc-locale-code in DISTRO_FEATURES, since then localedef isn't installed and it blows up trying to copy that file. p. ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On Tue, 2011-06-28 at 10:07 +0100, Phil Blundell wrote: On Tue, 2011-06-28 at 08:51 +0800, Xu, Dongxiao wrote: -Original Message- From: openembedded-core-boun...@lists.openembedded.org [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of Phil Blundell Sent: Monday, June 27, 2011 4:58 PM To: Patches and discussions about the oe-core layer Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe. On Mon, 2011-06-27 at 16:37 +0800, Dongxiao Xu wrote: -PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX} nscd${PKGSUFFIX} ldd${PKGSUFFIX} localedef${PKGSUFFIX} libcidn ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc eglibc-locale libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile libsotruss${PKGSUFFIX} - -PACKAGES_DYNAMIC = \ - glibc-gconv-*${PKGSUFFIX} glibc-charmap-* glibc-localedata-* glibc-binary-localedata-* \ - eglibc-gconv-* eglibc-charmap-* eglibc-localedata-* eglibc-binary-localedata-* \ - locale-base-*${PKGSUFFIX} +PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX} nscd${PKGSUFFIX} ldd${PKGSUFFIX} ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc libcidn libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile You seem to have made a bunch of changes here that are not related to locales. What are those about? They are locale related changes. Locale related stuffs in the above PACKAGES and PACKAGES_DYNAMIC are moved to eglibc-locale recipe. Is libsotruss${PKGSUFFIX} (for example) really a locale related stuff? It's not obvious to me what the connection is. Dongxiao: What Phil is referring to here is that libsotruss${PKGSUFFIX} disappeared from PACKAGES and that also ${libdir}/audit/.debug disappeared from FILES_${PN}-dbg. Specifically these changes came in as part of: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=77c2dd944db42e40cc8467e6bc5a175fed90c251 which I don't think your patch accounted for. Since this patch has been around for a while and it otherwise looks good, I've fixed up these couple of issues and merged it though. Cheers, Richard ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
Op 28 jun 2011, om 13:07 heeft Richard Purdie het volgende geschreven: On Tue, 2011-06-28 at 10:07 +0100, Phil Blundell wrote: On Tue, 2011-06-28 at 08:51 +0800, Xu, Dongxiao wrote: -Original Message- From: openembedded-core-boun...@lists.openembedded.org [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of Phil Blundell Sent: Monday, June 27, 2011 4:58 PM To: Patches and discussions about the oe-core layer Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe. On Mon, 2011-06-27 at 16:37 +0800, Dongxiao Xu wrote: -PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX} nscd${PKGSUFFIX} ldd${PKGSUFFIX} localedef${PKGSUFFIX} libcidn ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc eglibc-locale libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile libsotruss${PKGSUFFIX} - -PACKAGES_DYNAMIC = \ - glibc-gconv-*${PKGSUFFIX} glibc-charmap-* glibc-localedata-* glibc-binary-localedata-* \ - eglibc-gconv-* eglibc-charmap-* eglibc-localedata-* eglibc-binary-localedata-* \ - locale-base-*${PKGSUFFIX} +PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX} nscd${PKGSUFFIX} ldd${PKGSUFFIX} ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc libcidn libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile You seem to have made a bunch of changes here that are not related to locales. What are those about? They are locale related changes. Locale related stuffs in the above PACKAGES and PACKAGES_DYNAMIC are moved to eglibc-locale recipe. Is libsotruss${PKGSUFFIX} (for example) really a locale related stuff? It's not obvious to me what the connection is. Dongxiao: What Phil is referring to here is that libsotruss${PKGSUFFIX} disappeared from PACKAGES and that also ${libdir}/audit/.debug disappeared from FILES_${PN}-dbg. Specifically these changes came in as part of: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=77c2dd944db42e40cc8467e6bc5a175fed90c251 which I don't think your patch accounted for. Since this patch has been around for a while and it otherwise looks good, I've fixed up these couple of issues and merged it though. This breaks when using eglibc 2.12 :( ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On Tue, 2011-06-28 at 14:17 +0200, Koen Kooi wrote: Op 28 jun 2011, om 13:07 heeft Richard Purdie het volgende geschreven: On Tue, 2011-06-28 at 10:07 +0100, Phil Blundell wrote: Dongxiao: What Phil is referring to here is that libsotruss${PKGSUFFIX} disappeared from PACKAGES and that also ${libdir}/audit/.debug disappeared from FILES_${PN}-dbg. Specifically these changes came in as part of: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=77c2dd944db42e40cc8467e6bc5a175fed90c251 which I don't think your patch accounted for. Since this patch has been around for a while and it otherwise looks good, I've fixed up these couple of issues and merged it though. This breaks when using eglibc 2.12 :( Sorry, I've pushed some cleanup to resolve that. Cheers, Richard ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
Op 28 jun 2011, om 16:00 heeft Richard Purdie het volgende geschreven: On Tue, 2011-06-28 at 14:17 +0200, Koen Kooi wrote: Op 28 jun 2011, om 13:07 heeft Richard Purdie het volgende geschreven: On Tue, 2011-06-28 at 10:07 +0100, Phil Blundell wrote: Dongxiao: What Phil is referring to here is that libsotruss${PKGSUFFIX} disappeared from PACKAGES and that also ${libdir}/audit/.debug disappeared from FILES_${PN}-dbg. Specifically these changes came in as part of: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=77c2dd944db42e40cc8467e6bc5a175fed90c251 which I don't think your patch accounted for. Since this patch has been around for a while and it otherwise looks good, I've fixed up these couple of issues and merged it though. This breaks when using eglibc 2.12 :( Sorry, I've pushed some cleanup to resolve that. So after my shlib renaming patch I still can't build any image, since locale-base-* has disappeared. Is there anything related to libc and libc-locales this patch *didn't* break? ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
[OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
*libc's do_package will cost a lot of time due to the locale handing, which may delay the other recipe's do_package task and affect the build performance. This commit moves locale handling into a separate recipe *libc-locale. Signed-off-by: Dongxiao Xu dongxiao...@intel.com --- meta/classes/libc-common.bbclass | 23 meta/classes/libc-package.bbclass | 24 meta/conf/distro/include/tclibc-eglibc.inc |1 + meta/conf/distro/include/tclibc-glibc.inc |1 + meta/recipes-core/eglibc/eglibc-initial_2.13.bb|4 ++ meta/recipes-core/eglibc/eglibc-locale_2.13.bb | 56 meta/recipes-core/eglibc/eglibc-package.inc| 32 +-- meta/recipes-core/eglibc/eglibc.inc|2 +- meta/recipes-core/eglibc/eglibc_2.13.bb| 12 meta/recipes-core/glibc/glibc-initial_2.10.1.bb|4 ++ meta/recipes-core/glibc/glibc-locale_2.10.1.bb | 49 + meta/recipes-core/glibc/glibc-package.inc | 29 ++- meta/recipes-core/glibc/glibc_2.10.1.bb| 12 .../meta/external-csl-toolchain_2008q3-72.bb |1 + 14 files changed, 171 insertions(+), 79 deletions(-) create mode 100644 meta/classes/libc-common.bbclass create mode 100644 meta/recipes-core/eglibc/eglibc-locale_2.13.bb create mode 100644 meta/recipes-core/glibc/glibc-locale_2.10.1.bb diff --git a/meta/classes/libc-common.bbclass b/meta/classes/libc-common.bbclass new file mode 100644 index 000..bae0ace --- /dev/null +++ b/meta/classes/libc-common.bbclass @@ -0,0 +1,23 @@ +do_install() { + oe_runmake install_root=${D} install + for r in ${rpcsvc}; do + h=`echo $r|sed -e's,\.x$,.h,'` + install -m 0644 ${S}/sunrpc/rpcsvc/$h ${D}/${includedir}/rpcsvc/ + done + install -m 0644 ${WORKDIR}/etc/ld.so.conf ${D}/${sysconfdir}/ + install -d ${D}${libdir}/locale + make -f ${WORKDIR}/generate-supported.mk IN=${S}/localedata/SUPPORTED OUT=${WORKDIR}/SUPPORTED + # get rid of some broken files... + for i in ${GLIBC_BROKEN_LOCALES}; do + grep -v $i ${WORKDIR}/SUPPORTED ${WORKDIR}/SUPPORTED.tmp + mv ${WORKDIR}/SUPPORTED.tmp ${WORKDIR}/SUPPORTED + done + rm -f ${D}{sysconfdir}/rpc + rm -rf ${D}${datadir}/zoneinfo + rm -rf ${D}${libexecdir}/getconf +} + +def get_libc_fpu_setting(bb, d): +if bb.data.getVar('TARGET_FPU', d, 1) in [ 'soft' ]: +return --without-fp +return diff --git a/meta/classes/libc-package.bbclass b/meta/classes/libc-package.bbclass index 55e3d48..5737af4 100644 --- a/meta/classes/libc-package.bbclass +++ b/meta/classes/libc-package.bbclass @@ -37,11 +37,6 @@ python __anonymous () { break } -def get_libc_fpu_setting(bb, d): -if bb.data.getVar('TARGET_FPU', d, 1) in [ 'soft' ]: -return --without-fp -return - OVERRIDES_append = :${TARGET_ARCH}-${TARGET_OS} do_configure_prepend() { @@ -84,25 +79,6 @@ rm -rf ${TMP_LOCALE} } -do_install() { - oe_runmake install_root=${D} install - for r in ${rpcsvc}; do - h=`echo $r|sed -e's,\.x$,.h,'` - install -m 0644 ${S}/sunrpc/rpcsvc/$h ${D}/${includedir}/rpcsvc/ - done - install -m 0644 ${WORKDIR}/etc/ld.so.conf ${D}/${sysconfdir}/ - install -d ${D}${libdir}/locale - make -f ${WORKDIR}/generate-supported.mk IN=${S}/localedata/SUPPORTED OUT=${WORKDIR}/SUPPORTED - # get rid of some broken files... - for i in ${GLIBC_BROKEN_LOCALES}; do - grep -v $i ${WORKDIR}/SUPPORTED ${WORKDIR}/SUPPORTED.tmp - mv ${WORKDIR}/SUPPORTED.tmp ${WORKDIR}/SUPPORTED - done - rm -f ${D}${sysconfdir}/rpc - rm -rf ${D}${datadir}/zoneinfo - rm -rf ${D}${libexecdir}/getconf -} - TMP_LOCALE=/tmp/locale${libdir}/locale do_prep_locale_tree() { diff --git a/meta/conf/distro/include/tclibc-eglibc.inc b/meta/conf/distro/include/tclibc-eglibc.inc index a4c648e..e070aad 100644 --- a/meta/conf/distro/include/tclibc-eglibc.inc +++ b/meta/conf/distro/include/tclibc-eglibc.inc @@ -15,6 +15,7 @@ PREFERRED_PROVIDER_virtual/libiconv-nativesdk ?= eglibc-nativesdk PREFERRED_PROVIDER_virtual/libintl ?= eglibc PREFERRED_PROVIDER_virtual/libc ?= eglibc PREFERRED_PROVIDER_virtual/libc-nativesdk ?= eglibc-nativesdk +PREFERRED_PROVIDER_virtual/libc-locale ?= eglibc-locale CXXFLAGS += -fvisibility-inlines-hidden diff --git a/meta/conf/distro/include/tclibc-glibc.inc b/meta/conf/distro/include/tclibc-glibc.inc index e5843b0..5e7afc1 100644 --- a/meta/conf/distro/include/tclibc-glibc.inc +++ b/meta/conf/distro/include/tclibc-glibc.inc @@ -15,6 +15,7 @@ PREFERRED_PROVIDER_virtual/libiconv-nativesdk ?= glibc-nativesdk PREFERRED_PROVIDER_virtual/libintl ?= glibc PREFERRED_PROVIDER_virtual/libc ?= glibc PREFERRED_PROVIDER_virtual/libc-nativesdk
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On Mon, 2011-06-27 at 16:37 +0800, Dongxiao Xu wrote: -PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX} nscd${PKGSUFFIX} ldd${PKGSUFFIX} localedef${PKGSUFFIX} libcidn ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc eglibc-locale libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile libsotruss${PKGSUFFIX} - -PACKAGES_DYNAMIC = \ - glibc-gconv-*${PKGSUFFIX} glibc-charmap-* glibc-localedata-* glibc-binary-localedata-* \ - eglibc-gconv-* eglibc-charmap-* eglibc-localedata-* eglibc-binary-localedata-* \ - locale-base-*${PKGSUFFIX} +PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX} nscd${PKGSUFFIX} ldd${PKGSUFFIX} ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc libcidn libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile You seem to have made a bunch of changes here that are not related to locales. What are those about? -FILES_${PN}-dbg += ${libexecdir}/*/.debug ${libdir}/gconv/.debug ${libdir}/audit/.debug +FILES_${PN}-dbg += ${libexecdir}/*/.debug Likewise. Also, why is localedef getting packaged by libc-locale? Not that I think it's necessarily a problem but it seems a little bit strange. p. ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
-Original Message- From: openembedded-core-boun...@lists.openembedded.org [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of Phil Blundell Sent: Monday, June 27, 2011 4:58 PM To: Patches and discussions about the oe-core layer Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe. On Mon, 2011-06-27 at 16:37 +0800, Dongxiao Xu wrote: -PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX} nscd${PKGSUFFIX} ldd${PKGSUFFIX} localedef${PKGSUFFIX} libcidn ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc eglibc-locale libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile libsotruss${PKGSUFFIX} - -PACKAGES_DYNAMIC = \ - glibc-gconv-*${PKGSUFFIX} glibc-charmap-* glibc-localedata-* glibc-binary-localedata-* \ - eglibc-gconv-* eglibc-charmap-* eglibc-localedata-* eglibc-binary-localedata-* \ - locale-base-*${PKGSUFFIX} +PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX} nscd${PKGSUFFIX} ldd${PKGSUFFIX} ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc libcidn libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile You seem to have made a bunch of changes here that are not related to locales. What are those about? They are locale related changes. Locale related stuffs in the above PACKAGES and PACKAGES_DYNAMIC are moved to eglibc-locale recipe. -FILES_${PN}-dbg += ${libexecdir}/*/.debug ${libdir}/gconv/.debug ${libdir}/audit/.debug +FILES_${PN}-dbg += ${libexecdir}/*/.debug Likewise. Also, why is localedef getting packaged by libc-locale? Not that I think it's necessarily a problem but it seems a little bit strange. Originally I thought of putting all locale related things to eglibc-locale. Now actually virtual/libiconv belongs to eglibc. Thus I am OK to move localedef from eglibc-locale to eglibc, and will include this in my next pull request. Thanks, Dongxiao p. ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
-Original Message- From: Xu, Dongxiao Sent: Tuesday, June 28, 2011 8:52 AM To: Patches and discussions about the oe-core layer Subject: RE: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe. -Original Message- From: openembedded-core-boun...@lists.openembedded.org [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of Phil Blundell Sent: Monday, June 27, 2011 4:58 PM To: Patches and discussions about the oe-core layer Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe. On Mon, 2011-06-27 at 16:37 +0800, Dongxiao Xu wrote: -PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX} nscd${PKGSUFFIX} ldd${PKGSUFFIX} localedef${PKGSUFFIX} libcidn ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc eglibc-locale libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile libsotruss${PKGSUFFIX} - -PACKAGES_DYNAMIC = \ - glibc-gconv-*${PKGSUFFIX} glibc-charmap-* glibc-localedata-* glibc-binary-localedata-* \ - eglibc-gconv-* eglibc-charmap-* eglibc-localedata-* eglibc-binary-localedata-* \ - locale-base-*${PKGSUFFIX} +PACKAGES = ${PN}-dbg ${PN} catchsegv${PKGSUFFIX} sln${PKGSUFFIX} nscd${PKGSUFFIX} ldd${PKGSUFFIX} ${PN}-utils ${PN}-pic ${PN}-dev eglibc-doc libcidn libmemusage libsegfault${PKGSUFFIX} eglibc-extra-nss eglibc-thread-db${PKGSUFFIX} eglibc-pcprofile You seem to have made a bunch of changes here that are not related to locales. What are those about? They are locale related changes. Locale related stuffs in the above PACKAGES and PACKAGES_DYNAMIC are moved to eglibc-locale recipe. -FILES_${PN}-dbg += ${libexecdir}/*/.debug ${libdir}/gconv/.debug ${libdir}/audit/.debug +FILES_${PN}-dbg += ${libexecdir}/*/.debug Likewise. Also, why is localedef getting packaged by libc-locale? Not that I think it's necessarily a problem but it seems a little bit strange. Originally I thought of putting all locale related things to eglibc-locale. Now actually virtual/libiconv belongs to eglibc. Thus I am OK to move localedef from eglibc-locale to eglibc, and will include this in my next pull request. Just did an experiment to move localedef in eglibc instead of eglibc-locale, however eglibc-locale's do_package met errors since it needs localedef to generate locales, see libc-packages.inc. Therefore localedef still needs to be packaged in eglibc-locale recipe. Thanks, Dongxiao Thanks, Dongxiao p. ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
-Original Message- From: openembedded-core-boun...@lists.openembedded.org [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of Khem Raj Sent: Friday, June 24, 2011 7:43 AM To: openembedded-core@lists.openembedded.org Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe. On 06/23/2011 03:14 AM, Richard Purdie wrote: On Thu, 2011-06-23 at 10:40 +0100, Phil Blundell wrote: On Thu, 2011-06-23 at 12:08 +0800, Xu, Dongxiao wrote: -Original Message- From: openembedded-core-boun...@lists.openembedded.org [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of Phil Blundell Sent: Wednesday, June 22, 2011 7:45 PM To: Patches and discussions about the oe-core layer Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe. I still don't quite understand why the virtual/libiconv provider has moved from eglibc to eglibc-locale. Can you explain what's going on there? Generally, the purpose of this patch is to reduce eglibc's do_package time. Since many other recipes' do_package have dependency on eglibc's do_package, thus the sooner eglibc could finish its do_package, the ealier can other do_packages be executed, which improves the parallelism. From our study, we found most of eglibc's do_package time is used to handle locale. Therefore we split all the locale related stuffs into a separately recipe eglibc-locale. Yes, I understand that. But I don't think this answers the question about why, specifically, glibc-locale is now PROVIDEing virtual/libiconv. Does that recipe, in fact, implement libiconv? I must admit I was wondering about this too. Isn't the libiconv functionality still packaged as part of the libc recipe, not the locale part? yes it should be in libc Oh, it should be a mistake. I have changed that in my patch and doing test for it now. Thanks, Dongxiao Cheers, Richard ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On Thu, 2011-06-23 at 12:08 +0800, Xu, Dongxiao wrote: -Original Message- From: openembedded-core-boun...@lists.openembedded.org [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of Phil Blundell Sent: Wednesday, June 22, 2011 7:45 PM To: Patches and discussions about the oe-core layer Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe. I still don't quite understand why the virtual/libiconv provider has moved from eglibc to eglibc-locale. Can you explain what's going on there? Generally, the purpose of this patch is to reduce eglibc's do_package time. Since many other recipes' do_package have dependency on eglibc's do_package, thus the sooner eglibc could finish its do_package, the ealier can other do_packages be executed, which improves the parallelism. From our study, we found most of eglibc's do_package time is used to handle locale. Therefore we split all the locale related stuffs into a separately recipe eglibc-locale. Yes, I understand that. But I don't think this answers the question about why, specifically, glibc-locale is now PROVIDEing virtual/libiconv. Does that recipe, in fact, implement libiconv? p. Thanks, Dongxiao p. On Wed, 2011-06-22 at 17:01 +0800, Dongxiao Xu wrote: *libc's do_package will cost a lot of time due to the locale handing, which may delay the other recipe's do_package task and affect the build performance. This commit moves locale handling into a separate recipe *libc-locale. Signed-off-by: Dongxiao Xu dongxiao...@intel.com --- meta/classes/libc-common.bbclass | 23 meta/classes/libc-package.bbclass | 24 meta/conf/distro/include/tclibc-eglibc.inc |2 +- meta/conf/distro/include/tclibc-glibc.inc |2 +- meta/recipes-core/eglibc/eglibc-initial_2.13.bb|4 ++ meta/recipes-core/eglibc/eglibc-locale_2.13.bb | 58 meta/recipes-core/eglibc/eglibc-package.inc| 32 +-- meta/recipes-core/eglibc/eglibc.inc|4 +- meta/recipes-core/eglibc/eglibc_2.13.bb| 12 meta/recipes-core/glibc/glibc-initial_2.10.1.bb|4 ++ meta/recipes-core/glibc/glibc-locale_2.10.1.bb | 50 + meta/recipes-core/glibc/glibc-package.inc | 29 ++- meta/recipes-core/glibc/glibc.inc |2 +- meta/recipes-core/glibc/glibc_2.10.1.bb| 12 .../meta/external-csl-toolchain_2008q3-72.bb |1 + 15 files changed, 176 insertions(+), 83 deletions(-) create mode 100644 meta/classes/libc-common.bbclass create mode 100644 meta/recipes-core/eglibc/eglibc-locale_2.13.bb create mode 100644 meta/recipes-core/glibc/glibc-locale_2.10.1.bb diff --git a/meta/conf/distro/include/tclibc-eglibc.inc b/meta/conf/distro/include/tclibc-eglibc.inc index a4c648e..90de04f 100644 --- a/meta/conf/distro/include/tclibc-eglibc.inc +++ b/meta/conf/distro/include/tclibc-eglibc.inc @@ -10,7 +10,7 @@ TARGET_OS_powerpc = linux${@['','-gnuspe'][bb.data.getVar('BASE_PACKAGE_ARCH',d # Add glibc overrides to the overrides for eglibc. OVERRIDES .= :libc-glibc -PREFERRED_PROVIDER_virtual/libiconv ?= eglibc +PREFERRED_PROVIDER_virtual/libiconv ?= eglibc-locale PREFERRED_PROVIDER_virtual/libiconv-nativesdk ?= eglibc-nativesdk PREFERRED_PROVIDER_virtual/libintl ?= eglibc PREFERRED_PROVIDER_virtual/libc ?= eglibc diff --git a/meta/conf/distro/include/tclibc-glibc.inc b/meta/conf/distro/include/tclibc-glibc.inc index e5843b0..748c23f 100644 --- a/meta/conf/distro/include/tclibc-glibc.inc +++ b/meta/conf/distro/include/tclibc-glibc.inc @@ -10,7 +10,7 @@ TARGET_OS_powerpc = linux${@['','-gnuspe'][bb.data.getVar('BASE_PACKAGE_ARCH',d # Add glibc to the overrides. OVERRIDES =. libc-glibc: -PREFERRED_PROVIDER_virtual/libiconv ?= glibc +PREFERRED_PROVIDER_virtual/libiconv ?= glibc-locale PREFERRED_PROVIDER_virtual/libiconv-nativesdk ?= glibc-nativesdk PREFERRED_PROVIDER_virtual/libintl ?= glibc PREFERRED_PROVIDER_virtual/libc ?= glibc ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On 06/23/2011 03:14 AM, Richard Purdie wrote: On Thu, 2011-06-23 at 10:40 +0100, Phil Blundell wrote: On Thu, 2011-06-23 at 12:08 +0800, Xu, Dongxiao wrote: -Original Message- From: openembedded-core-boun...@lists.openembedded.org [mailto:openembedded-core-boun...@lists.openembedded.org] On Behalf Of Phil Blundell Sent: Wednesday, June 22, 2011 7:45 PM To: Patches and discussions about the oe-core layer Subject: Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe. I still don't quite understand why the virtual/libiconv provider has moved from eglibc to eglibc-locale. Can you explain what's going on there? Generally, the purpose of this patch is to reduce eglibc's do_package time. Since many other recipes' do_package have dependency on eglibc's do_package, thus the sooner eglibc could finish its do_package, the ealier can other do_packages be executed, which improves the parallelism. From our study, we found most of eglibc's do_package time is used to handle locale. Therefore we split all the locale related stuffs into a separately recipe eglibc-locale. Yes, I understand that. But I don't think this answers the question about why, specifically, glibc-locale is now PROVIDEing virtual/libiconv. Does that recipe, in fact, implement libiconv? I must admit I was wondering about this too. Isn't the libiconv functionality still packaged as part of the libc recipe, not the locale part? yes it should be in libc Cheers, Richard ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
I still don't quite understand why the virtual/libiconv provider has moved from eglibc to eglibc-locale. Can you explain what's going on there? p. On Wed, 2011-06-22 at 17:01 +0800, Dongxiao Xu wrote: *libc's do_package will cost a lot of time due to the locale handing, which may delay the other recipe's do_package task and affect the build performance. This commit moves locale handling into a separate recipe *libc-locale. Signed-off-by: Dongxiao Xu dongxiao...@intel.com --- meta/classes/libc-common.bbclass | 23 meta/classes/libc-package.bbclass | 24 meta/conf/distro/include/tclibc-eglibc.inc |2 +- meta/conf/distro/include/tclibc-glibc.inc |2 +- meta/recipes-core/eglibc/eglibc-initial_2.13.bb|4 ++ meta/recipes-core/eglibc/eglibc-locale_2.13.bb | 58 meta/recipes-core/eglibc/eglibc-package.inc| 32 +-- meta/recipes-core/eglibc/eglibc.inc|4 +- meta/recipes-core/eglibc/eglibc_2.13.bb| 12 meta/recipes-core/glibc/glibc-initial_2.10.1.bb|4 ++ meta/recipes-core/glibc/glibc-locale_2.10.1.bb | 50 + meta/recipes-core/glibc/glibc-package.inc | 29 ++- meta/recipes-core/glibc/glibc.inc |2 +- meta/recipes-core/glibc/glibc_2.10.1.bb| 12 .../meta/external-csl-toolchain_2008q3-72.bb |1 + 15 files changed, 176 insertions(+), 83 deletions(-) create mode 100644 meta/classes/libc-common.bbclass create mode 100644 meta/recipes-core/eglibc/eglibc-locale_2.13.bb create mode 100644 meta/recipes-core/glibc/glibc-locale_2.10.1.bb diff --git a/meta/conf/distro/include/tclibc-eglibc.inc b/meta/conf/distro/include/tclibc-eglibc.inc index a4c648e..90de04f 100644 --- a/meta/conf/distro/include/tclibc-eglibc.inc +++ b/meta/conf/distro/include/tclibc-eglibc.inc @@ -10,7 +10,7 @@ TARGET_OS_powerpc = linux${@['','-gnuspe'][bb.data.getVar('BASE_PACKAGE_ARCH',d # Add glibc overrides to the overrides for eglibc. OVERRIDES .= :libc-glibc -PREFERRED_PROVIDER_virtual/libiconv ?= eglibc +PREFERRED_PROVIDER_virtual/libiconv ?= eglibc-locale PREFERRED_PROVIDER_virtual/libiconv-nativesdk ?= eglibc-nativesdk PREFERRED_PROVIDER_virtual/libintl ?= eglibc PREFERRED_PROVIDER_virtual/libc ?= eglibc diff --git a/meta/conf/distro/include/tclibc-glibc.inc b/meta/conf/distro/include/tclibc-glibc.inc index e5843b0..748c23f 100644 --- a/meta/conf/distro/include/tclibc-glibc.inc +++ b/meta/conf/distro/include/tclibc-glibc.inc @@ -10,7 +10,7 @@ TARGET_OS_powerpc = linux${@['','-gnuspe'][bb.data.getVar('BASE_PACKAGE_ARCH',d # Add glibc to the overrides. OVERRIDES =. libc-glibc: -PREFERRED_PROVIDER_virtual/libiconv ?= glibc +PREFERRED_PROVIDER_virtual/libiconv ?= glibc-locale PREFERRED_PROVIDER_virtual/libiconv-nativesdk ?= glibc-nativesdk PREFERRED_PROVIDER_virtual/libintl ?= glibc PREFERRED_PROVIDER_virtual/libc ?= glibc ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On 06/22/2011 02:01 AM, Dongxiao Xu wrote: *libc's do_package will cost a lot of time due to the locale handing, which may delay the other recipe's do_package task and affect the build performance. you could also move the locale generation into a task of its own instead of new recipe This commit moves locale handling into a separate recipe *libc-locale. Signed-off-by: Dongxiao Xudongxiao...@intel.com --- meta/classes/libc-common.bbclass | 23 meta/classes/libc-package.bbclass | 24 meta/conf/distro/include/tclibc-eglibc.inc |2 +- meta/conf/distro/include/tclibc-glibc.inc |2 +- meta/recipes-core/eglibc/eglibc-initial_2.13.bb|4 ++ meta/recipes-core/eglibc/eglibc-locale_2.13.bb | 58 meta/recipes-core/eglibc/eglibc-package.inc| 32 +-- meta/recipes-core/eglibc/eglibc.inc|4 +- meta/recipes-core/eglibc/eglibc_2.13.bb| 12 meta/recipes-core/glibc/glibc-initial_2.10.1.bb|4 ++ meta/recipes-core/glibc/glibc-locale_2.10.1.bb | 50 + meta/recipes-core/glibc/glibc-package.inc | 29 ++- meta/recipes-core/glibc/glibc.inc |2 +- meta/recipes-core/glibc/glibc_2.10.1.bb| 12 .../meta/external-csl-toolchain_2008q3-72.bb |1 + 15 files changed, 176 insertions(+), 83 deletions(-) create mode 100644 meta/classes/libc-common.bbclass create mode 100644 meta/recipes-core/eglibc/eglibc-locale_2.13.bb create mode 100644 meta/recipes-core/glibc/glibc-locale_2.10.1.bb diff --git a/meta/classes/libc-common.bbclass b/meta/classes/libc-common.bbclass new file mode 100644 index 000..bae0ace --- /dev/null +++ b/meta/classes/libc-common.bbclass @@ -0,0 +1,23 @@ +do_install() { + oe_runmake install_root=${D} install + for r in ${rpcsvc}; do + h=`echo $r|sed -e's,\.x$,.h,'` + install -m 0644 ${S}/sunrpc/rpcsvc/$h ${D}/${includedir}/rpcsvc/ + done + install -m 0644 ${WORKDIR}/etc/ld.so.conf ${D}/${sysconfdir}/ + install -d ${D}${libdir}/locale + make -f ${WORKDIR}/generate-supported.mk IN=${S}/localedata/SUPPORTED OUT=${WORKDIR}/SUPPORTED + # get rid of some broken files... + for i in ${GLIBC_BROKEN_LOCALES}; do + grep -v $i ${WORKDIR}/SUPPORTED ${WORKDIR}/SUPPORTED.tmp + mv ${WORKDIR}/SUPPORTED.tmp ${WORKDIR}/SUPPORTED + done + rm -f ${D}{sysconfdir}/rpc + rm -rf ${D}${datadir}/zoneinfo + rm -rf ${D}${libexecdir}/getconf +} + +def get_libc_fpu_setting(bb, d): +if bb.data.getVar('TARGET_FPU', d, 1) in [ 'soft' ]: +return --without-fp +return diff --git a/meta/classes/libc-package.bbclass b/meta/classes/libc-package.bbclass index 55e3d48..5737af4 100644 --- a/meta/classes/libc-package.bbclass +++ b/meta/classes/libc-package.bbclass @@ -37,11 +37,6 @@ python __anonymous () { break } -def get_libc_fpu_setting(bb, d): -if bb.data.getVar('TARGET_FPU', d, 1) in [ 'soft' ]: -return --without-fp -return - OVERRIDES_append = :${TARGET_ARCH}-${TARGET_OS} do_configure_prepend() { @@ -84,25 +79,6 @@ rm -rf ${TMP_LOCALE} } -do_install() { - oe_runmake install_root=${D} install - for r in ${rpcsvc}; do - h=`echo $r|sed -e's,\.x$,.h,'` - install -m 0644 ${S}/sunrpc/rpcsvc/$h ${D}/${includedir}/rpcsvc/ - done - install -m 0644 ${WORKDIR}/etc/ld.so.conf ${D}/${sysconfdir}/ - install -d ${D}${libdir}/locale - make -f ${WORKDIR}/generate-supported.mk IN=${S}/localedata/SUPPORTED OUT=${WORKDIR}/SUPPORTED - # get rid of some broken files... - for i in ${GLIBC_BROKEN_LOCALES}; do - grep -v $i ${WORKDIR}/SUPPORTED ${WORKDIR}/SUPPORTED.tmp - mv ${WORKDIR}/SUPPORTED.tmp ${WORKDIR}/SUPPORTED - done - rm -f ${D}${sysconfdir}/rpc - rm -rf ${D}${datadir}/zoneinfo - rm -rf ${D}${libexecdir}/getconf -} - TMP_LOCALE=/tmp/locale${libdir}/locale do_prep_locale_tree() { diff --git a/meta/conf/distro/include/tclibc-eglibc.inc b/meta/conf/distro/include/tclibc-eglibc.inc index a4c648e..90de04f 100644 --- a/meta/conf/distro/include/tclibc-eglibc.inc +++ b/meta/conf/distro/include/tclibc-eglibc.inc @@ -10,7 +10,7 @@ TARGET_OS_powerpc = linux${@['','-gnuspe'][bb.data.getVar('BASE_PACKAGE_ARCH',d # Add glibc overrides to the overrides for eglibc. OVERRIDES .= :libc-glibc -PREFERRED_PROVIDER_virtual/libiconv ?= eglibc +PREFERRED_PROVIDER_virtual/libiconv ?= eglibc-locale PREFERRED_PROVIDER_virtual/libiconv-nativesdk ?= eglibc-nativesdk PREFERRED_PROVIDER_virtual/libintl ?= eglibc PREFERRED_PROVIDER_virtual/libc ?= eglibc diff --git a/meta/conf/distro/include/tclibc-glibc.inc b/meta/conf/distro/include/tclibc-glibc.inc index
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On Wed, 2011-06-22 at 07:44 -0700, Khem Raj wrote: On 06/22/2011 02:01 AM, Dongxiao Xu wrote: *libc's do_package will cost a lot of time due to the locale handing, which may delay the other recipe's do_package task and affect the build performance. you could also move the locale generation into a task of its own instead of new recipe It'd be hard to get the dependencies to work out right if you did that, though. I think a new recipe probably is the right answer. p. [ 10 quoted lines snipped ... ] ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On 6/22/11 9:47 AM, Phil Blundell wrote: On Wed, 2011-06-22 at 07:44 -0700, Khem Raj wrote: On 06/22/2011 02:01 AM, Dongxiao Xu wrote: *libc's do_package will cost a lot of time due to the locale handing, which may delay the other recipe's do_package task and affect the build performance. you could also move the locale generation into a task of its own instead of new recipe It'd be hard to get the dependencies to work out right if you did that, though. I think a new recipe probably is the right answer. And in the case of libc, the locales need to be done -after- the populate_sysroot, or we don't get the parallelism improvement. --Mark p. [ 10 quoted lines snipped ... ] ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On 06/22/2011 08:17 AM, Mark Hatle wrote: On 6/22/11 9:47 AM, Phil Blundell wrote: On Wed, 2011-06-22 at 07:44 -0700, Khem Raj wrote: On 06/22/2011 02:01 AM, Dongxiao Xu wrote: *libc's do_package will cost a lot of time due to the locale handing, which may delay the other recipe's do_package task and affect the build performance. you could also move the locale generation into a task of its own instead of new recipe It'd be hard to get the dependencies to work out right if you did that, though. I think a new recipe probably is the right answer. And in the case of libc, the locales need to be done -after- the populate_sysroot, or we don't get the parallelism improvement. ofcourse thats why I was suggesting a new task. --Mark p. [ 10 quoted lines snipped ... ] ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On Thu, 2011-06-09 at 12:14 +0100, Phil Blundell wrote: On Wed, 2011-06-08 at 16:35 +0100, Richard Purdie wrote: I'm not sure how this would reduce performance of builds of a few threads, it should just make better use of any available spare processing capacity throughout the build. If I'm reading the patch right, it does involve a certain amount of extra copying around of the locale-related files since they need to be stashed away in some place where libc-locale.bb can find them later. I don't think these files are especially big, but there are quite a few of them (which is the whole reason that libc's do_package was slow in the first place). I must admit that I'm slightly surprised that libc's packaging stage is becoming a bottleneck for anything other than the very smallest builds. If there's any substantial amount of other material being compiled then I would expect that there would be enough do_compile() work to keep the other threads busy until libc was done packaging. That's not to say that I think this change is a bad idea, but I would be interested to know what the actual impact is for real-world workloads. And, just on a point of principle, any time we're making a chance for performance-related reasons I think we should always have measurements to back it up. Totally agreed and this decision is being made on real world data even if its perhaps not clearly being presented here. $ cat buildstats/core-image-sato-qemux86/201106030912/eglibc-2.13-r1+svnr13356/do_package | grep time eglibc-2.13-r1+svnr13356: do_package: Elapsed time: 828.01 seconds As you can see, eglibc do_package takes about 14 minutes which is about 14% of our build time. That is a long time to block pretty much all packaging activity, particularly if you have access to something with several cores. When it does complete, even on my 4 core system you see a stampeding herd of packaging happening on the build charts suggesting a backlog does build up. For those reasons I do think its a reasonable change. Cheers, Richard ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On Thu, 2011-06-09 at 12:43 +0100, Phil Blundell wrote: On Thu, 2011-06-09 at 12:29 +0100, Richard Purdie wrote: As you can see, eglibc do_package takes about 14 minutes which is about 14% of our build time. That is a long time to block pretty much all packaging activity, particularly if you have access to something with several cores. When it does complete, even on my 4 core system you see a stampeding herd of packaging happening on the build charts suggesting a backlog does build up. Yeah, I can imagine that a backlog of packaging activity does build up. The thing I'm not entirely clear on is whether this is actually causing some threads to get starved of work (and hence the total build time to be longer than it needs to be) or whether we're really just shifting things around in the timeline without making much/any difference to the overall build duration. It will certainly be a net win on large core systems which I know of people running builds on as task starvation happens there. Last time I tested this on a 4 core I think we saw a couple of minutes improvement in build time so it appears beneficial there too. (I'm not familiar enough with bitbake's scheduler to know whether it will schedule tasks as early as possible, or as late as possible, or something else.) Roughly speaking, the default scheduler runs tasks with the most things depending upon them first. This means it will fill any spare threads with lower priority tasks such as packaging. Just as a matter of interest, are you using qemu-based locale generation or the cross localedef for your measurement? 14 minutes does sound like an awfully long time and I wonder whether there is anything we could do in absolute terms to just speed that process up. I'm using cross localedef. Its faster than with qemu but still slow. I'm very open to suggestions on how to speed it up as it does take an age. Cheers, Richard ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On Thu, 2011-06-09 at 14:15 +0100, Richard Purdie wrote: On Thu, 2011-06-09 at 12:43 +0100, Phil Blundell wrote: On Thu, 2011-06-09 at 12:29 +0100, Richard Purdie wrote: As you can see, eglibc do_package takes about 14 minutes which is about 14% of our build time. That is a long time to block pretty much all packaging activity, particularly if you have access to something with several cores. When it does complete, even on my 4 core system you see a stampeding herd of packaging happening on the build charts suggesting a backlog does build up. Yeah, I can imagine that a backlog of packaging activity does build up. The thing I'm not entirely clear on is whether this is actually causing some threads to get starved of work (and hence the total build time to be longer than it needs to be) or whether we're really just shifting things around in the timeline without making much/any difference to the overall build duration. It will certainly be a net win on large core systems which I know of people running builds on as task starvation happens there. Last time I tested this on a 4 core I think we saw a couple of minutes improvement in build time so it appears beneficial there too. Just as another data point, I'm running builds on my 4 core machine here and it is scheduling many do_package tasks including long running ones such as perl's in parallel with eglibc-locale's do_package task. It would appear the bitbake scheduler believes there is benefit in splitting these out at least. Cheers, Richard ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
Op 9 jun 2011, om 15:15 heeft Richard Purdie het volgende geschreven: On Thu, 2011-06-09 at 12:43 +0100, Phil Blundell wrote: On Thu, 2011-06-09 at 12:29 +0100, Richard Purdie wrote: As you can see, eglibc do_package takes about 14 minutes which is about 14% of our build time. That is a long time to block pretty much all packaging activity, particularly if you have access to something with several cores. When it does complete, even on my 4 core system you see a stampeding herd of packaging happening on the build charts suggesting a backlog does build up. Yeah, I can imagine that a backlog of packaging activity does build up. The thing I'm not entirely clear on is whether this is actually causing some threads to get starved of work (and hence the total build time to be longer than it needs to be) or whether we're really just shifting things around in the timeline without making much/any difference to the overall build duration. It will certainly be a net win on large core systems which I know of people running builds on as task starvation happens there. Last time I tested this on a 4 core I think we saw a couple of minutes improvement in build time so it appears beneficial there too. (I'm not familiar enough with bitbake's scheduler to know whether it will schedule tasks as early as possible, or as late as possible, or something else.) Roughly speaking, the default scheduler runs tasks with the most things depending upon them first. This means it will fill any spare threads with lower priority tasks such as packaging. Just as a matter of interest, are you using qemu-based locale generation or the cross localedef for your measurement? 14 minutes does sound like an awfully long time and I wonder whether there is anything we could do in absolute terms to just speed that process up. I'm using cross localedef. Its faster than with qemu but still slow. I'm very open to suggestions on how to speed it up as it does take an age. Does cross localegen obey parallel make or does it only do a single locale at a time? regards, Koen ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On Thu, 2011-06-09 at 15:53 +0200, Koen Kooi wrote: Does cross localegen obey parallel make or does it only do a single locale at a time? Just one at a time. It's run out of Python code in libc-package, not by make, so there is no completely straightforward way to get it to respect PARALLEL_MAKE. I guess you could teach libc-package how to generate a makefile and then run it; that'd probably be the easiest way to parallelize that particular process. p. ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On Wed, 2011-06-08 at 17:08 +0800, Dongxiao Xu wrote: *libc's do_package will cost a lot of time due to the locale handing, which may delay the other recipe's do_package task and affect the build performance. This commit moves locale handling into a separate recipe *libc-locale. Can you quantify the effect on build performance a bit? If I understand correctly, you're basically saying that the goal is to increase parallelism. Does that cause reduced performance for people running with few threads? For example, it would be interesting to know what the before and after times are for some representative image build using (say) BB_NUMBER_THREADS=1 and BB_NUMBER_THREADS=16. Also, see below. diff --git a/meta/conf/distro/include/tclibc-eglibc.inc b/meta/conf/distro/include/tclibc-eglibc.inc index a4c648e..8997d9e 100644 --- a/meta/conf/distro/include/tclibc-eglibc.inc +++ b/meta/conf/distro/include/tclibc-eglibc.inc @@ -10,7 +10,7 @@ TARGET_OS_powerpc = linux${@['','-gnuspe'][bb.data.getVar('BASE_PACKAGE_ARCH',d # Add glibc overrides to the overrides for eglibc. OVERRIDES .= :libc-glibc -PREFERRED_PROVIDER_virtual/libiconv ?= eglibc +PREFERRED_PROVIDER_virtual/libiconv ?= eglibc-locale That looks a bit strange. Is this really correct? - eglibc-localedata-i18n \ - eglibc-gconv-ibm850 \ - eglibc-gconv-cp1252 \ - eglibc-gconv-iso8859-1 \ - eglibc-gconv-iso8859-15 \ + eglibc-locale-localedata-i18n \ + eglibc-locale-gconv-ibm850 \ + eglibc-locale-gconv-cp1252 \ + eglibc-locale-gconv-iso8859-1 \ + eglibc-locale-gconv-iso8859-15 \ This kind of thing is definitely not very desirable. Can you find a way to keep the output package names the same? p. ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] libc-locale: split locale handling from libc recipe.
On Wed, 2011-06-08 at 10:36 +0100, Phil Blundell wrote: On Wed, 2011-06-08 at 17:08 +0800, Dongxiao Xu wrote: *libc's do_package will cost a lot of time due to the locale handing, which may delay the other recipe's do_package task and affect the build performance. This commit moves locale handling into a separate recipe *libc-locale. Can you quantify the effect on build performance a bit? If I understand correctly, you're basically saying that the goal is to increase parallelism. Does that cause reduced performance for people running with few threads? One of the side effects of debian.bbclass is that it requires all dependencies to do_package before any package itself can do_package so any renaming of dependencies can be accounted for. There is an issue if do_package for libc takes an age as it holds up any other tasks from writing out packages. Pretty much most things depend on libc. This patch therefore splits it into two stages and means that packaging of things depending on libc can happen sooner thereby increasing the potential parallelism of the packaging stages of builds. There is a very clear step on the bootchart graphs of builds I made showing this. I'm not sure how this would reduce performance of builds of a few threads, it should just make better use of any available spare processing capacity throughout the build. Cheers, Richard ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core