Re: [yocto] how to intelligently avoid OOM build errors with BB_PRESSURE_MAX_MEMORY?

2023-12-01 Thread Ferry Toth 

Hi

Op 29-11-2023 om 10:45 schreef Martin Jansa:

I was experimenting quite a bit with this in:
https://github.com/shr-project/test-oe-build-time/commits/results 
 e.g.:

https://github.com/shr-project/test-oe-build-time/commit/d5111f4472ac397c0f1197eb6366ac7d2e56453f
 

https://github.com/shr-project/test-oe-build-time/commit/e97d8fe55fcf848416a8494a90da4ae6418b19fe
 


but be aware that reasonable BB_PRESSURE_MAX_ values are different for 
kirkstone and master, see:
https://lists.openembedded.org/g/bitbake-devel/message/15499 


which wasn't merged yet.

And for my system and typical builds BB_PRESSURE_MAX_* isn't able to 
prevent OOMK, because pressure is still fine when bitbake is starting 
chromium.do_compile and nodejs.do_compile tasks at the same time and 
OOMK gets triggered later even when only these 2 tasks are running in 
parallel with other tasks not started thanks to BB_PRESSURE_MAX_*.


So I'm setting a bit lower PARALLEL_MAKE for chromium instead as in:
BB_PRESSURE_MAX_CPU = "20"
BB_NUMBER_THREADS = "8"
BB_NUMBER_PARSE_THREADS = "64"
PARALLEL_MAKE = "-j 70 -l 140"
PARALLEL_MAKE:pn-qtwebengine = "-j 40"
PARALLEL_MAKE:pn-webruntime = "-j 40"

which works well for my system and my typical builds, YMMV for yours.

Regards,

On Wed, Nov 29, 2023 at 10:21 AM Robert P. J. Day 
> wrote:



   a colleague is using Wind River Linux and is getting occasional OOM
errors when taking advantage of maximum parallelism, so he took the
quick way out and dialed things back using BB_NUMBER_THREADS and
PARALLEL_MAKE.

   however, i am aware of the bitbake variables
BB_PRESSURE_MAX_{IO,CPU,MEMORY} and was wondering if there is a decent
writeup on how to use, perhaps, BB_PRESSURE_MAX_MEMORY, to dynamically
detect imminent OOM and adjust things. that is, how to calculate a
sane value for that as i haven't yet dug into the values under
/proc/pressure and how to interpret them?

   i've seen mentions of combining the above with the "memstat"
command, and other approaches. thoughts?

rday

p.s. is there a docs page that expands on this stuff?

There was an older thread "bitbake controlling memory use", maybe that 
might help.


Myself, I found these pressure things don't help. And even though we can 
set the number of make threads we still can have a number of makes in 
parallel. I patched bitbake to have make and ninja use a jobserver to 
limit the total number of compile threads to the number of cores.

See https://github.com/htot/meta-intel-edison/tree/kirkstone

So, if you have 1GB ram x #cores you should be fine except for 1 thing:
nodejs is linking 5 executables that require 4GB ram simultaneously. So 
if you don't have 20GB you will start thrashing up to the point that you 
have no mouse movement. This I solved by modifying the recipe to 
serialize the linking.












-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61790): https://lists.yoctoproject.org/g/yocto/message/61790
Mute This Topic: https://lists.yoctoproject.org/mt/102868689/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[yocto] [yocto-autobuilder-helper v2] scripts/run-toaster-tests.py: run via pytest and fix environment setup

2023-12-01 Thread Alexander Lussier-Cullen 
Strip quotes from directory variables as they caused path errors.
Add environment variable for the toaster artifact directory.
Migrate from tox and django tools to using pytest.
Install python module requirements from the script as this is no
longer handled by tox.

Signed-off-by: Alexander Lussier-Cullen 

CC: Richard Purdie 
---
 scripts/run-toaster-tests | 10 ++
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/scripts/run-toaster-tests b/scripts/run-toaster-tests
index 516965e..b694e37 100755
--- a/scripts/run-toaster-tests
+++ b/scripts/run-toaster-tests
@@ -16,13 +16,15 @@ pokydir=$(realpath "$2")
 cd $builddir
 
 bitbake -e > bbenv
-export SSTATE_DIR=$(grep '^SSTATE_DIR=' bbenv | cut -d "=" -f2-)
-export DL_DIR=$(grep '^DL_DIR=' bbenv | cut -d "=" -f2-)
+export SSTATE_DIR=$(grep '^SSTATE_DIR=' bbenv | cut -d "=" -f2- | sed -e 
's/^"//' -e 's/"$//')
+export DL_DIR=$(grep '^DL_DIR=' bbenv | cut -d "=" -f2- | sed -e 's/^"//' -e 
's/"$//')
 export TOASTER_DJANGO_TMPDIR=$builddir
+export TOASTER_DIR=$builddir
 
 mkdir -p toaster_logs
+
 python3 -m venv venv --without-pip --system-site-packages
 source venv/bin/activate
-python3 -m pip install tox
+python3 -m pip install -r $pokydir/bitbake/toaster-requirements.txt -r 
$pokydir/bitbake/lib/toaster/tests/toaster-tests-requirements.txt
 
-tox -c $pokydir/bitbake/lib/toaster/tox.ini
+python3 -m pytest -c $pokydir/bitbake/lib/toaster/pytest.ini 
$pokydir/bitbake/lib/toaster/tests/
-- 
2.34.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61789): https://lists.yoctoproject.org/g/yocto/message/61789
Mute This Topic: https://lists.yoctoproject.org/mt/102916819/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[yocto] How to download file from Microsoft SharePoint?

2023-12-01 Thread Martin Leduc via lists.yoctoproject.org 
Hi I've a file to retrieve from Microsoft SharePoint.  I've a shared link to 
download the file.  If I cut'n paste the URL in the browser, it works but Yocto 
download the web page.

I've added the BB_STRICT_CHECKSUM = "0" to avoid the SRCURI checksum mismatch 
but the file isn't downloaded.

What esle I can do or try?

BR,

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61788): https://lists.yoctoproject.org/g/yocto/message/61788
Mute This Topic: https://lists.yoctoproject.org/mt/102914150/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [yocto] The postinstall intercept hook 'update_mime_database' failed

2023-12-01 Thread Alexander Kanavin 
You need to try plain poky then because the error does not happen
there. Then figure out where the difference is.

Alex

On Fri, 1 Dec 2023 at 12:56, Mans Zigher  wrote:
>
> I have now tried building a multitude of different machines and
> switched to core-minimal-image and still I get these postinstall
> intercept scripts failing. Depending what distro and machine I build I
> get different errors but It is always a postinstall intercept hook
> script that fails and it is always related to missing some directory
> under the rootfs of the image that I am currently trying to build.
>
> BR
>
> Den tors 30 nov. 2023 kl 19:11 skrev Måns :
> >
> > Hi,
> >
> > I have taken over a imx8mm project that is based on kirkstone. I don't
> > think it is an official release and some layers are not even a
> > kirkstone release. The project has a reference build but currently it
> > is not possible to build from scratch. I have been fixing some issues
> > on the way to try and build a complete build from scratch but now I
> > have encountered an issue when creating the rootfs that I am
> > struggling with. The issues that I am getting is
> >
> > ERROR: fsl-image-gui-1.0-r0 do_rootfs: The postinstall intercept hook
> > 'update_mime_database' failed, details in
> > /workspace/mender-imx8mm/builds/tmp/work/imx8mm_var_dart-fslc-linux/fsl-image-gui/1.0-r0/temp/log.do_rootfs
> > ERROR: Logfile of failure stored in:
> > /workspace/mender-imx8mm/builds/tmp/work/imx8mm_var_dart-fslc-linux/fsl-image-gui/1.0-r0/temp/log.do_rootfs.335539
> > ERROR: Task 
> > (/workspace/mender-imx8mm/sources/meta-variscite-sdk/recipes-fsl/images/fsl-image-gui.bb:do_rootfs)
> > failed with exit code '1'
> >
> > checking the log file
> >
> > NOTE: Running intercept scripts:
> > NOTE: > Executing update_mime_database intercept ...
> > NOTE: Exit code 1. Output:
> > Updating MIME database... this may take a while.
> > Directory 
> > '/workspace/mender-imx8mm/builds/tmp/work/imx8mm_var_dart-fslc-linux/fsl-image-gui/1.0-r0/rootfs/packages'
> > does not exist!
> >
> > In poky/meta/classes/mime.bbclass there is a comment that seems to
> > refer to this issue
> >
> > # $D${MIMEDIR}/packages belong to package shared-mime-info-data,
> > # packages like libfm-mime depend on shared-mime-info-data.
> > # after shared-mime-info-data uninstalled, $D${MIMEDIR}/packages
> > # is removed, but update-mime-database need this dir to update
> > # database, workaround to create one and remove it later
> > if [ ! -d $D${MIMEDIR}/packages ]; then
> >mkdir -p $D${MIMEDIR}/packages
> >update-mime-database $D${MIMEDIR}
> >rmdir --ignore-fail-on-non-empty $D${MIMEDIR}/packages
> >else
> >update-mime-database $D${MIMEDIR}
> >fi
> >
> > This is not called so no packages directory doesn't exist and the
> > intercept script failed. I can adjust the script
> > poky/scripts/postinst-intercepts/update_mime_database
> >
> > echo "Updating MIME database... this may take a while."
> > update-mime-database $D${mimedir}
> >
> > By adding a similar logic I will get forward but with my build but
> > that doesn't seem like the right solution. Also I just get some other
> > issue related to a second intercept script update_desktop_database so
> > to me it seems more like something is not correctly set up in the
> > build. I am trying to understand how these intercept scripts are
> > configured and why they are not working for my build. Any pointers on
> > how to move forward would be much appreciated.
> >
> > BR
>
> 
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61787): https://lists.yoctoproject.org/g/yocto/message/61787
Mute This Topic: https://lists.yoctoproject.org/mt/102901287/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [yocto] The postinstall intercept hook 'update_mime_database' failed

2023-12-01 Thread Mans Zigher 
I have now tried building a multitude of different machines and
switched to core-minimal-image and still I get these postinstall
intercept scripts failing. Depending what distro and machine I build I
get different errors but It is always a postinstall intercept hook
script that fails and it is always related to missing some directory
under the rootfs of the image that I am currently trying to build.

BR

Den tors 30 nov. 2023 kl 19:11 skrev Måns :
>
> Hi,
>
> I have taken over a imx8mm project that is based on kirkstone. I don't
> think it is an official release and some layers are not even a
> kirkstone release. The project has a reference build but currently it
> is not possible to build from scratch. I have been fixing some issues
> on the way to try and build a complete build from scratch but now I
> have encountered an issue when creating the rootfs that I am
> struggling with. The issues that I am getting is
>
> ERROR: fsl-image-gui-1.0-r0 do_rootfs: The postinstall intercept hook
> 'update_mime_database' failed, details in
> /workspace/mender-imx8mm/builds/tmp/work/imx8mm_var_dart-fslc-linux/fsl-image-gui/1.0-r0/temp/log.do_rootfs
> ERROR: Logfile of failure stored in:
> /workspace/mender-imx8mm/builds/tmp/work/imx8mm_var_dart-fslc-linux/fsl-image-gui/1.0-r0/temp/log.do_rootfs.335539
> ERROR: Task 
> (/workspace/mender-imx8mm/sources/meta-variscite-sdk/recipes-fsl/images/fsl-image-gui.bb:do_rootfs)
> failed with exit code '1'
>
> checking the log file
>
> NOTE: Running intercept scripts:
> NOTE: > Executing update_mime_database intercept ...
> NOTE: Exit code 1. Output:
> Updating MIME database... this may take a while.
> Directory 
> '/workspace/mender-imx8mm/builds/tmp/work/imx8mm_var_dart-fslc-linux/fsl-image-gui/1.0-r0/rootfs/packages'
> does not exist!
>
> In poky/meta/classes/mime.bbclass there is a comment that seems to
> refer to this issue
>
> # $D${MIMEDIR}/packages belong to package shared-mime-info-data,
> # packages like libfm-mime depend on shared-mime-info-data.
> # after shared-mime-info-data uninstalled, $D${MIMEDIR}/packages
> # is removed, but update-mime-database need this dir to update
> # database, workaround to create one and remove it later
> if [ ! -d $D${MIMEDIR}/packages ]; then
>mkdir -p $D${MIMEDIR}/packages
>update-mime-database $D${MIMEDIR}
>rmdir --ignore-fail-on-non-empty $D${MIMEDIR}/packages
>else
>update-mime-database $D${MIMEDIR}
>fi
>
> This is not called so no packages directory doesn't exist and the
> intercept script failed. I can adjust the script
> poky/scripts/postinst-intercepts/update_mime_database
>
> echo "Updating MIME database... this may take a while."
> update-mime-database $D${mimedir}
>
> By adding a similar logic I will get forward but with my build but
> that doesn't seem like the right solution. Also I just get some other
> issue related to a second intercept script update_desktop_database so
> to me it seems more like something is not correctly set up in the
> build. I am trying to understand how these intercept scripts are
> configured and why they are not working for my build. Any pointers on
> how to move forward would be much appreciated.
>
> BR

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61786): https://lists.yoctoproject.org/g/yocto/message/61786
Mute This Topic: https://lists.yoctoproject.org/mt/102901287/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-