I think I need to write down a bit of a summary status on where we (and I) am at with things.
The good news is that scarthgap is released. On the downside, we messed up the public hash server url in local.conf.sample so we have a point release in progress to fix that. The Sovereign Tech Fund (STF) work is drawing to a close now, some tweaks are needed on some patch series to get them over the line and able to merge but the basic pieces are there. We have the PoC for bitbake-setup too. I have not had the time to look at it in detail yet (see below for an idea of why). That work brings us many steps forward in many key areas of the project and takes a number of weights off my mind. It adds some new ones too but change is at least refreshing. It isn't all rosy, in particular the autobuilder is struggling. The quality of builds for scarthgap isn't what I'd have liked and Steve is already abandoning rc builds due to intermittent failures. We're seeing the same failures with master. In particular: A rust reproducibility issue lurking somewhere: https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/8318 https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/8906/steps/17/logs/stdio https://bugzilla.yoctoproject.org/show_bug.cgi?id=15185 [both master and scarthgap] Intermittent do_compile failure in libportal: https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4760/steps/12/logs/stdio debuginfod keeps breaking: https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3299/steps/14/logs/stdio https://bugzilla.yoctoproject.org/show_bug.cgi?id=14937 qemuppc with intermittent issues: https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/8906/steps/17/logs/stdio qemuarm eSDK intermittent failure: https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/8946/steps/15/logs/stdio (network glitch or a problem with git/curl/certs in the tools?) We did have an intermittent python3 test hang. We've taken the decision just to disable that test for now as we don't have the time to debug it. The weird hang in the 6.6 kernel in early boot also remains. Paul Gortmaker made some progress in debugging it but I haven't found any time to help/support him. There is some kind of issue where the longer the arm servers have uptime, the more broken the builds get too. It is hard to be more specific, someone needs to do some analysis of the failure and try and find the pattern. We then get to the non oe-core failures that seem to be piling up: meta-oe source mirroring giving warnings: https://autobuilder.yoctoproject.org/typhoon/#/builders/156/builds/353/steps/12/logs/warnings meta-oe source mirroring fetch failures: https://autobuilder.yoctoproject.org/typhoon/#/builders/156/builds/353/steps/16/logs/stdio kirkstone meta-oe metrics failures: https://autobuilder.yoctoproject.org/typhoon/#/builders/138/builds/1612/steps/16/logs/stdio meta-ti nightly layer check failures: https://autobuilder.yoctoproject.org/typhoon/#/builders/121/builds/2064/steps/19/logs/stdio Yes, some of these need reporting to the right people, e.g. meta-ti. *Someone* actually needs to do it. If someone doesn't do it, someone needs to notice and find/remind someone to do it. We expanded testing to meta-oe on the basiss there would be help in addressing the issues but few people seem to be paying attention :(. Meanwhile, we have a pile of gcc 14 fixes piling up and pressure to get gcc 14 in so we can build a new uninative to fix things for people on Fedora 40 and arch. There are simple things others could do to help like replying to patches about missing patch information. The current autobuilder hardware is failing. We have a plan for a new setup however that new setup has issues that need debugging. The one resource I can direct to help with that is me, everyone else just runs away from intermittent issues or says it is too hard or they don't have access/experience/whatever. We have challenges in the CVE space, I'm hoping Marta has some patches coming which can help there soon. The open letter idea is all well and good but it needs socialising with people outside the project. There are also some bigger changes I think the project needs to make, WORKDIR changes are one example, we also need to revisit the layer dependency situation and "Yocto Project Compatible" needs work too, amongst many other things. Personally, I'm now really struggling. The intensity of the last few months has taken a toll on me and I simply can't switch into handling all of the above, I'm running on empty. This means I need a break, everyone knows that and agrees. The twist is that yes, everyone encourages me to do that, as long as I fix the issues on the current autobuilder, the issues on the new one, review and merge patches, make architecture/development changes at the right point in the release cycle, ensure proposals made for key changes are well thought out and don't upset anyone and so on. I'm sharing this not as a complaint, but to try and show people that if their patches aren't merging, it isn't that I/we don't care, it is just that there simply aren't enough hours in the day and I'm at breaking point if I don't let some things slow down. One of the key issues is that I act like a traffic director in the middle of all this. Nobody else is willing to step into all those different streams and help try and tame/direct them. The downside to that is that if I stop or slow down, things do break. Obviously, help in any of the most regular autobuilder failures would be much appreciated. Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#199143): https://lists.openembedded.org/g/openembedded-core/message/199143 Mute This Topic: https://lists.openembedded.org/mt/105997972/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-