Re: pkg server for current/arm64 stopped ? [main-armv7 on ampere2, elapsed so far: 651:21:56]
On 2024-04-24 02:12:41 (+0800), Mark Millard wrote: On Apr 19, 2024, at 07:16, Philip Paeps wrote: On 2024-04-18 23:02:30 (+0800), Mark Millard wrote: void wrote on Date: Thu, 18 Apr 2024 14:08:36 UTC : Not sure where to post this.. The last bulk build for arm64 appears to have happened around mid-March on ampere2. Is it broken? main-armv7 building is broken and the last completed build was the one started on Mon, 19 Feb 2024 12:32:10 GMT. It gets stuck making no progress until manually forced to stop, which leads to huge elapsed times for the incomplete builds: pd5512ae7b8c6_s75464941dc 34472 12282 (+9196) 107 (+77) 4753 (+2247) 1390 (+529) 15940 parallel_build: Fri, 22 Mar 2024 11:05:01 GMT 651:21:56 p43e3af5f5763_sf5f08e41aa 19809 5919 (+3126) 137 (+100) 5363 (+2741) 1395 (+522) 6995 parallel_build: Wed, 28 Feb 2024 15:46:14 GMT 359:42:14 ampere2 ampere2 alternates between trying to build main-arm64 and main-armv7, so main-armv7 being stuck blocks main-arm64 from building. One can see that all 13 job ID's show over 570 hours: http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=pd5512ae7b8c6_s75464941dc It is not random which packages are building when this happens. Compare: http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=p43e3af5f5763_sf5f08e41aa By contrast, the 19 Feb 2024 from-scratch (full) build worked: http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=pe9c9c73181b5_sbd45bbe440 My guess is that FreeBSD has something that broken after bd45bbe440 that was broken as of f5f08e41aa and was still broken at 75464941dc . I'll kill the build on ampere2 again. Thanks for the nudge. We don't really have good monitoring for this. Also: builds should time out after 36 hours. The fact that this one does not is a bug in itself. Philip [hat: clusteradm] I'll note that I've never managed to replicate the problem for building for armv7 on aarch64. But my context never has the likes of: QUOTE Host OSVERSION: 156 Jail OSVERSION: 1500015 . . . !!! Jail is newer than host. (Jail: 1500015, Host: 156) !!! !!! This is not supported. !!! !!! Host kernel must be same or newer than jail. !!! !!! Expect build failures. !!! END QUOTE but always has the two OSVERSION's the same, such as: Host OSVERSION: 1500015 Jail OSVERSION: 1500015 or, recently, Host OSVERSION: 1500018 Jail OSVERSION: 1500018 My bulk runs do go through the sequence where the hangups have repeated for main-armv7 on ampere2. I wonder what would happen if "Host OSVERSION" was updated (modernized) to match the modern "Jail OSVERSION" that would be used? The package builders are due for a regular refresh to newer -CURRENT dogfood. I'll do the aarch64 builders first this time. I've set /root/stop-builds on them. I'll upgrade them when they go idle. Or I'll kill them if they take much longer to build what they're building. It annoys me that they do not stop building after 36 hours, like they're supposed to. They're currently running: n266879-6abee52e0d79 2023-12-09 01:06:28 jlduran strfmon: Silence scan-build warning Our current clusteradm build is: n269399-bbc6e6c5ec8c 2024-04-14 03:12:36 sigsys daemon: fix -R to enable supervision mode I may do a new build while waiting for them to go idle: - quarterly 140arm64 1b931669de11 parallel_build 28776 15299 33 588 985 0 11871 3D:01:08:29 https://pkg-status.freebsd.org/ampere1/build.html?mastername=140arm64-quarterly&build=1b931669de11 - default main-arm64 p1c7a816cd0ad_s1bd4f769caf parallel_build 34528 19888 65 669980 0 12926 4D:00:52:21 https://pkg-status.freebsd.org/ampere2/build.html?mastername=main-arm64-default&build=p1c7a816cd0ad_s1bd4f769caf - default 140releng-armv7 2910ff97e727 parallel_build 34543 14826 60 5539 1397 0 12721 1D:09:35:28 https://pkg-status.freebsd.org/ampere3/build.html?mastername=140releng-armv7-default&build=2910ff97e727 Philip
Re: April 2024 stabilization week
Hi FreeBSD/main users & developers, this stabilization week [likely final] status update: * Netflix testing didn't discover any stability issues with main-n269602-dd03eafacba9. * Netflix testing didn't discover any substantial performance degradations. The data is still being analyzed though. * A regression with ZFS reported in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=278494 has been addressed by ZFS 9f83eec03904b18e052fbe2c66542bd47254cf57. * An old (more than a month old) regression has been identified with accept_filter(9). Fixed by a8acc2bf5699556946dda2a37589d3c3bd9762c6. Since FreeBSD/main has been pushed with several non-documentation, non-a- trivial-bugfix commits during the days of the stabilization week, I can't guarantee that the above testing results are applicable to the current state of FreeBSD/main. That's why I created a temporary cherry-picking branch stabweek-2024-Apr that is published at https://github.com/glebius/FreeBSD.git. Users of FreeBSD/main are adviced with the following choices: - Pull up FreeBSD/main up to a8acc2bf5699556946dda2a37589d3c3bd9762c6 and use it as your stabilization point. There is tiny risk of untested changes added recently. - Pull stabweek-2024-Apr from https://github.com/glebius/FreeBSD.git. - Craft stabweek-2024-Apr yourself: # git checkout -b stabweek-2024-Apr dd03eafacba962c9dcec929c3ed9d63e7c43da3a # git cherry-pick -x --strategy=subtree -Xsubtree=sys/contrib/openzfs 9f83eec03904b18e052fbe2c66542bd47254cf57 # git cherry-pick -x a8acc2bf5699556946dda2a37589d3c3bd9762c6 I'm planning to end the advisory freeze on the main branch Wednesday morning at 8:00 UTC, unless somebody opposes that with a valid reason, e.g. a regression that I missed. -- Gleb Smirnoff signature.asc Description: PGP signature
Re: pkg server for current/arm64 stopped ? [main-armv7 on ampere2, elapsed so far: 651:21:56]
On Apr 19, 2024, at 07:16, Philip Paeps wrote: > On 2024-04-18 23:02:30 (+0800), Mark Millard wrote: > >> void wrote on >> Date: Thu, 18 Apr 2024 14:08:36 UTC : >> >>> Not sure where to post this.. >>> >>> The last bulk build for arm64 appears to have happened around >>> mid-March on ampere2. Is it broken? >> >> main-armv7 building is broken and the last completed build >> was the one started on Mon, 19 Feb 2024 12:32:10 GMT. It >> gets stuck making no progress until manually forced to stop, >> which leads to huge elapsed times for the incomplete builds: >> >> pd5512ae7b8c6_s75464941dc 34472 12282 (+9196) 107 (+77) 4753 (+2247) 1390 >> (+529) 15940 parallel_build: Fri, 22 Mar 2024 11:05:01 GMT 651:21:56 >> >> p43e3af5f5763_sf5f08e41aa 19809 5919 (+3126) 137 (+100) 5363 (+2741) 1395 >> (+522) 6995 parallel_build: Wed, 28 Feb 2024 15:46:14 GMT 359:42:14 ampere2 >> >> ampere2 alternates between trying to build main-arm64 and main-armv7, so >> main-armv7 being stuck blocks main-arm64 from building. >> >> One can see that all 13 job ID's show over 570 hours: >> >> http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=pd5512ae7b8c6_s75464941dc >> >> It is not random which packages are building when this happens. Compare: >> >> http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=p43e3af5f5763_sf5f08e41aa >> >> By contrast, the 19 Feb 2024 from-scratch (full) build worked: >> >> http://ampere2.nyi.freebsd.org/build.html?mastername=main-armv7-default&build=pe9c9c73181b5_sbd45bbe440 >> >> My guess is that FreeBSD has something that broken after bd45bbe440 >> that was broken as of f5f08e41aa and was still broken at 75464941dc . > > I'll kill the build on ampere2 again. Thanks for the nudge. > > We don't really have good monitoring for this. Also: builds should time out > after 36 hours. The fact that this one does not is a bug in itself. > > Philip [hat: clusteradm] I'll note that I've never managed to replicate the problem for building for armv7 on aarch64. But my context never has the likes of: QUOTE Host OSVERSION: 156 Jail OSVERSION: 1500015 . . !!! Jail is newer than host. (Jail: 1500015, Host: 156) !!! !!! This is not supported. !!! !!! Host kernel must be same or newer than jail. !!! !!! Expect build failures. !!! END QUOTE but always has the two OSVERSION's the same, such as: Host OSVERSION: 1500015 Jail OSVERSION: 1500015 or, recently, Host OSVERSION: 1500018 Jail OSVERSION: 1500018 My bulk runs do go through the sequence where the hangups have repeated for main-armv7 on ampere2. I wonder what would happen if "Host OSVERSION" was updated (modernized) to match the modern "Jail OSVERSION" that would be used? === Mark Millard marklmi at yahoo.com
Re: April 2024 stabilization week
Hi FreeBSD/main users & developers, On Mon, Apr 22, 2024 at 01:00:50AM -0700, Gleb Smirnoff wrote: T> This is an automated email to inform you that the April 2024 stabilization week T> started with FreeBSD/main at main-n269602-dd03eafacba9, which was tagged as T> main-stabweek-2024-Apr. T> T> The tag main-stabweek-2024-Apr has been published at T> https://github.com/glebius/FreeBSD/tags. Those who want to participate T> in the stabilization week are encouraged to update to the above T> revision/tag and test their systems. * Netflix testing didn't discover any stability issues with main-n269602-dd03eafacba9. We are still running the performance test, but preliminary results are that everything is fine. * My personal desktop/server experience with the tag neither has any problems. Feel free to reply with your reports if you participated in the testing, too. In bugzilla we have this submission, which looks important: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=278494 I want to hear from Alexander and Martin before thawing the advisory freeze. Don't want to declare the tag good if some ZFS systems fail to boot after upgrade. -- Gleb Smirnoff
