Bug#1012250: Fixing CI bugs for a package on the REJECT list
Hi Paul, On 01/10/2022 11:06, Paul Gevers wrote: I have triggered several runs (about 15 or so) and they all passed. I have removed the block and am lowering the severity of this bug for now. What bothers me about this is that these flaky tests do not occur with the buildd hosts: Well, the failure didn't happen that often, so maybe it just didn't happen on the buildd. Also, maybe it's something relatively new (and you only had X chances on the buildds). We'll keep monitoring. Thanks for this. I'll keep looking at the CI builds and will try to diagnose the timeouts when they happen. Regards Jeff OpenPGP_signature Description: OpenPGP digital signature
Bug#1012250: Fixing CI bugs for a package on the REJECT list
Control: severity -1 important Hi Jeff, On 27-09-2022 08:17, Jeff wrote: Please trigger a new run. I can't add more diagnostics until I know where. I have triggered several runs (about 15 or so) and they all passed. I have removed the block and am lowering the severity of this bug for now. What bothers me about this is that these flaky tests do not occur with the buildd hosts: Well, the failure didn't happen that often, so maybe it just didn't happen on the buildd. Also, maybe it's something relatively new (and you only had X chances on the buildds). We'll keep monitoring. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1012250: Fixing CI bugs for a package on the REJECT list
Hi Jeff, On 27-09-2022 08:17, Jeff wrote: I also offer to run the test (once or twice) manually and get information out of the testbed, if you tell me the exact commands you want me to run in the testbed. Please trigger a new run. I can't add more diagnostics until I know where. Done so a couple of times, we'll first need to hit one that times out of course. I'll continue to trigger until we find one for a while. What bothers me about this is that these flaky tests do not occur with the buildd hosts: https://buildd.debian.org/status/logs.php?pkg=gscan2pdf=all The only failure took 7 minutes, not 27 hours. So how do the CI hosts diff from the buildd ones? I don't know how buildd's look like, but an obvious delta is that autopkgtests on our host run inside lxc. Also, as your package is arch:all only, it always runs on amd64. I think one of the delta's we have for *several* hosts is that the amount of cores and RAM available for the test is significantly higher than on buildds. On a lot of our hosts, we also run multiple debci workers in parallel, so timing wise, you may be seeing varying performance even on the same host. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1012250: Fixing CI bugs for a package on the REJECT list
Hi Paul, On 26/09/2022 19:46, Paul Gevers wrote: The bug has a user specified for the usertag and explicitly mentions: """ Don't hesitate to reach out if you need help [...] """ So, either using debian...@lists.debian.org or the submitter's address (mine) seems appropriate. In which case, I evidently misunderstood. I had assumed that replying to the bug mail would do the same. In this case: we can trigger the test from the backside, such that you can get a fresh log, but I prefer to only do that coordinated and after you give it a try to enable more diagnostic logging, because apparently in the original logs there wasn't enough information for you. I also offer to run the test (once or twice) manually and get information out of the testbed, if you tell me the exact commands you want me to run in the testbed. Please trigger a new run. I can't add more diagnostics until I know where. What bothers me about this is that these flaky tests do not occur with the buildd hosts: https://buildd.debian.org/status/logs.php?pkg=gscan2pdf=all The only failure took 7 minutes, not 27 hours. So how do the CI hosts diff from the buildd ones? Thanks for your help. Regards Jeff OpenPGP_signature Description: OpenPGP digital signature
Bug#1012250: Fixing CI bugs for a package on the REJECT list
Hi Jeff, On 26-09-2022 12:53, Jeff wrote: Short of closing #1012250, how do I get CI pipeline to pick up gscan2pdf again to debug the flaky tests? I'd appreciate any pointers. The bug has a user specified for the usertag and explicitly mentions: """ Don't hesitate to reach out if you need help [...] """ So, either using debian...@lists.debian.org or the submitter's address (mine) seems appropriate. In this case: we can trigger the test from the backside, such that you can get a fresh log, but I prefer to only do that coordinated and after you give it a try to enable more diagnostic logging, because apparently in the original logs there wasn't enough information for you. I also offer to run the test (once or twice) manually and get information out of the testbed, if you tell me the exact commands you want me to run in the testbed. Paul PS: I propose we drop debian-devel from the replies and continue our discussion in the bug, but please keep me in CC as I'm not subscribed to the bug. Be reminded that the BTS doesn't send e-mail to the submitter unless asked explicitly or unless the bug is closed. OpenPGP_signature Description: OpenPGP digital signature
