Hello, after a short period of time when SNAPSHOTs worked fine after the recent refactoring, there emerged problems once again.
Since a few days ago, SNAPSHOT deployments started failing with HTTP 503 error, reported as https://issues.apache.org/jira/browse/INFRA-25807 Regards Jan On Thu, 16 May 2024 at 08:38, Jan Šťastný <[email protected]> wrote: > Hello, > > Once again, our SNAPSHOT deploying CI pipelines were not successful during > the past few days. > > This time, we're hitting timeouts too often during artifact uploads. > Incidentally we were working on unifying deploy procedures (also for > SNAPSHOTS) across KIE podling pipelines ( > https://github.com/apache/incubator-kie-issues/issues/1123), which should > bring a failover mechanism for such timeouts. > > I went ahead and merged the change for Jenkinsfile.buildchain file, which > is the one actually used in nightlies for drools and kogito-* projects. > Currently running a skipTests run to validate the change, if successful, > I'll chase reviewers for the rest of my relevant PRs to have this fixed for > tonight. > > Regards > Jan > > On Tue, 7 May 2024 at 15:00, Jan Šťastný <[email protected]> wrote: > >> Jobs have been successfully generated now. >> >> Nightlies including snapshot deployment have passed (after some reruns). >> >> Generally speaking, I don't think the Jenkins instance is healthy, so >> please report problems on the mailing list if they persist. Or you can >> rerun if you know where things reside, every committer has job execution >> permissions. >> >> Often snapshots uploads fail due to timeout, we already have one tweak to >> this on the way. >> >> Regards >> Jan >> >> On Mon, 6 May 2024 at 19:46, Jan Šťastný <[email protected]> wrote: >> >>> I tried selective removal of old jobs for drools and kogito pipelines, >>> but seed runs were still not going through. So I dropped all existing jobs >>> and now seeds are running correctly, already generating jobs for branches >>> (nighly, PR, ...). >>> Downsides are: loss of execution history and retrigger of PR checks for >>> all open PRs (again reminder to do regular clean up of stale PRs). >>> Upside is that the nightly build should trigger overnight. >>> >>> Regards >>> Jan >>> >>> Dne po 6. 5. 2024 16:05 uživatel Alex Porcelli <[email protected]> >>> napsal: >>> >>>> Thank you, Jan - really appreciate your proactiveness! >>>> >>>> Please keep us posted! >>>> >>>> On Mon, May 6, 2024 at 9:58 AM Jan Šťastný <[email protected]> >>>> wrote: >>>> > >>>> > The workaround resolves the issue. >>>> > >>>> > But to apply the workaround, it turns out the easiest way is to drop >>>> the >>>> > existing jobs. It's not clear to me why all of a sudden existing jobs >>>> are >>>> > not replaced during the DSL generation. Also there's no error message >>>> > signalling the possible reason. I only realized that this is a >>>> > differentiator between my working tests and failing seed execution >>>> after >>>> > the merge. >>>> > >>>> > Now I took OptaPlanner pipelines as a guinea pig to test this >>>> assumption, >>>> > and after removing existing jobs, the dsl code generation worked >>>> correctly. >>>> > I have triggered nightly build after the DSL generation and the >>>> problem is >>>> > gone. >>>> > >>>> > As a result though, the job execution history is lost, including the >>>> "age" >>>> > of a possible test failure. For the sake of timeliness of unblocking >>>> > nightly builds though, I am gonna replicate this approach for kogito >>>> and >>>> > drools pipelines too. >>>> > >>>> > Regards >>>> > Jan >>>> > >>>> > >>>> > >>>> > On Mon, 6 May 2024 at 09:25, Jan Šťastný <[email protected]> >>>> wrote: >>>> > >>>> > > Hello, >>>> > > it seems that since May 1st the nightly builds fail to trigger due >>>> to git >>>> > > clone error. >>>> > > >>>> > > The root problem of this failure is not clear at the moment, but >>>> there is >>>> > > supposed to be a workaround available. I need to figure out how to >>>> apply >>>> > > all across our ASF Jenkins CI jobs. >>>> > > >>>> > > Regards >>>> > > Jan >>>> > > >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>>>
