> +1 to fix precommit slowness! It's doing a *lot* of checks, Mike... So the tradeoff for disabling a precommit is to either push it later to a CI run or to skip a check that was otherwise doing something.
> It is really depressing to run precommit on my 128 core box and watch how > slow it is :) What are your generated gradle settings? I have a slightly less beefy box (ryzen, 64 threads) and precommit takes ~2mins. D. > > I'll try to add (read: open an issue!) a chart to our nightly benchmarks > tracking precommit time over time ... > > Mike McCandless > > http://blog.mikemccandless.com > > > On Fri, Jan 8, 2021 at 4:43 PM Dawid Weiss <[email protected]> wrote: >> >> I don't remember, to be honest. I think javadocs, html linters - they >> all contribute. We could remove some of them (via -x) on github if >> they're too heavy. >> >> On Fri, Jan 8, 2021 at 9:08 PM Michael Sokolov <[email protected]> wrote: >> > >> > Whoops! Well I guess we'll see. I wonder what other checks we are running >> > now that's were not part of precommit, aside from spotless? >> > >> > On Fri, Jan 8, 2021, 2:28 PM Dawid Weiss <[email protected]> wrote: >> >> >> >> The switch to full check (minus tests) makes the "precommit" really >> >> heavy... it's around 14 minutes on my not-so-fancy machine. I'm >> >> curious if it's going to pass on github or timeout somehow. >> >> >> >> On Fri, Jan 8, 2021 at 8:26 PM Houston Putman <[email protected]> >> >> wrote: >> >> > >> >> > Yeah, that should work Dawid. I'll create a PR that tests it out. >> >> > >> >> > - Houston >> >> > >> >> > On Fri, Jan 8, 2021 at 2:02 PM Dawid Weiss <[email protected]> >> >> > wrote: >> >> >> >> >> >> Can those jobs just run a sequence of two commands - >> >> >> >> >> >> ./gradlew localSettings >> >> >> ./gradlew check -x test >> >> >> >> >> >> This would solve the problem as proper JVM settings (tuned for the >> >> >> machine/ image it's running on) would be written and used on >> >> >> subsequent run. >> >> >> >> >> >> Dawid >> >> >> >> >> >> On Fri, Jan 8, 2021 at 7:44 PM Uwe Schindler <[email protected]> wrote: >> >> >> > >> >> >> > The problembcomes from the fact that Gradle sets a Xmx for itself in >> >> >> > the settings file. Jenkins installs a settings file before. >> >> >> > >> >> >> > As every GitHub run is a clean checkout in new working dir, the >> >> >> > settings never persist. >> >> >> > >> >> >> > IMHO, we should change the command line and pass JVM options to set >> >> >> > heap size as it is written to the settings file. >> >> >> > >> >> >> > Uwe >> >> >> > >> >> >> > Am January 8, 2021 6:13:01 PM UTC schrieb David Smiley >> >> >> > <[email protected]>: >> >> >> >> >> >> >> >> Perhaps the OOMs are because .github/workflows/gradle-precommit.yml >> >> >> >> yesterday switched from doing "gradlew check -x test" to "gradlew >> >> >> >> precomit" ? CC Michael Sokolov >> >> >> >> >> >> >> >> ~ David Smiley >> >> >> >> Apache Lucene/Solr Search Developer >> >> >> >> http://www.linkedin.com/in/davidwsmiley >> >> >> >> >> >> >> >> >> >> >> >> On Fri, Jan 8, 2021 at 1:06 PM Timothy Potter >> >> >> >> <[email protected]> wrote: >> >> >> >>> >> >> >> >>> Same for my PR too ... OOMs about 14 minutes in ... >> >> >> >>> >> >> >> >>> On Fri, Jan 8, 2021 at 9:45 AM Houston Putman >> >> >> >>> <[email protected]> wrote: >> >> >> >>>> >> >> >> >>>> Weirdly enough, Github PR precommit actions have started to OOM. >> >> >> >>>> Not sure if it's a github thing or something that changed on our >> >> >> >>>> end... >> >> >> >>>> >> >> >> >>>> On Fri, Jan 8, 2021 at 11:37 AM Joel Bernstein >> >> >> >>>> <[email protected]> wrote: >> >> >> >>>>> >> >> >> >>>>> It turned out to be this while I merged branches: >> >> >> >>>>> >> >> >> >>>>> warning: inexact rename detection was skipped due to too many >> >> >> >>>>> files. >> >> >> >>>>> >> >> >> >>>>> warning: you may want to set your merge.renamelimit variable to >> >> >> >>>>> at least 1639 and retry the command. >> >> >> >>>>> >> >> >> >>>>> >> >> >> >>>>> Joel Bernstein >> >> >> >>>>> http://joelsolr.blogspot.com/ >> >> >> >>>>> >> >> >> >>>>> >> >> >> >>>>> On Fri, Jan 8, 2021 at 11:16 AM Joel Bernstein >> >> >> >>>>> <[email protected]> wrote: >> >> >> >>>>>> >> >> >> >>>>>> Thanks Eric, I'll do a fresh clone, something must be out of >> >> >> >>>>>> wack with my local repo. >> >> >> >>>>>> >> >> >> >>>>>> >> >> >> >>>>>> Joel Bernstein >> >> >> >>>>>> http://joelsolr.blogspot.com/ >> >> >> >>>>>> >> >> >> >>>>>> >> >> >> >>>>>> On Fri, Jan 8, 2021 at 10:55 AM Eric Pugh >> >> >> >>>>>> <[email protected]> wrote: >> >> >> >>>>>>> >> >> >> >>>>>>> It ran for me just fine. I *think* you may not be up to >> >> >> >>>>>>> date, as dataimporthandler/ is no longer in master! >> >> >> >>>>>>> >> >> >> >>>>>>> >> >> >> >>>>>>> On Jan 8, 2021, at 10:08 AM, Joel Bernstein >> >> >> >>>>>>> <[email protected]> wrote: >> >> >> >>>>>>> >> >> >> >>>>>>> I'm getting failing gradle precommits in master: >> >> >> >>>>>>> >> >> >> >>>>>>> > Task :solr:contrib:validateSourcePatterns FAILED >> >> >> >>>>>>> tabs instead spaces: >> >> >> >>>>>>> /Users/joelbernstein/committer/lucene-solr/solr/contrib/dataimporthandler/build/test-results/test/TEST-org.apache.solr.handler.dataimport.TestDocBuilder.xml >> >> >> >>>>>>> tabs instead spaces: >> >> >> >>>>>>> /Users/joelbernstein/committer/lucene-solr/solr/contrib/dataimporthandler/build/test-results/test/TEST-org.apache.solr.handler.dataimport.TestSolrEntityProcessorEndToEnd.xml >> >> >> >>>>>>> tabs instead spaces: >> >> >> >>>>>>> /Users/joelbernstein/committer/lucene-solr/solr/contrib/dataimporthandler/build/test-results/test/TEST-org.apache.solr.handler.dataimport.TestErrorHandling.xml >> >> >> >>>>>>> tabs instead spaces: >> >> >> >>>>>>> /Users/joelbernstein/committer/lucene-solr/solr/contrib/dataimporthandler/build/test-results/test/TEST-org.apache.solr.handler.dataimport.TestScriptTransformer.xml >> >> >> >>>>>>> tabs instead spaces: >> >> >> >>>>>>> /Users/joelbernstein/committer/lucene-solr/solr/contrib/dataimporthandler/build/test-results/test/TEST-org.apache.solr.handler.dataimport.TestSqlEntityProcessor.xml >> >> >> >>>>>>> tabs instead spaces: >> >> >> >>>>>>> /Users/joelbernstein/committer/lucene-solr/solr/contrib/dataimporthandler/build/test-results/test/TEST-org.apache.solr.handler.dataimport.TestDocBuilder2.xml >> >> >> >>>>>>> tabs instead spaces: >> >> >> >>>>>>> /Users/joelbernstein/committer/lucene-solr/solr/contrib/dataimporthandler/build/test-results/test/TEST-org.apache.solr.handler.dataimport.TestZKPropertiesWriter.xml >> >> >> >>>>>>> tabs instead spaces: >> >> >> >>>>>>> /Users/joelbernstein/committer/lucene-solr/solr/contrib/dataimporthandler-extras/build/test-results/test/TEST-org.apache.solr.handler.dataimport.TestTikaEntityProcessor.xml >> >> >> >>>>>>> >> >> >> >>>>>>> FAILURE: Build failed with an exception. >> >> >> >>>>>>> >> >> >> >>>>>>> * Where: >> >> >> >>>>>>> Script >> >> >> >>>>>>> '/Users/joelbernstein/committer/lucene-solr/gradle/validation/validate-source-patterns.gradle' >> >> >> >>>>>>> line: 324 >> >> >> >>>>>>> >> >> >> >>>>>>> * What went wrong: >> >> >> >>>>>>> Execution failed for task >> >> >> >>>>>>> ':solr:contrib:validateSourcePatterns'. >> >> >> >>>>>>> > Found 8 violations in source files (tabs instead spaces). >> >> >> >>>>>>> >> >> >> >>>>>>> >> >> >> >>>>>>> Are others seeing this as well? I'm not seeing Jenkins emails >> >> >> >>>>>>> about this. >> >> >> >>>>>>> >> >> >> >>>>>>> >> >> >> >>>>>>> Joel Bernstein >> >> >> >>>>>>> http://joelsolr.blogspot.com/ >> >> >> >>>>>>> >> >> >> >>>>>>> >> >> >> >>>>>>> _______________________ >> >> >> >>>>>>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | >> >> >> >>>>>>> 434.466.1467 | http://www.opensourceconnections.com | My >> >> >> >>>>>>> Free/Busy >> >> >> >>>>>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed >> >> >> >>>>>>> This e-mail and all contents, including attachments, is >> >> >> >>>>>>> considered to be Company Confidential unless explicitly stated >> >> >> >>>>>>> otherwise, regardless of whether attachments are marked as >> >> >> >>>>>>> such. >> >> >> >>>>>>> >> >> >> > >> >> >> > -- >> >> >> > Uwe Schindler >> >> >> > Achterdiek 19, 28357 Bremen >> >> >> > https://www.thetaphi.de >> >> >> >> >> >> --------------------------------------------------------------------- >> >> >> To unsubscribe, e-mail: [email protected] >> >> >> For additional commands, e-mail: [email protected] >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: [email protected] >> >> For additional commands, e-mail: [email protected] >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
