Re: additional term meta data

2021-01-08 Thread Simon Willnauer
John, can you explain what the usecase for such a new API is? I don't see a user of the API in your code. Is there a query you can optimize with this or what is the reasoning behind this change? I personally think it's quite invasive to add this information and there must be a good reason to add th

Re: SOLR-15056 change circuit breaker metric

2021-01-08 Thread Walter Underwood
OK, I’ll do it against master. We’d love to see it in an 8.x release, though. This would protect against the major Solr outages we’ve had in the past few years. A question for someone who knows the Java metrics stuff better than I do: Is it OK to use com.sun.management.OperatingSystemMXBean inste

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

2021-01-08 Thread David Smiley
Jason, I know you've seen SOLR-15051 (Shared storage -- BlobDirectory), but perhaps not a part of the linked document: https://docs.google.com/document/d/1kjQPK80sLiZJyRjek_Edhokfc5q9S3ISvFRM2_YeL8M/edit# "ZK Data: Listings" that discusses an aspect of the design that is very similar to a part of

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

2021-01-08 Thread Mike Drob
Do the shard metadata files list all of the segments that make up the backup, or only the segments that were uploaded in this incremental update? On Fri, Jan 8, 2021 at 3:19 PM Jason Gerlowski wrote: > Sure thing. I put together a writeup on the file layout and formats > here: > https://cwiki.a

Re: Failing gradle precommits

2021-01-08 Thread Dawid Weiss
I don't remember, to be honest. I think javadocs, html linters - they all contribute. We could remove some of them (via -x) on github if they're too heavy. On Fri, Jan 8, 2021 at 9:08 PM Michael Sokolov wrote: > > Whoops! Well I guess we'll see. I wonder what other checks we are running now > th

Re: [DISCUSS] SIP-12: Incremental Backup and Restore

2021-01-08 Thread Jason Gerlowski
Sure thing. I put together a writeup on the file layout and formats here: https://cwiki.apache.org/confluence/display/SOLR/Incremental+Backup+File+Format The details get a little verbose, so I made it a subpage that the SIP-proper calls out to. Let me know what you think when you get a chance to

Re: Requesting a new GH repository for CrossDC modules

2021-01-08 Thread Anshum Gupta
The problem with a single repository is that it will/may conflict at times. Also, I still don't see the problem with having the extra repo as long as we aren't releasing anything. The problem with (b) is that you can't create a PR from a random repository to a repo it isn't a fork of. I also don't

Re: [JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-16-ea+30) - Build # 29209 - Still Unstable!

2021-01-08 Thread Michael Sokolov
I ran 3000+ iterations with that seed (on JDK11) and it did not reproduce. Then another 2000+ with JDK-16 runtime, and no fails there either. On Fri, Jan 8, 2021 at 10:32 AM David Smiley wrote: > > Hm; this is spooky > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com

Re: Failing gradle precommits

2021-01-08 Thread Michael Sokolov
Whoops! Well I guess we'll see. I wonder what other checks we are running now that's were not part of precommit, aside from spotless? On Fri, Jan 8, 2021, 2:28 PM Dawid Weiss wrote: > The switch to full check (minus tests) makes the "precommit" really > heavy... it's around 14 minutes on my not-

Re: Failing gradle precommits

2021-01-08 Thread Dawid Weiss
The switch to full check (minus tests) makes the "precommit" really heavy... it's around 14 minutes on my not-so-fancy machine. I'm curious if it's going to pass on github or timeout somehow. On Fri, Jan 8, 2021 at 8:26 PM Houston Putman wrote: > > Yeah, that should work Dawid. I'll create a PR t

Re: Failing gradle precommits

2021-01-08 Thread Houston Putman
Yeah, that should work Dawid. I'll create a PR that tests it out. - Houston On Fri, Jan 8, 2021 at 2:02 PM Dawid Weiss wrote: > Can those jobs just run a sequence of two commands - > > ./gradlew localSettings > ./gradlew check -x test > > This would solve the problem as proper JVM settings (tun

Re: Failing gradle precommits

2021-01-08 Thread Dawid Weiss
Can those jobs just run a sequence of two commands - ./gradlew localSettings ./gradlew check -x test This would solve the problem as proper JVM settings (tuned for the machine/ image it's running on) would be written and used on subsequent run. Dawid On Fri, Jan 8, 2021 at 7:44 PM Uwe Schindler

Re: Failing gradle precommits

2021-01-08 Thread Uwe Schindler
The problembcomes from the fact that Gradle sets a Xmx for itself in the settings file. Jenkins installs a settings file before. As every GitHub run is a clean checkout in new working dir, the settings never persist. IMHO, we should change the command line and pass JVM options to set heap size

Re: questions of FST

2021-01-08 Thread Michael McCandless
Hello! A few quick answers: In theory, an FST node can have inCount > 1 (when it is in the "suffix" part of the FST), but it sounds like maybe you found an exciting bug if indeed FST Builder fails to ever increment it. E.g. try building some FSTs here: http://examples.mikemccandless.com/fst.py t

Re: SOLR-15056 change circuit breaker metric

2021-01-08 Thread David Smiley
Glad to see you contributing Walter! Unless you know it only applies to 8x, you should branch against master. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Thu, Jan 7, 2021 at 4:26 PM Walter Underwood wrote: > Starting work on this change. Shoul

Re: Failing gradle precommits

2021-01-08 Thread David Smiley
Perhaps the OOMs are because .github/workflows/gradle-precommit.yml yesterday switched from doing "gradlew check -x test" to "gradlew precomit" ? CC Michael Sokolov ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Fri, Jan 8, 2021 at 1:06 PM Timothy

Re: Failing gradle precommits

2021-01-08 Thread Timothy Potter
Same for my PR too ... OOMs about 14 minutes in ... On Fri, Jan 8, 2021 at 9:45 AM Houston Putman wrote: > Weirdly enough, Github PR precommit actions have started to OOM. Not sure > if it's a github thing or something that changed on our end... > > On Fri, Jan 8, 2021 at 11:37 AM Joel Bernstein

Re: Failing gradle precommits

2021-01-08 Thread Houston Putman
Weirdly enough, Github PR precommit actions have started to OOM. Not sure if it's a github thing or something that changed on our end... On Fri, Jan 8, 2021 at 11:37 AM Joel Bernstein wrote: > It turned out to be this while I merged branches: > > warning: inexact rename detection was skipped due

Re: Failing gradle precommits

2021-01-08 Thread Joel Bernstein
It turned out to be this while I merged branches: warning: inexact rename detection was skipped due to too many files. warning: you may want to set your merge.renamelimit variable to at least 1639 and retry the command. Joel Bernstein http://joelsolr.blogspot.com/ On Fri, Jan 8, 2021 at 11:16

Re: Failing gradle precommits

2021-01-08 Thread Joel Bernstein
Thanks Eric, I'll do a fresh clone, something must be out of wack with my local repo. Joel Bernstein http://joelsolr.blogspot.com/ On Fri, Jan 8, 2021 at 10:55 AM Eric Pugh wrote: > It ran for me just fine. I *think* you may not be up to date, as > dataimporthandler/ is no longer in master!

Re: Failing gradle precommits

2021-01-08 Thread Eric Pugh
It ran for me just fine. I *think* you may not be up to date, as dataimporthandler/ is no longer in master! > On Jan 8, 2021, at 10:08 AM, Joel Bernstein wrote: > > I'm getting failing gradle precommits in master: > > > Task :solr:contrib:validateSourcePatterns FAILED > tabs instead spaces:

Re: [JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-16-ea+30) - Build # 29209 - Still Unstable!

2021-01-08 Thread David Smiley
Hm; this is spooky ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Fri, Jan 8, 2021 at 5:18 AM Policeman Jenkins Server wrote: > Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/29209/ > Java: 64bit/jdk-16-ea+30 -XX:+UseCompressedOops

Failing gradle precommits

2021-01-08 Thread Joel Bernstein
I'm getting failing gradle precommits in master: *> Task :solr:contrib:validateSourcePatterns* FAILED tabs instead spaces: /Users/joelbernstein/committer/lucene-solr/solr/contrib/dataimporthandler/build/test-results/test/TEST-org.apache.solr.handler.dataimport.TestDocBuilder.xml tabs instead spa

Re: Requesting a new GH repository for CrossDC modules

2021-01-08 Thread Ishan Chattopadhyaya
Not necessarily. Most people contribute to Apache Lucene/Solr using external repositories (forks) and raise pull requests against Apache owned repositories. There's no SGA needed on such occasions. I see two paths forward from here. a) Lets setup a single repository for all packages/plugins, say