Re: 8.6.1 Release
FYI I volunteered to do the docker 8.6.1 release already: https://github.com/docker-solr/docker-solr/issues/328#issuecomment-666494362 I recommend other committers interested in docker "Watch" that repo, just as you probably do for lucene-solr. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Sat, Aug 1, 2020 at 11:49 PM Houston Putman wrote: > Its a manual process, but there are scripts available in the docker-solr > repo to do it. > > The releases also have to be verified by the docker folks, but i think > that goes pretty quickly. > > Ill make sure to get that process kicked off whenever the release gets > cemented. > > - Houston > > On Sat, Aug 1, 2020 at 6:13 PM Marcus Eagan wrote: > >> >> >> I agree with Ishan on the need for the Docker image check. It’s an >> additional burden but most people are deploying Solr as a container these >> days. >> >> Marcus >> >> On Sat, Aug 1, 2020 at 13:18 Ishan Chattopadhyaya < >> ichattopadhy...@gmail.com> wrote: >> >>> For some reason, 8.6.0 docker images are not generated and pushed. After >>> the 8.6.1 release, can we ensure that the docker images are pushed? Any >>> idea how does that happen, generally? >>> Can we add this as a step in the release process please? >>> >>> On Fri, Jul 31, 2020 at 2:47 AM Houston Putman >>> wrote: >>> The first RC has been pushed and the vote has been started. Release note drafts can be found here: https://cwiki.apache.org/confluence/display/LUCENE/ReleaseNote861 https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote861 Any changes can be made there directly. - Houston On Thu, Jul 30, 2020 at 11:51 AM Houston Putman < houstonput...@gmail.com> wrote: > Ok, I'm finalizing the 8.6.1 release, so please do not push any > commits unless there is a large bug we have not yet identified. > > I'll be sending out the RC1 later today. > > - Houston > > On Thu, Jul 30, 2020 at 11:41 AM Atri Sharma wrote: > >> This seems like a patch release for targeting a specific blocking >> issue in 8.6. 8.7 would seem over arching? >> >> On Thu, 30 Jul 2020 at 20:33, Houston Putman >> wrote: >> >>> I started the process last night. Is there a reason why you want >>> this to make it into 8.6.1 and not 8.7? >>> >>> - Houston >>> >>> On Thu, Jul 30, 2020 at 8:13 AM Noble Paul >>> wrote: >>> Can I cherry-pick SOLR-14634: Limit the HTTP security headers to "/solr" end point ? if the build is not already made? On Thu, Jul 30, 2020 at 4:14 AM David Smiley wrote: > > I have a PR up: https://github.com/apache/lucene-solr/pull/1706 reviews welcome; I won't merge to a release branch without one. > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley > > > On Wed, Jul 29, 2020 at 10:43 AM Houston Putman < houstonput...@gmail.com> wrote: >> >> Thanks Jan and David. >> >> I think that's the last issue. So after you commit it I'll begin the release process. >> >> - Houston >> >> On Wed, Jul 29, 2020 at 10:29 AM David Smiley < dsmi...@apache.org> wrote: >>> >>> https://issues.apache.org/jira/browse/LUCENE-9443 >>> This regression on the highlighter is trivial; just revert a commit to a file and maybe suppress a warning. I'll get this into 8.6.1 today. >>> >>> ~ David Smiley >>> Apache Lucene/Solr Search Developer >>> http://www.linkedin.com/in/davidwsmiley >>> >>> >>> On Wed, Jul 29, 2020 at 6:04 AM Jan Høydahl < jan@cominvent.com> wrote: Merged! 28. jul. 2020 kl. 23:22 skrev Houston Putman < houstonput...@gmail.com>: +1 to the change. I think the last issue remaining is Jan's Zookeeper client port fix. After that is merged I will start with the release. - Houston On Tue, Jul 28, 2020 at 5:12 PM Gus Heck wrote: > > Ishan suggested I also supply the slight clarification to ref guide build docs to 8_6 https://github.com/apache/lucene-solr/pull/1704 if you want to include it. > > On Mon, Jul 27, 2020 at 9:25 PM Houston Putman < houstonput...@gmail.com> wrote: >> >> Both of those look good to include! >> >> Ill keep an eye out for when they get resolved. >> >> - Houston >> >>
Re: [VOTE] Release Lucene/Solr 8.6.1 RC1
Digging a little further, I notice that the deployment that had the error
has this autoscaling (whereas the working deployment does not).
"cluster-preferences":[{
"minimize":"cores",
"precision":1},
{"maximize":"freedisk"}],
"cluster-policy":[
{
"replica":"<2",
"shard":"#EACH",
"node":"#ANY",
"strict":"false"},
{
"replica":"#EQUAL",
"node":"#ANY",
"strict":"false"},
{
"cores":"#EQUAL",
"node":"#ANY",
"strict":"false"}],
So this may raise the question of whether or not we have an issue upgrading
an 8.6.0 version to 8.6.1... also, not very familiar with autoscaling's
error messages, but it kinda looks dodgy too since "one extra tag in cores"
appears to be referring to a cores attribute that has only one value, but
no idea yet if I'm reading that error message right. ... As to how I got
that, I'm pretty sure it was one of the times when my edits to cloud.sh
errored and tried to deploy an existing branch_8x build. Zk probably was
not clean, and retained the old config.
Tomorrow I'll try to deploy 8_6_0 and then upgrade it to 8_6_1 (late here
now) and see if I get a similar result.
On Sun, Aug 2, 2020 at 11:59 PM Gus Heck wrote:
> I Got:
>
> Ubuntu 18.04.4 LTS:
> SUCCESS! [0:53:02.203047]
> Mac OS 10.13:
>
> SUCCESS! [1:00:57.938586]
>
>
> BUT... when I deployed the tarball locally and tried to create a
> collection (single shard, _default config, via the solr UI), I got:
>
>
> 2020-08-03 02:55:15.585 INFO (zkCallback-14-thread-1) [ ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
>
> 2020-08-03 02:55:21.288 INFO (zkCallback-14-thread-1) [ ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
>
> 2020-08-03 02:55:26.705 INFO (zkCallback-14-thread-1) [ ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
>
> 2020-08-03 03:00:07.521 INFO
> (OverseerThreadFactory-22-thread-1-processing-n:192.168.2.106:8981_solr)
> [ ] o.a.s.c.a.c.CreateCollectionCmd Create collection test
>
> 2020-08-03 03:00:07.672 ERROR
> (OverseerThreadFactory-22-thread-1-processing-n:192.168.2.106:8981_solr)
> [ ] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: test
> operation: create failed:org.apache.solr.common.SolrException
>
> at
> org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:347)
>
> at
> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:264)
>
> at
> org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:517)
>
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:212)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.lang.RuntimeException: Only one extra tag supported for
> the tag cores in {
>
> "cores":"#EQUAL",
>
> "node":"#ANY",
>
> "strict":"false"}
>
> at
> org.apache.solr.client.solrj.cloud.autoscaling.Clause.(Clause.java:122)
>
> at
> org.apache.solr.client.solrj.cloud.autoscaling.Clause.create(Clause.java:235)
>
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
>
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
>
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
>
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
>
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
>
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>
> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>
> at
> org.apache.solr.client.solrj.cloud.autoscaling.Policy.(Policy.java:144)
>
> at
> org.apache.solr.client.solrj.cloud.autoscaling.AutoScalingConfig.getPolicy(AutoScalingConfig.java:372)
>
> at
> org.apache.solr.cloud.api.collections.Assign.usePolicyFramework(Assign.java:300)
>
> at
> org.apache.solr.cloud.api.collections.Assign.usePolicyFramework(Assign.java:277)
>
> at
> org.apache.solr.cloud.api.collections.Assign$AssignStrategyFactory.create(Assign.java:661)
>
> at
> org.apache.solr.cloud.api.collections.CreateCollectionCmd.buildReplicaPositions(CreateCollectionCmd.java:415)
>
> at
> org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:192)
>
> ... 6 more
>
>
> However, when I re-did everything a second time to double check creating a
> collection worked just fine and now I can't seem to reproduce this.
>
>
> If nobody else gets this I'll figure I just managed to mangle something
> while working on https://issues.apache.org/jira/browse/SOLR-14704
>
>
> But others should perhaps give it a spin to look for this, So I'll give it
> +0
>
>
>
> On Fri, Jul 31, 2020 at 8:14 AM Noble Paul wrote:
>
>> success SUCCESS! [1:03:21.78
Re: [VOTE] Release Lucene/Solr 8.6.1 RC1
I Got:
Ubuntu 18.04.4 LTS:
SUCCESS! [0:53:02.203047]
Mac OS 10.13:
SUCCESS! [1:00:57.938586]
BUT... when I deployed the tarball locally and tried to create a collection
(single shard, _default config, via the solr UI), I got:
2020-08-03 02:55:15.585 INFO (zkCallback-14-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2020-08-03 02:55:21.288 INFO (zkCallback-14-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
2020-08-03 02:55:26.705 INFO (zkCallback-14-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
2020-08-03 03:00:07.521 INFO
(OverseerThreadFactory-22-thread-1-processing-n:192.168.2.106:8981_solr) [
] o.a.s.c.a.c.CreateCollectionCmd Create collection test
2020-08-03 03:00:07.672 ERROR
(OverseerThreadFactory-22-thread-1-processing-n:192.168.2.106:8981_solr) [
] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: test operation:
create failed:org.apache.solr.common.SolrException
at
org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:347)
at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:264)
at
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:517)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:212)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Only one extra tag supported for the
tag cores in {
"cores":"#EQUAL",
"node":"#ANY",
"strict":"false"}
at
org.apache.solr.client.solrj.cloud.autoscaling.Clause.(Clause.java:122)
at
org.apache.solr.client.solrj.cloud.autoscaling.Clause.create(Clause.java:235)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at
org.apache.solr.client.solrj.cloud.autoscaling.Policy.(Policy.java:144)
at
org.apache.solr.client.solrj.cloud.autoscaling.AutoScalingConfig.getPolicy(AutoScalingConfig.java:372)
at
org.apache.solr.cloud.api.collections.Assign.usePolicyFramework(Assign.java:300)
at
org.apache.solr.cloud.api.collections.Assign.usePolicyFramework(Assign.java:277)
at
org.apache.solr.cloud.api.collections.Assign$AssignStrategyFactory.create(Assign.java:661)
at
org.apache.solr.cloud.api.collections.CreateCollectionCmd.buildReplicaPositions(CreateCollectionCmd.java:415)
at
org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:192)
... 6 more
However, when I re-did everything a second time to double check creating a
collection worked just fine and now I can't seem to reproduce this.
If nobody else gets this I'll figure I just managed to mangle something
while working on https://issues.apache.org/jira/browse/SOLR-14704
But others should perhaps give it a spin to look for this, So I'll give it
+0
On Fri, Jul 31, 2020 at 8:14 AM Noble Paul wrote:
> success SUCCESS! [1:03:21.786536]
> Ubuntu 20.04 LTS
>
>
> On Fri, Jul 31, 2020 at 7:34 AM Houston Putman
> wrote:
> >
> > Due to the weekend the vote will be open until 2020-08-03 22:00 UTC.
> That's 96 hours, and two business days.
> >
> > I can leave the vote open for longer if people want an additional
> business day, but will end it on Monday otherwise.
> >
> > - Houston
> >
> >
> >
> > On Thu, Jul 30, 2020 at 5:07 PM Houston Putman
> wrote:
> >>
> >> Please vote for release candidate 1 for Lucene/Solr 8.6.1
> >>
> >> The artifacts can be downloaded from:
> >>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.6.1-RC1-reva32a3ac4e43f629df71e5ae30a3330be94b095f2
> >>
> >> You can run the smoke tester directly with this command:
> >>
> >> python3 -u dev-tools/scripts/smokeTestRelease.py \
> >>
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.6.1-RC1-reva32a3ac4e43f629df71e5ae30a3330be94b095f2
> >>
> >> The vote will be open for at least 72 hours i.e. until 2020-08-02 22:00
> UTC.
> >>
> >> [ ] +1 approve
> >> [ ] +0 no opinion
> >> [ ] -1 disapprove (and reason why)
> >>
> >> Here is my +1
>
>
>
> --
> -
> Noble Paul
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-
Re: Welcome Namgyu Kim to the PMC
Welcome, Namgyu! Koji On 2020/08/03 8:18, Ishan Chattopadhyaya wrote: I am pleased to announce that Namgyu Kim has accepted the PMC's invitation to join. Congratulations and welcome, Namgyu! - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Welcome Munendra SN to the PMC
Welcome, Munendra! Koji On 2020/08/03 8:19, Ishan Chattopadhyaya wrote: I am pleased to announce that Munendra SN has accepted the PMC's invitation to join. Congratulations and welcome, Munendra! - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Welcome Gus Heck to the PMC
Welcome, Gus! Koji On 2020/08/03 8:20, Ishan Chattopadhyaya wrote: I am pleased to announce that Gus Heck has accepted the PMC's invitation to join. Congratulations and welcome, Gus! - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Welcome Gus Heck to the PMC
I am pleased to announce that Gus Heck has accepted the PMC's invitation to join. Congratulations and welcome, Gus!
Welcome Munendra SN to the PMC
I am pleased to announce that Munendra SN has accepted the PMC's invitation to join. Congratulations and welcome, Munendra!
Welcome Namgyu Kim to the PMC
I am pleased to announce that Namgyu Kim has accepted the PMC's invitation to join. Congratulations and welcome, Namgyu!
Re: SolrClient and making requests asynchronously
I am +1 to this approach. Some thoughts inline. How would query timeout be respected in this approach? > The default approach might be configured to throw > UnsupportedOperationException, or perhaps might simply use an Executor to > get it done in an obvious way (assuming we can get ahold of an Executor > somewhere?). > Would that mean that we use an Executor to execute a single thread? >> CompletableFuture, and which merely takes the SolrRequest parameter and nothing else. Alternatively the client could supply a CompletableFuture parameter that Solr will call complete() on etc. but that seems a bit less natural to the notion of a method that returns it's results, albeit with a wrapper. I would think that we allow users to specify their callback. One of the advantages of AsyncListener is that it a custom implementation can allow users to handle the behaviour of timeout and other events. We should retain that behaviour. -- Regards, Atri Apache Concerted
Re: SolrClient and making requests asynchronously
The new API seems good to me. it does be able to handle all cases : future,
listener and error handling.
On Sat, Aug 1, 2020 at 10:15 AM David Smiley
wrote:
> Noble: Those sound like some big and bold ideas that are deserving of a
> separate conversation? My proposal here is based on the APIs we have today.
>
> ~ David
>
>
> On Fri, Jul 31, 2020 at 10:06 PM Noble Paul wrote:
>
>> I would also wish to deprecate all the public APIs which uses concrete
>> classes and move over to interfaces.
>>
>> The objectives are.
>> * We should be able to use POJOs that help with strong typing
>> * We should be able to easily switch between JSON/javabin/XML or even
>> well known protocols like protobuf
>>
>> On Sat, Aug 1, 2020 at 12:02 PM Noble Paul wrote:
>> >
>> > Thanks David for bringing this up.
>> >
>> > I want us to stop using concrete classes. We should exclusively use
>> > interfaces in our public APIs.
>> >
>> > We should stop using NamedList and start using this interface
>> >
>> >
>> https://github.com/apache/lucene-solr/blob/b91b461632b19c9488926a9302126a2158df3298/solr/solrj/src/java/org/apache/solr/common/util/SimpleMap.java
>> >
>> > NamedList will implement this interface too
>> >
>> > The SolrRequest is an abstract class and it has a huge surface area.
>> > We must create a simple small interface for this as will
>> >
>> >
>> >
>> >
>> > On Sat, Aug 1, 2020 at 6:25 AM David Smiley wrote:
>> > >
>> > > Hello devs,
>> > >
>> > > I'd like to have an API design discussion here on the dev list
>> concerning adding asynchronous request semantics to SolrClient. I think
>> the SolrClient APIs require great peer-review consideration and so I'm
>> having this discussion here instead of a JIRA issue.
>> > >
>> > > Background: Sometimes, a SolrJ user (including Solr itself) may wish
>> to issue a request but not block/wait for the response because it can do
>> other useful work before eventually needing the results. For example,
>> maybe the user has multiple requests to send to Solr that can be done in
>> parallel and _then_ process the results. This is in fact what Solr needs
>> internally for during distributed-search (see HttpShardHandler) and during
>> distributed-indexing (see ConcurrentUpdateHttp2SolrClient).
>> > >
>> > > Why?: Greater efficiency. This can be accomplished easily in a
>> generic fashion with threads (e.g.
>> CompletableFuture.supplyAsync(Supplier,Executor)) with use of the
>> SolrClient concurrently. However, this can result in lots of threads and
>> context switching. I really like Cao Dat's explanation in the description
>> here: https://issues.apache.org/jira/browse/SOLR-14354 Thanks to
>> HTTP/2, there is a new approach in which the underlying HTTP client can
>> handle the asynchronous nature more efficiently with minimal threads.
>> Complete plumbing of this implies that SolrClient (and/or its derivatives)
>> need a similar async mechanism. Disclaimer: I haven't measured any of
>> this but am piggybacking off of Cao's's insights on SOLR-14354
>> > >
>> > > Where?: An async API _could_ go only on some SolrClient classes that
>> natively support it, avoiding changing SolrClient itself. Maybe this is
>> what should occur first before "graduating" the method to SolrClient where
>> we devise a default approach, although I would prefer just putting it on
>> SolrClient. The default approach might be configured to throw
>> UnsupportedOperationException, or perhaps might simply use an Executor to
>> get it done in an obvious way (assuming we can get ahold of an Executor
>> somewhere?). If you're writing a delegating SolrClient (which I've done in
>> the past), you might want to take-care to delegate this method.
>> > > Another aspect of "Where" is whether SolrRequest should additionally
>> have an API alternative to "process" which is currently synchronous and
>> calls out to SolrClient.request(this).
>> > >
>> > > What?: What should this API look like? This is my primary interest
>> right now and the meat of the discussion.
>> > >
>> > > SolrClient's primary signature that actually does the work
>> (synchronously) is this:
>> > >
>> > > public abstract NamedList
>> request(@SuppressWarnings({"rawtypes"})final SolrRequest request, String
>> collection)
>> > > throws SolrServerException, IOException;
>> > >
>> > > I don't like the "collection" there as it's redundant with either the
>> configured SolrClient default or a setting in SolrRequest, but I digress
>> from the important matter at hand.
>> > >
>> > > In SOLR-14354, recently committed by Cao Dat destined for Solr 8.7,
>> there is a new (undocumented) method on Http2SolrClient (*not* SolrClient):
>> > >
>> > > public Cancellable asyncRequest(@SuppressWarnings({"rawtypes"})
>> SolrRequest solrRequest, String collection,
>> AsyncListener> asyncListener) {
>> > >
>> > > So firstly, it's only on Http2SolrClient, which means if you're using
>> CloudHttp2SolrClient (which does not subclass it, counterintuitively to
>> us
