Re: Updating configset
Thanks for the help everyone. I'm hopeful for the PR, Tomás. That looks to be exactly what I was looking for! Best, Michael Carroll On 9/11/20, 8:44 PM, "Tomás Fernández Löbbe" wrote: *** External email: use caution *** I created https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Flucene-solr%2Fpull%2F1861data=02%7C01%7Cm.carroll%40elsevier.com%7C10d16615d6484b13f82008d856b4f94e%7C9274ee3f94254109a27f9fb15c10675d%7C0%7C0%7C637354682569763217sdata=%2BEfYbYVY94NTlRNJBxjhONpz4fcPkXOh1YgsGK4RF0A%3Dreserved=0 On Fri, Sep 11, 2020 at 11:43 AM Walter Underwood wrote: > I wrote some Python to get the Zookeeper address from CLUSTERSTATUS, then > use the Kazoo library to upload a configset. Then it goes back to the > cluster and > runs an async command to RELOAD. > > I really should open source that thing (in my copious free time). > > wunder > Walter Underwood > wun...@wunderwood.org > https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fobserver.wunderwood.org%2Fdata=02%7C01%7Cm.carroll%40elsevier.com%7C10d16615d6484b13f82008d856b4f94e%7C9274ee3f94254109a27f9fb15c10675d%7C0%7C0%7C637354682569763217sdata=qrnj0iratKmlu8hOosZ%2FC0Ji2P8k7CGE0AAHY9txTJ0%3Dreserved=0 (my blog) > > > On Sep 11, 2020, at 9:35 AM, Tomás Fernández Löbbe < > tomasflo...@gmail.com> wrote: > > > > I was in the same situation recently. I think it would be nice to have > the > > configset UPLOAD command be able to override the existing configset > instead > > of just fail (with a parameter such as override=true or something). We > need > > to be careful with the trusted/unstrusted flag there, but that should be > > possible. > > > >> If we can’t modify the configset wholesale this way, is it possible to > > create a new configset and swap the old collection to it? > > You can create a new one and then call MODIFYCOLLECTION on the collection > > that uses it: > > > https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucene.apache.org%2Fsolr%2Fguide%2F8_6%2Fcollection-management.html%23modifycollection-parametersdata=02%7C01%7Cm.carroll%40elsevier.com%7C10d16615d6484b13f82008d856b4f94e%7C9274ee3f94254109a27f9fb15c10675d%7C0%7C0%7C637354682569763217sdata=RyW%2BE9wbnWVKDjmABk1N82fVIqu10R6mxggsZ%2F7Lxdw%3Dreserved=0 > . > > I've never used that though. > > > > On Fri, Sep 11, 2020 at 7:26 AM Carroll, Michael (ELS-PHI) < > > m.carr...@elsevier.com> wrote: > > > >> Hello, > >> > >> I am running SolrCloud in Kubernetes with Solr version 8.5.2. > >> > >> Is it possible to update a configset being used by a collection using a > >> SolrCloud API directly? I know that this is possible using the zkcli > and a > >> collection RELOAD. We essentially want to be able to checkout our > configset > >> from source control, and then replace everything in the active > configset in > >> SolrCloud (other than the schema.xml). > >> > >> We have a couple of custom plugins that use config files that reside in > >> the configset, and we don’t want to have to rebuild the collection or > >> access zookeeper directly if we don’t have to. If we can’t modify the > >> configset wholesale this way, is it possible to create a new configset > and > >> swap the old collection to it? > >> > >> Best, > >> Michael Carroll > >> > >
Re: Updating configset
I created https://github.com/apache/lucene-solr/pull/1861 On Fri, Sep 11, 2020 at 11:43 AM Walter Underwood wrote: > I wrote some Python to get the Zookeeper address from CLUSTERSTATUS, then > use the Kazoo library to upload a configset. Then it goes back to the > cluster and > runs an async command to RELOAD. > > I really should open source that thing (in my copious free time). > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > On Sep 11, 2020, at 9:35 AM, Tomás Fernández Löbbe < > tomasflo...@gmail.com> wrote: > > > > I was in the same situation recently. I think it would be nice to have > the > > configset UPLOAD command be able to override the existing configset > instead > > of just fail (with a parameter such as override=true or something). We > need > > to be careful with the trusted/unstrusted flag there, but that should be > > possible. > > > >> If we can’t modify the configset wholesale this way, is it possible to > > create a new configset and swap the old collection to it? > > You can create a new one and then call MODIFYCOLLECTION on the collection > > that uses it: > > > https://lucene.apache.org/solr/guide/8_6/collection-management.html#modifycollection-parameters > . > > I've never used that though. > > > > On Fri, Sep 11, 2020 at 7:26 AM Carroll, Michael (ELS-PHI) < > > m.carr...@elsevier.com> wrote: > > > >> Hello, > >> > >> I am running SolrCloud in Kubernetes with Solr version 8.5.2. > >> > >> Is it possible to update a configset being used by a collection using a > >> SolrCloud API directly? I know that this is possible using the zkcli > and a > >> collection RELOAD. We essentially want to be able to checkout our > configset > >> from source control, and then replace everything in the active > configset in > >> SolrCloud (other than the schema.xml). > >> > >> We have a couple of custom plugins that use config files that reside in > >> the configset, and we don’t want to have to rebuild the collection or > >> access zookeeper directly if we don’t have to. If we can’t modify the > >> configset wholesale this way, is it possible to create a new configset > and > >> swap the old collection to it? > >> > >> Best, > >> Michael Carroll > >> > >
Re: Updating configset
I wrote some Python to get the Zookeeper address from CLUSTERSTATUS, then use the Kazoo library to upload a configset. Then it goes back to the cluster and runs an async command to RELOAD. I really should open source that thing (in my copious free time). wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Sep 11, 2020, at 9:35 AM, Tomás Fernández Löbbe > wrote: > > I was in the same situation recently. I think it would be nice to have the > configset UPLOAD command be able to override the existing configset instead > of just fail (with a parameter such as override=true or something). We need > to be careful with the trusted/unstrusted flag there, but that should be > possible. > >> If we can’t modify the configset wholesale this way, is it possible to > create a new configset and swap the old collection to it? > You can create a new one and then call MODIFYCOLLECTION on the collection > that uses it: > https://lucene.apache.org/solr/guide/8_6/collection-management.html#modifycollection-parameters. > I've never used that though. > > On Fri, Sep 11, 2020 at 7:26 AM Carroll, Michael (ELS-PHI) < > m.carr...@elsevier.com> wrote: > >> Hello, >> >> I am running SolrCloud in Kubernetes with Solr version 8.5.2. >> >> Is it possible to update a configset being used by a collection using a >> SolrCloud API directly? I know that this is possible using the zkcli and a >> collection RELOAD. We essentially want to be able to checkout our configset >> from source control, and then replace everything in the active configset in >> SolrCloud (other than the schema.xml). >> >> We have a couple of custom plugins that use config files that reside in >> the configset, and we don’t want to have to rebuild the collection or >> access zookeeper directly if we don’t have to. If we can’t modify the >> configset wholesale this way, is it possible to create a new configset and >> swap the old collection to it? >> >> Best, >> Michael Carroll >>
Re: Non-exists Field Query doesn't work when use defType=edismax
First, Solr does _not_ implement boolean logic. It can look like it, but that’s sugar. See: https://lucidworks.com/post/why-not-and-or-and-not/ As for (1), I haven’t a clue. If you’d provided the results of adding =query to the query, the parsed version would have given us some pointers. (2) Yes, See the reference above. the NOT (-) operator is really “from the results so far, remove everything that satisfies this clause”. But just -field_to_exclude:[* TO *] has nothing to remove things that satisfy this clause _from_. And a couple of nits: There’s no need for the AND, again, see the reference above. ((*:* -field_to_exclude:*)) would work just fine. Second, and just to confuse you, the *:* is not required in fq clauses, there’s some special logic to handle fq=-field_to_exclude:* that implies the *:* that you’d have to put in a “q” clause. Third, field_to_exclude:* is actually fairly inefficient, fq=-field_to_exclude:[* TO *] can be more efficient. The reasons are rather arcane... Best, Erick > On Sep 11, 2020, at 10:18 AM, Iana Bondarska wrote: > > Hi everyone, > I'm checking Solr query that contains condition "Exclude all documents that > contain ceratin field". > Currently, it looks like this: > *(*:* AND -field_to_exclude:*)* > > Full query is: > *((some_other_field:"value") AND ((*:* AND -field_to_exclude:*)))* > > If I use defType=edismax, nothing is found. If I use default defType - I > get correct results, same as when I do not specify *:* in non-exists > condition. > > My questions are: > 1) why defType "edismax" leads to empty results? > 2) do we really need to include clause "**:**" in non-exists condition? > Isn't *-field_to_exclude:* *enough? > -- > Best Regards, > Iana Bondarska
Re: Updating configset
Right, the problem is that both, bin/solr zk and ZkConfigManager require "direct access" to ZooKeeper (you have to have ZooKeeper exposed). I believe the original question was about how to achieve this without exposing ZooKeeper. On Fri, Sep 11, 2020 at 11:00 AM Andy C wrote: > Don't know if this is an option for you but the SolrJ Java Client library > has support for uploading a config set. If the config set already exists it > will overwrite it, and automatically RELOAD the dependent collection. > > See > > https://lucene.apache.org/solr/8_5_0/solr-solrj/org/apache/solr/common/cloud/ZkConfigManager.html > > On Fri, Sep 11, 2020 at 1:45 PM Jörn Franke wrote: > > > I would go for the Solr rest api ... especially if you have a secured zk > > (eg with Kerberos). Then you need to manage access for humans only in > Solr > > and not also in ZK. > > > > > Am 11.09.2020 um 19:41 schrieb Erick Erickson >: > > > > > > Bin/solr zk upconfig... > > > Bin/solr zk cp... For individual files. > > > > > > Not as convenient as a nice API, but might let you get by... > > > > > >> On Fri, Sep 11, 2020, 13:26 Houston Putman > > wrote: > > >> > > >> I completely agree, there should be a way to overwrite an existing > > >> configSet. > > >> > > >> Looks like https://issues.apache.org/jira/browse/SOLR-10391 already > > >> exists, > > >> so the work could be tracked there. > > >> > > >> On Fri, Sep 11, 2020 at 12:36 PM Tomás Fernández Löbbe < > > >> tomasflo...@gmail.com> wrote: > > >> > > >>> I was in the same situation recently. I think it would be nice to > have > > >> the > > >>> configset UPLOAD command be able to override the existing configset > > >> instead > > >>> of just fail (with a parameter such as override=true or something). > We > > >> need > > >>> to be careful with the trusted/unstrusted flag there, but that should > > be > > >>> possible. > > >>> > > If we can’t modify the configset wholesale this way, is it possible > to > > >>> create a new configset and swap the old collection to it? > > >>> You can create a new one and then call MODIFYCOLLECTION on the > > collection > > >>> that uses it: > > >>> > > >>> > > >> > > > https://lucene.apache.org/solr/guide/8_6/collection-management.html#modifycollection-parameters > > >>> . > > >>> I've never used that though. > > >>> > > >>> On Fri, Sep 11, 2020 at 7:26 AM Carroll, Michael (ELS-PHI) < > > >>> m.carr...@elsevier.com> wrote: > > >>> > > Hello, > > > > I am running SolrCloud in Kubernetes with Solr version 8.5.2. > > > > Is it possible to update a configset being used by a collection > using > > a > > SolrCloud API directly? I know that this is possible using the zkcli > > >> and > > >>> a > > collection RELOAD. We essentially want to be able to checkout our > > >>> configset > > from source control, and then replace everything in the active > > >> configset > > >>> in > > SolrCloud (other than the schema.xml). > > > > We have a couple of custom plugins that use config files that reside > > in > > the configset, and we don’t want to have to rebuild the collection > or > > access zookeeper directly if we don’t have to. If we can’t modify > the > > configset wholesale this way, is it possible to create a new > configset > > >>> and > > swap the old collection to it? > > > > Best, > > Michael Carroll > > > > >>> > > >> > > >
Re: Updating configset
Don't know if this is an option for you but the SolrJ Java Client library has support for uploading a config set. If the config set already exists it will overwrite it, and automatically RELOAD the dependent collection. See https://lucene.apache.org/solr/8_5_0/solr-solrj/org/apache/solr/common/cloud/ZkConfigManager.html On Fri, Sep 11, 2020 at 1:45 PM Jörn Franke wrote: > I would go for the Solr rest api ... especially if you have a secured zk > (eg with Kerberos). Then you need to manage access for humans only in Solr > and not also in ZK. > > > Am 11.09.2020 um 19:41 schrieb Erick Erickson : > > > > Bin/solr zk upconfig... > > Bin/solr zk cp... For individual files. > > > > Not as convenient as a nice API, but might let you get by... > > > >> On Fri, Sep 11, 2020, 13:26 Houston Putman > wrote: > >> > >> I completely agree, there should be a way to overwrite an existing > >> configSet. > >> > >> Looks like https://issues.apache.org/jira/browse/SOLR-10391 already > >> exists, > >> so the work could be tracked there. > >> > >> On Fri, Sep 11, 2020 at 12:36 PM Tomás Fernández Löbbe < > >> tomasflo...@gmail.com> wrote: > >> > >>> I was in the same situation recently. I think it would be nice to have > >> the > >>> configset UPLOAD command be able to override the existing configset > >> instead > >>> of just fail (with a parameter such as override=true or something). We > >> need > >>> to be careful with the trusted/unstrusted flag there, but that should > be > >>> possible. > >>> > If we can’t modify the configset wholesale this way, is it possible to > >>> create a new configset and swap the old collection to it? > >>> You can create a new one and then call MODIFYCOLLECTION on the > collection > >>> that uses it: > >>> > >>> > >> > https://lucene.apache.org/solr/guide/8_6/collection-management.html#modifycollection-parameters > >>> . > >>> I've never used that though. > >>> > >>> On Fri, Sep 11, 2020 at 7:26 AM Carroll, Michael (ELS-PHI) < > >>> m.carr...@elsevier.com> wrote: > >>> > Hello, > > I am running SolrCloud in Kubernetes with Solr version 8.5.2. > > Is it possible to update a configset being used by a collection using > a > SolrCloud API directly? I know that this is possible using the zkcli > >> and > >>> a > collection RELOAD. We essentially want to be able to checkout our > >>> configset > from source control, and then replace everything in the active > >> configset > >>> in > SolrCloud (other than the schema.xml). > > We have a couple of custom plugins that use config files that reside > in > the configset, and we don’t want to have to rebuild the collection or > access zookeeper directly if we don’t have to. If we can’t modify the > configset wholesale this way, is it possible to create a new configset > >>> and > swap the old collection to it? > > Best, > Michael Carroll > > >>> > >> >
Re: Updating configset
I would go for the Solr rest api ... especially if you have a secured zk (eg with Kerberos). Then you need to manage access for humans only in Solr and not also in ZK. > Am 11.09.2020 um 19:41 schrieb Erick Erickson : > > Bin/solr zk upconfig... > Bin/solr zk cp... For individual files. > > Not as convenient as a nice API, but might let you get by... > >> On Fri, Sep 11, 2020, 13:26 Houston Putman wrote: >> >> I completely agree, there should be a way to overwrite an existing >> configSet. >> >> Looks like https://issues.apache.org/jira/browse/SOLR-10391 already >> exists, >> so the work could be tracked there. >> >> On Fri, Sep 11, 2020 at 12:36 PM Tomás Fernández Löbbe < >> tomasflo...@gmail.com> wrote: >> >>> I was in the same situation recently. I think it would be nice to have >> the >>> configset UPLOAD command be able to override the existing configset >> instead >>> of just fail (with a parameter such as override=true or something). We >> need >>> to be careful with the trusted/unstrusted flag there, but that should be >>> possible. >>> If we can’t modify the configset wholesale this way, is it possible to >>> create a new configset and swap the old collection to it? >>> You can create a new one and then call MODIFYCOLLECTION on the collection >>> that uses it: >>> >>> >> https://lucene.apache.org/solr/guide/8_6/collection-management.html#modifycollection-parameters >>> . >>> I've never used that though. >>> >>> On Fri, Sep 11, 2020 at 7:26 AM Carroll, Michael (ELS-PHI) < >>> m.carr...@elsevier.com> wrote: >>> Hello, I am running SolrCloud in Kubernetes with Solr version 8.5.2. Is it possible to update a configset being used by a collection using a SolrCloud API directly? I know that this is possible using the zkcli >> and >>> a collection RELOAD. We essentially want to be able to checkout our >>> configset from source control, and then replace everything in the active >> configset >>> in SolrCloud (other than the schema.xml). We have a couple of custom plugins that use config files that reside in the configset, and we don’t want to have to rebuild the collection or access zookeeper directly if we don’t have to. If we can’t modify the configset wholesale this way, is it possible to create a new configset >>> and swap the old collection to it? Best, Michael Carroll >>> >>
Re: Updating configset
Bin/solr zk upconfig... Bin/solr zk cp... For individual files. Not as convenient as a nice API, but might let you get by... On Fri, Sep 11, 2020, 13:26 Houston Putman wrote: > I completely agree, there should be a way to overwrite an existing > configSet. > > Looks like https://issues.apache.org/jira/browse/SOLR-10391 already > exists, > so the work could be tracked there. > > On Fri, Sep 11, 2020 at 12:36 PM Tomás Fernández Löbbe < > tomasflo...@gmail.com> wrote: > > > I was in the same situation recently. I think it would be nice to have > the > > configset UPLOAD command be able to override the existing configset > instead > > of just fail (with a parameter such as override=true or something). We > need > > to be careful with the trusted/unstrusted flag there, but that should be > > possible. > > > > > If we can’t modify the configset wholesale this way, is it possible to > > create a new configset and swap the old collection to it? > > You can create a new one and then call MODIFYCOLLECTION on the collection > > that uses it: > > > > > https://lucene.apache.org/solr/guide/8_6/collection-management.html#modifycollection-parameters > > . > > I've never used that though. > > > > On Fri, Sep 11, 2020 at 7:26 AM Carroll, Michael (ELS-PHI) < > > m.carr...@elsevier.com> wrote: > > > > > Hello, > > > > > > I am running SolrCloud in Kubernetes with Solr version 8.5.2. > > > > > > Is it possible to update a configset being used by a collection using a > > > SolrCloud API directly? I know that this is possible using the zkcli > and > > a > > > collection RELOAD. We essentially want to be able to checkout our > > configset > > > from source control, and then replace everything in the active > configset > > in > > > SolrCloud (other than the schema.xml). > > > > > > We have a couple of custom plugins that use config files that reside in > > > the configset, and we don’t want to have to rebuild the collection or > > > access zookeeper directly if we don’t have to. If we can’t modify the > > > configset wholesale this way, is it possible to create a new configset > > and > > > swap the old collection to it? > > > > > > Best, > > > Michael Carroll > > > > > >
Re: Updating configset
I completely agree, there should be a way to overwrite an existing configSet. Looks like https://issues.apache.org/jira/browse/SOLR-10391 already exists, so the work could be tracked there. On Fri, Sep 11, 2020 at 12:36 PM Tomás Fernández Löbbe < tomasflo...@gmail.com> wrote: > I was in the same situation recently. I think it would be nice to have the > configset UPLOAD command be able to override the existing configset instead > of just fail (with a parameter such as override=true or something). We need > to be careful with the trusted/unstrusted flag there, but that should be > possible. > > > If we can’t modify the configset wholesale this way, is it possible to > create a new configset and swap the old collection to it? > You can create a new one and then call MODIFYCOLLECTION on the collection > that uses it: > > https://lucene.apache.org/solr/guide/8_6/collection-management.html#modifycollection-parameters > . > I've never used that though. > > On Fri, Sep 11, 2020 at 7:26 AM Carroll, Michael (ELS-PHI) < > m.carr...@elsevier.com> wrote: > > > Hello, > > > > I am running SolrCloud in Kubernetes with Solr version 8.5.2. > > > > Is it possible to update a configset being used by a collection using a > > SolrCloud API directly? I know that this is possible using the zkcli and > a > > collection RELOAD. We essentially want to be able to checkout our > configset > > from source control, and then replace everything in the active configset > in > > SolrCloud (other than the schema.xml). > > > > We have a couple of custom plugins that use config files that reside in > > the configset, and we don’t want to have to rebuild the collection or > > access zookeeper directly if we don’t have to. If we can’t modify the > > configset wholesale this way, is it possible to create a new configset > and > > swap the old collection to it? > > > > Best, > > Michael Carroll > > >
Re: Updating configset
I was in the same situation recently. I think it would be nice to have the configset UPLOAD command be able to override the existing configset instead of just fail (with a parameter such as override=true or something). We need to be careful with the trusted/unstrusted flag there, but that should be possible. > If we can’t modify the configset wholesale this way, is it possible to create a new configset and swap the old collection to it? You can create a new one and then call MODIFYCOLLECTION on the collection that uses it: https://lucene.apache.org/solr/guide/8_6/collection-management.html#modifycollection-parameters. I've never used that though. On Fri, Sep 11, 2020 at 7:26 AM Carroll, Michael (ELS-PHI) < m.carr...@elsevier.com> wrote: > Hello, > > I am running SolrCloud in Kubernetes with Solr version 8.5.2. > > Is it possible to update a configset being used by a collection using a > SolrCloud API directly? I know that this is possible using the zkcli and a > collection RELOAD. We essentially want to be able to checkout our configset > from source control, and then replace everything in the active configset in > SolrCloud (other than the schema.xml). > > We have a couple of custom plugins that use config files that reside in > the configset, and we don’t want to have to rebuild the collection or > access zookeeper directly if we don’t have to. If we can’t modify the > configset wholesale this way, is it possible to create a new configset and > swap the old collection to it? > > Best, > Michael Carroll >
Javascript pre-processing
I need to pre-process my data using Javascript, but I will need to make use of a JS library in order to accomplish this. Can I include other JS files in my pre-process script ? If so, how ?
Updating configset
Hello, I am running SolrCloud in Kubernetes with Solr version 8.5.2. Is it possible to update a configset being used by a collection using a SolrCloud API directly? I know that this is possible using the zkcli and a collection RELOAD. We essentially want to be able to checkout our configset from source control, and then replace everything in the active configset in SolrCloud (other than the schema.xml). We have a couple of custom plugins that use config files that reside in the configset, and we don’t want to have to rebuild the collection or access zookeeper directly if we don’t have to. If we can’t modify the configset wholesale this way, is it possible to create a new configset and swap the old collection to it? Best, Michael Carroll
Non-exists Field Query doesn't work when use defType=edismax
Hi everyone, I'm checking Solr query that contains condition "Exclude all documents that contain ceratin field". Currently, it looks like this: *(*:* AND -field_to_exclude:*)* Full query is: *((some_other_field:"value") AND ((*:* AND -field_to_exclude:*)))* If I use defType=edismax, nothing is found. If I use default defType - I get correct results, same as when I do not specify *:* in non-exists condition. My questions are: 1) why defType "edismax" leads to empty results? 2) do we really need to include clause "**:**" in non-exists condition? Isn't *-field_to_exclude:* *enough? -- Best Regards, Iana Bondarska
