How to emulate updateRequestProcessorChain with the Config API
Hello, I am developing a 'Solr management tool' that can configure Solr using its REST APIs and hit a small problem: I am able to configure pretty much everything I need except the updateRequestProcessorChain elements. According to the Config API page this is by design and I was wondering if there was a workaround I could use to create or edit updateRequestProcessorChain elements. One option I found in the documentation would be to create updateProcessor elements however I find this error prone because it would be up to users of the requestHandler to specify which processor they want to use (e.g.: /update/json?processor=remove_blanks,signature). Is there a way to create/edit updateRequestProcessorChain elements using Solr's REST API that I may not know about? I am trying to avoid directly modifying the solrconfig.xml file stored in ZooKeeper. Thanks, Alex
RE: Beginer's questions
Hi, I'm also a Solr beginner but I think I can answer a few of your questions: 5 - There's a bug in solr.cmd related to SSL where the settings defined in solr.in.cmd are ignored. You can see SOLR-8491 (https://issues.apache.org/jira/browse/SOLR-8491) for more information and fix. 6- From my understanding, the 'cUrl certificate' is only useful if your Solr instance is using a self-signed certificate; it is used by the client to validate that the server's certificate is valid. 7- The example (from your link) shows how to setups 2 Solr instances on the same machine using different ports (-p) and data directories (-s). When running in SolrCloud configuration, Solr's does not know which protocol to use when communicating with other Solr instances and uses HTTP by default. You can change this behavior by changing the urlScheme cluster property stored in ZooKeeper to HTTPS. Alex -Original Message- From: Kostas [mailto:k...@dataverse.gr] Sent: June 28, 2016 8:42 AM To: solr-user@lucene.apache.org Subject: RE: Beginer's questions Regarding the SSL question, it fails when I try this too : solr start -c -V ^ -Dsolr.ssl.checkPeerName=false ^ -Djavax.net.ssl.keyStorePassword=password ^ -Djavax.net.ssl.trustStorePassword=password ^ -Djavax.net.ssl.keyStore="F:/Users/me/Downloads/solr-6.1.0/server/etc/solr-s sl.keystore.jks" ^ -Djavax.net.ssl.trustStore="F:/Users/me/Downloads/solr-6.1.0/server/etc/solr -ssl.keystore.jks" But when I specify the exact same settings inside `solr-6.1.0\server\etc\ jetty-ssl.xml` it works. Do those two places do the same thing? This is the failure I get when jetty-ssl.xml has a wrong password: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl .java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.eclipse.jetty.start.Main.invokeMain(Main.java:214) at org.eclipse.jetty.start.Main.start(Main.java:457) at org.eclipse.jetty.start.Main.main(Main.java:75) Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:22 4) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.jav a:70) at java.security.KeyStore.load(KeyStore.java:1445) at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtil s.java:52) at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory. java:1016) at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java: 332) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.j ava:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle .java:132) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCyc le.java:114) at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.j ava:64) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.j ava:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle .java:132) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCyc le.java:114) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:26 0) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkCon nector.java:81) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:244) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.j ava:68) at org.eclipse.jetty.server.Server.doStart(Server.java:384) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.j ava:68) at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510) at java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435) ... 7 more Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778) ... 30 more >From a few tests that I've run, it seems that the `jetty-ssl.xml` overrides >the command-line arguments. I haven't seen you mention anywhere that I should update `jetty-ssl.xml` and the cmd-line doesn't seem to override those settings, so what am I missing? Kostas -Original Message- From: Kostas [m
How to create highlight search component using Config API
Hi,
I'm trying to create a highlight search component using the Config API of Solr
6.0.1 however I cannot figure out how to include the elements fragmenter,
formatter, encoder, etc...
Let's say I have the following component:
100
>From what I can see from the documentation my JSON should look a bit like this:
{
"add-searchcomponent":{
"name":"myHighlightingComponent",
"class":"solr.HighlightComponent",
??
}
}
However I have no idea how to defines the 2 fragmenters or the encoder. Any
help is appreciated.
Thanks
Alex
RE: How to create highlight search component using Config API
Hi,
Thanks for info, however I got an error when adding the field using curl (sorry
for the long lines):
Command I used:
curl -k -u user:password https://solr:8443/solr/q/config -H
'Content-type:application/json' -d '{ "add-searchcomponent": { "highlight": {
"name": "myHighlight", "class": "srl'.HighlightComponent", "": { "gap": {
"default": "true", "name": "gap", "class": "solr.highlight.GapFragmente,"'
"defaults": { "hl.fragsize": 100 } } }, "html": [{ "default": "true", "name":
"html", "class": "solr.highlith'.HtmlFormatter", "defaults": { "hl.simple.pre":
"before-", "hl.simple.post": "-after" }}, { "name": "html", "class":
"solr.highlight.HtmlEncoder"}] }}}'
The error I got was :
'name' is a required field", "'class' is a required field"
I was able to figure out the correct format:
curl -k -u user:password https://solr:8443/solr/q/config -H
'Content-type:application/json' -d '{ "add-searchcomponent": { "name":
"highlight", "class": "solr.HighlightComponent", "": { "gap": { "default":
"true", "name": "gap", "class": "solr.highlight.GapFragmenter", "defaults": {
"hl.fragsize": 100 } } }, "html": [{ "default": "true", "name": "html",
"class": "solr.highlight.HtmlFormatter", "defaults": { "hl.simple.pre":
before-", "hl.simple.post": "-after" }}, { "name": "html", "class":
"solr.highlight.HtmlEncoder"}] }}'
It was accepted by Solr and I can see it when going to the url
"solr//config". As you can see I use the 'highlight' name to
override the default one however my highlighter is not picked up by my request
handler. If I remove my highlight search component from the API and add it to
solrconfig.xml it is picked up by my request handler.
I compared the output of the url "solr//config" (solrconfig.xml
version) and it was identical to what I had before when I added the search
component using the API. I am at a loss why the search component works when
using solrconfig.xml but doesn't when using the Config API. Do you know why
this is the case?
I am using Solr 6.0.1 with ZooKeeper.
Thanks for any help
Alexandre Drouin
-Original Message-
From: Cassandra Targett [mailto:casstarg...@gmail.com]
Sent: July 8, 2016 5:04 PM
To: solr-user@lucene.apache.org
Subject: Re: How to create highlight search component using Config API
If you already have highlighting defined from one of the default configsets,
you can see an example of how the JSON is structured with a Config API request.
I assume you already tried that, but pointing it out just in case.
Defining a highlighter with the Config API is a bit confusing to be honest, but
I worked out something that works:
{"add-searchcomponent": {"highlight": {"name":"myHighlight",
"class":"solr.HighlightComponent","": {"gap": {"default":"true",
"name": "gap", "class":"solr.highlight.GapFragmenter",
"defaults":{"hl.fragsize":100}}},"html":[{"default": "true","name":
"html","class": "solr.highlight.HtmlFormatter","defaults":
{"hl.simple.pre":"",
"hl.simple.post":""}},{"name": "html","class":
"solr.highlight.HtmlEncoder"}]}}}
Note there is an empty string after the initial class definition (shown as "").
That lets you then add the fragmenters.
(I tried to prettify that, but my mail client isn't cooperating. I'm going to
add this example to the Solr Ref Guide, though so it might be easier to see
there in a few minutes.)
Hope it helps -
Cassandra
On Wed, Jun 29, 2016 at 8:00 AM, Alexandre Drouin
wrote:
> Hi,
>
> I'm trying to create a highlight search component using the Config API of
> Solr 6.0.1 however I cannot figure out how to include the elements
> fragmenter, formatter, encoder, etc...
>
> Let's say I have the following component:
>
>name="myHighlightingComponent">
>
>class="solr.highlight.GapFragmenter">
>
> 100
>
>
>class="solr.highlight.HtmlFormatter">
>
>
>
>
>
>
>
>
>
> From what I can see from the documentation my JSON should look a bit like
> this:
>
> {
> "add-searchcomponent":{
> "name":"myHighlightingComponent",
> "class":"solr.HighlightComponent",
> ??
> }
> }
>
> However I have no idea how to defines the 2 fragmenters or the encoder. Any
> help is appreciated.
>
> Thanks
> Alex
>
How to know if a core was reloaded without errors
Hi,
I am using the Collection API to reload a core and I was wondering if there is
a way to know if the core was reloaded without errors.
For my testing, I added a known error (an invalid field in a request handler)
in my configuration and I use the url
"solr/admin/collections?action=RELOAD&name=MyCollection&wt=json" to reload the
collection.
This url returns the following JSON:
{"responseHeader":{"status":0,"QTime":128},"success":{"localhost:8443_solr":{"responseHeader":{"status":0,"QTime":92.
And according to the documentation "status=0" means there was no errors with
the request. However if I look at the logs in Solr UI I can see the error I
created (org.apache.solr.common.SolrException: undefined field XYZ).
I have the following questions:
1) Is it possible to know if the reload of a core was completed without errors?
2) Does Solr have an API to download the logs? If #1 is not possible I could
hack something if I have access to Solr's logs using an API.
I am using Solr 6.0.1 with ZooKeeper.
Thanks,
Alexandre Drouin
RE: Can't load schema managed-schema: unknown field 'id'
Hello, You may have a uniqueKey that points to a field that do not exists anymore. You can try adding an "id" field using Solr's UI or the schema API since you are using the managed-schema. Alexandre Drouin -Original Message- From: Michael Joyner [mailto:mich...@newsrx.com] Sent: July 26, 2016 2:34 PM To: solr-user@lucene.apache.org Subject: Can't load schema managed-schema: unknown field 'id' |Help!| | | |What is the best way to recover from: | Can't load schema managed-schema: unknown field 'id' |I was managing the schema on a test collection, fat fingered it, but now I find out the schema ops seem to altering all collections on the core? SolrCloud 5.5.1 ||| | -Mike|||
RE: Can't load schema managed-schema: unknown field 'id'
Other than deleting the collection, I think you'll have to edit the manage-schema file manually. Since you are using SolrCloud you will need to use Solr's zkcli (https://cwiki.apache.org/confluence/display/solr/Command+Line+Utilities) utility to download and upload the file from ZooKeeper. Alexandre Drouin -Original Message- From: Michael Joyner [mailto:mich...@newsrx.com] Sent: July 26, 2016 3:48 PM To: solr-user@lucene.apache.org Subject: Re: Can't load schema managed-schema: unknown field 'id' Importance: High Same error via the UI: Can't load schema managed-schema: unknown field 'id' On 07/26/2016 03:05 PM, Alexandre Drouin wrote: > Hello, > > You may have a uniqueKey that points to a field that do not exists anymore. > You can try adding an "id" field using Solr's UI or the schema API since you > are using the managed-schema. > > > Alexandre Drouin > > -Original Message- > From: Michael Joyner [mailto:mich...@newsrx.com] > Sent: July 26, 2016 2:34 PM > To: solr-user@lucene.apache.org > Subject: Can't load schema managed-schema: unknown field 'id' > > |Help!| > > | > | > > |What is the best way to recover from: | > > Can't load schema managed-schema: unknown field 'id' > |I was managing the schema on a test collection, fat fingered it, but now > I find out the schema ops seem to altering all collections on the core? > SolrCloud 5.5.1 ||| > > | > -Mike|||
RE: Can't load schema managed-schema: unknown field 'id'
@Michael - there are GUI available for ZooKeeper: http://stackoverflow.com/questions/24551835/available-gui-for-zookeeper I used the Eclipse plugin before and while it is a bit clunky it gets the job done. Alexandre Drouin -Original Message- From: John Bickerstaff [mailto:j...@johnbickerstaff.com] Sent: July 26, 2016 4:21 PM To: solr-user@lucene.apache.org Subject: Re: Can't load schema managed-schema: unknown field 'id' Importance: High @Michael - somewhere there should be a "conf" directory for your SOLR instance. For my Dev efforts, I moved it to a different directory and I forget where it was, originally -- but if you search for solrconfig.xml or schema.xml, you should find it. It could be on your servers (or on only one of them) or, if someone has done a really good job, it's in source control somewhere... On Tue, Jul 26, 2016 at 2:17 PM, John Bickerstaff wrote: > type="string" > indexed="true" stored="true" required="true" multiValued="false" /> > > and further on in the file... > > id > > > On Tue, Jul 26, 2016 at 2:17 PM, John Bickerstaff < > j...@johnbickerstaff.com> wrote: > >> I don't see a managed schema file. As far as I understand it, id is >> set as a "uniqueKey" in the schema.xml file... >> >> On Tue, Jul 26, 2016 at 2:11 PM, Michael Joyner >> wrote: >> >>> ok, I think I need to do a manual edit on the managed-schema file >>> but I get "NoNode" for /managed-schema when trying to use the zkcli.sh file? >>> >>> >>> How can I get to this file and edit it? >>> >>> >>> On 07/26/2016 03:05 PM, Alexandre Drouin wrote: >>> >>>> Hello, >>>> >>>> You may have a uniqueKey that points to a field that do not exists >>>> anymore. You can try adding an "id" field using Solr's UI or the >>>> schema API since you are using the managed-schema. >>>> >>>> >>>> Alexandre Drouin >>>> >>>> -Original Message- >>>> From: Michael Joyner [mailto:mich...@newsrx.com] >>>> Sent: July 26, 2016 2:34 PM >>>> To: solr-user@lucene.apache.org >>>> Subject: Can't load schema managed-schema: unknown field 'id' >>>> >>>> |Help!| >>>> >>>> | >>>> | >>>> >>>> |What is the best way to recover from: | >>>> >>>> Can't load schema managed-schema: unknown field 'id' >>>> |I was managing the schema on a test collection, fat fingered it, >>>> |but >>>> now >>>> I find out the schema ops seem to altering all collections on the core? >>>> SolrCloud 5.5.1 ||| >>>> >>>> | >>>> -Mike||| >>>> >>> >>> >> >
Unable to create core
Hi, While I was testing I deleted my collection (customer) using the collection API and this step succeeded, however if I try to create a new collection using the same name I get the following error: Core with name 'customer_shard1_replica1' already exists." I am using SolrCloud 6.0.1 and ZK 3.4.8. How can I remove the core that should have been deleted when I removed my collection? Thanks, Alexandre Drouin
RE: Unable to create core
My deleted collection is not present in the Admin UI. I was able to fix my problem by deleting the core using the Core API and its unload command. Alexandre Drouin -Original Message- From: Alexandre Rafalovitch [mailto:arafa...@gmail.com] Sent: July 28, 2016 10:17 AM To: solr-user Subject: Re: Unable to create core Importance: High What do you see in the Admin UI? Is it there? Also, deleting the core - I believe - still leaves the underlying directory on the filesystem in place. So perhaps it is actually complaining about the leftover filesystem directory when you try to create a new core with the same name/location. Check the filesystem for that directory (customer_shard1_replica1). Regards, Alex. Newsletter and resources for Solr beginners and intermediates: http://www.solr-start.com/ On 29 July 2016 at 00:06, Alexandre Drouin wrote: > Hi, > > While I was testing I deleted my collection (customer) using the collection > API and this step succeeded, however if I try to create a new collection > using the same name I get the following error: Core with name > 'customer_shard1_replica1' already exists." > I am using SolrCloud 6.0.1 and ZK 3.4.8. > > How can I remove the core that should have been deleted when I removed my > collection? > > Thanks, > Alexandre Drouin
Where should the listeners be defined in solrconfig.xml (Solr 6.0.1)
Hello, I was wondering where I should put the configuration. I can see from the sample solrconfig.xml that they are defined under the and elements. The Schema API for listeners does not specify a parent of type updateHandler or query so I wanted to know if I also define them directly under the root of the xml document (config)? Alexandre Drouin
RE: Where should the listeners be defined in solrconfig.xml (Solr 6.0.1)
Does anyone knows where the listener should be defined in solrconfig.xml? Alexandre Drouin -Original Message- From: Alexandre Drouin [mailto:alexandre.dro...@orckestra.com] Sent: July 29, 2016 10:46 AM To: solr-user@lucene.apache.org Subject: Where should the listeners be defined in solrconfig.xml (Solr 6.0.1) Importance: High Hello, I was wondering where I should put the configuration. I can see from the sample solrconfig.xml that they are defined under the and elements. The Schema API for listeners does not specify a parent of type updateHandler or query so I wanted to know if I also define them directly under the root of the xml document (config)? Alexandre Drouin
RE: Where should the listeners be defined in solrconfig.xml (Solr 6.0.1)
That's good to know thanks! I should have thought to check the java code before asking. Alexandre Drouin -Original Message- From: Mikhail Khludnev [mailto:m...@apache.org] Sent: August 3, 2016 10:52 AM To: solr-user Subject: Re: Where should the listeners be defined in solrconfig.xml (Solr 6.0.1) Importance: High As far as I remember the code it captures everywhere https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/core/SolrConfig.java#L331 Double slash "//listener" means "everywhere". On Wed, Aug 3, 2016 at 4:38 PM, Alexandre Drouin < alexandre.dro...@orckestra.com> wrote: > Does anyone knows where the listener should be defined in solrconfig.xml? > > > Alexandre Drouin > > > -Original Message- > From: Alexandre Drouin [mailto:alexandre.dro...@orckestra.com] > Sent: July 29, 2016 10:46 AM > To: solr-user@lucene.apache.org > Subject: Where should the listeners be defined in solrconfig.xml (Solr > 6.0.1) > Importance: High > > Hello, > > I was wondering where I should put the configuration. I > can see from the sample solrconfig.xml that they are defined under the > and elements. > The Schema API for listeners does not specify a parent of type > updateHandler or query so I wanted to know if I also define them > directly under the root of the xml document (config)? > > Alexandre Drouin > > -- Sincerely yours Mikhail Khludnev
RE: Solr 6.1 :: language specific analysis
Can you use Solr's synonym feature? You can find a German synonym file here: https://sites.google.com/site/kevinbouge/synonyms-lists Alexandre Drouin -Original Message- From: Rainer Gnan [mailto:rainer.g...@bsb-muenchen.de] Sent: Wednesday, August 10, 2016 10:21 AM To: solr-user@lucene.apache.org Subject: Solr 6.1 :: language specific analysis Hello, I wonder if solr offers a feature (class) to handle different orthogaphy versions? For the German language for example ... in order to find the same documents when searching after "Foto" or "Photo". I appreachiate any help! Rainer Rainer Gnan Bayerische Staatsbibliothek BibliotheksVerbund Bayern Verbundnahe Dienste 80539 München Tel.: +49(0)89/28638-2445 Fax: +49(0)89/28638-2665 E-Mail: rainer.g...@bsb-muenchen.de
Create collection on all nodes using the Collection API
Hi, What would be the best/easiest way to create a collection (only one shard) using the Collection API and have a replica created on all live nodes? Using the 'create collection' API, I can use the 'replicationFactor' parameter and specify the number of replica I want for my collection. So if I have 3 lives nodes I can say 'replicationFactor=3' and my collection will have a replica on all lives nodes. However I do not want to 'hardcode' my number of live nodes for obvious reasons, so because of that I have the following questions: 1) Is there a way to create a collection (only one shard) and having a replica of the shard on all live nodes? 2) Assuming #1 is not possible, is it possible to have the list of live nodes ? If I can have the list of live nodes I could detect the number required for the replicationFactor parameter. Thanks Alexandre Drouin
Getting "collection already exists" when creating collection in admin UI
Hello,
I am running SolrCloud with 2 nodes (Solr 6.1 with SSL and basic auth) and with
one Zookeeper node (for development purposes) and when I try to create a new
collection in the admin UI with 'replicationFactor=2' I get a "Connection to
Solr lost" message and another message telling me " collection already exists:
MyNewNode". I made sure that a collection with the same name does not exists
and the issue does not appear with a replication factor of 1.
While debugging I saw that the create action is called twice with the following
parameters:
/solr/admin/collections?_=1471010473184&action=CREATE&collection.configName=DefaultConfig&maxShardsPerNode=1&name=aaa&numShards=1&replicationFactor=2&router.name=compositeId&routerName=compositeId&wt=json
Can anyone replicate this issue? I have not found it in JIRA.
Below is the relevant log (if useful) and I posted the full logs here
https://gist.github.com/orck-adrouin/690d485ba0835320273e7b2e09fb3771
63549 ERROR
(OverseerThreadFactory-5-thread-5-processing-n:orc-dev-solr-cd.local:8444_solr)
[ ] o.a.s.c.OverseerCollectionMessageHandler Collection: MyNewNode operation:
create failed:org.apache.solr.common.SolrException: collection already exists:
MyNewNode
at
org.apache.solr.cloud.OverseerCollectionMessageHandler.createCollection(OverseerCollectionMessageHandler.java:1832)
at
org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:224)
at
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:463)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Thanks,
Alexandre Drouin
RE: Getting "collection already exists" when creating collection in admin UI
Hi Esther-Melaine, The collection exists in Zookeeper under the /collections node and I can see the shardX_replicaX folders under $SOLR_HOME/server/solr of both servers. I was not able to replicate the issue using the collection API. Here are the logs where I added the 'MyNewerNode' https://gist.github.com/orck-adrouin/4d074cbb60141cba90c0aae9c55360d4 I took a closer look at the admin UI and here are my findings: - In Chrome's devtool I can see the first create request - After 10 seconds the request getting aborted and a second create request is sent to the server - In Fiddler I can see that the first request completes successfully without any issues. The second request is sent a few seconds before the first one ends so it looks like a admin UI issue. Is it possible that the admin UI has some kind of TTL for requests set to 10 seconds? You mentioned something about the nodes going into recovery. Any idea how I can fix this issue? My development environment (if it makes a difference): - OS: Windows - 2 Solr 6.1 nodes using SolrCloud. They both are running on the same server using different ports. - Zookeeper 3.4.8 Alexandre Drouin -Original Message- From: Esther-Melaine Quansah [mailto:esther.quan...@lucidworks.com] Sent: August 12, 2016 10:46 AM To: solr-user@lucene.apache.org Subject: Re: Getting "collection already exists" when creating collection in admin UI Importance: High Hi Alexandre, The question here is why the create action is called twice. You’re getting that “collection already exists” error after the second action is called. Can you verify if MyNewNode exists in /collections in ZK or on the machines running Solr at $SOLR_HOME/server/solr/ Your logs show a lot of issues around the overseer and it looks like those nodes are going into recovery pretty frequently. Can you replicate this issue by creating a collection through the API (not through the UI): http://localhost:8983/admin/collections?action=CREATE&name=MyNewerNode&numShards=1&replicationFactor=2&maxShardsPerNode=1&collection.configName=DefaultConfig Thanks, Esther > On Aug 12, 2016, at 10:05 AM, Alexandre Drouin > wrote: > > Hello, > > I am running SolrCloud with 2 nodes (Solr 6.1 with SSL and basic auth) and > with one Zookeeper node (for development purposes) and when I try to create a > new collection in the admin UI with 'replicationFactor=2' I get a > "Connection to Solr lost" message and another message telling me " collection > already exists: MyNewNode". I made sure that a collection with the same name > does not exists and the issue does not appear with a replication factor of 1. > > > While debugging I saw that the create action is called twice with the > following parameters: > /solr/admin/collections?_=1471010473184&action=CREATE&collection.confi > gName=DefaultConfig&maxShardsPerNode=1&name=aaa&numShards=1&replicatio > nFactor=2&router.name=compositeId&routerName=compositeId&wt=json > > Can anyone replicate this issue? I have not found it in JIRA. > > > Below is the relevant log (if useful) and I posted the full logs here > https://gist.github.com/orck-adrouin/690d485ba0835320273e7b2e09fb3771 > > 63549 ERROR > (OverseerThreadFactory-5-thread-5-processing-n:orc-dev-solr-cd.local:8444_solr) > [ ] o.a.s.c.OverseerCollectionMessageHandler Collection: MyNewNode > operation: create failed:org.apache.solr.common.SolrException: collection > already exists: MyNewNode > at > org.apache.solr.cloud.OverseerCollectionMessageHandler.createCollection(OverseerCollectionMessageHandler.java:1832) > at > org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:224) > at > org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:463) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Thanks, > Alexandre Drouin
RE: Getting "collection already exists" when creating collection in admin UI
Thanks for the offer however I think I have a different issue. I reinstalled my ZK and Solr servers between each tests so I didn't have any unwanted files. Alexandre Drouin -Original Message- From: John Bickerstaff [mailto:j...@johnbickerstaff.com] Sent: August 12, 2016 1:43 PM To: solr-user@lucene.apache.org Subject: Re: Getting "collection already exists" when creating collection in admin UI Importance: High I don't know if this helps, but I had trouble creating collections due to a number of issues and I think I got this error (I was using the command line, not the UI) As I recall, if it exists in Zookeeper, it will error out. It was a while ago, but I think the way I had to solve it was to go into Zookeeper and delete the "node". This was easier for me because I was using "chroot" in Zookeeper such that each collection was separate - so all I had to do was delete the entire node and start over. Take me with a grain of salt - it was a while ago. If you want, I have linux command lines for most / all of this... let me know. On Fri, Aug 12, 2016 at 11:10 AM, Alexandre Drouin < alexandre.dro...@orckestra.com> wrote: > Hi Esther-Melaine, > > The collection exists in Zookeeper under the /collections node and I > can see the shardX_replicaX folders under $SOLR_HOME/server/solr of > both servers. > > I was not able to replicate the issue using the collection API. Here > are the logs where I added the 'MyNewerNode' > https://gist.github.com/orck- > adrouin/4d074cbb60141cba90c0aae9c55360d4 > > I took a closer look at the admin UI and here are my findings: > - In Chrome's devtool I can see the first create request > - After 10 seconds the request getting aborted and a second create > request is sent to the server > - In Fiddler I can see that the first request completes successfully > without any issues. The second request is sent a few seconds before > the first one ends so it looks like a admin UI issue. > > Is it possible that the admin UI has some kind of TTL for requests set > to > 10 seconds? > > You mentioned something about the nodes going into recovery. Any idea > how I can fix this issue? > > My development environment (if it makes a difference): > - OS: Windows > - 2 Solr 6.1 nodes using SolrCloud. They both are running on the > same server using different ports. > - Zookeeper 3.4.8 > > Alexandre Drouin > > > -Original Message- > From: Esther-Melaine Quansah [mailto:esther.quan...@lucidworks.com] > Sent: August 12, 2016 10:46 AM > To: solr-user@lucene.apache.org > Subject: Re: Getting "collection already exists" when creating > collection in admin UI > Importance: High > > Hi Alexandre, > > The question here is why the create action is called twice. You’re > getting that “collection already exists” error after the second action is > called. > Can you verify if MyNewNode exists in /collections in ZK or on the > machines running Solr at $SOLR_HOME/server/solr/ Your logs show a lot > of issues around the overseer and it looks like those nodes are going > into recovery pretty frequently. Can you replicate this issue by > creating a collection through the API (not through the UI): > > http://localhost:8983/admin/collections?action=CREATE&; > name=MyNewerNode&numShards=1&replicationFactor=2& > maxShardsPerNode=1&collection.configName=DefaultConfig > > Thanks, > Esther > > > > On Aug 12, 2016, at 10:05 AM, Alexandre Drouin < > alexandre.dro...@orckestra.com> wrote: > > > > Hello, > > > > I am running SolrCloud with 2 nodes (Solr 6.1 with SSL and basic > > auth) > and with one Zookeeper node (for development purposes) and when I try > to create a new collection in the admin UI with 'replicationFactor=2' > I get a "Connection to Solr lost" message and another message telling me " > collection already exists: MyNewNode". I made sure that a collection > with the same name does not exists and the issue does not appear with > a replication factor of 1. > > > > While debugging I saw that the create action is called twice with > > the following parameters: > > /solr/admin/collections?_=1471010473184&action=CREATE&collection.con > > fi > > gName=DefaultConfig&maxShardsPerNode=1&name=aaa&numShards=1&replicat > > io nFactor=2&router.name=compositeId&routerName=compositeId&wt=json > > > > Can anyone replicate this issue? I have not found it in JIRA. > > > > > > Below is the relevant log (if useful) and I posted the full logs > > here > > https://gist.gi
RE: Getting "collection already exists" when creating collection in admin UI
I removed everything related to Zookeeper or Solr between each of my test, including the data directory. Alexandre Drouin -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: August 12, 2016 3:24 PM To: solr-user Subject: RE: Getting "collection already exists" when creating collection in admin UI Importance: High Reinstalling ZK wouldn't help if the data directory weren't purged..m On Aug 12, 2016 11:35, "Alexandre Drouin" wrote: > Thanks for the offer however I think I have a different issue. I > reinstalled my ZK and Solr servers between each tests so I didn't have > any unwanted files. > > > Alexandre Drouin > > -Original Message- > From: John Bickerstaff [mailto:j...@johnbickerstaff.com] > Sent: August 12, 2016 1:43 PM > To: solr-user@lucene.apache.org > Subject: Re: Getting "collection already exists" when creating > collection in admin UI > Importance: High > > I don't know if this helps, but I had trouble creating collections due > to a number of issues and I think I got this error (I was using the > command line, not the UI) > > As I recall, if it exists in Zookeeper, it will error out. It was a > while ago, but I think the way I had to solve it was to go into > Zookeeper and delete the "node". > > This was easier for me because I was using "chroot" in Zookeeper such > that each collection was separate - so all I had to do was delete the > entire node and start over. > > Take me with a grain of salt - it was a while ago. > > If you want, I have linux command lines for most / all of this... let > me know. > > On Fri, Aug 12, 2016 at 11:10 AM, Alexandre Drouin < > alexandre.dro...@orckestra.com> wrote: > > > Hi Esther-Melaine, > > > > The collection exists in Zookeeper under the /collections node and I > > can see the shardX_replicaX folders under $SOLR_HOME/server/solr of > > both servers. > > > > I was not able to replicate the issue using the collection API. > > Here are the logs where I added the 'MyNewerNode' > > https://gist.github.com/orck- > > adrouin/4d074cbb60141cba90c0aae9c55360d4 > > > > I took a closer look at the admin UI and here are my findings: > > - In Chrome's devtool I can see the first create request > > - After 10 seconds the request getting aborted and a second create > > request is sent to the server > > - In Fiddler I can see that the first request completes > > successfully without any issues. The second request is sent a few > > seconds before the first one ends so it looks like a admin UI issue. > > > > Is it possible that the admin UI has some kind of TTL for requests > > set to > > 10 seconds? > > > > You mentioned something about the nodes going into recovery. Any > > idea how I can fix this issue? > > > > My development environment (if it makes a difference): > > - OS: Windows > > - 2 Solr 6.1 nodes using SolrCloud. They both are running on the > > same server using different ports. > > - Zookeeper 3.4.8 > > > > Alexandre Drouin > > > > > > -Original Message- > > From: Esther-Melaine Quansah [mailto:esther.quan...@lucidworks.com] > > Sent: August 12, 2016 10:46 AM > > To: solr-user@lucene.apache.org > > Subject: Re: Getting "collection already exists" when creating > > collection in admin UI > > Importance: High > > > > Hi Alexandre, > > > > The question here is why the create action is called twice. You’re > > getting that “collection already exists” error after the second > > action > is called. > > Can you verify if MyNewNode exists in /collections in ZK or on the > > machines running Solr at $SOLR_HOME/server/solr/ Your logs show a > > lot of issues around the overseer and it looks like those nodes are > > going into recovery pretty frequently. Can you replicate this issue > > by creating a collection through the API (not through the UI): > > > > http://localhost:8983/admin/collections?action=CREATE&; > > name=MyNewerNode&numShards=1&replicationFactor=2& > > maxShardsPerNode=1&collection.configName=DefaultConfig > > > > Thanks, > > Esther > > > > > > > On Aug 12, 2016, at 10:05 AM, Alexandre Drouin < > > alexandre.dro...@orckestra.com> wrote: > > > > > > Hello, > > > > > > I am running SolrCloud with 2 nodes (Solr 6.1 with SSL and basic > > > auth) > > and with one Zookeeper node (for developme
Always add the marker when elevating documents
Hi, I followed the instructions on the wiki (https://wiki.apache.org/solr/QueryElevationComponent) to add a QueryElevationComponent searchComponent in my Solr 4.10.2 server and it is working as expected. I saw in the documentation that it is possible to see which documents were elevated by adding [elevated] to the fl parameter and I would like to know if there is a way to always have the [elevated] property the results without having to add it to the fl parameter. If it is not possible, is it safe to use "fl=*,[elevated]" to specify all fields plus the elevated marker? Thanks! Alexandre Drouin
RE: Always add the marker when elevating documents
Thank you Emir. Alexandre Drouin -Original Message- From: Emir Arnautovic [mailto:emir.arnauto...@sematext.com] Sent: September 2, 2016 5:32 AM To: solr-user@lucene.apache.org Subject: Re: Always add the marker when elevating documents Importance: High Hi Alexandre, You can specify default fl paramter for search handler in Solr config. You can use *,[elevated] to return all fields + elevated, but it is recommended to limit fl to fields needed - if you truly need all fields, then using * is ok. Regards, Emir On 01.09.2016 22:11, Alexandre Drouin wrote: > Hi, > > I followed the instructions on the wiki > (https://wiki.apache.org/solr/QueryElevationComponent) to add a > QueryElevationComponent searchComponent in my Solr 4.10.2 server and it is > working as expected. > > I saw in the documentation that it is possible to see which documents were > elevated by adding [elevated] to the fl parameter and I would like to know if > there is a way to always have the [elevated] property the results without > having to add it to the fl parameter. > If it is not possible, is it safe to use "fl=*,[elevated]" to specify all > fields plus the elevated marker? > > Thanks! > Alexandre Drouin -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/
RE: Solr 4.10.2 Enabling Basic Authentication in HTTP
I forgot to mention that I also tried this connector without success: 5 1500 false If I remove the basic authentication related configuration from web.xml I can access Solr without issue so I know the connector is working, I just can't get it to work with basic auth. Alexandre Drouin -Original Message- From: Alexandre Drouin [mailto:alexandre.dro...@orckestra.com] Sent: October 5, 2016 2:07 PM To: solr-user@lucene.apache.org Subject: Solr 4.10.2 Enabling Basic Authentication in HTTP Importance: High Hi, I currently have basic authentication enabled with HTTPS and it is working great. I want to remove SSL from my Solr installation (master/slave) while keeping the basic authentication. To this effect, I removed the SslSelectChannelConnector connector from my jetty.xml file and I replaced it with the SelectChannelConnector connector like so: 5 2 false 5000 5000 However I get a 403 error when I try to access the Solr server (url: /solr). I did not modify the security-constraint nodes of my web.xml file since the basic authentication was working in HTTPS and I can provide the file if necessary. How can I enable basic authentication for HTTP in Solr 4.10.2 ? Alexandre Drouin
Solr 4.10.2 Enabling Basic Authentication in HTTP
Hi, I currently have basic authentication enabled with HTTPS and it is working great. I want to remove SSL from my Solr installation (master/slave) while keeping the basic authentication. To this effect, I removed the SslSelectChannelConnector connector from my jetty.xml file and I replaced it with the SelectChannelConnector connector like so: 5 2 false 5000 5000 However I get a 403 error when I try to access the Solr server (url: /solr). I did not modify the security-constraint nodes of my web.xml file since the basic authentication was working in HTTPS and I can provide the file if necessary. How can I enable basic authentication for HTTP in Solr 4.10.2 ? Alexandre Drouin
IndexWriter exception
java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is
closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:698)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:712)
at
org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4653)
at
org.apache.lucene.index.StandardDirectoryReader.isCurrent(StandardDirectoryReader.java:357)
at
org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:562)
at
org.apache.solr.handler.admin.CoreAdminHandler.getCoreStatus(CoreAdminHandler.java:1152)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:698)
... 31 more
Alexandre Drouin
RE: IndexWriter exception
I checked my source control history and "6" was the original value that was
checked-in. I'll investigate lowering this value in our next iteration.
Thanks for the hint.
Alexandre Drouin
-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: December 6, 2016 5:42 PM
To: solr-user
Subject: Re: IndexWriter exception
Importance: High
bq: maxWarmingSearchers is set to 6
Red flag ref. If this was done to avoid the warning in the logs about too many
warming searchers, it's a clear indication that you're committing far too
often. Let's see exactly what you're using to post when you say you're "using
the REST API". My bet: each one does a commit. If this is the post.jar tool
there's an option to _not_ commit and I'd be sure to set that and let your
autocommit settings handle committing.
My guess is that you are adding more and more documents to Solr, thus making it
more likely that you are opening a bunch of searchers at once (see above) and
running into a race condition. So the first thing I'd do is straighten that out
and see if the problem goes away.
Best,
Erick
On Tue, Dec 6, 2016 at 1:34 PM, Alexandre Drouin
wrote:
> Hello,
>
> I have an error that has been popping up randomly since 3 weeks ago and the
> randomness of the issue makes it hard to troubleshoot.
>
> I have a service that use the REST API to index documents (1000 docs at a
> time) and in this process I often call the core status API
> (/solr/admin/cores?action=STATUS) to get the statuses of the different cores.
> This process has been working flawlessly since 2014 however it has been
> failing recently with the exception: " this IndexWriter is closed".
>
> I did a few search on Google for this exception but I did not see anything
> relevant. Does anyone have an idea how to troubleshoot/fix this issue?
>
> This is my configuration:
> - Solr 4.10.2 on Windows. I am not using SolrCloud.
> - Java 1.7.0_79 24.79-b02
> - useColdSearcher is set to true
> - maxWarmingSearchers is set to 6
> - I changed my Solr configuration about 2-3 months ago: I disabled HTTPS and
> enabled the logging (INFO level) but I do not think this could cause the
> issue.
>
> Relevant stack trace:
>
> INFO - 2016-12-06 18:43:23.854; org.apache.solr.update.CommitTracker;
> Hard AutoCommit: if uncommited for 9ms; if 75000 uncommited docs
> INFO - 2016-12-06 18:43:23.856; org.apache.solr.update.CommitTracker;
> Soft AutoCommit: if uncommited for 15000ms; INFO - 2016-12-06
> 18:43:23.929; org.apache.solr.update.processor.LogUpdateProcessor;
> [coreENCA] webapp=/solr path=/update params={commit=false}
> {add=[Global_44235 (1552993270510911488), Global_44236Pony
> (1552993270516154368), Global_44236Magnum (1552993270518251520),
> Global_44237Pony (1552993270519300096), Global_44237Split
> (1552993270521397249), Global_44237Standard (1552993270523494401),
> Global_44238Pony (1552993270525591553), Global_44238Standard
> (1552993270527688704), Global_44238Magnum (1552993270529785856),
> Global_44239Standard (1552993270531883008), ... (2102 adds)]} 0 8292
> INFO - 2016-12-06 18:43:23.933; org.apache.solr.core.SolrCore;
> [coreENCA] CLOSING SolrCore org.apache.solr.core.SolrCore@5730eaaf
> INFO - 2016-12-06 18:43:23.935;
> org.apache.solr.update.DirectUpdateHandler2; closing
> DirectUpdateHandler2{commits=0,autocommit maxDocs=75000,autocommit
> maxTime=9ms,autocommits=0,soft autocommit maxTime=15000ms,soft
> autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=417
> 6,adds=4176,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=41
> 76,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_err
> ors=0,transaction_logs_total_size=84547858,transaction_logs_total_numb
> er=1} INFO - 2016-12-06 18:43:23.936; org.apache.solr.core.SolrCore;
> [coreENCA] Closing main searcher on request.
> INFO - 2016-12-06 18:43:24.044;
> org.apache.solr.search.SolrIndexSearcher; Opening Searcher@73a40fbb[coreENCA]
> main ERROR - 2016-12-06 18:43:24.045; org.apache.solr.common.SolrException;
> org.apache.solr.common.SolrException: Error handling 'status' action
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:710)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:214)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:188)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at
> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:729)
> at
&g
Promote documents if they are present in the search results
Hello, I am developing a new feature in my product where an administrator can select a list of documents for different sections of the application. You can think of this feature as the classic categorization in an e-commerce site (e.g.: Boots, Headgear, etc). Since this categorization is different depending on the section/category I cannot easily store which documents are promoted in Solr itself. I know ahead of time which documents should be promoted for which section so I would like to know if there is a way to "pass" these documents to Solr and have them at the top of the list if they are found in the search results. For example let's say that I have the following documents: - Doc1 (name: A, Age: 12) - Doc2 (name: B, Age: 24) - Doc3 (name: C, Age: 36) I would like to promote Doc3 at the top of the list and if someone search for "name:B" or "Age:12" the results should not include Doc3 because it did not match the query. I looked at QueryElevationComponent and it doesn't do exactly what I want: the component is based on a "query" and this query can be different in my use case. I know I can use the "elevatedIds" parameter to elevate documents however they are always elevated even if they are not present in the search results. Is it possible to achieve what I need with Solr? I am using Solr 4.10.2 Thanks! Alexandre Drouin
RE: Promote documents if they are present in the search results
Hi Alexandre, A boost should works for my needs. Thanks! Alexandre Drouin -Original Message- From: Alexandre Rafalovitch [mailto:arafa...@gmail.com] Sent: February 3, 2017 12:14 PM To: solr-user Subject: Re: Promote documents if they are present in the search results Importance: High Could you do a basic boost query with a list of all those document ids. If they are in results, their relevance is boosted. If they are not, they are just ignored. You could have that boost query configured in the solrconfig.xml or passed in by a client. Regards, Alex http://www.solr-start.com/ - Resources for Solr users, new and experienced On 3 February 2017 at 12:06, Alexandre Drouin wrote: > Hello, > > I am developing a new feature in my product where an administrator can select > a list of documents for different sections of the application. You can think > of this feature as the classic categorization in an e-commerce site (e.g.: > Boots, Headgear, etc). Since this categorization is different depending on > the section/category I cannot easily store which documents are promoted in > Solr itself. I know ahead of time which documents should be promoted for > which section so I would like to know if there is a way to "pass" these > documents to Solr and have them at the top of the list if they are found in > the search results. > > For example let's say that I have the following documents: > - Doc1 (name: A, Age: 12) > - Doc2 (name: B, Age: 24) > - Doc3 (name: C, Age: 36) > > I would like to promote Doc3 at the top of the list and if someone search for > "name:B" or "Age:12" the results should not include Doc3 because it did not > match the query. > > I looked at QueryElevationComponent and it doesn't do exactly what I want: > the component is based on a "query" and this query can be different in my use > case. I know I can use the "elevatedIds" parameter to elevate documents > however they are always elevated even if they are not present in the search > results. > > Is it possible to achieve what I need with Solr? I am using Solr > 4.10.2 > > Thanks! > > > Alexandre Drouin
