Is it possible to force a Shard Leader change?

[Tim Chen]

Hi Guys,

I am running a Solr Cloud 4.10, with 4 Solr servers and 5 Zookeeper setup.

Solr servers:
solr01, solr02, solr03, solr04

I have around 20 collections in Solr cloud, and there are 4 Shards for each 
Collection. For each Shard, I have 4 Replicas, and sitting on each Solr server, 
with one of them is the Shard Leader.

The issue I am having right now is all the Shard Leader are pointing to the 
same server, eg: solr01.  When there are documents update, they are all pushed 
to the Leader. I really want to distribute the Shard Leader across all 4 Solr 
servers.

I noticed Solr 6 has a "REBALANCELEADERS" command to do that, but not available 
in Solr 4.

Questions:

1, Is my setup OK? with 4 Shards for each Collection and 4 Replicas for each 
Shard. Each Solr server has full set of documents.
2, To distribute the Shard Leader to different Solr servers, can I somehow 
shutdown a single Replica that is currently a Shard Leader and force Solr to 
elect a different replica to be new Shard Leader?

Thanks guys!

Regards,
Tim


[Roots Wednesday 27 July 8.30pm]

The Query Elevation Component

[Ryan Yacyshyn]

Hi everyone,

I'm reading the docs on the query elevation component and some questions
came up:

Can I specify a field that the elevate component will look at, such as only
looking at the title field? My search handler (using eDisMax) is searching
across multiple fields, but if I only want the elevate component to look at
one field, is this possible? I'd like the search request to search multiple
fields, but only elevate if the query is found in one of the fields.

Also, is there a recommended way to analyze the query? For example, when
using the queryFieldType parameter, I'd think I'd only want to use the
KeywordTokenizer and maybe lowercasing.

Thanks,
Ryan

Re: Search sort depth limited to 4?

[Joel Bernstein]

Yes, currently there is the 4 sort field limit. A custom handler could be
built that allows for unlimited sorts or you could provide a patch to the
export handler.

I think though that you'll find that performance drops off quite a bit as
the number of sort fields increases. This is because each field needs to be
retrieved from the docValues cache in order to perform the sort.

Joel Bernstein
http://joelsolr.blogspot.com/

On Tue, Jul 26, 2016 at 2:58 PM, tedsolr  wrote:

> So I found the limit in the Ref Doc p. 394, under the /export request
> handler:
>
> "Up to four sort fields can be specified per request, with the 'asc' or
> 'desc' properties"
>
> Yikes I'm in trouble. Does anyone know if this can be circumvented? Can I
> write a custom handler that could handle up to 20? Oh boy.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Search-sort-depth-limited-to-4-tp4289049p4289053.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: SolrCloud create_collection not uploading configs to zookeeper

[Nirav Patel]

OK, I can see '/configs' directory in Solr UI and under that I can see
configuration fo my 'test' collection. BUt this all seemed to be disjointed
information. Doc is definitely not clear.
And what that Tree represent anyway where;s information for that. I would
ideally put link to that document.

Also where are those configs file on filesystem anyway?

On Tue, Jul 26, 2016 at 4:24 PM, Nirav Patel  wrote:

> Hi Erick,
>
> I did read that paragraph. It says: First, if you don't provide the -d or
> -n options, then the default configuration
> ($SOLR_HOME/server/solr/configsets/data_driven_schema_configs/conf) is
> uploaded to ZooKeeper using the same name as the collection. For example,
> the following command will result in the data_driven_schema_configs
> configuration being uploaded to /configs/contacts in ZooKeeper:  bin/solr
> create -c contacts.
>
> Here's what I did :
>
> /bin/solr create_collection -c test
> INFO  - 2016-07-26 15:59:15.408;
> org.apache.solr.client.solrj.impl.CloudSolrClient; Final constructed zkHost
> string: 10.250.72.130:2181
>
> Connecting to ZooKeeper at 10.250.72.130:2181 ...
> Uploading
> /home/np/solr-6.1.0/server/solr/configsets/data_driven_schema_configs/conf
> for config test to ZooKeeper at 10.250.72.130:2181
>
> Creating new collection 'test' using command:
>
> http://localhost:8983/solr/admin/collections?action=CREATE=test=1=1=1=test
>
> {
>   "responseHeader":{
> "status":0,
> "QTime":5006},
>   "success":{"10.250.70.120:8983_solr":{
>   "responseHeader":{
> "status":0,
> "QTime":3667},
>   "core":"test_shard1_replica1"}}}
>
>
> So according to doc I should see data_driven_schema_configs being uploaded
> to  ZooKeeper under "/configs/test" directory. It seems like doc is
> mentioning a literal "configs" directory. BUt I can't find that under my
> zookeeper installation.
>
>
> On Tue, Jul 26, 2016 at 4:18 PM, Erick Erickson 
> wrote:
>
>> From the doc you referenced where it outlines the parameters for the
>> create command:
>>
>> -d  : The configuration directory. This defaults to
>> data_driven_schema_configs.
>>
>> You should also review the linked section about what configuration
>> directories are all about:
>>
>> https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference#SolrStartScriptReference-ConfigurationDirectoriesandSolrCloud
>>
>> Best,
>> Erick
>>
>>
>> On Tue, Jul 26, 2016 at 4:11 PM, Nirav Patel 
>> wrote:
>> > Hi,
>> >
>> > I have 2 nodes solr cluster with 1 node standalone zookeeper.
>> >
>> > I tried following from following doc to create collection:
>> >
>> > bin/solr create -c contacts.
>> >
>> > According to doc should upload config date into /configs/contacts in
>> > ZooKeeper. But I can't find any configs directory under zookeeper.
>> >
>> >
>> https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference#SolrStartScriptReference-Create
>> >
>> > Where did config files go? I want to modify them using vi and then
>> upload
>> > it again. But first I want to locate them on filesystem. According to
>> doc
>> > on "Configuration Directories and SolrCloud" it should be under zookeepr
>> > but it's not!
>> >
>> > Thanks
>> >
>> > --
>> >
>> >
>> > [image: What's New with Xactly] > >
>> >
>> >   [image: LinkedIn]
>> >   [image: Twitter]
>> >   [image: Facebook]
>> >   [image: YouTube]
>> > 
>>
>
>

-- 


[image: What's New with Xactly] 

  [image: LinkedIn] 
  [image: Twitter] 
  [image: Facebook] 
  [image: YouTube] 

Re: SolrCloud create_collection not uploading configs to zookeeper

[Nirav Patel]

Hi Erick,

I did read that paragraph. It says: First, if you don't provide the -d or
-n options, then the default configuration
($SOLR_HOME/server/solr/configsets/data_driven_schema_configs/conf) is
uploaded to ZooKeeper using the same name as the collection. For example,
the following command will result in the data_driven_schema_configs
configuration being uploaded to /configs/contacts in ZooKeeper:  bin/solr
create -c contacts.

Here's what I did :

/bin/solr create_collection -c test
INFO  - 2016-07-26 15:59:15.408;
org.apache.solr.client.solrj.impl.CloudSolrClient; Final constructed zkHost
string: 10.250.72.130:2181

Connecting to ZooKeeper at 10.250.72.130:2181 ...
Uploading
/home/np/solr-6.1.0/server/solr/configsets/data_driven_schema_configs/conf
for config test to ZooKeeper at 10.250.72.130:2181

Creating new collection 'test' using command:
http://localhost:8983/solr/admin/collections?action=CREATE=test=1=1=1=test

{
  "responseHeader":{
"status":0,
"QTime":5006},
  "success":{"10.250.70.120:8983_solr":{
  "responseHeader":{
"status":0,
"QTime":3667},
  "core":"test_shard1_replica1"}}}


So according to doc I should see data_driven_schema_configs being uploaded
to  ZooKeeper under "/configs/test" directory. It seems like doc is
mentioning a literal "configs" directory. BUt I can't find that under my
zookeeper installation.


On Tue, Jul 26, 2016 at 4:18 PM, Erick Erickson 
wrote:

> From the doc you referenced where it outlines the parameters for the
> create command:
>
> -d  : The configuration directory. This defaults to
> data_driven_schema_configs.
>
> You should also review the linked section about what configuration
> directories are all about:
>
> https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference#SolrStartScriptReference-ConfigurationDirectoriesandSolrCloud
>
> Best,
> Erick
>
>
> On Tue, Jul 26, 2016 at 4:11 PM, Nirav Patel 
> wrote:
> > Hi,
> >
> > I have 2 nodes solr cluster with 1 node standalone zookeeper.
> >
> > I tried following from following doc to create collection:
> >
> > bin/solr create -c contacts.
> >
> > According to doc should upload config date into /configs/contacts in
> > ZooKeeper. But I can't find any configs directory under zookeeper.
> >
> >
> https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference#SolrStartScriptReference-Create
> >
> > Where did config files go? I want to modify them using vi and then upload
> > it again. But first I want to locate them on filesystem. According to doc
> > on "Configuration Directories and SolrCloud" it should be under zookeepr
> > but it's not!
> >
> > Thanks
> >
> > --
> >
> >
> > [image: What's New with Xactly] 
> >
> >   [image: LinkedIn]
> >   [image: Twitter]
> >   [image: Facebook]
> >   [image: YouTube]
> > 
>

-- 


[image: What's New with Xactly] 

  [image: LinkedIn] 
  [image: Twitter] 
  [image: Facebook] 
  [image: YouTube] 

Re: SolrCloud create_collection not uploading configs to zookeeper

[Erick Erickson]

>From the doc you referenced where it outlines the parameters for the
create command:

-d  : The configuration directory. This defaults to
data_driven_schema_configs.

You should also review the linked section about what configuration
directories are all about:
https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference#SolrStartScriptReference-ConfigurationDirectoriesandSolrCloud

Best,
Erick


On Tue, Jul 26, 2016 at 4:11 PM, Nirav Patel  wrote:
> Hi,
>
> I have 2 nodes solr cluster with 1 node standalone zookeeper.
>
> I tried following from following doc to create collection:
>
> bin/solr create -c contacts.
>
> According to doc should upload config date into /configs/contacts in
> ZooKeeper. But I can't find any configs directory under zookeeper.
>
> https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference#SolrStartScriptReference-Create
>
> Where did config files go? I want to modify them using vi and then upload
> it again. But first I want to locate them on filesystem. According to doc
> on "Configuration Directories and SolrCloud" it should be under zookeepr
> but it's not!
>
> Thanks
>
> --
>
>
> [image: What's New with Xactly] 
>
>   [image: LinkedIn]
>   [image: Twitter]
>   [image: Facebook]
>   [image: YouTube]
> 

SolrCloud create_collection not uploading configs to zookeeper

[Nirav Patel]

Hi,

I have 2 nodes solr cluster with 1 node standalone zookeeper.

I tried following from following doc to create collection:

bin/solr create -c contacts.

According to doc should upload config date into /configs/contacts in
ZooKeeper. But I can't find any configs directory under zookeeper.

https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference#SolrStartScriptReference-Create

Where did config files go? I want to modify them using vi and then upload
it again. But first I want to locate them on filesystem. According to doc
on "Configuration Directories and SolrCloud" it should be under zookeepr
but it's not!

Thanks

-- 


[image: What's New with Xactly] 

  [image: LinkedIn] 
  [image: Twitter] 
  [image: Facebook] 
  [image: YouTube] 

Re: Solr MapReduce Indexer : go-live option throwing exception

[Erick Erickson]

Can't really deal with the security issues, but...

The resulting indexes created by MRIT are just plain vanilla
Solr/Lucene indexes. All the --go-live step does is issue a
MERGEINDEXES command from the core where they live to the directory
MRIT leaves them in, you might get some joy there, see:
https://cwiki.apache.org/confluence/display/solr/CoreAdmin+API#CoreAdminAPI-MERGEINDEXES

Or you can copy them around by hand and start Solr.

You have to be _really_ sure that you get the right index for each
replica though, if you get the index intended for a replica on shard1
on a replica for shard2 it's A Bad Thing.

Best,
Erick

On Tue, Jul 26, 2016 at 10:52 AM, Archana Satheesh Kumar
 wrote:
> Hi,
>
>
> I was trying to use the Mapreduce Indexer tool from cloudera, to index my 
> data in Hive table using Solr.
>
>
> hadoop jar /path/to/lib/solr/contrib/mr/search-mr-*-job.jar  
> org.apache.solr.hadoop.MapReduceIndexerTool -Djute.maxbuffer= size>--morphline-file /path/to/morphlines.conf --output-dir 
> hdfs://path/to/output/dir --reducers -1 --mappers -1 --verbose --go-live 
> --zk-host :2181/solr --shards 2 --collection  name> hdfs://location/of/hive/table
>
> My MR job runs successfully and I am able to view _SUCCESS flag in the 
> specified output loc
>
>  hadoop fs -ls /path/to/output/results
> Found 2 items
> -rwxrwx--x+  3 hive hive  0 2016-07-26 11:35 
> /path/to/output/results/_SUCCESS
> drwxrwx--x+  - hive hive  0 2016-07-26 11:20 
> /path/to/output/results/part-0
>
> But my go-live option is not working.
>
> Exception:
> java.util.concurrent.ExecutionException: 
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: 
> Expected mime type application/octet-stream but got text/html
>
> I also tried using jaas-client.conf
>
> Client {
>  com.sun.security.auth.module.Krb5LoginModule required
>  useKeyTab=false
>  useTicketCache=true
>  principal="";
>  };
>
> So, before executing the Mapreduce job, HADOOP_OPTS was set to the 
> jass-client.conf
> export 
> HADOOP_OPTS="-Djava.security.auth.login.config=/path/to/jaas-client.conf"
>
>
> 1. What could be the issue?
> 2. Am I missing something?
> 3. Since I have my data indexed locally, is there a way to perform the 
> go-live option separately?
>
> Thanks in advance
>
> Archana
>
>
>
>
>

Re: Example of posting to /stream in SolrJ?

[Joel Bernstein]

I posted this also to another thread, but I'll cross post to this ticket:

Take a look at org.apache.solr.client.solrj.io.sql.
StatementImpl.constructStream()

This uses a SolrStream to connect to the /sql handler. You can use the same
approach to send a request to the /stream handler just by changing the
parameters. Then you can open and read the SolrStream.





Joel Bernstein
http://joelsolr.blogspot.com/

On Tue, Jul 26, 2016 at 3:58 PM, Timothy Potter 
wrote:

> Does anyone have an example of just POST'ing a streaming expression to
> the /stream handler from SolrJ client code? i.e. I don't want to parse
> and execute the streaming expression on the client side, rather, I
> want to post the expression to the server side.
>
> Currently, my client code is a big copy and paste of the /stream
> request handler, but I'd rather not do that. Specifically, I wasn't
> able to figure out how to parse the tuple
> stream coming back using SolrJ code if I just post the expression to
> /stream.
>
> Thanks.
>

Re: Can't load schema managed-schema: unknown field 'id'

[Erick Erickson]

Hmm, this is odd. You should not have had to restart Zookeeper, are
you 100% sure
you looked in the same place you downloaded to?

Of course you would have to reload the collection to get them to "take".

BTW, in Solr 5.4+ there's upconfig/downconfig from the bin/solr
script, it was put
there to try to keep from having to go to zkcli.sh with it's own
distinct syntax. They
both work, whichever you like most. And in 6.2+ it's been extended to
upload/download
arbitrary files/directories.

Best,
Erick

On Tue, Jul 26, 2016 at 1:47 PM, Michael Joyner  wrote:
> Finally got it to straighten out.
>
> So I have two collections, my test collection and my production collection.
>
> I "fat fingered" the test collection and both collections were complaining
> about the missing "id" field.
>
> I downloaded the config from both collections and it was showing the id
> field in place (?)
>
> I restarted the zookeeper I was talking to and then redownloaded the configs
> and now it was gone.
>
> Added it (and _version_) back, re-upped, restarted the solr node local to
> that zookeeper and it stopped complaining about the missing id field.
>
> Now waiting on the node I restarted to show "green".
>
> -MIke
>
>
>
> On 07/26/2016 04:32 PM, Alexandre Drouin wrote:
>>
>> @Michael - there are GUI available for ZooKeeper:
>> http://stackoverflow.com/questions/24551835/available-gui-for-zookeeper
>> I used the Eclipse plugin before and while it is a bit clunky it gets the
>> job done.
>>
>>
>> Alexandre Drouin
>>
>>
>> -Original Message-
>> From: John Bickerstaff [mailto:j...@johnbickerstaff.com]
>> Sent: July 26, 2016 4:21 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Can't load schema managed-schema: unknown field 'id'
>> Importance: High
>>
>> @Michael - somewhere there should be a "conf" directory for your SOLR
>> instance.  For my Dev efforts, I moved it to a different directory and I
>> forget where it was, originally -- but if you search for solrconfig.xml or
>> schema.xml, you should find it.
>>
>> It could be on your servers (or on only one of them) or, if someone has
>> done a really good job, it's in source control somewhere...
>>
>> On Tue, Jul 26, 2016 at 2:17 PM, John Bickerstaff
>> 
>> wrote:
>>
>>>  >> type="string"
>>> indexed="true" stored="true" required="true" multiValued="false" />
>>>
>>> and further on in the file...
>>>
>>>  id
>>>
>>>
>>> On Tue, Jul 26, 2016 at 2:17 PM, John Bickerstaff <
>>> j...@johnbickerstaff.com> wrote:
>>>
 I don't see a managed schema file.  As far as I understand it, id is
 set as a "uniqueKey" in the schema.xml file...

 On Tue, Jul 26, 2016 at 2:11 PM, Michael Joyner 
 wrote:

> ok, I think I need to do a manual edit on the managed-schema file
> but I get "NoNode" for /managed-schema when trying to use the zkcli.sh
> file?
>
>
> How can I get to this file and edit it?
>
>
> On 07/26/2016 03:05 PM, Alexandre Drouin wrote:
>
>> Hello,
>>
>> You may have a uniqueKey that points to a field that do not exists
>> anymore.  You can try adding an "id" field using Solr's UI or the
>> schema API since you are using the managed-schema.
>>
>>
>> Alexandre Drouin
>>
>> -Original Message-
>> From: Michael Joyner [mailto:mich...@newsrx.com]
>> Sent: July 26, 2016 2:34 PM
>> To: solr-user@lucene.apache.org
>> Subject: Can't load schema managed-schema: unknown field 'id'
>>
>> |Help!|
>>
>> |
>> |
>>
>> |What is the best way to recover from: |
>>
>> Can't load schema managed-schema: unknown field 'id'
>> |I was managing the schema on a test collection, fat fingered it,
>> |but
>> now
>> I find out the schema ops seem to altering all collections on the
>> core?
>> SolrCloud 5.5.1 |||
>>
>> |
>> -Mike|||
>>
>
>

Re: Can't load schema managed-schema: unknown field 'id'

[Michael Joyner]

Finally got it to straighten out.

So I have two collections, my test collection and my production collection.

I "fat fingered" the test collection and both collections were 
complaining about the missing "id" field.


I downloaded the config from both collections and it was showing the id 
field in place (?)


I restarted the zookeeper I was talking to and then redownloaded the 
configs and now it was gone.


Added it (and _version_) back, re-upped, restarted the solr node local 
to that zookeeper and it stopped complaining about the missing id field.


Now waiting on the node I restarted to show "green".

-MIke


On 07/26/2016 04:32 PM, Alexandre Drouin wrote:

@Michael - there are GUI available for ZooKeeper: 
http://stackoverflow.com/questions/24551835/available-gui-for-zookeeper
I used the Eclipse plugin before and while it is a bit clunky it gets the job 
done.


Alexandre Drouin


-Original Message-
From: John Bickerstaff [mailto:j...@johnbickerstaff.com]
Sent: July 26, 2016 4:21 PM
To: solr-user@lucene.apache.org
Subject: Re: Can't load schema managed-schema: unknown field 'id'
Importance: High

@Michael - somewhere there should be a "conf" directory for your SOLR instance. 
 For my Dev efforts, I moved it to a different directory and I forget where it was, 
originally -- but if you search for solrconfig.xml or schema.xml, you should find it.

It could be on your servers (or on only one of them) or, if someone has done a 
really good job, it's in source control somewhere...

On Tue, Jul 26, 2016 at 2:17 PM, John Bickerstaff 
wrote:


 

and further on in the file...

 id


On Tue, Jul 26, 2016 at 2:17 PM, John Bickerstaff <
j...@johnbickerstaff.com> wrote:


I don't see a managed schema file.  As far as I understand it, id is
set as a "uniqueKey" in the schema.xml file...

On Tue, Jul 26, 2016 at 2:11 PM, Michael Joyner 
wrote:


ok, I think I need to do a manual edit on the managed-schema file
but I get "NoNode" for /managed-schema when trying to use the zkcli.sh file?


How can I get to this file and edit it?


On 07/26/2016 03:05 PM, Alexandre Drouin wrote:


Hello,

You may have a uniqueKey that points to a field that do not exists
anymore.  You can try adding an "id" field using Solr's UI or the
schema API since you are using the managed-schema.


Alexandre Drouin

-Original Message-
From: Michael Joyner [mailto:mich...@newsrx.com]
Sent: July 26, 2016 2:34 PM
To: solr-user@lucene.apache.org
Subject: Can't load schema managed-schema: unknown field 'id'

|Help!|

|
|

|What is the best way to recover from: |

Can't load schema managed-schema: unknown field 'id'
|I was managing the schema on a test collection, fat fingered it,
|but
now
I find out the schema ops seem to altering all collections on the core?
SolrCloud 5.5.1 |||

|
-Mike|||

Re: Can't load schema managed-schema: unknown field 'id'

[Michael Joyner]

@John

I am using a managed schema with zookeeper/solrcloud.


On 07/26/2016 04:21 PM, John Bickerstaff wrote:

@Michael - somewhere there should be a "conf" directory for your SOLR
instance.  For my Dev efforts, I moved it to a different directory and I
forget where it was, originally -- but if you search for solrconfig.xml or
schema.xml, you should find it.

It could be on your servers (or on only one of them) or, if someone has
done a really good job, it's in source control somewhere...

On Tue, Jul 26, 2016 at 2:17 PM, John Bickerstaff 
wrote:


 

and further on in the file...


id


On Tue, Jul 26, 2016 at 2:17 PM, John Bickerstaff <
j...@johnbickerstaff.com> wrote:


I don't see a managed schema file.  As far as I understand it, id is set
as a "uniqueKey" in the schema.xml file...

On Tue, Jul 26, 2016 at 2:11 PM, Michael Joyner 
wrote:


ok, I think I need to do a manual edit on the managed-schema file but I
get "NoNode" for /managed-schema when trying to use the zkcli.sh file?


How can I get to this file and edit it?


On 07/26/2016 03:05 PM, Alexandre Drouin wrote:


Hello,

You may have a uniqueKey that points to a field that do not exists
anymore.  You can try adding an "id" field using Solr's UI or the schema
API since you are using the managed-schema.


Alexandre Drouin

-Original Message-
From: Michael Joyner [mailto:mich...@newsrx.com]
Sent: July 26, 2016 2:34 PM
To: solr-user@lucene.apache.org
Subject: Can't load schema managed-schema: unknown field 'id'

|Help!|

|
|

|What is the best way to recover from: |

Can't load schema managed-schema: unknown field 'id'
|I was managing the schema on a test collection, fat fingered it, but
now
I find out the schema ops seem to altering all collections on the core?
SolrCloud 5.5.1 |||

|
-Mike|||

Re: Can't load schema managed-schema: unknown field 'id'

[Michael Joyner]

ok...

I downloaded the config for both of my collections and the downloaded 
managed-schema file shows "id" as defined? But the online view in the UI 
shows it as not defined?


I've tried re-upping the config and nothing changes.

-Mike



On 07/26/2016 04:11 PM, John Bickerstaff wrote:

@Michael - if you're on Linux and decide to take Alexandre's advice, I can
possibly save you some time.  I wrestled with getting the data in and out
of zookeeper a while ago...

sudo /opt/solr/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -confdir
/home/john/conf/ -confname collectionName -z 192.168.56.5/solr5_4

Explanation:

sudo /opt/solr/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig = run
the code that sends config files (whatever files you modify)over to
Zookeeper

-confdir /home/john/conf/ = find the configuration directory here

-confname collectionName  = apply the configuration to this collection name

-z 192.168.56.5/solr5_4 - find Zookeeper here - and use the solr5_4
"chroot" which already exists in Zookeeper  (If you don't have chroot in
Zookeeper, ignore and don't use the slash)





On Tue, Jul 26, 2016 at 1:55 PM, Alexandre Drouin <
alexandre.dro...@orckestra.com> wrote:


Other than deleting the collection, I think you'll have to edit the
manage-schema file manually.

Since you are using SolrCloud you will need to use Solr's zkcli (
https://cwiki.apache.org/confluence/display/solr/Command+Line+Utilities)
utility to download and upload the file from ZooKeeper.


Alexandre Drouin


-Original Message-
From: Michael Joyner [mailto:mich...@newsrx.com]
Sent: July 26, 2016 3:48 PM
To: solr-user@lucene.apache.org
Subject: Re: Can't load schema managed-schema: unknown field 'id'
Importance: High

Same error via the UI:

Can't load schema managed-schema: unknown field 'id'


On 07/26/2016 03:05 PM, Alexandre Drouin wrote:

Hello,

You may have a uniqueKey that points to a field that do not exists

anymore.  You can try adding an "id" field using Solr's UI or the schema
API since you are using the managed-schema.


Alexandre Drouin

-Original Message-
From: Michael Joyner [mailto:mich...@newsrx.com]
Sent: July 26, 2016 2:34 PM
To: solr-user@lucene.apache.org
Subject: Can't load schema managed-schema: unknown field 'id'

|Help!|

|
|

|What is the best way to recover from: |

Can't load schema managed-schema: unknown field 'id'
|I was managing the schema on a test collection, fat fingered it, but now
I find out the schema ops seem to altering all collections on the core?
SolrCloud 5.5.1 |||

|
-Mike|||

RE: Can't load schema managed-schema: unknown field 'id'

[Alexandre Drouin]

@Michael - there are GUI available for ZooKeeper: 
http://stackoverflow.com/questions/24551835/available-gui-for-zookeeper
I used the Eclipse plugin before and while it is a bit clunky it gets the job 
done.


Alexandre Drouin


-Original Message-
From: John Bickerstaff [mailto:j...@johnbickerstaff.com] 
Sent: July 26, 2016 4:21 PM
To: solr-user@lucene.apache.org
Subject: Re: Can't load schema managed-schema: unknown field 'id'
Importance: High

@Michael - somewhere there should be a "conf" directory for your SOLR instance. 
 For my Dev efforts, I moved it to a different directory and I forget where it 
was, originally -- but if you search for solrconfig.xml or schema.xml, you 
should find it.

It could be on your servers (or on only one of them) or, if someone has done a 
really good job, it's in source control somewhere...

On Tue, Jul 26, 2016 at 2:17 PM, John Bickerstaff 
wrote:

>   type="string"
> indexed="true" stored="true" required="true" multiValued="false" />
>
> and further on in the file...
>
>  id
>
>
> On Tue, Jul 26, 2016 at 2:17 PM, John Bickerstaff < 
> j...@johnbickerstaff.com> wrote:
>
>> I don't see a managed schema file.  As far as I understand it, id is 
>> set as a "uniqueKey" in the schema.xml file...
>>
>> On Tue, Jul 26, 2016 at 2:11 PM, Michael Joyner 
>> wrote:
>>
>>> ok, I think I need to do a manual edit on the managed-schema file 
>>> but I get "NoNode" for /managed-schema when trying to use the zkcli.sh file?
>>>
>>>
>>> How can I get to this file and edit it?
>>>
>>>
>>> On 07/26/2016 03:05 PM, Alexandre Drouin wrote:
>>>
 Hello,

 You may have a uniqueKey that points to a field that do not exists 
 anymore.  You can try adding an "id" field using Solr's UI or the 
 schema API since you are using the managed-schema.


 Alexandre Drouin

 -Original Message-
 From: Michael Joyner [mailto:mich...@newsrx.com]
 Sent: July 26, 2016 2:34 PM
 To: solr-user@lucene.apache.org
 Subject: Can't load schema managed-schema: unknown field 'id'

 |Help!|

 |
 |

 |What is the best way to recover from: |

 Can't load schema managed-schema: unknown field 'id'
 |I was managing the schema on a test collection, fat fingered it, 
 |but
 now
 I find out the schema ops seem to altering all collections on the core?
 SolrCloud 5.5.1 |||

 |
 -Mike|||

>>>
>>>
>>
>

Re: Can't load schema managed-schema: unknown field 'id'

[John Bickerstaff]

@Michael - somewhere there should be a "conf" directory for your SOLR
instance.  For my Dev efforts, I moved it to a different directory and I
forget where it was, originally -- but if you search for solrconfig.xml or
schema.xml, you should find it.

It could be on your servers (or on only one of them) or, if someone has
done a really good job, it's in source control somewhere...

On Tue, Jul 26, 2016 at 2:17 PM, John Bickerstaff 
wrote:

>   indexed="true" stored="true" required="true" multiValued="false" />
>
> and further on in the file...
>
> 
> id
>
>
> On Tue, Jul 26, 2016 at 2:17 PM, John Bickerstaff <
> j...@johnbickerstaff.com> wrote:
>
>> I don't see a managed schema file.  As far as I understand it, id is set
>> as a "uniqueKey" in the schema.xml file...
>>
>> On Tue, Jul 26, 2016 at 2:11 PM, Michael Joyner 
>> wrote:
>>
>>> ok, I think I need to do a manual edit on the managed-schema file but I
>>> get "NoNode" for /managed-schema when trying to use the zkcli.sh file?
>>>
>>>
>>> How can I get to this file and edit it?
>>>
>>>
>>> On 07/26/2016 03:05 PM, Alexandre Drouin wrote:
>>>
 Hello,

 You may have a uniqueKey that points to a field that do not exists
 anymore.  You can try adding an "id" field using Solr's UI or the schema
 API since you are using the managed-schema.


 Alexandre Drouin

 -Original Message-
 From: Michael Joyner [mailto:mich...@newsrx.com]
 Sent: July 26, 2016 2:34 PM
 To: solr-user@lucene.apache.org
 Subject: Can't load schema managed-schema: unknown field 'id'

 |Help!|

 |
 |

 |What is the best way to recover from: |

 Can't load schema managed-schema: unknown field 'id'
 |I was managing the schema on a test collection, fat fingered it, but
 now
 I find out the schema ops seem to altering all collections on the core?
 SolrCloud 5.5.1 |||

 |
 -Mike|||

>>>
>>>
>>
>

Re: Can't load schema managed-schema: unknown field 'id'

[John Bickerstaff]

 

and further on in the file...

 <
uniqueKey>id


On Tue, Jul 26, 2016 at 2:17 PM, John Bickerstaff 
wrote:

> I don't see a managed schema file.  As far as I understand it, id is set
> as a "uniqueKey" in the schema.xml file...
>
> On Tue, Jul 26, 2016 at 2:11 PM, Michael Joyner 
> wrote:
>
>> ok, I think I need to do a manual edit on the managed-schema file but I
>> get "NoNode" for /managed-schema when trying to use the zkcli.sh file?
>>
>>
>> How can I get to this file and edit it?
>>
>>
>> On 07/26/2016 03:05 PM, Alexandre Drouin wrote:
>>
>>> Hello,
>>>
>>> You may have a uniqueKey that points to a field that do not exists
>>> anymore.  You can try adding an "id" field using Solr's UI or the schema
>>> API since you are using the managed-schema.
>>>
>>>
>>> Alexandre Drouin
>>>
>>> -Original Message-
>>> From: Michael Joyner [mailto:mich...@newsrx.com]
>>> Sent: July 26, 2016 2:34 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Can't load schema managed-schema: unknown field 'id'
>>>
>>> |Help!|
>>>
>>> |
>>> |
>>>
>>> |What is the best way to recover from: |
>>>
>>> Can't load schema managed-schema: unknown field 'id'
>>> |I was managing the schema on a test collection, fat fingered it, but now
>>> I find out the schema ops seem to altering all collections on the core?
>>> SolrCloud 5.5.1 |||
>>>
>>> |
>>> -Mike|||
>>>
>>
>>
>

Re: Can't load schema managed-schema: unknown field 'id'

[John Bickerstaff]

I don't see a managed schema file.  As far as I understand it, id is set as
a "uniqueKey" in the schema.xml file...

On Tue, Jul 26, 2016 at 2:11 PM, Michael Joyner  wrote:

> ok, I think I need to do a manual edit on the managed-schema file but I
> get "NoNode" for /managed-schema when trying to use the zkcli.sh file?
>
>
> How can I get to this file and edit it?
>
>
> On 07/26/2016 03:05 PM, Alexandre Drouin wrote:
>
>> Hello,
>>
>> You may have a uniqueKey that points to a field that do not exists
>> anymore.  You can try adding an "id" field using Solr's UI or the schema
>> API since you are using the managed-schema.
>>
>>
>> Alexandre Drouin
>>
>> -Original Message-
>> From: Michael Joyner [mailto:mich...@newsrx.com]
>> Sent: July 26, 2016 2:34 PM
>> To: solr-user@lucene.apache.org
>> Subject: Can't load schema managed-schema: unknown field 'id'
>>
>> |Help!|
>>
>> |
>> |
>>
>> |What is the best way to recover from: |
>>
>> Can't load schema managed-schema: unknown field 'id'
>> |I was managing the schema on a test collection, fat fingered it, but now
>> I find out the schema ops seem to altering all collections on the core?
>> SolrCloud 5.5.1 |||
>>
>> |
>> -Mike|||
>>
>
>

Re: Can't load schema managed-schema: unknown field 'id'

[John Bickerstaff]

@Michael - if you're on Linux and decide to take Alexandre's advice, I can
possibly save you some time.  I wrestled with getting the data in and out
of zookeeper a while ago...

sudo /opt/solr/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -confdir
/home/john/conf/ -confname collectionName -z 192.168.56.5/solr5_4

Explanation:

sudo /opt/solr/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig = run
the code that sends config files (whatever files you modify)over to
Zookeeper

-confdir /home/john/conf/ = find the configuration directory here

-confname collectionName  = apply the configuration to this collection name

-z 192.168.56.5/solr5_4 - find Zookeeper here - and use the solr5_4
"chroot" which already exists in Zookeeper  (If you don't have chroot in
Zookeeper, ignore and don't use the slash)





On Tue, Jul 26, 2016 at 1:55 PM, Alexandre Drouin <
alexandre.dro...@orckestra.com> wrote:

> Other than deleting the collection, I think you'll have to edit the
> manage-schema file manually.
>
> Since you are using SolrCloud you will need to use Solr's zkcli (
> https://cwiki.apache.org/confluence/display/solr/Command+Line+Utilities)
> utility to download and upload the file from ZooKeeper.
>
>
> Alexandre Drouin
>
>
> -Original Message-
> From: Michael Joyner [mailto:mich...@newsrx.com]
> Sent: July 26, 2016 3:48 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Can't load schema managed-schema: unknown field 'id'
> Importance: High
>
> Same error via the UI:
>
> Can't load schema managed-schema: unknown field 'id'
>
>
> On 07/26/2016 03:05 PM, Alexandre Drouin wrote:
> > Hello,
> >
> > You may have a uniqueKey that points to a field that do not exists
> anymore.  You can try adding an "id" field using Solr's UI or the schema
> API since you are using the managed-schema.
> >
> >
> > Alexandre Drouin
> >
> > -Original Message-
> > From: Michael Joyner [mailto:mich...@newsrx.com]
> > Sent: July 26, 2016 2:34 PM
> > To: solr-user@lucene.apache.org
> > Subject: Can't load schema managed-schema: unknown field 'id'
> >
> > |Help!|
> >
> > |
> > |
> >
> > |What is the best way to recover from: |
> >
> > Can't load schema managed-schema: unknown field 'id'
> > |I was managing the schema on a test collection, fat fingered it, but now
> > I find out the schema ops seem to altering all collections on the core?
> > SolrCloud 5.5.1 |||
> >
> > |
> > -Mike|||
>
>

Re: Can't load schema managed-schema: unknown field 'id'

[Michael Joyner]

ok, I think I need to do a manual edit on the managed-schema file but I 
get "NoNode" for /managed-schema when trying to use the zkcli.sh file?



How can I get to this file and edit it?


On 07/26/2016 03:05 PM, Alexandre Drouin wrote:

Hello,

You may have a uniqueKey that points to a field that do not exists anymore.  You can try 
adding an "id" field using Solr's UI or the schema API since you are using the 
managed-schema.


Alexandre Drouin

-Original Message-
From: Michael Joyner [mailto:mich...@newsrx.com]
Sent: July 26, 2016 2:34 PM
To: solr-user@lucene.apache.org
Subject: Can't load schema managed-schema: unknown field 'id'

|Help!|

|
|

|What is the best way to recover from: |

Can't load schema managed-schema: unknown field 'id'
|I was managing the schema on a test collection, fat fingered it, but now
I find out the schema ops seem to altering all collections on the core?
SolrCloud 5.5.1 |||

|
-Mike|||

Re: Streaming expression - workers is zero somehow?

[Joel Bernstein]

Take a look at StatementImpl.constructStream()

This uses a SolrStream to connect to the /sql handler. You can use the same
approach to send a request to the /stream handler just by changing the
parameters. Then you can open and read the SolrStream.

We don't yet have a load balancing SolrStream.

Joel Bernstein
http://joelsolr.blogspot.com/

On Tue, Jul 26, 2016 at 11:27 AM, Timothy Potter 
wrote:

> Ok, makes sense now, thanks Joel. We should probably add some earlier
> error checking vs. letting the code get all the way into the
> HashQParser.
>
> So this raises a separate question that I haven't been able to figure
> out, namely, do we have an example of just POST'ing the expression to
> the /stream handler from SolrJ client code? i.e. I don't want to parse
> and execute the streaming expression on the client side, rather, I
> want to post it to the server side. Currently, my client code is a big
> copy and paste of the /stream request handler (with 1 obvious omission
> ;-) Specifically, I wasn't able to figure out how to parse the tuple
> stream coming back using SolrJ code if I just post the expression to
> /stream.
>
> Tim
>
> On Tue, Jul 26, 2016 at 8:54 AM, Joel Bernstein 
> wrote:
> > The difference would be if you are compiling and running the expression
> in
> > a java class or sending it to the /stream handler to be compiled.
> >
> > If you're compiling it and running it locally you could get this error
> > because the StreamContext would not have the numWorkers variable set.
> >
> > The /stream handler always sets the numWorkers variable, so in theory you
> > would never see this error if the /stream handler is executing the
> > expression.
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> > On Tue, Jul 26, 2016 at 10:44 AM, Timothy Potter 
> > wrote:
> >
> >> it's from a unit test, but not sure why that matters? If I wrap the
> >> expression in a parallel expression with explicit workers=1, then it
> >> works
> >>
> >> On Thu, Jul 21, 2016 at 11:13 AM, Joel Bernstein 
> >> wrote:
> >> > Are you getting this error from a test case you've setup or from a
> manual
> >> > call to the /stream handler?
> >> >
> >> > Joel Bernstein
> >> > http://joelsolr.blogspot.com/
> >> >
> >> > On Thu, Jul 21, 2016 at 12:28 PM, Timothy Potter <
> thelabd...@gmail.com>
> >> > wrote:
> >> >
> >> >> I'm working with 6.1.0 release and I have a single SolrCloud instance
> >> >> with 1 shard / 1 replica. Somehow I'm triggering this, which from
> what
> >> >> I can see, means workers == 0, but how? Shouldn't workers default to
> 1
> >> >>
> >> >> I should mention that my streaming expression doesn't include any
> >> >> workers, i.e. it is simply:
> >> >>
> >> >> val hashJoinExpr =
> >> >>   s"""
> >> >>  | hashJoin(
> >> >>  |search(${ratingsCollection},
> >> >>  |   q="*:*",
> >> >>  |   fl="movie_id,user_id,rating",
> >> >>  |   sort="movie_id asc",
> >> >>  |   qt="/export",
> >> >>  |   partitionKeys="movie_id"),
> >> >>  |hashed=search(${moviesCollection},
> >> >>  |  q="*:*",
> >> >>  |  fl="movie_id,title",
> >> >>  |  sort="movie_id asc",
> >> >>  |  qt="/export",
> >> >>  |  partitionKeys="movie_id"),
> >> >>  |on="movie_id"
> >> >>  |  )
> >> >>""".stripMargin
> >> >>
> >> >>
> >> >>
> >> >> 2016-07-21 10:08:44,596 [qtp2125832297-1073] ERROR RequestHandlerBase
> >> >> - java.io.IOException: java.lang.RuntimeException:
> >> >> java.lang.ArithmeticException: / by zero
> >> >> at
> >> >>
> >>
> org.apache.solr.search.HashQParserPlugin$HashQuery.createWeight(HashQParserPlugin.java:130)
> >> >> at
> >> >>
> >>
> org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:752)
> >> >> at
> >> >>
> >>
> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:735)
> >> >> at
> >> >> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
> >> >> at
> >> >>
> >>
> org.apache.solr.search.DocSetUtil.createDocSetGeneric(DocSetUtil.java:102)
> >> >> at
> >> org.apache.solr.search.DocSetUtil.createDocSet(DocSetUtil.java:91)
> >> >> at
> >> >>
> >>
> org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:1386)
> >> >> at
> >> >>
> >>
> org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:1064)
> >> >> at
> >> >>
> >>
> org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSearcher.java:1234)
> >> >> at
> >> >>
> >>
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1751)
> >> >> at
> >> >>
> >>
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1627)
> >> >> at
> >> >>
> >>
> 

Example of posting to /stream in SolrJ?

[Timothy Potter]

Does anyone have an example of just POST'ing a streaming expression to
the /stream handler from SolrJ client code? i.e. I don't want to parse
and execute the streaming expression on the client side, rather, I
want to post the expression to the server side.

Currently, my client code is a big copy and paste of the /stream
request handler, but I'd rather not do that. Specifically, I wasn't
able to figure out how to parse the tuple
stream coming back using SolrJ code if I just post the expression to /stream.

Thanks.

RE: Can't load schema managed-schema: unknown field 'id'

[Alexandre Drouin]

Other than deleting the collection, I think you'll have to edit the 
manage-schema file manually.

Since you are using SolrCloud you will need to use Solr's zkcli 
(https://cwiki.apache.org/confluence/display/solr/Command+Line+Utilities) 
utility to download and upload the file from ZooKeeper.


Alexandre Drouin


-Original Message-
From: Michael Joyner [mailto:mich...@newsrx.com] 
Sent: July 26, 2016 3:48 PM
To: solr-user@lucene.apache.org
Subject: Re: Can't load schema managed-schema: unknown field 'id'
Importance: High

Same error via the UI:

Can't load schema managed-schema: unknown field 'id'


On 07/26/2016 03:05 PM, Alexandre Drouin wrote:
> Hello,
>
> You may have a uniqueKey that points to a field that do not exists anymore.  
> You can try adding an "id" field using Solr's UI or the schema API since you 
> are using the managed-schema.
>
>
> Alexandre Drouin
>
> -Original Message-
> From: Michael Joyner [mailto:mich...@newsrx.com]
> Sent: July 26, 2016 2:34 PM
> To: solr-user@lucene.apache.org
> Subject: Can't load schema managed-schema: unknown field 'id'
>
> |Help!|
>
> |
> |
>
> |What is the best way to recover from: |
>
> Can't load schema managed-schema: unknown field 'id'
> |I was managing the schema on a test collection, fat fingered it, but now
> I find out the schema ops seem to altering all collections on the core?
> SolrCloud 5.5.1 |||
>
> |
> -Mike|||

Re: Can't load schema managed-schema: unknown field 'id'

[Michael Joyner]

Same error via the UI:

Can't load schema managed-schema: unknown field 'id'


On 07/26/2016 03:05 PM, Alexandre Drouin wrote:

Hello,

You may have a uniqueKey that points to a field that do not exists anymore.  You can try 
adding an "id" field using Solr's UI or the schema API since you are using the 
managed-schema.


Alexandre Drouin

-Original Message-
From: Michael Joyner [mailto:mich...@newsrx.com]
Sent: July 26, 2016 2:34 PM
To: solr-user@lucene.apache.org
Subject: Can't load schema managed-schema: unknown field 'id'

|Help!|

|
|

|What is the best way to recover from: |

Can't load schema managed-schema: unknown field 'id'
|I was managing the schema on a test collection, fat fingered it, but now
I find out the schema ops seem to altering all collections on the core?
SolrCloud 5.5.1 |||

|
-Mike|||

Re: Can't load schema managed-schema: unknown field 'id'

[Michael Joyner]

The schema API is failing with the the unknown field "id" error.

Where in the UI could I try adding this field back at?


On 07/26/2016 03:05 PM, Alexandre Drouin wrote:

Hello,

You may have a uniqueKey that points to a field that do not exists anymore.  You can try 
adding an "id" field using Solr's UI or the schema API since you are using the 
managed-schema.


Alexandre Drouin

-Original Message-
From: Michael Joyner [mailto:mich...@newsrx.com]
Sent: July 26, 2016 2:34 PM
To: solr-user@lucene.apache.org
Subject: Can't load schema managed-schema: unknown field 'id'

|Help!|

|
|

|What is the best way to recover from: |

Can't load schema managed-schema: unknown field 'id'
|I was managing the schema on a test collection, fat fingered it, but now
I find out the schema ops seem to altering all collections on the core?
SolrCloud 5.5.1 |||

|
-Mike|||

RE: Can't load schema managed-schema: unknown field 'id'

[Alexandre Drouin]

Hello,

You may have a uniqueKey that points to a field that do not exists anymore.  
You can try adding an "id" field using Solr's UI or the schema API since you 
are using the managed-schema.


Alexandre Drouin

-Original Message-
From: Michael Joyner [mailto:mich...@newsrx.com] 
Sent: July 26, 2016 2:34 PM
To: solr-user@lucene.apache.org
Subject: Can't load schema managed-schema: unknown field 'id'

|Help!|

|
|

|What is the best way to recover from: |

Can't load schema managed-schema: unknown field 'id'
|I was managing the schema on a test collection, fat fingered it, but now 
I find out the schema ops seem to altering all collections on the core? 
SolrCloud 5.5.1 |||

|
-Mike|||

Re: Search sort depth limited to 4?

[tedsolr]

So I found the limit in the Ref Doc p. 394, under the /export request
handler:

"Up to four sort fields can be specified per request, with the 'asc' or
'desc' properties"

Yikes I'm in trouble. Does anyone know if this can be circumvented? Can I
write a custom handler that could handle up to 20? Oh boy.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Search-sort-depth-limited-to-4-tp4289049p4289053.html
Sent from the Solr - User mailing list archive at Nabble.com.

How to know if a core was reloaded without errors

[Alexandre Drouin]

Hi,

I am using the Collection API to reload a core and I was wondering if there is 
a way to know if the core was reloaded without errors.  

For my testing, I added a known error (an invalid field in a request handler) 
in my configuration and I use the url 
"solr/admin/collections?action=RELOAD=MyCollection=json" to reload the 
collection.  

This url returns the following JSON: 
{"responseHeader":{"status":0,"QTime":128},"success":{"localhost:8443_solr":{"responseHeader":{"status":0,"QTime":92.
 And according to the documentation "status=0" means there was no errors with 
the request.  However if I look at the logs in Solr UI I can see the error I 
created (org.apache.solr.common.SolrException: undefined field XYZ).

I have the following questions:

1) Is it possible to know if the reload of a core was completed without errors?
2) Does Solr have an API to download the logs?  If #1 is not possible I could 
hack something if I have access to Solr's logs using an API.

I am using Solr 6.0.1 with ZooKeeper.

Thanks,
Alexandre Drouin

Search sort depth limited to 4?

[tedsolr]

Hi, I'm trying to group search results by fields using the streaming API. I
don't see a sort limit mentioned in the Solr Ref Doc, but when I use 4
fields I get results and when I use 5 or more I get an exception:

java.util.concurrent.ExecutionException: java.io.IOException:
JSONTupleStream: expected OBJECT_START but got STRING

I've got to be able to group docs by an unlimited number of like fields
(well, at least up to 20), which requires sorting the stream. Is there a
limit to the number of sort params in a single search?

thanks!
Ted
v5.2.1



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Search-sort-depth-limited-to-4-tp4289049.html
Sent from the Solr - User mailing list archive at Nabble.com.

Can't load schema managed-schema: unknown field 'id'

[Michael Joyner]

|Help!|

|
|

|What is the best way to recover from: |

Can't load schema managed-schema: unknown field 'id'
|I was managing the schema on a test collection, fat fingered it, but now 
I find out the schema ops seem to altering all collections on the core? 
SolrCloud 5.5.1 |||


|
-Mike|||

Solr is locked after killed - hdfs

[Chaushu, Shani]

Hi,
I have issue that I saw it's a bug that really didn't resolved:
https://issues.apache.org/jira/browse/SOLR-8335

I worked with solr 6.1, on HDFS but I see it's also exists in solr 5.X
I started my solr, and because of out of memory exception, it was killed.
Now when I start it again, I have errors of

org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: 
Index dir 'hdfs://XXX/' of core yyy  is already locked. The most likely cause 
is another Solr server (or another solr core in this server) also configured to 
use this directory; other possible causes may be specific to lockType: hdfs

I saw in the index hdfs files, that the write.lock is exists even after killed.
I have multiple servers and multiple collection, and a lot of processes that 
work on solr, and when it killed I want it to be up without errors.

Is there any way to kill the solr and free the locks? Or any other kind of lock 
that will work with hdfs?

Thanks a lot,
Shani


-
Intel Electronics Ltd.

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Solr MapReduce Indexer : go-live option throwing exception

[Archana Satheesh Kumar]

Hi,


I was trying to use the Mapreduce Indexer tool from cloudera, to index my data 
in Hive table using Solr.


hadoop jar /path/to/lib/solr/contrib/mr/search-mr-*-job.jar  
org.apache.solr.hadoop.MapReduceIndexerTool -Djute.maxbuffer=--morphline-file /path/to/morphlines.conf --output-dir 
hdfs://path/to/output/dir --reducers -1 --mappers -1 --verbose --go-live 
--zk-host :2181/solr --shards 2 --collection  
hdfs://location/of/hive/table

My MR job runs successfully and I am able to view _SUCCESS flag in the 
specified output loc

 hadoop fs -ls /path/to/output/results
Found 2 items
-rwxrwx--x+  3 hive hive  0 2016-07-26 11:35 
/path/to/output/results/_SUCCESS
drwxrwx--x+  - hive hive  0 2016-07-26 11:20 
/path/to/output/results/part-0

But my go-live option is not working.

Exception:
java.util.concurrent.ExecutionException: 
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected 
mime type application/octet-stream but got text/html

I also tried using jaas-client.conf

Client {
 com.sun.security.auth.module.Krb5LoginModule required
 useKeyTab=false
 useTicketCache=true
 principal="";
 };

So, before executing the Mapreduce job, HADOOP_OPTS was set to the 
jass-client.conf
export HADOOP_OPTS="-Djava.security.auth.login.config=/path/to/jaas-client.conf"


1. What could be the issue?
2. Am I missing something?
3. Since I have my data indexed locally, is there a way to perform the go-live 
option separately?

Thanks in advance

Archana

Re: solr.NRTCachingDirectoryFactory

[Erick Erickson]

And, I might add, you should look through your old logs
and see how long it takes to open a searcher. Let's
say Shawn's lower bound is what you see, i.e.
it takes a minute each to execute all the autowarming
in filterCache and queryResultCache... So you're current
latency is _at least_ 2 minutes between the time something
is indexed and it's available for search just for autowarming.

Plus up to another 2 minutes for your soft commit interval
to expire.

So if your business people haven't noticed a 4 minute
latency yet, tell them they don't know what they're talking
about when they insist on the NRT interval being a few
seconds ;).

Best,
Erick

On Tue, Jul 26, 2016 at 7:20 AM, Rallavagu  wrote:
>
>
> On 7/26/16 5:46 AM, Shawn Heisey wrote:
>>
>> On 7/22/2016 10:15 AM, Rallavagu wrote:
>>>
>>> >>  size="5000"
>>>  initialSize="5000"
>>>  autowarmCount="500"/>
>>>
>>
>>> >>  size="2"
>>>  initialSize="2"
>>>  autowarmCount="500"/>
>>
>>
>> As Erick indicated, these settings are incompatible with Near Real Time
>> updates.
>>
>> With those settings, every time you commit and create a new searcher,
>> Solr will execute up to 1000 queries (potentially 500 for each of the
>> caches above) before that new searcher will begin returning new results.
>>
>> I do not know how fast your filter queries execute when they aren't
>> cached... but even if they only take 100 milliseconds each, that's could
>> take up to a minute for filterCache warming.  If each one takes two
>> seconds and there are 500 entries in the cache, then autowarming the
>> filterCache would take nearly 17 minutes. You would also need to wait
>> for the warming queries on queryResultCache.
>>
>> The autowarmCount on my filterCache is 4, and warming that cache *still*
>> sometimes takes ten or more seconds to complete.
>>
>> If you want true NRT, you need to set all your autowarmCount values to
>> zero.  The tradeoff with NRT is that your caches are ineffective
>> immediately after a new searcher is created.
>
> Will look into this and make changes as suggested.
>
>>
>> Looking at the "top" screenshot ... you have plenty of memory to cache
>> the entire index.  Unless your queries are extreme, this is usually
>> enough for good performance.
>>
>> One possible problem is that cache warming is taking far longer than
>> your autoSoftCommit interval, and the server is constantly busy making
>> thousands of warming queries.  Reducing autowarmCount, possibly to zero,
>> *might* fix that. I would expect higher CPU load than what your
>> screenshot shows if this were happening, but it still might be the
>> problem.
>
> Great point. Thanks for the help.
>
>>
>> Thanks,
>> Shawn
>>
>

Re: Streaming expression - workers is zero somehow?

[Timothy Potter]

Ok, makes sense now, thanks Joel. We should probably add some earlier
error checking vs. letting the code get all the way into the
HashQParser.

So this raises a separate question that I haven't been able to figure
out, namely, do we have an example of just POST'ing the expression to
the /stream handler from SolrJ client code? i.e. I don't want to parse
and execute the streaming expression on the client side, rather, I
want to post it to the server side. Currently, my client code is a big
copy and paste of the /stream request handler (with 1 obvious omission
;-) Specifically, I wasn't able to figure out how to parse the tuple
stream coming back using SolrJ code if I just post the expression to
/stream.

Tim

On Tue, Jul 26, 2016 at 8:54 AM, Joel Bernstein  wrote:
> The difference would be if you are compiling and running the expression in
> a java class or sending it to the /stream handler to be compiled.
>
> If you're compiling it and running it locally you could get this error
> because the StreamContext would not have the numWorkers variable set.
>
> The /stream handler always sets the numWorkers variable, so in theory you
> would never see this error if the /stream handler is executing the
> expression.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Tue, Jul 26, 2016 at 10:44 AM, Timothy Potter 
> wrote:
>
>> it's from a unit test, but not sure why that matters? If I wrap the
>> expression in a parallel expression with explicit workers=1, then it
>> works
>>
>> On Thu, Jul 21, 2016 at 11:13 AM, Joel Bernstein 
>> wrote:
>> > Are you getting this error from a test case you've setup or from a manual
>> > call to the /stream handler?
>> >
>> > Joel Bernstein
>> > http://joelsolr.blogspot.com/
>> >
>> > On Thu, Jul 21, 2016 at 12:28 PM, Timothy Potter 
>> > wrote:
>> >
>> >> I'm working with 6.1.0 release and I have a single SolrCloud instance
>> >> with 1 shard / 1 replica. Somehow I'm triggering this, which from what
>> >> I can see, means workers == 0, but how? Shouldn't workers default to 1
>> >>
>> >> I should mention that my streaming expression doesn't include any
>> >> workers, i.e. it is simply:
>> >>
>> >> val hashJoinExpr =
>> >>   s"""
>> >>  | hashJoin(
>> >>  |search(${ratingsCollection},
>> >>  |   q="*:*",
>> >>  |   fl="movie_id,user_id,rating",
>> >>  |   sort="movie_id asc",
>> >>  |   qt="/export",
>> >>  |   partitionKeys="movie_id"),
>> >>  |hashed=search(${moviesCollection},
>> >>  |  q="*:*",
>> >>  |  fl="movie_id,title",
>> >>  |  sort="movie_id asc",
>> >>  |  qt="/export",
>> >>  |  partitionKeys="movie_id"),
>> >>  |on="movie_id"
>> >>  |  )
>> >>""".stripMargin
>> >>
>> >>
>> >>
>> >> 2016-07-21 10:08:44,596 [qtp2125832297-1073] ERROR RequestHandlerBase
>> >> - java.io.IOException: java.lang.RuntimeException:
>> >> java.lang.ArithmeticException: / by zero
>> >> at
>> >>
>> org.apache.solr.search.HashQParserPlugin$HashQuery.createWeight(HashQParserPlugin.java:130)
>> >> at
>> >>
>> org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:752)
>> >> at
>> >>
>> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:735)
>> >> at
>> >> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
>> >> at
>> >>
>> org.apache.solr.search.DocSetUtil.createDocSetGeneric(DocSetUtil.java:102)
>> >> at
>> org.apache.solr.search.DocSetUtil.createDocSet(DocSetUtil.java:91)
>> >> at
>> >>
>> org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:1386)
>> >> at
>> >>
>> org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:1064)
>> >> at
>> >>
>> org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSearcher.java:1234)
>> >> at
>> >>
>> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1751)
>> >> at
>> >>
>> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1627)
>> >> at
>> >>
>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:643)
>> >> at
>> >>
>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:529)
>> >> at
>> >>
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:293)
>> >> at
>> >>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
>> >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
>> >> at
>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
>> >> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
>> >> at
>> >>
>> 

Re: Streaming expression - workers is zero somehow?

[Joel Bernstein]

The difference would be if you are compiling and running the expression in
a java class or sending it to the /stream handler to be compiled.

If you're compiling it and running it locally you could get this error
because the StreamContext would not have the numWorkers variable set.

The /stream handler always sets the numWorkers variable, so in theory you
would never see this error if the /stream handler is executing the
expression.

Joel Bernstein
http://joelsolr.blogspot.com/

On Tue, Jul 26, 2016 at 10:44 AM, Timothy Potter 
wrote:

> it's from a unit test, but not sure why that matters? If I wrap the
> expression in a parallel expression with explicit workers=1, then it
> works
>
> On Thu, Jul 21, 2016 at 11:13 AM, Joel Bernstein 
> wrote:
> > Are you getting this error from a test case you've setup or from a manual
> > call to the /stream handler?
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> > On Thu, Jul 21, 2016 at 12:28 PM, Timothy Potter 
> > wrote:
> >
> >> I'm working with 6.1.0 release and I have a single SolrCloud instance
> >> with 1 shard / 1 replica. Somehow I'm triggering this, which from what
> >> I can see, means workers == 0, but how? Shouldn't workers default to 1
> >>
> >> I should mention that my streaming expression doesn't include any
> >> workers, i.e. it is simply:
> >>
> >> val hashJoinExpr =
> >>   s"""
> >>  | hashJoin(
> >>  |search(${ratingsCollection},
> >>  |   q="*:*",
> >>  |   fl="movie_id,user_id,rating",
> >>  |   sort="movie_id asc",
> >>  |   qt="/export",
> >>  |   partitionKeys="movie_id"),
> >>  |hashed=search(${moviesCollection},
> >>  |  q="*:*",
> >>  |  fl="movie_id,title",
> >>  |  sort="movie_id asc",
> >>  |  qt="/export",
> >>  |  partitionKeys="movie_id"),
> >>  |on="movie_id"
> >>  |  )
> >>""".stripMargin
> >>
> >>
> >>
> >> 2016-07-21 10:08:44,596 [qtp2125832297-1073] ERROR RequestHandlerBase
> >> - java.io.IOException: java.lang.RuntimeException:
> >> java.lang.ArithmeticException: / by zero
> >> at
> >>
> org.apache.solr.search.HashQParserPlugin$HashQuery.createWeight(HashQParserPlugin.java:130)
> >> at
> >>
> org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:752)
> >> at
> >>
> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:735)
> >> at
> >> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
> >> at
> >>
> org.apache.solr.search.DocSetUtil.createDocSetGeneric(DocSetUtil.java:102)
> >> at
> org.apache.solr.search.DocSetUtil.createDocSet(DocSetUtil.java:91)
> >> at
> >>
> org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:1386)
> >> at
> >>
> org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:1064)
> >> at
> >>
> org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSearcher.java:1234)
> >> at
> >>
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1751)
> >> at
> >>
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1627)
> >> at
> >>
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:643)
> >> at
> >>
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:529)
> >> at
> >>
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:293)
> >> at
> >>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
> >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
> >> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
> >> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
> >> at
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
> >> at
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
> >> at
> >>
> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> >> at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
> >> at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> >> at
> >>
> 

Re: Streaming expression - workers is zero somehow?

[Timothy Potter]

it's from a unit test, but not sure why that matters? If I wrap the
expression in a parallel expression with explicit workers=1, then it
works

On Thu, Jul 21, 2016 at 11:13 AM, Joel Bernstein  wrote:
> Are you getting this error from a test case you've setup or from a manual
> call to the /stream handler?
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Thu, Jul 21, 2016 at 12:28 PM, Timothy Potter 
> wrote:
>
>> I'm working with 6.1.0 release and I have a single SolrCloud instance
>> with 1 shard / 1 replica. Somehow I'm triggering this, which from what
>> I can see, means workers == 0, but how? Shouldn't workers default to 1
>>
>> I should mention that my streaming expression doesn't include any
>> workers, i.e. it is simply:
>>
>> val hashJoinExpr =
>>   s"""
>>  | hashJoin(
>>  |search(${ratingsCollection},
>>  |   q="*:*",
>>  |   fl="movie_id,user_id,rating",
>>  |   sort="movie_id asc",
>>  |   qt="/export",
>>  |   partitionKeys="movie_id"),
>>  |hashed=search(${moviesCollection},
>>  |  q="*:*",
>>  |  fl="movie_id,title",
>>  |  sort="movie_id asc",
>>  |  qt="/export",
>>  |  partitionKeys="movie_id"),
>>  |on="movie_id"
>>  |  )
>>""".stripMargin
>>
>>
>>
>> 2016-07-21 10:08:44,596 [qtp2125832297-1073] ERROR RequestHandlerBase
>> - java.io.IOException: java.lang.RuntimeException:
>> java.lang.ArithmeticException: / by zero
>> at
>> org.apache.solr.search.HashQParserPlugin$HashQuery.createWeight(HashQParserPlugin.java:130)
>> at
>> org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:752)
>> at
>> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:735)
>> at
>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:473)
>> at
>> org.apache.solr.search.DocSetUtil.createDocSetGeneric(DocSetUtil.java:102)
>> at org.apache.solr.search.DocSetUtil.createDocSet(DocSetUtil.java:91)
>> at
>> org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:1386)
>> at
>> org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:1064)
>> at
>> org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSearcher.java:1234)
>> at
>> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1751)
>> at
>> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1627)
>> at
>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:643)
>> at
>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:529)
>> at
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:293)
>> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
>> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>> at
>> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> at
>> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>> at org.eclipse.jetty.server.Server.handle(Server.java:518)
>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
>> at
>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>> at

Re: solr.NRTCachingDirectoryFactory

[Rallavagu]

On 7/26/16 5:46 AM, Shawn Heisey wrote:

On 7/22/2016 10:15 AM, Rallavagu wrote:









As Erick indicated, these settings are incompatible with Near Real Time
updates.

With those settings, every time you commit and create a new searcher,
Solr will execute up to 1000 queries (potentially 500 for each of the
caches above) before that new searcher will begin returning new results.

I do not know how fast your filter queries execute when they aren't
cached... but even if they only take 100 milliseconds each, that's could
take up to a minute for filterCache warming.  If each one takes two
seconds and there are 500 entries in the cache, then autowarming the
filterCache would take nearly 17 minutes. You would also need to wait
for the warming queries on queryResultCache.

The autowarmCount on my filterCache is 4, and warming that cache *still*
sometimes takes ten or more seconds to complete.

If you want true NRT, you need to set all your autowarmCount values to
zero.  The tradeoff with NRT is that your caches are ineffective
immediately after a new searcher is created.

Will look into this and make changes as suggested.



Looking at the "top" screenshot ... you have plenty of memory to cache
the entire index.  Unless your queries are extreme, this is usually
enough for good performance.

One possible problem is that cache warming is taking far longer than
your autoSoftCommit interval, and the server is constantly busy making
thousands of warming queries.  Reducing autowarmCount, possibly to zero,
*might* fix that. I would expect higher CPU load than what your
screenshot shows if this were happening, but it still might be the problem.

Great point. Thanks for the help.



Thanks,
Shawn

Re: solr extends query

[Erik Hatcher]

Ah, I see - thanks for explaining that you’re not operating on tokens like that.

Given that, I think the best place to implement this is as a QParserPlugin - 
that gives you the query string and allows you to return a standard Lucene 
Query.  

Erik


> On Jul 25, 2016, at 9:44 AM, sara hajili  wrote:
> 
> i saw synonym Filter but it is not sufficient for me.as i saw this, i must
> build synonym filter map . that this is built like a map i mean i must put
> exactly in synonym map for example "home" is synonym of "house" (for
> example).
> so in query when user insert home it filter add house to it and expand this.
> but i don't want this. i have a algorithm that get n words and according to
> it and use of wordnet ,it make a decision that what word must be add to
> expand this query. it depends on all words exist on query.
> so i can not tokenize my query word by word.
> because of if i tokenize query word by word .then in synonym filter factory
> i have just one token and i can not make a decision what word must be added
> to query. i need to have all word together.
> and also i can not tokenize my query with "." for example because i wanna
> to tokenize my query word by word to normalize every word. so i can not
> change tokenize format.
> and i can not use synonymfilterfactory.
> how i can do this?i got all query words and expand this with my algorithm
> based on using wordnet?
> 
> On Mon, Jul 25, 2016 at 5:32 AM, Erik Hatcher 
> wrote:
> 
>> You’re going to need to tokenize to look up words, so a TokenFilter is a
>> better place to put this sort of thing, I think.
>> 
>> Build off of Lucene’s SynonymFilter (and corresponding
>> SynonymFilterFactory)
>> https://github.com/apache/lucene-solr/blob/5e5fd662575105de88d8514b426bccdcb4c76948/lucene/analysis/common/src/java/org/apache/lucene/analysis/synonym/SynonymFilter.java
>> <
>> https://github.com/apache/lucene-solr/blob/5e5fd662575105de88d8514b426bccdcb4c76948/lucene/analysis/common/src/java/org/apache/lucene/analysis/synonym/SynonymFilter.java
>>> 
>> 
>> I’m not sure exactly what your code is trying to do (please share that as
>> well, for best assistance), but I do not recommend putting custom code into
>> Solr’s package namespaces (and obviously that has issues here, because of
>> the separate JAR and visibility/access).
>> 
>>Erik
>> 
>> 
>>> On Jul 25, 2016, at 2:25 AM, sara hajili  wrote:
>>> 
>>> i use solr 6-1-0.
>>> and i write my own search Handler .i got this error:
>>> 
>>> java.lang.IllegalAccessError: tried to access field
>>> org.apache.solr.handler.component.ResponseBuilder.requestInfo from
>>> class org.apache.solr.handler.component.MyResponseBuilder
>>>  at
>> org.apache.solr.handler.component.MyResponseBuilder.getRequestInfo(MyResponseBuilder.java:19)
>>>  at
>> org.apache.solr.handler.component.MySearchHandler.handleRequestBody(MySearchHandler.java:94)
>>>  at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
>>>  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
>>>  at
>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
>>>  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
>>>  at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
>>>  at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
>>>  at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>>>  at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>>>  at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>>  at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>>  at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>>>  at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>>>  at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>>>  at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>>  at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
>>>  at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>  at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>>>  at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>>>  at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>>>  at org.eclipse.jetty.server.Server.handle(Server.java:518)
>>>  at
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>>>  at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
>>> 

Re: facet.query on text - recommended field type

[Erik Hatcher]

facet.query is really just a short-cut for numFound using that query standalone.

How many facet.query’s are you issuing?   And what is the QTime for all those 
queries when individually made like this:

   /select?q==0

If one of your queries is “slow” - you mention wildcards and complex phrase 
query, so definitely some red flags!  If you’re making a bunch of those types 
of slow queries as facet.query’s, then it adds up (or maybe worse if we’re 
talking a sharded environment?)

No, the docValues/string/keyword considerations do no apply to facet.query.
The field(s) you need to query for facet.query need to be typed per the needs 
of the query itself, which often is useful to be tokenized text fields.  By all 
means, use the right field types for what you’d use to issue the same 
standalone query as above.

Consider this sort of example, using facet.query to give counts on precise to 
fuzzy queries:

  /select?q=*:*
_facet_q=smith
=on
={!field f=name_s v=$my_facet_q}
={!field f=name_t v=$my_facet_q}
={!field f=name_phonetic v=$my_facet_q}

Assuming “name” is copyFielded into a string (name_s), basic tokenized 
(name_t), and phonetic (name_phonetic), this example will show the counts of 
each of those _levels_ of matching.

If you’re getting heavy into wildcards and complex phrases, I’d love to see 
some examples and compare notes.  I love what facet.query offers, and it’s one 
of Solr’s powerful little known gems.

Erik


> On Jul 26, 2016, at 8:48 AM, lochri  wrote:
> 
> I am experiencing very slow query time when using multiple facet.query on
> large result sets (like 5 minutes).
> I don't know how to optimize this, since it is not completely clear to me
> how the facet.query works.
> Currently my facet queries use tokenized text fields and contain wildcards
> or even use complex phrase query.
> The normal q query part uses the same queries but is acceptable fast (1
> second).
> Does the facet.query use exact same search (through term index) as the q
> query does ?
> I am always reading that fields used for faceting should be string or
> keyword tokenized and docValues should be enabled.
> Is this true for facet.query as well or is this meant for facet.field only ?
> 
> 
> 
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/facet-query-on-text-recommended-field-type-tp4288940.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr 5.5.2 loadOnStartUp does not work

[elisabeth benoit]

Hello,

Thanks for your answer.

Yes, it seems a little tricky to me.

Best regards,
Elisabeth

2016-07-25 18:06 GMT+02:00 Erick Erickson :

> "Load" is a little tricky here, it means "load the core and open a
> searcher.
> The core _descriptor_ which is the internal structure of
> core.properties (plus some other info) _is_ loaded and is what's
> used to show the list of available cores. Else how would you
> even know the core existed?
>
> It's not until you actually try to do anything (even click on the
> item in the "cores" drop-down) that the heavy-duty
> work of opening the core actually executes.
>
> So I think it's working as expected,. But do note
> that this whole area (transient cores, loading on
> startup true/false) is intended for stand-alone
> Solr and is unsupported in SolrCloud.
>
> Best,
> Erick
>
> On Mon, Jul 25, 2016 at 6:09 AM, elisabeth benoit
>  wrote:
> > Hello,
> >
> > I have a core.properties with content
> >
> > name=indexer
> > loadOnStartup=false
> >
> >
> > but the core is loaded on start up (it appears on the admin interface).
> >
> > I thougth the core would be unloaded on startup. did I miss something?
> >
> >
> > best regards,
> >
> > elisabeth
>

facet.query on text - recommended field type

[lochri]

I am experiencing very slow query time when using multiple facet.query on
large result sets (like 5 minutes).
I don't know how to optimize this, since it is not completely clear to me
how the facet.query works.
Currently my facet queries use tokenized text fields and contain wildcards
or even use complex phrase query.
The normal q query part uses the same queries but is acceptable fast (1
second).
Does the facet.query use exact same search (through term index) as the q
query does ?
I am always reading that fields used for faceting should be string or
keyword tokenized and docValues should be enabled.
Is this true for facet.query as well or is this meant for facet.field only ?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/facet-query-on-text-recommended-field-type-tp4288940.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr.NRTCachingDirectoryFactory

[Shawn Heisey]

On 7/22/2016 10:15 AM, Rallavagu wrote:
>   size="5000"
>  initialSize="5000"
>  autowarmCount="500"/>
>

>   size="2"
>  initialSize="2"
>  autowarmCount="500"/>

As Erick indicated, these settings are incompatible with Near Real Time
updates.

With those settings, every time you commit and create a new searcher,
Solr will execute up to 1000 queries (potentially 500 for each of the
caches above) before that new searcher will begin returning new results.

I do not know how fast your filter queries execute when they aren't
cached... but even if they only take 100 milliseconds each, that's could
take up to a minute for filterCache warming.  If each one takes two
seconds and there are 500 entries in the cache, then autowarming the
filterCache would take nearly 17 minutes. You would also need to wait
for the warming queries on queryResultCache.

The autowarmCount on my filterCache is 4, and warming that cache *still*
sometimes takes ten or more seconds to complete.

If you want true NRT, you need to set all your autowarmCount values to
zero.  The tradeoff with NRT is that your caches are ineffective
immediately after a new searcher is created.

Looking at the "top" screenshot ... you have plenty of memory to cache
the entire index.  Unless your queries are extreme, this is usually
enough for good performance.

One possible problem is that cache warming is taking far longer than
your autoSoftCommit interval, and the server is constantly busy making
thousands of warming queries.  Reducing autowarmCount, possibly to zero,
*might* fix that. I would expect higher CPU load than what your
screenshot shows if this were happening, but it still might be the problem.

Thanks,
Shawn

solr-6.1.0 - Using different client and server certificates for authentication doesn't work

[Kostas]

Hello.

 

I have setup Solr 6.1.0 to use SSL (on Windows) and to do client
authentication based on the client certificate.

When I use the same certificate for both the server and the client
authentication, everything works OK :

 



== solr.in.cmd

set SOLR_SSL_KEY_STORE=%ROO%/server/etc/solr-ssl.keystore.jks

set SOLR_SSL_KEY_STORE_PASSWORD=password

set SOLR_SSL_TRUST_STORE=%ROO%/server/etc/solr-ssl.keystore.jks

set SOLR_SSL_TRUST_STORE_PASSWORD=password

set SOLR_SSL_NEED_CLIENT_AUTH=true

set SOLR_SSL_WANT_CLIENT_AUTH=false

REM (Client settings residing below are commented out.)

 

== server\etc\jetty-ssl.xml

  

  

  

  

  

  

 

==  This works :

curl ^

--cert "solr-ssl.keystore.pem" ^

--cacert "solr-ssl.keystore.pem" ^

"https://localhost:8898/solr/admin/collections?action=CLUSTERSTATUS=json;
indent=on"



 

However, when I try to use different server and client certificates, it
doesn't work (it seems that it still uses the server certificate for client
authorizations) :

 



== solr.in.cmd

set SOLR_SSL_KEY_STORE=%ROO%/server/etc/solr-ssl.keystore.jks

set SOLR_SSL_KEY_STORE_PASSWORD=password

set SOLR_SSL_TRUST_STORE=%ROO%/server/etc/solr-ssl.keystore.jks

set SOLR_SSL_TRUST_STORE_PASSWORD=password

set SOLR_SSL_NEED_CLIENT_AUTH=true

set SOLR_SSL_WANT_CLIENT_AUTH=false

 

set SOLR_SSL_CLIENT_KEY_STORE=%ROO%/server/etc/solr-ssl-client.keystore.jks

set SOLR_SSL_CLIENT_KEY_STORE_PASSWORD=password

set
SOLR_SSL_CLIENT_TRUST_STORE=%ROO%/server/etc/solr-ssl-client.keystore.jks

set SOLR_SSL_CLIENT_TRUST_STORE_PASSWORD=password

 

 

== server\etc\jetty-ssl.xml

  

  

  

  

  

  

 

 

== This fails (!!!):

curl ^

--cert "solr-ssl-client.keystore.pem" ^

--cacert "solr-ssl.keystore.pem" ^

"https://localhost:8898/solr/admin/collections?action=CLUSTERSTATUS=json;
indent=on"

 

== This STILL works (!!!):

curl ^

--cert "solr-ssl.keystore.pem" ^

--cacert "solr-ssl.keystore.pem" ^

"https://localhost:8898/solr/admin/collections?action=CLUSTERSTATUS=json;
indent=on"



 

I run Solr like this:

 

"%ROO%\bin\solr" start -c -V -f -p 8898^

-Dsolr.ssl.checkPeerName=false

 

>From what I can tell, Solr uses the values from ` server\etc\jetty-ssl.xml `
and totally discards the ones form `solr.in.cmd`.

Naturally, I would try to set the client certificate inside there
(jetty-ssl.xml), but I don't see any setting available for that.

Is what I am trying to do (use different certificates for server and client
authentication) supported or I waste my time?

Also, why don't the docs say that jetty-ssl.xml overrides the settings in
`solr.in.cmd`? Am I missing something?

 

Thanks,
Kostas

 

Re: Issue with Block Join Parent Query Parser

[Mikhail Khludnev]

How it's parsed? You can check with debugQuery=true

On Tue, Jul 26, 2016 at 10:53 AM, Zheng Lin Edwin Yeo 
wrote:

> Hi,
>
> I'm using Solr 6.1.0
>
> Would like to find out, can we use the Block Join Parent Query Parser to
> filter the parents when I search for a field in the child document?
>
> For example, when I just filter by child query like this, I get 8 results.
> q={!parent which="*content_type:parentDocument*"}+range_f:[2 TO 8]
>
>
> But when I try to filter further by parents as well by this query, I get a
> much more number of results.
> q=* +title:join *+{!parent
> which="*content_type:parentDocument*"}+range_f:[2
> TO 8]
>
>
> This results is weird, as by adding an additional filter, the number of
> results returned should be further reduced. Like out of the original 8
> results, only 2 of them has the title:join, so I should be getting 2
> results.
>
> But instead, I am getting records that are neither with the title:join or
> range that is between 2 to 8 in my returned result.
>
> What could be wrong here?
>
> Regards,
> Edwin
>



-- 
Sincerely yours
Mikhail Khludnev

group.ngroups=true does not work well in distributed searches

[manisha marken]

Hi ,

I am using solr 4.7.2 verison.

I am using group.ngroups=true in my solr queries and recently shifted to
distributed architeture of solr, where i add shards param in my solr
queries as follow :

http://localhost:8983/solr/foo/select?wt=json=2=true=dcterms_source=true=localhost:8983/solr/foo,localhost:8983/solr/bar=true

But group.ngroups=true does not work well if i have documents of one group
on all shards rather than on same shard.

i have seen various jiras of solr, where it is written that issue is fixed
in 4.9 version of solr.
Would like to know , in 4.9 of solr , can we use same queries as earlier or
i will have to change the queries to have facets instead of groups ?

And also, if I have to upgrade to 4.9 version of solr, i would like to know
the changes to be done to upgrade to 4.9 as well.


Regards,
Manisha

Issue with Block Join Parent Query Parser

[Zheng Lin Edwin Yeo]

Hi,

I'm using Solr 6.1.0

Would like to find out, can we use the Block Join Parent Query Parser to
filter the parents when I search for a field in the child document?

For example, when I just filter by child query like this, I get 8 results.
q={!parent which="*content_type:parentDocument*"}+range_f:[2 TO 8]


But when I try to filter further by parents as well by this query, I get a
much more number of results.
q=* +title:join *+{!parent which="*content_type:parentDocument*"}+range_f:[2
TO 8]


This results is weird, as by adding an additional filter, the number of
results returned should be further reduced. Like out of the original 8
results, only 2 of them has the title:join, so I should be getting 2
results.

But instead, I am getting records that are neither with the title:join or
range that is between 2 to 8 in my returned result.

What could be wrong here?

Regards,
Edwin

Re: Using log4j.xml in Solr6

[marotosg]

After a bit of testing I got it working. Basically all the configuration for
log4j by default is under server/resources/log4j.properites. 
By default log4j should be able to find log4j.xml if you delete
log4j.properties. I tried it and that's no the case. 

I figured out this is due the fact that solr script "solr start" sets the
configuration file manually. I replaced in the script "log4j.properties" by
"log4j.xml" and it works well.

I basically need it because some extensions don't work without log4j.xml. 

Please correct me if what I am doing doesn't make sense.

Thanks 
Sergio



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Using-log4j-xml-in-Solr6-tp4288742p4288895.html
Sent from the Solr - User mailing list archive at Nabble.com.