date:20200821

All cores gone along with all solr configuration upon reboot

2020-08-21 Thread yaswanth kumar

Can someone help me on the below issue??

I have configured solr 8.2 with one zookeeper 3.4 and 3 solr nodes

All the configs were pushed initially and Also Indexed all the data into 
multiple collections with 3 replicas on each collection 

Now part of server maintenance these solr nodes were restarted and once they 
came back solr could became empty.. lost all the collections .. all collections 
specific instance directories  in the path /solr/server/solr Were deleted ..but 
data folders are intact nothing lost.. not really sure on how to recover from 
this situation.

Did make sure that the zoo.cfg was properly configured (permanent paths for zoo 
data and logs instead of /tmp )as I am using external zoo instead of the one 
that comes with solr.

Solr data path is a nas storage which is a common for all three solr nodes

Another data point is that I enabled solr basic authentication as well if 
that’s making any difference. Even clusterstate , schema’s, security Json were 
all lost.. really looking for a help in understanding to prevent this happening 
again.

Sent from my iPhone

Re: Kerberos on windows device

2020-08-21 Thread Jörn Franke

Hi,

You can use ktpass, if you are AD Administrator. The security json does not 
change from Linux.

Please note that there are a lot of things to consider with Kerberos that can 
go wrong which is not a Solr issue but Kerberos complexity (eg correct DNS 
names, correct encryption type selected in ktpass, correct attribute set in AD) 
- contact your AD administrator with Kerberos experience to get the parameters 
for your AD.

> Am 21.08.2020 um 22:28 schrieb Vanalli, Ali A - DOT :
> 
> Is there way to create a keytab file from windows using the ktpass utility?  
> We are running SOLR from a Windows device and the example provided in the 
> documentation is only for a Linux server.
> 
> Also, please provide the example of security.json for the Kerberos 
> Authentication and authorization on windows device.
> 
> -Thanks
> 
> Ali Vanalli
> Enterprise Business Appl Services Unit
> Enterprise Services Section
> Bureau of Information Technology Services
> Wisconsin Department of Transportation
> ofc (608) 264-9960
> ali.vana...@dot.wi.gov
>

Kerberos on windows device

2020-08-21 Thread Vanalli, Ali A - DOT

Is there way to create a keytab file from windows using the ktpass utility?  We 
are running SOLR from a Windows device and the example provided in the 
documentation is only for a Linux server.

Also, please provide the example of security.json for the Kerberos 
Authentication and authorization on windows device.

-Thanks

Ali Vanalli
Enterprise Business Appl Services Unit
Enterprise Services Section
Bureau of Information Technology Services
Wisconsin Department of Transportation
ofc (608) 264-9960
ali.vana...@dot.wi.gov

RE: Kerberos on windows device

2020-08-21 Thread Vanalli, Ali A - DOT

Hi,

Wondering, if there is a way to create a keytab file from windows using the 
ktpass utility?  We are running SOLR from a Windows device and the example 
provided in the documentation is only for a Linux server.

Also, please provide the example of security.json for the Kerberos 
Authentication and authorization on windows device.

-Thanks

Ali Vanalli

Re: Solr 8.3.1 longer query latency over 6.4.2

2020-08-21 Thread Michael Gibney

Hmm... if you're manually constructing phrase queries during
pre-parsing, and those are set sow=true,
autogeneratePhraseQueries=true, then despite lack of pf, phrase
queries could still be a key to this. Would any of the phrase queries
explicitly introduced by your pre-parsing hactually  trigger
autogeneratePhraseQueries to kick in? (i.e., would any of the
whitespace-separated tokens in your phrases be further split by your
Solr-internal analysis chain -- WordDelimiter, (Solr-internal)
Synonym, etc.?). Would you be able to share the analysis chain on the
relevant fields, and perhaps (forgiving readability challenges) an
example of pre-parsed input that suffers particularly from performance
degradation?

On Thu, Aug 20, 2020 at 2:28 PM Elaine Cario  wrote:
>
> Thanks Michael, I took a look, but we don't have any pf or pf1,2,3 phrase
> params set at all.  Also, we don't add synonyms through Solr filters,
> rather we parse the user's query in our own application and add synonyms
> there, before it gets to Solr.
>
> Some additional info:  we have sow=true (to be compatible with Solr 6), and
> autogeneratePhraseQueries=true.  In our A/B testing, we didn't see any
> difference in search results (aside from some minor scoring variations), so
> functionally everything is working fine.
>
> I compared the debugQuery results between Solr 6 and 8 on a somewhat
> simplified query (they quickly become unreadable otherwise):
>
> Solr 6:
>   (+(DisjunctionMaxQuery((wkxmlsource:"new york" |
> title:"new york")~1.0) DisjunctionMaxQuery((wkxmlsource:ny | title:ny)~1.0)
> DisjunctionMaxQuery((wkxmlsource:"big apple" | title:"big
> apple")~1.0)))/no_coord
>   +((wkxmlsource:"new york" | title:"new
> york")~1.0 (wkxmlsource:ny | title:ny)~1.0 (wkxmlsource:"big apple" |
> title:"big apple")~1.0)
>
> Solr 8:
>   +(DisjunctionMaxQuery((wkxmlsource:"new york" |
> title:"new york")~1.0) DisjunctionMaxQuery((wkxmlsource:ny | title:ny)~1.0)
> DisjunctionMaxQuery((wkxmlsource:"big apple" | title:"big
> apple")~1.0))
>   +((wkxmlsource:"new york" | title:"new
> york")~1.0 (wkxmlsource:ny | title:ny)~1.0 (wkxmlsource:"big apple" |
> title:"big apple")~1.0)
>
> The only substantial difference is the removal of /no_coord (which is
> probably a result of LUCENE-7347 and likely accounts also for scoring
> variations).
>
> We do see generally higher CPU load with Solr 8 (although it is well within
> tolerance), and we do see much higher thread count (60 for Solr 6 vs 150
> for Solr 8 on average) even on a relatively quiet system.  That seems an
> interesting statistic, but not really sure what it signifies.  We mostly
> take the OOTB defaults for most everything, and config changes were
> minimal, mostly to maintain Solr 6 query behavior (uf=*_query_, sow=true).
>
> On Wed, Aug 19, 2020 at 5:46 PM Michael Gibney 
> wrote:
>
> > Hi Elaine,
> > I'm curious what happens if you remove "pf" (phrase field) setting
> > from your edismax config?
> >
> > This question brought to mind
> >
> > https://issues.apache.org/jira/browse/SOLR-12243?focusedCommentId=16836448#comment-16836448
> > and https://issues.apache.org/jira/browse/LUCENE-8531. This *could*
> > have directly explained the behavior you're observing, except for the
> > fact that pre-6.5.0, analyzeGraphPhrase(...) generated a
> > fully-enumerated Lucene "GraphQuery" (since removed, but afaict
> > similar to MultiPhraseQuery). But the direct topic of SOLR-12243 was
> > that SpanNearQuery, nevermind its performance characteristics, was
> > getting completely ignored by edismax. Curious about your case, I
> > looked at ExtendedDismaxQParser for 6.4.2, and it appears that
> > GraphQuery was similarly ignored?:
> >
> >
> > https://github.com/apache/lucene-solr/blob/releases/lucene-solr/6.4.2/solr/core/src/java/org/apache/solr/search/ExtendedDismaxQParser.java#L1219-L1252
> >
> > If this is in fact the case (and I could well be overlooking
> > something), then it's possible that 6.4.2 was more performant mainly
> > because edismax was completely ignoring the more complex phrase
> > queries generated by analyzeGraphPhrase(...).
> >
> > I'll be curious to hear what you find, and eager to be corrected if
> > the above speculation is off-base!
> >
> > Michael
> >
> >
> > On Wed, Aug 19, 2020 at 10:56 AM Elaine Cario  wrote:
> > >
> > > Hi Solr experts,
> > >
> > > We're in the process of upgrading SolrCloud from 6.4.2 to 8.3.1, and our
> > > performance testing is consistently showing search latencies are
> > measurably
> > > higher in 8.3.1, for certain kinds of queries it may be as much as 200 ms
> > > higher on average.
> > >
> > > We've seen this now in 2 different environments.  In one environment, we
> > > effectively doubled the OS memory for Solr 8 (by removing a replica set),
> > > and saw little improvement.
> > >
> > > The specs on the VM's we're using are the same from Solr 6 and 8, and the
> > > index sizes and shard distribution are also the same.  We reviewed
> > garbage
> > > collection

How to Write Autoscaling Policy changes to Zookeeper/SolrCloud using the autoscaling Java API

2020-08-21 Thread Howard Gonzalez

Hello. I am trying to use the autoscaling Java API to write some cluster policy 
changes to a Zookeeper/SolrCloud cluster. However, I can't find the right way 
to do it. I can get all the autoscaling cluster policy clauses using:

autoScalingConfig.getPolicy.getClusterPolicy

However, after getting all the right List of clauses, I don't know how to write 
those changes to the Zookeeper/Solr cluster using the Java API.

Any guidance please? I know I can use the HTTP solr client to send a json 
request, but just wondering how to do it using the provided Java API.

Thanks in advance

Re: JsonLayout breaks logging?

2020-08-21 Thread t spam

Hi, is there anyone out there that could give me at least a direction? I
just can't seem to figure this one out. Kind regards,

Tijmen

On Wed, Jul 29, 2020 at 9:13 AM t spam  wrote:

>
>> Hi Naz and other solr-users (now with solr-user in to),
>>
>> Excuse my ignorance here (just getting started) but let's take the
>> techproducts example. As you proposed I included the latest jackson-core
>> and jackson-databind jars in the "solr install dir/lib/" directory:
>>
>> [tijmen@solr-1 solr-7.7.3]$ ls -la lib/
>> total 1732
>> drwxrwxr-x.  2 tijmen tijmen  72 Jul 25 08:06 .
>> drwxr-xr-x. 10 tijmen tijmen 212 Jul 25 07:45 ..
>> -rw-rw-r--.  1 tijmen tijmen  351575 Jul 25 07:48 jackson-core-2.11.1.jar
>> -rw-rw-r--.  1 tijmen tijmen 1419800 Jul 25 07:48
>> jackson-databind-2.11.1.jar
>>
>> I then added a lib directive to include the jackson jars
>> in: example/techproducts/solr/techproducts/conf/solrconfig.xml
>>
>>   > regex=".*\.jar" />
>>   > regex="solr-cell-\d.*\.jar" />
>>
>>   > regex=".*\.jar" />
>>   > regex="solr-clustering-\d.*\.jar" />
>>
>>   > regex=".*\.jar" />
>>   > regex="solr-langid-\d.*\.jar" />
>>
>>   > regex="solr-ltr-\d.*\.jar" />
>>
>>   > regex=".*\.jar" />
>>   > regex="solr-velocity-\d.*\.jar" />
>>
>>   > />
>>
>> I start solr using:
>>
>> [tijmen@solr-1 solr-7.7.3]$ ./bin/solr stop -e techproducts
>>
>> Unfortunately I get the same result. Solr starts but no logging.
>>
>> Whenever I remove the JsonLayout from the log4j2.xml it starts logging as
>> expected.
>>
>> Thanks,
>>
>> Tijmen
>>
>> On Fri, Jul 24, 2020 at 6:48 PM Naz S  wrote:
>>
>>> Hi Tijmen,
>>>
>>> If you use maven, for example, you can add dependencies in pom.xml.
>>>
>>> For example,
>>> 
>>> 
>>> com.fasterxml.jackson.core
>>> jackson-core
>>> 2.11.1
>>> 
>>> 
>>> com.fasterxml.jackson.core
>>> jackson-databind
>>> 2.11.1
>>> 
>>> 
>>>
>>> On Fri, Jul 24, 2020 at 1:37 PM t spam  wrote:
>>>
 Hi Naz,

 Could you give me some directions in where or how I should provide
 these dependencies? I can see these dependencies are already in various
 places by default:

 [tijmen@solr-1 solr-7.7.3]$ find . -name jackson*
 ./contrib/clustering/lib/jackson-annotations-2.9.8.jar
 ./contrib/clustering/lib/jackson-databind-2.9.8.jar
 ./contrib/prometheus-exporter/lib/jackson-annotations-2.9.8.jar
 ./contrib/prometheus-exporter/lib/jackson-core-2.9.8.jar
 ./contrib/prometheus-exporter/lib/jackson-databind-2.9.8.jar
 ./contrib/prometheus-exporter/lib/jackson-jq-0.0.8.jar
 ./licenses/jackson-annotations-2.9.8.jar.sha1
 ./licenses/jackson-annotations-LICENSE-ASL.txt
 ./licenses/jackson-annotations-NOTICE.txt
 ./licenses/jackson-core-2.9.8.jar.sha1
 ./licenses/jackson-core-LICENSE-ASL.txt
 ./licenses/jackson-core-NOTICE.txt
 ./licenses/jackson-core-asl-1.9.13.jar.sha1
 ./licenses/jackson-core-asl-LICENSE-ASL.txt
 ./licenses/jackson-core-asl-NOTICE.txt
 ./licenses/jackson-databind-2.9.8.jar.sha1
 ./licenses/jackson-databind-LICENSE-ASL.txt
 ./licenses/jackson-databind-NOTICE.txt
 ./licenses/jackson-dataformat-smile-2.9.8.jar.sha1
 ./licenses/jackson-dataformat-smile-LICENSE-ASL.txt
 ./licenses/jackson-dataformat-smile-NOTICE.txt
 ./licenses/jackson-jq-0.0.8.jar.sha1
 ./licenses/jackson-jq-LICENSE-ASL.txt
 ./licenses/jackson-jq-NOTICE.txt
 ./licenses/jackson-mapper-asl-1.9.13.jar.sha1
 ./licenses/jackson-mapper-asl-LICENSE-ASL.txt
 ./licenses/jackson-mapper-asl-NOTICE.txt
 ./server/solr-webapp/webapp/WEB-INF/lib/jackson-annotations-2.9.8.jar
 ./server/solr-webapp/webapp/WEB-INF/lib/jackson-core-2.9.8.jar
 ./server/solr-webapp/webapp/WEB-INF/lib/jackson-core-asl-1.9.13.jar
 ./server/solr-webapp/webapp/WEB-INF/lib/jackson-databind-2.9.8.jar

 ./server/solr-webapp/webapp/WEB-INF/lib/jackson-dataformat-smile-2.9.8.jar
 ./server/solr-webapp/webapp/WEB-INF/lib/jackson-mapper-asl-1.9.13.jar

 Thanks for your time.

 Tijmen

 On Fri, Jul 24, 2020 at 1:16 PM Naz S  wrote:

>
> You should explicitly provide the jackson dependencies: jackson-core,
> jackson-databind and/or jackson-annotations.
>
> On Fri, Jul 24, 2020 at 8:24 AM t spam  wrote:
>
>> Hi,
>>
>> I'm having difficulty configuring JsonLayout for appenders. I have the
>> following in my log4j2.xml:
>>
>> 
>> 
>>   
>>
>> 
>>   
>> 
>>   %d{-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection}
>> %X{shard}
>> %X{replica} %X{core}] %c{1.} %m%n
>> 
>>   
>> 
>>
>> > name="RollingFile"
>> fileName="${sys:solr.log.dir}/solr.log"
>> filePattern="${sys:solr.log.dir}/solr.log.%i" >
>>   
>>   
>> 
>> 
>>   
>>   
>> 
>>
>> >

Re: Solr v8.6.x fails with multipart MIME in commands

2020-08-21 Thread Markus Kalkbrenner

I created https://issues.apache.org/jira/browse/SOLR-14768

@Joe Maye you should add your findings there.

Von: Markus Kalkbrenner
Antworten an: "solr-user@lucene.apache.org"
Datum: Donnerstag, 20. August 2020 um 16:24
An: "solr-user@lucene.apache.org"
Betreff: Re: Solr v8.6.x fails with multipart MIME in commands

Hi,

Since the update from Solr 8.5 to 8.6 the automated tests for the solarium PHP
library and the Search API Solr module for Drupal fail.
In both cases the error message is:

/solr/5f3e65e0106b9/update/extract => java.lang.NoClassDefFoundError:
org/eclipse/jetty/server/MultiParts

at
org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)

You can see the test results at

https://github.com/solariumphp/solarium/actions

https://github.com/mkalkbrenner/search_api_solr/actions

On stackoverflow someone reports the same:
https://stackoverflow.com/questions/63070107/index-pdf-documents-to-solr-error-500-java-lang-noclassdeffounderror-org-ecli

The tests for the PHP library and drupal both use the official Solr docker
image.

If I force Solr 8.5.1 instead of 8.6.1 the tests pass again.

Von: Markus Kalkbrenner
Datum: Donnerstag, 20. August 2020 um 16:23
An: "solr-user@lucene.apache.org"
Betreff: Re: Solr v8.6.x fails with multipart MIME in commands

This is the same error I reported one hour earlier here at the list.

Von: Joe Doupnik
Antworten an: "solr-user@lucene.apache.org"
Datum: Donnerstag, 20. August 2020 um 15:09
An: "solr-user@lucene.apache.org"
Betreff: Solr v8.6.x fails with multipart MIME in commands

Where as this works with Solr v8.5.0, on v8.6.0 and 8.6.1, operations fail
when Solr encounters multipart mime formatted commands used with POST. Commands
are being issued by PHP v5 and v7 programs. A PHP source program snippet:

[cid:image001.png@01D6770E.5986EAF0]
This produces output of Deleteindex errno=22 and the index is not deleted.

More, when submitting files for indexing. Again error 22 is returned, and
the curl response is "false".
While debugging the root of this is handling multipart mime encoded commands
and the HTTP ERROR 500 message is the tell-tale:
First the generating program part:

[cid:image002.png@01D6770E.5986EAF0]

Then the wireshark analysis of the exchange. A piece of the update/extract
request packet showing MIME parts:

[cid:image003.png@01D6770E.5986EAF0]

Below is what the 500 Server Error packet contains:

[cid:image004.png@01D6770E.5986EAF0]
...
In this file submission case the file is actually submitted and indexed,
but the server's response is failure with error code 22.
Again, one key to this puzzle is that NoClassDefFoundError for MultiParts. The
java code for that area differs dramatically between Solr 8.5 and 8.6.
Thanks,
Joe D.

Markus Kalkbrenner
Dipl.-Ing. (FH) techn. Informatik
CTO

bio.logis Genetic Information Management GmbH
Zentrale:
Olof-Palme-Straße 15
60439 Frankfurt am Main

T: +4969348739 110

markus.kalkbren...@biologis.com
biologis.com

Geschäftsführung: Prof. Dr. med. Daniela Steinberger
Sitz des Unternehmens: Altenhöferallee 3, 60438 Frankfurt am Main
Registergericht Frankfurt am Main, HRB 97945

Markus Kalkbrenner
Dipl.-Ing. (FH) techn. Informatik
CTO

bio.logis Genetic Information Management GmbH
Zentrale:
Olof-Palme-Straße 15
60439 Frankfurt am Main

T: +4969348739 110

markus.kalkbren...@biologis.com
biologis.com

Geschäftsführung: Prof. Dr. med. Daniela Steinberger
Sitz des Unternehmens: Altenhöferallee 3, 60438 Frankfurt am Main
Registergericht Frankfurt am Main, HRB 97945

Solr docker container logging certain entries to syslog

2020-08-21 Thread Mohan, Sowmya

Hello,

I have a solr (version 6.6.6) docker container running where a few logs are 
being written to /var/log/messages. My log4j.properties has the rootLogger set 
to file and console. I have attached it for reference. My question is why are 
some logs still being written to /var/log/messages and what can I do to stop 
that?

Example of logs in /var/log/messages:
Aug 21 03:49:45 devstack 8fa87b1b73a8: 2020-08-21 03:49:45.209 INFO  
(qtp1282788025-249) [   x:ask_a_question] o.a.s.c.S.Request [ask_a_question]  
webapp=/solr path=/dataimport 
params={RequestTimeout=60&qt=/dataimport&clean=false&command=full-import} 
status=0 QTime=207
Aug 21 03:49:45 devstack e0ee5f7c16d0: 2020-08-21 03:49:45.700 INFO  
(qtp1282788025-76) [   x:ta_portal] o.a.s.u.p.LogUpdateProcessorFactory 
[ta_portal]  webapp=/solr path=/dataimport 
params={RequestTimeout=60&qt=/dataimport&clean=false&command=full-import}{} 0 
15698
Aug 21 03:49:47 devstack 8fa87b1b73a8: 2020-08-21 03:49:47.823 INFO  
(qtp1282788025-220) [   x:ta_portal] o.a.s.u.p.LogUpdateProcessorFactory 
[ta_portal]  webapp=/solr path=/dataimport 
params={RequestTimeout=60&qt=/dataimport&clean=false&command=full-import}{} 0 
17822

Thanks,
Sowmya.

Re: Solr 7.7 restore issue

2020-08-21 Thread mirei

Unfortunately, upon further testing, my above suggestion about using only the
set-policy does not actually solve the issue.

The reason my testing above worked with restore was only because I left out
the problematic set-cluster-policy autoscaling rule for replica count. And
the truth is that restore was already always working when that cluster
autoscaling rule was missing. The key point is that I only tested restore
functionality above. When I tested to see if autoscaling itself still worked
by adding/placing replicas within our intended limits, that part wasn't
working.

Now it seems we are back to the two workarounds mentioned previously:
1. Clear out the cluster-policy for replica count, restore, then add back
the cluster-policy.
2. Create or modify your collections attaching a 'rule=replica:<2,node:*' to
match your autoscaling policy.


Out of curiousity, I did try testing using both the set-cluster-policy and
the set-policy where I used MODIFYCOLLECTION to attach the policy to my
collection similar to attaching a rule to the collection, but that produced
the same error when attempting to restore:

"Error getting replica locations :  No node can satisfy the rules
"[{replica=<2, node=#ANY, shard=#EACH, collection=gettingstarted}]"




--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html