Re: Joining more than 2 collections

[Zheng Lin Edwin Yeo]

Hi Joel,

I have managed to get the Join to work, but so far it is only working when
I use qt="/select". It is not working when I use qt="/export".

For the display of the field, is there a way to allow it to list them in
the order which I want?
Currently, the display is quite random, and I can get a field in
collection1, followed by a field in collection3, then collection1 again,
and then collection2.

It will be good if we can arrange the field to display in the order that we
want.

Regards,
Edwin



On 4 May 2017 at 09:56, Zheng Lin Edwin Yeo  wrote:

> Hi Joel,
>
> It works when I started off with just one expression.
>
> Could it be that the data size is too big for export after the join, which
> causes the error?
>
> Regards,
> Edwin
>
> On 4 May 2017 at 02:53, Joel Bernstein  wrote:
>
>> I was just testing with the query below and it worked for me. Some of the
>> error messages I was getting with the syntax was not what I was expecting
>> though, so I'll look into the error handling. But the joins do work when
>> the syntax correct. The query below is joining to the same collection
>> three
>> times, but the mechanics are exactly the same joining three different
>> tables. In this example each join narrows down the result set.
>>
>> hashJoin(parallel(collection2,
>> workers=3,
>> sort="id asc",
>> innerJoin(search(collection2, q="*:*",
>> fl="id",
>> sort="id asc", qt="/export", partitionKeys="id"),
>> search(collection2,
>> q="year_i:42", fl="id, year_i", sort="id asc", qt="/export",
>> partitionKeys="id"),
>> on="id")),
>> hashed=search(collection2, q="day_i:7", fl="id, day_i",
>> sort="id asc", qt="/export"),
>> on="id")
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>> On Wed, May 3, 2017 at 1:29 PM, Joel Bernstein 
>> wrote:
>>
>> > Start off with just this expression:
>> >
>> > search(collection2,
>> > q=*:*,
>> > fl="a_s,b_s,c_s,d_s,e_s",
>> > sort="a_s asc",
>> > qt="/export")
>> >
>> > And then check the logs for exceptions.
>> >
>> > Joel Bernstein
>> > http://joelsolr.blogspot.com/
>> >
>> > On Wed, May 3, 2017 at 12:35 PM, Zheng Lin Edwin Yeo <
>> edwinye...@gmail.com
>> > > wrote:
>> >
>> >> Hi Joel,
>> >>
>> >> I am getting this error after I change add qt=/export and removed the
>> rows
>> >> param. Do you know what could be the reason?
>> >>
>> >> {
>> >>   "error":{
>> >> "metadata":[
>> >>   "error-class","org.apache.solr.common.SolrException",
>> >>   "root-error-class","org.apache.http.MalformedChunkCodingExce
>> >> ption"],
>> >> "msg":"org.apache.http.MalformedChunkCodingException: CRLF
>> expected
>> >> at
>> >> end of chunk",
>> >> "trace":"org.apache.solr.common.SolrException:
>> >> org.apache.http.MalformedChunkCodingException: CRLF expected at end of
>> >> chunk\r\n\tat
>> >> org.apache.solr.client.solrj.io.stream.TupleStream.lambda$wr
>> >> iteMap$0(TupleStream.java:79)\r\n\tat
>> >> org.apache.solr.response.JSONWriter.writeIterator(JSONRespon
>> >> seWriter.java:523)\r\n\tat
>> >> org.apache.solr.response.TextResponseWriter.writeVal(TextRes
>> >> ponseWriter.java:175)\r\n\tat
>> >> org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter
>> >> .java:559)\r\n\tat
>> >> org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(
>> >> TupleStream.java:64)\r\n\tat
>> >> org.apache.solr.response.JSONWriter.writeMap(JSONResponseWri
>> >> ter.java:547)\r\n\tat
>> >> org.apache.solr.response.TextResponseWriter.writeVal(TextRes
>> >> ponseWriter.java:193)\r\n\tat
>> >> org.apache.solr.response.JSONWriter.writeNamedListAsMapWithD
>> >> ups(JSONResponseWriter.java:209)\r\n\tat
>> >> org.apache.solr.response.JSONWriter.writeNamedList(JSONRespo
>> >> nseWriter.java:325)\r\n\tat
>> >> org.apache.solr.response.JSONWriter.writeResponse(JSONRespon
>> >> seWriter.java:120)\r\n\tat
>> >> org.apache.solr.response.JSONResponseWriter.write(JSONRespon
>> >> seWriter.java:71)\r\n\tat
>> >> org.apache.solr.response.QueryResponseWriterUtil.writeQueryR
>> >> esponse(QueryResponseWriterUtil.java:65)\r\n\tat
>> >> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrC
>> >> all.java:732)\r\n\tat
>> >> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:
>> 473)\r\n\tat
>> >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>> >> atchFilter.java:345)\r\n\tat
>> >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>> >> atchFilter.java:296)\r\n\tat
>> >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte
>> >> r(ServletHandler.java:1691)\r\n\tat
>> >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
>> >> dler.java:582)\r\n\tat
>> >> org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>> >> Handler.java:143)\r\n\tat
>> >> org.eclipse.jetty.security.Sec

Re: logging in SolrCloud

[Bernd Fehling]

Hi Edwin,

I'm using Solr 6.5.1


Am 04.05.2017 um 04:35 schrieb Zheng Lin Edwin Yeo:
> Which version of Solr are you using?
> 
> I am using Solr 6.4.2, it seems that both nodes are trying to write to the
> same archived file.
> 
> 
> Exception in thread "main" java.nio.file.FileSystemException:
> C:\edwin\solr\server\logs\solr_gc.log.0.current ->
> C:\edwin\solr\server\logs\archived\solr_gc.log.0.current: The process
>  cannot access the file because it is being used by another process.
> 
> 
> Regards,
> Edwin
> 
> 
> On 3 May 2017 at 23:42, Erick Erickson  wrote:
> 
>> That does look weird. Does the 7574 console log really get archived or
>> is the 8983 console log archived twice? If 7574 doesn't get moved to
>> the archive, this sounds like a JIRA, I'd go ahead and raise it.
>>
>> Actually either way I think it needs a JIRA. Either the wrong log is
>> getting moved or the message needs to be fixed.
>>
>> Best,
>> Erick
>>
>> On Wed, May 3, 2017 at 5:29 AM, Bernd Fehling
>>  wrote:
>>> While looking into SolrCloud I noticed that my logging
>>> gets moved to archived dir by starting a new node.
>>>
>>> E.g.:
>>> bin/solr start -cloud -p 8983
>>> -> server/logs/ has solr-8983-console.log
>>>
>>> bin/solr start -cloud -p 7574
>>> -> solr-8983-console.log is moved to server/logs/archived/
>>> -> server/logs/ has solr-7574-console.log
>>>
>>> Is this how it should be or do I have a misconfig?
>>>
>>> Regards
>>> Bernd
>>
> 

-- 
*
Bernd FehlingBielefeld University Library
Dipl.-Inform. (FH)LibTec - Library Technology
Universitätsstr. 25  and Knowledge Management
33615 Bielefeld
Tel. +49 521 106-4060   bernd.fehling(at)uni-bielefeld.de

BASE - Bielefeld Academic Search Engine - www.base-search.net
*

Graph Query Parser

[mganeshs]

All, Is any one using graph query parser with Solr 6+ versions? Is that
working fine as expected ? Can you guys guide me with some working data
model and configurations to set ?

I tried with sample provided over here
https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-GraphQueryParser
 

But it's not working as expected.





--
View this message in context: 
http://lucene.472066.n3.nabble.com/Graph-Query-Parser-tp405.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SessionExpiredException

[Zheng Lin Edwin Yeo]

Are you using SolrCloud with external ZooKeeper, or Solr's internal
ZooKeeper?

Also, which version of Solr are you using?

Regards,
Edwin

On 3 May 2017 at 21:32, Satya Marivada  wrote:

> Hi,
>
> I see below exceptions in my logs sometimes. What could be causing it?
>
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>
> Thanks,
> Satya
>

Re: Verify unloading of a Solr Core

[Erick Erickson]

Please do not kill Solr with "kill -9". Sure it stops the process but
it's A Bad Practice exactly because it dis-allows graceful shutdowns,
Solr may be forcefully killed when in the middle of operations.

You are not, by chance, specifying an async ID are you? If so, the
command will return immediately and you have to check the id via
REQUESTSTATUS to know when the command is completed.

Finally, I'd look at the Solr logs to be sure the command reached the
Solr instance in question and completed successfully. My bet: the
command didn't go where and/or do what you expected.

Best,
Erick

On Wed, May 3, 2017 at 4:21 PM, Shawn Heisey  wrote:
> On 5/3/2017 1:52 PM, Shashank Pedamallu wrote:
>> I unload a core from Solr using the CoreAdmin page on UI or send unload 
>> request via the api, it works perfectly fine. In my code, I’m trying to 
>> unload a core using the call:
>>
>> CoreAdminRequest.unloadCore(, );
>>
>> I get following response:
>>
>>  {responseHeader={status=0,QTime=0}}
>>
>> At this step, I had put a debug point and Solr UI still shows the core that 
>> I have unloaded. Also, I killed the solr process using kill –9 command and 
>> restarted Solr. The core is still present in Solr. When does it get unloaded 
>> then?
>
> Can you show how the SolrClient is created?  If it's HttpSolrClient,
> then include the URL that's used to create the client.  Change the
> hostname if you need to in order to disguise anything that should remain
> hidden.
>
> Ideally, I'd like to see all of the SolrJ code, and the log entry in
> solr.log from the unload call.
>
> Thanks,
> Shawn
>

Re: logging in SolrCloud

[Erick Erickson]

Bernd:

Do check two things:

1> your core.properties files. Do you have properties set in the
core.properties files that could possibly confuse things?

2> when you start your Solr instances, do you define any sysvars that
could confuse the archive directories?

These are wild shots in the dark mind you...

Best,
Erick

On Wed, May 3, 2017 at 7:35 PM, Zheng Lin Edwin Yeo
 wrote:
> Which version of Solr are you using?
>
> I am using Solr 6.4.2, it seems that both nodes are trying to write to the
> same archived file.
>
>
> Exception in thread "main" java.nio.file.FileSystemException:
> C:\edwin\solr\server\logs\solr_gc.log.0.current ->
> C:\edwin\solr\server\logs\archived\solr_gc.log.0.current: The process
>  cannot access the file because it is being used by another process.
>
>
> Regards,
> Edwin
>
>
> On 3 May 2017 at 23:42, Erick Erickson  wrote:
>
>> That does look weird. Does the 7574 console log really get archived or
>> is the 8983 console log archived twice? If 7574 doesn't get moved to
>> the archive, this sounds like a JIRA, I'd go ahead and raise it.
>>
>> Actually either way I think it needs a JIRA. Either the wrong log is
>> getting moved or the message needs to be fixed.
>>
>> Best,
>> Erick
>>
>> On Wed, May 3, 2017 at 5:29 AM, Bernd Fehling
>>  wrote:
>> > While looking into SolrCloud I noticed that my logging
>> > gets moved to archived dir by starting a new node.
>> >
>> > E.g.:
>> > bin/solr start -cloud -p 8983
>> > -> server/logs/ has solr-8983-console.log
>> >
>> > bin/solr start -cloud -p 7574
>> > -> solr-8983-console.log is moved to server/logs/archived/
>> > -> server/logs/ has solr-7574-console.log
>> >
>> > Is this how it should be or do I have a misconfig?
>> >
>> > Regards
>> > Bernd
>>

Re: logging in SolrCloud

[Zheng Lin Edwin Yeo]

Which version of Solr are you using?

I am using Solr 6.4.2, it seems that both nodes are trying to write to the
same archived file.


Exception in thread "main" java.nio.file.FileSystemException:
C:\edwin\solr\server\logs\solr_gc.log.0.current ->
C:\edwin\solr\server\logs\archived\solr_gc.log.0.current: The process
 cannot access the file because it is being used by another process.


Regards,
Edwin


On 3 May 2017 at 23:42, Erick Erickson  wrote:

> That does look weird. Does the 7574 console log really get archived or
> is the 8983 console log archived twice? If 7574 doesn't get moved to
> the archive, this sounds like a JIRA, I'd go ahead and raise it.
>
> Actually either way I think it needs a JIRA. Either the wrong log is
> getting moved or the message needs to be fixed.
>
> Best,
> Erick
>
> On Wed, May 3, 2017 at 5:29 AM, Bernd Fehling
>  wrote:
> > While looking into SolrCloud I noticed that my logging
> > gets moved to archived dir by starting a new node.
> >
> > E.g.:
> > bin/solr start -cloud -p 8983
> > -> server/logs/ has solr-8983-console.log
> >
> > bin/solr start -cloud -p 7574
> > -> solr-8983-console.log is moved to server/logs/archived/
> > -> server/logs/ has solr-7574-console.log
> >
> > Is this how it should be or do I have a misconfig?
> >
> > Regards
> > Bernd
>

Re: Joining more than 2 collections

[Zheng Lin Edwin Yeo]

Hi Joel,

It works when I started off with just one expression.

Could it be that the data size is too big for export after the join, which
causes the error?

Regards,
Edwin

On 4 May 2017 at 02:53, Joel Bernstein  wrote:

> I was just testing with the query below and it worked for me. Some of the
> error messages I was getting with the syntax was not what I was expecting
> though, so I'll look into the error handling. But the joins do work when
> the syntax correct. The query below is joining to the same collection three
> times, but the mechanics are exactly the same joining three different
> tables. In this example each join narrows down the result set.
>
> hashJoin(parallel(collection2,
> workers=3,
> sort="id asc",
> innerJoin(search(collection2, q="*:*", fl="id",
> sort="id asc", qt="/export", partitionKeys="id"),
> search(collection2,
> q="year_i:42", fl="id, year_i", sort="id asc", qt="/export",
> partitionKeys="id"),
> on="id")),
> hashed=search(collection2, q="day_i:7", fl="id, day_i",
> sort="id asc", qt="/export"),
> on="id")
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Wed, May 3, 2017 at 1:29 PM, Joel Bernstein  wrote:
>
> > Start off with just this expression:
> >
> > search(collection2,
> > q=*:*,
> > fl="a_s,b_s,c_s,d_s,e_s",
> > sort="a_s asc",
> > qt="/export")
> >
> > And then check the logs for exceptions.
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> > On Wed, May 3, 2017 at 12:35 PM, Zheng Lin Edwin Yeo <
> edwinye...@gmail.com
> > > wrote:
> >
> >> Hi Joel,
> >>
> >> I am getting this error after I change add qt=/export and removed the
> rows
> >> param. Do you know what could be the reason?
> >>
> >> {
> >>   "error":{
> >> "metadata":[
> >>   "error-class","org.apache.solr.common.SolrException",
> >>   "root-error-class","org.apache.http.MalformedChunkCodingExce
> >> ption"],
> >> "msg":"org.apache.http.MalformedChunkCodingException: CRLF expected
> >> at
> >> end of chunk",
> >> "trace":"org.apache.solr.common.SolrException:
> >> org.apache.http.MalformedChunkCodingException: CRLF expected at end of
> >> chunk\r\n\tat
> >> org.apache.solr.client.solrj.io.stream.TupleStream.lambda$wr
> >> iteMap$0(TupleStream.java:79)\r\n\tat
> >> org.apache.solr.response.JSONWriter.writeIterator(JSONRespon
> >> seWriter.java:523)\r\n\tat
> >> org.apache.solr.response.TextResponseWriter.writeVal(TextRes
> >> ponseWriter.java:175)\r\n\tat
> >> org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter
> >> .java:559)\r\n\tat
> >> org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(
> >> TupleStream.java:64)\r\n\tat
> >> org.apache.solr.response.JSONWriter.writeMap(JSONResponseWri
> >> ter.java:547)\r\n\tat
> >> org.apache.solr.response.TextResponseWriter.writeVal(TextRes
> >> ponseWriter.java:193)\r\n\tat
> >> org.apache.solr.response.JSONWriter.writeNamedListAsMapWithD
> >> ups(JSONResponseWriter.java:209)\r\n\tat
> >> org.apache.solr.response.JSONWriter.writeNamedList(JSONRespo
> >> nseWriter.java:325)\r\n\tat
> >> org.apache.solr.response.JSONWriter.writeResponse(JSONRespon
> >> seWriter.java:120)\r\n\tat
> >> org.apache.solr.response.JSONResponseWriter.write(JSONRespon
> >> seWriter.java:71)\r\n\tat
> >> org.apache.solr.response.QueryResponseWriterUtil.writeQueryR
> >> esponse(QueryResponseWriterUtil.java:65)\r\n\tat
> >> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrC
> >> all.java:732)\r\n\tat
> >> org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:473)\r\n\tat
> >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
> >> atchFilter.java:345)\r\n\tat
> >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
> >> atchFilter.java:296)\r\n\tat
> >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte
> >> r(ServletHandler.java:1691)\r\n\tat
> >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
> >> dler.java:582)\r\n\tat
> >> org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
> >> Handler.java:143)\r\n\tat
> >> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHa
> >> ndler.java:548)\r\n\tat
> >> org.eclipse.jetty.server.session.SessionHandler.doHandle(
> >> SessionHandler.java:226)\r\n\tat
> >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(
> >> ContextHandler.java:1180)\r\n\tat
> >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
> >> ler.java:512)\r\n\tat
> >> org.eclipse.jetty.server.session.SessionHandler.doScope(
> >> SessionHandler.java:185)\r\n\tat
> >> org.eclipse.jetty.server.handler.ContextHandler.doScope(
> >> ContextHandler.java:1112)\r\n\tat
> >> org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
> >> Handler.java:141)\r\n\tat
> >> org.eclipse.jetty.server.handler

Re: Solr performance on EC2 linux

[Walter Underwood]

Already have a Jira issue for next week. I have a script to run prod logs 
against a cluster. I’ll be testing a four shard by two replica cluster with 17 
million docs and very long queries. We are working on getting the 95th 
percentile under one second, so we should exercise the timeAllowed feature.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On May 3, 2017, at 3:53 PM, Rick Leir  wrote:
> 
> +Walter test it
> 
> Jeff,
> How much CPU does the EC2 hypervisor use? I have heard 5% but that is for a 
> normal workload, and is mostly consumed during system calls or context 
> changes. So it is quite understandable that frequent time calls would take a 
> bigger bite in the AWS cloud compared to bare metal. Sorry, my words are 
> mostly conjecture so please ignore. Cheers -- Rick
> 
> On May 3, 2017 2:35:33 PM EDT, Jeff Wartes  wrote:
>> 
>> It’s presumably not a small degradation - this guy very recently
>> suggested it’s 77% slower:
>> https://blog.packagecloud.io/eng/2017/03/08/system-calls-are-much-slower-on-ec2/
>> 
>> The other reason that blog post is interesting to me is that his
>> benchmark utility showed the work of entering the kernel as high system
>> time, which is also what I was seeing.
>> 
>> I really want to go back and try some more tests, including (now)
>> disabling the timeAllowed param in my query corpus. 
>> I think I’m still a few weeks of higher priority issues away from that
>> though.
>> 
>> 
>> On 5/2/17, 1:45 PM, "Tomás Fernández Löbbe" 
>> wrote:
>> 
>> I remember seeing some performance impact (even when not using it) and
>> it
>> was attributed to the calls to System.nanoTime. See SOLR-7875 and
>> SOLR-7876
>> (fixed for 5.3 and 5.4). Those two Jiras fix the impact when
>> timeAllowed is
>>  not used, but I don't know if there were more changes to improve the
>> performance of the feature itself. The problem was that System.nanoTime
>> may
>> be called too many times on indices with many different terms. If this
>> is
>> the problem Jeff is seeing, a small degradation of System.nanoTime
>> could
>>   have a big impact.
>> 
>>   Tomás
>> 
>> On Tue, May 2, 2017 at 10:23 AM, Walter Underwood
>> 
>>   wrote:
>> 
>>> Hmm, has anyone measured the overhead of timeAllowed? We use it all
>> the
>>> time.
>>> 
>>> If nobody has, I’ll run a benchmark with and without it.
>>> 
>>> wunder
>>> Walter Underwood
>>> wun...@wunderwood.org
>>> 
>> https://linkprotect.cudasvc.com/url?a=http://observer.wunderwood.org/&c=E,1,7uGY1VtJPqam-MhMKpspcb31C9NQ_Jh4nI0gzkQP4gVJkhcC5l031vMIHH0j38EdMESOM5Chjav3lUu1rpTdohTNTPdchTkl4TGNEHWJpJFJ-MR6RrjnTQ,,&typo=0
>> (my blog)
>>> 
>>> 
 On May 2, 2017, at 9:52 AM, Chris Hostetter
>> 
>>> wrote:
 
 
 : I specify a timeout on all queries, 
 
 Ah -- ok, yeah -- you mean using "timeAllowed" correct?
 
 If the root issue you were seeing is in fact clocksource related,
 then using timeAllowed would probably be a significant compounding
 factor there since it would involve a lot of time checks in a
>> single
 request (even w/o any debugging enabled)
 
 (did your coworker's experiements with ES use any sort of
>> equivilent
 timeout feature?)
 
 
 
 
 
 -Hoss
 
>> https://linkprotect.cudasvc.com/url?a=http://www.lucidworks.com/&c=E,1,DwDibSb7PG6wpqsnn-u9uKdCuujyokjeyc6ero6bEdoUjs4Hn_X1jj_z6QAEDmorDqAP_TcaEJX8k5HYYJI7bJ7jQxTDpKUX9MvWAaP6ICoyVmpmQ8X7&typo=0
>>> 
>>> 
>> 
> 
> -- 
> Sorry for being brief. Alternate email is rickleir at yahoo dot com

Re: Verify unloading of a Solr Core

[Shawn Heisey]

On 5/3/2017 1:52 PM, Shashank Pedamallu wrote:
> I unload a core from Solr using the CoreAdmin page on UI or send unload 
> request via the api, it works perfectly fine. In my code, I’m trying to 
> unload a core using the call:
>
> CoreAdminRequest.unloadCore(, );
>
> I get following response:
>
>  {responseHeader={status=0,QTime=0}}
>
> At this step, I had put a debug point and Solr UI still shows the core that I 
> have unloaded. Also, I killed the solr process using kill –9 command and 
> restarted Solr. The core is still present in Solr. When does it get unloaded 
> then?

Can you show how the SolrClient is created?  If it's HttpSolrClient,
then include the URL that's used to create the client.  Change the
hostname if you need to in order to disguise anything that should remain
hidden.

Ideally, I'd like to see all of the SolrJ code, and the log entry in
solr.log from the unload call.

Thanks,
Shawn

Re: Solr performance on EC2 linux

[Rick Leir]

+Walter test it

Jeff,
How much CPU does the EC2 hypervisor use? I have heard 5% but that is for a 
normal workload, and is mostly consumed during system calls or context changes. 
So it is quite understandable that frequent time calls would take a bigger bite 
in the AWS cloud compared to bare metal. Sorry, my words are mostly conjecture 
so please ignore. Cheers -- Rick

On May 3, 2017 2:35:33 PM EDT, Jeff Wartes  wrote:
>
>It’s presumably not a small degradation - this guy very recently
>suggested it’s 77% slower:
>https://blog.packagecloud.io/eng/2017/03/08/system-calls-are-much-slower-on-ec2/
>
>The other reason that blog post is interesting to me is that his
>benchmark utility showed the work of entering the kernel as high system
>time, which is also what I was seeing.
>
>I really want to go back and try some more tests, including (now)
>disabling the timeAllowed param in my query corpus. 
>I think I’m still a few weeks of higher priority issues away from that
>though.
>
>
>On 5/2/17, 1:45 PM, "Tomás Fernández Löbbe" 
>wrote:
>
>I remember seeing some performance impact (even when not using it) and
>it
>was attributed to the calls to System.nanoTime. See SOLR-7875 and
>SOLR-7876
>(fixed for 5.3 and 5.4). Those two Jiras fix the impact when
>timeAllowed is
>   not used, but I don't know if there were more changes to improve the
>performance of the feature itself. The problem was that System.nanoTime
>may
>be called too many times on indices with many different terms. If this
>is
>the problem Jeff is seeing, a small degradation of System.nanoTime
>could
>have a big impact.
>
>Tomás
>
>On Tue, May 2, 2017 at 10:23 AM, Walter Underwood
>
>wrote:
>
>> Hmm, has anyone measured the overhead of timeAllowed? We use it all
>the
>> time.
>>
>> If nobody has, I’ll run a benchmark with and without it.
>>
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>>
>https://linkprotect.cudasvc.com/url?a=http://observer.wunderwood.org/&c=E,1,7uGY1VtJPqam-MhMKpspcb31C9NQ_Jh4nI0gzkQP4gVJkhcC5l031vMIHH0j38EdMESOM5Chjav3lUu1rpTdohTNTPdchTkl4TGNEHWJpJFJ-MR6RrjnTQ,,&typo=0
> (my blog)
>>
>>
>> > On May 2, 2017, at 9:52 AM, Chris Hostetter
>
>> wrote:
>> >
>> >
>> > : I specify a timeout on all queries, 
>> >
>> > Ah -- ok, yeah -- you mean using "timeAllowed" correct?
>> >
>  > > If the root issue you were seeing is in fact clocksource related,
> > > then using timeAllowed would probably be a significant compounding
>> > factor there since it would involve a lot of time checks in a
>single
>> > request (even w/o any debugging enabled)
>> >
>> > (did your coworker's experiements with ES use any sort of
>equivilent
>> > timeout feature?)
>> >
>> >
>> >
>> >
>> >
>> > -Hoss
>> >
>https://linkprotect.cudasvc.com/url?a=http://www.lucidworks.com/&c=E,1,DwDibSb7PG6wpqsnn-u9uKdCuujyokjeyc6ero6bEdoUjs4Hn_X1jj_z6QAEDmorDqAP_TcaEJX8k5HYYJI7bJ7jQxTDpKUX9MvWAaP6ICoyVmpmQ8X7&typo=0
>>
>>
>

-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Re: Verify unloading of a Solr Core

[Shashank Pedamallu]

Sorry, I take back my ignore email. Can someone please respond to the initial 
question:

I unload a core from Solr using the CoreAdmin page on UI or send unload request 
via the api, it works perfectly fine. In my code, I’m trying to unload a core 
using the call:

CoreAdminRequest.unloadCore(, );


I get following response:

{responseHeader={status=0,QTime=0}}


At this step, I had put a debug point and Solr UI still shows the core that I 
have unloaded. Also, I killed the solr process using kill –9 command and 
restarted Solr. The core is still present in Solr. When does it get unloaded 
then? Please note that, I’m making these requests from a CoreAdminHandler.


Thanks,
Shashank Pedamallu






On 5/3/17, 1:09 PM, "Shashank Pedamallu"  wrote:

>Please ignore this question. I’m not able to reproduce this, it should have 
>been a mistake by me.
>
>Thanks,
>Shashank Pedamallu
>
>
>
>
>
>
>On 5/3/17, 12:52 PM, "Shashank Pedamallu"  wrote:
>
>>Hi all,
>>
>>I unload a core from Solr using the CoreAdmin page on UI or send unload 
>>request via the api, it works perfectly fine. In my code, I’m trying to 
>>unload a core using the call:
>>
>>CoreAdminRequest.unloadCore(, );
>>
>>
>>I get following response:
>>
>> {responseHeader={status=0,QTime=0}}
>>
>>
>>At this step, I had put a debug point and Solr UI still shows the core that I 
>>have unloaded. Also, I killed the solr process using kill –9 command and 
>>restarted Solr. The core is still present in Solr. When does it get unloaded 
>>then?
>>
>>Thanks in advance!
>>
>>Thanks,
>>Shashank Pedamallu

Re: Verify unloading of a Solr Core

[Shashank Pedamallu]

Please ignore this question. I’m not able to reproduce this, it should have 
been a mistake by me.

Thanks,
Shashank Pedamallu






On 5/3/17, 12:52 PM, "Shashank Pedamallu"  wrote:

>Hi all,
>
>I unload a core from Solr using the CoreAdmin page on UI or send unload 
>request via the api, it works perfectly fine. In my code, I’m trying to unload 
>a core using the call:
>
>CoreAdminRequest.unloadCore(, );
>
>
>I get following response:
>
> {responseHeader={status=0,QTime=0}}
>
>
>At this step, I had put a debug point and Solr UI still shows the core that I 
>have unloaded. Also, I killed the solr process using kill –9 command and 
>restarted Solr. The core is still present in Solr. When does it get unloaded 
>then?
>
>Thanks in advance!
>
>Thanks,
>Shashank Pedamallu

Verify unloading of a Solr Core

[Shashank Pedamallu]

Hi all,

I unload a core from Solr using the CoreAdmin page on UI or send unload request 
via the api, it works perfectly fine. In my code, I’m trying to unload a core 
using the call:

CoreAdminRequest.unloadCore(, );


I get following response:

 {responseHeader={status=0,QTime=0}}


At this step, I had put a debug point and Solr UI still shows the core that I 
have unloaded. Also, I killed the solr process using kill –9 command and 
restarted Solr. The core is still present in Solr. When does it get unloaded 
then?

Thanks in advance!

Thanks,
Shashank Pedamallu

solr 6.3.0 monitoring

[Satya Marivada]

Hi,

We stood up solr 6.3.0 with external zookeeper 3.4.9. We are moving to
production and setting up monitoring for solr, to check on all cores of a
collection to see they are up. Similary any other pointers towards the
entire collection monitoring or any other suggestions would be useful.

For zookeeper, planning to use MNTR command to check on its status.

Thanks,
Satya

Tests for ZooKeeper Access Control

[bay chae]

Can anyone provide any coherent tests at API level to test ZooKeeper Access 
Control?

Or have I missed the point on what ZooKeeper Access Control is?

In fact what is ZooKeeper Access Control and how should I see its affects?

Re: Joining more than 2 collections

[Joel Bernstein]

I was just testing with the query below and it worked for me. Some of the
error messages I was getting with the syntax was not what I was expecting
though, so I'll look into the error handling. But the joins do work when
the syntax correct. The query below is joining to the same collection three
times, but the mechanics are exactly the same joining three different
tables. In this example each join narrows down the result set.

hashJoin(parallel(collection2,
workers=3,
sort="id asc",
innerJoin(search(collection2, q="*:*", fl="id",
sort="id asc", qt="/export", partitionKeys="id"),
search(collection2,
q="year_i:42", fl="id, year_i", sort="id asc", qt="/export",
partitionKeys="id"),
on="id")),
hashed=search(collection2, q="day_i:7", fl="id, day_i",
sort="id asc", qt="/export"),
on="id")

Joel Bernstein
http://joelsolr.blogspot.com/

On Wed, May 3, 2017 at 1:29 PM, Joel Bernstein  wrote:

> Start off with just this expression:
>
> search(collection2,
> q=*:*,
> fl="a_s,b_s,c_s,d_s,e_s",
> sort="a_s asc",
> qt="/export")
>
> And then check the logs for exceptions.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Wed, May 3, 2017 at 12:35 PM, Zheng Lin Edwin Yeo  > wrote:
>
>> Hi Joel,
>>
>> I am getting this error after I change add qt=/export and removed the rows
>> param. Do you know what could be the reason?
>>
>> {
>>   "error":{
>> "metadata":[
>>   "error-class","org.apache.solr.common.SolrException",
>>   "root-error-class","org.apache.http.MalformedChunkCodingExce
>> ption"],
>> "msg":"org.apache.http.MalformedChunkCodingException: CRLF expected
>> at
>> end of chunk",
>> "trace":"org.apache.solr.common.SolrException:
>> org.apache.http.MalformedChunkCodingException: CRLF expected at end of
>> chunk\r\n\tat
>> org.apache.solr.client.solrj.io.stream.TupleStream.lambda$wr
>> iteMap$0(TupleStream.java:79)\r\n\tat
>> org.apache.solr.response.JSONWriter.writeIterator(JSONRespon
>> seWriter.java:523)\r\n\tat
>> org.apache.solr.response.TextResponseWriter.writeVal(TextRes
>> ponseWriter.java:175)\r\n\tat
>> org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter
>> .java:559)\r\n\tat
>> org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(
>> TupleStream.java:64)\r\n\tat
>> org.apache.solr.response.JSONWriter.writeMap(JSONResponseWri
>> ter.java:547)\r\n\tat
>> org.apache.solr.response.TextResponseWriter.writeVal(TextRes
>> ponseWriter.java:193)\r\n\tat
>> org.apache.solr.response.JSONWriter.writeNamedListAsMapWithD
>> ups(JSONResponseWriter.java:209)\r\n\tat
>> org.apache.solr.response.JSONWriter.writeNamedList(JSONRespo
>> nseWriter.java:325)\r\n\tat
>> org.apache.solr.response.JSONWriter.writeResponse(JSONRespon
>> seWriter.java:120)\r\n\tat
>> org.apache.solr.response.JSONResponseWriter.write(JSONRespon
>> seWriter.java:71)\r\n\tat
>> org.apache.solr.response.QueryResponseWriterUtil.writeQueryR
>> esponse(QueryResponseWriterUtil.java:65)\r\n\tat
>> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrC
>> all.java:732)\r\n\tat
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:473)\r\n\tat
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>> atchFilter.java:345)\r\n\tat
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>> atchFilter.java:296)\r\n\tat
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte
>> r(ServletHandler.java:1691)\r\n\tat
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
>> dler.java:582)\r\n\tat
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>> Handler.java:143)\r\n\tat
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHa
>> ndler.java:548)\r\n\tat
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(
>> SessionHandler.java:226)\r\n\tat
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(
>> ContextHandler.java:1180)\r\n\tat
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
>> ler.java:512)\r\n\tat
>> org.eclipse.jetty.server.session.SessionHandler.doScope(
>> SessionHandler.java:185)\r\n\tat
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(
>> ContextHandler.java:1112)\r\n\tat
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>> Handler.java:141)\r\n\tat
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.ha
>> ndle(ContextHandlerCollection.java:213)\r\n\tat
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(
>> HandlerCollection.java:119)\r\n\tat
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>> erWrapper.java:134)\r\n\tat
>> org.eclipse.jetty.server.Server.handle(Server.java:534)\r\n\tat
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\r\n\tat
>> org.eclipse.jetty.server.HttpConnection.onFilla

Re: Solr performance on EC2 linux

[Jeff Wartes]

It’s presumably not a small degradation - this guy very recently suggested it’s 
77% slower:
https://blog.packagecloud.io/eng/2017/03/08/system-calls-are-much-slower-on-ec2/

The other reason that blog post is interesting to me is that his benchmark 
utility showed the work of entering the kernel as high system time, which is 
also what I was seeing.

I really want to go back and try some more tests, including (now) disabling the 
timeAllowed param in my query corpus. 
I think I’m still a few weeks of higher priority issues away from that though.


On 5/2/17, 1:45 PM, "Tomás Fernández Löbbe"  wrote:

I remember seeing some performance impact (even when not using it) and it
was attributed to the calls to System.nanoTime. See SOLR-7875 and SOLR-7876
(fixed for 5.3 and 5.4). Those two Jiras fix the impact when timeAllowed is
not used, but I don't know if there were more changes to improve the
performance of the feature itself. The problem was that System.nanoTime may
be called too many times on indices with many different terms. If this is
the problem Jeff is seeing, a small degradation of System.nanoTime could
have a big impact.

Tomás

On Tue, May 2, 2017 at 10:23 AM, Walter Underwood 
wrote:

> Hmm, has anyone measured the overhead of timeAllowed? We use it all the
> time.
>
> If nobody has, I’ll run a benchmark with and without it.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> 
https://linkprotect.cudasvc.com/url?a=http://observer.wunderwood.org/&c=E,1,7uGY1VtJPqam-MhMKpspcb31C9NQ_Jh4nI0gzkQP4gVJkhcC5l031vMIHH0j38EdMESOM5Chjav3lUu1rpTdohTNTPdchTkl4TGNEHWJpJFJ-MR6RrjnTQ,,&typo=0
  (my blog)
>
>
> > On May 2, 2017, at 9:52 AM, Chris Hostetter 
> wrote:
> >
> >
> > : I specify a timeout on all queries, 
> >
> > Ah -- ok, yeah -- you mean using "timeAllowed" correct?
> >
> > If the root issue you were seeing is in fact clocksource related,
> > then using timeAllowed would probably be a significant compounding
> > factor there since it would involve a lot of time checks in a single
> > request (even w/o any debugging enabled)
> >
> > (did your coworker's experiements with ES use any sort of equivilent
> > timeout feature?)
> >
> >
> >
> >
> >
> > -Hoss
> > 
https://linkprotect.cudasvc.com/url?a=http://www.lucidworks.com/&c=E,1,DwDibSb7PG6wpqsnn-u9uKdCuujyokjeyc6ero6bEdoUjs4Hn_X1jj_z6QAEDmorDqAP_TcaEJX8k5HYYJI7bJ7jQxTDpKUX9MvWAaP6ICoyVmpmQ8X7&typo=0
>
>

Re: Joining more than 2 collections

[Joel Bernstein]

Start off with just this expression:

search(collection2,
q=*:*,
fl="a_s,b_s,c_s,d_s,e_s",
sort="a_s asc",
qt="/export")

And then check the logs for exceptions.

Joel Bernstein
http://joelsolr.blogspot.com/

On Wed, May 3, 2017 at 12:35 PM, Zheng Lin Edwin Yeo 
wrote:

> Hi Joel,
>
> I am getting this error after I change add qt=/export and removed the rows
> param. Do you know what could be the reason?
>
> {
>   "error":{
> "metadata":[
>   "error-class","org.apache.solr.common.SolrException",
>   "root-error-class","org.apache.http.MalformedChunkCodingException"],
> "msg":"org.apache.http.MalformedChunkCodingException: CRLF expected at
> end of chunk",
> "trace":"org.apache.solr.common.SolrException:
> org.apache.http.MalformedChunkCodingException: CRLF expected at end of
> chunk\r\n\tat
> org.apache.solr.client.solrj.io.stream.TupleStream.lambda$
> writeMap$0(TupleStream.java:79)\r\n\tat
> org.apache.solr.response.JSONWriter.writeIterator(
> JSONResponseWriter.java:523)\r\n\tat
> org.apache.solr.response.TextResponseWriter.writeVal(
> TextResponseWriter.java:175)\r\n\tat
> org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter.java:559)\
> r\n\tat
> org.apache.solr.client.solrj.io.stream.TupleStream.
> writeMap(TupleStream.java:64)\r\n\tat
> org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547)\
> r\n\tat
> org.apache.solr.response.TextResponseWriter.writeVal(
> TextResponseWriter.java:193)\r\n\tat
> org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(
> JSONResponseWriter.java:209)\r\n\tat
> org.apache.solr.response.JSONWriter.writeNamedList(
> JSONResponseWriter.java:325)\r\n\tat
> org.apache.solr.response.JSONWriter.writeResponse(
> JSONResponseWriter.java:120)\r\n\tat
> org.apache.solr.response.JSONResponseWriter.write(
> JSONResponseWriter.java:71)\r\n\tat
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(
> QueryResponseWriterUtil.java:65)\r\n\tat
> org.apache.solr.servlet.HttpSolrCall.writeResponse(
> HttpSolrCall.java:732)\r\n\tat
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:473)\r\n\tat
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:345)\r\n\tat
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:296)\r\n\tat
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1691)\r\n\tat
> org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:582)\r\n\tat
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)\r\n\tat
> org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)\r\n\tat
> org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)\r\n\tat
> org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1180)\r\n\tat
> org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:512)\r\n\tat
> org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)\r\n\tat
> org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1112)\r\n\tat
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)\r\n\tat
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> ContextHandlerCollection.java:213)\r\n\tat
> org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)\r\n\tat
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)\r\n\tat
> org.eclipse.jetty.server.Server.handle(Server.java:534)\r\n\tat
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\r\n\tat
> org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)\r\n\tat
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)\r\n\tat
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\r\n\tat
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)\r\n\tat
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> executeProduceConsume(ExecuteProduceConsume.java:303)\r\n\tat
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(ExecuteProduceConsume.java:148)\r\n\tat
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
> ExecuteProduceConsume.java:136)\r\n\tat
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:671)\r\n\tat
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:589)\r\n\tat
> java.lang.Thread.run(Thread.java:745)\r\nCaused by:
> org.apache.http.MalformedChunkCodingException: CRLF expected at end of
> chunk\r\n\tat
> org.apache.http.impl.io.ChunkedInputStream.getChunkSize(
> ChunkedInputStream.java:255)\r\n\tat
> org.apache.http.impl.io.ChunkedInputStream.nextChunk(
> ChunkedInputStream.java:227)\r\n\tat
> org.apache.ht

Throttle CDCR packets?

[Webster Homer]

We are trying to move to solrcloud in production. Our production servers
are in the Google Cloud, we load data to an on premises cloud and use CDCR
to replicate to two solr clouds hosted in GCloud. The problem is that its
sending packets at too high a rate for our VPN to Google. Is there a way to
throttle the packet rate in CDCR?

is there some combination of replicator schedule and batchSize that would
slow it down? This is a stopper for us, if we cannot address this we may
have to abandon plans for using solrcloud

This is our current configuration for source CDCR

"/cdcr": {"name": "/cdcr","class": "solr.CdcrRequestHandler","replica": [{"
zkHost": "
ae1b-ecom-mzk01:2181,ae1b-ecom-mzk02:2181,ae1b-ecom-mzk03:2181/solr","source
": "bb-catalog-material","target": "bb-catalog-material"},{"zkHost": "
uc1f-ecom-mzk01:2181,uc1f-ecom-mzk02:2181,uc1f-ecom-mzk03:2181/solr","source
": "bb-catalog-material","target": "bb-catalog-material"}],"replicator": {"
threadPoolSize": 4,"schedule": 1000,"batchSize": 250},"updateLogSynchronizer
": {"schedule": 6}}

-- 


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 
therewith.

Click http://www.emdgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

Unstable

[mostafa shahdadi]

Sent from my iPad

Re: Joining more than 2 collections

[Zheng Lin Edwin Yeo]

Hi Joel,

I am getting this error after I change add qt=/export and removed the rows
param. Do you know what could be the reason?

{
  "error":{
"metadata":[
  "error-class","org.apache.solr.common.SolrException",
  "root-error-class","org.apache.http.MalformedChunkCodingException"],
"msg":"org.apache.http.MalformedChunkCodingException: CRLF expected at
end of chunk",
"trace":"org.apache.solr.common.SolrException:
org.apache.http.MalformedChunkCodingException: CRLF expected at end of
chunk\r\n\tat
org.apache.solr.client.solrj.io.stream.TupleStream.lambda$writeMap$0(TupleStream.java:79)\r\n\tat
org.apache.solr.response.JSONWriter.writeIterator(JSONResponseWriter.java:523)\r\n\tat
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:175)\r\n\tat
org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter.java:559)\r\n\tat
org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(TupleStream.java:64)\r\n\tat
org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547)\r\n\tat
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:193)\r\n\tat
org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:209)\r\n\tat
org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:325)\r\n\tat
org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:120)\r\n\tat
org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:71)\r\n\tat
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)\r\n\tat
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:732)\r\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:473)\r\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)\r\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)\r\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)\r\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\r\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\r\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\r\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\r\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\r\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\r\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\r\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\r\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\r\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)\r\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)\r\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\r\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:534)\r\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\r\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\r\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\r\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\r\n\tat
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\r\n\tat
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\r\n\tat
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\r\n\tat
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\r\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\r\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\r\n\tat
java.lang.Thread.run(Thread.java:745)\r\nCaused by:
org.apache.http.MalformedChunkCodingException: CRLF expected at end of
chunk\r\n\tat
org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:255)\r\n\tat
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:227)\r\n\tat
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:186)\r\n\tat
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:215)\r\n\tat
org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:316)\r\n\tat
org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:164)\r\n\tat
org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)\r\n\tat
org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:174)\r\n\tat
sun.nio.cs.StreamDecoder.implClose(Stream

Re: solr-6.3.0 error port is running already

[Satya Marivada]

Hi Rick and Erick,

Thanks for responding. I made sure that there is no other process running
on that port.

Also when this happened, the admin page had all the nodes as live nodes
though some of them are down. So I went ahead and emptied the zookeeper
data directory where all the configuration is stored in version-2 folder. I
then had to upload the configuration again and placed my index fresh in the
solr.

It then came up fine. I was playing with JMX parameters to be passed to jvm
before this started to happen. Not sure if this got to do something with it.

Thanks,
Satya

On Wed, May 3, 2017 at 7:01 AM Rick Leir  wrote:

> Here it is on Fedora/Redhat/Centos (similar on Unix like systems). Look
> for other processes which might already be listening on the port you
> want to listen on:
>
> $ sudo netstat --inet -lp -4
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address   Foreign Address
> State   PID/Program name
> tcp0  0 0.0.0.0:imaps 0.0.0.0:*   LISTEN
> 2577/dovecot
> tcp0  0 0.0.0.0:pop3s 0.0.0.0:*   LISTEN
> 2577/dovecot
>
> ...
>
> $ grep pop3s /etc/services
> pop3s   995/tcp # POP-3 over SSL
> pop3s   995/udp # POP-3 over SSL
>
> I mention pop3s just because you will run into misleading port numbers
> which alias some well known services which are listed in the services file.
>
> cheers -- Rick
>
> On 2017-05-02 05:09 PM, Rick Leir wrote:
> > Satya
> > Say netstat --inet -lP
> > You might need to add -ipv4 to that command. The P might be lower case
> (I am on the bus!). And the output might show misleading service names, see
> /etc/services.
> > Cheers-- Rick
> >
> > On May 2, 2017 3:10:30 PM EDT, Satya Marivada 
> wrote:
> >> Hi,
> >>
> >> I am getting the below exception all of a sudden with solr-6.3.0.
> >> "null:org.apache.solr.common.SolrException: A previous ephemeral live
> >> node
> >> still exists. Solr cannot continue. Please ensure that no other Solr
> >> process using the same port is running already."
> >>
> >> We are using external zookeeper and have restarted solr many times.
> >> There
> >> is no solr running on those ports already. Any suggestions. Looks like
> >> a
> >> bug. Had started using jmx option and then started getting it. Turned
> >> jmx
> >> off, still getting the same issue.
> >>
> >> We are in crunch of time, any workaround to get it started would be
> >> helpful. Not sure where solr is seeing that port, when everything is
> >> started clean.
> >>
> >> Thanks,
> >> Satya
>
>

Re: Joining more than 2 collections

[Joel Bernstein]

I've reformatted the expression below and made a few changes. You have put
things together properly. But these are MapReduce joins that require
exporting the entire result sets. So you will need to add qt=/export to all
the searches and remove the rows param. In Solr 6.6. there is a new
"shuffle" expression that does this automatically.

To test things you'll want to break down each expression and make sure it's
behaving as expected.

For example first run each search. Then run the innerJoin, not in parallel
mode. Then run it in parallel mode. Then try the whole thing.

hashJoin(parallel(collection2,
innerJoin(search(collection2,
   q=*:*,

 fl="a_s,b_s,c_s,d_s,e_s",
   sort="a_s asc",
   partitionKeys="a_s",
   qt="/export"),
   search(collection1,
   q=*:*,

 fl="a_s,f_s,g_s,h_s,i_s,j_s",
   sort="a_s asc",
  partitionKeys="a_s",
  qt="/export"),
   on="a_s"),
 workers="2",
 sort="a_s asc"),
   hashed=search(collection3,
 q=*:*,
 fl="a_s,k_s,l_s",
 sort="a_s asc",
 qt="/export"),
  on="a_s")

Joel Bernstein
http://joelsolr.blogspot.com/

On Wed, May 3, 2017 at 11:26 AM, Zheng Lin Edwin Yeo 
wrote:

> Hi Joel,
>
> Thanks for the clarification.
>
> Would like to check, is this the correct way to do the join? Currently, I
> could not get any results after putting in the hashJoin for the 3rd,
> smallerStream collection (collection3).
>
> http://localhost:8983/solr/collection1/stream?expr=
> hashJoin(parallel(collection2
> ,
> innerJoin(
>  search(collection2,
> q=*:*,
> fl="a_s,b_s,c_s,d_s,e_s",
>  sort="a_s asc",
> partitionKeys="a_s",
> rows=200),
>  search(collection1,
> q=*:*,
> fl="a_s,f_s,g_s,h_s,i_s,j_s",
>  sort="a_s asc",
> partitionKeys="a_s",
> rows=200),
>  on="a_s"),
> workers="2",
>  sort="a_s asc"),
>  hashed=search(collection3,
> q=*:*,
> fl="a_s,k_s,l_s",
> sort="a_s asc",
> rows=200),
> on="a_s")
> &indent=true
>
>
> Regards,
> Edwin
>
>
> On 3 May 2017 at 20:59, Joel Bernstein  wrote:
>
> > Sorry, it's just called hashJoin
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> > On Wed, May 3, 2017 at 2:45 AM, Zheng Lin Edwin Yeo <
> edwinye...@gmail.com>
> > wrote:
> >
> > > Hi Joel,
> > >
> > > I am getting this error when I used the innerHashJoin.
> > >
> > >  "EXCEPTION":"Invalid stream expression innerHashJoin(parallel(
> innerJoin
> > >
> > > I also can't find the documentation on innerHashJoin for the Streaming
> > > Expressions.
> > >
> > > Are you referring to hashJoin?
> > >
> > > Regards,
> > > Edwin
> > >
> > >
> > > On 3 May 2017 at 13:20, Zheng Lin Edwin Yeo 
> > wrote:
> > >
> > > > Hi Joel,
> > > >
> > > > Thanks for the info.
> > > >
> > > > Regards,
> > > > Edwin
> > > >
> > > >
> > > > On 3 May 2017 at 02:04, Joel Bernstein  wrote:
> > > >
> > > >> Also take a look at the documentation for the "fetch" streaming
> > > >> expression.
> > > >>
> > > >> Joel Bernstein
> > > >> http://joelsolr.blogspot.com/
> > > >>
> > > >> On Tue, May 2, 2017 at 2:03 PM, Joel Bernstein 
> > > >> wrote:
> > > >>
> > > >> > Yes you join more then one collection with Streaming Expressions.
> > Here
> > > >> are
> > > >> > a few things to keep in mind.
> > > >> >
> > > >> > * You'll likely want to use the parallel function around the
> largest
> > > >> join.
> > > >> > You'll need to use the join keys as the partitionKeys.
> > > >> > * innerJoin: requires that the streams be sorted on the join keys.
> > > >> > * innerHashJoin: has no sorting requirement.
> > > >> >
> > > >> > So a strategy for a three collection join might look like this:
> > > >> >
> > > >> > innerHashJoin(parallel(innerJoin(bigStream, bigStream)),
> > > smallerStream)
> > > >> >
> > > >> > The largest join can be done in parallel using an innerJoin. You
> can
> > > >> then
> > > >> > wrap the stream coming out of the parallel function in an
> > > innerHashJoin
> > > >> to
> > > >> > join it to another stream.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > Joel Bernstein
> > > >> > http://joelsolr.blogspot.com/
> > > >> >
> > > >> > On Mon, May 1, 2017 at 9:42 PM, Zheng Lin Edwin Yeo <
> 

Re: logging in SolrCloud

[Erick Erickson]

That does look weird. Does the 7574 console log really get archived or
is the 8983 console log archived twice? If 7574 doesn't get moved to
the archive, this sounds like a JIRA, I'd go ahead and raise it.

Actually either way I think it needs a JIRA. Either the wrong log is
getting moved or the message needs to be fixed.

Best,
Erick

On Wed, May 3, 2017 at 5:29 AM, Bernd Fehling
 wrote:
> While looking into SolrCloud I noticed that my logging
> gets moved to archived dir by starting a new node.
>
> E.g.:
> bin/solr start -cloud -p 8983
> -> server/logs/ has solr-8983-console.log
>
> bin/solr start -cloud -p 7574
> -> solr-8983-console.log is moved to server/logs/archived/
> -> server/logs/ has solr-7574-console.log
>
> Is this how it should be or do I have a misconfig?
>
> Regards
> Bernd

Re: Joining more than 2 collections

[Zheng Lin Edwin Yeo]

Hi Joel,

Thanks for the clarification.

Would like to check, is this the correct way to do the join? Currently, I
could not get any results after putting in the hashJoin for the 3rd,
smallerStream collection (collection3).

http://localhost:8983/solr/collection1/stream?expr=hashJoin(parallel(collection2
,
innerJoin(
 search(collection2,
q=*:*,
fl="a_s,b_s,c_s,d_s,e_s",
 sort="a_s asc",
partitionKeys="a_s",
rows=200),
 search(collection1,
q=*:*,
fl="a_s,f_s,g_s,h_s,i_s,j_s",
 sort="a_s asc",
partitionKeys="a_s",
rows=200),
 on="a_s"),
workers="2",
 sort="a_s asc"),
 hashed=search(collection3,
q=*:*,
fl="a_s,k_s,l_s",
sort="a_s asc",
rows=200),
on="a_s")
&indent=true


Regards,
Edwin


On 3 May 2017 at 20:59, Joel Bernstein  wrote:

> Sorry, it's just called hashJoin
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Wed, May 3, 2017 at 2:45 AM, Zheng Lin Edwin Yeo 
> wrote:
>
> > Hi Joel,
> >
> > I am getting this error when I used the innerHashJoin.
> >
> >  "EXCEPTION":"Invalid stream expression innerHashJoin(parallel(innerJoin
> >
> > I also can't find the documentation on innerHashJoin for the Streaming
> > Expressions.
> >
> > Are you referring to hashJoin?
> >
> > Regards,
> > Edwin
> >
> >
> > On 3 May 2017 at 13:20, Zheng Lin Edwin Yeo 
> wrote:
> >
> > > Hi Joel,
> > >
> > > Thanks for the info.
> > >
> > > Regards,
> > > Edwin
> > >
> > >
> > > On 3 May 2017 at 02:04, Joel Bernstein  wrote:
> > >
> > >> Also take a look at the documentation for the "fetch" streaming
> > >> expression.
> > >>
> > >> Joel Bernstein
> > >> http://joelsolr.blogspot.com/
> > >>
> > >> On Tue, May 2, 2017 at 2:03 PM, Joel Bernstein 
> > >> wrote:
> > >>
> > >> > Yes you join more then one collection with Streaming Expressions.
> Here
> > >> are
> > >> > a few things to keep in mind.
> > >> >
> > >> > * You'll likely want to use the parallel function around the largest
> > >> join.
> > >> > You'll need to use the join keys as the partitionKeys.
> > >> > * innerJoin: requires that the streams be sorted on the join keys.
> > >> > * innerHashJoin: has no sorting requirement.
> > >> >
> > >> > So a strategy for a three collection join might look like this:
> > >> >
> > >> > innerHashJoin(parallel(innerJoin(bigStream, bigStream)),
> > smallerStream)
> > >> >
> > >> > The largest join can be done in parallel using an innerJoin. You can
> > >> then
> > >> > wrap the stream coming out of the parallel function in an
> > innerHashJoin
> > >> to
> > >> > join it to another stream.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Joel Bernstein
> > >> > http://joelsolr.blogspot.com/
> > >> >
> > >> > On Mon, May 1, 2017 at 9:42 PM, Zheng Lin Edwin Yeo <
> > >> edwinye...@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> Hi,
> > >> >>
> > >> >> Is it possible to join more than 2 collections using one of the
> > >> streaming
> > >> >> expressions (Eg: innerJoin)? If not, is there other ways we can do
> > it?
> > >> >>
> > >> >> Currently, I may need to join 3 or 4 collections together, and to
> > >> output
> > >> >> selected fields from all these collections together.
> > >> >>
> > >> >> I'm using Solr 6.4.2.
> > >> >>
> > >> >> Regards,
> > >> >> Edwin
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>

Solr 6.5.1 process crash after jshort_disjoint_arraycopy error

[Yago Riveiro]

# A fatal error has been detected by the Java Runtime Environment:
Hi,

I'm running 6.5.1 using Java 8 build 1.8.0_131-b11 and solr's process crash
with this log

#  SIGBUS (0x7) at pc=0x7fd2c87ea014, pid=4468, tid=0x7fd1f487e700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build
1.8.0_131-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode
linux-amd64 compressed oops)
# Problematic frame:
# v  ~StubRoutines::jshort_disjoint_arraycopy
#
# Failed to write core dump. Core dumps have been disabled. To enable core
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /opt/solr/solr-6.5.1/server/hs_err_pid4468.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp


Any idea what is going on?



-
Best regards

/Yago
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-6-5-1-process-crash-after-jshort-disjoint-arraycopy-error-tp4333150.html
Sent from the Solr - User mailing list archive at Nabble.com.

SessionExpiredException

[Satya Marivada]

Hi,

I see below exceptions in my logs sometimes. What could be causing it?

org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /overseer
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)

Thanks,
Satya

Newbie Zookeeper Access Control

[bay chae]

Hi I amredacted following this guide:

https://cwiki.apache.org/confluence/display/solr/ZooKeeper+Access+Control 


My solr version is 6.5.1. My operating system is OS X macSierra.

My startup command is:

• bin/solr start -v -cloud -s ~/var/solr

in ~/var/solr I have default zoo.cfg and my solr.xml is:



  

${host:}
${jetty.port:8983}
${hostContext:solr}

${genericCoreNodeNames:true}

${zkClientTimeout:3}
${distribUpdateSoTimeout:60}
${distribUpdateConnTimeout:6}

${org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider}
${zkDigestUsername:admin-user}
${zkDigestPassword:secret}
${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}

  

  
${socketTimeout:60}
${connTimeout:6}
  


in solr.in.sh I have:
SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider
 \
  
-DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider
 \
  -DzkDigestUsername=admin-user -DzkDigestPassword=secret \
  -DzkDigestReadonlyUsername=readonly-user 
-DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD"
But I am getting an error in the logs (please see below). Any help?
2017-05-03 13:17:20.689 INFO  (main) [   ] o.e.j.s.Server jetty-9.3.14.v20161028
2017-05-03 13:17:21.031 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter  ___  
_   Welcome to Apache Solr™ version 6.5.1
2017-05-03 13:17:21.031 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter / __| 
___| |_ _   Starting in cloud mode on port 8983
2017-05-03 13:17:21.032 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter \__ \/ _ 
\ | '_|  Install dir: /usr/local/solr-6.5.1
2017-05-03 13:17:21.044 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter 
|___/\___/_|_|Start time: 2017-05-03T13:17:21.033Z
2017-05-03 13:17:21.045 INFO  (main) [   ] o.a.s.s.StartupLoggingUtils Property 
solr.log.muteconsole given. Muting ConsoleAppender named CONSOLE
2017-05-03 13:17:21.045 INFO  (main) [   ] o.a.s.s.StartupLoggingUtils Log 
level override, property solr.log.level=DEBUG
2017-05-03 13:17:21.058 DEBUG (main) [   ] o.a.s.c.SolrResourceLoader JNDI not 
configured for solr (NoInitialContextEx)
2017-05-03 13:17:21.059 INFO  (main) [   ] o.a.s.c.SolrResourceLoader Using 
system property solr.solr.home: /Users/mredacted/var/solr
2017-05-03 13:17:21.060 DEBUG (main) [   ] o.a.s.c.SolrResourceLoader new 
SolrResourceLoader for directory: '/Users/mrsasdasdr/var/solr'
2017-05-03 13:17:21.060 DEBUG (main) [   ] o.a.s.c.SolrResourceLoader JNDI not 
configured for solr (NoInitialContextEx)
2017-05-03 13:17:21.065 INFO  (main) [   ] o.a.s.c.SolrXmlConfig Loading 
container configuration from /Users/mredacted/var/solr/solr.xml
2017-05-03 13:17:21.104 DEBUG (main) [   ] o.a.s.c.Config null missing optional 
solr/@coreLoadThreads
2017-05-03 13:17:21.105 DEBUG (main) [   ] o.a.s.c.Config null missing optional 
solr/@persistent
2017-05-03 13:17:21.106 DEBUG (main) [   ] o.a.s.c.Config null missing optional 
solr/@sharedLib
2017-05-03 13:17:21.107 DEBUG (main) [   ] o.a.s.c.Config null missing optional 
solr/@zkHost
2017-05-03 13:17:21.109 DEBUG (main) [   ] o.a.s.c.Config null missing optional 
solr/cores
2017-05-03 13:17:21.116 ERROR (main) [   ] o.a.s.s.SolrDispatchFilter Could not 
start Solr. Check solr/home property and the logs
2017-05-03 13:17:21.137 ERROR (main) [   ] o.a.s.c.SolrCore 
null:org.apache.solr.common.SolrException: No system property or default value 
specified for 
org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider
 
value:${org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider}
   at 
org.apache.solr.util.PropertiesUtil.substituteProperty(PropertiesUtil.java:65)
   at org.apache.solr.util.DOMUtil.substituteProperties(DOMUtil.java:303)
   at org.apache.solr.util.DOMUtil.substituteProperties(DOMUtil.java:311)
   at org.apache.solr.util.DOMUtil.substituteProperties(DOMUtil.java:311)
   at org.apache.solr.util.DOMUtil.substituteProperties(DOMUtil.java:311)
   at org.apache.solr.core.Config.substituteProperties(Config.java:231)
   at org.apache.solr.core.SolrXmlConfig.fromConfig(SolrXmlConfig.java:63)
   at org.apache.solr.core.SolrXmlConfig.fromInputStream(SolrXmlConfig.java:131)
   at org.apache.solr.core.SolrXmlConfig.fromFile(SolrXmlConfig.java:113)
   at org.apache.solr.core.SolrXmlConfig.fromSolrHome(SolrXmlConfig.java:141)
   at 
org.apache.solr.servlet.SolrDispatchFilter.loadNodeConfig(SolrDispatchFilter.java:267)
   at 
org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:235)
   at 
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:169)
   at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
   at 
org.eclipse.jetty.servlet.ServletHandler.initi

Re: Joining more than 2 collections

[Joel Bernstein]

Sorry, it's just called hashJoin

Joel Bernstein
http://joelsolr.blogspot.com/

On Wed, May 3, 2017 at 2:45 AM, Zheng Lin Edwin Yeo 
wrote:

> Hi Joel,
>
> I am getting this error when I used the innerHashJoin.
>
>  "EXCEPTION":"Invalid stream expression innerHashJoin(parallel(innerJoin
>
> I also can't find the documentation on innerHashJoin for the Streaming
> Expressions.
>
> Are you referring to hashJoin?
>
> Regards,
> Edwin
>
>
> On 3 May 2017 at 13:20, Zheng Lin Edwin Yeo  wrote:
>
> > Hi Joel,
> >
> > Thanks for the info.
> >
> > Regards,
> > Edwin
> >
> >
> > On 3 May 2017 at 02:04, Joel Bernstein  wrote:
> >
> >> Also take a look at the documentation for the "fetch" streaming
> >> expression.
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >> On Tue, May 2, 2017 at 2:03 PM, Joel Bernstein 
> >> wrote:
> >>
> >> > Yes you join more then one collection with Streaming Expressions. Here
> >> are
> >> > a few things to keep in mind.
> >> >
> >> > * You'll likely want to use the parallel function around the largest
> >> join.
> >> > You'll need to use the join keys as the partitionKeys.
> >> > * innerJoin: requires that the streams be sorted on the join keys.
> >> > * innerHashJoin: has no sorting requirement.
> >> >
> >> > So a strategy for a three collection join might look like this:
> >> >
> >> > innerHashJoin(parallel(innerJoin(bigStream, bigStream)),
> smallerStream)
> >> >
> >> > The largest join can be done in parallel using an innerJoin. You can
> >> then
> >> > wrap the stream coming out of the parallel function in an
> innerHashJoin
> >> to
> >> > join it to another stream.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Joel Bernstein
> >> > http://joelsolr.blogspot.com/
> >> >
> >> > On Mon, May 1, 2017 at 9:42 PM, Zheng Lin Edwin Yeo <
> >> edwinye...@gmail.com>
> >> > wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> Is it possible to join more than 2 collections using one of the
> >> streaming
> >> >> expressions (Eg: innerJoin)? If not, is there other ways we can do
> it?
> >> >>
> >> >> Currently, I may need to join 3 or 4 collections together, and to
> >> output
> >> >> selected fields from all these collections together.
> >> >>
> >> >> I'm using Solr 6.4.2.
> >> >>
> >> >> Regards,
> >> >> Edwin
> >> >>
> >> >
> >> >
> >>
> >
> >
>

logging in SolrCloud

[Bernd Fehling]

While looking into SolrCloud I noticed that my logging
gets moved to archived dir by starting a new node.

E.g.:
bin/solr start -cloud -p 8983
-> server/logs/ has solr-8983-console.log

bin/solr start -cloud -p 7574
-> solr-8983-console.log is moved to server/logs/archived/
-> server/logs/ has solr-7574-console.log

Is this how it should be or do I have a misconfig?

Regards
Bernd

RE: CloudDescriptor.getNumShards() sometimes returns null

[Markus Jelsma]

Any ideas on this one? I am still puzzled by this oddity.

Many thanks,
Markus

 
 
-Original message-
> From:Markus Jelsma 
> Sent: Monday 24th April 2017 17:10
> To: solr-user@lucene.apache.org; solr-user 
> Subject: RE: CloudDescriptor.getNumShards() sometimes returns null
> 
> Sorry, forgot to mention the version, it is on 6.5.0.
> 
> Thanks,
> Markus
> 
>  
>  
> -Original message-
> > From:Erick Erickson 
> > Sent: Monday 24th April 2017 16:50
> > To: solr-user 
> > Subject: Re: CloudDescriptor.getNumShards() sometimes returns null
> > 
> > What version of Solr? This has been reworked pretty heavily lately no
> > 6x and trunk.
> > 
> > On Mon, Apr 24, 2017 at 2:24 AM, Markus Jelsma
> >  wrote:
> > > Hi - that (RE: Overseer session expires on multiple collection creation) 
> > > was the wrong thread. I meant, any ideas on this one?
> > >
> > > Many thanks,
> > > Markus
> > >
> > >
> > > -Original message-
> > >> From:Markus Jelsma 
> > >> Sent: Friday 14th April 2017 17:25
> > >> To: solr-user 
> > >> Subject: CloudDescriptor.getNumShards() sometimes returns null
> > >>
> > >> Hi - I've got this 2 shard/2 replica cluster. In handler i need the 
> > >> number of shards of the cluster.
> > >>
> > >> cloudDescriptor = core.getCoreDescriptor().getCloudDescriptor();
> > >> return cloudDescriptor.getNumShards();
> > >>
> > >> It is, however, depending on which node is executing this, sometimes 
> > >> null! This code only runs on shard leaders. First replica of the first 
> > >> shard always returns 2, but second replica of shard one, even when it is 
> > >> the leader, always gets me null. Same appears to be true for the second 
> > >> shard.
> > >>
> > >> I am clearly missing something, any ideas to share?
> > >>
> > >> Thanks,
> > >> Markus
> > >>
> > 
> 

Re: solr-6.3.0 error port is running already

[Rick Leir]

Here it is on Fedora/Redhat/Centos (similar on Unix like systems). Look 
for other processes which might already be listening on the port you 
want to listen on:


$ sudo netstat --inet -lp -4
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address   Foreign Address 
State   PID/Program name
tcp0  0 0.0.0.0:imaps 0.0.0.0:*   LISTEN  
2577/dovecot
tcp0  0 0.0.0.0:pop3s 0.0.0.0:*   LISTEN  
2577/dovecot


...

$ grep pop3s /etc/services
pop3s   995/tcp # POP-3 over SSL
pop3s   995/udp # POP-3 over SSL

I mention pop3s just because you will run into misleading port numbers 
which alias some well known services which are listed in the services file.


cheers -- Rick

On 2017-05-02 05:09 PM, Rick Leir wrote:

Satya
Say netstat --inet -lP
You might need to add -ipv4 to that command. The P might be lower case (I am on 
the bus!). And the output might show misleading service names, see 
/etc/services.
Cheers-- Rick

On May 2, 2017 3:10:30 PM EDT, Satya Marivada  wrote:

Hi,

I am getting the below exception all of a sudden with solr-6.3.0.
"null:org.apache.solr.common.SolrException: A previous ephemeral live
node
still exists. Solr cannot continue. Please ensure that no other Solr
process using the same port is running already."

We are using external zookeeper and have restarted solr many times.
There
is no solr running on those ports already. Any suggestions. Looks like
a
bug. Had started using jmx option and then started getting it. Turned
jmx
off, still getting the same issue.

We are in crunch of time, any workaround to get it started would be
helpful. Not sure where solr is seeing that port, when everything is
started clean.

Thanks,
Satya

Re: Collection backup fails with java.nio.file.NoSuchFileException

[Bernd Fehling]

Hi Ere,

https://issues.apache.org/jira/browse/SOLR-9120

Regards,
Bernd

Am 03.05.2017 um 08:22 schrieb Ere Maijala:
> I'm running a three-node SolrCloud (tested with versions 6.4.2 and 6.5.0) 
> with 1 shard and 3 replicas, and I'm having trouble getting the
> collection backup API to actually do the job. This is the request I use to 
> initiate the backup:
> 
> http://localhost:8983/solr/admin/collections?action=BACKUP&name=biblioprod&collection=biblio1&location=/data/backup/solr
> 
> The result is always something like this:
> 
> 
> 
> 500 name="QTime">840 name="[servername]:8983_solr">org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
>  from server at
> http://[servername]:8983/solr: Failed to backup core=biblio1_shard1_replica3 
> because java.nio.file.NoSuchFileException:
> /data/solr1-1/vufind/biblio1_shard1_replica3/data/index.20170424115540663/segments_y1  name="Operation backup caused
> exception:">org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
>  Could not backup all replicas name="exception">Could not backup all replicas name="rspCode">500 name="metadata"> name="error-class">org.apache.solr.common.SolrException name="root-error-class">org.apache.solr.common.SolrException name="msg">Could not backup all replicas name="trace">org.apache.solr.common.SolrException: Could not backup all 
> replicas
> at 
> org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:287)
> [snip stack]
> 500
> 
> 
> The index seems to be working fine. I've also tested optimizing it first 
> (shouldn't need to do that, right?), but it always fails with
> segments_[something] missing. Also going to Solr Admin UI, selecting the 
> collection and clicking e.g. Schema in left hand menu causes the
> following to be written into solr.log:
> 
> 2017-04-27 11:19:50.724 WARN  (qtp1205044462-1590) [c:biblio1 s:shard1 
> r:core_node1 x:biblio1_shard1_replica3] o.a.s.h.a.LukeRequestHandler
> Error getting file length for [segments_y1]
> java.nio.file.NoSuchFileException: 
> /data/solr1-1/vufind/biblio1_shard1_replica3/data/index.20170424115540663/segments_y1
> at 
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at 
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at 
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at 
> sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at 
> sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
> at 
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.nio.file.Files.readAttributes(Files.java:1737)
> at java.nio.file.Files.size(Files.java:2332)
> at 
> org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
> at 
> org.apache.lucene.store.NRTCachingDirectory.fileLength(NRTCachingDirectory.java:128)
> at 
> org.apache.solr.handler.admin.LukeRequestHandler.getFileLength(LukeRequestHandler.java:614)
> at 
> org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:587)
> at 
> org.apache.solr.handler.admin.LukeRequestHandler.handleRequestBody(LukeRequestHandler.java:138)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:166)
> [snip]
> 
> Looks like the backup only works intermittently after the server has been 
> sitting for a while without any update requests being made.
> 
> Any ideas?
> 
> Regards,
> Ere