Re: RegexReplaceProcessorFactory pattern to detect multiple \n
Hi,
Thanks for the reply.
Do you know of any regex online tool that works correctly for Java regex?
I tried to find some, but they are not working properly.
Yes, our plan is to replace more than one \n with , and single \n
with single .
Regards,
Edwin
On Wed, 20 Feb 2019 at 14:59, Jörn Franke wrote:
> Solr uses Java regex matching, so i doubt there is a bug - it would then
> be in the JDK. Try out in a regex online Tool that supports Java regex for
> your solution.
>
> I believe you want to have 2 regex process factories:
> One that deals with single \n and one that deals with more than one \n
>
> > Am 20.02.2019 um 06:17 schrieb Zheng Lin Edwin Yeo >:
> >
> > Hi,
> >
> > We have tried with the following pattern ([ \t]*\r?\n){2,} and
> > configuration:
> >
> >
> > content
> > ([ \t]*\r?\n){2,}
> >
> > true
> >
> >
> > However, the issue is still occurring.
> >
> > Anyone else is able to help?
> >
> > Regards,
> > Edwin
> >
> > On Fri, 15 Feb 2019 at 11:47, Zheng Lin Edwin Yeo
> > wrote:
> >
> >> Hi,
> >>
> >> For your info, this issue is occurring in Solr 7.7.0 as well.
> >>
> >> Regards,
> >> Edwin
> >>
> >> On Tue, 12 Feb 2019 at 00:10, Zheng Lin Edwin Yeo >
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Should we report this as a bug in Solr?
> >>>
> >>> Regards,
> >>> Edwin
> >>>
> >>> On Fri, 8 Feb 2019 at 22:18, Zheng Lin Edwin Yeo >
> >>> wrote:
> >>>
> Hi Paul,
>
> Regarding the regex (\n\s*){2,} that we are using, when we try in on
> https://regex101.com/, it is able to give us the correct result for
> all
> the examples (ie: All of them will only have , and not more
> than
> that like what we are getting in Solr in our earlier examples).
>
> Could there be a possibility of a bug in Solr?
>
> Regards,
> Edwin
>
> On Fri, 8 Feb 2019 at 00:33, Zheng Lin Edwin Yeo <
> edwinye...@gmail.com>
> wrote:
>
> > Hi Paul,
> >
> > We have tried it with the space preceeding the \n i.e. > name="pattern">(\s*\n){2,}, with the following regex pattern:
> >
> >
> > content
> > (\s*\n){2,}
> >
> >
> >
> > However, we are also getting the exact same results as the earlier
> > Example 1, 2 and 3.
> >
> > As for your point 2 on perhaps in the data you have other (non
> > printing) characters than \n, we have find that there are no non
> printing
> > characters. It is just next line with a space. You can refer to the
> > original content in the same examples below.
> >
> >
> > Example 1: The sentence that the above regex pattern is working
> > correctly
> > *Original content in EML file:*
> > Dear Sir,
> >
> >
> > I am terminating
> > *Original content:*Dear Sir, \n\n \n \n\n I am terminating
> > *Index content: *Dear Sir, I am terminating
> >
> > Example 2: The sentence that the above regex pattern is partially
> > working (as you can see, instead of 2 , there are 4 )
> > *Original content in EML file:*
> >
> > *exalted*
> >
> > *Psalm 89:17*
> >
> >
> > 3 Choa Chu Kang Avenue 4
> > *Original content:* exalted \n \n\n Psalm 89:17 \n\n \n\n 3
> > Choa Chu Kang Avenue 4, Singapore
> > *Index content: *exalted Psalm 89:17 3
> > Choa Chu Kang Avenue 4, Singapore
> >
> > Example 3: The sentence that the above regex pattern is partially
> > working (as you can see, instead of 2 , there are 4 )
> > *Original content in EML file:*
> >
> > http://www.concordpri.moe.edu.sg/
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Dec 18, 2018 at 10:07 AM
> > *Original content:* http://www.concordpri.moe.edu.sg/ \n\n \n\n
> \n
> > \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n\n \n\n\n On Tue,
> Dec 18,
> > 2018 at 10:07 AM
> > *Index content: *http://www.concordpri.moe.edu.sg/
> > On Tue, Dec 18, 2018 at 10:07 AM
> >
> >
> > Appreciate any other ideas or suggestions that you may have.
> >
> > Thank you.
> >
> > Regards,
> > Edwin
> >
> >> On Thu, 7 Feb 2019 at 22:49, wrote:
> >>
> >> Hi Edwin
> >>
> >>
> >>
> >> 1. Sorry, the pattern was wrong, the space should preceed the \n
> >> i.e. (\s*\n){2,}
> >> 2. Perhaps in the data you have other (non printing) characters
> >> than \n?
> >>
> >>
> >>
> >> Gesendet von Mail
> für
> >> Windows 10
> >>
> >>
> >>
> >> Von: Zheng Lin Edwin Yeo
> >> Gesendet: Donnerstag, 7. Februar 2019 15:23
> >> An: solr-user@lucene.apache.org
> >> Betreff: Re: RegexReplaceProcessorFactory pattern to detect
> multiple \n
> >>
> >>
> >>
> >> Hi Paul,
> >>
> >> We
Re: RegexReplaceProcessorFactory pattern to detect multiple \n
Solr uses Java regex matching, so i doubt there is a bug - it would then be in
the JDK. Try out in a regex online Tool that supports Java regex for your
solution.
I believe you want to have 2 regex process factories:
One that deals with single \n and one that deals with more than one \n
> Am 20.02.2019 um 06:17 schrieb Zheng Lin Edwin Yeo :
>
> Hi,
>
> We have tried with the following pattern ([ \t]*\r?\n){2,} and
> configuration:
>
>
> content
> ([ \t]*\r?\n){2,}
>
> true
>
>
> However, the issue is still occurring.
>
> Anyone else is able to help?
>
> Regards,
> Edwin
>
> On Fri, 15 Feb 2019 at 11:47, Zheng Lin Edwin Yeo
> wrote:
>
>> Hi,
>>
>> For your info, this issue is occurring in Solr 7.7.0 as well.
>>
>> Regards,
>> Edwin
>>
>> On Tue, 12 Feb 2019 at 00:10, Zheng Lin Edwin Yeo
>> wrote:
>>
>>> Hi,
>>>
>>> Should we report this as a bug in Solr?
>>>
>>> Regards,
>>> Edwin
>>>
>>> On Fri, 8 Feb 2019 at 22:18, Zheng Lin Edwin Yeo
>>> wrote:
>>>
Hi Paul,
Regarding the regex (\n\s*){2,} that we are using, when we try in on
https://regex101.com/, it is able to give us the correct result for all
the examples (ie: All of them will only have , and not more than
that like what we are getting in Solr in our earlier examples).
Could there be a possibility of a bug in Solr?
Regards,
Edwin
On Fri, 8 Feb 2019 at 00:33, Zheng Lin Edwin Yeo
wrote:
> Hi Paul,
>
> We have tried it with the space preceeding the \n i.e. name="pattern">(\s*\n){2,}, with the following regex pattern:
>
>
> content
> (\s*\n){2,}
>
>
>
> However, we are also getting the exact same results as the earlier
> Example 1, 2 and 3.
>
> As for your point 2 on perhaps in the data you have other (non
> printing) characters than \n, we have find that there are no non printing
> characters. It is just next line with a space. You can refer to the
> original content in the same examples below.
>
>
> Example 1: The sentence that the above regex pattern is working
> correctly
> *Original content in EML file:*
> Dear Sir,
>
>
> I am terminating
> *Original content:*Dear Sir, \n\n \n \n\n I am terminating
> *Index content: *Dear Sir, I am terminating
>
> Example 2: The sentence that the above regex pattern is partially
> working (as you can see, instead of 2 , there are 4 )
> *Original content in EML file:*
>
> *exalted*
>
> *Psalm 89:17*
>
>
> 3 Choa Chu Kang Avenue 4
> *Original content:* exalted \n \n\n Psalm 89:17 \n\n \n\n 3
> Choa Chu Kang Avenue 4, Singapore
> *Index content: *exalted Psalm 89:17 3
> Choa Chu Kang Avenue 4, Singapore
>
> Example 3: The sentence that the above regex pattern is partially
> working (as you can see, instead of 2 , there are 4 )
> *Original content in EML file:*
>
> http://www.concordpri.moe.edu.sg/
>
>
>
>
>
>
>
>
> On Tue, Dec 18, 2018 at 10:07 AM
> *Original content:* http://www.concordpri.moe.edu.sg/ \n\n \n\n \n
> \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n\n \n\n\n On Tue, Dec
> 18,
> 2018 at 10:07 AM
> *Index content: *http://www.concordpri.moe.edu.sg/
> On Tue, Dec 18, 2018 at 10:07 AM
>
>
> Appreciate any other ideas or suggestions that you may have.
>
> Thank you.
>
> Regards,
> Edwin
>
>> On Thu, 7 Feb 2019 at 22:49, wrote:
>>
>> Hi Edwin
>>
>>
>>
>> 1. Sorry, the pattern was wrong, the space should preceed the \n
>> i.e. (\s*\n){2,}
>> 2. Perhaps in the data you have other (non printing) characters
>> than \n?
>>
>>
>>
>> Gesendet von Mail für
>> Windows 10
>>
>>
>>
>> Von: Zheng Lin Edwin Yeo
>> Gesendet: Donnerstag, 7. Februar 2019 15:23
>> An: solr-user@lucene.apache.org
>> Betreff: Re: RegexReplaceProcessorFactory pattern to detect multiple \n
>>
>>
>>
>> Hi Paul,
>>
>> We have tried this suggested regex pattern as follow:
>>
>> content
>> (\n\s*){2,}
>>
>>
>>
>> But we still have exactly the same problem of Example 1,2 and 3 below.
>>
>> Example 1: The sentence that the above regex pattern is working
>> correctly
>> *Original content:*Dear Sir, \n\n \n \n\n I am terminating
>> *Index content: *Dear Sir, I am terminating
>>
>> Example 2: The sentence that the above regex pattern is partially
>> working
>> (as you can see, instead of 2 , there are 4 )
>> *Original content:* exal
Re: RegexReplaceProcessorFactory pattern to detect multiple \n
Hi,
We have tried with the following pattern ([ \t]*\r?\n){2,} and
configuration:
content
([ \t]*\r?\n){2,}
true
However, the issue is still occurring.
Anyone else is able to help?
Regards,
Edwin
On Fri, 15 Feb 2019 at 11:47, Zheng Lin Edwin Yeo
wrote:
> Hi,
>
> For your info, this issue is occurring in Solr 7.7.0 as well.
>
> Regards,
> Edwin
>
> On Tue, 12 Feb 2019 at 00:10, Zheng Lin Edwin Yeo
> wrote:
>
>> Hi,
>>
>> Should we report this as a bug in Solr?
>>
>> Regards,
>> Edwin
>>
>> On Fri, 8 Feb 2019 at 22:18, Zheng Lin Edwin Yeo
>> wrote:
>>
>>> Hi Paul,
>>>
>>> Regarding the regex (\n\s*){2,} that we are using, when we try in on
>>> https://regex101.com/, it is able to give us the correct result for all
>>> the examples (ie: All of them will only have , and not more than
>>> that like what we are getting in Solr in our earlier examples).
>>>
>>> Could there be a possibility of a bug in Solr?
>>>
>>> Regards,
>>> Edwin
>>>
>>> On Fri, 8 Feb 2019 at 00:33, Zheng Lin Edwin Yeo
>>> wrote:
>>>
Hi Paul,
We have tried it with the space preceeding the \n i.e. >>> name="pattern">(\s*\n){2,}, with the following regex pattern:
content
(\s*\n){2,}
However, we are also getting the exact same results as the earlier
Example 1, 2 and 3.
As for your point 2 on perhaps in the data you have other (non
printing) characters than \n, we have find that there are no non printing
characters. It is just next line with a space. You can refer to the
original content in the same examples below.
Example 1: The sentence that the above regex pattern is working
correctly
*Original content in EML file:*
Dear Sir,
I am terminating
*Original content:*Dear Sir, \n\n \n \n\n I am terminating
*Index content: *Dear Sir, I am terminating
Example 2: The sentence that the above regex pattern is partially
working (as you can see, instead of 2 , there are 4 )
*Original content in EML file:*
*exalted*
*Psalm 89:17*
3 Choa Chu Kang Avenue 4
*Original content:* exalted \n \n\n Psalm 89:17 \n\n \n\n 3
Choa Chu Kang Avenue 4, Singapore
*Index content: *exalted Psalm 89:17 3
Choa Chu Kang Avenue 4, Singapore
Example 3: The sentence that the above regex pattern is partially
working (as you can see, instead of 2 , there are 4 )
*Original content in EML file:*
http://www.concordpri.moe.edu.sg/
On Tue, Dec 18, 2018 at 10:07 AM
*Original content:* http://www.concordpri.moe.edu.sg/ \n\n \n\n \n
\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n\n \n\n\n On Tue, Dec 18,
2018 at 10:07 AM
*Index content: *http://www.concordpri.moe.edu.sg/
On Tue, Dec 18, 2018 at 10:07 AM
Appreciate any other ideas or suggestions that you may have.
Thank you.
Regards,
Edwin
On Thu, 7 Feb 2019 at 22:49, wrote:
> Hi Edwin
>
>
>
> 1. Sorry, the pattern was wrong, the space should preceed the \n
> i.e. (\s*\n){2,}
> 2. Perhaps in the data you have other (non printing) characters
> than \n?
>
>
>
> Gesendet von Mail für
> Windows 10
>
>
>
> Von: Zheng Lin Edwin Yeo
> Gesendet: Donnerstag, 7. Februar 2019 15:23
> An: solr-user@lucene.apache.org
> Betreff: Re: RegexReplaceProcessorFactory pattern to detect multiple \n
>
>
>
> Hi Paul,
>
> We have tried this suggested regex pattern as follow:
>
>content
>(\n\s*){2,}
>
>
>
> But we still have exactly the same problem of Example 1,2 and 3 below.
>
> Example 1: The sentence that the above regex pattern is working
> correctly
> *Original content:*Dear Sir, \n\n \n \n\n I am terminating
> *Index content: *Dear Sir, I am terminating
>
> Example 2: The sentence that the above regex pattern is partially
> working
> (as you can see, instead of 2 , there are 4 )
> *Original content:* exalted \n \n\n Psalm 89:17 \n\n \n\n 3
> Choa
> Chu Kang Avenue 4, Singapore
> *Index content: *exalted Psalm 89:17 3
> Choa
> Chu Kang Avenue 4, Singapore
>
> Example 3: The sentence that the above regex pattern is partially
> working
> (as you can see, instead of 2 , there are 4 )
> *Original content:* http://www.concordpri.moe.edu.sg/ \n\n \n\n
> \n \n\n
> \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n\n \n\n\n On Tue, Dec 18,
> 2018
> at 10:07 AM
> *Index content: *http://www.concordpri.moe.edu.sg/
> On
> Tue
Re: Newbie question - Error loading an existing config file
I used the front end admin (see attached) thanks On Tue, Feb 19, 2019 at 3:54 PM Erick Erickson wrote: > Hmmm, that’s not very helpful….. > > Don’t quite know what to say. There should be something more helpful > in the logs. > > Hmmm, How did you create the core? > > Best, > Erick > > > > On Feb 19, 2019, at 1:29 PM, Greg Robinson > wrote: > > > > Thanks for your direction regarding the log. > > > > I was able to locate it and these two lines stood out: > > > > Caused by: org.apache.solr.common.SolrException: Could not load conf for > > core new_solr_core: Error loading solr config from > > /home/solr/server/solr/new_solr_core/conf/solrconfig.xml > > > > Caused by: org.apache.solr.common.SolrException: Error loading solr > config > > from /home/solr/server/solr/new_solr_core/conf/solrconfig.xml > > > > which seems to point to the same issue. > > > > I also went ahead and updated permissions/owner to "solr" on all > > directories and files within "/home/solr/server/solr/new_solr_core". > > > > Still no luck. This is currently the same message that I'm getting on the > > admin front end: > > > > new_solr_core: > > > org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: > > Could not load conf for core new_solr_core: Error loading solr config > from > > /home/solr/server/solr/new_solr_core/conf/solrconfig.xml. > > > > thanks! > > > > > > > > On Tue, Feb 19, 2019 at 1:55 PM Erick Erickson > > wrote: > > > >> do a recursive seach for “solr.log" under SOLR_HOME……. > >> > >> Best, > >> ERick > >> > >>> On Feb 19, 2019, at 8:08 AM, Greg Robinson > >> wrote: > >>> > >>> Hi Erick, > >>> > >>> Thanks for the quick response. > >>> > >>> Here is what is currently contained within the conf dir: > >>> > >>> drwxr-xr-x 2 root root 4096 Feb 18 17:51 lang > >>> -rw-r--r-- 1 root root 54513 Feb 18 17:51 managed-schema > >>> -rw-r--r-- 1 root root 329 Feb 18 17:51 params.json > >>> -rw-r--r-- 1 root root 894 Feb 18 17:51 protwords.txt > >>> -rwxrwxrwx 1 root root 55323 Feb 18 17:51 solrconfig.xml > >>> -rw-r--r-- 1 root root 795 Feb 18 17:51 stopwords.txt > >>> -rw-r--r-- 1 root root 1153 Feb 18 17:51 synonyms.txt > >>> > >>> As far as the log, where exactly might I find the specific log that > would > >>> give more info in regards to this error? > >>> > >>> thanks again! > >>> > >>> On Tue, Feb 19, 2019 at 9:06 AM Erick Erickson < > erickerick...@gmail.com> > >>> wrote: > >>> > Are all the other files there in your conf dir? Solrconfig.xml > >> references > things like nanaged-schema etc. > > Also, your log file might contain more clues... > > On Tue, Feb 19, 2019, 08:03 Greg Robinson >> wrote: > > > Hello, > > > > We have Solr 7.4 up and running on a Linux machine. > > > > I'm just trying to add a new core so that I can eventually point a > >> Drupal > > site to the Solr Server for indexing. > > > > When attempting to add a core, I'm getting the following error: > > > > new_solr_core: > > > > >> > org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: > > Could not load conf for core new_solr_core: Error loading solr config > from > > /home/solr/server/solr/new_solr_core/conf/solrconfig.xml > > > > I've confirmed that > > /home/solr/server/solr/new_solr_core/conf/solrconfig.xml exists but > I'm > > still getting the error. > > > > Any direction is appreciated. > > > > Thanks! > > > > >>> > >>> > >>> -- > >>> Greg Robinson > >>> CEO - Mobile*Enhanced* > >>> www.mobileenhanced.com > >>> g...@mobileenhanced.com > >>> 303-598-1865 > >> > >> > > > > -- > > Greg Robinson > > CEO - Mobile*Enhanced* > > www.mobileenhanced.com > > g...@mobileenhanced.com > > 303-598-1865 > > -- Greg Robinson CEO - Mobile*Enhanced* www.mobileenhanced.com g...@mobileenhanced.com 303-598-1865
Re: Newbie question - Error loading an existing config file
Hmmm, that’s not very helpful….. Don’t quite know what to say. There should be something more helpful in the logs. Hmmm, How did you create the core? Best, Erick > On Feb 19, 2019, at 1:29 PM, Greg Robinson wrote: > > Thanks for your direction regarding the log. > > I was able to locate it and these two lines stood out: > > Caused by: org.apache.solr.common.SolrException: Could not load conf for > core new_solr_core: Error loading solr config from > /home/solr/server/solr/new_solr_core/conf/solrconfig.xml > > Caused by: org.apache.solr.common.SolrException: Error loading solr config > from /home/solr/server/solr/new_solr_core/conf/solrconfig.xml > > which seems to point to the same issue. > > I also went ahead and updated permissions/owner to "solr" on all > directories and files within "/home/solr/server/solr/new_solr_core". > > Still no luck. This is currently the same message that I'm getting on the > admin front end: > > new_solr_core: > org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: > Could not load conf for core new_solr_core: Error loading solr config from > /home/solr/server/solr/new_solr_core/conf/solrconfig.xml. > > thanks! > > > > On Tue, Feb 19, 2019 at 1:55 PM Erick Erickson > wrote: > >> do a recursive seach for “solr.log" under SOLR_HOME……. >> >> Best, >> ERick >> >>> On Feb 19, 2019, at 8:08 AM, Greg Robinson >> wrote: >>> >>> Hi Erick, >>> >>> Thanks for the quick response. >>> >>> Here is what is currently contained within the conf dir: >>> >>> drwxr-xr-x 2 root root 4096 Feb 18 17:51 lang >>> -rw-r--r-- 1 root root 54513 Feb 18 17:51 managed-schema >>> -rw-r--r-- 1 root root 329 Feb 18 17:51 params.json >>> -rw-r--r-- 1 root root 894 Feb 18 17:51 protwords.txt >>> -rwxrwxrwx 1 root root 55323 Feb 18 17:51 solrconfig.xml >>> -rw-r--r-- 1 root root 795 Feb 18 17:51 stopwords.txt >>> -rw-r--r-- 1 root root 1153 Feb 18 17:51 synonyms.txt >>> >>> As far as the log, where exactly might I find the specific log that would >>> give more info in regards to this error? >>> >>> thanks again! >>> >>> On Tue, Feb 19, 2019 at 9:06 AM Erick Erickson >>> wrote: >>> Are all the other files there in your conf dir? Solrconfig.xml >> references things like nanaged-schema etc. Also, your log file might contain more clues... On Tue, Feb 19, 2019, 08:03 Greg Robinson > wrote: > Hello, > > We have Solr 7.4 up and running on a Linux machine. > > I'm just trying to add a new core so that I can eventually point a >> Drupal > site to the Solr Server for indexing. > > When attempting to add a core, I'm getting the following error: > > new_solr_core: > >> org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: > Could not load conf for core new_solr_core: Error loading solr config from > /home/solr/server/solr/new_solr_core/conf/solrconfig.xml > > I've confirmed that > /home/solr/server/solr/new_solr_core/conf/solrconfig.xml exists but I'm > still getting the error. > > Any direction is appreciated. > > Thanks! > >>> >>> >>> -- >>> Greg Robinson >>> CEO - Mobile*Enhanced* >>> www.mobileenhanced.com >>> g...@mobileenhanced.com >>> 303-598-1865 >> >> > > -- > Greg Robinson > CEO - Mobile*Enhanced* > www.mobileenhanced.com > g...@mobileenhanced.com > 303-598-1865
Re: UpdateHandler batch size / search solr-user
Sending batches in parallel is perfectly fine. _However_, if you’re updating the same document, there’s no guarantee which would win. Imagine you have two processes sending batches. The order of execution depends on way too many variables. If nothing else, if process 1 sends a document then some time later process 2 sends the same document, the one from process2 would “win”. The optimistic locking scenario wouldn’t come into the picture unless you took control of assigning the _version_ number. Best, Erick > On Feb 19, 2019, at 9:23 AM, David '-1' Schmid wrote: > > Hi! > > On 2019-02-18T20:36:35, Erick Erickson wrote: >> Typically, people set their autocommit (hard) settings in >> solrconfig.xml and forget about it. I usually use a time-based trigger >> and don’t use documents as a trigger. > I added a timed autoCommit and it seems to work out nicely. Thank you! > >> Until you do a hard commit, all the incoming documents are held in the >> transaction log, > Ah, yes. Somehow I did not draw the link to transactions. > I've noticed that solr is using only one of my four CPUs for applying > the update. With that in mind, could I submit my batches in parallel, > or would that be worse? To be honest, I've never seen what kind of > transaction or coherency model is used in solr. > > I think it's touched briefly by the solr-ref-guide for applying updates > to single document fields; but I can't say for sure if it's using an > optimistic strategy or if the parallel updates would produce more > overhead by pessimistic locking. > > regards, > =1
Re: Newbie question - Error loading an existing config file
Thanks for your direction regarding the log. I was able to locate it and these two lines stood out: Caused by: org.apache.solr.common.SolrException: Could not load conf for core new_solr_core: Error loading solr config from /home/solr/server/solr/new_solr_core/conf/solrconfig.xml Caused by: org.apache.solr.common.SolrException: Error loading solr config from /home/solr/server/solr/new_solr_core/conf/solrconfig.xml which seems to point to the same issue. I also went ahead and updated permissions/owner to "solr" on all directories and files within "/home/solr/server/solr/new_solr_core". Still no luck. This is currently the same message that I'm getting on the admin front end: new_solr_core: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Could not load conf for core new_solr_core: Error loading solr config from /home/solr/server/solr/new_solr_core/conf/solrconfig.xml. thanks! On Tue, Feb 19, 2019 at 1:55 PM Erick Erickson wrote: > do a recursive seach for “solr.log" under SOLR_HOME……. > > Best, > ERick > > > On Feb 19, 2019, at 8:08 AM, Greg Robinson > wrote: > > > > Hi Erick, > > > > Thanks for the quick response. > > > > Here is what is currently contained within the conf dir: > > > > drwxr-xr-x 2 root root 4096 Feb 18 17:51 lang > > -rw-r--r-- 1 root root 54513 Feb 18 17:51 managed-schema > > -rw-r--r-- 1 root root 329 Feb 18 17:51 params.json > > -rw-r--r-- 1 root root 894 Feb 18 17:51 protwords.txt > > -rwxrwxrwx 1 root root 55323 Feb 18 17:51 solrconfig.xml > > -rw-r--r-- 1 root root 795 Feb 18 17:51 stopwords.txt > > -rw-r--r-- 1 root root 1153 Feb 18 17:51 synonyms.txt > > > > As far as the log, where exactly might I find the specific log that would > > give more info in regards to this error? > > > > thanks again! > > > > On Tue, Feb 19, 2019 at 9:06 AM Erick Erickson > > wrote: > > > >> Are all the other files there in your conf dir? Solrconfig.xml > references > >> things like nanaged-schema etc. > >> > >> Also, your log file might contain more clues... > >> > >> On Tue, Feb 19, 2019, 08:03 Greg Robinson wrote: > >> > >>> Hello, > >>> > >>> We have Solr 7.4 up and running on a Linux machine. > >>> > >>> I'm just trying to add a new core so that I can eventually point a > Drupal > >>> site to the Solr Server for indexing. > >>> > >>> When attempting to add a core, I'm getting the following error: > >>> > >>> new_solr_core: > >>> > >> > org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: > >>> Could not load conf for core new_solr_core: Error loading solr config > >> from > >>> /home/solr/server/solr/new_solr_core/conf/solrconfig.xml > >>> > >>> I've confirmed that > >>> /home/solr/server/solr/new_solr_core/conf/solrconfig.xml exists but I'm > >>> still getting the error. > >>> > >>> Any direction is appreciated. > >>> > >>> Thanks! > >>> > >> > > > > > > -- > > Greg Robinson > > CEO - Mobile*Enhanced* > > www.mobileenhanced.com > > g...@mobileenhanced.com > > 303-598-1865 > > -- Greg Robinson CEO - Mobile*Enhanced* www.mobileenhanced.com g...@mobileenhanced.com 303-598-1865
Re: Newbie question - Error loading an existing config file
do a recursive seach for “solr.log" under SOLR_HOME……. Best, ERick > On Feb 19, 2019, at 8:08 AM, Greg Robinson wrote: > > Hi Erick, > > Thanks for the quick response. > > Here is what is currently contained within the conf dir: > > drwxr-xr-x 2 root root 4096 Feb 18 17:51 lang > -rw-r--r-- 1 root root 54513 Feb 18 17:51 managed-schema > -rw-r--r-- 1 root root 329 Feb 18 17:51 params.json > -rw-r--r-- 1 root root 894 Feb 18 17:51 protwords.txt > -rwxrwxrwx 1 root root 55323 Feb 18 17:51 solrconfig.xml > -rw-r--r-- 1 root root 795 Feb 18 17:51 stopwords.txt > -rw-r--r-- 1 root root 1153 Feb 18 17:51 synonyms.txt > > As far as the log, where exactly might I find the specific log that would > give more info in regards to this error? > > thanks again! > > On Tue, Feb 19, 2019 at 9:06 AM Erick Erickson > wrote: > >> Are all the other files there in your conf dir? Solrconfig.xml references >> things like nanaged-schema etc. >> >> Also, your log file might contain more clues... >> >> On Tue, Feb 19, 2019, 08:03 Greg Robinson > >>> Hello, >>> >>> We have Solr 7.4 up and running on a Linux machine. >>> >>> I'm just trying to add a new core so that I can eventually point a Drupal >>> site to the Solr Server for indexing. >>> >>> When attempting to add a core, I'm getting the following error: >>> >>> new_solr_core: >>> >> org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: >>> Could not load conf for core new_solr_core: Error loading solr config >> from >>> /home/solr/server/solr/new_solr_core/conf/solrconfig.xml >>> >>> I've confirmed that >>> /home/solr/server/solr/new_solr_core/conf/solrconfig.xml exists but I'm >>> still getting the error. >>> >>> Any direction is appreciated. >>> >>> Thanks! >>> >> > > > -- > Greg Robinson > CEO - Mobile*Enhanced* > www.mobileenhanced.com > g...@mobileenhanced.com > 303-598-1865
Atomic updates in streaming expressions?
Hi,
Solr supports atomic updates as described at
https://lucene.apache.org/solr/guide/7_6/updating-parts-of-documents.html#atomic-updates
But I wonder how to create a streaming expression that does atomic updates. We
want to search for documents matching a given criteria and update a particular
of them.
If I do a „full“ update it works, but I have no idea about how to run an atomic
update.
For example I try to set all fieldB to „foo“, this full update works
commit(
update(
select(
export( … ),
id,
fieldA,
val(foo) as fieldB
)
)
Something like this doesn’t:
commit(
update(
select(
export( … ),
id,
val({„set“: foo}) as fieldB
)
)
Any ideas?
Markus
Re: UpdateHandler batch size / search solr-user
Hi! On 2019-02-18T20:36:35, Erick Erickson wrote: > Typically, people set their autocommit (hard) settings in > solrconfig.xml and forget about it. I usually use a time-based trigger > and don’t use documents as a trigger. I added a timed autoCommit and it seems to work out nicely. Thank you! > Until you do a hard commit, all the incoming documents are held in the > transaction log, Ah, yes. Somehow I did not draw the link to transactions. I've noticed that solr is using only one of my four CPUs for applying the update. With that in mind, could I submit my batches in parallel, or would that be worse? To be honest, I've never seen what kind of transaction or coherency model is used in solr. I think it's touched briefly by the solr-ref-guide for applying updates to single document fields; but I can't say for sure if it's using an optimistic strategy or if the parallel updates would produce more overhead by pessimistic locking. regards, =1
Re: Newbie question - Error loading an existing config file
Hi Erick, Thanks for the quick response. Here is what is currently contained within the conf dir: drwxr-xr-x 2 root root 4096 Feb 18 17:51 lang -rw-r--r-- 1 root root 54513 Feb 18 17:51 managed-schema -rw-r--r-- 1 root root 329 Feb 18 17:51 params.json -rw-r--r-- 1 root root 894 Feb 18 17:51 protwords.txt -rwxrwxrwx 1 root root 55323 Feb 18 17:51 solrconfig.xml -rw-r--r-- 1 root root 795 Feb 18 17:51 stopwords.txt -rw-r--r-- 1 root root 1153 Feb 18 17:51 synonyms.txt As far as the log, where exactly might I find the specific log that would give more info in regards to this error? thanks again! On Tue, Feb 19, 2019 at 9:06 AM Erick Erickson wrote: > Are all the other files there in your conf dir? Solrconfig.xml references > things like nanaged-schema etc. > > Also, your log file might contain more clues... > > On Tue, Feb 19, 2019, 08:03 Greg Robinson > > Hello, > > > > We have Solr 7.4 up and running on a Linux machine. > > > > I'm just trying to add a new core so that I can eventually point a Drupal > > site to the Solr Server for indexing. > > > > When attempting to add a core, I'm getting the following error: > > > > new_solr_core: > > > org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: > > Could not load conf for core new_solr_core: Error loading solr config > from > > /home/solr/server/solr/new_solr_core/conf/solrconfig.xml > > > > I've confirmed that > > /home/solr/server/solr/new_solr_core/conf/solrconfig.xml exists but I'm > > still getting the error. > > > > Any direction is appreciated. > > > > Thanks! > > > -- Greg Robinson CEO - Mobile*Enhanced* www.mobileenhanced.com g...@mobileenhanced.com 303-598-1865
Re: Newbie question - Error loading an existing config file
Are all the other files there in your conf dir? Solrconfig.xml references things like nanaged-schema etc. Also, your log file might contain more clues... On Tue, Feb 19, 2019, 08:03 Greg Robinson Hello, > > We have Solr 7.4 up and running on a Linux machine. > > I'm just trying to add a new core so that I can eventually point a Drupal > site to the Solr Server for indexing. > > When attempting to add a core, I'm getting the following error: > > new_solr_core: > org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: > Could not load conf for core new_solr_core: Error loading solr config from > /home/solr/server/solr/new_solr_core/conf/solrconfig.xml > > I've confirmed that > /home/solr/server/solr/new_solr_core/conf/solrconfig.xml exists but I'm > still getting the error. > > Any direction is appreciated. > > Thanks! >
Newbie question - Error loading an existing config file
Hello, We have Solr 7.4 up and running on a Linux machine. I'm just trying to add a new core so that I can eventually point a Drupal site to the Solr Server for indexing. When attempting to add a core, I'm getting the following error: new_solr_core: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Could not load conf for core new_solr_core: Error loading solr config from /home/solr/server/solr/new_solr_core/conf/solrconfig.xml I've confirmed that /home/solr/server/solr/new_solr_core/conf/solrconfig.xml exists but I'm still getting the error. Any direction is appreciated. Thanks!
graph query parser: depth dependent score?
Dear reader,
I'll have a hierarchical graph "like a book":
{ id:solr_doc1; title:book }
{ id:solr_doc2; title:chapter; parent_ids: solr_doc1 }
{ id:solr_doc3; title:subchapter; parent_ids: solr_doc2 }
etc.
Now to match all docs with "title" and "chapter" I could do:
+_query_:"{!graph from=parent_ids to=id}title:book"
+_query_:"{!graph from=parent_ids to=id}title:chapter",
The result would be solr_doc2 and solr_doc3;
but is there a way to "boost" or "put a higher score" on solr_doc2 than
on solr_doc3 because of direct match (and not via {!graph... ) ?
The only way to do so seems a {!boost before {!graph, but what I can do
there is not dependent on the match nor {!graph, I think.
Kind regards,
Jochen
--
Jochen Barth * Universitätsbibliothek Heidelberg, IT * Telefon 06221 54-2580
RE: Re: Delayed/waiting requests
Quick update just in case someone comes on this thread someday: we did lower the autowarm but it didn't have effect on the performance issues we are seeing. We are still investigating... Regards, Gaël De : Gael Jourdan-Weil Envoyé : mardi 15 janvier 2019 18:33 À : solr-user@lucene.apache.org Objet : RE: Re: Delayed/waiting requests @Erick: We will try to lower the autowarm and run some tests to compare. If I get your point, having a big cache might cause more troubles than help if the cache hit ratio is not high enough because the cache is constantly evicting/inserting entries? @Jeremy: Index size: ~20G and ~14M documents Server memory available: 256G from which ~30G used and ~100G system cache Server CPU count: 32, ~10% usage JVM memory settings: -Xms12G -Xmx12G We have 3 servers and 3 clusters of 3 Solr instances. That is each server hosts 1 Solr instance for each cluster. And, indeed, each cluster only has 1 shard with replication factor 3. Among all these Solr instances, the pauses are observed on only one single cluster but on every server at different times (sometimes on all servers at the same time but I would say it's very rare). We do observe the traffic is evenly balanced across the 3 servers, around 30-40 queries per second sent to each server. Regards, Gaël De : Branham, Jeremy (Experis) Envoyé : mardi 15 janvier 2019 17:59:56 À : solr-user@lucene.apache.org Objet : Re: Re: Delayed/waiting requests Hi Gael – Could you share this information? Size of the index Server memory available Server CPU count JVM memory settings You mentioned a cloud configuration of 3 replicas. Does that mean you have 1 shard with a replication factor of 3? Do the pauses occur on all 3 servers? Is the traffic evenly balanced across those servers? Jeremy Branham jb...@allstate.com On 1/15/19, 9:50 AM, "Erick Erickson" wrote: Well, it was a nice theory anyway. "Other collections with the same settings" doesn't really mean much unless those other collections are very similar, especially in terms of numbers of docs. You should only see a new searcher opening when you do a hard-commit-with-opensearcher-true or soft commit. So what happens when you just try lowering the autowarm count? I'm assuming you're free to test in some non-prod system. Focusing on the hit ratio is something of a red herring. Remember that each entry in your filterCache is roughly maxDoc/8 + a little overhead, the increase in GC pressure has to be balanced against getting the hits from the cache. Now, all that said if there's no correlation, then you need to put a profiler on the system when you see this kind of thing and find out where the hotspots are, otherwise it's guesswork and I'm out of ideas. Best, Erick On Tue, Jan 15, 2019 at 12:06 AM Gael Jourdan-Weil wrote: > > Hi Erick, > > > Thank you for your detailed answer, I better understand autowarming. > > > We have an autowarming time of ~10s for filterCache (queryResultCache is not used at all, ratio = 0.02). > > We increased the size of the filterCache from 6k to 12k (and autowarming size set to same values) to have a better ratio which is _only_ around 0.85/0.90. > > > The thing I don't understand is I should see "Opening new searcher" in the logs everytime a new searcher is opened and thus an autowarming happens, right? > > But I don't see "Opening new searcher" very often, and I don't see it being correlated with the response time peaks. > > > Also, I didn't mention it earlier but, we have other SolrCloud clusters with similar settings and load (~10s filterCache autowarming, 10k entries) and we don't observe the same behavior. > > > Regards, > > > De : Erick Erickson > Envoyé : lundi 14 janvier 2019 17:44:38 > À : solr-user > Objet : Re: Delayed/waiting requests > > Gael: > > bq. Nevertheless, our filterCache is set to autowarm 12k entries which > is also the maxSize > > That is far, far, far too many. Let's assume you actually have 12K > entries in the filterCache. > Every time you open a new searcher, 12K queries are executed _before_ > the searcher > accepts any new requests. While being able to re-use a filterCache > entry is useful, one of > the primary purposes is to pre-load index data from disk into memory > which can be > the event that takes the most time. > > The queryResultCache has a similar function. I often find that this > cache doesn't have a > very high hit ratio, but again executing a _few_ of these queries > warms the index from > disk. > > I think of both caches as a map, where the key is the "thing", (fq > clause in the case >
Re: only error logging in solr
After looking into the source code there seams nothing in there for error logging together with the request which produced the error. I think there is a need for this to log the request along with the error. Could be done at o.a.s.core.SolrCore.execute() where the INFO logging is also located. And the response from o.a.s.handler.RequestHandlerBase.handleRequest() is setting rsp.setException(e) which could be used to select logging only requests which produced an ERROR. Are there any opinions about this? Regards Bernd Am 18.02.19 um 14:43 schrieb Bernd Fehling: Hi list, logging in solr sounds easy but the problem is logging only errors and the request which produced the error. I want to log all 4xx and 5xx http and also solr ERROR. My request_logs from jetty show nothing useful because of POST requests. Only that a request got HTTP 4xx or 5xx from solr. INFO log level for solr_logs is not used because of to much log writing at high QPS. My solr_logs should report ERRORs the request which produced the ERROR. Has anyone an idea or solved this problem? Is it possible to raise the level of a request from INFO to ERROR if the request produced an ERROR in solr_logs? Regards Bernd
RE: solr cloud version upgrade 7.6 to 7.7 collection indexes all marked as down
Hello, We just witnessed this too with 7.7. No no obvious messages in the logs, the replica status would not come out of 'down'. Meanwhile we got another weird exception from a neighbouring collection sharing the same nodes: 2019-02-18 13:47:20.622 ERROR (updateExecutor-3-thread-1-processing-n:idx1:8983_solr x:search_20180717_shard1_replica_t81 c:search_20180717 s:shard1 r:core_node82 ) [c:search_20180717 s:shard1 r:core_node82 x:search_20180717_shard1_replica_t81] o.a.s.u.SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Er ror from server at http://idx5:8983/solr/search_20180717_shard1_replica_t91: invalid boolean value: replicas at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient.request(ConcurrentUpdateSolrClient.java:491) at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260) at org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:326) at org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:315) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Although probably not related, is this a known thing? Or shall i open an issue. Thanks, Markus -Original message- > From:Jeff Courtade > Sent: Friday 15th February 2019 21:54 > To: solr-user@lucene.apache.org > Subject: Re: solr cloud version upgrade 7.6 to 7.7 collection indexes all > marked as down > > Yes... nothing in the logs does mean that there was nothing of interest. I > have actual entries. > > This is a test environment so this isn't an emergency. Thanks for the > clarification about what I should be seeing. > > I was just so flabbergasted by this because it's so strange I had to tell > somebody and yell at the universe basically so I yelled at the solar > mailing list. > > This is an automated upgrading so the next step is to go through and > manually perform all the steps and see if I get the same behavior. > > I am fairly certain I just going to be some dumb thing that I'm doing and I > will be happy to update the mailing list when I figure this out for > everyone's Mutual entertainment. > -- > Jeff Courtade > M: 240.507.6116 > > On Fri, Feb 15, 2019, 12:33 PM Erick Erickson wrote: > > > Hmmm. I'm assuming that "nothing in the logs" is node/logs/solr.log, and > > that > > you're not finding errors/exceptipons. Just sanity checking here. > > > > My guess: you're picking up the default SOLR_HOME which is in your new > > installation directory and all your > > replicas are under the old install directory. > > > > There should be some kind of message in the log files indicating that > > Solr is at least trying to load replicas, something similar to: > > > > Using system property solr.solr.home: > > /Users/Erick/apache/solrVersions/playspace/solr/example/cloud/node1/solr > > > > and/or: > > > > CorePropertiesLocator Found 3 core definitions underneath > > /Users/Erick/apache/solrVersions/playspace/solr/example/cloud/node1/solr > > > > A bit of background: When Solr starts up, it recursively descends from > > SOLR_HOME and whenever it finds a "core.properties" file > > it says "Aha, this must be a core, I'll try to load it". So if > > SOLR_HOME is doesn't point to an ancestor of your existing replicas, > > Solr won't find any replicas and everything will stay down. _If_ > > SOLR_HOME is defined in solr.in.sh, this should just be picked up. > > > > Best, > > Erick > > > > On Thu, Feb 14, 2019 at 7:43 PM Zheng Lin Edwin Yeo > > wrote: > > > > > > Hi, > > > > > > Which version of zookeeper are you using? > > > > > > Also, if you tried to query the index, did you get any error message? > > > > > > Regards, > > > Edwin > > > > > > > > > On Fri, 15 Feb 2019 at 02:34, Jeff Courtade > > wrote: > > > > > > > Hi, > > > > > > > > I am working n doing a simple point upgrade from solr 7.6 to 7.7 cloud. > > > > > > > > 6 servers > > > > 3 zookeepers > > > > one simple test collection using the prepackages _default config. > > > > > > > > i stop all solr servers le
