Creating Solr cores dynamically with config files using solrj

[Gimantha Bandara]

Hi all,

I have requirement where multiple solr cores need to be created with
different schema. Using CoreAdminAPI, I can create a core dynamically but,
I have to create a folder and copy the schema.xml and solrconfig.xml there
before invoking the API. Is there a way to create solr cores with some kind
of default configurations using solrj?

I tried using configSets, but it seems all the cores are referring to the
same ConfigSet and changes done to the schema of the cores, reflect on the
configSet, which will ultimately change the schema of all the cores.

Any help is appreciated.

Thanks,
Gimantha

Re: error while working solr 6.2 cdcr

[Neeraj Bhatt]

Hi Eric

Source have only target zookeeper ensemble (only 3 ips)
  192.169.35.53:2181,192.169.35.249:2181,
192.169.35.251:2181> , because of some mail formatting ip were repeated(see
sent item attached image). Error in log files is
Action LASTPROCESSEDVERSION sent to non-leader replica at
org.apache.solr.handler.CdcrRequestHandler.handleLastProcess
edVersionAction(CdcrRequestHandler.java:498)
I will start with empty collection with  3 nodes 3 shards one replication
factor, It seems from log that request from souce is going to non leader
replica.

Please suggest

Thanks



On Wed, Nov 9, 2016 at 9:23 PM, Erick Erickson 
wrote:

> Why do you have 6 zookeepers listed in the zkHost parameter? My guess
> is that you've listed the ZK addresses for both the source and target
> clusters. This should only be the ZK nodes for the target cluster and
> only present on the source.
>
> The source already knows its own ZK addresses, it needs to know about
> the target cluster in order to know what nodes it should send updates
> to.
>
> Best,
> Erick
>
> On Wed, Nov 9, 2016 at 4:44 AM, Neeraj Bhatt 
> wrote:
> > Hello All
> >
> > I have a source cluster (3 nodes, 3 shards and 3 replica) and same for
> > target cluster. On posting of documents to source I am getting below
> error
> > in target solr log files when i turned log to debug level
> >
> >
> >
> >
> > *6538 ERROR (qtp110456297-20) [c:multi_dc_poc s:shard1 r:core_node3
> > x:multi_dc_poc_shard1_replica2] o.a.s.h.RequestHandlerBase
> > org.apache.solr.common.SolrException: Action LASTPROCESSEDVERSION sent
> to
> > non-leader replicaat
> > org.apache.solr.handler.CdcrRequestHandler.handleLastProcess
> edVersionAction(CdcrRequestHandler.java:498)*
> >
> > *should I reconfigure my replicas to one in target ?*
> >
> > solr config of source and target is same as given in wiki
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pa
> geId=62687462
> >
> > *source *
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > * > class="solr.CdcrRequestHandler">  > name="replica">  > name="zkHost">192.169.35.53:2181
> > ,192.169.35.249:2181
> > ,192.169.35.251:2181
> > 
> >   > name="source">multi_dc_poc  > name="target">multi_dc_poc
>  > name="replicator"> > name="threadPoolSize">9 > name="schedule">1000 > name="batchSize">128 > name="updateLogSynchronizer"> > name="schedule">1000  
> *
> >
> >
> >
> >
> >
> >
> >
> > *source and target update log   > class="solr.CdcrUpdateLog">   > name="dir">${solr.ulog.dir:}   > name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}
> > *
> >
> >
> > *target solr config*
> >
> > 
> >   
> > disabled
> >   
> > 
> >
> > 
> >   
> > cdcr-processor-chain
> >   
> > 
> >
> > 
> >   
> >   
> > 
> >
> > *target status*
> >
> > 0 > name="QTime">0 > name="process">stopped > name="buffer">disabled
> >
> > *source status*
> >
> > 0 > name="QTime">0 > name="process">started > name="buffer">enabled
>

Re: Basic Auth for Solr Streaming Expressions

[Joel Bernstein]

Thanks for digging into this, let's create a jira ticket for this.

Joel Bernstein
http://joelsolr.blogspot.com/

On Wed, Nov 9, 2016 at 6:23 PM, sandeep mukherjee <
wiredcit...@yahoo.com.invalid> wrote:

> I have more progress since my last mail. I figured out that  in the
> StreamContext object there is a way to set the SolrClientCache object which
> keep reference to all the CloudSolrClient where I can set a reference to
> HttpClient which sets the Basic Auth header. However the problem is, inside
> the SolrClientCache there is no way to set your own version of
> CloudSolrClient with BasicAuth enabled. Unfortunately, SolrClientCache has
> no set method which takes a CloudSolrClient object.
> So long story short we need an API in SolrClientCache to
> accept CloudSolrClient object from user.
> Please let me know if there is a better way to enable Basic Auth when
> using StreamFactory as mentioned in my previous email.
> Thanks much,Sandeep
>
> On Wednesday, November 9, 2016 11:44 AM, sandeep mukherjee
>  wrote:
>
>
>  Hello everyone,
> I trying to find the documentation for Basic Auth plugin for Solr
> Streaming expressions. But I'm not able to find it in the documentation
> anywhere. Could you please point me in right direction of how to enable
> Basic auth for Solr Streams?
> I'm creating StreamFactory as follows: I wonder how and where can I
> specify Basic Auth username and password
> @Bean
> public StreamFactory streamFactory() {
> SolrConfig solrConfig = ConfigManager.getNamedConfig("solr",
> SolrConfig.class);
>
> return new StreamFactory().withDefaultZkHost(solrConfig.
> getConnectString())
> .withFunctionName("gatherNodes", GatherNodesStream.class);
> }
>
>
>

Re: Basic Auth for Solr Streaming Expressions

[sandeep mukherjee]

I have more progress since my last mail. I figured out that  in the 
StreamContext object there is a way to set the SolrClientCache object which 
keep reference to all the CloudSolrClient where I can set a reference to 
HttpClient which sets the Basic Auth header. However the problem is, inside the 
SolrClientCache there is no way to set your own version of CloudSolrClient with 
BasicAuth enabled. Unfortunately, SolrClientCache has no set method which takes 
a CloudSolrClient object.
So long story short we need an API in SolrClientCache to accept CloudSolrClient 
object from user.
Please let me know if there is a better way to enable Basic Auth when using 
StreamFactory as mentioned in my previous email.
Thanks much,Sandeep 

On Wednesday, November 9, 2016 11:44 AM, sandeep mukherjee 
 wrote:
 

 Hello everyone,
I trying to find the documentation for Basic Auth plugin for Solr Streaming 
expressions. But I'm not able to find it in the documentation anywhere. Could 
you please point me in right direction of how to enable Basic auth for Solr 
Streams?
I'm creating StreamFactory as follows: I wonder how and where can I specify 
Basic Auth username and password
@Bean
public StreamFactory streamFactory() {
    SolrConfig solrConfig = ConfigManager.getNamedConfig("solr", 
SolrConfig.class);

    return new StreamFactory().withDefaultZkHost(solrConfig.getConnectString())
            .withFunctionName("gatherNodes", GatherNodesStream.class);
}

   

Re: solrj Https problem

[sandeep mukherjee]

Thanks Bryan for the response. That seem to have solved it. 

On Monday, October 31, 2016 6:58 PM, Bryan Bende  wrote:
 

 A possible problem might be that your certificate was generated for
"localhost" which is why it works when you go to https://localhost:8985/solr
in your browser, but when SolrJ gets the cluster information from ZooKeeper
the hostnames of the Solr nodes might be using an IP address which won't
work when the SSL/TLS negotiation happens.

If this is the problem you will want to specify the hostname for Solr to
use when starting each node by passing "-h localhost".

-Bryan

On Mon, Oct 31, 2016 at 1:05 PM, sandeep mukherjee <
wiredcit...@yahoo.com.invalid> wrote:

> I followed the steps to make the solr SSL enabled. I'm able to hit solr
> at: https://localhost:8985/solr/problem/select?indent=on=*:*=json And
> for accessing it through Solr Client I created it as
> follows:System.setProperty("javax.net.ssl.keyStore",
> "/path/to/solr/server/etc/solr-ssl.keystore.jks");
> System.setProperty("javax.net.ssl.keyStorePassword", "secret");
> System.setProperty("javax.net.ssl.trustStore", "/path/to/solr/server/etc/
> solr-ssl.keystore.jks");
> System.setProperty("javax.net.ssl.trustStorePassword", "secret");
> return new CloudSolrClient.Builder()
>        .withZkHost(solrConfig.getConnectString()).build(); The path to
> the keystore and truststore is correct.  However I still get the following
> error:Caused by: javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to
> find valid certification path to requested target
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_45]
> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1937)
> ~[na:1.8.0_45]
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[na:1.8.0_45]
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[na:1.8.0_45]
> at 
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1478)
> ~[na:1.8.0_45]
> at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212)
> ~[na:1.8.0_45]
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
> ~[na:1.8.0_45]
> at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
> ~[na:1.8.0_45]
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1050)
> ~[na:1.8.0_45]
> at 
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363)
> ~[na:1.8.0_45]
> at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1391)
> ~[na:1.8.0_45]
> at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1375)
> ~[na:1.8.0_45]
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
> ~[httpclient-4.5.1.jar:4.5.1]
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
> ~[httpclient-4.5.1.jar:4.5.1]
> at org.apache.http.impl.conn.DefaultClientConnectionOperato
> r.openConnection(DefaultClientConnectionOperator.java:177)
> ~[httpclient-4.5.1.jar:4.5.1]
> at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(
> ManagedClientConnectionImpl.java:304) ~[httpclient-4.5.1.jar:4.5.1]
> at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(
> DefaultRequestDirector.java:611) ~[httpclient-4.5.1.jar:4.5.1]
> at org.apache.http.impl.client.DefaultRequestDirector.execute(
> DefaultRequestDirector.java:446) ~[httpclient-4.5.1.jar:4.5.1]
> at 
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> ~[httpclient-4.5.1.jar:4.5.1]
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> ~[httpclient-4.5.1.jar:4.5.1]
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
> ~[httpclient-4.5.1.jar:4.5.1]
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> ~[httpclient-4.5.1.jar:4.5.1]
> at org.apache.solr.client.solrj.impl.HttpSolrClient.
> executeMethod(HttpSolrClient.java:495) ~[solr-solrj-6.1.0.jar:6.1.0
> 4726c5b2d2efa9ba160b608d46a977d0a6b83f94 - jpountz - 2016-06-13 09:46:59]
> ... 26 common frames omitted
> Caused by: sun.security.validator.ValidatorException: PKIX path building
> failed: sun.security.provider.certpath.SunCertPathBuilderException:
> unable to find valid certification path to requested target
> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
> ~[na:1.8.0_45]
> at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
> ~[na:1.8.0_45]
> at sun.security.validator.Validator.validate(Validator.java:260)
> ~[na:1.8.0_45]
> at 
> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> ~[na:1.8.0_45]
> at 
> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
> ~[na:1.8.0_45]
> at 
> 

Re: Editing schema and solrconfig files

[Erick Erickson]

We had the bright idea of allowing editing of the config files through
the UI... but the ability to upload arbitrary XML is a security
vulnerability, so that idea was nixed.

The solr/bin script has an upconfig and downconfig command that are (I
hope) easier to use than zkcli, I think from 5.5. In Solr 6.2 the
solr/bin script has been enhanced to allow other ZK operations. Not
quite what you were looking for, but I thought I'd mention it.

There are some ZK clients out there that'll let you edit files
directly in ZK, and I know IntelliJ also has a plugin that'll allow
you to do that from the IDE, don't know about Eclipse but I expect it
does.

I usually edit them locally and set up a shell script to push them up
as necessary...

FWIW,
Erick

On Wed, Nov 9, 2016 at 2:09 PM, John Bickerstaff
 wrote:
> I never found a way to do it through the UI... and ended up using "nano" on
> linux for simple things.
>
> For more complex stuff, I scp'd the file (or the whole conf directory) up
> to my dev box (a Mac in my case) and edited in a decent UI tool, then scp'd
> the whole thing back...  I wrote a simple bash script to automate the scp
> process on both ends once I got tired of typing it over and over...
>
> On Wed, Nov 9, 2016 at 3:05 PM, Reth RM  wrote:
>
>> What are some easiest ways to edit/modify/add conf files, such as
>> solrconfig.xml and schema.xml other than APIs end points or using zk
>> commands to re-upload modified file?
>>
>> In other words, can we edit conf files through solr admin (GUI)
>> interface(add new filed by click on button or add new request handler on
>> click?)  with feature of enabling/disabling same feature as required?
>>

Re: Rolling backups of a collection

[Mike Thomsen]

Thanks. If we write such a process, I'll see if I can get permission to
release it. It might be a moot point because I found out we're stuck on
4.10.3 for the time being. Haven't used that version in a while and forgot
it didn't even have the collection backup API.

On Wed, Nov 9, 2016 at 2:18 PM, Hrishikesh Gadre 
wrote:

> Hi Mike,
>
> I filed SOLR-9744  to
> track this work. Please comment on this jira if you have any suggestions.
>
> Thanks
> Hrishikesh
>
>
> On Wed, Nov 9, 2016 at 11:07 AM, Hrishikesh Gadre 
> wrote:
>
> > Hi Mike,
> >
> > Currently we don't have capability to take rolling backups for the Solr
> > collections. I think it should be fairly straightforward to write a
> script
> > that implements this functionality outside of Solr. If you post that
> > script, may be we can even ship it as part of Solr itself (for the
> benefit
> > of the community).
> >
> > Thanks
> > Hrishikesh
> >
> >
> >
> > On Wed, Nov 9, 2016 at 9:17 AM, Mike Thomsen 
> > wrote:
> >
> >> I read over the docs (
> >> https://cwiki.apache.org/confluence/display/solr/Making+and+
> >> Restoring+Backups)
> >> and am not quite sure what route to take. My team is looking for a way
> to
> >> backup the entire index of a SolrCloud collection with regular rotation
> >> similar to the backup option available in a single node deployment.
> >>
> >> We have plenty of space in our HDFS cluster. Resources are not an issue
> in
> >> the least to have a rolling back up of say, the last seven days. Is
> there
> >> a
> >> good way to implement this sort of rolling backup with the APIs or will
> we
> >> have to roll some of the functionality ourselves?
> >>
> >> I'm not averse to using the API to dump a copy of each shard to HDFS.
> >> Something like this:
> >>
> >> /solr/collection/replication?command=backup=shard_1_1&
> numberToKeep=7
> >>
> >> Is that a viable route to achieve this or do we need to do something
> else?
> >>
> >> Thanks,
> >>
> >> Mike
> >>
> >
> >
>

Re: Editing schema and solrconfig files

[John Bickerstaff]

I never found a way to do it through the UI... and ended up using "nano" on
linux for simple things.

For more complex stuff, I scp'd the file (or the whole conf directory) up
to my dev box (a Mac in my case) and edited in a decent UI tool, then scp'd
the whole thing back...  I wrote a simple bash script to automate the scp
process on both ends once I got tired of typing it over and over...

On Wed, Nov 9, 2016 at 3:05 PM, Reth RM  wrote:

> What are some easiest ways to edit/modify/add conf files, such as
> solrconfig.xml and schema.xml other than APIs end points or using zk
> commands to re-upload modified file?
>
> In other words, can we edit conf files through solr admin (GUI)
> interface(add new filed by click on button or add new request handler on
> click?)  with feature of enabling/disabling same feature as required?
>

Editing schema and solrconfig files

[Reth RM]

What are some easiest ways to edit/modify/add conf files, such as
solrconfig.xml and schema.xml other than APIs end points or using zk
commands to re-upload modified file?

In other words, can we edit conf files through solr admin (GUI)
interface(add new filed by click on button or add new request handler on
click?)  with feature of enabling/disabling same feature as required?

Re: OOM Error

[Susheel Kumar]

Thanks, Shawn for looking into. Your summption is right, the end of graph
is the OOM. I am trying to collect all the queries & ingestion numbers
around 9:12 but one more observation and a question from today

Observed that on 2-3 VM's out of 12, shows high usage of heap even though
heavy ingestion stopped more than an hour back while on other machines
shows normal usage.  Does that tells anything?

Snapshot 1 showing high usage of heap
===
https://www.dropbox.com/s/c1qy1s5nc9uo6cp/2016-11-09_15-55-24.png?dl=0

Snapshot  2 showing normal usage of heap
===
https://www.dropbox.com/s/9v016ilmhcahs28/2016-11-09_15-58-28.png?dl=0

The other question is we found that our ingestion batch size varies (goes
from 200 to 4000+ docs depending on  queue size). I am asking the ingestion
folks to fix the batch size but wondering does it matter in terms of load
on solr/heap usage if we submit small batches (like 500 docs max) more
frequently, than submitting bigger batches less frequently.  So far bigger
batch size has not caused any issues except these two incidents.

Thanks,
Susheel





On Wed, Nov 9, 2016 at 10:19 AM, Shawn Heisey  wrote:

> On 11/8/2016 12:49 PM, Susheel Kumar wrote:
> > Ran into OOM Error again right after two weeks. Below is the GC log
> > viewer graph. The first time we run into this was after 3 months and
> > then second time in two weeks. After first incident reduced the cache
> > size and increase heap from 8 to 10G. Interestingly query and
> > ingestion load is like normal other days and heap utilisation remains
> > stable and suddenly jumps to x2.
>
> It looks like something happened at about 9:12:30 on that graph.  Do you
> know what that was?  Starting at about that time, GC times went through
> the roof and the allocated heap began a steady rise.  At about 9:15, a
> lot of garbage was freed up and GC times dropped way down again.  At
> about 9:18, the GC once again started taking a long time, and the used
> heap was still going up steadily. At about 9:21, the full GCs started --
> the wide black bars.  I assume that the end of the graph is the OOM.
>
> > We are looking to reproduce this in test environment by producing
> > similar queries/ingestion but wondering if running into some memory
> > leak or bug like "SOLR-8922 - DocSetCollector can allocate massive
> > garbage on large indexes" which can cause this issue. Also we have
> > frequent updates and wondering if not optimizing the index can result
> > into this situation
>
> It looks more like a problem with allocated memory that's NOT garbage
> than a problem with garbage, but I can't really rule anything out, and
> even what I've said below could be wrong.
>
> Most of the allocated heap is in the old generation.  If there's a bug
> in Solr causing this problem, it would probably be a memory leak, but
> SOLR-8922 doesn't talk about a leak.  A memory leak is always possible,
> but those have been rare in Solr.  The most likely problem is that
> something changed in your indexing or query patterns which required a
> lot more memory than what happened before that point.
>
> Thanks,
> Shawn
>
>

Basic Auth for Solr Streaming Expressions

[sandeep mukherjee]

Hello everyone,
I trying to find the documentation for Basic Auth plugin for Solr Streaming 
expressions. But I'm not able to find it in the documentation anywhere. Could 
you please point me in right direction of how to enable Basic auth for Solr 
Streams?
I'm creating StreamFactory as follows: I wonder how and where can I specify 
Basic Auth username and password
@Bean
public StreamFactory streamFactory() {
SolrConfig solrConfig = ConfigManager.getNamedConfig("solr", 
SolrConfig.class);

return new StreamFactory().withDefaultZkHost(solrConfig.getConnectString())
.withFunctionName("gatherNodes", GatherNodesStream.class);
}

Re: Rolling backups of a collection

[Hrishikesh Gadre]

Hi Mike,

I filed SOLR-9744  to
track this work. Please comment on this jira if you have any suggestions.

Thanks
Hrishikesh


On Wed, Nov 9, 2016 at 11:07 AM, Hrishikesh Gadre 
wrote:

> Hi Mike,
>
> Currently we don't have capability to take rolling backups for the Solr
> collections. I think it should be fairly straightforward to write a script
> that implements this functionality outside of Solr. If you post that
> script, may be we can even ship it as part of Solr itself (for the benefit
> of the community).
>
> Thanks
> Hrishikesh
>
>
>
> On Wed, Nov 9, 2016 at 9:17 AM, Mike Thomsen 
> wrote:
>
>> I read over the docs (
>> https://cwiki.apache.org/confluence/display/solr/Making+and+
>> Restoring+Backups)
>> and am not quite sure what route to take. My team is looking for a way to
>> backup the entire index of a SolrCloud collection with regular rotation
>> similar to the backup option available in a single node deployment.
>>
>> We have plenty of space in our HDFS cluster. Resources are not an issue in
>> the least to have a rolling back up of say, the last seven days. Is there
>> a
>> good way to implement this sort of rolling backup with the APIs or will we
>> have to roll some of the functionality ourselves?
>>
>> I'm not averse to using the API to dump a copy of each shard to HDFS.
>> Something like this:
>>
>> /solr/collection/replication?command=backup=shard_1_1=7
>>
>> Is that a viable route to achieve this or do we need to do something else?
>>
>> Thanks,
>>
>> Mike
>>
>
>

Re: Rolling backups of a collection

[Hrishikesh Gadre]

Hi Mike,

Currently we don't have capability to take rolling backups for the Solr
collections. I think it should be fairly straightforward to write a script
that implements this functionality outside of Solr. If you post that
script, may be we can even ship it as part of Solr itself (for the benefit
of the community).

Thanks
Hrishikesh



On Wed, Nov 9, 2016 at 9:17 AM, Mike Thomsen  wrote:

> I read over the docs (
> https://cwiki.apache.org/confluence/display/solr/
> Making+and+Restoring+Backups)
> and am not quite sure what route to take. My team is looking for a way to
> backup the entire index of a SolrCloud collection with regular rotation
> similar to the backup option available in a single node deployment.
>
> We have plenty of space in our HDFS cluster. Resources are not an issue in
> the least to have a rolling back up of say, the last seven days. Is there a
> good way to implement this sort of rolling backup with the APIs or will we
> have to roll some of the functionality ourselves?
>
> I'm not averse to using the API to dump a copy of each shard to HDFS.
> Something like this:
>
> /solr/collection/replication?command=backup=shard_1_1=7
>
> Is that a viable route to achieve this or do we need to do something else?
>
> Thanks,
>
> Mike
>

Rolling backups of a collection

[Mike Thomsen]

I read over the docs (
https://cwiki.apache.org/confluence/display/solr/Making+and+Restoring+Backups)
and am not quite sure what route to take. My team is looking for a way to
backup the entire index of a SolrCloud collection with regular rotation
similar to the backup option available in a single node deployment.

We have plenty of space in our HDFS cluster. Resources are not an issue in
the least to have a rolling back up of say, the last seven days. Is there a
good way to implement this sort of rolling backup with the APIs or will we
have to roll some of the functionality ourselves?

I'm not averse to using the API to dump a copy of each shard to HDFS.
Something like this:

/solr/collection/replication?command=backup=shard_1_1=7

Is that a viable route to achieve this or do we need to do something else?

Thanks,

Mike

Re: 5.5.3: fieldValueCache auto-warming error

[Erick Erickson]

That is strange. If you turn off autowarming for fieldValueCache and
this goes away that would be _really_ strange indeed.

But my bet is that you _are_ doing something that uninverts the text
field (obviously inadvertently). If you restart Solr and monitor the
log until the first time you see this exception, what do the queries
show? My guess is that once you get some query in your
queryResultCache or filterCache it gets recycled and produces this on
autowarm rather than the fieldValueCache, but that's a total guess.

And my guess is correct, there's probably _is_ a JIRA here to not put
anything in caches (filter or queryResult) that's an exception..

Puzzled,
Erick

On Wed, Nov 9, 2016 at 1:33 AM, Bram Van Dam  wrote:
> Hey folks,
>
> I'm frequently getting the following error, which has me a little puzzled:
>
> Error during auto-warming of
> key:text:org.apache.solr.common.SolrException:
> java.lang.IllegalStateException: Too many values for UnInvertedField
> faceting on field text
>
> This is strange because the field in question is a full text field,
> which will never ever be used in faceting. I don't understand why Solr
> is trying to build an uninverted index for faceting on this field during
> auto-warming.
>
> Should I turn off the auto-warming of fieldValueCache? But then that
> will probably affect overall performance of other fields, which is
> something I'd like to avoid.
>
> Is there any way to exclude this field from being cached in the manner?
>
> Thanks a bunch. Stack trace and field definitions below.
>
>  - Bram
>
> Field:
>
>  positionIncrementGap="100">
>  
>   
>   
>  
> 
>
>  stored="false" required="false" multiValued="false" omitNorms="true"
> termVectors="false" termPositions="false" termOffsets="false"  />
>
> Cache definition:
>
>  autowarmCount="1024" showItems="32" />
>
> Full stack trace:
>
> 2016-11-08
> 03:35:57.196/UTC|ERROR|searcherExecutor-27-thread-1-processing-x:myIndexx:myIndex|org.apache.solr.search.FastLRUCache|Error
> during auto-warming of key:text:org.apache.solr.common.SolrException:
> java.lang.IllegalStateException: Too many values for UnInvertedField
> faceting on field text
> at
> org.apache.solr.search.facet.UnInvertedField.(UnInvertedField.java:194)
> at
> org.apache.solr.search.facet.UnInvertedField.getUnInvertedField(UnInvertedField.java:595)
> at
> org.apache.solr.search.SolrIndexSearcher$1.regenerateItem(SolrIndexSearcher.java:523)
> at org.apache.solr.search.FastLRUCache.warm(FastLRUCache.java:163)
> at
> org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:2320)
> at org.apache.solr.core.SolrCore$4.call(SolrCore.java:1851)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: Too many values for
> UnInvertedField faceting on field text
> at 
> org.apache.lucene.uninverting.DocTermOrds.uninvert(DocTermOrds.java:489)
> at
> org.apache.solr.search.facet.UnInvertedField.(UnInvertedField.java:192)
> ... 10 more
>
>

Re: error while working solr 6.2 cdcr

[Erick Erickson]

Why do you have 6 zookeepers listed in the zkHost parameter? My guess
is that you've listed the ZK addresses for both the source and target
clusters. This should only be the ZK nodes for the target cluster and
only present on the source.

The source already knows its own ZK addresses, it needs to know about
the target cluster in order to know what nodes it should send updates
to.

Best,
Erick

On Wed, Nov 9, 2016 at 4:44 AM, Neeraj Bhatt  wrote:
> Hello All
>
> I have a source cluster (3 nodes, 3 shards and 3 replica) and same for
> target cluster. On posting of documents to source I am getting below error
> in target solr log files when i turned log to debug level
>
>
>
>
> *6538 ERROR (qtp110456297-20) [c:multi_dc_poc s:shard1 r:core_node3
> x:multi_dc_poc_shard1_replica2] o.a.s.h.RequestHandlerBase
> org.apache.solr.common.SolrException: Action LASTPROCESSEDVERSION sent to
> non-leader replicaat
> org.apache.solr.handler.CdcrRequestHandler.handleLastProcessedVersionAction(CdcrRequestHandler.java:498)*
>
> *should I reconfigure my replicas to one in target ?*
>
> solr config of source and target is same as given in wiki
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62687462
>
> *source *
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> * class="solr.CdcrRequestHandler">  name="replica">  name="zkHost">192.169.35.53:2181
> ,192.169.35.249:2181
> ,192.169.35.251:2181
> 
>   name="source">multi_dc_poc  name="target">multi_dc_poc name="replicator"> name="threadPoolSize">9 name="schedule">1000 name="batchSize">128 name="updateLogSynchronizer"> name="schedule">1000  *
>
>
>
>
>
>
>
> *source and target update log   class="solr.CdcrUpdateLog">   name="dir">${solr.ulog.dir:}   name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}
> *
>
>
> *target solr config*
>
> 
>   
> disabled
>   
> 
>
> 
>   
> cdcr-processor-chain
>   
> 
>
> 
>   
>   
> 
>
> *target status*
>
> 0 name="QTime">0 name="process">stopped name="buffer">disabled
>
> *source status*
>
> 0 name="QTime">0 name="process">started name="buffer">enabled

Re: OOM Error

[Shawn Heisey]

On 11/8/2016 12:49 PM, Susheel Kumar wrote:
> Ran into OOM Error again right after two weeks. Below is the GC log
> viewer graph. The first time we run into this was after 3 months and
> then second time in two weeks. After first incident reduced the cache
> size and increase heap from 8 to 10G. Interestingly query and
> ingestion load is like normal other days and heap utilisation remains
> stable and suddenly jumps to x2. 

It looks like something happened at about 9:12:30 on that graph.  Do you
know what that was?  Starting at about that time, GC times went through
the roof and the allocated heap began a steady rise.  At about 9:15, a
lot of garbage was freed up and GC times dropped way down again.  At
about 9:18, the GC once again started taking a long time, and the used
heap was still going up steadily. At about 9:21, the full GCs started --
the wide black bars.  I assume that the end of the graph is the OOM.

> We are looking to reproduce this in test environment by producing
> similar queries/ingestion but wondering if running into some memory
> leak or bug like "SOLR-8922 - DocSetCollector can allocate massive
> garbage on large indexes" which can cause this issue. Also we have
> frequent updates and wondering if not optimizing the index can result
> into this situation

It looks more like a problem with allocated memory that's NOT garbage
than a problem with garbage, but I can't really rule anything out, and
even what I've said below could be wrong.

Most of the allocated heap is in the old generation.  If there's a bug
in Solr causing this problem, it would probably be a memory leak, but
SOLR-8922 doesn't talk about a leak.  A memory leak is always possible,
but those have been rare in Solr.  The most likely problem is that
something changed in your indexing or query patterns which required a
lot more memory than what happened before that point.

Thanks,
Shawn

Re: SolrJ optimize method -- not returning immediately when the "wait" options are false

[Shawn Heisey]

On 11/8/2016 4:27 PM, Yonik Seeley wrote:
> So no, you should not expect to see an immediate return with
> waitSearcher=false since it only represents the
> open-and-register-searcher part.

That's too bad.  I already have the SolrJ call happening in a separate
thread, so it doesn't really affect my *current* program much.

For a future version of the program, I have a question:  If I have a
SolrJ optimize running in a background thread, can I call close() on
SolrClient and HttpClient objects (and remove all references to them)
while that's happening and have all the objects properly turn into
garbage, possibly at a later time when the optimize finishes?  The 40+
seconds for the optimizes I have mentioned is the hot shard.  I've seen
optimizes on a cold shard take well over an hour.

Basically, I want to be able to have my program dismantle all the
objects (threads, SolrClient, HttpClient, JDBC, etc) related to building
an individual index, reload its configuration, and set everything back
up again, without stopping the program.  Currently I do that by
restarting the program entirely, but I want something much more dynamic,
with a web interface to control it.  I'd like it to be able to do this
even when a LONG background optimize is happening.

I do have a possible alternate plan:  Use a client with a very short
socket timeout for optimizes and ignore the exception.  I'm worried that
I won't be able to tell the difference between a problem with the
optimize call and the socket timeout being reached.

Thanks,
Shawn

error while working solr 6.2 cdcr

[Neeraj Bhatt]

Hello All

I have a source cluster (3 nodes, 3 shards and 3 replica) and same for
target cluster. On posting of documents to source I am getting below error
in target solr log files when i turned log to debug level




*6538 ERROR (qtp110456297-20) [c:multi_dc_poc s:shard1 r:core_node3
x:multi_dc_poc_shard1_replica2] o.a.s.h.RequestHandlerBase
org.apache.solr.common.SolrException: Action LASTPROCESSEDVERSION sent to
non-leader replicaat
org.apache.solr.handler.CdcrRequestHandler.handleLastProcessedVersionAction(CdcrRequestHandler.java:498)*

*should I reconfigure my replicas to one in target ?*

solr config of source and target is same as given in wiki
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62687462

*source *




















*  192.169.35.53:2181
,192.169.35.249:2181
,192.169.35.251:2181

 multi_dc_poc multi_dc_poc910001281000  *







*source and target update log${solr.ulog.dir:}  ${solr.ulog.numVersionBuckets:65536}
*


*target solr config*


  
disabled
  



  
cdcr-processor-chain
  



  
  


*target status*

00stoppeddisabled

*source status*

00startedenabled

Re: CDCR (Solr6.x) does not start

[neerajbhatt]

Hi Uwe

I am facing the same error as you , I am getting error 6538 ERROR
(qtp110456297-20) [c:multi_dc_poc s:shard1 r:core_node3
x:multi_dc_poc_shard1_replica2] o.a.s.h.RequestHandlerBase
org.apache.solr.common.SolrException: Action LASTPROCESSEDVERSION sent to
non-leader replica
at
org.apache.solr.handler.CdcrRequestHandler.handleLastProcessedVersionAction(CdcrRequestHandler.java:498)

- I have 3 shards and 3 replicas in 3 solr machines

Can you please help

Thanks
Neeraj



--
View this message in context: 
http://lucene.472066.n3.nabble.com/CDCR-Solr6-x-does-not-start-tp4284742p4305205.html
Sent from the Solr - User mailing list archive at Nabble.com.

5.5.3: fieldValueCache auto-warming error

[Bram Van Dam]

Hey folks,

I'm frequently getting the following error, which has me a little puzzled:

Error during auto-warming of
key:text:org.apache.solr.common.SolrException:
java.lang.IllegalStateException: Too many values for UnInvertedField
faceting on field text

This is strange because the field in question is a full text field,
which will never ever be used in faceting. I don't understand why Solr
is trying to build an uninverted index for faceting on this field during
auto-warming.

Should I turn off the auto-warming of fieldValueCache? But then that
will probably affect overall performance of other fields, which is
something I'd like to avoid.

Is there any way to exclude this field from being cached in the manner?

Thanks a bunch. Stack trace and field definitions below.

 - Bram

Field:


 
  
  
 




Cache definition:



Full stack trace:

2016-11-08
03:35:57.196/UTC|ERROR|searcherExecutor-27-thread-1-processing-x:myIndexx:myIndex|org.apache.solr.search.FastLRUCache|Error
during auto-warming of key:text:org.apache.solr.common.SolrException:
java.lang.IllegalStateException: Too many values for UnInvertedField
faceting on field text
at
org.apache.solr.search.facet.UnInvertedField.(UnInvertedField.java:194)
at
org.apache.solr.search.facet.UnInvertedField.getUnInvertedField(UnInvertedField.java:595)
at
org.apache.solr.search.SolrIndexSearcher$1.regenerateItem(SolrIndexSearcher.java:523)
at org.apache.solr.search.FastLRUCache.warm(FastLRUCache.java:163)
at
org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:2320)
at org.apache.solr.core.SolrCore$4.call(SolrCore.java:1851)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Too many values for
UnInvertedField faceting on field text
at 
org.apache.lucene.uninverting.DocTermOrds.uninvert(DocTermOrds.java:489)
at
org.apache.solr.search.facet.UnInvertedField.(UnInvertedField.java:192)
... 10 more