Re: Error upgrading from 6.0 to 6.1

[Erick Erickson]

That's usually an indication that your classpath has old and new jars
in it. When you start Solr,
the directories from which all the jars are listed in the log file. My
guess is that if you examine them
you'll see jar files loaded from both 6.0 and some from 6.1 and you
need to figure out how
that happened and undo it

Best,
Erick

On Mon, Aug 22, 2016 at 8:31 PM, Stephen Lewis  wrote:
> In particular, I see this line. Was there perhaps a deprecation of a method
> or something changed about cluster properties?
>
> 
> 
> Error 500
> {msg=org.apache.solr.common.cloud.ZkStateReader.getClusterProps()Ljava/util/Map;,trace=java.lang.NoSuchMethodError:
> org.apache.solr.common.cloud.ZkStateReader.getClusterProps()Ljava/util/Map;
>
> On Mon, Aug 22, 2016 at 8:18 PM, Stephen Lewis 
> wrote:
>
>> Oops, apologies for my confusing grammar and for missing the attachment.
>> The intro sentence should have read "I have a question about upgrading a
>> solr cloud cluster in place." I've actually attached the log below this
>> time.
>>
>> Thanks again,
>> Stephen
>>
>> On Mon, Aug 22, 2016 at 7:41 PM, Stephen Lewis 
>> wrote:
>>
>>> Hello,
>>>
>>> I have a question about updating a solr cloud cluster servers in place. I
>>> have a scripted method for updating a solr cloud in place, which works
>>> consistently to up/down grade between 6.0.0 and 6.0.1 (in our test
>>> environment), but hits an error consistently when going from either to solr
>>> 6.1.0. Each server is hosting a single solr node, and each shard has a
>>> replication factor of 3.
>>>
>>> The way the script works is as follows. For each instance:
>>>
>>> 1. Pull the instance from serving requests and drain.
>>> 2. Delete the replica from the collection (but leave the index and data)
>>> 3. If that node was the leader, force a leader election (solr is not
>>> accepting writes at this time, so this is safe)
>>> 4. Run a bootstrapping script on the remote machine (which installs a
>>> particular solr version, and is otherwise idempotent)
>>> 5. Once the instance is updated and solr is confirmed, add the node as a
>>> replica where it used to be
>>> 6. Wait for recovery
>>> 7. Reserve requests from this node.
>>>
>>> As mentioned, this hasn't shown any problem switching between versions
>>> 6.0.0 and 6.0.1, but when I try to use this to upgrade to solr 6.1.0, the
>>> "ADDREPLICA" command fails as follows:
>>>
>>> "status":500,"QTime":65},"failure":{"172.18.6.68:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
>>>  from server at http://172.18.6.68:8983/solr: Expected mime type 
>>> application/octet-stream but got text/html
>>>
>>> I've included the full log below as an attachment. The exact request
>>> being served is the following:
>>>
>>> http://52.91.138.30:8983/solr/admin/collections?wt=json&acti
>>> on=ADDREPLICA&collection=panopto&shard=shard2&node=172.18.6.68:8983_solr
>>>
>>> I didn't see any special actions which needed to be taken when upgrading
>>> to 6.1.0. Is there perhaps something wrong in my upgrade methodology or
>>> anything else you're aware of which may be related?
>>>
>>> Thanks for your help!
>>> Stephen
>>>
>>> --
>>> www.stephen-lewis.net
>>>
>>
>>
>>
>> --
>> www.stephen-lewis.net
>>
>
>
>
> --
> www.stephen-lewis.net

Re: Error upgrading from 6.0 to 6.1

[Stephen Lewis]

In particular, I see this line. Was there perhaps a deprecation of a method
or something changed about cluster properties?



Error 500
{msg=org.apache.solr.common.cloud.ZkStateReader.getClusterProps()Ljava/util/Map;,trace=java.lang.NoSuchMethodError:
org.apache.solr.common.cloud.ZkStateReader.getClusterProps()Ljava/util/Map;

On Mon, Aug 22, 2016 at 8:18 PM, Stephen Lewis 
wrote:

> Oops, apologies for my confusing grammar and for missing the attachment.
> The intro sentence should have read "I have a question about upgrading a
> solr cloud cluster in place." I've actually attached the log below this
> time.
>
> Thanks again,
> Stephen
>
> On Mon, Aug 22, 2016 at 7:41 PM, Stephen Lewis 
> wrote:
>
>> Hello,
>>
>> I have a question about updating a solr cloud cluster servers in place. I
>> have a scripted method for updating a solr cloud in place, which works
>> consistently to up/down grade between 6.0.0 and 6.0.1 (in our test
>> environment), but hits an error consistently when going from either to solr
>> 6.1.0. Each server is hosting a single solr node, and each shard has a
>> replication factor of 3.
>>
>> ​The way the script works is as follows. For each instance:
>>
>> 1. Pull the instance from serving requests and drain.
>> 2. Delete the replica from the collection (but leave the index and data)
>> 3. If that node was the leader, force a leader election (solr is not
>> accepting writes at this time, so this is safe)
>> 4. Run a bootstrapping script on the remote machine (which installs a
>> particular solr version, and is otherwise idempotent)
>> 5. Once the instance is updated and solr is confirmed, add the node as a
>> replica where it used to be
>> 6. Wait for recovery
>> 7. Reserve requests from this node.
>>
>> ​As mentioned, this hasn't shown any problem switching between versions
>> 6.0.0 and 6.0.1, but when I try to use this to upgrade to solr 6.1.0, the
>> "ADDREPLICA" command fails as follows:
>>
>> "status":500,"QTime":65},"failure":{"172.18.6.68:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
>>  from server at http://172.18.6.68:8983/solr: Expected mime type 
>> application/octet-stream but got text/html
>>
>> I've included​ the full log below as an attachment. The exact request
>> being served is the following:
>>
>> http://52.91.138.30:8983/solr/admin/collections?wt=json&acti
>> on=ADDREPLICA&collection=panopto&shard=shard2&node=172.18.6.68:8983_solr
>>
>> I didn't see any special actions which needed to be taken when upgrading
>> to 6.1.0. Is there perhaps something wrong in my upgrade methodology or
>> anything else you're aware of which may be related?
>>
>> Thanks for your help!
>> Stephen
>>
>> --
>> www.stephen-lewis.net
>>
>
>
>
> --
> www.stephen-lewis.net
>



-- 
www.stephen-lewis.net

Re: Error upgrading from 6.0 to 6.1

[Stephen Lewis]

Oops, apologies for my confusing grammar and for missing the attachment.
The intro sentence should have read "I have a question about upgrading a
solr cloud cluster in place." I've actually attached the log below this
time.

Thanks again,
Stephen

On Mon, Aug 22, 2016 at 7:41 PM, Stephen Lewis 
wrote:

> Hello,
>
> I have a question about updating a solr cloud cluster servers in place. I
> have a scripted method for updating a solr cloud in place, which works
> consistently to up/down grade between 6.0.0 and 6.0.1 (in our test
> environment), but hits an error consistently when going from either to solr
> 6.1.0. Each server is hosting a single solr node, and each shard has a
> replication factor of 3.
>
> ​The way the script works is as follows. For each instance:
>
> 1. Pull the instance from serving requests and drain.
> 2. Delete the replica from the collection (but leave the index and data)
> 3. If that node was the leader, force a leader election (solr is not
> accepting writes at this time, so this is safe)
> 4. Run a bootstrapping script on the remote machine (which installs a
> particular solr version, and is otherwise idempotent)
> 5. Once the instance is updated and solr is confirmed, add the node as a
> replica where it used to be
> 6. Wait for recovery
> 7. Reserve requests from this node.
>
> ​As mentioned, this hasn't shown any problem switching between versions
> 6.0.0 and 6.0.1, but when I try to use this to upgrade to solr 6.1.0, the
> "ADDREPLICA" command fails as follows:
>
> "status":500,"QTime":65},"failure":{"172.18.6.68:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
>  from server at http://172.18.6.68:8983/solr: Expected mime type 
> application/octet-stream but got text/html
>
> I've included​ the full log below as an attachment. The exact request
> being served is the following:
>
> http://52.91.138.30:8983/solr/admin/collections?wt=json&acti
> on=ADDREPLICA&collection=panopto&shard=shard2&node=172.18.6.68:8983_solr
>
> I didn't see any special actions which needed to be taken when upgrading
> to 6.1.0. Is there perhaps something wrong in my upgrade methodology or
> anything else you're aware of which may be related?
>
> Thanks for your help!
> Stephen
>
> --
> www.stephen-lewis.net
>



-- 
www.stephen-lewis.net

Error upgrading from 6.0 to 6.1

[Stephen Lewis]

Hello,

I have a question about updating a solr cloud cluster servers in place. I
have a scripted method for updating a solr cloud in place, which works
consistently to up/down grade between 6.0.0 and 6.0.1 (in our test
environment), but hits an error consistently when going from either to solr
6.1.0. Each server is hosting a single solr node, and each shard has a
replication factor of 3.

​The way the script works is as follows. For each instance:

1. Pull the instance from serving requests and drain.
2. Delete the replica from the collection (but leave the index and data)
3. If that node was the leader, force a leader election (solr is not
accepting writes at this time, so this is safe)
4. Run a bootstrapping script on the remote machine (which installs a
particular solr version, and is otherwise idempotent)
5. Once the instance is updated and solr is confirmed, add the node as a
replica where it used to be
6. Wait for recovery
7. Reserve requests from this node.

​As mentioned, this hasn't shown any problem switching between versions
6.0.0 and 6.0.1, but when I try to use this to upgrade to solr 6.1.0, the
"ADDREPLICA" command fails as follows:

"status":500,"QTime":65},"failure":{"172.18.6.68:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
from server at http://172.18.6.68:8983/solr: Expected mime type
application/octet-stream but got text/html

I've included​ the full log below as an attachment. The exact request being
served is the following:

http://52.91.138.30:8983/solr/admin/collections?wt=json&action=ADDREPLICA&collection=panopto&shard=shard2&node=172.18.6.68:8983_solr

I didn't see any special actions which needed to be taken when upgrading to
6.1.0. Is there perhaps something wrong in my upgrade methodology or
anything else you're aware of which may be related?

Thanks for your help!
Stephen

-- 
www.stephen-lewis.net

Re: help with DIH transformer to add a suffix to column names

[Wendy]

Hi Emlr,

I use the example of "A General TrimTransformer" in the following link:

https://wiki.apache.org/solr/DIHCustomTransformer#transformer

But instead of trim the field value, I wanted to change the table column
name to columnName_stem.
So I can use *_stem to copy all fields.  

Here is my code, but just not working. I don't what is the problem with the
code? Any ideas? Thanks!

public class RowTransformer extends Transformer  {
public Map transformRow(Map row, Context
context) {
List> fields = ((Context)
context).getAllEntityFields();
 
System.out.println("fields = " + fields.size());

for (Map field : fields) {
String columnName = field.get(DataImporter.COLUMN);
System.out.println("columnName = "+ columnName);
// Get this field's value from the current row
Object value = row.get(columnName);
if (value != null && !value.toString().trim().equals("")) {
   row.put(columnName + "_stem", value.toString().trim());
   System.out.println("positive columnName = "+ columnName);
   System.out.println("positive columnValue = "+
value.toString());
}
}
return row;
}
}



--
View this message in context: 
http://lucene.472066.n3.nabble.com/help-with-DIH-transformer-to-add-a-suffix-to-column-names-tp4292448p4292796.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: help with DIH transformer to add a suffix to column names

[Wendy]

Hi Alex,Thank you for your response. I include the details below. Hope it
makes more clear. I wanted to append _stem to each column name, so I can
specify fields dynamically with *_stem w/o specifying each field
names.Thanks!I use solr 6.1.0  to index data from mysql database.I wrote a
java class of FieldTransformer as following. But it didn't work. It got null
when retrieving column name. How to fix the java code?
Thanks!---package
my.solr.transformer;import java.util.ArrayList; public class
FieldTransformer {  public Object transformRow(Map row) 
{   
int s = row.size(); System.out.println("row size = 
" + s);  
for (Entry entry : row.entrySet()) {String
columnName = entry.getKey();row.put(columnName + "_stem",
entry.getValue());  System.out.println("column name = " +
columnName);System.out.println("value = " + entry.getKey());
   
row.remove(columnName); }   System.out.println("after 
processed row
size = " + row.size()); return row; }   }---Other solr
config files--1. part of db-data-config.xml 
 
 
.  2. part of managed-schema file .. ..3.
part of solrconfig.xml file   db-data-config.xml  true  
 
explicit  edismax   pdb_id^20.0
classification_stem^10.0*_stem^0.3
rest_fields_stem ^0.3   7   
1000text  



--
View this message in context: 
http://lucene.472066.n3.nabble.com/help-with-DIH-transformer-to-add-a-suffix-to-column-names-tp4292448p4292627.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr - search score and tf-idf vector from individual fields

[Alessandro Benedetti]

Hi govind,
let's analyse your request step by step :

On Tue, Aug 16, 2016 at 7:54 AM, govind nitk  wrote:

> Hi Developers,
>
>
> down votefavorite
>  vector-from-individual-fields#>
>
> This is a fundamental question which I was unable to get from the solr help
> and other related Stackoverflow queries.
>
> I have few hundred thousand documents which have 12 fields in them (to be
> indexed). All of these fields have text in them (each field can have
> varying length text in them - may be from 10 to 5000 characters). For e.g ,
> lets say these fields are named A, B . L (12 in all)
>
> Now, when I search for documents, my query comes from 3 fields. X1 , X2 and
> X3. Now X1 (conceptually) closely matches with fields C, D , and E. X2
> (conceptually) closely matches with fields F, G and J. And X3 is basically
> the same field as A. But X1 and X2 should be searched for, all over the
> fields (including A). Just filtering against their conceptually matching
> fields will not do.
>

This logic needs to be defined :
1) in your search API, rewriting the query OR
2) with a specific query parser, that will take different parameters and
rewrite the query properly

Out of the box solr doesn't allow you to map one field to another/ more
than one .

>
> So when designing the schema, my only criterion is the ranking and the
> search. I also want (can I ? ) get scores of my query against individual
> fields. Something like this
>
> Query : X1 , Score against C , E and over all score (for all returned
> documents)
>
> Query : X2 , Score against M , N , O and over all score (for all returned
> documents)
>
> Query : X1 + X2 , Score against C , E, M, N and O, and over all score (for
> all returned documents)
>

Given a document and a query, the score in Solr is only one.
Anyway if you activate debug you can see the different components of the
score.
This can be potentially the closest thing to what you need .

>
> The reason I want those individual scores is I want to further use those
> scores for ML algorithms to further reshuffle/fit the rankings against a
> training set.
>

This is interesting, can you give more details ?
Maybe the query re-rank component and Learning To Rank could be interesting
to you.

Cheers

>
> I also want want the tf-idf vector components of X1 and X2 against C, E and
> M,N,O respectively.
>
> Can anyone please let me know if this is possible ?
>



-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Re: commit it taking 1300 ms

[Alessandro Benedetti]

I agree with the suggestions so far.
The cache auto-warming doesn't seem the problem as the index is not massive
and the auto-warm is for only 10 docs.
Are you using any warming query for the new searcher ?

Are you using soft or hard commit ?
This can make the difference ( soft are much cheaper, not free but cheaper)
.
You said :
" Actually earlier it was taking less but suddenly it has increased "

What happened ?
Anyway, there are a lot of questions to answer before we can help you...

Cheers

On Fri, Aug 12, 2016 at 4:58 AM, Esther-Melaine Quansah <
esther.quan...@lucidworks.com> wrote:

> Midas,
>
> I’d like further clarification as well. Are you sending commits along with
> each document that you’re POSTing to Solr? If so, you’re essentially either
> opening a new searcher or flushing to disk with each POST which could
> explain latency between each request.
>
> Thanks,
>
> Esther
> > On Aug 11, 2016, at 12:19 PM, Erick Erickson 
> wrote:
> >
> > bq:  we post json documents through the curl it takes the time (same
> time i
> > would like to say that we are not hard committing ). that curl takes time
> > i.e. 1.3 sec.
> >
> > OK, I'm really confused. _what_ is taking 1.3 seconds? When you said
> > commit, I was thinking of Solr's commit operation, which is totally
> distinct
> > from just adding a doc to the index. But I read the above statement
> > as you're saying it takes 1.3 seconds just to send a doc to Solr.
> >
> > Let's see the exact curl command you're using please?
> >
> > Best,
> > Erick
> >
> >
> > On Thu, Aug 11, 2016 at 5:32 AM, Emir Arnautovic
> >  wrote:
> >> Hi Midas,
> >>
> >> 1. How many indexing threads?
> >> 2. Do you batch documents and what is your batch size?
> >> 3. How frequently do you commit?
> >>
> >> I would recommend:
> >> 1. Move commits to Solr (set auto soft commit to max allowed time)
> >> 2. Use batches (bulks)
> >> 3. tune bulk size and number of threads to achieve max performance.
> >>
> >> Thanks,
> >> Emir
> >>
> >>
> >>
> >> On 11.08.2016 08:21, Midas A wrote:
> >>>
> >>> Emir,
> >>>
> >>> other queries:
> >>>
> >>> a) Solr cloud : NO
> >>> b)  >>> size="5000" initialSize="5000" autowarmCount="10"/>
> >>> c)   >>> size="1000" initialSize="1000" autowarmCount="10"/>
> >>> d)  >>> size="1000" initialSize="1000" autowarmCount="10"/>
> >>> e) we are using multi threaded system.
> >>>
> >>> On Thu, Aug 11, 2016 at 11:48 AM, Midas A 
> wrote:
> >>>
>  Emir,
> 
>  we post json documents through the curl it takes the time (same time i
>  would like to say that we are not hard committing ). that curl takes
> time
>  i.e. 1.3 sec.
> 
>  On Wed, Aug 10, 2016 at 2:29 PM, Emir Arnautovic <
>  emir.arnauto...@sematext.com> wrote:
> 
> > Hi Midas,
> >
> > According to your autocommit configuration and your worry about
> commit
> > time I assume that you are doing explicit commits from client code
> and
> > that
> > 1.3s is client observed commit time. If that is the case, than it
> might
> > be
> > opening searcher that is taking time.
> >
> > How do you index data - single threaded or multithreaded? How
> frequently
> > do you commit from client? Can you let Solr do soft commits instead
> of
> > explicitly committing? Do you have warmup queries? Is this SolrCloud?
> > What
> > is number of servers (what spec), shards, docs?
> >
> > In any case monitoring can give you more info about server/Solr
> behavior
> > and help you diagnose issues more easily/precisely. One such
> monitoring
> > tool is our SPM .
> >
> > Regards,
> > Emir
> >
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> > On 10.08.2016 05:20, Midas A wrote:
> >
> >> Thanks for replying
> >>
> >> index size:9GB
> >> 2000 docs/sec.
> >>
> >> Actually earlier it was taking less but suddenly it has increased .
> >>
> >> Currently we do not have any monitoring  tool.
> >>
> >> On Tue, Aug 9, 2016 at 7:00 PM, Emir Arnautovic <
> >> emir.arnauto...@sematext.com> wrote:
> >>
> >> Hi Midas,
> >>>
> >>> Can you give us more details on your index: size, number of new
> docs
> >>> between commits. Why do you think 1.3s for commit is to much and
> why
> >>> do
> >>> you
> >>> need it to take less? Did you do any system/Solr monitoring?
> >>>
> >>> Emir
> >>>
> >>>
> >>> On 09.08.2016 14:10, Midas A wrote:
> >>>
> >>> please reply it is urgent.
> 
>  On Tue, Aug 9, 2016 at 11:17 AM, Midas A 
>  wrote:
> 
>  Hi ,
> 
> > commit is taking more than 1300 ms . what should i check on
> server.
> >
> > below is my configuration .
> >
> >  ${solr.autoCommit.maxTime:15000}
> <
> 

Re: SolrCloud: Collection creation failed due to timeout

[Chris Rogers]

Thanks Shawn, that was indeed the issue. Upgraded and everything is working as 
expected.

Much appreciated.
Chris

On 22/08/2016, 14:50, "Shawn Heisey"  wrote:

On 8/22/2016 7:27 AM, Chris Rogers wrote:
> Solr version: 6.1.0 Zookeeper: 3.3.6 

This is probably the root of the issue.

Solr includes Zookeeper 3.4.6, for the client.  The server you're
running is significantly older than the client version.  The WARN
messages I quoted previously indicates that the server dropped a packet
type (number 14) that it didn't understand, which seems to indicate that
the client code in Solr is trying to use functionality that it knows
about, but is not provided by the 3.3.6 server version.

You should upgrade your zookeeper server, probably to the latest
version, which is 3.4.8.  Don't use 3.4.7 if you manage to find it -- it
has an extremely serious bug, so it was removed from the mirror system.

Thanks,
Shawn

Re: SolrCloud: Collection creation failed due to timeout

[Shawn Heisey]

On 8/22/2016 7:27 AM, Chris Rogers wrote:
> Solr version: 6.1.0 Zookeeper: 3.3.6 

This is probably the root of the issue.

Solr includes Zookeeper 3.4.6, for the client.  The server you're
running is significantly older than the client version.  The WARN
messages I quoted previously indicates that the server dropped a packet
type (number 14) that it didn't understand, which seems to indicate that
the client code in Solr is trying to use functionality that it knows
about, but is not provided by the 3.3.6 server version.

You should upgrade your zookeeper server, probably to the latest
version, which is 3.4.8.  Don't use 3.4.7 if you manage to find it -- it
has an extremely serious bug, so it was removed from the mirror system.

Thanks,
Shawn

Re: SolrCloud: Collection creation failed due to timeout

[Chris Rogers]

Hi Shawn,

Thanks for replying.

Solr version: 6.1.0
Zookeeper: 3.3.6

Solr log errors below are from around the 21:33:39 timestamp:


2016-08-21 21:33:37.135 WARN  (main-SendThread(172.28.128.3:2181)) [   ] 
o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server 
172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and 
attempting reconnect
java.io.IOException: Xid out of order. Got Xid 1438 with err 0 expected Xid 
1437 for a packet with details: clientPath:null serverPath:null finished:false 
header:: 1437,14  replyHeader:: 0,0,-4  request:: 
org.apache.zookeeper.MultiTransactionRecord@95ad06de response:: 
org.apache.zookeeper.MultiResponse@0
at 
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
at 
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-21 21:33:37.144 ERROR 
(recoveryExecutor-3-thread-1-processing-n:10.0.2.15:8983_solr 
x:mycollection_shard1_replica1 s:shard1 c:mycollection r:core_node1) 
[c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1] 
o.a.s.c.RecoveryStrategy Error while trying to recover. 
core=mycollection_shard1_replica1:org.apache.solr.common.SolrException: No 
registered leader was found after waiting for 4000ms , collection: mycollection 
slice: shard1
at 
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:718)
at 
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:704)
at 
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:305)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

2016-08-21 21:33:37.144 INFO  
(recoveryExecutor-3-thread-1-processing-n:10.0.2.15:8983_solr 
x:mycollection_shard1_replica1 s:shard1 c:mycollection r:core_node1) 
[c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1] 
o.a.s.c.RecoveryStrategy Replay not started, or was not successful... still 
buffering updates.
2016-08-21 21:33:37.144 INFO  
(recoveryExecutor-3-thread-1-processing-n:10.0.2.15:8983_solr 
x:mycollection_shard1_replica1 s:shard1 c:mycollection r:core_node1) 
[c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1] 
o.a.s.c.RecoveryStrategy RecoveryStrategy has been closed
2016-08-21 21:33:37.149 INFO  
(recoveryExecutor-3-thread-1-processing-n:10.0.2.15:8983_solr 
x:mycollection_shard1_replica1 s:shard1 c:mycollection r:core_node1) 
[c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1] 
o.a.s.c.RecoveryStrategy Finished recovery process, successful=[false]
2016-08-21 21:33:37.237 INFO  
(zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [   ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@7d41da47 
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent 
state:Disconnected type:None path:null path:null type:None
2016-08-21 21:33:37.237 INFO  
(zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [   ] 
o.a.s.c.c.ConnectionManager zkClient has disconnected
2016-08-21 21:33:39.062 WARN  (main-SendThread(172.28.128.3:2181)) [   ] 
o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2016-08-21 21:33:39.063 INFO  
(zkCallback-4-thread-171-processing-n:10.0.2.15:8983_solr) [   ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@7d41da47 
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
2016-08-21 21:33:39.071 WARN  (main-SendThread(172.28.128.3:2181)) [   ] 
o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server 
172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and 
attempting reconnect
java.io.IOException: Xid out of order. Got Xid 1441 with err 0 expected Xid 
1440 for a packet with details: clientPath:null serverPath:null finished:false 
header:: 1440,14  replyHeader:: 0,0,-4  request:: 
org.apache.zookeeper.MultiTransactionRecord@95ad06de response:: 
org.apache.zookeeper.MultiResponse@0
at 
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
at 
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at 
org.apache.zookeeper.ClientCn

Re: Document removed from index after Delta import

[Or Gerson]

i would have understood if the index stayed the same, but the document is
removed entirely.



On Mon, Aug 22, 2016 at 5:48 AM, Erick Erickson 
wrote:

> You can't rely on any searches with that autocommit configuration.
>
> openSearcher is set to false. Therefore you will not see any changes to
> your index as a result of an expiring autoCommit interval.
>
> I'm not sure whether DIH issues its own commit when done, but your
> tests so far aren't particularly dependable. I'd try again after issuing
> a manual hard commit or a soft commit.
>
> You can do this in a URL by
> /collection/update?commit=true
>
> Here's the whole run-down on the various commit options.
> https://lucidworks.com/blog/2013/08/23/understanding-
> transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> Best,
> Erick
>
> On Sun, Aug 21, 2016 at 7:43 AM, Or Gerson  wrote:
> > Hello,
> >
> > I have Solr version 4.3.0.
> >
> > I have encountered a problem where document is not returning from queries
> > after delta import although the delta import does not report that a
> > document has been deleted.
> >
> > i have a document that is composed of several fields , the delta import
> > looks for a field
> > called "update_date" and checks whether the date is after the last time
> > delta import was ran.
> >
> > so something like
> >
> > select d.ID as UNIQUE_ID
> >from document d
> >where d.UPDATE_DATE >= '${dataimporter.last_index_time}' and
> > d.OWNER_ID IS NOT NULL and d.DELETE_DATE IS NULL
> >
> >
> > deletePKQuery looks like
> >
> > select d.ID as UNIQUE_ID from document d
> >   where d.DELETE_DATE >= '${dataimporter.last_index_time}'
> >
> > doing full import will successfully fetch the documents
> >
> > then changing the update_date on a document will remove it from the index
> > (but this is not reported in the log)
> >
> > "07 Aug 2016 18:29:24,278 [Thread-53] INFO  DocBuilder - Completed
> > DeletedRowKey for Entity: permission_set rows obtained : 0"
> >
> > 07 Aug 2016 18:29:24,438 [Thread-53] INFO  MetadataOnImportEndEventListen
> er
> > - metadata import process end: {deletedDocCount=0, docCount=1,
> > queryCount=6, rowCount=6, skipDocCount=0}
> >
> >
> > merge policy is :
> >
> > 
> >   10
> >   10
> > 
> >
> >
> > 
> > 
> >  1
> > 
> >  1
> >
> >
> > only hard commit is configured, soft commit is commented out:
> >
> > 
> >   1000
> >   3
> >   false
> > 
> >
> >
> > will greatly appreciate your help
> >
> > Thanks,
> > Or Gerson
>

Re: SolrCloud: Collection creation failed due to timeout

[Shawn Heisey]

On 8/22/2016 6:20 AM, Chris Rogers wrote:
> It’s then that I start seeing lots of errors in the Solr logs, and lots of 
> repetitive messages appearing in Zookeeper:
>
> These two Solr errors over and over:
>
> java.io.IOException: Xid out of order. Got Xid 1299 with err 0 expected Xid 
> 1298 for a packet with details: clientPath:null serverPath:null 
> finished:false header:: 1298,14  replyHeader:: 0,0,-4 request:: 
> org.apache.zookeeper.MultiTransactionRecord@95acc4f3 response:: 
> org.apache.zookeeper.MultiResponse@0

That appears to be one log message, but you said there were two.  Also,
this message is incomplete.  It is missing the timestamp at the
beginning and appears to have been cut off at the end too.  I think the
message probably had *many* more lines of output that weren't included.

> And this from Zookeeper:

> 2016-08-21 21:33:39,147 - WARN  
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@593] - Dropping 
> packet at server of type 14
> 2016-08-21 21:33:39,154 - WARN  
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] - 
> EndOfStreamException: Unable to read additional data from client sessionid 
> 0x156aefeba2d0004, likely client has closed socket

That seems to be saying that Solr closed the connection to zookeeper.  I
have no idea what might be wrong, based just on what's been provided
here.  This section of logging seems to contain everything related to
the specific connection from port 54548, and if that's true, then it
does not appear to have been a timeout.

Is there anything in the solr.log file at the timestamp at or near
21:33:39.154(when zookeeper thought the connection was closed)?

What version of Solr?  What version of zookeeper did you install on the
other node?

Thanks,
Shawn

SolrCloud: Collection creation failed due to timeout

[Chris Rogers]

Hi there,

I'm struggling with setting up a test implementation of SolrCloud using Vagrant.

I've got two vagrant boxes on a private network:

* One Ubuntu machine running Zookeeper
* One Ubuntu machine running Solr

My zookeeper config is as follows:


tickTime=4000

dataDir=/home/vagrant/zookeeper-data

clientPort=2181

server.1=localhost:2888:3888


Zookeeper is running as expected. I’m starting it in foreground mode with the 
command below:


bin/zkServer.sh start-foreground



I’m then starting Solr in cloud mode:


 bin/solr start -c -z 172.28.128.3:2181


At this point, all seems well. Nothing bad shows up in the Solr logs, and no 
alarming messages coming through in the console for Zookeeper.

I then try and create a collection with the command below:

  bin/solr create -c mycollection -d data_driven_schema_configs


It’s then that I start seeing lots of errors in the Solr logs, and lots of 
repetitive messages appearing in Zookeeper:

These two Solr errors over and over:

java.io.IOException: Xid out of order. Got Xid 1299 with err 0 expected Xid 
1298 for a packet with details: clientPath:null serverPath:null finished:false 
header:: 1298,14  replyHeader:: 0,0,-4 request:: 
org.apache.zookeeper.MultiTransactionRecord@95acc4f3 response:: 
org.apache.zookeeper.MultiResponse@0

There was a problem trying to register as the 
leader:org.apache.solr.common.SolrException: Could not register as the leader 
because creating the ephemeral registration node in ZooKeeper failed


And this from Zookeeper:

[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - 
Accepted socket connection from /172.28.128.4:54548
2016-08-21 21:33:39,137 - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@770] - Client 
attempting to renew session 0x156aefeba2d0004 at /172.28.128.4:54548
2016-08-21 21:33:39,145 - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1580] - Established 
session 0x156aefeba2d0004 with negotiated timeout 15000 for client 
/172.28.128.4:54548
2016-08-21 21:33:39,147 - WARN  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@593] - Dropping 
packet at server of type 14
2016-08-21 21:33:39,154 - WARN  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] - 
EndOfStreamException: Unable to read additional data from client sessionid 
0x156aefeba2d0004, likely client has closed socket
2016-08-21 21:33:39,155 - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket 
connection for client /172.28.128.4:54548 which had sessionid 0x156aefeba2d0004
2016-08-21 21:33:40,698 - INFO


Finally, the collection creation fails with the following message:

ERROR: Failed to create collection 'mycollection' due to: create the collection 
time out:180s


I can't find any useful references to these errors in SO or in the docs. Can 
anyone help me diagnose the problem?

Many thanks,
Chris

Chris Rogers
Digital Projects Manager
Bodleian Digital Library Systems and Services
chris.rog...@bodleian.ox.ac.uk

Question about highlighting html-tags

[Jannicke Husby]

Hi,


I have a question about searchterms and highlighting, hope you can help me.

The problem is that the HTML-tags get highlighted as well.


Example:

I search for "google". It hits the text "google" and that is a hyperlink like 
this https://www.google.com";>google.  I only want the text between 
the a-tags to get highlighted but it highlight the value in the 
"href"-attribute too.

I want it like this: https://www.google.com";>google
It get like this: https://www.google.com">google

So my question is: How can I prevent Solr to hits the HTML-tags in highlighted 
fields?



Jannicke Husby
Consultant

jannicke.hu...@acando.no

Mobile: +47 950 50 682

[http://www.acando.no/Content/101276/Bilde%20signatur.png]
Acando Norge
Tordenskioldsgate 8-10
0160 Oslo
+47 93 00 10 00
www.acando.no
Follow Acando
Facebook 
Twitter 
Linkedin

Re: help with DIH transformer to add a suffix to column names

[Emir Arnautovic]

Hi Wendy,

It seems to me that you misunderstood concept of dynamic fields. It is 
something that is defined in Solr schema, e.g. *_text, and then in your 
DIH config you define fields that match that pattern, e.g. name_text, 
desc_text etc.


HTH,
Emir

On 20.08.2016 00:58, Alexandre Rafalovitch wrote:

Can you give an example of what SQL column name and what Solr field name
you want correspondingly.

Because 'name_*' is not a valid field name.

Also, why specifically you are doing this.

Regards,
 Alex

On 20 Aug 2016 6:04 AM, "Wendy"  wrote:

Hi,How can I append a suffix  _* to all column names from a mysql database.I
am working on a project index data from mysql . I would like to use dynamic
field to dynamically index fields without specifying each field/column
names. I have been tried DIH customer transformer to append a suffix to
column name. But no error, no data. Does anyone has a good working
example?Thanks!



--
View this message in context: http://lucene.472066.n3.
nabble.com/help-with-DIH-transformer-to-add-a-suffix-
to-column-names-tp4292448.html
Sent from the Solr - User mailing list archive at Nabble.com.



--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/