Re: SOLR 4.4 - Slave always replicates full index
Erick: I now that. I didn't optimize the index frequently. The problem was more that a lot of documents have been added (without commit or autoCommit configured) to the index and MergePolicy kicked in and started merging the segments (i guess). This led to all segments beeing replicated because all of them had changed. I didn't find a solution yet so I am *now* optimizing the index after every full import (once a day) and replicate then to save the bandwidth.. I was more interested in the change to index strategy that Suresh mentioned. ~ Dom 2014-06-26 1:57 GMT+02:00 Erick Erickson erickerick...@gmail.com: Dominik: If you optimize your index, then the entire thing will be replicated from the master to the slave every time. In general, optimizing isn't necessary even though it sounds like something that's A Good Thing. I suspect that's the nub of the issue. Erick On Tue, Jun 24, 2014 at 11:14 PM, Dominik Siebel m...@dsiebel.de wrote: Hey Suresh, could you get a little more specific on what solved your problem here? I am currently facing the same problem and am trying to find a proper solution. Thanks! ~ Dom 2014-02-28 7:46 GMT+01:00 sureshrk19 sureshr...@gmail.com: Thanks Shawn and Erick. I followed SOLR configuration document and modified index strategy. Looks good now. I haven't seen any problems in last 1 week. Thanks for your suggestions. -- View this message in context: http://lucene.472066.n3.nabble.com/SOLR-4-4-Slave-always-replicates-full-index-tp4113089p4120337.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: SOLR 4.4 - Slave always replicates full index
Hey Suresh, could you get a little more specific on what solved your problem here? I am currently facing the same problem and am trying to find a proper solution. Thanks! ~ Dom 2014-02-28 7:46 GMT+01:00 sureshrk19 sureshr...@gmail.com: Thanks Shawn and Erick. I followed SOLR configuration document and modified index strategy. Looks good now. I haven't seen any problems in last 1 week. Thanks for your suggestions. -- View this message in context: http://lucene.472066.n3.nabble.com/SOLR-4-4-Slave-always-replicates-full-index-tp4113089p4120337.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr-4663 - Alternatives to use same data dir in different cores for optimal cache performance
Maybe you're right. The problem is that with the different types of queries it is hard to properly size document- and queryResultCaches (one query requests 10 results per page, others up to 12000). We tried different approaches, cache sizes and spend hours with JVM configuration (OutOfMemory problems and lot of stop the world collections). Having multiple cores, on the other hand, would drastically increase indexing time. I'm just looking at possible ways to improve query performance without increasing indexing time (only full imports possible). Regards Dom 2013/7/27 Erick Erickson erickerick...@gmail.com You can certainly have multiple Solrs pointing to the same underlying physical index if (and only if) you absolutely guarantee that only one Solr will write to the index at a time. But I'm not sure this is premature optimization or not. Problem is that your multiple Solrs are eating up the same physical memory, so I'm not quite sure whether the fact that you have different query characteristics is best served by multiple cores or not. Have you measured improvements with your proposed architecture? Best Erick On Fri, Jul 26, 2013 at 3:23 AM, Dominik Siebel m...@dsiebel.de wrote: Hi, I just found SOLR-4663 beeing patched in the latest update I did. Does anyone know any other solution to use ONE physical index for various purposes? Why? I would like to use different solconfig.xmls in terms of cache sizes, result window size, etc. per business case for optimal performance, while relying on the same data. This is due to the fact the queries are mostly completely different in structure and result size and we only have one unified search index (indexing performance). Any suggestions (besides replicating the index to another core on the same machine, of course ;) )? Cheers! Dom
Solr-4663 - Alternatives to use same data dir in different cores for optimal cache performance
Hi, I just found SOLR-4663 beeing patched in the latest update I did. Does anyone know any other solution to use ONE physical index for various purposes? Why? I would like to use different solconfig.xmls in terms of cache sizes, result window size, etc. per business case for optimal performance, while relying on the same data. This is due to the fact the queries are mostly completely different in structure and result size and we only have one unified search index (indexing performance). Any suggestions (besides replicating the index to another core on the same machine, of course ;) )? Cheers! Dom
Re: DIH throws NullPointerException when using dataimporter.functions.escapeSql with parent entities
In fact there are fields that have a NULL value but they are already
taken care of in the SQL Query like: IF(field_name IS NULL, '',
field_name).
Also it's not just single rows that fail. It's all of them.
It does not seem to have anything to do with the data that's coming
from the database. If I omit the dataimporter.functions.escapeSql the
importer at least processes those rows that don't have any SQL meta
characters in them.
Sadly, I haven't had time to put together a test to verify that the
error really emerges from the DIH itself.
Do you have any experience with bugreporting or submitting patches
(just in case there really IS a bug)?
2012/10/27 Lance Norskog goks...@gmail.com:
Which database rows cause the problem? The bug report talks about fields with
an empty string. Do your rows have empty string values?
- Original Message -
| From: Dominik Siebel m...@dsiebel.de
| To: solr-user@lucene.apache.org
| Sent: Monday, October 22, 2012 3:15:29 AM
| Subject: Re: DIH throws NullPointerException when using
dataimporter.functions.escapeSql with parent entities
|
| That's what I thought.
| I'm just curious that nobody else seems to have this problem although
| I found the exact same issue description in the issue tracker
| (https://issues.apache.org/jira/browse/SOLR-2141) which goes back to
| October 2010 and is flagged as Resolved: Cannot Reproduce.
|
|
| 2012/10/20 Lance Norskog goks...@gmail.com:
| If it worked before and does not work now, I don't think you are
| doing anything wrong :)
|
| Do you have a different version of your JDBC driver?
| Can you make a unit test with a minimal DIH script and schema?
| Or, scan through all of the JIRA issues against the DIH from your
| old Solr capture date.
|
|
| - Original Message -
| | From: Dominik Siebel m...@dsiebel.de
| | To: solr-user@lucene.apache.org
| | Sent: Thursday, October 18, 2012 11:22:54 PM
| | Subject: Fwd: DIH throws NullPointerException when using
| | dataimporter.functions.escapeSql with parent entities
| |
| | Hi folks,
| |
| | I am currently migrating our Solr servers from a 4.0.0 nightly
| | build
| | (aprox. November 2011, which worked very well) to the newly
| | released
| | 4.0.0 and am running into some issues concerning the existing
| | DataImportHandler configuratiions. Maybe you have an idea where I
| | am
| | going wrong here.
| |
| | The following lines are a highly simplified excerpt from one of
| | the
| | problematic imports:
| |
| | entity name=path rootEntity=false query=SELECT p.id,
| | IF(p.name
| | IS NULL, '', p.name) AS name FROM path p GROUP BY p.id
| |
| | entity name=item rootEntity=true query=
| | SELECT
| | i.*,
| |
| | CONVERT('${dataimporter.functions.escapeSql(path.name)}' USING
| | utf8) AS path_name
| | FROM items i
| | WHERE i.path_id = ${path.id} /
| |
| | /entity
| |
| | While this configuration worked without any problem for over half
| | a
| | year now, when upgrading to 4.0.0-BETA AND 4.0.0 the Import
| | throws
| | the
| | followeing Stacktrace and exits:
| |
| | SEVERE: Exception while processing: path document :
| | null:org.apache.solr.handler.dataimport.DataImportHandlerException:
| | java.lang.NullPointerException
| |
| | which is caused by
| |
| | Caused by: java.lang.NullPointerException
| | at
| |
org.apache.solr.handler.dataimport.EvaluatorBag$1.evaluate(EvaluatorBag.java:79)
| |
| | In other words: The EvaluatorBag doesn't seem to resolve the
| | given
| | path.name variable properly and returns null.
| |
| | Does anyone have any idea?
| | Appreciate your input!
| |
| | Regards
| | Dom
| |
|
Fwd: DIH throws NullPointerException when using dataimporter.functions.escapeSql with parent entities
Hi folks,
I am currently migrating our Solr servers from a 4.0.0 nightly build
(aprox. November 2011, which worked very well) to the newly released
4.0.0 and am running into some issues concerning the existing
DataImportHandler configuratiions. Maybe you have an idea where I am
going wrong here.
The following lines are a highly simplified excerpt from one of the
problematic imports:
entity name=path rootEntity=false query=SELECT p.id, IF(p.name
IS NULL, '', p.name) AS name FROM path p GROUP BY p.id
entity name=item rootEntity=true query=
SELECT
i.*,
CONVERT('${dataimporter.functions.escapeSql(path.name)}' USING
utf8) AS path_name
FROM items i
WHERE i.path_id = ${path.id} /
/entity
While this configuration worked without any problem for over half a
year now, when upgrading to 4.0.0-BETA AND 4.0.0 the Import throws the
followeing Stacktrace and exits:
SEVERE: Exception while processing: path document :
null:org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.NullPointerException
which is caused by
Caused by: java.lang.NullPointerException
at
org.apache.solr.handler.dataimport.EvaluatorBag$1.evaluate(EvaluatorBag.java:79)
In other words: The EvaluatorBag doesn't seem to resolve the given
path.name variable properly and returns null.
Does anyone have any idea?
Appreciate your input!
Regards
Dom
DIH throws NullPointerException when using dataimporter.functions.escapeSql with parent entities
Hi folks,
I am currently migrating our Solr servers from a 4.0.0 nightly build
(aprox. November 2011, which worked very well) to the newly released
4.0.0 and am running into some issues concerning the existing
DataImportHandler configuratiions. Maybe you have an idea where I am
going wrong here.
The following lines are a highly simplified excerpt from one of the
problematic imports:
entity name=path rootEntity=false query=SELECT p.id, IF(p.name
IS NULL, '', p.name) AS name FROM path p GROUP BY p.id
entity name=item rootEntity=true query=
SELECT
i.*,
CONVERT('${dataimporter.functions.escapeSql(path.name)}' USING
utf8) AS path_name
FROM items i
WHERE i.path_id = ${path.id} /
/entity
While this configuration worked without any problem for over half a
year now, when upgrading to 4.0.0-BETA AND 4.0.0 the Import throws the
followeing Stacktrace and exits:
SEVERE: Exception while processing: path document :
null:org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.NullPointerException
which is caused by
Caused by: java.lang.NullPointerException
at
org.apache.solr.handler.dataimport.EvaluatorBag$1.evaluate(EvaluatorBag.java:79)
In other words: The EvaluatorBag doesn't seem to resolve the given
path.name variable properly and returns null.
Does anyone have any idea?
Appreciate your input!
Regards
Dom
