Re: solr 4.2.1 still has problems with index version and index generation
Hi Hoss, we don't use autoCommit and autoSoftCommit. We don't use openSearcher. We don't use transaction log. I can see it in the AdminGUI and with http://master_host:port/solr/replication?command=indexversion All files are replicated from master to slave, nothing lost. It is just that the gen/version differs and breaks our cronjobs which worked since solr 2.x. As you mentioned, it seams that the latest commit is fetched. Strange thing is, we start with a clean, empty index on master. With all commands we send a commit=true and, where applicable, an optimze=true. The master is always in state optimzed and current when replicating. How can it be that the searcher on master is referring to an older commit point if there is no such point. The logs show _AFTER_ the last optimize has finished a new searcher is started and the old one is closed. Also, we have replicateAfter startup, commit and optimize set but the AdminGUI and replication details only report replicateAfter commit and startup. Not really an error, but not that what is really set in the config! Very strange, I will try the patch. Regards Bernd Am 08.04.2013 20:12, schrieb Chris Hostetter: : I know there was some effort to fix this but I must report : that solr 4.2.1 has still problems with index version and : index generation numbering in master/slave mode with replication. ... : RESULT: slave has different (higher) version number and is with generation 1 ahead :-( Can you please provide more details... * are you using autocommit? with what settings? * are you using openSearcher=false in any of your commits? * where exactly are you looking that you see the master/slave out of sync? * are you observing any actual problems, or just seeing that the gen/version are reported as different? As Joel mentioned, there is an open Jira related purely to the *display* of information about gen/version between master slave, because in many cases the searcher in use on the master may refer to an older commit point, but it doesn't mean there is any actual problem in replication -- the slave is still fetching seraching the latest commit from the master as intended https://issues.apache.org/jira/browse/SOLR-4661 -Hoss
Re: solr 4.2.1 still has problems with index version and index generation
Looking a bit deeper showed that replication?command=commit reports the right indexversion, generation and filelist. arr name=commits lst long name=indexVersion1365357951589/long long name=generation198/long arr name=filelist ... And with replication?command=details I also see the correct commit part as above, BUT where the hell are the wrong info below the commit array are coming from? str name=isMastertrue/str str name=isSlavefalse/str long name=indexVersion1365357774190/long long name=generation197/long The command replication?command=filelistgeneration=197 replies with str name=statusinvalid index generation/str Have a look into the sources: Ahh, it is build in getReplicationDetails with: details.add(isMaster, String.valueOf(isMaster)); details.add(isSlave, String.valueOf(isSlave)); long[] versionAndGeneration = getIndexVersion(); details.add(indexVersion, versionAndGeneration[0]); details.add(GENERATION, versionAndGeneration[1]); So getIndexVersion() gets a wrong version and generation, but why? It first gets the searcher from the core and then tries to get via the IndexReader the IndexCommit and then the commitData. I think I should use remote debugging on master. At least I now know that it is the master. Regards Bernd Am 09.04.2013 08:35, schrieb Bernd Fehling: Hi Hoss, we don't use autoCommit and autoSoftCommit. We don't use openSearcher. We don't use transaction log. I can see it in the AdminGUI and with http://master_host:port/solr/replication?command=indexversion All files are replicated from master to slave, nothing lost. It is just that the gen/version differs and breaks our cronjobs which worked since solr 2.x. As you mentioned, it seams that the latest commit is fetched. Strange thing is, we start with a clean, empty index on master. With all commands we send a commit=true and, where applicable, an optimze=true. The master is always in state optimzed and current when replicating. How can it be that the searcher on master is referring to an older commit point if there is no such point. The logs show _AFTER_ the last optimize has finished a new searcher is started and the old one is closed. Also, we have replicateAfter startup, commit and optimize set but the AdminGUI and replication details only report replicateAfter commit and startup. Not really an error, but not that what is really set in the config! Very strange, I will try the patch. Regards Bernd Am 08.04.2013 20:12, schrieb Chris Hostetter: : I know there was some effort to fix this but I must report : that solr 4.2.1 has still problems with index version and : index generation numbering in master/slave mode with replication. ... : RESULT: slave has different (higher) version number and is with generation 1 ahead :-( Can you please provide more details... * are you using autocommit? with what settings? * are you using openSearcher=false in any of your commits? * where exactly are you looking that you see the master/slave out of sync? * are you observing any actual problems, or just seeing that the gen/version are reported as different? As Joel mentioned, there is an open Jira related purely to the *display* of information about gen/version between master slave, because in many cases the searcher in use on the master may refer to an older commit point, but it doesn't mean there is any actual problem in replication -- the slave is still fetching seraching the latest commit from the master as intended https://issues.apache.org/jira/browse/SOLR-4661 -Hoss -- * Bernd FehlingBielefeld University Library Dipl.-Inform. (FH)LibTec - Library Technology Universitätsstr. 25 and Knowledge Management 33615 Bielefeld Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de BASE - Bielefeld Academic Search Engine - www.base-search.net *
Re: solr 4.2.1 still has problems with index version and index generation
: And with replication?command=details I also see the correct commit part as : above, BUT where the hell are the wrong info below the commit array are : coming from? Please read the details in the previously mentioned Jira issue... https://issues.apache.org/jira/browse/SOLR-4661 The indexVersion and generation you are looking at refer to the speciics of the IndexReader as used by the *seracher* on the master server -- but in addition to situations like openSearcher=false, there are some optimizations in place such that Solr/Lucene is smart enough to realize when an empty commit doesn't change the IndexReader it continues to use the previous commit point... https://issues.apache.org/jira/browse/SOLR-4661?focusedCommentId=13620195page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13620195 ...but from the perspective of the slave, this is still a commit that needs replicated and loaded. Hence the current objective of the patch in SOLR-3855: add more details to the command=details response (as well as the Admin UI) to clearly distinguish between the gen/ver of the currently replicatable commit and the gen/ver of the currently open searcher. All available information suggests that this is purely a problem of conveying information to users via command=details -- Replication is behaving as designed using the correct information about hte commit points. -Hoss
solr 4.2.1 still has problems with index version and index generation
I know there was some effort to fix this but I must report that solr 4.2.1 has still problems with index version and index generation numbering in master/slave mode with replication. Test was: 1. installed solr 4.2.1 on master and build index from scratch 2. installed solr 4.2.1 on slave with empty index 3. replicated master to slave, everything was fine, both in sync 4. deleted index on master with *:* and build index from scratch 5. replicated index from master to slave RESULT: slave has different (higher) version number and is with generation 1 ahead :-( Regards Bernd
Re: solr 4.2.1 still has problems with index version and index generation
I'm on 4.1 and I have a similar problem. Except for the version number everything else seems to be fine. Is that what other people are seeing? -- View this message in context: http://lucene.472066.n3.nabble.com/solr-4-2-1-still-has-problems-with-index-version-and-index-generation-tp405p4054463.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr 4.2.1 still has problems with index version and index generation
This is the jira issue that addresses this: https://issues.apache.org/jira/browse/SOLR-4661 I'll try to find some time today and test out the patch and see how things look. On Mon, Apr 8, 2013 at 7:18 AM, Tom Gullo springmeth...@gmail.com wrote: I'm on 4.1 and I have a similar problem. Except for the version number everything else seems to be fine. Is that what other people are seeing? -- View this message in context: http://lucene.472066.n3.nabble.com/solr-4-2-1-still-has-problems-with-index-version-and-index-generation-tp405p4054463.html Sent from the Solr - User mailing list archive at Nabble.com. -- Joel Bernstein Professional Services LucidWorks
Re: solr 4.2.1 still has problems with index version and index generation
: I know there was some effort to fix this but I must report : that solr 4.2.1 has still problems with index version and : index generation numbering in master/slave mode with replication. ... : RESULT: slave has different (higher) version number and is with generation 1 ahead :-( Can you please provide more details... * are you using autocommit? with what settings? * are you using openSearcher=false in any of your commits? * where exactly are you looking that you see the master/slave out of sync? * are you observing any actual problems, or just seeing that the gen/version are reported as different? As Joel mentioned, there is an open Jira related purely to the *display* of information about gen/version between master slave, because in many cases the searcher in use on the master may refer to an older commit point, but it doesn't mean there is any actual problem in replication -- the slave is still fetching seraching the latest commit from the master as intended https://issues.apache.org/jira/browse/SOLR-4661 -Hoss
