Re: documents are not commited distributively in solr cloud tomcat with core discovery, range is null for shards in clusterstate.json
I've solved this problem myself. If you use core discovery, you must specify the "numShards" parameter in core.properties. or else solr won't be allocate range for each shards and then documents won't be distributed properly. Using core discovery to set up solr cloud in tomcat is much easier and clean than coreAdmin described in the wiki: http://wiki.apache.org/solr/SolrCloudTomcat. It costs me some time to move from jetty to tomcat, but I think our IT team will like this way. :) On 6 October 2013 23:53, Liu Bo wrote: > Hi all > > I've sent out this mail before, but I only subscribed to lucene-user but > not solr-user at that time. Sorry for repeating if any and your help will > be much of my appreciation. > > I'm trying out the tutorial about solrcloud, and then I manage to write my > own plugin to import data from our set of databases, I use SolrWriter from > DataImporter package and the docs could be distributed commit to shards. > > Every thing works fine using jetty from the solr example, but when I move > to tomcat, solrcloud seems not been configured right. As the documents are > just committed to the shard where update requested goes to. > > The cause probably is the range is null for shards in clusterstate.json. > The router is "implicit" instead of "compositeId" as well. > > Is there anything missed or configured wrong in the following steps? How > could I fix it. Your help will be much of my appreciation. > > PS, solr cloud tomcat wiki page isn't up to 4.4 with core discovery, I'm > trying out after reading SoclrCloud, SolrCloudJboss, and CoreAdmin wiki > pages. > > Here's what I've done and some useful logs: > > 1. start three zookeeper server. > 2. upload configuration files to zookeeper, the collection name is > "content_collection" > 3. start three tomcat instants on three server with core discovery > > a) core file: > name=content > loadOnStartup=true > transient=false > shard=shard1 (differrent on servers) > collection=content_collection > b) solr.xml > > > > > > ${host:} > > ${hostContext:solr} > > 8080 > > ${zkClientTimeout:15000} > > 10.199.46.176:2181,10.199.46.165:2181, > 10.199.46.158:2181 > > ${genericCoreNodeNames:true} > > > > > > class="HttpShardHandlerFactory"> > > ${socketTimeout:0} > > ${connTimeout:0} > > > > > > 4. In the solr.log, I see the three shards are recognized, and the > solrcloud can see the content_collection has three shards as well. > 5. write documents to content_collection using my update request, the > documents only commits to the shard the request goes to, in the log I can > see the DistributedUpdateProcessorFactory is in the processorChain and > disribute commit is triggered: > > INFO - 2013-09-30 16:31:43.205; > com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler; > updata request processor factories: > > INFO - 2013-09-30 16:31:43.206; > com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler; > org.apache.solr.update.processor.LogUpdateProcessorFactory@4ae7b77 > > INFO - 2013-09-30 16:31:43.207; > com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler; > org.apache.solr.update.processor.*DistributedUpdateProcessorFactory* > @5b2bc407 > > INFO - 2013-09-30 16:31:43.207; > com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler; > org.apache.solr.update.processor.RunUpdateProcessorFactory@1652d654 > > INFO - 2013-09-30 16:31:43.283; org.apache.solr.core.SolrDeletionPolicy; > SolrDeletionPolicy.onInit: commits: num=1 > > > commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1} > > INFO - 2013-09-30 16:31:43.284; org.apache.solr.core.SolrDeletionPolicy; > newest commit generation = 1 > > INFO - 2013-09-30 16:31:43.440; *org.apache.solr.update.SolrCmdDistributor; > Distrib commit to*:[StdNode: http://10.199.46.176:8080/solr/content/, > StdNode: http://10.199.46.165:8080/solr/content/] > params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false > > but the documents won't go to other shards, the other shards only has a > request with not documents: > > INFO - 2013-09-30 16:31:43.841; > org.apache.solr.update.DirectUpdateHandler2; start > commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} > > INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy; > SolrDeletionPolicy.onInit: commits: num=1 > > > commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1} > > INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy; > newest commit generation = 1 > > INFO - 2013-09-30 16:31:43.856; > org.apache.solr.update.DirectUpdateHandler2; No uncommitted changes. > Skipping IW.commit. > > INFO - 2013-09-30 1
documents are not commited distributively in solr cloud tomcat with core discovery, range is null for shards in clusterstate.json
Hi all I've sent out this mail before, but I only subscribed to lucene-user but not solr-user at that time. Sorry for repeating if any and your help will be much of my appreciation. I'm trying out the tutorial about solrcloud, and then I manage to write my own plugin to import data from our set of databases, I use SolrWriter from DataImporter package and the docs could be distributed commit to shards. Every thing works fine using jetty from the solr example, but when I move to tomcat, solrcloud seems not been configured right. As the documents are just committed to the shard where update requested goes to. The cause probably is the range is null for shards in clusterstate.json. The router is "implicit" instead of "compositeId" as well. Is there anything missed or configured wrong in the following steps? How could I fix it. Your help will be much of my appreciation. PS, solr cloud tomcat wiki page isn't up to 4.4 with core discovery, I'm trying out after reading SoclrCloud, SolrCloudJboss, and CoreAdmin wiki pages. Here's what I've done and some useful logs: 1. start three zookeeper server. 2. upload configuration files to zookeeper, the collection name is "content_collection" 3. start three tomcat instants on three server with core discovery a) core file: name=content loadOnStartup=true transient=false shard=shard1 (differrent on servers) collection=content_collection b) solr.xml ${host:} ${hostContext:solr} 8080 ${zkClientTimeout:15000} 10.199.46.176:2181,10.199.46.165:2181, 10.199.46.158:2181 ${genericCoreNodeNames:true} ${socketTimeout:0} ${connTimeout:0} 4. In the solr.log, I see the three shards are recognized, and the solrcloud can see the content_collection has three shards as well. 5. write documents to content_collection using my update request, the documents only commits to the shard the request goes to, in the log I can see the DistributedUpdateProcessorFactory is in the processorChain and disribute commit is triggered: INFO - 2013-09-30 16:31:43.205; com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler; updata request processor factories: INFO - 2013-09-30 16:31:43.206; com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler; org.apache.solr.update.processor.LogUpdateProcessorFactory@4ae7b77 INFO - 2013-09-30 16:31:43.207; com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler; org.apache.solr.update.processor.*DistributedUpdateProcessorFactory* @5b2bc407 INFO - 2013-09-30 16:31:43.207; com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler; org.apache.solr.update.processor.RunUpdateProcessorFactory@1652d654 INFO - 2013-09-30 16:31:43.283; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits: num=1 commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1} INFO - 2013-09-30 16:31:43.284; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 1 INFO - 2013-09-30 16:31:43.440; *org.apache.solr.update.SolrCmdDistributor; Distrib commit to*:[StdNode: http://10.199.46.176:8080/solr/content/, StdNode: http://10.199.46.165:8080/solr/content/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false but the documents won't go to other shards, the other shards only has a request with not documents: INFO - 2013-09-30 16:31:43.841; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits: num=1 commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1} INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 1 INFO - 2013-09-30 16:31:43.856; org.apache.solr.update.DirectUpdateHandler2; No uncommitted changes. Skipping IW.commit. INFO - 2013-09-30 16:31:43.865; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@3c74c144 main INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener; QuerySenderListener sending requests to Searcher@3c74c144main{StandardDirectoryReader(segments_1:1:nrt)} INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener; QuerySenderListener done. INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.SolrCore; [content] Registered new searcher Searcher@3c74c144main{StandardDirectoryReader(segments_1:1:nrt)} INFO - 2013-09-30 16:31:43.870; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush INFO - 2013-09-30 16:31:43.870; org.apache.solr.update.processor.LogUpdateProcessor; [content] webapp=/solr path=/update params={waitSearcher=true&commit=tru
documents are not commited distributively in solr cloud tomcat with core discovery, range is null for shards in clusterstate.json
Hi all I'm trying out the tutorial about solrcloud, and then I manage to write my own plugin to import data from our set of databases, I use SolrWriter from DataImporter package and the docs could be distributed commit to shards. Every thing works fine using jetty from the solr example, but when I move to tomcat, solrcloud seems not been configured right. As the documents are just committed to the shard where update requested goes to. The cause probably is the range is null for shards in clusterstate.json. The router is "implicit" instead of "compositeId" as well. Is there anything missed or configured wrong in the following steps? How could I fix it. Your help will be much of my appreciation. PS, solr cloud tomcat wiki page isn't up to 4.4 with core discovery, I'm trying out after reading SoclrCloud, SolrCloudJboss, and CoreAdmin wiki pages. Here's what I've done and some useful logs: 1. start three zookeeper server. 2. upload configuration files to zookeeper, the collection name is "content_collection" 3. start three tomcat instants on three server with core discovery a) core file: name=content loadOnStartup=true transient=false shard=shard1 (differrent on servers) collection=content_collection b) solr.xml ${host:} ${hostContext:solr} 8080 ${zkClientTimeout:15000} 10.199.46.176:2181,10.199.46.165:2181, 10.199.46.158:2181 ${genericCoreNodeNames:true} ${socketTimeout:0} ${connTimeout:0} 4. In the solr.log, I see the three shards are recognized, and the solrcloud can see the content_collection has three shards as well. 5. write documents to content_collection using my update request, the documents only commits to the shard the request goes to, in the log I can see the DistributedUpdateProcessorFactory is in the processorChain and disribute commit is triggered: INFO - 2013-09-30 16:31:43.205; com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler; updata request processor factories: INFO - 2013-09-30 16:31:43.206; com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler; org.apache.solr.update.processor.LogUpdateProcessorFactory@4ae7b77 INFO - 2013-09-30 16:31:43.207; com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler; org.apache.solr.update.processor.*DistributedUpdateProcessorFactory* @5b2bc407 INFO - 2013-09-30 16:31:43.207; com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler; org.apache.solr.update.processor.RunUpdateProcessorFactory@1652d654 INFO - 2013-09-30 16:31:43.283; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits: num=1 commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1} INFO - 2013-09-30 16:31:43.284; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 1 INFO - 2013-09-30 16:31:43.440; *org.apache.solr.update.SolrCmdDistributor; Distrib commit to*:[StdNode: http://10.199.46.176:8080/solr/content/, StdNode: http://10.199.46.165:8080/solr/content/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false but the documents won't go to other shards, the other shards only has a request with not documents: INFO - 2013-09-30 16:31:43.841; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits: num=1 commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1} INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 1 INFO - 2013-09-30 16:31:43.856; org.apache.solr.update.DirectUpdateHandler2; No uncommitted changes. Skipping IW.commit. INFO - 2013-09-30 16:31:43.865; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@3c74c144 main INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener; QuerySenderListener sending requests to Searcher@3c74c144main{StandardDirectoryReader(segments_1:1:nrt)} INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener; QuerySenderListener done. INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.SolrCore; [content] Registered new searcher Searcher@3c74c144main{StandardDirectoryReader(segments_1:1:nrt)} INFO - 2013-09-30 16:31:43.870; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush INFO - 2013-09-30 16:31:43.870; org.apache.solr.update.processor.LogUpdateProcessor; [content] webapp=/solr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 42 6) later I found the range is null in clusterstate.json which might have ca