Re: setup solrcloud from scratch vie web-ui
> Shawn Heiseyhat am 17. Mai 2017 um 15:10 geschrieben: > > > On 5/17/2017 6:18 AM, Thomas Porschberg wrote: > > Thank you. I am now a step further. > > I could import data into the new collection with the DIH. However I > > observed the following exception > > in solr.log: > > > > request: > > http://127.0.1.1:8983/solr/hugo_shard1_replica1/update?update.distrib=TOLEADER=http%3A%2F%2F127.0.1.1%3A8983%2Fsolr%2Fhugo_shard2_replica1%2F=javabin=2 > > Remote error message: This IndexSchema is not mutable. > > This probably means that the configuration has an update processor that > adds unknown fields, but is using the classic schema instead of the > managed schema. If you want unknown fields to automatically be guessed > and added, then you need the managed schema. If not, then remove the > custom update processor chain. If this doesn't sound like what's wrong, > then we will need the entire error message including the full Java > stacktrace. That may be in the other instance's solr.log file. Ok, commenting out the "update processor chain" was a solution. I use classic schema. > > > I imagine to split my data per day of the year. My idea was to create 365 > > shards of type compositeKey. > > You cannot control shard routing explicitly with the compositeId > router. That router uses a hash of the uniqueKey field to decide which > shard gets the document. As its name implies, the hash can be composite > -- parts of the hash can be decided by multiple parts of the value in > the field, but it's still hashed. > > You must use the implicit router (which means all routing is manual) if > you want to explicitly name the shard that receives the data. I was now able to create 365 shards with the 'implicit' router. In the collection-API call I also specified router.field=part_crit which is the day of the year 1..365 I added this field in my SQL-statement and in schema.xml. Next step I thought would be to trigger the dataimport. However I get: 2017-05-18 05:41:37.417 ERROR (Thread-14) [c:hansi s:308 r:core_node76 x:hansi_308_replica1] o.a.s.h.d.DataImporter Full Import failed:java.lang.RuntimeException: org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: hansi slice: 230 at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:270) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:475) at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:458) at java.lang.Thread.run(Thread.java:745) when I start the import. What could be the reason? Thank you Thomas
Re: setup solrcloud from scratch vie web-ui
On 5/17/2017 6:18 AM, Thomas Porschberg wrote: > Thank you. I am now a step further. > I could import data into the new collection with the DIH. However I observed > the following exception > in solr.log: > > request: > http://127.0.1.1:8983/solr/hugo_shard1_replica1/update?update.distrib=TOLEADER=http%3A%2F%2F127.0.1.1%3A8983%2Fsolr%2Fhugo_shard2_replica1%2F=javabin=2 > Remote error message: This IndexSchema is not mutable. This probably means that the configuration has an update processor that adds unknown fields, but is using the classic schema instead of the managed schema. If you want unknown fields to automatically be guessed and added, then you need the managed schema. If not, then remove the custom update processor chain. If this doesn't sound like what's wrong, then we will need the entire error message including the full Java stacktrace. That may be in the other instance's solr.log file. > I imagine to split my data per day of the year. My idea was to create 365 > shards of type compositeKey. You cannot control shard routing explicitly with the compositeId router. That router uses a hash of the uniqueKey field to decide which shard gets the document. As its name implies, the hash can be composite -- parts of the hash can be decided by multiple parts of the value in the field, but it's still hashed. You must use the implicit router (which means all routing is manual) if you want to explicitly name the shard that receives the data. Thanks, Shawn
Re: setup solrcloud from scratch vie web-ui
> Tom Evanshat am 17. Mai 2017 um 11:48 geschrieben: > > > On Wed, May 17, 2017 at 6:28 AM, Thomas Porschberg > wrote: > > Hi, > > > > I did not manipulating the data dir. What I did was: > > > > 1. Downloaded solr-6.5.1.zip > > 2. ensured no solr process is running > > 3. unzipped solr-6.5.1.zip to ~/solr_new2/solr-6.5.1 > > 3. started an external zookeeper > > 4. copied a conf directory from a working non-cloudsolr (6.5.1) to > >~/solr_new2/solr-6.5.1 so that I have ~/solr_new2/solr-6.5.1/conf > > (see http://randspringer.de/solrcloud_test/my.zip for content) > > ..in which you've manipulated the dataDir! :) > > The problem (I think) is that you have set a fixed data dir, and when > Solr attempts to create a second core (for whatever reason, in your > case it looks like you are adding a shard), Solr puts it exactly where > you have told it to, in the same directory as the previous one. It > finds the lock and blows up, because each core needs to be in a > separate directory, but you've instructed Solr to put them in the same > one. > > Start with a the solrconfig from basic_configs configset that ships > with Solr and add the special things that your installation needs. I > am not massively surprised that your non cloud config does not work in > cloud mode, when we moved to SolrCloud, we rewrote from scratch > solrconfig.xml and schema.xml, starting from basic_configs and adding > anything particular that we needed from our old config, checking every > difference that we have from stock config and noting/discerning why, > and ensuring that our field types are using the same names for the > same types as basic_config wherever possible. > > I only say all that because to fix this issue is a single thing, but > you should spend the time comparing configs because this will not be > the only issue. Anyway, to fix this problem, in your solrconfig.xml > you have: > > data > > It should be > > ${solr.data.dir:} > > Which is still in your config, you've just got it commented out :) Thank you. I am now a step further. I could import data into the new collection with the DIH. However I observed the following exception in solr.log: request: http://127.0.1.1:8983/solr/hugo_shard1_replica1/update?update.distrib=TOLEADER=http%3A%2F%2F127.0.1.1%3A8983%2Fsolr%2Fhugo_shard2_replica1%2F=javabin=2 Remote error message: This IndexSchema is not mutable. at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:345) I also noticed that only one shard is filled. The wiki describes how to populate data with the rest api. However I use the data importer. I imagine to split my data per day of the year. My idea was to create 365 shards of type compositeKey. In my SQL I have a date field and it is no problem to overwrite data after one year. However I'm looking for a good example how to achieve this. May be I need in this case 365 dataimport.xml files under each shard one... with some modulo-expression for the specific day. Currently the dataimport.xml is in the conf directory. So I'm looking for a good example how to use the DIH with solrcloud. Should it work to create a implicit router instead of compositeKey router (with 365 shards) and simply specfiy as router.field= ? Thomas
Re: setup solrcloud from scratch vie web-ui
On Wed, May 17, 2017 at 6:28 AM, Thomas Porschbergwrote: > Hi, > > I did not manipulating the data dir. What I did was: > > 1. Downloaded solr-6.5.1.zip > 2. ensured no solr process is running > 3. unzipped solr-6.5.1.zip to ~/solr_new2/solr-6.5.1 > 3. started an external zookeeper > 4. copied a conf directory from a working non-cloudsolr (6.5.1) to >~/solr_new2/solr-6.5.1 so that I have ~/solr_new2/solr-6.5.1/conf > (see http://randspringer.de/solrcloud_test/my.zip for content) ..in which you've manipulated the dataDir! :) The problem (I think) is that you have set a fixed data dir, and when Solr attempts to create a second core (for whatever reason, in your case it looks like you are adding a shard), Solr puts it exactly where you have told it to, in the same directory as the previous one. It finds the lock and blows up, because each core needs to be in a separate directory, but you've instructed Solr to put them in the same one. Start with a the solrconfig from basic_configs configset that ships with Solr and add the special things that your installation needs. I am not massively surprised that your non cloud config does not work in cloud mode, when we moved to SolrCloud, we rewrote from scratch solrconfig.xml and schema.xml, starting from basic_configs and adding anything particular that we needed from our old config, checking every difference that we have from stock config and noting/discerning why, and ensuring that our field types are using the same names for the same types as basic_config wherever possible. I only say all that because to fix this issue is a single thing, but you should spend the time comparing configs because this will not be the only issue. Anyway, to fix this problem, in your solrconfig.xml you have: data It should be ${solr.data.dir:} Which is still in your config, you've just got it commented out :) Cheers Tom
Re: setup solrcloud from scratch vie web-ui
Hi, I did not manipulating the data dir. What I did was: 1. Downloaded solr-6.5.1.zip 2. ensured no solr process is running 3. unzipped solr-6.5.1.zip to ~/solr_new2/solr-6.5.1 3. started an external zookeeper 4. copied a conf directory from a working non-cloudsolr (6.5.1) to ~/solr_new2/solr-6.5.1 so that I have ~/solr_new2/solr-6.5.1/conf (see http://randspringer.de/solrcloud_test/my.zip for content) 5. postd the conf to zookeeper with: bin/solr zk upconfig -n heise -d ./conf -z localhost:2181 6. started solr in cloud mode with bin/solr -c -z localhost:2181 7. tried to create a acollection with bin/solr create -c heise -shards 2 -->failed with: Connecting to ZooKeeper at localhost:2181 ... INFO - 2017-05-17 07:06:38.249; org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider; Cluster at localhost:2181 ready Re-using existing configuration directory heise Creating new collection 'heise' using command: http://localhost:8983/solr/admin/collections?action=CREATE=heise=2=1=2=heise ERROR: Failed to create collection 'heise' due to: {127.0.1.1:8983_solr=org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://127.0.1.1:8983/solr: Error CREATEing SolrCore 'heise_shard2_replica1': Unable to create core [heise_shard2_replica1] Caused by: Lock held by this virtual machine: /home/pberg/solr_new2/solr-6.5.1/server/data/index/write.lock} 8. Tried with 1 shard, worked --> pberg@porschberg:~/solr_new2/solr-6.5.1$ bin/solr create -c heise -shards 1 Connecting to ZooKeeper at localhost:2181 ... INFO - 2017-05-17 07:21:01.632; org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider; Cluster at localhost:2181 ready Re-using existing configuration directory heise Creating new collection 'heise' using command: http://localhost:8983/solr/admin/collections?action=CREATE=heise=1=1=1=heise { "responseHeader":{ "status":0, "QTime":2577}, "success":{"127.0.1.1:8983_solr":{ "responseHeader":{ "status":0, "QTime":1441}, "core":"heise_shard1_replica1"}}} What did I wrong? I want to use multiple shards on ONE node. Best regards Thomas > Shawn Heiseyhat am 16. Mai 2017 um 16:30 geschrieben: > > > On 5/12/2017 8:49 AM, Thomas Porschberg wrote: > > ERROR: Failed to create collection 'cat' due to: > > {127.0.1.1:8983_solr=org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error > > from server at http://127.0.1.1:8983/solr: Error CREATEing SolrCore > > 'cat_shard1_replica1': Unable to create core [cat_shard1_replica1] Caused > > by: Lock held by this virtual machine: > > /home/pberg/solr_new2/solr-6.5.1/server/data/bestand/index/write.lock} > > The same Solr instance is already holding the lock on the index at > /home/pberg/solr_new2/solr-6.5.1/server/data/bestand/index. This means > that Solr already has a core using that index directory. > > If the write.lock were present but wasn't being held by the same > instance, then the message would have said it was held by another program. > > This sounds like you are manually manipulating settings like dataDir. > When you start the server from an extracted download (not as a service) > and haven't messed with any configurations, the index directory for a > single-shard single-replica "cat" collection should be something like > the following, and should not be overridden unless you understand > *EXACTLY* how SolrCloud functions and have a REALLY good reason for > changing it: > > /home/pberg/solr_new2/solr-6.5.1/server/solr/cat_shard1_replica1/data/index > > On the "Sorry, no dataimport-handler defined!" problem, this is > happening because the solrconfig.xml file being used by the collection > does not have any configuration for the dataimport handler. It's not > enough to add a DIH config file, solrconfig.xml must have a dataimport > handler defined that references the DIH config file. > > Thanks, > Shawn >
Re: setup solrcloud from scratch vie web-ui
On 5/12/2017 8:49 AM, Thomas Porschberg wrote: > ERROR: Failed to create collection 'cat' due to: > {127.0.1.1:8983_solr=org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error > from server at http://127.0.1.1:8983/solr: Error CREATEing SolrCore > 'cat_shard1_replica1': Unable to create core [cat_shard1_replica1] Caused by: > Lock held by this virtual machine: > /home/pberg/solr_new2/solr-6.5.1/server/data/bestand/index/write.lock} The same Solr instance is already holding the lock on the index at /home/pberg/solr_new2/solr-6.5.1/server/data/bestand/index. This means that Solr already has a core using that index directory. If the write.lock were present but wasn't being held by the same instance, then the message would have said it was held by another program. This sounds like you are manually manipulating settings like dataDir. When you start the server from an extracted download (not as a service) and haven't messed with any configurations, the index directory for a single-shard single-replica "cat" collection should be something like the following, and should not be overridden unless you understand *EXACTLY* how SolrCloud functions and have a REALLY good reason for changing it: /home/pberg/solr_new2/solr-6.5.1/server/solr/cat_shard1_replica1/data/index On the "Sorry, no dataimport-handler defined!" problem, this is happening because the solrconfig.xml file being used by the collection does not have any configuration for the dataimport handler. It's not enough to add a DIH config file, solrconfig.xml must have a dataimport handler defined that references the DIH config file. Thanks, Shawn
Re: setup solrcloud from scratch vie web-ui
Hi, I think I did one mistake when I started in step 3 solr without zookeeper-option. I did: bin/solr start -c but I think it should: bin/solr start -c -z localhost:2181 The problem is now when repeat step 4 (creating a collection) I get the following error: //I uploaded my cat-config again to zookeeper with // bin/solr zk upconfig -n cat -d $HOME/solr-6.5.1/server/solr/tommy/conf -z // localhost:2181 bin/solr create -c cat -shards 2 Connecting to ZooKeeper at localhost:2181 ... INFO - 2017-05-12 16:38:06.593; org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider; Cluster at localhost:2181 ready Re-using existing configuration directory cat Creating new collection 'cat' using command: http://localhost:8983/solr/admin/collections?action=CREATE=cat=2=1=2=cat ERROR: Failed to create collection 'cat' due to: {127.0.1.1:8983_solr=org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://127.0.1.1:8983/solr: Error CREATEing SolrCore 'cat_shard1_replica1': Unable to create core [cat_shard1_replica1] Caused by: Lock held by this virtual machine: /home/pberg/solr_new2/solr-6.5.1/server/data/bestand/index/write.lock} This "data/bestand" is configured in solrconfig.xml (from tommy standalone) with data/bestand I tried to create the directory /home/pberg/solr_new2/solr-6.5.1/server/data/bestand/index/ manually , but nothing changed. What is the reason for this CREATE-error? Thomas > ANNAMANENI RAVEENDRAhat am 12. Mai 2017 um 15:54 > geschrieben: > > > Hi , > > If there is a request handler configured in solrconfig.xml and update the > Conf in zookeeper it should show up > > If already did it try reloading configuration > > Thanks > Ravi > > > On Fri, 12 May 2017 at 9:46 AM, Thomas Porschberg > wrote: > > > > > This is another problem I see: With my non-cloud core I have a > > conf-directory where I have dataimport.xml, schema.xml and solrconfig.xml. > > > > I think these 3 files are enough to import my data from my relational > > database. > > > > Under example/cloud I could not find one of them. How to setup DIH for > > the solrcould? > > > > > > The entire configuration (what would normally be in the conf directory) > > > is in zookeeper when you're in cloud mode, not in the core directories. > > > You must upload a directory containing the same files that would > > > normally be in a conf directory as a named configset to zookeeper before > > > you try to create your collection. This is something that the "bin/solr > > > create" command does for you in cloud mode, typically using one of the > > > configsets included on the disk as a source. > > > > > > > > https://cwiki.apache.org/confluence/display/solr/Using+ZooKeeper+to+Manage+Configuration+Files > > > > > Ok, thank you. I did the following steps. > > > > 1. Started an external zookeeper > > 2. Copied a conf-directory to zookeeper: > > bin/solr zk upconfig -n books -d $HOME/solr-6.5.1/server/solr/tommy/conf > > -z localhost:2181 > > // This is a conf-directory from a standalone solr when dataimport was > > working! > > --> Connecting to ZooKeeper at localhost:2181 ... > > Uploading <> for config books to ZooKeeper at localhost:2181 > > // I think no errors, but how can I check it in zookeeper? I found no > > files solrconfig.xml ... > > in the zookeeper directories (installation dir and data dir) > > 3. Started solr: > > bin/solr start -c > > 4. Created a books collection with 2 shards > > bin/solr create -c books -shards 2 > > > > Result: I see in the web-ui my books collection with the 2 shards. No > > errors so far. > > However, the Dataimport-entry says: > > "Sorry, no dataimport-handler defined!" > > > > What could be the reason? > > > > Thomas > >
Re: setup solrcloud from scratch vie web-ui
Hi , If there is a request handler configured in solrconfig.xml and update the Conf in zookeeper it should show up If already did it try reloading configuration Thanks Ravi On Fri, 12 May 2017 at 9:46 AM, Thomas Porschbergwrote: > > > This is another problem I see: With my non-cloud core I have a > conf-directory where I have dataimport.xml, schema.xml and solrconfig.xml. > > > I think these 3 files are enough to import my data from my relational > database. > > > Under example/cloud I could not find one of them. How to setup DIH for > the solrcould? > > > > The entire configuration (what would normally be in the conf directory) > > is in zookeeper when you're in cloud mode, not in the core directories. > > You must upload a directory containing the same files that would > > normally be in a conf directory as a named configset to zookeeper before > > you try to create your collection. This is something that the "bin/solr > > create" command does for you in cloud mode, typically using one of the > > configsets included on the disk as a source. > > > > > https://cwiki.apache.org/confluence/display/solr/Using+ZooKeeper+to+Manage+Configuration+Files > > > Ok, thank you. I did the following steps. > > 1. Started an external zookeeper > 2. Copied a conf-directory to zookeeper: > bin/solr zk upconfig -n books -d $HOME/solr-6.5.1/server/solr/tommy/conf > -z localhost:2181 > // This is a conf-directory from a standalone solr when dataimport was > working! > --> Connecting to ZooKeeper at localhost:2181 ... > Uploading <> for config books to ZooKeeper at localhost:2181 > // I think no errors, but how can I check it in zookeeper? I found no > files solrconfig.xml ... > in the zookeeper directories (installation dir and data dir) > 3. Started solr: > bin/solr start -c > 4. Created a books collection with 2 shards > bin/solr create -c books -shards 2 > > Result: I see in the web-ui my books collection with the 2 shards. No > errors so far. > However, the Dataimport-entry says: > "Sorry, no dataimport-handler defined!" > > What could be the reason? > > Thomas >
Re: setup solrcloud from scratch vie web-ui
> > This is another problem I see: With my non-cloud core I have a > > conf-directory where I have dataimport.xml, schema.xml and solrconfig.xml. > > I think these 3 files are enough to import my data from my relational > > database. > > Under example/cloud I could not find one of them. How to setup DIH for the > > solrcould? > > The entire configuration (what would normally be in the conf directory) > is in zookeeper when you're in cloud mode, not in the core directories. > You must upload a directory containing the same files that would > normally be in a conf directory as a named configset to zookeeper before > you try to create your collection. This is something that the "bin/solr > create" command does for you in cloud mode, typically using one of the > configsets included on the disk as a source. > > https://cwiki.apache.org/confluence/display/solr/Using+ZooKeeper+to+Manage+Configuration+Files > Ok, thank you. I did the following steps. 1. Started an external zookeeper 2. Copied a conf-directory to zookeeper: bin/solr zk upconfig -n books -d $HOME/solr-6.5.1/server/solr/tommy/conf -z localhost:2181 // This is a conf-directory from a standalone solr when dataimport was working! --> Connecting to ZooKeeper at localhost:2181 ... Uploading <> for config books to ZooKeeper at localhost:2181 // I think no errors, but how can I check it in zookeeper? I found no files solrconfig.xml ... in the zookeeper directories (installation dir and data dir) 3. Started solr: bin/solr start -c 4. Created a books collection with 2 shards bin/solr create -c books -shards 2 Result: I see in the web-ui my books collection with the 2 shards. No errors so far. However, the Dataimport-entry says: "Sorry, no dataimport-handler defined!" What could be the reason? Thomas
Re: setup solrcloud from scratch vie web-ui
On 5/12/2017 3:12 AM, Thomas Porschberg wrote: > I want to setup a solrcloud. I want to test sharding with one node, no > replication. > I have some experience with the non-cloud solr and I also run the cloud > examples. > I also have to use the DIH for importing. I think I can live with the > internal zookeeper. > > I did my first steps with solr-6.5.1. > > My first question is: Is it possible to setup a new solrcloud with the web-ui > only? >From what I can tell, something you cannot do with the admin UI is upload a configuration to zookeeper. If you don't already have one uploaded, then that must be accomplished separately, which you should be able to do at the commandline with the "bin/solr zk" commands. Because the DIH setup is in the configuration that you will be uploading, you also cannot set up DIH from within the UI. > When I start solr with: 'bin/solr start -c' > > I get a menu on the left side where I can create new collections and cores. > I think when I have only one node with no replication a collection maps to > one core, right? What you are describing is one replica, or replicationFactor=1. The leader is also a replica. A one-shard one-replica collection will consist of a single core, but you do not need to worry about that core. > Should I create first the core or the collection? You are only concerned with collections when running in cloud mode. Let Solr worry about the cores. > What should I fill in as instanceDir? Nothing. You won't be using the part of the UI that asks about instanceDir and dataDir. That is only for non-cloud mode, and the configuration must already exist in the core directory that you wish to create. > This is another problem I see: With my non-cloud core I have a conf-directory > where I have dataimport.xml, schema.xml and solrconfig.xml. > I think these 3 files are enough to import my data from my relational > database. > Under example/cloud I could not find one of them. How to setup DIH for the > solrcould? The entire configuration (what would normally be in the conf directory) is in zookeeper when you're in cloud mode, not in the core directories. You must upload a directory containing the same files that would normally be in a conf directory as a named configset to zookeeper before you try to create your collection. This is something that the "bin/solr create" command does for you in cloud mode, typically using one of the configsets included on the disk as a source. https://cwiki.apache.org/confluence/display/solr/Using+ZooKeeper+to+Manage+Configuration+Files Thanks, Shawn