It is worth noting that the ref guide page on configsets refers to non-cloud mode (a useful new feature) whereas people may confuse this with configsets in cloud mode, which use Zookeeper.
Upayavira On Sun, Sep 20, 2015, at 04:59 AM, Ravi Solr wrote: > Cant thank you enough for clarifying it at length. Yeah its pretty > confusing even for experienced Solr users. I used the upconfig and > linkconfig commands to update 4 collections into zookeeper...As you > described, I lucked out as I used the same name for configset and the > collection and hence did not have to use the collections API :-) > > Thanks, > > Ravi Kiran Bhaskar > > On Sat, Sep 19, 2015 at 11:22 PM, Erick Erickson > <erickerick...@gmail.com> > wrote: > > > Let's back up a second. Configsets are what _used_ to be in the conf > > directory for each core on a local drive, it's just that they're now > > kept up on Zookeeper. Otherwise, you'd have to put them on each > > instance in SolrCloud, and bringing up a new replica on a new machine > > would look a lot like adding a core with the old core admin API. > > > > So instead, configurations are kept on zookeeper. A config set > > consists of, essentially, a named old-style "conf" directory. There's > > no a-priori limit to the number of config sets you can have. Look in > > the admin UI, Cloud>>tree>>configs and you'll see each name you've > > pushed to ZK. If you explore that tree, you'll see a lot of old > > familiar faces, schema.xml, solrconfig.xml, etc. > > > > So now we come to associating configs with collections. You've > > probably done one of the examples where some things happen under the > > covers, including explicitly pushing the configset to Zookeeper. > > Currently, there's no option in the bin/solr script to push a config, > > although I know there's a JIRA to do that. > > > > So, to put a new config set up you currently need to use the zkCli.sh > > script see: > > https://cwiki.apache.org/confluence/display/solr/Command+Line+Utilities, > > the "upconfig" command. That pushes the configset up to ZK and gives > > it a name. > > > > Now, you create a collection and it needs a configset stored in ZK. > > It's a little tricky in that if you do _not_ explicitly specify a > > configest (using the collection.configName parameter to the > > collections API CREATE command), then by default it'll look for a > > configset with the same name as the collection. If it doesn't find > > one, _and_ there is one and only one configset, then it'll use that > > one (personally I find that confusing, but that's the way it works). > > See: https://cwiki.apache.org/confluence/display/solr/Collections+API > > > > If you have two or more configsets in ZK, then either the configset > > name has to be identical to the collection name (if you don't specify > > collection.configName), _or_ you specify collection.configName at > > create time. > > > > NOTE: there are _no_ config files on the local disk! When a replica of > > a collection loads, it "knows" what collection it's part of and pulls > > the corresponding configset from ZK. > > > > So typically the process is this. > > > you create the config set by editing all the usual suspects, schema.xml, > > solrconfig.xml, DIH config etc. > > > you put those configuration files into some version control system (you > > are using one, right?) > > > you push the configs to Zookeeper > > > you create the collection > > > you figure out you need to change the configs so you > > > check the code out of your version control > > > edit them > > > put the current version back into version control > > > push the configs up to zookeeper, overwriting the ones already > > there with that name > > > reload the collection or bounce all the servers. As each replica > > in the collection comes up, > > it downloads the latest configs from Zookeeper to memory (not to > > disk) and uses them. > > > > Seems like a long drawn-out process, but pretty soon it's automatic. > > And really, the only extra step is the push to Zookeeper, the rest is > > just like old-style cores with the exception that you don't have to > > manually push all the configs to all the machines hosting cores. > > > > Notice that I have mostly avoided talking about "cores" here. Although > > it's true that a replica in a collection is just another core, it's > > "special" in that it has certain very specific properties set. I > > _strongly_ advise you stop thinking about old-style Solr cores and > > instead thing about collections and replicas. And above all, do _not_ > > use the admin core API to try to create members of a collection > > (cores), use the collections API to ADDREPLICA/DELETEREPLICA instead. > > Loading/unloading cores is less "fraught", but I try to avoid that too > > and use > > > > Best, > > Erick > > > > On Sat, Sep 19, 2015 at 9:08 PM, Ravi Solr <ravis...@gmail.com> wrote: > > > Thanks Erick, I will report back once the reindex is finished. Oh, your > > > answer reminded me of another question - Regarding configsets the > > > documentation says > > > > > > "On a multicore Solr instance, you may find that you want to share > > > configuration between a number of different cores." > > > > > > Can the same be used to push disparate mutually exclusive configs ? I ask > > > this as I have 4 mutually exclusive apps each with a 4 single core index > > on > > > a single machine which I am trying to convert to SolrCloud with single > > > shard approach. Just being lazy and trying to find a way to update and > > link > > > configs to zookeeper ;-) > > > > > > Thanks > > > > > > Rvai Kiran Bhaskar > > > > > > On Sat, Sep 19, 2015 at 6:54 PM, Erick Erickson <erickerick...@gmail.com > > > > > > wrote: > > > > > >> Just pushing up the entire configset would be easiest, but the > > >> Zookeeper command line tools allow you to push up a single > > >> file if you want. > > >> > > >> Yeah, it puzzles me too that the import worked yesterday, not really > > >> sure what happened, the file shouldn't just disappear.... > > >> > > >> Erick > > >> > > >> On Sat, Sep 19, 2015 at 2:46 PM, Ravi Solr <ravis...@gmail.com> wrote: > > >> > Thank you for the prompt response Erick. I did a full-import > > yesterday, > > >> you > > >> > are correct that I did not push dataimport.properties to ZK, should it > > >> have > > >> > not worked even for a full import ?. You may be right about 'clean' > > >> option, > > >> > I will reindex again today. BTW how do we push a single file to a > > >> specific > > >> > config name in zookeeper ? > > >> > > > >> > > > >> > Thanks, > > >> > > > >> > Ravi Kiran Bhaskar > > >> > > > >> > > > >> > On Sat, Sep 19, 2015 at 1:48 PM, Erick Erickson < > > erickerick...@gmail.com > > >> > > > >> > wrote: > > >> > > > >> >> Could not read DIH properties from > > >> >> /configs/sitesearchcore/dataimport.properties > > >> >> > > >> >> This looks like somehow you didn't push this file up to Zookeeper. > > You > > >> >> can check what files are there in the admin UI. How you indexed > > >> >> yesterday is a mystery though, unless somehow this file was removed > > >> >> from ZK. > > >> >> > > >> >> As for why you lost all the docs, my suspicion is that you have the > > >> >> clean param set up for delta import.... > > >> >> > > >> >> FWIW, > > >> >> Erick > > >> >> > > >> >> On Sat, Sep 19, 2015 at 10:36 AM, Ravi Solr <ravis...@gmail.com> > > wrote: > > >> >> > I am facing a weird problem. As part of upgrade from 4.7.2 > > >> (Master-Slave) > > >> >> > to 5.3.0 (Solrcloud) I re-indexed 1.5 million records via DIH using > > >> >> > SolrEntityProcessor yesterday, all of them indexed properly. Today > > >> >> morning > > >> >> > I just ran the DIH again with delta import and I lost all > > docs...what > > >> am > > >> >> I > > >> >> > missing ? Did anybody face similar issue ? > > >> >> > > > >> >> > Here are the errors in the logs > > >> >> > > > >> >> > 9/19/2015, 2:41:17 AM ERROR null SolrCore Previous SolrRequestInfo > > was > > >> >> not > > >> >> > closed! > > >> >> > req=waitSearcher=true&distrib.from= > > >> >> > > >> > > http://10.128.159.32:8983/solr/sitesearchcore/&update.distrib=FROMLEADER&openSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false > > >> >> > 9/19/2015, > > >> >> > 2:41:17 AM ERROR null SolrCore prev == info : false 9/19/2015, > > >> 2:41:17 AM > > >> >> > WARN null ZKPropertiesWriter Could not read DIH properties from > > >> >> > /configs/sitesearchcore/dataimport.properties :class > > >> >> > org.apache.zookeeper.KeeperException$NoNodeException > > >> >> > > > >> >> > org.apache.zookeeper.KeeperException$NoNodeException: > > KeeperErrorCode > > >> >> > = NoNode for /configs/sitesearchcore/dataimport.properties > > >> >> > at > > >> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > > >> >> > at > > >> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > > >> >> > at > > org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) > > >> >> > at > > >> >> > > org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:349) > > >> >> > at > > >> >> > > >> > > org.apache.solr.handler.dataimport.ZKPropertiesWriter.readIndexerProperties(ZKPropertiesWriter.java:91) > > >> >> > at > > >> >> > > >> > > org.apache.solr.handler.dataimport.ZKPropertiesWriter.persist(ZKPropertiesWriter.java:65) > > >> >> > at > > >> >> > > >> > > org.apache.solr.handler.dataimport.DocBuilder.finish(DocBuilder.java:307) > > >> >> > at > > >> >> > > >> > > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:253) > > >> >> > at > > >> >> > > >> > > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416) > > >> >> > at > > >> >> > > >> > > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480) > > >> >> > at > > >> >> > > >> > > org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461) > > >> >> > > > >> >> > 9/19/2015, 11:16:43 AM ERROR null SolrCore Previous SolrRequestInfo > > >> was > > >> >> not > > >> >> > closed! > > >> >> > req=waitSearcher=true&distrib.from= > > >> >> > > >> > > http://10.128.159.32:8983/solr/sitesearchcore/&update.distrib=FROMLEADER&openSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false > > >> >> > 9/19/2015, > > >> >> > 11:16:43 AM ERROR null SolrCore prev == info : false > > >> >> > > > >> >> > > > >> >> > > > >> >> > Thanks > > >> >> > > > >> >> > Ravi Kiran Bhaskar > > >> >> > > >> > >
