Custom shard key
HI, i am using explicit shading by creating custom shard key for my application using hbase util mumurhash (added snippet below). int hash = MurmurHash.getInstance().hash(sharekey.getBytes()); hash = Math.abs(hash); int routingValue = hash % shards; I noticed only 5 out of 8 shards used and 3 shards are empty. Please let me know if you see any issues or suggest any new shard key generation mechanism. Thanks. Regards, Anil
Re: Custom shard key, shard partitioning
On Feb 19, 2013, at 9:16 AM, Markus Jelsma wrote: > Ah, thanks. Got a Jira? I don't think i'm watching that one right now. https://issues.apache.org/jira/browse/SOLR-3154 - Mark
RE: Custom shard key, shard partitioning
-Original message- > From:Mark Miller > Sent: Mon 18-Feb-2013 16:27 > To: solr-user@lucene.apache.org > Subject: Re: Custom shard key, shard partitioning > > Yeah, I think we are missing some docs on this… > > I think the info is in here: https://issues.apache.org/jira/browse/SOLR-2592 > > But it's not so easy to pick out - I'd been considering going through and > writing up some wiki doc for that feature (unless I'm somehow missing it), > but just been too busy with other stuff.. It's not easy to pick it out indeed. Some docs, even very brief, would be very helpful :) > > Concerning CloudSolrServer, there is a JIRA to make it hash and send updates > to the "right" leader, but currently it still doesn't - it just favors > leaders in general over non leaders currently. Ah, thanks. Got a Jira? I don't think i'm watching that one right now. > > - Mark > > On Feb 18, 2013, at 7:34 AM, Markus Jelsma wrote: > > > Hi, > > > > By defaut SolrCloud partitions records by the hash of the uniqueKey field > > but we want to do some tests and partition the records by a signed integer > > field but keep the current uniqueKey unique. I've scanned through several > > issues concerning distributed index, custom hashing, shard policies etc but > > i have not found some concise examples or documentation or even blog post > > on this matter. > > > > How do we set up shard partitioning via another than the default uniqueKey > > field? > > > > According to some older resolved issue CloudSolrServer should be cloud > > aware and send updates to the leader of the correct shards, how does it > > know this? Must we set up the same partitioning in SolrServer client as > > well? If so, how? The apidocs do not reveal a lot when i look through them. > > > > I probably totally missed an issue or discussion or wiki page. > > > > Thanks, > > Markus > >
Re: Custom shard key, shard partitioning
Yeah, I think we are missing some docs on this… I think the info is in here: https://issues.apache.org/jira/browse/SOLR-2592 But it's not so easy to pick out - I'd been considering going through and writing up some wiki doc for that feature (unless I'm somehow missing it), but just been too busy with other stuff.. Concerning CloudSolrServer, there is a JIRA to make it hash and send updates to the "right" leader, but currently it still doesn't - it just favors leaders in general over non leaders currently. - Mark On Feb 18, 2013, at 7:34 AM, Markus Jelsma wrote: > Hi, > > By defaut SolrCloud partitions records by the hash of the uniqueKey field but > we want to do some tests and partition the records by a signed integer field > but keep the current uniqueKey unique. I've scanned through several issues > concerning distributed index, custom hashing, shard policies etc but i have > not found some concise examples or documentation or even blog post on this > matter. > > How do we set up shard partitioning via another than the default uniqueKey > field? > > According to some older resolved issue CloudSolrServer should be cloud aware > and send updates to the leader of the correct shards, how does it know this? > Must we set up the same partitioning in SolrServer client as well? If so, > how? The apidocs do not reveal a lot when i look through them. > > I probably totally missed an issue or discussion or wiki page. > > Thanks, > Markus
Re: Custom shard key, shard partitioning
Hi, I was able to implement custom hashing with the use of "_shard_" field. It contains the name of shard a document should go to. Works fine. Maybe there's some other method to do the same with the use of solrconfig.xml, but I have not found any docs about it so far. Regards. On 18 February 2013 13:34, Markus Jelsma wrote: > Hi, > > By defaut SolrCloud partitions records by the hash of the uniqueKey field > but we want to do some tests and partition the records by a signed integer > field but keep the current uniqueKey unique. I've scanned through several > issues concerning distributed index, custom hashing, shard policies etc but > i have not found some concise examples or documentation or even blog post > on this matter. > > How do we set up shard partitioning via another than the default uniqueKey > field? > > According to some older resolved issue CloudSolrServer should be cloud > aware and send updates to the leader of the correct shards, how does it > know this? Must we set up the same partitioning in SolrServer client as > well? If so, how? The apidocs do not reveal a lot when i look through them. > > I probably totally missed an issue or discussion or wiki page. > > Thanks, > Markus >
Custom shard key, shard partitioning
Hi, By defaut SolrCloud partitions records by the hash of the uniqueKey field but we want to do some tests and partition the records by a signed integer field but keep the current uniqueKey unique. I've scanned through several issues concerning distributed index, custom hashing, shard policies etc but i have not found some concise examples or documentation or even blog post on this matter. How do we set up shard partitioning via another than the default uniqueKey field? According to some older resolved issue CloudSolrServer should be cloud aware and send updates to the leader of the correct shards, how does it know this? Must we set up the same partitioning in SolrServer client as well? If so, how? The apidocs do not reveal a lot when i look through them. I probably totally missed an issue or discussion or wiki page. Thanks, Markus
