Re: Problems with maxShardsPerNode in 4.5
I have committed a fix for 4.6 http://issues.apache.org/jira/browse/SOLR-5295 On Wed, Oct 2, 2013 at 10:34 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: Thanks for reporting this Brett. This is indeed a bug. A workaround is to specify replicationFactor=1 with the createShard command which will create only one replica even if maxShardsPerNode=1000 at collection level. I'll open an issue. On Wed, Oct 2, 2013 at 7:25 AM, Brett Hoerner br...@bretthoerner.comwrote: Related, 1 more try: Created collection starting with 4 shards on 1 box. Had to set maxShardsPerNode to 4 to do this. Now I want to roll over my time window, so to attempt to deal with the problems noted above I delete the oldest shard first. That works fine. Now I try to add my new shard, which works, but again it defaults to maxShardsPerNode # of replicas, so I'm left with: * [deleted by me] hour0 * hour1 - 1 replica * hour2 - 1 replica * hour3 - 1 replica * hour4 - 4 replicas [ the one I created after deleting hour0] Still at a loss as to how I would create 1 new shard with 1 replica on any server in 4.5? Thanks! On Tue, Oct 1, 2013 at 8:14 PM, Brett Hoerner br...@bretthoerner.com wrote: It seems that changes in 4.5 collection configuration now require users to set a maxShardsPerNode (or it defaults to 1). Maybe this was the case before, but with the new CREATESHARD API it seems a very restrictive. I've just created a very simple test collection on 3 machines where I set maxShardsPerNode at collection creation time to 1, and I made 3 shards. Everything is good. Now I want a 4th shard, it seems impossible to create because the cluster knows I should only have 1 shard per node. Yet my problem doesn't require more hardware, I just my new shard to exist on one of the existing servers. So I try again -- I create a collection with 3 shards and set maxShardsPerNode to 1000 (just as a silly test). Everything is good. Now I add shard4 and it immediately tries to add 1000 replicas of shard4... You can see my earlier email today about time-oriented data in 4.5 to see what I'm trying to do. I was hoping to have 1 shard per hour/day with the ability to easily add/drop them as I move the time window (say, a week of data, 1 per day). Am I missing something? Thanks! -- Regards, Shalin Shekhar Mangar. -- Regards, Shalin Shekhar Mangar.
Re: Problems with maxShardsPerNode in 4.5
Shalin, Thanks for the fix. There's still part of the underlying issue that I consider a bug or a documentation problem: how do I adjust maxShardsPerNode after my collection has been created, and/or how can I disable it being checked/used at all? It seems odd to me that I have to set it to an odd number like 1000 just to get around this? Thanks! On Wed, Oct 2, 2013 at 12:04 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: Thanks for reporting this Brett. This is indeed a bug. A workaround is to specify replicationFactor=1 with the createShard command which will create only one replica even if maxShardsPerNode=1000 at collection level. I'll open an issue. On Wed, Oct 2, 2013 at 7:25 AM, Brett Hoerner br...@bretthoerner.com wrote: Related, 1 more try: Created collection starting with 4 shards on 1 box. Had to set maxShardsPerNode to 4 to do this. Now I want to roll over my time window, so to attempt to deal with the problems noted above I delete the oldest shard first. That works fine. Now I try to add my new shard, which works, but again it defaults to maxShardsPerNode # of replicas, so I'm left with: * [deleted by me] hour0 * hour1 - 1 replica * hour2 - 1 replica * hour3 - 1 replica * hour4 - 4 replicas [ the one I created after deleting hour0] Still at a loss as to how I would create 1 new shard with 1 replica on any server in 4.5? Thanks! On Tue, Oct 1, 2013 at 8:14 PM, Brett Hoerner br...@bretthoerner.com wrote: It seems that changes in 4.5 collection configuration now require users to set a maxShardsPerNode (or it defaults to 1). Maybe this was the case before, but with the new CREATESHARD API it seems a very restrictive. I've just created a very simple test collection on 3 machines where I set maxShardsPerNode at collection creation time to 1, and I made 3 shards. Everything is good. Now I want a 4th shard, it seems impossible to create because the cluster knows I should only have 1 shard per node. Yet my problem doesn't require more hardware, I just my new shard to exist on one of the existing servers. So I try again -- I create a collection with 3 shards and set maxShardsPerNode to 1000 (just as a silly test). Everything is good. Now I add shard4 and it immediately tries to add 1000 replicas of shard4... You can see my earlier email today about time-oriented data in 4.5 to see what I'm trying to do. I was hoping to have 1 shard per hour/day with the ability to easily add/drop them as I move the time window (say, a week of data, 1 per day). Am I missing something? Thanks! -- Regards, Shalin Shekhar Mangar.
Re: Problems with maxShardsPerNode in 4.5
You are right Brett. There's plenty more to be done. I can't promise when all these improvements will be done but at least some should make it to 4.6 https://issues.apache.org/jira/browse/SOLR-5128 On Wed, Oct 2, 2013 at 9:35 PM, Brett Hoerner br...@bretthoerner.comwrote: Shalin, Thanks for the fix. There's still part of the underlying issue that I consider a bug or a documentation problem: how do I adjust maxShardsPerNode after my collection has been created, and/or how can I disable it being checked/used at all? It seems odd to me that I have to set it to an odd number like 1000 just to get around this? Thanks! On Wed, Oct 2, 2013 at 12:04 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: Thanks for reporting this Brett. This is indeed a bug. A workaround is to specify replicationFactor=1 with the createShard command which will create only one replica even if maxShardsPerNode=1000 at collection level. I'll open an issue. On Wed, Oct 2, 2013 at 7:25 AM, Brett Hoerner br...@bretthoerner.com wrote: Related, 1 more try: Created collection starting with 4 shards on 1 box. Had to set maxShardsPerNode to 4 to do this. Now I want to roll over my time window, so to attempt to deal with the problems noted above I delete the oldest shard first. That works fine. Now I try to add my new shard, which works, but again it defaults to maxShardsPerNode # of replicas, so I'm left with: * [deleted by me] hour0 * hour1 - 1 replica * hour2 - 1 replica * hour3 - 1 replica * hour4 - 4 replicas [ the one I created after deleting hour0] Still at a loss as to how I would create 1 new shard with 1 replica on any server in 4.5? Thanks! On Tue, Oct 1, 2013 at 8:14 PM, Brett Hoerner br...@bretthoerner.com wrote: It seems that changes in 4.5 collection configuration now require users to set a maxShardsPerNode (or it defaults to 1). Maybe this was the case before, but with the new CREATESHARD API it seems a very restrictive. I've just created a very simple test collection on 3 machines where I set maxShardsPerNode at collection creation time to 1, and I made 3 shards. Everything is good. Now I want a 4th shard, it seems impossible to create because the cluster knows I should only have 1 shard per node. Yet my problem doesn't require more hardware, I just my new shard to exist on one of the existing servers. So I try again -- I create a collection with 3 shards and set maxShardsPerNode to 1000 (just as a silly test). Everything is good. Now I add shard4 and it immediately tries to add 1000 replicas of shard4... You can see my earlier email today about time-oriented data in 4.5 to see what I'm trying to do. I was hoping to have 1 shard per hour/day with the ability to easily add/drop them as I move the time window (say, a week of data, 1 per day). Am I missing something? Thanks! -- Regards, Shalin Shekhar Mangar. -- Regards, Shalin Shekhar Mangar.
Re: Problems with maxShardsPerNode in 4.5
Related, 1 more try: Created collection starting with 4 shards on 1 box. Had to set maxShardsPerNode to 4 to do this. Now I want to roll over my time window, so to attempt to deal with the problems noted above I delete the oldest shard first. That works fine. Now I try to add my new shard, which works, but again it defaults to maxShardsPerNode # of replicas, so I'm left with: * [deleted by me] hour0 * hour1 - 1 replica * hour2 - 1 replica * hour3 - 1 replica * hour4 - 4 replicas [ the one I created after deleting hour0] Still at a loss as to how I would create 1 new shard with 1 replica on any server in 4.5? Thanks! On Tue, Oct 1, 2013 at 8:14 PM, Brett Hoerner br...@bretthoerner.comwrote: It seems that changes in 4.5 collection configuration now require users to set a maxShardsPerNode (or it defaults to 1). Maybe this was the case before, but with the new CREATESHARD API it seems a very restrictive. I've just created a very simple test collection on 3 machines where I set maxShardsPerNode at collection creation time to 1, and I made 3 shards. Everything is good. Now I want a 4th shard, it seems impossible to create because the cluster knows I should only have 1 shard per node. Yet my problem doesn't require more hardware, I just my new shard to exist on one of the existing servers. So I try again -- I create a collection with 3 shards and set maxShardsPerNode to 1000 (just as a silly test). Everything is good. Now I add shard4 and it immediately tries to add 1000 replicas of shard4... You can see my earlier email today about time-oriented data in 4.5 to see what I'm trying to do. I was hoping to have 1 shard per hour/day with the ability to easily add/drop them as I move the time window (say, a week of data, 1 per day). Am I missing something? Thanks!
Re: Problems with maxShardsPerNode in 4.5
Thanks for reporting this Brett. This is indeed a bug. A workaround is to specify replicationFactor=1 with the createShard command which will create only one replica even if maxShardsPerNode=1000 at collection level. I'll open an issue. On Wed, Oct 2, 2013 at 7:25 AM, Brett Hoerner br...@bretthoerner.comwrote: Related, 1 more try: Created collection starting with 4 shards on 1 box. Had to set maxShardsPerNode to 4 to do this. Now I want to roll over my time window, so to attempt to deal with the problems noted above I delete the oldest shard first. That works fine. Now I try to add my new shard, which works, but again it defaults to maxShardsPerNode # of replicas, so I'm left with: * [deleted by me] hour0 * hour1 - 1 replica * hour2 - 1 replica * hour3 - 1 replica * hour4 - 4 replicas [ the one I created after deleting hour0] Still at a loss as to how I would create 1 new shard with 1 replica on any server in 4.5? Thanks! On Tue, Oct 1, 2013 at 8:14 PM, Brett Hoerner br...@bretthoerner.com wrote: It seems that changes in 4.5 collection configuration now require users to set a maxShardsPerNode (or it defaults to 1). Maybe this was the case before, but with the new CREATESHARD API it seems a very restrictive. I've just created a very simple test collection on 3 machines where I set maxShardsPerNode at collection creation time to 1, and I made 3 shards. Everything is good. Now I want a 4th shard, it seems impossible to create because the cluster knows I should only have 1 shard per node. Yet my problem doesn't require more hardware, I just my new shard to exist on one of the existing servers. So I try again -- I create a collection with 3 shards and set maxShardsPerNode to 1000 (just as a silly test). Everything is good. Now I add shard4 and it immediately tries to add 1000 replicas of shard4... You can see my earlier email today about time-oriented data in 4.5 to see what I'm trying to do. I was hoping to have 1 shard per hour/day with the ability to easily add/drop them as I move the time window (say, a week of data, 1 per day). Am I missing something? Thanks! -- Regards, Shalin Shekhar Mangar.
