[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-10-10 Thread Cao Manh Dat (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199766#comment-16199766 ] Cao Manh Dat commented on SOLR-10265: - I linked two child issues for this issue which is SOLR-11443

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-10-10 Thread Shawn Heisey (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199169#comment-16199169 ] Shawn Heisey commented on SOLR-10265: - No idea how this got assigned to me. I probably was trying to

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-10-06 Thread Shawn Heisey (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194970#comment-16194970 ] Shawn Heisey commented on SOLR-10265: - Guava has something really cool that I think we could use for a

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-10-06 Thread Cao Manh Dat (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194319#comment-16194319 ] Cao Manh Dat commented on SOLR-10265: - Maybe the problem here is Overseer is processing all messages

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-07-30 Thread Noble Paul (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106729#comment-16106729 ] Noble Paul commented on SOLR-10265: --- bq.Would a watcher have to be set for every shard of every

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-07-29 Thread Erick Erickson (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106167#comment-16106167 ] Erick Erickson commented on SOLR-10265: --- Hmm, it seems to me that another problem with splitting any

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-07-29 Thread Noble Paul (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106038#comment-16106038 ] Noble Paul commented on SOLR-10265: --- bq. If for some reason a per-replica znode is unworkable, would a

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-07-28 Thread David Smiley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106015#comment-16106015 ] David Smiley commented on SOLR-10265: - I haven't participated in these overseer related conversations

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-04-10 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963527#comment-15963527 ] Mark Miller commented on SOLR-10265: There are perhaps other wins we could get in the short term as

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-03-19 Thread Noble Paul (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932032#comment-15932032 ] Noble Paul commented on SOLR-10265: --- Yeah, the overseer messages and collection action messages are

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-03-19 Thread Erick Erickson (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931754#comment-15931754 ] Erick Erickson commented on SOLR-10265: --- Hmmm, could we take a step in that direction now? Off the

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-03-18 Thread Mark Miller (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931560#comment-15931560 ] Mark Miller commented on SOLR-10265: The main reason a zk queue was used instead of in memory was that

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-03-18 Thread Erick Erickson (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931544#comment-15931544 ] Erick Erickson commented on SOLR-10265: --- bq: My suspicion is that it's problematic because of the

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-03-18 Thread Shawn Heisey (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931532#comment-15931532 ] Shawn Heisey commented on SOLR-10265: - bq. We should seriously reconsider the idea of using ZK queue

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-03-18 Thread Noble Paul (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931501#comment-15931501 ] Noble Paul commented on SOLR-10265: --- Order of events between two clients don't matter > Overseer can

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-03-18 Thread Erick Erickson (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931500#comment-15931500 ] Erick Erickson commented on SOLR-10265: --- bq: The client should take care of the ordering. The case

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-03-18 Thread Noble Paul (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931437#comment-15931437 ] Noble Paul commented on SOLR-10265: --- bq.Depending on the timing of the retry logic, the clients could

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-03-18 Thread Erick Erickson (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931336#comment-15931336 ] Erick Erickson commented on SOLR-10265: --- [~noble.paul] Interesting. You're essentially replacing the

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-03-17 Thread Noble Paul (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931070#comment-15931070 ] Noble Paul commented on SOLR-10265: --- We should seriously reconsider the idea of using ZK queue for the

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-03-11 Thread Noble Paul (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906381#comment-15906381 ] Noble Paul commented on SOLR-10265: --- bq.I haven't debugged why the transaction logs ran into terabytes

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-03-11 Thread Shawn Heisey (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906245#comment-15906245 ] Shawn Heisey commented on SOLR-10265: - Side comment, need its own issue if we decide to pursue. Only

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

2017-03-11 Thread Erick Erickson (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906241#comment-15906241 ] Erick Erickson commented on SOLR-10265: --- Trying to think a little OOB here: > Is there any way to