[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-04-20 Thread Per Steffensen (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249790#comment-15249790 ] Per Steffensen commented on SOLR-8922: -- bq. I don't know how well Per Steffensen teste

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-04-10 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234092#comment-15234092 ] Yonik Seeley commented on SOLR-8922: bq. I thought this was just about saving memory. W

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-04-09 Thread David Smiley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233876#comment-15233876 ] David Smiley commented on SOLR-8922: Wow; I'm (pleasantly) surprised to see such a gene

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-04-09 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233771#comment-15233771 ] ASF subversion and git services commented on SOLR-8922: --- Commit 301e1

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-04-09 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233768#comment-15233768 ] ASF subversion and git services commented on SOLR-8922: --- Commit cfba5

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-04-09 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233682#comment-15233682 ] Yonik Seeley commented on SOLR-8922: OK, it took me *much* longer to do the benchmarks

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-04-06 Thread Yonik Seeley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229286#comment-15229286 ] Yonik Seeley commented on SOLR-8922: I quickly tried some patches in the past that resu

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-04-06 Thread Jeff Wartes (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229260#comment-15229260 ] Jeff Wartes commented on SOLR-8922: --- I stumbled onto SJK recently, which provides me a mo

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-31 Thread Joel Bernstein (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220687#comment-15220687 ] Joel Bernstein commented on SOLR-8922: -- Yep, that is cool! > DocSetCollector can allo

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-31 Thread David Smiley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220633#comment-15220633 ] David Smiley commented on SOLR-8922: I spoke too soon; I like your utility :-) > DocSe

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-31 Thread Jeff Wartes (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220342#comment-15220342 ] Jeff Wartes commented on SOLR-8922: --- Both of those appear to add capacity by declaring a

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-31 Thread Jeff Wartes (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220299#comment-15220299 ] Jeff Wartes commented on SOLR-8922: --- With some tweaking, I was able to get G1 pause to ab

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-31 Thread Joel Bernstein (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220270#comment-15220270 ] Joel Bernstein commented on SOLR-8922: -- Also check out ArrayUtil in Lucene > DocSetCo

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-31 Thread David Smiley (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220259#comment-15220259 ] David Smiley commented on SOLR-8922: Solr uses HPPC for it's native collections. Just

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-31 Thread Keith Laban (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220252#comment-15220252 ] Keith Laban commented on SOLR-8922: --- Just out of curiosity, have you run tests with G1? W

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-31 Thread Jeff Wartes (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220136#comment-15220136 ] Jeff Wartes commented on SOLR-8922: --- Ok, so after a little more than 12 hours on one of m

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-31 Thread Jeff Wartes (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220052#comment-15220052 ] Jeff Wartes commented on SOLR-8922: --- Incidentally, I had one or two other findings from m

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-31 Thread Jeff Wartes (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220045#comment-15220045 ] Jeff Wartes commented on SOLR-8922: --- Absolutely. Memory pools were my first thought, betw

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-30 Thread Erick Erickson (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219235#comment-15219235 ] Erick Erickson commented on SOLR-8922: -- I'm also wondering if there are some tricks wh

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-30 Thread Jeff Wartes (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219066#comment-15219066 ] Jeff Wartes commented on SOLR-8922: --- Not yet. The major risk area would be the new Expand

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-30 Thread Erick Erickson (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219003#comment-15219003 ] Erick Erickson commented on SOLR-8922: -- Jeff: Thanks for pointing that out, we need

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-30 Thread Jeff Wartes (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218974#comment-15218974 ] Jeff Wartes commented on SOLR-8922: --- For my index, (86M-doc shards and a per-shard 99.9th

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-03-30 Thread Jeff Wartes (JIRA)
[ https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218910#comment-15218910 ] Jeff Wartes commented on SOLR-8922: --- SOLR-5444 had a patch to help with this, (SOLR-5444