OK thanks Erick, I have put a story in our jira backlog to investigate the G1GC issues with the Lucene test suite. I don't know if we'll be able to shed any light on the issue, but since we're using Lucene with Java 7 G1GC, I guess we better investigate it.
Jim ________________________________________ From: Erick Erickson [erickerick...@gmail.com] Sent: Saturday, February 07, 2015 2:22 PM To: java-user Subject: Re: Lucene Version Upgrade (3->4) and Java JVM Versions(6->8) The G1C1 issue reference by Robert Muir on the Wiki page is at a Lucene level. Lucene, of course, is critically important to Solr so from that perspective it is about Solr too. https://wiki.apache.org/lucene-java/JavaBugs And, I assume, it also applies to your custom app. FWIW, Erick On Fri, Feb 6, 2015 at 12:10 PM, McKinley, James T <james.mckin...@cengage.com> wrote: > Just to be clear in case there was any confusion about my previous message > regarding G1GC, we do not use Solr, my team works on a proprietary > Lucene-based search engine. Consequently, I can't really give any advice > regarding Solr with G1GC, but for our uses (so far anyway), G1GC seems to > work well with Lucene. > > Jim > ________________________________________ > From: Piotr Idzikowski [piotridzikow...@gmail.com] > Sent: Friday, February 06, 2015 5:35 AM > To: java-user@lucene.apache.org > Subject: Re: Lucene Version Upgrade (3->4) and Java JVM Versions(6->8) > > Hello. > A little bit delayed question. But recently I have found this articles: > https://wiki.apache.org/solr/SolrPerformanceProblems > https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning > > Especially this part from first url: > *Using the ConcurrentMarkSweep (CMS) collector with tuning parameters is a > very good option for for Solr, but with the latest Java 7 releases (7u72 at > the time of this writing), G1 is looking like a better option, if the > -XX:+ParallelRefProcEnabled option is used.* > > How does it play with *"Do not, under any circumstances, run Lucene with > the G1 garbage collector."* > from https://wiki.apache.org/lucene-java/JavaBugs? > > Regards > Piotr > > On Tue, Jan 27, 2015 at 9:55 PM, McKinley, James T < > james.mckin...@cengage.com> wrote: > >> Hi Uwe, >> >> OK, thanks for the info. We'll see if we can download the Lucene test >> suite and check it out. >> >> FWIW, we use G1GC in our production runtime (~70 12-16 core Cisco UCS and >> HP Gen7/Gen8 nodes with 20+ GB heaps using Java 7 and Lucene 4.8.1 with >> pairs of 30 index partitions with 15M-23M docs each) and have not >> experienced any VM crashes (well, maybe a couple, but not directly >> traceable to G1 to my knowledge). We have found some undocumented pauses >> in G1 due to very large object arrays and filed a bug report which was >> confirmed and also affects CMS (we worked around this in our code using >> memory mapping of some files whose contents we previously held all in >> RAM). I think the only index corruption we've ever seen was in our index >> creation workflow (~30 HP Gen7 nodes with 27GB heaps) but this was using >> Parallel GC since it is a batch system, so that corruption (which we've not >> seen recently and never found a cause for) was definitely not due to G1GC. >> >> G1GC has bugs as does CMS but we've found it to work pretty well so far in >> our runtime system. Of course YMMV, thanks again for the info. >> >> Jim >> ________________________________________ >> From: Uwe Schindler [u...@thetaphi.de] >> Sent: Tuesday, January 27, 2015 3:02 PM >> To: java-user@lucene.apache.org >> Subject: RE: Lucene Version Upgrade (3->4) and Java JVM Versions(6->8) >> >> Hi., >> >> About G1GC. We consistently see problems when running the Lucene Testsuite >> with G1GC enabled. The people from Elasticsearch concluded: >> >> "There is a newer GC called the Garbage First GC (G1GC). This newer GC is >> designed to minimize pausing even more than CMS, and operate on large >> heaps. It works by dividing the heap into regions and predicting which >> regions contain the most reclaimable space. By collecting those regions >> first (garbage first), it can minimize pauses and operate on very large >> heaps. >> >> Sounds great! Unfortunately, G1GC is still new, and fresh bugs are found >> routinely. These bugs are usually of the segfault variety, and will cause >> hard crashes. The Lucene test suite is brutal on GC algorithms, and it >> seems that G1GC hasn’t had the kinks worked out yet. >> >> We would like to recommend G1GC someday, but for now, it is simply not >> stable enough to meet the demands of Elasticsearch and Lucene." >> ( >> http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_don_8217_t_touch_these_settings.html >> ) >> >> In fact, the problems with G1GC can sometimes lead to index corruption, >> and are hard to reproduce. So better don't use... >> >> Uwe >> >> ----- >> Uwe Schindler >> H.-H.-Meier-Allee 63, D-28213 Bremen >> http://www.thetaphi.de >> eMail: u...@thetaphi.de >> >> >> > -----Original Message----- >> > From: McKinley, James T [mailto:james.mckin...@cengage.com] >> > Sent: Tuesday, January 27, 2015 8:58 PM >> > To: java-user@lucene.apache.org >> > Subject: RE: Lucene Version Upgrade (3->4) and Java JVM Versions(6->8) >> > >> > Why do you say not to use G1GC? We are using Java 7 & G1GC with Lucene >> > 4.8.1 in production. Thanks. >> > >> > Jim >> > ________________________________________ >> > From: Uwe Schindler [u...@thetaphi.de] >> > Sent: Tuesday, January 27, 2015 2:49 PM >> > To: java-user@lucene.apache.org; 'kiwi clive' >> > Subject: RE: Lucene Version Upgrade (3->4) and Java JVM Versions(6->8) >> > >> > Java 8 update 20 or later is also fine. At current time, always use >> latest update >> > release and you are be fine with Java 7 and Java 8. Don't use older >> releases >> > and don't use G1 Garbage Collector. >> > >> > ----- >> > Uwe Schindler >> > H.-H.-Meier-Allee 63, D-28213 Bremen >> > http://www.thetaphi.de >> > eMail: u...@thetaphi.de >> > >> > >> > > -----Original Message----- >> > > From: kiwi clive [mailto:kiwi_cl...@yahoo.com.INVALID] >> > > Sent: Tuesday, January 27, 2015 8:03 PM >> > > To: java-user@lucene.apache.org >> > > Subject: Re: Lucene Version Upgrade (3->4) and Java JVM Versions(6->8) >> > > >> > > Hi Hoss, >> > > Many thanks for the information. This looks very encouraging as the >> > > Java7 bug I remember was fixed and as far as I know, we should not be >> > > affected by the others. >> > > I'll put a few tests together and put my toe in the water :-) Clive >> > > >> > > From: Chris Hostetter <hossman_luc...@fucit.org> >> > > To: "java-user@lucene.apache.org" <java-user@lucene.apache.org>; kiwi >> > > clive <kiwi_cl...@yahoo.com> >> > > Sent: Tuesday, January 27, 2015 4:01 PM >> > > Subject: Re: Lucene Version Upgrade (3->4) and Java JVM >> > > Versions(6->8) >> > > >> > > >> > > >> > > >> > > : I seem to remember reading that certain versions of lucene were >> > > : incompatible with some java versions although I cannot find anything >> > > to >> > > : verify this. As we have tens of thousands of large indexes, >> > > backwards >> > > : compatibility without the need to reindex on an upgrade is of prime >> > > : importance to us. >> > > >> > > All known JVM bugs affecting Lucene are listed here... >> > > >> > > https://wiki.apache.org/lucene-java/JavaBugs >> > > >> > > >> > > -Hoss >> > > http://www.lucidworks.com/ >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> > > For additional commands, e-mail: java-user-h...@lucene.apache.org >> > > >> > > >> > > >> > > >> > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> > For additional commands, e-mail: java-user-h...@lucene.apache.org >> > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> > For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
