On Sat, Apr 13, 2013 at 3:02 AM, Sanne Grinovero <sa...@hibernate.org>wrote:
> that's right, as suggested by Emmanuel I plan to separate the JGroups > Sync/Async options from the worker.execution property so you can play > with the two independently. > I think the JGroups option's default could depend on the backend - if > not otherwise specified, and if we all agree it doesn't make it too > confusing. > > @All, the performance problem seemed to be caused by a problem in > JGroups, which I've logged here: > https://issues.jboss.org/browse/JGRP-1617 > > For the record, the first operation was indeed triggering some lazy > initialization of indexes, which in turn would trigger a Lucene > Directory being started, triggering 3 Cache starts which in turn would > trigger 6 state transfer processes: so indeed the first operation > would not be exactly "cheap" performance wise, still this would > complete in about 120 milliseconds. > The same cost is paid again when the second node is hit the first > time, after that index write operations block the writer for <1ms (not > investigated further on potential throughput). > > Not being sure about the options of depending to a newer JGroups > release or the complexity of a fix, I'll implement a workaround in > HSearch in the scope of HSEARCH-1296. > > As a lesson learned, I think we need to polish some of our TRACE level > messaged to include the cache name: Sanne, we already push the cache name in the NDC if trace is enabled for the "entry point" of the thread. So for your purpose, I think enabling trace for org.infinispan.interceptors.InvocationContextInterceptor and including %x in your pattern layout should work. > to resolve this we had not just > many threads and components but also 4 of them where using JGroups > (interleaving messages of all sorts) and 9 different caches where > involved for each simple write operation in CD: made it interesting to > figure what was going on! Also I'm wondering how hard it would be to > have a log parser which converts my 10GB of text log from today in a > graphical sequence diagram. > Big thanks to Mircea who helped me figuring this out. > > Sanne > > On 12 April 2013 21:10, Ales Justin <ales.jus...@gmail.com> wrote: > > I think we need more fine-grained config for this new JGroups sync > feature. > > > > I added this to our cache config > > > > <property > name="hibernate.search.default.worker.execution">async</property> > > > > and it broke our tests. > > > > Where previous (old / non JGroups sync) behavior worked. > > > > It of course also works without this async config, > > but in this case we don't need sync / ACK JGroups message. > > (we didn't have one before and it worked ;-) > > > > -Ales > > > > On Apr 11, 2013, at 11:51 PM, Sanne Grinovero <sa...@hibernate.org> > wrote: > > > >> There is a "blackhole" indexing backend, which pipes all indexing > >> requests > /dev/null > >> > >> Set this as an Infinispan Query configuration property: > >> > >> default.worker.backend = blackhole > >> > >> Of course that means that the index will not be updated: you might > >> need to adapt your test to tolerate that, but the point is not > >> functional testing but to verify how much the SYNC option on the > >> JGroups backend is actually slowing you down. I suspect the > >> performance penalty is not in the network but in the fact you're now > >> waiting for the index operations, while in async you where not waiting > >> for them to be flushed. > >> > >> If you can identify which part is slow, then we can help you with > >> better configuration options. > >> > >> > >> On 11 April 2013 20:47, Ales Justin <ales.jus...@gmail.com> wrote: > >>> What do you mean? > >>> > >>> On Apr 11, 2013, at 21:41, Sanne Grinovero <sa...@hibernate.org> > wrote: > >>> > >>> You could try the new sync version but setting the blackhole backend > on the > >>> master node to remove the indexing overhead from the picture. > >>> > >>> On Apr 11, 2013 8:39 PM, "Sanne Grinovero" <sa...@hibernate.org> > wrote: > >>>> > >>>> Are you sure that the async version actually had applied all writes > to the > >>>> index in the measured interval? > >>>> > >>>> On Apr 11, 2013 8:13 PM, "Ales Justin" <ales.jus...@gmail.com> wrote: > >>>>> > >>>>> Although this change fixes query lookup, > >>>>> it adds horrible performance: > >>>>> > >>>>> Running CapeDwarf cluster QueryTest: > >>>>> > >>>>> with HSEARCH-1296 > >>>>> > >>>>> 21:00:27,188 INFO > >>>>> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager] > >>>>> (http-/192.168.1.102:8080-1) HSEARCH000168: Serialization service > Avro > >>>>> SerializationProvider v1.0 being used for index > >>>>> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity' > >>>>> 21:01:17,911 INFO [org.jboss.web] (ServerService Thread Pool -- 49) > >>>>> JBAS018224: Unregister web context: /capedwarf-tests > >>>>> > >>>>> 50sec > >>>>> > >>>>> old 4.2.0.Final HS > >>>>> > >>>>> 21:08:19,988 INFO > >>>>> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager] > >>>>> (http-/192.168.1.102:8080-2) HSEARCH000168: Serialization service > Avro > >>>>> SerializationProvider v1.0 being used for index > >>>>> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity' > >>>>> 21:08:20,829 INFO [org.jboss.web] (ServerService Thread Pool -- 49) > >>>>> JBAS018224: Unregister web context: /capedwarf-tests > >>>>> > >>>>> 841ms > >>>>> > >>>>> --- > >>>>> > >>>>> I added > >>>>> > >>>>> <property name="enable_bundling">true</property> > >>>>> > >>>>> to AS jgroups transport config, but no improvement. > >>>>> > >>>>> Any (other) idea? > >>>>> > >>>>> -Ales > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> hibernate-dev mailing list > >>>>> hibernate-...@lists.jboss.org > >>>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev > > > > > > _______________________________________________ > > hibernate-dev mailing list > > hibernate-...@lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hibernate-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev >
_______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev