Yes but Lukasz do see data disappearing (see initial email). So there is a problem somewhere? Lukasz, can you maybe share the code and test with us so someone can have a closer look with you?
Emmanuel On 13 août 09, at 18:56, Sanne Grinovero wrote: > I am not an expert on this part of Lucene, but it looks like to me > that the IndexWriter is the "driver/coordinator", and it's decisions > are affected by a pluggable MergeScheduler; they do stuff on the > internal buffers of the IndexWriter (dequeue the pending segments to > be written to the index), but it shouldn't matter what they exactly do > as the internal status of these classes are unaffected by our > transactions. > They take some decision about writing segments to the Directory and > committing changes ("sync()") : as you implement this Directory you > should only have to take care of this class, I don't think the > MergeScheduler(s) are relevant: it just happens that the thread going > to apply changes to the index might be a different one than the one > pushing changes to the IndexWriter. > > In the Directory implementation you should use transactions to push > state changes to the "underlying storage": as FSDirectory is playing > with file descriptors and flushes, you do the same with Infinispan > transactions. > > 2009/8/14 Łukasz Moreń <lukasz.mo...@gmail.com>: >> Yes, right, MergeSchedulers. >> >> 2009/8/14 Sanne Grinovero <sanne.grinov...@gmail.com> >>> >>> what are these "other" threads? Are you speaking about the >>> MergeSchedulers? >>> >>> 2009/8/13 Łukasz Moreń <lukasz.mo...@gmail.com>: >>>> IndexWriter processes index update and delegates some job to other >>>> threads and waits when they finish. These "other" threads works on >>>> data modified >>>> in IndexWriter transaction. So I think if I use transaction per >>>> thread, "others" would not see data modified by IndexWriter until >>>> commit. >>>> >>>> 2009/8/13, Emmanuel Bernard <emman...@hibernate.org>: >>>>> Ah I thought it was using multiple threads because of your mass >>>>> indexing. I did not know some threads were span specifically for >>>>> the >>>>> Infinispan directory. >>>>> >>>>> On 13 août 09, at 17:34, Sanne Grinovero wrote: >>>>> >>>>>> Hi Łukasz, >>>>>> what is your usage of these threads? did you consider using one >>>>>> transaction per thread? >>>>>> >>>>>> Sanne >>>>>> >>>>>> 2009/8/13 Łukasz Moreń <lukasz.mo...@gmail.com>: >>>>>>> Newly created threads were not associated with any >>>>>>> transaction, so I >>>>>>> suppose it was a problem. Sharing transaction between threads >>>>>>> seems >>>>>>> to >>>>>>> be a good solution. >>>>>>> Thanks for help! >>>>>>> >>>>>>> 2009/8/13, Jason T. Greene <jason.gre...@redhat.com>: >>>>>>>> Correct. Also there could be read races as well, so if you are >>>>>>>> going to >>>>>>>> share a tx between threads, i would use some shared lock to >>>>>>>> gaurantee >>>>>>>> that only one thread can use it at a time. BTW this means you >>>>>>>> have >>>>>>>> to >>>>>>>> properly suspend/resume the TX via the TM API as well. >>>>>>>> >>>>>>>> Emmanuel Bernard wrote: >>>>>>>>> Modifying a transaction means applying muations (like SQL >>>>>>>>> INSERT / >>>>>>>>> UPDATE / DELETE) to the transactional resource? >>>>>>>>> >>>>>>>>> On 13 août 09, at 15:07, Jason T. Greene wrote: >>>>>>>>> >>>>>>>>>> When using transactions, the context is bound to the >>>>>>>>>> transaction, and >>>>>>>>>> you can move a transaction between threads. However, you >>>>>>>>>> should >>>>>>>>>> only >>>>>>>>>> be modifying a transaction with one thread at a time. >>>>>>>>>> >>>>>>>>>> Emmanuel Bernard wrote: >>>>>>>>>>> Could it be that you are not using the same transaction >>>>>>>>>>> between >>>>>>>>>>> different threads (ie you physically start different ones or >>>>>>>>>>> different "Infinispan contexts")? >>>>>>>>>>> Infini guys, do you support transactional operation spanning >>>>>>>>>>> several >>>>>>>>>>> concurrent threads? >>>>>>>>>>> On 13 août 09, at 14:04, Łukasz Moreń wrote: >>>>>>>>>>>> I've tried with JBoss AS transaction manager and >>>>>>>>>>>> JBossStandaloneTM. >>>>>>>>>>>> The result is this same in all cases - error during merge. >>>>>>>>>>>> >>>>>>>>>>>> 2009/8/12, Emmanuel Bernard <emman...@hibernate.org>: >>>>>>>>>>>>> Ok I understand better now. >>>>>>>>>>>>> Do your tests in JBoss AS with it's decent transaction >>>>>>>>>>>>> manager >>>>>>>>>>>>> (infinispan should have a config for it) >>>>>>>>>>>>> For unit testing, force the indexing process in >>>>>>>>>>>>> hibernate to >>>>>>>>>>>>> use a >>>>>>>>>>>>> single thread (I ghnk it's possible ask Sanne of you don't >>>>>>>>>>>>> know how). >>>>>>>>>>>>> >>>>>>>>>>>>> Exposing some configuration to infinispan makes sense. >>>>>>>>>>>>> can you >>>>>>>>>>>>> start a >>>>>>>>>>>>> thread explainig what is configurable and which one you >>>>>>>>>>>>> think >>>>>>>>>>>>> we >>>>>>>>>>>>> should expose to hsearch users. Ideally I would like to >>>>>>>>>>>>> offer >>>>>>>>>>>>> one or >>>>>>>>>>>>> two defaut config scenarios and allow to fallback to a >>>>>>>>>>>>> custom >>>>>>>>>>>>> config. >>>>>>>>>>>>> >>>>>>>>>>>>> Emmanuel >>>>>>>>>>>>> >>>>>>>>>>>>> On 12 août 2009, at 11:58, Łukasz Moreń >>>>>>>>>>>>> <lukasz.mo...@gmail.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Sorry, but my wifi does not work well today. I will try >>>>>>>>>>>>>> to >>>>>>>>>>>>>> explain >>>>>>>>>>>>>> it more clear. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm using DummyTransactionManager available for >>>>>>>>>>>>>> Infinispan. >>>>>>>>>>>>>> It associates transaction with the calling thread. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Steps to update index: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. index writer acquires lock - begin of transaction >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2. if it is necessary, index writer delegates new >>>>>>>>>>>>>> threads to >>>>>>>>>>>>>> do >>>>>>>>>>>>>> merge work. >>>>>>>>>>>>>> Those merge threads do not see changes made so far from >>>>>>>>>>>>>> begin of >>>>>>>>>>>>>> transaction, >>>>>>>>>>>>>> and are looking for segments which are not yet in index. >>>>>>>>>>>>>> Changes will be visible when AD.3 is completed. >>>>>>>>>>>>>> For tests i tried to commit transaction when merge starts >>>>>>>>>>>>>> and then >>>>>>>>>>>>>> everything worked well. But then i need to start it >>>>>>>>>>>>>> again. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3. index writer releases lock - transaction is >>>>>>>>>>>>>> commited, all >>>>>>>>>>>>>> changes >>>>>>>>>>>>>> made in this transaction are visible for other threads. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Maybe using some other transaction manager could help? >>>>>>>>>>>>>> >>>>>>>>>>>>>> What about Infinispan cache configuration? Some >>>>>>>>>>>>>> configuration >>>>>>>>>>>>>> mechanism should be exposed to the user, >>>>>>>>>>>>>> or we can hardcoded one in InfinispanDirectoryProvider is >>>>>>>>>>>>>> enough? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2009/8/12 Emmanuel Bernard <emman...@hibernate.org> >>>>>>>>>>>>>> why? >>>>>>>>>>>>>> Emmanuel Bernard >>>>>>>>>>>>>> Pending >>>>>>>>>>>>>> you there? >>>>>>>>>>>>>> Emmanuel Bernard >>>>>>>>>>>>>> Pending >>>>>>>>>>>>>> Ok please describe in details what is going on. From what >>>>>>>>>>>>>> you are >>>>>>>>>>>>>> describing the tx cannot see all segments which looks >>>>>>>>>>>>>> like an >>>>>>>>>>>>>> infinispan bug to me. >>>>>>>>>>>>>> Pending >>>>>>>>>>>>>> >>>>>>>>>>>>>> As a back up you can try wo transaction and see if that >>>>>>>>>>>>>> works >>>>>>>>>>>>>> Emmanuel Bernard >>>>>>>>>>>>>> Pending >>>>>>>>>>>>>> technically the lucene index should cope with that >>>>>>>>>>>>>> Emmanuel Bernard >>>>>>>>>>>>>> 11:16 >>>>>>>>>>>>>> but I like this approach less >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Let's try and chat by email IF I'm not online, I need >>>>>>>>>>>>>> to run >>>>>>>>>>>>>> on some >>>>>>>>>>>>>> errands today. >>>>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> infinispan-dev mailing list >>>>>>>>>>> infinispan-...@lists.jboss.org >>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Jason T. Greene >>>>>>>>>> JBoss, a division of Red Hat >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Jason T. Greene >>>>>>>> JBoss, a division of Red Hat >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-...@lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>>> _______________________________________________ >>>>>> hibernate-dev mailing list >>>>>> hibernate-dev@lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev >>>>> >>>>> >>>> >> >> _______________________________________________ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev