:) Keep your mind open, because there is no single solution for all the problems.
On Tue, May 29, 2012 at 1:15 PM, Alberto R. Galdo <arga...@gmail.com> wrote: > Thank you very much Mauricio. You've been of great help. Now is time for > us to cook all this valuable information and go for a solution. xD > > Alberto R. Galdo > arga...@gmail.com > > > > On Tue, May 29, 2012 at 5:59 PM, Mauricio Salatino <sala...@gmail.com>wrote: > >> Comments inline, >> >> On Tue, May 29, 2012 at 12:24 PM, Alberto R. Galdo <arga...@gmail.com>wrote: >> >>> Sure!. I didn't mention that we're running in a OSGi environment and >>> we're getting the session from JPAKnowledgeService like that: >>> >>> >>> >>> EntityManagerFactory emf = >>> Persistence.createEntityManagerFactory("com.package"); >>> >>> Environment env = KnowledgeBaseFactory.newEnvironment(); >>> >>> env.set(EnvironmentName.ENTITY_MANAGER_FACTORY, emf); >>> >>> env.set(EnvironmentName.TRANSACTION_MANAGER,(TransactionManager) >>> bundleContext.getService(this.tmServiceRef)); >>> >>> env.set(EnvironmentName.TRANSACTION, (UserTransaction) >>> bundleContext.getService(this.userTransactionServiceRef)); >>> >>> env.set(EnvironmentName.GLOBALS, new MapGlobalResolver()); >>> >>> env.set(EnvironmentName.OBJECT_MARSHALLING_STRATEGIES, new >>> ObjectMarshallingStrategy[] { >>> MarshallerFactory.newSerializeMarshallingStrategy() }); >>> >>> final StatefulKnowledgeSession ksession = >>> JPAKnowledgeService.newStatefulKnowledgeSession(kbase, sessionConfig, env); >>> >>> >>> We are just following the manual ( and some acquired knowledge >>> through the days ... ) .... I understand that inserting an event and >>> starting a process at the same time would lead to the current situation, >>> but Isn't this the reason why we are using drools with drools fusion + jbpm >>> to build a CEP? >>> >> I'm not sure about your reasons :) >> >>> >>> We chose Drools and a statefulsession in a fireUntilHalt loop just >>> because we would be able to have our rules and processes reacting to events >>> that enter the system at their own pace ... and found ourselves in this >>> nightmare when trying to make the system fault-tolerant through >>> persistence... :( >>> >>> FireUntilHalt and Persistence will cause "conflicts", because >> fireUntilHalts by default creates a different thread == race conditions. >> >> >>> Do you think we should change our fault-tolerance strategy by >>> reconstructing the knowledge session, process and tasks state by our own >>> means leaving Drools and JBPM JPA persistence appart? >>> >>> A common problem is to think that a session will do everything for you. >> I'm not saying that you need to separate your processes from your rules, >> I'm just saying that probably decoupling responsibilities will help you to >> tackle down some of the problems that you are having. If you have in memory >> processes and fusion entry points, you can keep those together and in a >> separate persistence session handle all the human task interactions and >> long running processes. If you are getting events in real time and you want >> to process them, you cannot expect to persist all the session state and >> have no performance impacts. >> >>> >>> Alberto R. Galdo >>> arga...@gmail.com >>> >>> >>> Cheers >> >>> >>> On Tue, May 29, 2012 at 4:43 PM, Mauricio Salatino <sala...@gmail.com>wrote: >>> >>>> Ok, so probably I'm not understanding your scenario, but are you using >>>> the JPAKnowledgeService to create persistent sessions? >>>> If you use that everything is aware of everything :) Drools and jBPM5 >>>> will run using the same persistence resources. For each interaction: >>>> startProcess, insert, update, fireAllRules a transaction will be created. >>>> If you are starting a process and at the same time inserting a Fusion event >>>> you will be generating two transaction that will fight for the resources >>>> (the same resources because you have only one session). >>>> Obviously if you take the persistence mechanisms outside of the >>>> discussion everything will work perfectly fine, because you don't have any >>>> transactional resource to fight for :) >>>> >>>> Hope it helps. >>>> Cheers >>>> >>>> >>>> On Tue, May 29, 2012 at 11:36 AM, Alberto R. Galdo >>>> <arga...@gmail.com>wrote: >>>> >>>>> Your answer is confusing me, I think I'm not getting it right. >>>>> >>>>> It seems that we you're saying that will have to modify JBPM or >>>>> Drools behaviour to get our architecture done ... but I don't think we are >>>>> doing nothing special than what Drools & JBPM are able to do in a >>>>> non-persisting enviroment ( at least without JBPM persisted ). >>>>> >>>>> Maybe deeping a bit more the architecture will lead to a better >>>>> understanding of the problem: >>>>> >>>>> We have a Drools StatefulKnowledgeSession wich is populated with >>>>> packages of rules and different BPMN 2.0 process definitions. That session >>>>> in Drools is persisted using JPA and our JTA provider. Our solution is a >>>>> CEP that uses events that get into the session using Drools Fusion. There, >>>>> several rules fire and then several consequences start processes wich >>>>> contain both Tasks and HumanTasks. Those rule activations are already in a >>>>> queue, the Agenda. Given that JBPM acts as the processmanager provider for >>>>> Drools, our processes get started in JBPM, and it is persisted using it's >>>>> own JPA manager. Just rules, that start processes in response to events >>>>> in >>>>> a fireUntilHalt loop in a persistent enviroment, both for Drools & JBPM. >>>>> Then some processes interact with the knowledge session using >>>>> BussinessTask >>>>> nodes( which make other rules fire and maybe launch a different set of >>>>> processes ) and other simply get to the Stop node. I see nothing wrong >>>>> here, in fact, taking the persistence appart from the equation, everything >>>>> runs as beautifuly as expected. We've tested that. >>>>> >>>>> I'm just trying to undersand the problem, I'm pretty 90% sure I'm >>>>> wrong, but what I think is happening here is that Drools isn't aware that >>>>> JBPM may be using the same resources and vice-versa, Isn't it? >>>>> >>>>> Shouldn't the solution be to make Drools & JBPM aware of each other >>>>> in what is related to the syncronization of the transaction management to >>>>> avoid race conditions? >>>>> >>>>> >>>>> >>>>> >>>>> Alberto R. Galdo >>>>> arga...@gmail.com >>>>> >>>>> >>>>> On Tue, May 29, 2012 at 3:58 PM, Mauricio Salatino >>>>> <sala...@gmail.com>wrote: >>>>> >>>>>> I'm in no way saying this: >>>>>> " Let me see if I'm getting this right, What you mean is that there >>>>>> is no way for the JPA persistence of Drools & JBPM to coexist in the same >>>>>> knowledge session because of race conditions between them competing for >>>>>> their shared resources? I'm afraid so.. :( >>>>>> >>>>>> They can coexist. As you can see they are working together. If you >>>>>> take a look at the underlying layers you will see that each operation >>>>>> that >>>>>> you run against the session is wrapped in a transaction. That leads to >>>>>> your >>>>>> architecture, If you have a single entry point for your session >>>>>> everything >>>>>> will work correctly. You can achieve this by receiving all the operations >>>>>> in a queue and then execute one after the other. >>>>>> If you have multiple threads interacting with the session, you can >>>>>> implement a retrying mechanism if the transaction gets rolled back and as >>>>>> soon as the transaction win the right to commit it will work. It really >>>>>> depends on the problem that you are trying to solve. >>>>>> >>>>>> Cheers >>>>>> >>>>>> On Tue, May 29, 2012 at 10:42 AM, Alberto R. Galdo <arga...@gmail.com >>>>>> > wrote: >>>>>> >>>>>>> Thanks Mauricio for your kind response, and kick, as usual ;-) >>>>>>> >>>>>>> Sure, we see lots of rolled back transactions ... Everywhere! >>>>>>> >>>>>>> Let me see if I'm getting this right, What you mean is that there >>>>>>> is no way for the JPA persistence of Drools & JBPM to coexist in the >>>>>>> same >>>>>>> knowledge session because of race conditions between them competing for >>>>>>> their shared resources? I'm afraid so.. :( >>>>>>> >>>>>>> If it is so, Are there any plans for the integration of a shared >>>>>>> JPA persistence management of both Drools & JBPM, wich may be the >>>>>>> solution >>>>>>> for this problem? ... Will we be better thinking to change our >>>>>>> architecture >>>>>>> ASAP? >>>>>>> >>>>>>> Greets, >>>>>>> >>>>>>> Alberto R. Galdo >>>>>>> arga...@gmail.com >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, May 29, 2012 at 1:54 PM, Mauricio Salatino < >>>>>>> sala...@gmail.com> wrote: >>>>>>> >>>>>>>> Ok, so what is happening is that two or more threads are fighting >>>>>>>> for the resources, causing that two or more transactions wants to be >>>>>>>> completed, but just one can win. All the other must be retried. Are you >>>>>>>> seeing rolled back exceptions in your stack trace? >>>>>>>> My question to you is if you really need to handle everything in >>>>>>>> just one big persistent session. I've solved similar issues in the past >>>>>>>> using more than one session, for example one session to run business >>>>>>>> processes(persistent) and one or more for receiving and reacting to >>>>>>>> events. >>>>>>>> If you have an async way to communicate both sessions there should be >>>>>>>> no >>>>>>>> problem. >>>>>>>> Hope it helps! >>>>>>>> Cheers >>>>>>>> >>>>>>>> On Tue, May 29, 2012 at 7:30 AM, Alberto R. Galdo < >>>>>>>> arga...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> We've been for long time now developing a complex event >>>>>>>>> processing system that ( simplified version ahead !! ) involves a set >>>>>>>>> of >>>>>>>>> rules that in turn activates a set processes that fulfill human tasks >>>>>>>>> and >>>>>>>>> other kind of tasks.. This all is running in a >>>>>>>>> StatefulKnowledgeSession >>>>>>>>> with JPA persistence configured both for Drools and JBPM. We are >>>>>>>>> running an >>>>>>>>> stack of Drools, JBPM, Drools Integration, Drools fussion, etc.. >>>>>>>>> >>>>>>>>> We've been able to persist Drools sessions & JBPM processes in >>>>>>>>> the same Persistence Context ( not without pain :( ) using different >>>>>>>>> JTA >>>>>>>>> implementations including ( but not limited to ) Bitronix & Atomikos. >>>>>>>>> >>>>>>>>> What we are observing here is what seems to be some kind of >>>>>>>>> race condition between Drools and JBPM when running the knowledge >>>>>>>>> session >>>>>>>>> when JPA&JTA persistence is configured. Very often, as soon as after >>>>>>>>> 2-3 >>>>>>>>> processes get created as rule's consequences are fired in response of >>>>>>>>> events inside the session we see how JBPM finds its instance >>>>>>>>> nullified by >>>>>>>>> Drools when it tries to end a process and persist it. >>>>>>>>> >>>>>>>>> We've been able to find where Drools decides to delete an >>>>>>>>> instance of the process ... at a given time Drools executes >>>>>>>>> JPAProcessInstanceManager.clearProcessInstances() [1] when it >>>>>>>>> finalizes a >>>>>>>>> SingleSessionCommand wich in turn calls disconnect() for *all* the >>>>>>>>> local-stored processinstances ( wich gets populated with instances of >>>>>>>>> processes every time a process is started in the knowledge session ): >>>>>>>>> >>>>>>>>> public void clearProcessInstances() { >>>>>>>>> >>>>>>>>> >>>>>>>>> for (ProcessInstance processInstance: new >>>>>>>>> ArrayList<ProcessInstance>(processInstances.values())) { >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ((ProcessInstanceImpl) processInstance).disconnect(); >>>>>>>>> >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> So, Drools decides to disconnect all process instances in it's >>>>>>>>> JPA context without taking in account the state the process is in, >>>>>>>>> and when >>>>>>>>> an processinstance that is not stopped gets removed then JBPM finds >>>>>>>>> it's >>>>>>>>> NullPointerException... >>>>>>>>> >>>>>>>>> We've modified the code to make Drools aware of the state of >>>>>>>>> the process before wiping it from the context ( no problem here, >>>>>>>>> there >>>>>>>>> will be no leak as a running processinstance will be removed in future >>>>>>>>> calls of "clearProcessInstances" given the process is closed ). But >>>>>>>>> unfortunatelly this seems to resolve this problem, but lots of other >>>>>>>>> problems ( wich seems also race conditions arise : for instance: >>>>>>>>> Drools >>>>>>>>> closes connections to the database and JBPM finds the connection >>>>>>>>> closed, >>>>>>>>> >>>>>>>>> >>>>>>>>> So, we are really worried about using Drools & JBPM in a >>>>>>>>> persisted environment. Maybe our asumptions are wrong... Is it >>>>>>>>> possible to >>>>>>>>> have an scenario like ours given the current Drools & JBPM integration >>>>>>>>> status for a persistent statefulKnowledge Session? Did anyone build a >>>>>>>>> complex event processing system like ours in a unaltered persistence >>>>>>>>> environment such as provided in Drools and JBPM by default? >>>>>>>>> >>>>>>>>> >>>>>>>>> Greets, >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> https://github.com/droolsjbpm/jbpm/blob/master/jbpm-persistence-jpa/src/main/java/org/jbpm/persistence/processinstance/JPAProcessInstanceManager.java >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Alberto R. Galdo >>>>>>>>> arga...@gmail.co <arga...@gmail.com> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rules-users mailing list >>>>>>>>> rules-users@lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> - MyJourney @ http://salaboy.wordpress.com >>>>>>>> - Co-Founder @ http://www.jugargentina.org >>>>>>>> - Co-Founder @ http://www.jbug.com.ar >>>>>>>> >>>>>>>> - Salatino "Salaboy" Mauricio - >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rules-users mailing list >>>>>>>> rules-users@lists.jboss.org >>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> rules-users mailing list >>>>>>> rules-users@lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> - MyJourney @ http://salaboy.wordpress.com >>>>>> - Co-Founder @ http://www.jugargentina.org >>>>>> - Co-Founder @ http://www.jbug.com.ar >>>>>> >>>>>> - Salatino "Salaboy" Mauricio - >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> rules-users mailing list >>>>>> rules-users@lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> rules-users mailing list >>>>> rules-users@lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>> >>>>> >>>> >>>> >>>> -- >>>> - MyJourney @ http://salaboy.wordpress.com >>>> - Co-Founder @ http://www.jugargentina.org >>>> - Co-Founder @ http://www.jbug.com.ar >>>> >>>> - Salatino "Salaboy" Mauricio - >>>> >>>> >>>> _______________________________________________ >>>> rules-users mailing list >>>> rules-users@lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>> >>>> >>> >>> _______________________________________________ >>> rules-users mailing list >>> rules-users@lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/rules-users >>> >>> >> >> >> -- >> - MyJourney @ http://salaboy.wordpress.com >> - Co-Founder @ http://www.jugargentina.org >> - Co-Founder @ http://www.jbug.com.ar >> >> - Salatino "Salaboy" Mauricio - >> >> >> _______________________________________________ >> rules-users mailing list >> rules-users@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/rules-users >> >> > > _______________________________________________ > rules-users mailing list > rules-users@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users > > -- - MyJourney @ http://salaboy.wordpress.com - Co-Founder @ http://www.jugargentina.org - Co-Founder @ http://www.jbug.com.ar - Salatino "Salaboy" Mauricio -
_______________________________________________ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users