Hi What is the exact url you use in the consumer template? And are you 100% sure you only create and use exactly one instance of the consumer template?
On Mon, Feb 17, 2014 at 9:26 AM, Chirag Dewan <chirag.dewa...@yahoo.in> wrote: > > > Hi Claus, > > I had this implementation > before. With this I saw an unusual behaviour. A thread was created for > every file consumed by the template and even though I called > template.doneUoW the thread is not stopped. So if I call doneUoW(and not > stop) the thread should be destroyed? And also on doneUoW the > inprogressRepository cache is cleared. Which is not getting cleared at > the moment(the reason I believe is causing the OOM) . Is there a possibility > that doneUoW is not working as it should? > > Thanks . > > Chirag > > ________________________________ > From: Chirag Dewan <chirag.dewa...@yahoo.in> > To: "users@camel.apache.org" <users@camel.apache.org> > Sent: Monday, 17 February 2014 1:53 PM > Subject: Re: OOM issue due to MemoryIdempotentRepository > > > Hi Claus, > > I had this implementation before. With this I saw an unusual behaviour. A > thread was created for every file consumed by the template and even though I > called template.doneUoW the thread is not stopped. So if I call doneUoW(and > not stop) the thread should be destroyed? And also on doneUoW the > inprogressRepository cache is cleared. Which is not getting cleared at the > moment(the reason I believe is causing the . Is there a possibility that > doneUoW is not working as it should? > > Thanks . > > Chirag > > > > > ________________________________ > > From: Claus Ibsen <claus.ib...@gmail.com> > To: "users@camel.apache.org" <users@camel.apache.org>; Chirag Dewan > <chirag.dewa...@yahoo.in> > Sent: Monday, 17 February 2014 1:28 PM > Subject: Re: OOM issue due to MemoryIdempotentRepository > > > On Mon, Feb 17, 2014 at 8:49 AM, Chirag Dewan <chirag.dewa...@yahoo.in> wrote: >> Hi Claus, >> >> So I dont need to start/stop the template for every file? >> >> Right now I create a consumerTemplate at satrtup and then on every method >> call(responsible for consuming file) I start the template,consume the >> file(template.receive) and stop the template. >> >> Is this the right approach? >> > > No, only start|stop the template once on startup and shutdown of your app. > And then call doneUoW for each file you consume. > > >> Thanks. >> >> Chirag >> >> >> >> >> ________________________________ >> From: Claus Ibsen <claus.ib...@gmail.com> >> To: "users@camel.apache.org" <users@camel.apache.org> >> Sent: Monday, 17 February 2014 12:33 PM >> Subject: Re: OOM issue due to MemoryIdempotentRepository >> >> >> Hi >> >> You should likely reuse the consumer template instead. As this FAQ >> applies to it too >> http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html >> >> On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <chirag.dewa...@yahoo.in> >> wrote: >>> Hi All, >>> >>> Any views on this? >>> >>> Actually my previous mail is not understandable. Let me rephrase the issue. >>> >>> I am using consumer template to consume single file using fileName >>> property. I have a single instance of consumerTemplate and I call doneUoW >>> and stop after consuming the file. >>> >>> I am running a test case where 10000 files are consumed by >>> consumerTemplate(one file at a time) . After a while JVM goes OOM and >>> crashes. In heap dump,I can see large number of MemoryIdempotentRepository >>> instances. I am not using idempotent=true. I figured out that >>> inProgressRepository also uses this cache. >>> >>> Is there something I am missing here? >>> >>> Thanks in advance. >>> >>> Chirag Dewan >>> >>> >>> >>> >>> ________________________________ >>> From: Chirag Dewan <chirag.dewa...@yahoo.in> >>> To: "users@camel.apache.org" <users@camel.apache.org>; Chirag Dewan >>> <chirag.dewa...@yahoo.in> >>> Sent: Friday, 14 February 2014 3:22 PM >>> Subject: Re: OOM issue due to MemoryIdempotentRepository >>> >>> >>> Hi All, >>> >>> I have been looking around why a large number of heap is being consumed by >>> MemoryIdempotentRepository. I saw one JIRA as well regarding >>> inProgressRepository not getting cleared. Can that be the reason here? >>> >>> Also,I am not using consumerTemplate to consume 1000o files. Does that also >>> use inProgressRepository? And when is it cleared from the memory? >>> >>> Any help is much appreciated. >>> >>> Thanks! >>> >>> Chirag Dewan >>> >>> >>> >>> >>> ________________________________ >>> >>> From: Chirag Dewan <chirag.dewa...@yahoo.in> >>> To: "users@camel.apache.org" <users@camel.apache.org>; Chirag Dewan >>> <chirag.dewa...@yahoo.in> >>> Sent: Wednesday, 12 February 2014 6:31 PM >>> Subject: Re: OOM issue due to MemoryIdempotentRepository >>> >>> >>> Hi, >>> >>> And I am using readLock=changed. So that can be a reason too? >>> >>> Chirag Dewan >>> >>> >>> >>> >>> ________________________________ >>> >>> From: Chirag Dewan <chirag.dewa...@yahoo.in> >>> To: "users@camel.apache.org" <users@camel.apache.org> >>> Sent: Wednesday, 12 February 2014 6:07 PM >>> Subject: Re: OOM issue due to MemoryIdempotentRepository >>> >>> >>> Hi Claus, >>> >>> Thanks for the quick reply. >>> >>> It might well be. But there is one thing I would like to get some insight >>> into. >>> >>> If idempotent=false(default) is used,will the LRU Cache still store the >>> file referrence? Because that is what I can see in my memory dump. >>> >>> org.apache.camel.util.LRUCache >>> >>> >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap >>> >>> I maybe doing something wrong here. Is there anything else I need to do to >>> make sure that file is deleted after it has been processed and not stored >>> in the cache? >>> >>> Thanks. >>> >>> Chirag Dewan >>> >>> >>> >>> >>> >>> ________________________________ >>> >>> From: Claus Ibsen <claus.ib...@gmail.com> >>> To: "users@camel.apache.org" <users@camel.apache.org> >>> Sent: Wednesday, 12 February 2014 5:42 PM >>> Subject: Re: OOM issue due to MemoryIdempotentRepository >>> >>> >>> Hi >>> >>> Some days ago someone else posted about a OOME issue when using Storm. >>> It smells like a Storm + Camel issue somewhere. >>> >>> >>> >>> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <chirag.dewa...@yahoo.in> >>> wrote: >>>> Hi All, >>>> >>>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in >>>> my application. >>>> >>>> >>>> Now I intent to delete the file after consuming it. So I have delete=true >>>> in my route. After consuming around 6000 files(out of 10000,as my test >>>> case) JVM goes Out of memory. >>>> >>>> >>>> In my heap dump I can see the following trace as the major memory >>>> consumers: >>>> >>>> java.util.ArrayList >>>> >java.lang.Object >>>> >>>> >>>> >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService >>>> >>>> >org.apache.camel.component.file.FileEndpoint >>>> >>>> >>>> >org.apache.camel.processor.idempotent.MemoryIdempotentRepository >>>> >org.apache.camel.util.LRUCache >>>> >>>> >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap >>>> >>>> >>>> So if I am not using indempotent,should it still store the files in the >>>> memory? Is this memory repository used,even if I am deleting the files and >>>> not storing in memory? Is there a way I can avoid that? >>>> >>>> I am testing this scenario under Storm as execution environment and with >>>> 2GB max heap space. >>>> >>>> Thanks in advance >>>> >>>> Chirag Dewan >>> >>> >>> >>> -- >>> Claus Ibsen >>> ----------------- >>> Red Hat, Inc. >>> Email: cib...@redhat.com >>> Twitter: davsclaus >>> Blog: http://davsclaus.com >>> Author of Camel in Action: http://www.manning.com/ibsen >>> Make your Camel applications look hawt, try: http://hawt.io > >> >> >> >> >> -- >> Claus Ibsen >> ----------------- >> Red Hat, Inc. >> Email: cib...@redhat.com >> Twitter: davsclaus >> Blog: http://davsclaus.com >> Author of Camel in Action: http://www.manning.com/ibsen >> Make your Camel applications look hawt, try: http://hawt.io > > > > -- > Claus Ibsen > ----------------- > Red Hat, Inc. > Email: cib...@redhat.com > Twitter: davsclaus > Blog: http://davsclaus.com > Author of Camel in Action: http://www.manning.com/ibsen > Make your Camel applications look hawt, try: http://hawt.io -- Claus Ibsen ----------------- Red Hat, Inc. Email: cib...@redhat.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen Make your Camel applications look hawt, try: http://hawt.io
