Hi Tickets ====== I am looking into two issues related to Camel eating memory https://issues.apache.org/activemq/browse/CAMEL-1771 https://issues.apache.org/activemq/browse/CAMEL-1738
The problem ========= It all boils down to using a lot of http endpoints with unique urls. So over time Camel accumulate a lot of created endpoints in its internal endpoint registry and as well as JMX Beans That consumes memory, and for people with millions unique endpoints over time that consumes to much memory Solutions ======= To remedy this I have addressed in two areas a) failsafe with a limited size of 1000 entries - Using the LRUCache for ProducerCache/ConsumerCache (verified by end user that its much better) - Using the LRUCache in CamelContext for its endpoint reigstry I assume having more than 1000 unique endpoints, producers or consumers is not within normal usage for a given CamelContext? And it does not bring harm as Camel will just recreate the object if not already cached. b) - Not registering endpoints with lenient properties in JMX or camel context. That is if only they use properties that Camel does not know about (= lenient properties) as these endpoints is highly not reusable and short lived. And its only a few endpoints that support lenient properties (http, restlet, cxf, atom, rss) So if you setup an endpoint with custom parameters (not Camel options) then its a lenient property and Camel will not cache/register it. For example: http://myserver/mypath?id=1 http://myserver/mypath?id=2 http://myserver/mypath?id=3 ... where id is a lenient property that is not a Camel option. So these will not be registered. But if you use http://myserver/mypath then it will be registered as its does not contain any lenient properties at all. c) - Only registering a single JMX bean. Otherwise the JMX registry will be cluttered with millions endpoints. So what we do not is to register the same parent endpoint for the endpoints. We use the new getEndpointKey() on Endpoint to provide the key to use for JMX registration. This allows us to provide the same key for all the http endpoints. Questions ======== 1) Option a and c is already nearly done. I am running further unit tests and do a bit of code polish. 2) As option b is a bit controversial, I am wondering if that is feasible. Should we just go ahead with the LRUCache being able to filter out the eldest endpoints? And I wonder if people just create a few endpoints with lenient properties then we still have "room" to store them in the cache so I wonder if we should do this at all? I do think we could keep this as a thought for the future, and just let the LRUCache handle it, so when people use millions of unqiue http endpoints we let the chace filter out the not used ones. -- Claus Ibsen Apache Camel Committer Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/ Twitter: http://twitter.com/davsclaus
