On Tue, Jun 30, 2009 at 1:50 PM, James Strachan<james.strac...@gmail.com> wrote: > I'm liking a) and allowing c) (a single mbean for all endpoints of a > certain type) make sense to me. Just for the record. Option a + c is currently implemented and reported as fixed the issues by the end user.
> > Using an LRU cache for non-singleton endpoints makes sense too; though > I wonder if an endpoint might want to mark itself as not cachable if > it knows its not really worth the trouble (as it has no state?). > > > 2009/6/30 Willem Jiang <willem.ji...@gmail.com>: >> Hi Claus, >> >> I think we need to implements a method for get unique name from the >> endpoint. It will base on the endpoint's parameter not the customer >> parameter. With this method we could avoid to register to much temp endpoint >> with different customer parameter. >> >> In this case we need find a way to trim the customer parameter from the >> endpoint URI. If we could do that, it will save us lots room for the store >> the fake temp http endpoints (we could share the endpoints in the >> ProducerCache/ConsumerCache with different customer parameter) and make the >> LRUCache more effective. >> >> Just my two cents. >> >> Willem >> >> Claus Ibsen wrote: >>> >>> Hi >>> >>> Tickets >>> ====== >>> I am looking into two issues related to Camel eating memory >>> https://issues.apache.org/activemq/browse/CAMEL-1771 >>> https://issues.apache.org/activemq/browse/CAMEL-1738 >>> >>> >>> The problem >>> ========= >>> It all boils down to using a lot of http endpoints with unique urls. >>> So over time Camel accumulate a lot of created endpoints in its >>> internal endpoint registry and as well as JMX Beans >>> That consumes memory, and for people with millions unique endpoints >>> over time that consumes to much memory >>> >>> >>> Solutions >>> ======= >>> To remedy this I have addressed in two areas >>> >>> a) failsafe with a limited size of 1000 entries >>> - Using the LRUCache for ProducerCache/ConsumerCache (verified by end >>> user that its much better) >>> - Using the LRUCache in CamelContext for its endpoint reigstry >>> >>> I assume having more than 1000 unique endpoints, producers or >>> consumers is not within normal usage for a given CamelContext? >>> And it does not bring harm as Camel will just recreate the object if >>> not already cached. >>> >>> >>> b) >>> - Not registering endpoints with lenient properties in JMX or camel >>> context. That is if only they use properties that Camel does not know >>> about (= lenient properties) >>> as these endpoints is highly not reusable and short lived. And its >>> only a few endpoints that support lenient properties (http, restlet, >>> cxf, atom, rss) >>> >>> So if you setup an endpoint with custom parameters (not Camel options) >>> then its a lenient property and Camel will not cache/register it. >>> For example: >>> >>> http://myserver/mypath?id=1 >>> http://myserver/mypath?id=2 >>> http://myserver/mypath?id=3 >>> ... >>> >>> where id is a lenient property that is not a Camel option. >>> So these will not be registered. >>> >>> >>> But if you use >>> http://myserver/mypath >>> then it will be registered as its does not contain any lenient >>> properties at all. >>> >>> >>> c) >>> - Only registering a single JMX bean. >>> Otherwise the JMX registry will be cluttered with millions endpoints. >>> So what we do not is to register the same parent endpoint for the >>> endpoints. >>> We use the new getEndpointKey() on Endpoint to provide the key to use >>> for JMX registration. This allows us to provide the same key for all >>> the http endpoints. >>> >>> >>> Questions >>> ======== >>> 1) >>> Option a and c is already nearly done. I am running further unit tests >>> and do a bit of code polish. >>> >>> 2) >>> As option b is a bit controversial, I am wondering if that is >>> feasible. Should we just go ahead with the LRUCache being able to >>> filter out the eldest endpoints? >>> And I wonder if people just create a few endpoints with lenient >>> properties then we still have "room" to store them in the cache so I >>> wonder if we should do this at all? >>> I do think we could keep this as a thought for the future, and just >>> let the LRUCache handle it, so when people use millions of unqiue http >>> endpoints we let the chace >>> filter out the not used ones. >>> >>> >>> >>> >> >> > > > > -- > James > ------- > http://macstrac.blogspot.com/ > > Open Source Integration > http://fusesource.com/ > -- Claus Ibsen Apache Camel Committer Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/ Twitter: http://twitter.com/davsclaus
