On Tue, Jun 30, 2009 at 11:15 AM, Willem Jiang<willem.ji...@gmail.com> wrote: > Hi Claus, > > I think we need to implements a method for get unique name from the > endpoint. It will base on the endpoint's parameter not the customer > parameter. With this method we could avoid to register to much temp endpoint > with different customer parameter. Yeah good idea. I kinda added that already with the getEndpointKey(). You can then override the default implementation on demand.
> > In this case we need find a way to trim the customer parameter from the > endpoint URI. If we could do that, it will save us lots room for the store > the fake temp http endpoints (we could share the endpoints in the > ProducerCache/ConsumerCache with different customer parameter) and make the > LRUCache more effective. The problem with the http endpoints is that they require the full uri to be useable as the producer created by it will send the camel message to the full uri provided on the endpoint. So if the http endpoint is configured as: http://someserver/somepath/foo?id=1234 Then we cannot create/reuse a single endpoint with the uri: http://someserver/somepath/foo As then we will loose the ?id=1234 parameter. So there is no quick win to only cache 1 shared http endpoint and creating producers that know to use ?id=1234 when sending to it. But as James hinted we could add a IsCacheable method/interface to mark a given endpoint/producer whatever as being capable of being cached. Then we could deny caching those http endpoints with lenient properties, as they are highly not reuseable. BTW: then end user confirmed that the memory issue is fixed in 1.6.2 with the patches committed today. > > Just my two cents. > > Willem > > Claus Ibsen wrote: >> >> Hi >> >> Tickets >> ====== >> I am looking into two issues related to Camel eating memory >> https://issues.apache.org/activemq/browse/CAMEL-1771 >> https://issues.apache.org/activemq/browse/CAMEL-1738 >> >> >> The problem >> ========= >> It all boils down to using a lot of http endpoints with unique urls. >> So over time Camel accumulate a lot of created endpoints in its >> internal endpoint registry and as well as JMX Beans >> That consumes memory, and for people with millions unique endpoints >> over time that consumes to much memory >> >> >> Solutions >> ======= >> To remedy this I have addressed in two areas >> >> a) failsafe with a limited size of 1000 entries >> - Using the LRUCache for ProducerCache/ConsumerCache (verified by end >> user that its much better) >> - Using the LRUCache in CamelContext for its endpoint reigstry >> >> I assume having more than 1000 unique endpoints, producers or >> consumers is not within normal usage for a given CamelContext? >> And it does not bring harm as Camel will just recreate the object if >> not already cached. >> >> >> b) >> - Not registering endpoints with lenient properties in JMX or camel >> context. That is if only they use properties that Camel does not know >> about (= lenient properties) >> as these endpoints is highly not reusable and short lived. And its >> only a few endpoints that support lenient properties (http, restlet, >> cxf, atom, rss) >> >> So if you setup an endpoint with custom parameters (not Camel options) >> then its a lenient property and Camel will not cache/register it. >> For example: >> >> http://myserver/mypath?id=1 >> http://myserver/mypath?id=2 >> http://myserver/mypath?id=3 >> ... >> >> where id is a lenient property that is not a Camel option. >> So these will not be registered. >> >> >> But if you use >> http://myserver/mypath >> then it will be registered as its does not contain any lenient >> properties at all. >> >> >> c) >> - Only registering a single JMX bean. >> Otherwise the JMX registry will be cluttered with millions endpoints. >> So what we do not is to register the same parent endpoint for the >> endpoints. >> We use the new getEndpointKey() on Endpoint to provide the key to use >> for JMX registration. This allows us to provide the same key for all >> the http endpoints. >> >> >> Questions >> ======== >> 1) >> Option a and c is already nearly done. I am running further unit tests >> and do a bit of code polish. >> >> 2) >> As option b is a bit controversial, I am wondering if that is >> feasible. Should we just go ahead with the LRUCache being able to >> filter out the eldest endpoints? >> And I wonder if people just create a few endpoints with lenient >> properties then we still have "room" to store them in the cache so I >> wonder if we should do this at all? >> I do think we could keep this as a thought for the future, and just >> let the LRUCache handle it, so when people use millions of unqiue http >> endpoints we let the chace >> filter out the not used ones. >> >> >> >> > > -- Claus Ibsen Apache Camel Committer Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/ Twitter: http://twitter.com/davsclaus
