Hi Claus,
I think we need to implements a method for get unique name from the
endpoint. It will base on the endpoint's parameter not the customer
parameter. With this method we could avoid to register to much temp
endpoint with different customer parameter.
In this case we need find a way to trim the customer parameter from the
endpoint URI. If we could do that, it will save us lots room for the
store the fake temp http endpoints (we could share the endpoints in the
ProducerCache/ConsumerCache with different customer parameter) and make
the LRUCache more effective.
Just my two cents.
Willem
Claus Ibsen wrote:
Hi
Tickets
======
I am looking into two issues related to Camel eating memory
https://issues.apache.org/activemq/browse/CAMEL-1771
https://issues.apache.org/activemq/browse/CAMEL-1738
The problem
=========
It all boils down to using a lot of http endpoints with unique urls.
So over time Camel accumulate a lot of created endpoints in its
internal endpoint registry and as well as JMX Beans
That consumes memory, and for people with millions unique endpoints
over time that consumes to much memory
Solutions
=======
To remedy this I have addressed in two areas
a) failsafe with a limited size of 1000 entries
- Using the LRUCache for ProducerCache/ConsumerCache (verified by end
user that its much better)
- Using the LRUCache in CamelContext for its endpoint reigstry
I assume having more than 1000 unique endpoints, producers or
consumers is not within normal usage for a given CamelContext?
And it does not bring harm as Camel will just recreate the object if
not already cached.
b)
- Not registering endpoints with lenient properties in JMX or camel
context. That is if only they use properties that Camel does not know
about (= lenient properties)
as these endpoints is highly not reusable and short lived. And its
only a few endpoints that support lenient properties (http, restlet,
cxf, atom, rss)
So if you setup an endpoint with custom parameters (not Camel options)
then its a lenient property and Camel will not cache/register it.
For example:
http://myserver/mypath?id=1
http://myserver/mypath?id=2
http://myserver/mypath?id=3
...
where id is a lenient property that is not a Camel option.
So these will not be registered.
But if you use
http://myserver/mypath
then it will be registered as its does not contain any lenient
properties at all.
c)
- Only registering a single JMX bean.
Otherwise the JMX registry will be cluttered with millions endpoints.
So what we do not is to register the same parent endpoint for the
endpoints.
We use the new getEndpointKey() on Endpoint to provide the key to use
for JMX registration. This allows us to provide the same key for all
the http endpoints.
Questions
========
1)
Option a and c is already nearly done. I am running further unit tests
and do a bit of code polish.
2)
As option b is a bit controversial, I am wondering if that is
feasible. Should we just go ahead with the LRUCache being able to
filter out the eldest endpoints?
And I wonder if people just create a few endpoints with lenient
properties then we still have "room" to store them in the cache so I
wonder if we should do this at all?
I do think we could keep this as a thought for the future, and just
let the LRUCache handle it, so when people use millions of unqiue http
endpoints we let the chace
filter out the not used ones.
