On Fri, Nov 27, 2009 at 5:33 AM, Pete Mueller <p...@routecloud.com> wrote:
>
> Thanks very much for your answers, I also found an excellent programming
> guide at the FUSEsource site. That helped a lot as well.
>
> I understand what you wrote below, but it raises some concerns in my mind
> with the reliability of Camel when dealing with unreliable external
> entities. I'm guessing that just stems from a lack of understanding on my
> part. If it helps, I come from an EDI background where systems are designed
> to gaurantee delivery of millions of messages a day to thousands of
> different endpoints. I'll try and illustrate my concern.
>
> I have two routes, the input, from a custom component, is a graphic image
> that is analyized (which can take 30-90 seconds) and them placed on a queue.
> The second route takes items off the queue and emails them to a destination
> AND puts the modified image back on the server.
> Something like:
>
> from("mycomp:host")
> .to(new MyImageProcessor())
> .to("amq:input_queue");
>
> from("amq:"input_queue")
> multicast().to("smtp:t...@test.com", "mycomp:host");
>
> Here what I need to account for:
> 1. Most importantly, I cannot lose any messages. So let's say that a
> message is taken off the queue and passed to the smtp component, but the
> mail server is not up, so it throws an exception.
> - What happens to the message? Does it get placed back on the queue? Is
> there a concept of transactions around routes so that only after BOTH
> endpoints confirm a sucessful delivery does the message get removed from the
> queue?
> - What if the smtp server sucessfully delivers and "mycomp" throws an
> exception?
>
> I am aware of the exceptionHandler() idea, but it seems exceptionHandlers
> are placed on a RouteCollection, not a single route. Each instance of
> RouteBuilder could have one route in it. But that creates a lot of extra
> classes.
If you bought the book and read chapter 5 about error handling you got
35 good pages on this matter.
Yes just referring to the fact that the book has a lot of information
about error handling.
Even though the online docu as well, but its not structure as well and
walking you through it so you would understand it well, as the book
does.
Or if you started reading from here
http://camel.apache.org/error-handling-in-camel.html
You will see that it supports 2 scopes
Camel supports 2 scopes for the non transactional type:
- global
- per route
And if you looked at this page
http://camel.apache.org/error-handler.html
There is a few examples on scopes.
>
> 2. After ensuring that i did not lose the message pending at the smtp
> component, I need to shut down the routes since the smtp service is dead
> (for the moment). This would seem to be a job for exceptionHandler(),
> however that handler would need to know which object threw the exception (in
> this case the smtp Producer) and what route it belongs to. then, it would
> need to scan all other route that use the same endpoint, and stop them too.
> I see no way of doing this from within the handler, since it is not notified
> which Producer threw the exception. Also, I see no way of consistently
> getting from a Producer to it's owner endpoint.
:) Have you actually looked at the Producer interface? There is a
getEndpoint() which is the owner endpoint.
public interface Producer extends Processor, Service, IsSingleton {
/**
* Gets the endpoint this producer sends to.
*
* @return the endpoint
*/
Endpoint getEndpoint();
> - How do you gracefully and without data loss bring down a route?
> - Does bringing down a route destroy the consumer/producer objects? or just
> stop() them?
>
> 3. Assuming I could figure out which routes to shut down. I can execute
> route.stop(). But what happens to messages that are "in process" take for
> example a message from mycomp that is currently being analyzed, when the
> analyzer finishes (say 20-30 seconds after the route.stop() was called:
> - Is the queue Producer still available?
> - Will route.stop() block until all parts of the route are shut down?
> - Will route.stop() stop ONLY the consumer that is part of it's route, or
> the entire route?
> - Really, I would not need to bring down the route, just stop() the consumer
> at the beginning. Do the Producers/Consumers/Endpoint have any knowledge
> about the route they are on? Or can they get it some how?
> - Can a Consumer/Producer be shared amongst several routes?
>
> It could be that I have the wrong idea of what Camel is meant to do. All of
> the examples I've seen deal with short routes from some system to some other
> system where it seems the assumption is made that everything is up all the
> time. That is the nature of examples. When dealing with Enterprise
> Integrations (well at least the integration I deal with), the biggest issue
> is always data integrity during connection failures. It could be that all
> this transactional nature and data integrity is in Camel, I just don't see
> it. It is also a valid answer to say that I need to "roll my own" system,
> but that lessens the usefulness of Camel as an "Enterprise" tool.
>
> BTW, if we can figure all this out with Camel, then I'll be happy to add an
> example to the collection that shows how to setup message "integrity".
> -p
>
>
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> A lot of questions. Let me give a start and try answering a couple of
>> those.
>>
>>
>>
>> On Thu, Nov 26, 2009 at 1:42 AM, Pete Mueller <p...@routecloud.com> wrote:
>>>
>>> Does any one have any documentation on the lifecycle of a camel component
>>> and
>>> it's associated entities (Endpoint,
>>> Consumer, Producer). I am attempting to write a component for a
>>> event-driving wireline protocol. I understand the basics of:
>>> 0. Component is created.
>>> 1. URI in DSL gets passed to to component.
>>> 2. Component creates Endpoint for each unique URI (if using a singleton)
>>> 3. Endpoint creates Consumers and Producers.
>>>
>>> But I'm looking for a little more detail as to what triggers the creation
>>> and removal of say Consumers. Specifically,
>>> - Is a single Consumer around for the life of the CamelContext?
>>
>> Consumers and Producers are in reality more dynamic as you can have
>> create a consumer, use it once and then discard it.
>> CamelContext does not hold a repository for all created
>> consumers/producers etc. As you can create then without involvement
>> from CamelContext.
>>
>> On the other hand when a consumer is used as a input to a route, then
>> that consumer is around for the life of that route.
>>
>>
>>> - Does an Endpoint get notified if a Consumer it created is stop()'d?
>>
>> No there is no such callback. Camel have a very liberal and light API
>> for Endpoint, Consumer, Producer so its easy to work with an implement
>> custom components etc.
>>
>> There is a event API added in Camel 2.1 which have notifications when
>> routes are started/stopped etc. This can be used in case you need to
>> do some custom stuff.
>>
>>> - What is Consumer.stop() used for? Only during shutdown of the
>>> CamelContext? Or is it more like a "pause" than a stop?
>>
>> Both. You can stop a route at runtime. But when Camel shutdown it will
>> go through all routes, components, endpoints, services etc. Basically
>> all what it has and stop them one by one.
>>
>>
>>> - Is there a way (say in the case of a loss of network connection) to
>>> stop()
>>> an Endpoint and all associated Consumers and Producers? Then later
>>> restart
>>> them after connection has been regained?
>>
>> Yeah you can for example use Camel error handling and stop a route in
>> case of an ConnectionException or what you like.
>>
>>
>> In Camel 2.1 you can use RoutePolicy to that as well. However you must
>> implement some logic yourself that monitors that particular connection
>> and can start/stop the route depending on its state. Camel does not do
>> that for you.
>>
>>
>>> - Is there a general example/recommendation on how to handle connections
>>> to
>>> unreliable external entities?
>>>
>>
>> No not 2 business are the same. Some wants to self heal and just try
>> at next interval. Others want to try 5 times with X delays and then
>> alert someone if still a problem.
>>
>>
>>> I've been using the IRC component as somewhat of a model, but my glance
>>> through the code doesn't really indicate how the component deals with
>>> connection loss or net-splits and other networking pit-falls.
>>>
>>
>> Again connectivity is not that easy to encompass into a single model
>> for dealing with lost connections.
>> Often you only got the wacky java Exceptions that may or may not
>> indicate a io/connection problem.
>>
>> However when you have a Camel consumer = input to a route then you got
>> a chicken egg problem as Camel does not take over
>> before the consumer has successfully got a message and created an
>> Exchange to be routed in Camel. At this point Camel take over and you
>> can use Camel error handling. But before that all error handling etc.
>> is component specify.
>>
>>
>> We love contributions so dig into it and help out.
>> http://camel.apache.org/contributing.html
>>
>>
>>> Here's a short outline of the wire protocol I have to implement if it
>>> helps:
>>>
>>> 1. Open TWO TCP connections with Server
>>> 2. Register Myself with Server, twice, once as a SENDER, once as a
>>> RECEIVER.
>>> Both connections are bi-directional, but the functions I can do on each
>>> connection is limited by how I register myself.
>>> 3. At this point I will receive data packets (messages) for some length
>>> of
>>> time over the Receiver connection.
>>> 4. At some point in the future the Server will send me an "ALL DONE"
>>> message. At which point I must un-register my sender and receiver, but
>>> the
>>> TCP connections remail open.
>>> 5. After the unregister I need to notify Consumers that there is no more
>>> messages coming and stop Producers from attempting to send more messages.
>>> 6. After waiting X seconds (X in most cases is 180) I will attempt to
>>> re-register myself and if successful need to re-activate Consumers and
>>> Producers.
>>> 7. I also need to account for the fact that at any time, I may lose a
>>> network connection to the Server, and will need to stop all Consumers and
>>> Producers until connection is restored.
>>>
>>> Thanks for any help
>>> -p
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Lifecycle-of-a-Camel-Component-tp26522808p26522808.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context:
> http://old.nabble.com/Lifecycle-of-a-Camel-Component-tp26522808p26535877.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
