Hi Vadim,
I am glad we are on the same page now.
That's the thing, I don't think we spent enough time to nail down all
the things your asking. We may not have the right answer or an answer
at all (yet) for what a well behaved component should do with respect
to faults. Afaik, even now, the sql component you refer to does not
distinguish between transient and persistent faults, nor the http
component. So the overloading of the server does take place already.
We do have a mechanism in place today to force treating faults as
exceptions, but for a less than trivial route I am not sure how
practical that is. As an example, the same fault thrown by different
endpoints in the route may have to be handled differently, and I am
not sure enough information is provided right now to make that
possible (maybe a new header with the url of the endpoint throwing the
fault?).
Let's use camel-316 and add all ideas there of what a well behaved
component should do with respect to faults.
Thanks for your comments and help,
Hadrian
On Sep 22, 2008, at 11:37 PM, Vadim Chekan wrote:
Hi Hadrian,
Thanks for explaining, I understand it now what you mean.
About terminology. May be we'd better stick with "transient" and
"persistent" because with "faults" and "exceptions" I'll have to
look it up email each time to remember which one is it.
I totally agree with you on having this distinguish between two
types of faults (I hope I got it right this time :). Many reties of
inserting large batch into SQL when there is foreign key violation
can overload sql server.
As I understand it, components will be responsible for deciding what
is transient and what is not. For example jdbc component can
interpret the same SQL exception class differently. If it is a
timeout or a deadlock, there is sense to try it again. But if it is
a key violation or wrong login then it is persistent.
So what are (will be) the rules for well behaved component? Perhaps
we want to keep it backward compatible and in case if exception is
not handled by component it is interpreted as transient error?
So if component decides that error is persistent then what? Set
"fault" of exchange to an exception? Or there will be a special
property in runtime exception?
Vadim.
Hadrian Zbarcea wrote:
I reluctantly pick the gauntlet. I feel your passion, and it looks
to me more like violent agreement than disagreement (at least of
the fact that there is some cleanup to be done). So let's clarify
a few things.
I am not sure how carefully you read https://issues.apache.org/activemq/browse/CAMEL-316
and the thread referenced there. I think the consensus is to
combine faults and exceptions, we just have to nail out the details.
So distinguishing fault and exception in Camel is senseless. There
is no difference between two.
I certainly understand this (your) point of view. I wish things
were as clear cut as you suggest, but I am not convinced that's the
case, on the contrary.
I'll move away from the terms fault and exception for a bit. There
are two kinds of errors: transient and persistent. Transient
errors indicate some environmental problem that prevented carrying
out the request. Persistent errors indicate something wrong with
the request. In the former case, if the caller were to issue the
request again at a later time, it may get it processed
successfully. In the latter case, no amount of retries would help.
Some protocols/technologies distinguish between the two kinds of
errors, some do not and some are in between (think http:, on a 500
you might want to retry, on a 404, 403, 401 one shouldn't!). Wsdl
for instance provides the faults. Some (architects) use them to
model the persistent errors a system may return, some do not. What
is the intended audience for faults? The caller. From this point
of view yes, camel, which is part of the infrastructure should not
care about faults, they are an output like any other, which happens
to be an exception, which should be sent back to the caller to take
appropriate action. You might say that from a Camel perspective a
Fault is *not* an Exception, it's a valid reply to be sent back to
the caller.
You ask "What "not in the way intended" mean?", well it means that
on a financial transfer for instance, you may get an "insufficient
funds" fault that indicates that the server got the request and for
the reason provided decided not to carry out the request in the
intended way, i.e. to actually complete the transfer and no amount
of retries would change that until somebody opens the purse and
deposits more funds in the said account. More precisely for me
"the way intended" means such as to get a valid response/output,
not a fault. That is more often than not, when requests are sent,
it is done with the intention of receiving a valid output, not a
fault (which may not be true for a test application, but I think
you got the point, sorry for not being more clear in my past post).
Good infrastructure relieves the caller as much as possible of
dealing with transient errors taking the appropriate measures
(retries, whatever), we have the ErrorHandler for that. Great
infrastructure allows one to deal with (at least some of the)
persistent errors. Why? Because in many cases they way to handle
them is the same, so that could be automated. So in that case you
could see the Fault as *being* an Exception and the infrastructure
could use predefined rules to handle them.
The issue is not if a Fault is of a java.lang.Exception type (I
think there was a comment stating that: of course Faults are
Exceptions, meaning of java.lang.Exception type). It may very well
be. From a web services framework point of view, they can be
modeled as of java.lang.Exception type. But Camel is a router, the
question is if and how should Camel handle Faults. My point is
that I would prefer to not lose the semantical distinction between
fault and exception, always react to exceptions and optionally
react to fault (and we do some of that).
As far as implementation goes, I could see faults going as outputs
(and being of java.lang.Exception type) so one can have rules to
handle them *if* desired. Or we can a header saying that this
output is actually a fault, so handling could be based on that.
(That means that in my mind a Fault is more of an output than an
Exception in Camel Exchange sense).
To your second point about In/Out. The In is always read-only.
One never modifies the in, this is always the message that was sent
by the producer. It's history, doesn't change. Out is always the
response. And there is the exchange pattern: InOnly, the Out is
always null, InOut, then well the out is null. There is that
ambiguity by convention that the next in is either last out of *if
null* the last in. This should be clarified indeed and produced
confusion more than once.
In your filter example, yeah, of course the result is a message,
but think of the exchange as having the whole context, the message
before the filter, the one after, the exception (if any) and some
context/properties/headers (maybe). I am not sure if a null output
should mean "stop" the pipeline. For an InOut exchange patter,
maybe, although I doubt this happens a lot in practice. For an
InOnly pattern (such as log, wiretap, etc, that don't have an
output, they're not an 'echo') I see nothing wrong with doing what
we already do, which is sending the same In to the next processor.
Keep in mind that some technologies do not have a way to formalize
the exchange pattern (such as wsdl does for instance) so the only
way to distinguish between an InOnly and InOut is based on
convention, such as the Out being null.
I wholeheartedly agree that this (more or less) edge cases should
be properly clarified and documented, and I hope you'll keep the
ball rolling, so we could hopefully get it done in 1.5. I don't
think it's too hard to do, but one, we need to agree on how to do
it and two, minimize the impact of the api change. (I think it's
reasonable tough to keep the api, deprecate and never use the fault
for 1.5).
Thoughts?
Hadrian
On Sep 14, 2008, at 5:23 PM, Vadim Chekan wrote:
I even didn't know about FAULT :(
Does it mean that all my processors are buggy because I do not
check fault?
My understanding of Fault in SOAP is that *it is* an exception. WS
frameworks generate proxy classes which check fault and throw
exception. The reason Soap use fault is that it is the only way
you can serialize an exception.
The same approach on server side: any business exception is caught
by WS handler, wrapped into fault code and sent back.
So distinguishing fault and exception in Camel is senseless. There
is no difference between two.
Fault code is nothing else but serialized exception.
I'm afraid I can't agree with Hadrian:
> Outputs provide a response after
> successful processing. Faults are responses that indicate to the
> client that the expected goal was not achieved, and they usually
> provide a reason. Faults are messages that tell a client that the
> server did receive the request and successfully processed it,
but not
> in the way intended and the client may need to take further action
What "not in the way intended" mean? What should Camel do if
something went not in the way intended? The answer is: nothing.
Camel knows nothing about application business stuff. From Camel
perspective message is either processed or not. Period. It is
application which should know that certain component can return
two types of responses and act accordingly. From Camel perspective
it is a perfectly valid response which is indistinguishable from
Out.
Now, lets talk use cases. Lets say we have a batch to process.
Like any programmer I'm all about ACID so I design my system to
rollback and fail the whole batch if any item failed.
But my users are unhappy. They come to me and say that if I just
create a list of errors and keep batch executing they would rather
fix errors manually instead of cleaning source data and re-running
the batch.
So now I'm facing dilemma. If before knew that if no exception
happen then batch succeeded but now I have a "failed" response
with list of failed items. So I can not throw exception anymore. I
need to process this "failed" list by publishing it.
But it is very important to understand that from service design
point of view there was no errors! I got a response with a list of
items which need to be published. Sounds like typical workflow. It
is error only from customer perspective, not from the Camel or
even application perspective!
I provided this long example because I suspect that this "fault"
conception is interpretation of application-level faults. I want
to show that there exist no "fault" message as opposite to
exception. If you have a use case which demonstrates "fault"
message, lets discuss.
In/Out
==============
I must admit I had some confusion grasping "exchange" conception.
The word exchange assumes that there is 2-directional flow between
2 entities. But Filter has only one direction: from one component
to another. So exchange between what and what is happening?
In/Out is perspective of component. The component is given In
message and must provide the Out message. But it is not "exchange"
it is rather replacing In with Out.
Now, on practical side, there is no need to provide both In and
Out at the same time. We can assume that component will return its
answer in the same object. But I'm failing to see any practical
benefits. IMHO it is the same, just matter of variable allocation.
I think Roman's (and mine) frustration comes not from the fact
that there are in/out but from the fact that they are too relaxed
and it is not clear when In is Out :) So stricter rules should
address this concern.
I have implemented a small message routing system for internal
needs at works and I used Soap idea of Envelope with properties
and payload - body. So I have no Exchange entity as such. All
service information, like number of faults, etc is in the envelope
header. The processor function itself is an exchange:
"public MessageEnvelope Process(MessageEnvelope env);"
Later I faced a need to implement a splitter which produces more
then one message so I modified interface to return array of
messages:
"public MessageEnvelope[] Process(MessageEnvelope env);"
In Camel there is a special type of processor to handle this. In
my design any component can do broadcast by returning multiple
messages.
You can do all the things you do in camel in/out exchange.
You can return null which would mean "no outputs, stop
processing"; you can return exact the same object you received for
logging for example; you can create a copy of object, modify it
and return; you can create a bunch of new messages and return then.
In/Out notation is pretty much the same as described above but
ability to modify In message and leave Out null brings
uncertainty. Should I modify In or create Out? Why the heck In
work as Out? Does componentX use In or Out? etc.
I would think about making rules more strict: result is always in
Out. Providing many ways of doing the same thing without benefits
is a bad idea. You'll spend infinite amount of time explaining
newbies what in/out thing is. Principle of the least surprise :)
Another note about HTTP request/response.
I think it has nothing to do with Camel. Camel In/Out implements
conception of filter. You have a message asking to convert X $USD
to EUR. Your component use http to do it. So what the output of
your component is, http response? I doubt it. It is a message with
EUR amount. The whole http request-response is internal business
of component. Http request-response in an Exchange, when Camel
Filter is not.
So my opinion is:
1. Faults must be removed and exceptions used instead like any WS
framework does.
2. "In" must be In and Out must be Out, meaning In can not be
modified and Out *must* be set; no tricks "if out is null then
interpret In as Out". We should agree what to do with "null" out.
Is it an error or it means "no messages to process, stop the pipe"?
Vadim.
P.S. Great discussion btw.
Claus Ibsen wrote:
Hi
Just a quick mail to get the discussion rolling.
I think we should @deprecated exchange.fault in Camel 1.5 and
remove it in Camel 2.0.
It is a confusing concept that isn't used much, and many
components doesn't handle the FAULT message. For instance it was
missing in camel-jms a very central component.
Med venlig hilsen
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
