I'll jump in and try to clarify a few things. From the subject of this tread, it looks like your not quite sure if you're using Camel in the intended way. You already got a few good suggestions. In particular, Ashwin's 1b is what I would have recommended myself too. Yet, Denis Krizanovic's suggestion seems to resonate with you more. I am not sure what Denis' experience with Camel is, he only posted a question once in the past on this users@ list. That said, Camel, like most of the other technologies you use, is just a tool and it's up to you to decide if it's the right tool for the job or not. I also understand your frustrations, some things seem overly complicated in Camel. I myself share a part of that frustration and agree that some things could be simpler. However, there is a reason why Camel works the way it does (with an emphasis on "works"), but if it doesn't fit your needs, you may want to look at other choices.

Camel prides itself in simplifying integration. That is for the larger integration application, which, in my experience, is pretty much any serious application nowadays. It does so in two main ways: 1. Hiding/abstracting the underlying communication protocols and allow one to use a uniform api (hence the 1b. proposal that is usually my first choice for scenarios like the one you describe) 2. Probably even more important, is the use of EIPs that allow developers to convey the intent behind the integration scenario (plus they are already implemented and come ootb with the Camel distro). What I mean by that is that a "filter(expression)" is (at least imo) far more suggestive to those who *read* the code than an "if (condition)" statement, especially coupled with other things from "setBody" to "choice" or "routingSlip". It tells readers 'what' you're trying to do, instead of 'how'. Then there are other goodies like TypeConverters, support for multiple languages, etc.

Another set of integration requirements that Camel excels at fulfilling are related to change. That is not just the initial implementation of an integration application, but the way the application is operated and maintained and grows in time to address new requirements. All this comes down to costs and risks and Camel is very good at adapting to such needs and make things *look* quite simple (although under the hood, as you correctly mentioned, it's quite complicated). Easily adapting to change is a priority for Camel. Matt touched on that too. (Looks like that may not be a concern for you though).

So... I am not sure I can offer more concrete advice, but:
1. If the only useful thing out of Camel is using the camel:dot diagrams then it may not be "proper usage of Camel" 2. If there are many couplings between processors, and got into "header hell" you may have a bigger problem with the architecture/design of your app. That thing won't scale no matter what you do and that may not even be your biggest problem. 3. The "direct" component connections are actually synchronous in-memory calls and as such, quite efficient. However if you are using many of those, again, I would revisit the design. 4. Testing requires a bit of experience indeed, but it's actually not that hard and once you get used do it everything starts to make more sense.

5. A bit of an aside, addressing Denis' analogy with the String manipulation calls. If the String in question is a search query, than having its processing by the search engine distributed in the Cloud using a bunch of algorithms and indexes will be a very complex task, although both the input and the output are fairly simple (and the tool for that kind of jobs would be Apache Lucene, not Camel). My only point, again, is that Camel may or may not be the tool you want and that is up to you to decide. The choice has a subjective component too.

Good luck with your choice, thanks for trying Camel and if you have any other questions in the future, feel free to ask the community on this list.

I hope this helps, at least a tiny bit,
Hadrian




On 02/12/2012 05:50 PM, Grzegorz Borkowski wrote:
Ioannis, Denis,

Thanks for you input.
What you say is exactly the same what I think now, that we tried to
"overuse" Camel. We actually used it as a workflow engine for the message
processing, instead of integration engine. Instead of using Java
if-then-else logic inside a processor, we used Camel routers, end that's
why we ended with 20 processors, where in fact it should be definitely less
than 5.
For the questions about throughput and external system outages: the load is
really small in this system, and it's very unlikely to grow to the point
where it's of some concern for us. When the systems B or C are down, we
return immediate error message to the system A, but we also try to
reprocess the message after some time (we use timers which trigger
reprocessing).

W dniu 12 lutego 2012 22:34 użytkownik Denis Krizanovic<
denis.krizano...@gmail.com>  napisał:

If all your doing is using an API, and you don't really care what is
happening underneath, then I can see no need for Camel.

It would be the equivalent of trying to use Camel to do a series of String
manipulation calls. For all you know, the String class makes remote calls
over proprietary Oracle protocols to their Cloud platform to offer you the
result of substring. Of course it doesn't, but the point is, the API
ensures you don't care, just like in your case.

Seems to me that you're just importing a lot of conceptual overhead, (ie
having to understand Camel) for no reason.

regards,
dk-

ps. What is questionable in this case, is why someone would bother writing
a wrapper library these days.


On 10 February 2012 22:34, Grzegorz Borkowski<grzegorz...@gmail.com>
wrote:

Hi All,

In our project we've been using Camel for a year, but with very mixed
feelings. I'd like to ask you if the way we use it is correct - maybe our
problems come from the wrong usage of the tool. Camel experts, please
share
your opinions on this. Below you'll find description of the project.


Out project is an integration tool, integrating 3 other systems, call
them
A, B and C. We receive the message from system A, query for some data
from
system B and send the message to system C. Camel seemed to be ideal tool
for such scenario.
In practice, there are couple of problems though.

1. It turned out that we don't use any typical communication channels to
integrate with the systems. All those systems provided us with a library
(jar) which is simply a stub, which presents us with an API to call
remote
systems synchronously. Even if those stubs use JMS or other channels
underneath, we don't interact with them directly. The only thing we do,
is
just calling simple synchronous Java method on the stub. Something
similar
to the old Java RMI. Thus, the whole power of Camel components is of no
value for us.

2. There is one place where we use Camel JMS component - we have a test
input, for receiving test data, which uses JMS communication. So
currently
this is the only place in the code we use Camel components for real
integration with external systems. We're also now developing additional
interface, and this will be the second entry point based on JMS. So those
are only places when we use Camel capabilities.

3. Inside the application, we have a workflow build on the Camel routers,
predicates, and processors. All of them are glued by the "direct"
components. So when XML message comes, we pass it to a router, which
passes
it to a proper processor, then to next router etc. There are around 20
processors and 10 routers in the flow. In practice however, we see only
one
advantage of using Camel for this: ability to generate visual diagram of
the flow (using camel:dot feature). There are many downsides though.
First,
there are many couplings between the processors, so we end up passing
tens
of headers in the messages between processors. We already call it
"header-oriented programming" or simply "header hell". It's a nightmare.
Then, it's horror to debug the code when for example the router passed
the
message to the wrong path. If it was a normal java code written by us,
with
"if then else", we could just put a breakpoint there and debug it. But in
camel, if you configure routers using the built-in predicates, like:
from(...).choide().when(header(..).isEqualsTo(...)).to(...) - then you're
not able to debug it easily, because the if-then-else logic at runtime is
performed by camel engine, not our code.

4. Using the "direct" component connections everywhere seems to be
extremely inefficient. Say I have a flow: message goes through 2 routers
and two processors, linked by direct components. If you generate stack
trace at the end of this path, you will see literally hundreds of camel
calls. Where do they come from? It's especially frustrating when you try
to
debug it - it's infeasible. I hate debugging Spring proxies because they
insert about 5-10 framework method calls between my classes. Camel does
the
same, but it uses hundreds of framework calls instead. The stack trace is
littered with DelegateAsyncProcessor and AsyncProcessorHelper calls. I
wouldn't really be surprised if I saw StackOverflowError somewhere in the
flow.

5. Testing the Camel flow is not a nice experience as well. For every
test,
you have to set up those dummy endpoints, and record expected number of
incoming messages. Then, when there is an exception thrown in the flow,
instead of just breaking the test where it was thrown, we see the cryptic
message at the end "expected receiving 1 message, received 0". Then you
have to analyze the log to find what was the real reason. Also, Camel
waiting for the messages, make the tests much slower, because  time is
lost
on this waiting.


Thoughts? Recommendations? What do we do wrong?




--
Hadrian Zbarcea
Principal Software Architect
Talend, Inc
http://coders.talend.com/
http://camelbot.blogspot.com/

Reply via email to