Routing new data from a DB query, updates from JMS
A quick rundown of our situation...Our system operates in a very dynamic environment where accessing the data to route will have to be done in two 'phases': 1. Perform an initial query to a database(s) and route any new records 2. Listen for updates pushed to a JMS Queue and route I have the book and have been reading as much as I can but I can't figure out how the functionality of querying for new data should best be handled. One thought is to have a component (outside of Camel) that queries for data and pushes to a JMS Queue...at that point Camel could take it from there (using the JMS Queue as a 'from' in a route). My other thought is to integrate this query logic in a Camel route but I get really lost when I think of the best way to do it. Maybe I could use a JPA component to get new data and at the same time have a route that pulls from a JMS Queue? It seems that Camel is exactly what we need for the routing/filtering capabilities but I'm having serious problems in figuring out how to get the data to the Camel infrastructure. Any help/advice would be greatly appreciated! -- View this message in context: http://old.nabble.com/Routing-new-data-from-a-DB-query%2C-updates-from-JMS-tp28893315p28893315.html Sent from the Camel - Users mailing list archive at Nabble.com.
Help with filter behavior
I am trying to build a route using Java DSL and I'm getting a confusing error
in my IDE (IntelliJ). Here's what I'm trying to do:
1. Receive a message from a JMS Queue
2. Enrich the message by querying a database to retrieve additional content
(claim check pattern)
3. Filter the messages to exclude any 'test' messages
4. Print to the console
Here's my Java DSL:
public class MyRouteBuilder extends RouteBuilder {
public void configure() throws Exception {
from("jms:xmlOrders")
.bean(enricher, "enrichData")
.filter().xpath("/order[not(@test)]")
.to("stream:out"); // ERROR on this line in IDE
}
}
The IDE highlights the last line (.to("stream:out");) as an error and I'm
having a hard time understanding why. If I remove the bean doing the
enrichment it works fine. ie:
from("jms:xmlOrders")
//.bean(enricher, "enrichData") COMMENTED OUT
.filter().xpath("/order[not(@test)]")
.to("stream:out"); // NOW everything is fine
I also tried replacing the bean with a new Processor that does the same
thing in case the issue has to do with keeping the Exchange in tact but it
still didn't make the error go away.
I'm sure this is how it's supposed to work and is due to a lack of my
understanding. I would be grateful if anyone could help me understand why
this is happening!
--
View this message in context:
http://camel.465427.n5.nabble.com/Help-with-filter-behavior-tp2256745p2256745.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Help with filter behavior
I was using Camel 2.3. I switched to version 2.4 as you recommended and the
error highlighting went away.
With version 2.4, it now works for '.filter().xpath("/order[not(@test)]")'
and also for Claus's suggestion of '.filter(xpath("/order[not(@test)]"))'.
Before trying the 2.4 version, I put the xpath logic in a custom Processor
which updates the header based on the xpath expression evaluations (of which
there could be many). Based on the header, the route will send it
accordingly.
I wanted to use the built-in filtering/evaluation support in Camel so I may
come back and put the filtering in the route itself (instead of using my
Processor). Camel is so flexible that I find it difficult to decide when to
use one approach over another!
I really appreciate the help!
--
View this message in context:
http://camel.465427.n5.nabble.com/Help-with-filter-behavior-tp2256745p2261890.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Help with filter behavior
Claus- Thanks for taking the time to respond!
I upgraded to 2.4 and my issue is resolved but just to follow-up, I tried
your suggestion of using '.filter(xpath("/order[not(@test)]"))' but it still
didn't work with 2.3.
I'm seeing that using Java to define the routes is limiting but the
application we're creating allows on-the-fly route building/configuration
so, maybe mistakenly, I saw this as the best approach. The only alternative
I see is to create the routes manually (building routes using
RouteDefinition, ProcessorDefinition, etc). Would love to hear any thoughts
you may have on this...
--
View this message in context:
http://camel.465427.n5.nabble.com/Help-with-filter-behavior-tp2256745p2261902.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Odd route start behavior after context is started
I'm seeing an issue when starting routes (pulling from a multiple-consumer SEDA Queue) after the context has been started. After the 3rd route is started, the 3rd route does not receive most of the data while the other two routes receive all the data. Am I doing the wrong thing by adding routes after the Camel Context has been started? I was under the impression that I could start/stop routes after the context starts so it must have something to do with my implementation. I created a simple program to illustrate the problems observed where each route has a processor that simply prints the message to the screen and keeps a count of the messages it has seen. Here's a summary of what I'm doing: 1. Start a route that takes data off a SEDA Queue and send to a SEDA multiple-consumer Queue 2. Start route 'r1' that pulls data from the multiple consumer queue 3. Send 10 messages. RESULT: Route 'r1' receives all 10 messages 4. Start route 'r2' that pulls data from the mutliple consumer queue 5. Send 10 messages. RESULT: Route 'r1' and 'r2' each receive all 10 messages 6. Start route 'r3' that pulls data from the multiple consumer queue 7. Send 10 messages RESULT: Routes 'r1' and 'r2' each receive all 10 message, 'r3' only receives 1 message To summarize, after adding three routes over a period of time, the last route added does not receive most of the data. The odd thing is that if I start all the routes after starting the context and send them data, all the routes get all messages. It's definitely something to do with the routes being started sequentially...is it something I'm doing wrong? Thanks, Glenn -- View this message in context: http://camel.465427.n5.nabble.com/Odd-route-start-behavior-after-context-is-started-tp2835464p2835464.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Odd route start behavior after context is started
http://camel.465427.n5.nabble.com/file/n2835470/Simple_Route_Test.jpg Sorry for the bad diagram but I just wanted to convey the configuration. The route that takes data from the SEDA Queue on the left and sends to the SEDA Queue on the right isn't necessary but I wanted to simulate one of the routes in our code. -- View this message in context: http://camel.465427.n5.nabble.com/Odd-route-start-behavior-after-context-is-started-tp2835464p2835470.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Odd route start behavior after context is started
If I replace the multiple-consumer SEDA Queue with a JMS Topic, everything works fine...ie: All three routes get the data. Is there a known problem with dynamically adding consumers to a multiple-consumer SEDA Queue? I need to do some more digging, but I turned on the Camel debug logging and saw some odd behavior with the size of the thread pool. After adding the 3rd route, it kept switching from a thread pool of size 1 to size 3. I don't know if that had anything to do with the issue I was seeing but it definitely stood out. I'll keep looking into it but if anyone has any insight I would be grateful! -- View this message in context: http://camel.465427.n5.nabble.com/Odd-route-start-behavior-after-context-is-started-tp2835464p2838501.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: How to implement (dynamic) rule engine with camel.
Naira & Kobo wrote: > > This means, if I have endpoints A, B, C, D, E, F > The first use case can be: A -> D -> E -> F > The second use case: A -> B -> C -> F > Adding the second use case should require a restart of the camel > application. > Does Camel need to be restarted to switch between the first use case and the second? I thought this could be done programmatically? Maybe the difference is that you are wanting to do it via a scripting file... -- View this message in context: http://camel.465427.n5.nabble.com/How-to-implement-dynamic-rule-engine-with-camel-tp2837427p2838590.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: How to implement (dynamic) rule engine with camel.
Claus Ibsen-2 wrote: > > Hi > > The dynamic router EIP pattern > http://camel.apache.org/dynamic-router.html > > And you can add/remove routes at runtime using the CamelContext API. > > Nice! This will probably work for the issue I posted about also! -- View this message in context: http://camel.465427.n5.nabble.com/How-to-implement-dynamic-rule-engine-with-camel-tp2837427p2838592.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Odd route start behavior after context is started
Thanks! I'll use the JMS Topic as a temporary workaround until this is fixed. If this doesn't make it in the 2.5 release, I'll try using the DynamicRouter pattern. -- View this message in context: http://camel.465427.n5.nabble.com/Odd-route-start-behavior-after-context-is-started-tp2835464p2839640.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Odd route start behavior after context is started
Many thanks! I'll try it out in the next couple of days and reply to confirm. -- View this message in context: http://camel.465427.n5.nabble.com/Odd-route-start-behavior-after-context-is-started-tp2835464p2850015.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Odd route start behavior after context is started
I downloaded the 2.5 Snapshot jar file and ran through a series of unit tests it was previously failing...now everything seems to be working as expected. I can add/remove routes from the CamelContext during runtime which I couldn't do with 2.4. Thanks for the fix! We will be using this snapshot jar until the official 2.5 release, which hopefully will be in November. In the unit test, I incrementally add routes and in between adding, I send 10 or more messages to make sure each route gets the messages. Then I stop a route, send data, and then re-add the route and send more data. -- View this message in context: http://camel.465427.n5.nabble.com/Odd-route-start-behavior-after-context-is-started-tp2835464p283.html Sent from the Camel - Users mailing list archive at Nabble.com.
File component not working with 2.5 snapshot
I posted on the developer forum (under the thread about JIRA 3223) but it
appears my post didn't get submitted or something. I think this bug is
related to what I'm seeing but it's more severe than what the JIRA ticket
describes.
Anyway, I'm having issues using the File component. Thinking it was my
code, I created a very simple route based on an example:
from("file://inputdir/?delete=true").to("file://output")
When I drop a file into the directory, it doesn't get processed and I get a
loop of exceptions...here's a portion of what I see:
13:41:15,150 ERROR [GenericFileOnCompletion] Caused by:
[org.apache.camel.component.file.GenericFileOperationFailedException -
Cannot store file: output\classpath.sh]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
store file: output\classpath.sh
at
org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:215)
at
org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:222)
at
org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:138)
at
org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:59)
at
org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:50)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
at
org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:104)
at
org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:272)
at
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:98)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:99)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:297)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:206)
at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:256)
at
org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:99)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:70)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:68)
at
org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:322)
at
org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:155)
at
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:121)
at
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:97)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.camel.InvalidPayloadException: No body available of
type: java.io.InputStream but has value: GenericFile[classpath.sh] of type:
org.apache.camel.component.file.GenericFile on: Message:
GenericFile[classpath
Re: File component not working with 2.5 snapshot
Interesting...here's what I'm seeing: INFO [org.apache.camel.impl.converter.AnnotationTypeConverterLoader] (main) Found 4 packages with 0 @Converter classes to load INFO [org.apache.camel.impl.converter.DefaultTypeConverter] (main) Loaded 0 type converters in 0.032 seconds Obviously something is wrong with my environment. Any ideas of where I need to start looking? I haven't used the File component in a couple months but when I used it last it worked. I'm using the 2.5 snapshot release. Thanks, Glenn Willem.Jiang wrote: > > > Here is is mine > > Oct 14, 2010 10:34:31 PM > org.apache.camel.impl.converter.AnnotationTypeConverterLoader load > INFO: Found 3 packages with 13 @Converter classes to load > Oct 14, 2010 10:34:31 PM > org.apache.camel.impl.converter.DefaultTypeConverter loadTypeConverters > INFO: Loaded 146 type converters in 0.429 seconds > > -- View this message in context: http://camel.465427.n5.nabble.com/File-component-not-working-with-2-5-snapshot-tp3212019p3218428.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File component not working with 2.5 snapshot
I'm running Java 1.6.0_18-b07 on Windows 7 and creating the Camel Context in Spring (using IntelliJ as my IDE). I downloaded the 2.5 snapshot about a week ago and was using that and just (15 minutes ago) downloaded the latest 2.5 snapshot from the link provided. Here's my log: 2010-10-19 10:05:51,156 INFO [org.apache.camel.impl.converter.AnnotationTypeConverterLoader] (main) Found 23 packages with 0 @Converter classes to load 2010-10-19 10:05:51,177 INFO [org.apache.camel.impl.converter.DefaultTypeConverter] (main) Loaded 0 type converters in 0.082 seconds Previously I was selectively including jar files from the Camel 2.5 snapshot but am now including all jar files for troubleshooting purposes. I'm using all jar files in the 'lib' directory and in the 'spring' directory under the snapshot release. My next step is to create a fresh project and running this single route (file copy from input directory to output directory) to see if the issue is due to my environment for some reason. It just seems odd to me that I'm not picking up any converters...maybe I'm doing something in the code that is clearing them. -- View this message in context: http://camel.465427.n5.nabble.com/File-component-not-working-with-2-5-snapshot-tp3212019p3219190.html Sent from the Camel - Users mailing list archive at Nabble.com.
Custom idempotent repository?
I see the 5 different types of idempotent repositories that are currently provided by Camel but is it possible to create a custom repository? I don't see anything preventing it but then again I'm not seeing anything about creating a custom solution. There are a couple options being discussed for how to handle duplicate messages including basing the logic on what is currently stored in our database. I'm going to attempt a quick proof-in-concept of using a different idempotent repository but thought I would ask in case it's a simple yes/no answer. Thanks! -- View this message in context: http://camel.465427.n5.nabble.com/Custom-idempotent-repository-tp4925593p4925593.html Sent from the Camel - Users mailing list archive at Nabble.com.
Inconsistent results with File2 'readLock=changed'
Using Camel 2.5 in JBoss 5.1. When dropping large files (over 200MB) into a directory I expected that the Exchange wouldn't kick off until the file had completed copying. More often than not, an Exchange is created before it finishes copying. We're using the 'readLock=changed' option btw. Is the reason that it's processing the file before it's done is that the file size doesn't change over a 1 second interval? This has to be the case but I just wanted to confirm. I might try upgrading to 2.6 or later to see if the 'readLockCheckInterval' option might resolve the inconsistencies. Thoughts? -- View this message in context: http://camel.465427.n5.nabble.com/Inconsistent-results-with-File2-readLock-changed-tp5030004p5030004.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Inconsistent results with File2 'readLock=changed'
I upgraded my dev box to version 2.8.3 so I could use the readLockCheckInterval option and set it to 3 seconds. The Exchange still fires off even though the file hasn't completely copied. Details: 1) My custom Idempotent Repository gets queried as soon as I initiate the file copy (OK) 2) 3 seconds later the incoming Exchange is logged and processes even though the 1GB file is not completely copied. In the Windows Explorer 'details' view, the file size shows 1GB even though only maybe 200MB have been transferred. It seems like for my case increasing the readLockCheckInterval doesn't help b/c Windows reports the final file size even though it hasn't finished copying. From searching (and from documentation in the Camel File component), it's a common problem but I've never run into it before. I'm going to look at the code from the current changed read lock as you suggested to see if I can figure something out for our scenario. We will have write permissions for the file so maybe I can determine file completion that way. I'll post back on the status. This issue appears when I'm copying between local directories and from a network drive btw. -- View this message in context: http://camel.465427.n5.nabble.com/Inconsistent-results-with-File2-readLock-changed-tp5030004p5033493.html Sent from the Camel - Users mailing list archive at Nabble.com.
