Re: How to insert Date String as date in MongoDB
I did a little more playing around. Discovered and verified a few different
things. First, I recall reading somewhere that the type mapping is being
cached (I think I read that). So when I played around with the JSON payload
being sent in I started to get mixed results and what was previously not
throwing any errors was now throwing a conversion error. I reloaded the
bundle, went back to the JSON that we agreed upon, and no more errors. So I
suspect that caching of the type conversion is happening somewhere (just a
hunch based on my observations though).
More importantly, when I changed the JSON to this:
/{ "time" : { "$date" : "2016-02-19T17:37:57.673Z" } }/
replacing the /+/ offset with /Z/, the document actually persisted
correctly as a DateTime type. Very cool. Now I need to figure out how to
manipulate the JSON in Spring DSL from /ISODate(...)/ (which is how the
callers will be sending the payload in to the route) to /{ "$date" : "...Z"
}/ so that I can put this piece to rest.
Does any of that make sense to you?
--
View this message in context:
http://camel.465427.n5.nabble.com/How-to-insert-Date-String-as-date-in-MongoDB-tp5778310p5779103.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: How to insert Date String as date in MongoDB
Thanks for confirming. I tested the logic and while there are no errors, the value of the /time /property is being persisted as /null /in Mongo. I am going to take a look at the converter classes to see if I can figure out how that is possible. Any thoughts? Again, thanks for the help on this! -- View this message in context: http://camel.465427.n5.nabble.com/How-to-insert-Date-String-as-date-in-MongoDB-tp5778310p5779102.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: How to insert Date String as date in MongoDB
Yes I can try that. To make sure I am on the same page is this what you are
asking for?
*Route:*
/
/
*JSON to send:*
/{ "time" : { "$date" : "2016-02-19T17:37:57.673+" } }/
Or do think we should not perform the convertBodyTo string?
Thank you for the quick response! I seriously appreciate the attention.
--
View this message in context:
http://camel.465427.n5.nabble.com/How-to-insert-Date-String-as-date-in-MongoDB-tp5778310p5779043.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: How to insert Date String as date in MongoDB
Any thoughts on my last post as to how we can use ISODates with camel-mongo? Thanks! -- View this message in context: http://camel.465427.n5.nabble.com/How-to-insert-Date-String-as-date-in-MongoDB-tp5778310p5779035.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: How to insert Date String as date in MongoDB
Thank you for the response and information. I believe I tried what is
recommended in the link that you provided. Below is a copy from the Logs via
the Karaf console. I hope the formatting is not too terrible. I can try and
provide it another way if necessary.
Pretty much the same result as I was seeing before as I believe I already
tried this approach. Here is the document that I used: /{ "time" : { "$date"
: "2016-02-19T17:37:57.673+" } }/
Here is what is going on in the route (the reason for the tracing is that I
wanted to see what type the body was being converted to and from...I was
toying around with different conversions to see if I could come up with some
magic incantation that would work):
/
/
And the jsonMarshaller dataformat definition is this:
/
/
And the Mongo bean definition:
/
/
Hope this information is helpful. I certainly appreciate someone looking
into this. We are not Java developers here, but at the moment it feels like
my next course of action is to write some Java code to customize the
serializer.
*From the Logs available via the Karaf Console:*
*3/4/2016, 4:29:33 PM WARN* /Execution of JMS message listener failed.
Caused by: [org.apache.camel.RuntimeCamelException -
org.apache.camel.component.mongodb.CamelMongoDbException: MongoDB operation
= insert, Body is not conversible to type DBObject nor List]
camel-core org.apache.camel.RuntimeCamelException:
org.apache.camel.component.mongodb.CamelMongoDbException: MongoDB operation
= insert, Body is not conversible to type DBObject nor List/
*3/4/2016, 4:29:33 PM ERROR* /Failed delivery for (MessageId:
ID:devvlapifuse01-42307-1456773862280-9:8:3:1:1 on ExchangeId:
ID-devvlapifuse01-49834-1456773865165-1-9). Exhausted after delivery
attempt: 1 caught: org.apache.camel.component.mongodb.CamelMongoDbException:
MongoDB operation = insert, Body is not conversible to type DBObject nor
List Message History
---
RouteId ProcessorId Processor Elapsed (ms) [oec.common.logger.]
[oec.common.logger.] [activemq://queue:oec.common.logger.elmah.request ] [
2] [oec.common.logger.] [validate4 ]
[validate[simple{${in.header.MongoDatabase}}] ] [ 0] [oec.common.logger.]
[validate5 ] [validate[simple{${in.header.MongoCollection}}] ] [ 0]
[oec.common.logger.] [log2 ] [log ] [ 0] [oec.common.logger.]
[convertBodyTo1 ] [convertBodyTo[String] ] [ 0] [oec.common.logger.] [log3 ]
[log ] [ 1] [oec.common.logger.] [log4 ] [log ] [ 0] [oec.common.logger.]
[to1 ]
[mongodb:mongoBean?database=PMA&collection=Elmah&writeResultAsHeader=true&opera]
[ 1] Exchange
---
Exchange[ Id ID-devvlapifuse01-49834-1456773865165-1-9 ExchangePattern
InOnly Headers {Accept=*/*, Accept-Encoding=gzip, deflate,
Accept-Language=en-US,en;q=0.8, AppKey=NfQFDRu7Ogq4xF4hZpMGV0CxdgQCnDPW,
breadcrumbId=ID-devvlapifuse01-49834-1456773865165-1-8,
CamelRedelivered=false, CamelRedeliveryCounter=0,
CamelServletContextPath=/private/common/logger/elmah, Content-Length=59,
Content-Type=application/json, JMSCorrelationID=null,
JMSCorrelationIDAsBytes=null, JMSDeliveryMode=2,
JMSDestination=queue://oec.common.logger.elmah.request, JMSExpiration=0,
JMSMessageID=ID:devvlapifuse01-42307-1456773862280-9:8:3:1:1, JMSPriority=4,
JMSRedelivered=false, JMSReplyTo=null, JMSTimestamp=1457126973022,
JMSType=null, JMSXGroupID=null, JMSXUserID=null, MongoCollection=Elmah,
MongoDatabase=PMA,
Origin=chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop,
Postman-Token=fc4ab341-5fbb-fb41-d0ad-c30cc7c3c30a, TestType=2} BodyType
String Body { "time" : { "$date" : "2016-02-19T17:37:57.673+" } }
] Stacktrace
---
camel-core org.apache.camel.component.mongodb.CamelMongoDbException:
MongoDB
operation = insert, Body is not conversible to type DBObject nor
List/
*3/4/2016, 4:29:33 PM WARN* /Conversion has fallen back to generic Object ->
DBObject, but unable to convert type org.apache.camel.impl.DefaultMessage.
Returning null. camel-mongodb/
*3/4/2016, 4:29:33 PM INFO* /[TRACE] After unmarshall - { "time" : { "$date"
: "2016-02-19T17:37:57.673+" } }
camel-core /
*3/4/2016, 4:29:33 PM INFO* /[TRACE] Before unmarshall - { "time" : {
"$date" : "2016-02-19T17:37:57.673+" } }
camel-core /
--
View this message in context:
http://camel.465427.n5.nabble.com/How-to-insert-Date-String-as-date-in-MongoDB-tp5778310p5778652.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: How to insert Date String as date in MongoDB
I would be interested in knowing the answer on how to do this as well. I posted in a different thread and the datatype I am trying to do this with is the ISODate built-in mongo type. I am using Spring DSL as well. Thanks! -- View this message in context: http://camel.465427.n5.nabble.com/How-to-insert-Date-String-as-date-in-MongoDB-tp5778310p5778555.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: MongoDB Jackson Date Mapping Option
I am wondering if there is a good option for getting ISODates or any of the
other built-in MongoDB types to work within the context of Spring. For
example, given a very simple document:
/{ "time" : ISODate("2016-02-19T17:37:57.673+") }/
How might one get the ISODate to properly work with camel-mongo? I have been
round and round with variations with no success. It appears according to the
camel-mongo docs that I can send in a string and that string will be
converted to a DBObject, but that does not appear to work with built-in
types unless I am missing something obvious.
/
/
*Error received:*
/org.apache.camel.RuntimeCamelException:
org.apache.camel.component.mongodb.CamelMongoDbException: MongoDB operation
= insert, Body is not conversible to type DBObject nor List/
I have also attempted to convert to a string and then unmarshall using the
Jackson parser, but the parser chokes on the ISODate as I suspect that it
really does not understand the mongo built-in types.
We do not code in Java and really do everything via Spring DSL. I can likely
manipulate the JSON prior to sending on the exchange to camel-mongo, but
still not sure what that manipulation might look like.
I attempted to change it to this based on my very loose understanding that
this was a possible solution:
/{ "time" : { "$date" : "2016-02-19T17:37:57.673+" }}/
But that still did not work as I got the same error as mentioned above. I
could try to get the Object --> DBObject conversion to work, but honestly I
am not sure how to get the JSON string into the form of an Object within
Spring before passing on the exchange to camel-mongo.
Any advice / suggestions are VERY much appreciated.
Thank you!
--
View this message in context:
http://camel.465427.n5.nabble.com/MongoDB-Jackson-Date-Mapping-Option-tp5727548p5778438.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Is it possible to read the contents of a file in Camel Spring DSL w/o File component?
I just discovered this article (http://www.davsclaus.com/2009/05/on-road-to-camel-20-concurrency-with.html). I think this answers my questions above. It is the that takes over the file exchange as handles the file completion piece. That tells me that the deletion of the file will happen in the message exchange and that the File component is free to pick p another file. If I am understanding that correctly, then that is exactly what i needed. -- View this message in context: http://camel.465427.n5.nabble.com/Is-it-possible-to-read-the-contents-of-a-file-in-Camel-Spring-DSL-w-o-File-component-tp5772942p5772946.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Is it possible to read the contents of a file in Camel Spring DSL w/o File component?
I think threads may be the way to go because i am using the File component in a way that deletes the file upon successful processing. The WireTap route, should it fail, means that my file is gone and I lose knowledge that the File ever existed (I am still learning so perhaps that in and of itself is a poor approach). Also, I am stuck on Camel 2.8 so not sure how much flexibility I have with threads, but essentially it looks like I would do something like this: My question is, does this free up the single threaded file component to pick up another file? If so, what happens if the processing inside the thread fails? I suspect that the File component will have already deleted the file at that point? Thank you Claus for the quick response! -- View this message in context: http://camel.465427.n5.nabble.com/Is-it-possible-to-read-the-contents-of-a-file-in-Camel-Spring-DSL-w-o-File-component-tp5772942p5772945.html Sent from the Camel - Users mailing list archive at Nabble.com.
Is it possible to read the contents of a file in Camel Spring DSL w/o File component?
I have searched through the user forum and done some research through Camel docs as well, so if I missed the answer somewhere then I apologize. I read that the File component is single threaded and what I would like to be able to do, all while staying within Spring DSL, is to read file and fire off a call to a web service. I do have this working, however I really want to process the files in a multi-threaded fashion so as to speed up the process. I can scale out using more servers, but that is less than ideal. is there a way to queue up the name and location of the file, pull that info off of a queue (i.e. from(activemq) and then read the file contents to fire off to a web service, without getting back into a single threaded scenario? Just looking for general direction and I should be able to take it from there. Much appreciated! -- View this message in context: http://camel.465427.n5.nabble.com/Is-it-possible-to-read-the-contents-of-a-file-in-Camel-Spring-DSL-w-o-File-component-tp5772942.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Can I perform a move file operation in camel?
Any idea as to why this does not seem to work?
/C move
${body[FileDetails][FilePath]}\\${body[FileDetails][FileName]}
E:\\temp
I am attempting to build the set of arguments dynamically. I found a thread
that seemed to indicate that I could set the ExecBinding.EXE_COMMAND_ARGS
header to a space delimited string, but that does not appear to work.
When I turn on tracing I get the following:
Mon Nov 12 2012 20:28:45 GMT-0500 (Eastern Standard Time) INFOThe
command
ExecCommand [args=[${in.headers.EXEC_COMMAND_ARGS}], executable=cmd,
timeout=9223372036854775807, outFile=null, workingDir=null,
useStderrOnEmptyStdout=false] had exit value 0
Mon Nov 12 2012 20:28:45 GMT-0500 (Eastern Standard Time) INFO
Executing
ExecCommand [args=[${in.headers.EXEC_COMMAND_ARGS}], executable=cmd,
timeout=9223372036854775807, outFile=null, workingDir=null,
useStderrOnEmptyStdout=false]
Mon Nov 12 2012 20:28:45 GMT-0500 (Eastern Standard Time) INFO
ID-WMORMANS-27499-1352751723623-32-1 >>> (oec.publishing.data.filemove)
setHeader[EXEC_COMMAND_ARGS] -->
exec://cmd?args=%24%7Bin.headers.EXEC_COMMAND_ARGS%7D <<< Pattern:InOnly,
Headers:{JMSMessageID=ID:WMORMANS-27498-1352751721636-8:183126:1:2:1,
Cache-Control=private, Content-Length=329, JMSReplyTo=null,
JMSDeliveryMode=2, JMSCorrelationID=null, JMSExpiration=0,
CamelHttpResponseCode=200, JMSPriority=1,
JMSDestination=queue://oec.publishing.data.filemove, EXEC_COMMAND_ARGS=/C
move
E:\InventoryDrop\Publishing\NewDataReceived\ControlData\Manufacturer\.received\\Manufacturers.csv
E:\\temp, ExchangeId=d35d69c6-bf94-45de-a58d-1762c974ce8d, JMSType=null,
breadcrumbId=ID:WMORMANS-27498-1352751721636-8:183126:1:2:1,
Connection=Close, Content-Type=application/json; charset=utf-8,
JMSTimestamp=1352770124948, JMSRedelivered=false,
X-AspNet-Version=4.0.30319, JMSXGroupID=null, Date=Tue, 13 Nov 2012 01:28:44
GMT, Server=ASP.NET Development Server/10.0.0.0},
BodyType:java.util.HashMap,
Body:{FileDetails={FilePath=E:\InventoryDrop\Publishing\NewDataReceived\ControlData\Manufacturer\.received,
FileName=Manufacturers.csv, FileExtension=csv, FileSize=3526},
ExchangeState=FileStaged, ExchangeId=d35d69c6-bf94-45de-a58d-1762c974ce8d}
--
View this message in context:
http://camel.465427.n5.nabble.com/Can-I-perform-a-move-file-operation-in-camel-tp5722620p5722629.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Can I perform a move file operation in camel?
Nice! It looks like camel-exec will work for me. I just need to figure out
how to dynamically build up arguments for the move command. I tried using
the $simple{...} syntax, but that doesn't seem to be working. Thanks again
for help!
--
View this message in context:
http://camel.465427.n5.nabble.com/Can-I-perform-a-move-file-operation-in-camel-tp5722620p5722626.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Can I perform a move file operation in camel?
I am hoping that this is possible, but I am struggling with figuring out a solution. What I need to know is, can I move a file that already exists on the file system and to another folder using Spring DSL? I have a route that gets kicked off by having a JSON message placed onto an ActiveMQ queue. The JSON message contains the name and path of the file that I want to move. I am not sure what do to at this point. The route to pull off of the queue looks like this: Do I need to dynamically create a FileExchange in order to move the file? Is there a simpler approach? Is this even possible to do in Spring DSL? Any hints would be greatly appreciated! Thank you. -- View this message in context: http://camel.465427.n5.nabble.com/Can-I-perform-a-move-file-operation-in-camel-tp5722620.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: String replacement via Spring
Thank you everyone for helping me work through this issue. I finally figured
out how to use groovy to solve for it.
This seems to be working for me.
request.body.replace("\\","")
Thanks again!
--
View this message in context:
http://camel.465427.n5.nabble.com/String-replacement-via-Spring-tp5719498p5719623.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: String replacement via Spring
Yes, I started to realize that when I received an Ambiguous method call
exception in Camel. This is what I was trying to refer to when I said that
there are probably some limitations to the Simple language in that I am
unable to essentially add a cast to show my intent. For example I am
assuming something like this is not possible:
transform>
${body.replace((char)'\\',(char)'-')}
This would be an attempt to indicate to the Simple language parser that I
want the char method of the replace API and not the CharSequence method,
thus trying to eliminate the Ambiguous call exception.
Any ideas why the .replaceAll won't work even though I got a simple java
code sample working? For example, the following works fine, but in the
Simple language it throws a java.util.regex.PatternSyntaxException: Illegal
repetition exception.
class DoubleTheBackslash {
public static void main(String[] args) {
String path = "E:\\root\\dira\\dirb";
System.out.println(path);
System.out.println(path.replaceAll("foo","bar"));
}
}
--
View this message in context:
http://camel.465427.n5.nabble.com/String-replacement-via-Spring-tp5719498p5719603.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: String replacement via Spring
Hi Henryk,
I apologize for not providing more information. I guess I thought there was
enough context within the thread that one could understand what I was
referring to. Sounds like that was not the case.
The two attempts I made using Spring are as follows:
What Claus suggested:
${body.replaceAll("foo", "bar")}
Same thing as what Claus suggested with the exception of using .replace
instead of .replaceAll.
${body.replace("foo", "bar")}
The replacement strings that I am using in the example above are not really
what I want in the end, but I wanted to just get anything working at this
point. The point of my last response is that neither solution appears to be
behaving as I would expect, but again I am wondering if this just isn't
something that the Simple language can handle. I would like to know if the
errors I posted earlier are expected or if there is some other underlying
problem with the Simple language parser.
The real transform that I am trying to get to is something like this:
${body.replace("\\", "")}
I just need to replace all single backslashes in a windows-based fully
qualified file path with double backslashes such that I can send the JSON
packet to a restful service. Right now, the request is failing to send to
the .NET restful service due to the backslashes in the file path. If I
manually change the single backslashes to double backslashes, the JSON
packet is accepted and processed by the restful service. I also thought
about using headers to send the information, but this doesn't seem like a
"restful"way of doing things.
Another approach would be to URL encode the file path, but I can't seem to
find a way to do that in Spring either. I am very new to Camel, so I am just
hoping that I am missing something simple.
Thank you for the response and sorry for not being clear enough in my
previous post.
--
View this message in context:
http://camel.465427.n5.nabble.com/String-replacement-via-Spring-tp5719498p5719590.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: String replacement via Spring
Thank you for the response Claus. Cool that this is possible. I have been
trying to work with the Simple language in order to not have to dive into
the other scripting languages. I have had mixed results, none of which have
actually worked yet.
If I just take your example verbatim, then I getting the following error:
/Failed delivery for exchangeId: ID-WMORMANS-63776-1347901580693-114-1.
Exhausted after delivery attempt: 1 caught:
org.apache.camel.language.bean.RuntimeBeanExpressionException: Failed to
invoke method: .replaceAll("foo","bar") on null due to:
org.apache.camel.language.bean.RuntimeBeanExpressionException: Failed to
invoke method: replaceAll("foo","bar") on null due to:
java.util.regex.PatternSyntaxException: Illegal repetition
{"FilePath":"E:\Publishing\NewDataReceived\ReferenceData\Ford","FileName":"TestFile.txt","FileSize":"14"}/
I also tried using the .replace() method, but that throws this error:
/Failed delivery for exchangeId: ID-WMORMANS-63776-1347901580693-116-1.
Exhausted after delivery attempt: 1 caught:
org.apache.camel.language.bean.RuntimeBeanExpressionException: Failed to
invoke method: .replace("foo","bar") on null due to:
org.apache.camel.language.bean.RuntimeBeanExpressionException: Failed to
invoke method: replace("foo","bar") on null due to:
org.apache.camel.component.bean.AmbiguousMethodCallException: Ambiguous
method invocations possible: [public java.lang.String
java.lang.String.replace(java.lang.CharSequence,java.lang.CharSequence),
public java.lang.String java.lang.String.replace(char,char)].
Exchange[Message:
{"FilePath":"E:\Publishing\NewDataReceived\ReferenceData\Ford","FileName":"TestFile.txt","FileSize":"14"}]/
I am guessing that I am hitting some limitations of the Simple language?
Both .replaceAll and .replace seems to work fine when put into a simple Java
program.
--
View this message in context:
http://camel.465427.n5.nabble.com/String-replacement-via-Spring-tp5719498p5719540.html
Sent from the Camel - Users mailing list archive at Nabble.com.
String replacement via Spring
Is it possible to replace characters in a string via Spring DSL? I have a file path and want to send that path to a resful service in the body; however slashes in the path are preventing the packet from making it to the restful service. If I escape the slashes, then the call works just fine. I need a way to do this via Spring DSL if at all possible. I could try doing this via Java, but at this point have very little Java experience and everything we have done has been done in Spring DSL. Is there anyway that anyone knows how to manipulate strings in Spring DSL? Thank you. -- View this message in context: http://camel.465427.n5.nabble.com/String-replacement-via-Spring-tp5719498.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Processing large files in Camel
Hi Claus, Thank you for the response. After someone hijacked my thread I deleted my postings as I figured no one would respond. Plus, I figured out the problem. You actually touched on the issue I discovered. When I send the message to AMQ you are correct in that the entire file content was being delivered to the queue. This is definitely not what I wanted. Since the files are so large, at least for my test cases, "something" was blowing up under the covers to cause the java.lang.NullPointerException. My assumption is that due to the volume of bytes being written to the queue either a timeout or some sort of buffer overflow was happening, but the error was just manifesting itself in a weird way (i.e. the java.lang.NullPointException). I would love to know the actual problem, but in the end all I wanted to write to the queue was some basic information about the file and NOT the entire file itself. Performing a transform did the trick, so at this point I have things working as suspected. Just had to hit my head against the desk a few times to figure out what was going on. Again, Thank you for your response. It is very much appreciated. --Scott On Sat, Sep 15, 2012 at 2:25 AM, Claus Ibsen-2 [via Camel] < ml-node+s465427n5719403...@n5.nabble.com> wrote: > Hi > > I suggest to take a moment to look more at the docs > http://camel.apache.org/file2 > > For example on that link there is a red box on top that talks about > the issue of reading files which are currently being written by > another process. There is a number of options you can configure to > chose among. And if that don't suit you, you can implement your own > custom read lock strategy. > > And for sending the "message" to AMQ you would need to do a message > transformation so you don't send the entire file content with it, but > only the meta-data you want. And in the format you want. > > See the EIP for that > http://camel.apache.org/message-translator.html > > There is more EIPs here > http://camel.apache.org/enterprise-integration-patterns.html > > And since you have to leave the files in place, you would have to > consider at "what to do" in case the server is restarted. > Then the previous "scanned" files may be picked up again. Is this > okay, or do you need to keep "book" of that. > If so check out the idempotent consumer option on the file component, > as well as the EIP > http://camel.apache.org/idempotent-consumer.html > > > On Fri, Sep 14, 2012 at 5:03 PM, gramanero <[hidden > email]<http://user/SendEmail.jtp?type=node&node=5719403&i=0>> > wrote: > > > I am attempting to understand how the Camel File/File2 component works > as I > > have a need to use this type of capability in an upcoming project. > > > > Goal: Write a Camel spring-based route that will detect files in a > folder > > structure, wait until the entire file has been written, and place an > entry > > onto an ActiveMQ queue that indicates the file name, path, and length, > and > > leave the file in place (i.e. do not let Camel move it to a .camel > folder). > > > > Camel Route: > > > >> > uri="file:E:\NewDataReceived?recursive=true&maxMessagesPerPoll=500&noop=true&delay=6"/> > > > > > > > > > Problem: For small files this route works as expected. For large files, > I am > > seeing a .camelLock file appear every N seconds (which corresponds to > the > > configured 'delay'setting) and then disappear. I am assuming that this > is > > Camel trying to get a read lock on the file, but I cannot find any docs > on > > this type of behavior. I am also guessing that it keeps appearing and > > disappearing due to the file size?? The size of the files that I am > dealing > > with are anywhere from 7Gb and up. This may not be a realistic use case > for > > what we will be dealing with, but I want to verify that writing files > will > > only ever put an entry onto the queue once the file have been completely > > written and not before. > > > > Camel version: Whatever is baked into apache-servicemix-4.4.1-fuse-03-06 > > > > Any suggestions/help is greatly appreciated. > > > > Thank you. > > > > > > > > > > > > -- > > View this message in context: > http://camel.465427.n5.nabble.com/Processing-large-files-in-Camel-tp5719376.html > > Sent from the Camel - Users mailing list archive at Nabble.com. > > > > -- > Claus Ibsen > - > FuseSource > Email: [hidden email]<http://user/SendEmail.jtp?type=node&node=5719403&i=1&g
Re: Processing large files in Camel
I found this in the log as it relates to the issue with the .camelLock file.
Looks like there is a java.lang.NullPointerException being thrown. No clue
as to why this is happening though.
Fri Sep 14 2012 12:43:21 GMT-0400 (Eastern Daylight Time) ERROR Failed
delivery for exchangeId: ID-WMORMANS-7972-1346359758983-12-77. Exhausted
after delivery attempt: 1 caught: java.lang.NullPointerException
java.lang.NullPointerException: null
Fri Sep 14 2012 12:43:21 GMT-0400 (Eastern Daylight Time) WARN
Rollback file
strategy:
org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@612a3291
for file:
GenericFile[E:\Publishing\NewDataReceived\ReferenceData\Ford\OEC.Publishing.Service.FileWatcher-7-19-2012.DMP]
Fri Sep 14 2012 12:43:20 GMT-0400 (Eastern Daylight Time) INFO
ID-WMORMANS-7972-1346359758983-12-77 >>>
(publishing.dataWatchDog.FileReceived)
from(file://E:\Publishing\NewDataReceived?delay=6&maxMessagesPerPoll=500&recursive=true)
--> activemq://queue:data.received <<< Pattern:InOnly,
Headers:{CamelFileNameOnly=OEC.Publishing.Service.FileWatcher-7-19-2012.DMP,
breadcrumbId=ID-WMORMANS-7972-1346359758983-12-78,
CamelFileLength=7623493727,
CamelFileAbsolutePath=E:\Publishing\NewDataReceived\ReferenceData\Ford\OEC.Publishing.Service.FileWatcher-7-19-2012.DMP,
CamelFileRelativePath=ReferenceData\Ford\OEC.Publishing.Service.FileWatcher-7-19-2012.DMP,
CamelFilePath=E:\Publishing\NewDataReceived\ReferenceData\Ford\OEC.Publishing.Service.FileWatcher-7-19-2012.DMP,
CamelFileAbsolute=true,
CamelFileName=ReferenceData\Ford\OEC.Publishing.Service.FileWatcher-7-19-2012.DMP,
CamelFileLastModified=Thu Jul 19 16:02:52 EDT 2012,
CamelFileParent=E:\Publishing\NewDataReceived\ReferenceData\Ford},
BodyType:org.apache.camel.component.file.GenericFile, Body:[Body is file
based:
GenericFile[E:\Publishing\NewDataReceived\ReferenceData\Ford\OEC.Publishing.Service.FileWatcher-7-19-2012.DMP]]
--
View this message in context:
http://camel.465427.n5.nabble.com/Processing-large-files-in-Camel-tp5719376p5719386.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Processing large files in Camel
Sorry, once more piece of information...the appearing and disappearing of the .camelLock file is continuous. It never stops unless I stop the route. Also, I am running on Windows 7 O.S. -- View this message in context: http://camel.465427.n5.nabble.com/Processing-large-files-in-Camel-tp5719376p5719378.html Sent from the Camel - Users mailing list archive at Nabble.com.
Processing large files in Camel
I am attempting to understand how the Camel File/File2 component works as I have a need to use this type of capability in an upcoming project. Goal: Write a Camel spring-based route that will detect files in a folder structure, wait until the entire file has been written, and place an entry onto an ActiveMQ queue that indicates the file name, path, and length, and leave the file in place (i.e. do not let Camel move it to a .camel folder). Camel Route: Problem: For small files this route works as expected. For large files, I am seeing a .camelLock file appear every N seconds (which corresponds to the configured 'delay'setting) and then disappear. I am assuming that this is Camel trying to get a read lock on the file, but I cannot find any docs on this type of behavior. I am also guessing that it keeps appearing and disappearing due to the file size?? The size of the files that I am dealing with are anywhere from 7Gb and up. This may not be a realistic use case for what we will be dealing with, but I want to verify that writing files will only ever put an entry onto the queue once the file have been completely written and not before. Camel version: Whatever is baked into apache-servicemix-4.4.1-fuse-03-06 Any suggestions/help is greatly appreciated. Thank you. -- View this message in context: http://camel.465427.n5.nabble.com/Processing-large-files-in-Camel-tp5719376.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
How do you get the message to "bounce off the front of the queue"? Can you control whether the message is on the front or back of the queue? You brought up an interesting point at the beginning of this thread that indicated that a message could get stuck on the front of the queue thus preventing all other messages rom processing, so I would be interested to know how you are essentially pulling the message of and putting it back on the end of the queue. I thought a rollback would essentially keep the message on the front of the queue. Its a FIFO queue, right? Overall, an interesting approach. I like the idea of getting messages temporarily out of the way until they can "play nice" again. I can see in a connection lost scenario that all of the messages will potentially end up in purgatory and then be put back on the original queue, hopefully when the service is available again. Essentially a holding tank until the situation resolves itself. I think the difficulty with any approach is with determining why a message is not playing nice. It is bad message content or is there some sort of system failure. Bad message content most likely means that the message needs to be taken out of rotation permanently (probably not to many cases where you can "fix" the message content), but a system failure (i.e. consuming service is down) is when we would want to try and re-play the message at a later time. Thanks for sharing. I'm interesting to see how this pans out for you. -- View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714205.html Sent from the Camel - Users mailing list archive at Nabble.com.
Question about .NET Exceptions and the onException clause
Does anyone know if it is possible to interrogate an exception in Camel that was thrown from a .NET-based endpoint (i.e. a .NET restful service)? I'm thinking that since the restful service will throw the exception in either JSON or XML format, that being a string based format (as opposed to binary) that somehow the underlying .NET exception would be available. All I am getting from the service is a org.apache.camel.component.cxf.CxfOperationException exception. I can trap this in the onException clause, but beyond that I don't know what really happened. Probably the downside of having to communicate with a .NET service from a java technology I suppose. Oh, and I am hoping to be able to do this in Spring DSL, but would be interested to know if there is a Java DSL way to do it as well. Thanks. -- View this message in context: http://camel.465427.n5.nabble.com/Question-about-NET-Exceptions-and-the-onException-clause-tp5714201.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Perhaps you are correct in that the onException gets fired within the bounds of the transaction thus making the operation, as a whole, atomic. However, and keep in mind that I am still learning Camel, from what I can tell the onException fires within the bounds of the transaction and the true elements are basically telling Camle that "everything is ok" which means that things do not get rolled back. In my case, I am trying to catch specific exceptions that are thrown from the restulful service (which means that no work happened...i.e. no DB updates or anything), and push the message onto the DLQ to get it out of the way. Essentially saying that those messages are bad and I don't/can't process them. If I let the transaction rollback, then I have the message on the queue still and then run the risk of trying to process it a second time which will result in the same scenario and most likely end up in an endless loop. Unless I configure the redelivery policy to only attempt to redeliver N times as opposed to an infinite numer of times. On Fri, Jun 8, 2012 at 7:30 AM, James Carman-2 [via Camel] < ml-node+s465427n5714184...@n5.nabble.com> wrote: > You want it to be atomic with the get from the input queue don't you? It > would make sense to me. > On Jun 8, 2012 7:13 AM, "gramanero" <[hidden > email]<http://user/SendEmail.jtp?type=node&node=5714184&i=0>> > wrote: > > > Now that you mention it, one might think that, but that does not appear > to > > be the case. With the DLQ put in the onException clause, the DLQ put > does > > not rollback. I think that is where the Handled clause comes into play. > > > > On Fri, Jun 8, 2012 at 7:10 AM, James Carman-2 [via Camel] < > > [hidden email] <http://user/SendEmail.jtp?type=node&node=5714184&i=1>> > wrote: > > > > > Won't the dlc "put" be in the transaction too? That would rollback > too, > > > thus nothing ever happened. > > > On Jun 8, 2012 6:56 AM, "James Carman" <[hidden email]< > > http://user/SendEmail.jtp?type=node&node=5714181&i=0>> > > > wrote: > > > > > > > Try it with client cache control. Take a look at my example. > > > > > > > > On Fri, Jun 8, 2012 at 6:47 AM, gramanero <[hidden email]< > > http://user/SendEmail.jtp?type=node&node=5714181&i=1>> > > > wrote: > > > > > No, it is not rolling back if you use the Handles element with a > > > > constant value of true. If you use the Continue element then I > believe > > > it > > > > will roll back. > > > > > > > > > > Sent from my iPod > > > > > > > > > > On Jun 7, 2012, at 11:46 PM, "James Carman [via Camel]" < > > > > [hidden email] <http://user/SendEmail.jtp?type=node&node=5714181&i=2>> > > > > wrote: > > > > > > > > > >> Your transaction isn't rolling back if you "handle" the > exception, > > is > > > > it? > > > > >> > > > > >> On Thu, Jun 7, 2012 at 12:21 PM, gramanero <[hidden email]> > wrote: > > > > >> > > > > >> > I have tested the case of using a route specific onException > > clause > > > > within a > > > > >> > transaction and it appears to work as I would expect (or hope). > So > > > I > > > > have a > > > > >> > route that is transactional and the final endpoint in the route > > > > throws an > > > > >> > exception I forced my restful service to just throw an > exception). > > > > Without > > > > >> > the onException clause the message lands back in the queue as > you > > > > would > > > > >> > expect due to it running within a transaction. With the > > onException > > > > clause, > > > > >> > I look for specific exceptions and if it is one of the > exceptions > > > > that I > > > > >> > have specified I tell tell Camel that the exception has been > > > > "handled" (via > > > > >> > the handled clause) and I route the message to the dead letter > > > queue, > > > > thus > > > > >> > moving the "bad message" out of the way of the messages > remaining > > > on > > > > the > > > > >> > queue. I think the key here is the use of the "handled" clause > > that
Re: Transacted vs DeadLetterQueue
Blueprint? Hmmm...I have not heard or seen that yet. lol...great. Another rabbit hole for me to dive down into. ;-) -- View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714190.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
So in your example, what is the difference between the Purgatory count and using the Redelivery capabilities built into ActiveMQ and Camel policies? Maybe I'm missing something, but it seems like the logic wrapped around the Purgatory concept is just keeping track of how many times a message has been delivered and caused an exception to happen. This could happen if the "to" endpoint is unavailable (i.e. a restful service) and until the service does become available the count will go up until the message ends up in the DLQ. Like I said, maybe I'm just missing something here. -- View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714188.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
I'm going through your example now. Only one problem for me, we are a Micosoft shop. Much of the skill set of the developers is Microsoft-only and very little Java experience. Having said that, I myself have developed in Java for 5-years, but that was seven years ago. While the language is not foreign to me, I need to try and build a system that others can maintain which means that I am looking to do everything in Spring DSL and not Java DSL. I "might" be able to take your concepts though and apply them to Spring DSL. We actually took a look at using NServiceBus, but fell in love with the Apache stack and all the Camel has to offer (+ServiceMi, ActiveMQ, etc). -- View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714186.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Now that you mention it, one might think that, but that does not appear to be the case. With the DLQ put in the onException clause, the DLQ put does not rollback. I think that is where the Handled clause comes into play. On Fri, Jun 8, 2012 at 7:10 AM, James Carman-2 [via Camel] < ml-node+s465427n5714181...@n5.nabble.com> wrote: > Won't the dlc "put" be in the transaction too? That would rollback too, > thus nothing ever happened. > On Jun 8, 2012 6:56 AM, "James Carman" <[hidden > email]<http://user/SendEmail.jtp?type=node&node=5714181&i=0>> > wrote: > > > Try it with client cache control. Take a look at my example. > > > > On Fri, Jun 8, 2012 at 6:47 AM, gramanero <[hidden > > email]<http://user/SendEmail.jtp?type=node&node=5714181&i=1>> > wrote: > > > No, it is not rolling back if you use the Handles element with a > > constant value of true. If you use the Continue element then I believe > it > > will roll back. > > > > > > Sent from my iPod > > > > > > On Jun 7, 2012, at 11:46 PM, "James Carman [via Camel]" < > > [hidden email] <http://user/SendEmail.jtp?type=node&node=5714181&i=2>> > wrote: > > > > > >> Your transaction isn't rolling back if you "handle" the exception, is > > it? > > >> > > >> On Thu, Jun 7, 2012 at 12:21 PM, gramanero <[hidden email]> wrote: > > >> > > >> > I have tested the case of using a route specific onException clause > > within a > > >> > transaction and it appears to work as I would expect (or hope). So > I > > have a > > >> > route that is transactional and the final endpoint in the route > > throws an > > >> > exception I forced my restful service to just throw an exception). > > Without > > >> > the onException clause the message lands back in the queue as you > > would > > >> > expect due to it running within a transaction. With the onException > > clause, > > >> > I look for specific exceptions and if it is one of the exceptions > > that I > > >> > have specified I tell tell Camel that the exception has been > > "handled" (via > > >> > the handled clause) and I route the message to the dead letter > queue, > > thus > > >> > moving the "bad message" out of the way of the messages remaining > on > > the > > >> > queue. I think the key here is the use of the "handled" clause that > > tells > > >> > Camel that the message has been handled and therefore to NOT > rollback > > the > > >> > transaction. The alternative choice is to tell Camel to "continue" > on > > with > > >> > its normal processing which would have rolled back the transaction > > and put > > >> > the message back onto the queue (via the "continue" clause...at > least > > I > > >> > think it is a clause). > > >> > > > >> > Hope that helps. > > >> > > > >> > -- > > >> > View this message in context: > > > http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714139.html > > >> > Sent from the Camel - Users mailing list archive at Nabble.com. > > >> > > >> > > >> If you reply to this email, your message will be added to the > > discussion below: > > >> > > > http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714151.html > > >> To unsubscribe from Transacted vs DeadLetterQueue, click here. > > >> NAML > > > > > > > > > -- > > > View this message in context: > > > http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714179.html > > > Sent from the Camel - Users mailing list archive at Nabble.com. > > > > > -- > If you reply to this email, your message will be added to the discussion > below: > > http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714181.html > To unsubscribe from Transacted vs DeadLetterQueue, click > here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5713992&code=Z3JhbWFuZXJvQGdtYWlsLmNvbXw1NzEzOTkyfC0xNjAyMDYxMDQz> > . > NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714182.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
No, it is not rolling back if you use the Handles element with a constant value of true. If you use the Continue element then I believe it will roll back. Sent from my iPod On Jun 7, 2012, at 11:46 PM, "James Carman [via Camel]" wrote: > Your transaction isn't rolling back if you "handle" the exception, is it? > > On Thu, Jun 7, 2012 at 12:21 PM, gramanero <[hidden email]> wrote: > > > I have tested the case of using a route specific onException clause within > > a > > transaction and it appears to work as I would expect (or hope). So I have a > > route that is transactional and the final endpoint in the route throws an > > exception I forced my restful service to just throw an exception). Without > > the onException clause the message lands back in the queue as you would > > expect due to it running within a transaction. With the onException clause, > > I look for specific exceptions and if it is one of the exceptions that I > > have specified I tell tell Camel that the exception has been "handled" (via > > the handled clause) and I route the message to the dead letter queue, thus > > moving the "bad message" out of the way of the messages remaining on the > > queue. I think the key here is the use of the "handled" clause that tells > > Camel that the message has been handled and therefore to NOT rollback the > > transaction. The alternative choice is to tell Camel to "continue" on with > > its normal processing which would have rolled back the transaction and put > > the message back onto the queue (via the "continue" clause...at least I > > think it is a clause). > > > > Hope that helps. > > > > -- > > View this message in context: > > http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714139.html > > Sent from the Camel - Users mailing list archive at Nabble.com. > > > If you reply to this email, your message will be added to the discussion > below: > http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714151.html > To unsubscribe from Transacted vs DeadLetterQueue, click here. > NAML -- View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714179.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
I have tested the case of using a route specific onException clause within a transaction and it appears to work as I would expect (or hope). So I have a route that is transactional and the final endpoint in the route throws an exception I forced my restful service to just throw an exception). Without the onException clause the message lands back in the queue as you would expect due to it running within a transaction. With the onException clause, I look for specific exceptions and if it is one of the exceptions that I have specified I tell tell Camel that the exception has been "handled" (via the handled clause) and I route the message to the dead letter queue, thus moving the "bad message" out of the way of the messages remaining on the queue. I think the key here is the use of the "handled" clause that tells Camel that the message has been handled and therefore to NOT rollback the transaction. The alternative choice is to tell Camel to "continue" on with its normal processing which would have rolled back the transaction and put the message back onto the queue (via the "continue" clause...at least I think it is a clause). Hope that helps. -- View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714139.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Yes, that is correct and actually in my case that "may" be what I want to have happen (i.e. the message gets put back onto queue A). However yesterday I did some digging into the onException clause and I'm finding it very useful. For example, if when "something bad happens" an exception is thrown, I can essentially catch that exception in Spring DSL and if it is a specific exception that I an looking for then I can decide what to do with the message. I can put it onto the deadletter queue, perhaps in the case of an argument exception, or I can stick it back onto the queue and have the route utilize my redelivery policy, especially if the exception was a connectivity issue with the cxfrs endpoint that I am trying to route the message to. Putting the message onto the DLQ move "poison pills" out of the way for the good messages to be routed to the cxfrs endpoint thus allowing message to flow through the route again. Seems to work pretty well. The issue I have now is that my cxfrs endpoint is a .NET restful service and if I throw a .NET exception from the service, it does not get translated to an equivalent java exception. Makes sense because I'm not sure that there would be a simple translation, but it does make it more difficult to determine which exceptions I want to look for a then route to the DLQ. Not sure if that helps you, but it seems to be a possible approach for me. -- View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714128.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Thanks Claus. So is collision avoidance a different mechanism than the exponential back-off settings or do they work together? -- View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714060.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Transacted vs DeadLetterQueue
Thanks for the response Christian. I have been reading through the Camel in Action and ActiveMQ in Action books most of the day today. I'm starting to get a better understanding of how transactions work and that definitely seems like the way to go. Also, seems like the DLQ and transactions work together, so that I can dump off messages in the DLQ if "something bad happens". I think what I want in my case is to set up the redelivery policy so that messages stay on the queue until the cxfrs endpoint becomes available again. Looks like I can achieve this with a value of -1 for the maximumRedelivery configuration. I just need to determine a good balance for the other settings (i.e. redeliveryDelay) so that I'm not trashing the system by constantly trying to resend messages to an unavailable endpoint. Can someone tell me what the useCollisionAvoidence setting is for? The only explanation that I can find is: "Should the redelivery policy use collision avoidance" This really isn't telling me a whole lot. I can't seem to find anything in either book that discusses it either. Thanks. -- View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992p5714032.html Sent from the Camel - Users mailing list archive at Nabble.com.
Transacted vs DeadLetterQueue
I'm wondering if there are any sort of best practices or pros/cons to using either the dead letter queue or transactions when errors occur while pulling off messages from a queue and attempting to route them to a cxfrs endpoint that may not be available. For example, say I have the following route: http://localhost:/MyService"; pattern="InOnly"/> I have played around with using the dead letter queue and that seems to work well, but then I started to think about getting the messages off the dead letter queue and putting them back onto myQueue for processing once the service becomes available. One way to accomplish this (possibly) is to use the CronScheduledRoutePolicy such that it wakes up every hour and starts a route that will pull messages off the dead letter queue (if there are any), put them onto the myQueue and then stop the route after N number of minutes. I'm not sure that this is a good idea, but it "one" way to build a more auto-maintained system. Seems like once I start getting into configuring the CronScheduledRoutePolicy that the solution becomes more complex then it really needs to be in terms of the various moving parts. The other thought is to use the ActiveMQ transaction manager. If the cxfrs endpoint is not available, then the transaction rolls back which puts the message back on the queue (or just never really pulls it off) and then waits a while (via a delay configuration) and tries to route the message again. One thing I don't really like about this is that if the service is down for a long period of time, then it seems like we will be in this mode where messages are constantly trying to be routed to the unavailable cxfrs endpoint thus sort of thrashing the system. I'm just looking for some opinions to how this type of problem is typically solved or some sort of best practice to this use case. Thanks! -- View this message in context: http://camel.465427.n5.nabble.com/Transacted-vs-DeadLetterQueue-tp5713992.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: ThrottlingInflightRoutePolicy
Perfect. Thank you Claus! Your quick response is very much appreciated. -- View this message in context: http://camel.465427.n5.nabble.com/ThrottlingInflightRoutePolicy-tp5713970p5713973.html Sent from the Camel - Users mailing list archive at Nabble.com.
ThrottlingInflightRoutePolicy
I need to verify my understanding of the ThrottlingInflightRoutePolicy. I have the following policy and route defined: http://localhost:/MyService"; pattern="InOnly"/> My understanding is that, after each completed exchange the policy will check the current number of exchanges that are "in-flight" and if that number exceeds the maximumInflightExchanges, then it will suspend the route until the percentage of "in-flight" exchanges reaches the resumePercentOfMax number of exchanges. At that point, the route will be resumed. This will have the impact of preventing the overloading of the cxfrs endpoint. Is this correct? If so, what does that mean for other routes that have a , essentially dumping entries into myQueue? Are the entries still pushed onto the queue, but just not picked up until the route is resumed? Lastly, is the "value" of the scope element for the policy specified as "Route" or the name of the actual route? I'm assuming "Route" because the actual route itself will specify the policy that it refers to. Thanks for any assistance in answering these questions! -- View this message in context: http://camel.465427.n5.nabble.com/ThrottlingInflightRoutePolicy-tp5713970.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: HOW-TO: Jetty, JSON, and Restful Services
Ok, after toying around with it I realized that I messed up the uri for the cxfrs endpoint. I have it working now. I would still be interested in seeing the route that send the request via the http endpoint, just to educate myself more. Thank you for all the help. The final route ends up looking like this: http://localhost:/MetricsEndpoint"/> application/json http://localhost:5477/MetricsService/"; pattern="InOnly"/> -- View this message in context: http://camel.465427.n5.nabble.com/HOW-TO-Jetty-JSON-and-Restful-Services-tp5713702p5713738.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: HOW-TO: Jetty, JSON, and Restful Services
Currently there is no need to perform any transforms on the request. Just a straight pass through camel. Sorry for my ignorance, but what would the route look like if I were to take your suggestion of using the http endpoint to send the request? Thank you for your response. -- View this message in context: http://camel.465427.n5.nabble.com/HOW-TO-Jetty-JSON-and-Restful-Services-tp5713702p5713735.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: HOW-TO: Jetty, JSON, and Restful Services
Thank you for the tip about removeHeaders. I was wondering if there was an easier way to accomplish that. Very much appreciated! What I noticed when initially trying to route the message from jetty to the restulf svc is that the CamelHttpUrl was incorrect. What was logged in the servicemix.log file was: http://localhost:5477/MetricsService/Create/MetricsEndpoint/ To me it looks like the jetty endpoint path is being appended to the cxfrs endpoint and then camel is trying to communicate with that, which obviously won't work. That is when I started messing with the headers to try and get them "cleaned up" to what I thought made more sense. Of course, I very well may be doing more harm than good. When I take out the removeHeader elements in the route I get the following error: Caused by: java.lang.IllegalArgumentException: Illegal character(s) in message header value: HTTP/1.1 200 As far as verifying the restful svc itself, I was able to use the Composer tab in Fiddler to fire off a JSON request and it all worked just fine. I basically threw a breakpoint in the .NET restful svc and verified that the callers data was passed correctly, so based on that it doesn't appear to be a problem with the service itself. I also got it to work via a powershell script. I tried using soapUI (and had success with it in the past), but when the restful svc method gets invoked, the input argument is null. I'm sure I'm doing something stupid in soapUI, but I didn't feel like fighting with the tool so I just ended up using Fiddler. Currently I am running everything on a local workstation. Thank you for the response! -- View this message in context: http://camel.465427.n5.nabble.com/HOW-TO-Jetty-JSON-and-Restful-Services-tp5713702p5713712.html Sent from the Camel - Users mailing list archive at Nabble.com.
HOW-TO: Jetty, JSON, and Restful Services
I am "very" new to FuseESB and the underlying technology stack, so I apologize up front if I am not using the correct terminology when trying to describe the issues we are running into. We are working on a proof of concept to determine if FuseESB is the direction we want to go for some internal projects. We are very excited about the various capabilities that are readily available and can envision how these pre-built pieces will allow us to build solutions quickly. Having said that, we have having some difficulty in getting a particular route to work and could use some assistance. There are two problems that we are running in to. 1) Routing XML based messages from a jetty or ActiveMQ endpoint to a .NET restful service without going through a file endpoint. 2) Routing JSON based messages from jetty or ActiveMQ endpoint to a .NET restful svc The route that we are trying to, ideally, get up and running is as follows: (client application) --> jetty:http --> ActiveMQ --> .NET Restful Service (JSON) We are able to send a JSON or XML message to the jetty endpoint; however trying to get the body of the message routed directly to the restul svc has been a bit of a challenge. Even attempting to route the body of the JSON or XML message to a queue, and then to the restful svc is showing similar problems as well. What have been able to get working is the following: (client application) --> jetty:http --> ActiveMQ --> File --> .NET Restful Service (XML) We want to get rid of the use of the file system as it slows up the process and is an unnecessary hop. With respect to trying to get the message from jetty to the restul svc or from ActiveMQ to the restful svc, it "feels" like there is data coming out of the "from" endpoint that is negatively colliding with our attempts to forward the message to the restful svc. In the case of jetty:http directly to the restul svc, here is the route: http://localhost:/MetricsEndpoint"/> http://localhost:5477/MetricsService/Create/"; pattern="InOnly"/> The reason for all of the "removeHeader" elements is that it appears as though the output message from jetty is adding in all of the input message headers, so this is my attempt at clearing them out. I suspect that something similar is happening when routing directly from ActiveMQ to the restful svc. I also suspect that this is why routing through the file system works, because the only information in the file that gets written to disk is the body of the message minus any header information from jetty or ActiveMQ. I attempted to use the Jackson library as a data format for marshalling/unmarshalling, but I'm not quite sure I need it since we are really trying to transform the data between JSON and XML (nor do we really want to). We really want to keep the messages as JSON al the way through the route. I'm not exactly how much information to provide with post, so I'll give you the current error that we are running into and see where the thread goes from there. When the restful svc is NOT up and running then I get an error that I would expect: Caused by: java.net.ConnectException: ConnectException invoking http://localhost:5477/MetricsService/Create/: Connection refused: connect This tells me (I think) that I am getting to the jetty endpoint and that camel is routing something to the restful svc, but just cannot connect to it. When I start the restful svc, I get the following: org.apache.camel.component.cxf.CxfOperationException: JAXRS operation failed invoking http://localhost:/MetricsEndpoint with statusCode: 404 I uploaded the service mix log file, so I'm hoping that provided more context. http://camel.465427.n5.nabble.com/file/n5713702/servicemix.log servicemix.log ServiceMix Version: 4.4.1-fuse-03-06 (standard out of the box configuration) One last piece of information is that while we have Java development experience in-house, we are mostly an MS shop. We are trying to get these routes to work with no java development, but rather through the use of Spring. Any help would be greatly appreciated and obviously if I can provide more information please let me know. Thanks! -- View this message in context: http://camel.465427.n5.nabble.com/HOW-TO-Jetty-JSON-and-Restful-Services-tp5713702.html Sent from the Camel - Users mailing list archive at Nabble.com.
