Re: How to consume selected files from FTP endpoint?
Claus, I found I could filter the files using the “include" option on the FTP end points. Thank you for reminding this Camel noob that FTP extends File. For anyone interested: Using the Java fluent DSL, I was using something like this: from(“ftp:server/someFolder&logonOptions&move=.done") .filter(predicateMatchingPrefixAndSomeSuffix) .process(myProcessor) .to(destinationEp) That was consuming every file in someFolder, but only processing those with matching names. That led to my earlier question about filter unexpectedly being applied after all files were consumed from the FTP folder. After doing some research into File and File2, I tried the “include" option: from(“ftp:server/someFolder&[logonOptions]&include=(?i)SomePrefix_.*_SomeSuffix.xml&move=.done”) .process(myProcessor) .to(destinationEp) Using the include option as above, Camel only reads and moves files that case insensitively match the filename pattern. I suppose I was thinking the ftp fetch was being lazily applied after the filter() processed the file names. (I'll blame it on my recently doing some functional programming with Java 8! :-) Thanks for the help. — Jack Frosch On 11/12/15, 10:08 AM, "Claus Ibsen" wrote: >The filter should work while scanning for files - not after. eg if you >configure the filter on the ftp endpoint. There is a bunch of options >for filtering with include/exclude/ant/filter etc. > >As ftp extends file you can find all the options in the file docs >http://camel.apache.org/file2 > >On Thu, Nov 12, 2015 at 5:04 PM, Jack Frosch wrote: >> Our application consumes files from a folder on a supplier's FTP server. >> Unfortunately, the supplier uses the same folder for documents intended to >> be consumed by other applications. We’d like to consume (i.e. read, then >> delete or move) our files which can be identified by file name pattern, >> leaving the other files we don’t care about in the folder. >> >> I was hoping a filter would do this, but it seems the filter gets applied >> after the file is consumed. >> >> I’m sure I’m overlooking something, but am Googled out. :-) >> >> Is there a way to configure the FTP endpoint using Camel 2.15.1 to only >> consume a subset of files from a folder on an FTP server? >> >> Thanks in advance! >> >> — >> Jack >> > > > >-- >Claus Ibsen >- >http://davsclaus.com @davsclaus >Camel in Action 2: https://www.manning.com/ibsen2
Re: Problem with bean method calls and parameter types
Hi Ryan,
The following Camel DSL:
to("bean:beanClass?method=methodToCall(${body})")
or
bean(BeanClass.class, "methodToCall(${body})")
Does not convert the body to String when the type of body corresponds to that
of the first method parameter.
Is there any step upfront in your route that may convert the body to String?
The error that you have can occur when quotes are added to type declaration
like:
to("bean:beanClass?method=processBody('org.apache.camel.cdi.se.bean.Pojo')")
^
^
instead of:
to("bean:beanClass?method=processBody(org.apache.camel.cdi.se.bean.Pojo)")
Could you share you exact piece of route and bean class?
Antonin
> On 12 Nov 2015, at 18:20, Ryan Moquin wrote:
>
> Hi, I'm using Camel 2.16.0 and trying to invoke a bean method using spring
> xml and having some issues. It seems like any time I try to invoke a bean
> method with a camel message body, camel tries to pass it as a string to the
> method (even though the method takes the class as an argument) I'm calling
> even though the body type is correctly set to the java class type and the
> body itself is a class. I've called bean methods a lot from Camel and
> don't remember this being normal behavior? Or am I wrong?
>
> Caused by: org.apache.camel.component.bean.ParameterBindingException: Error
> during parameter binding on method: public java.lang.String
> com.test.TestService.retrieve(com.test.bo.TestObject,java.util.Date,com.test.TestParam)
> at parameter #0 with type: class com.test.bo.TestObject with value type:
> class java.lang.String and value: com.test.bo.TestObject
>at
> org.apache.camel.component.bean.MethodInfo$2.evaluateParameterValue(MethodInfo.java:582)
>...
> Caused by: org.apache.camel.NoTypeConversionAvailableException: No type
> converter available to convert from type: java.lang.String to the required
> type: com.test.bo.TestObject with value com.test.bo.TestObject
>at
> org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:1
> 85)
>
> I'm just calling the bean method like you normally would. I can log the
> value of a property of the java class in the body using a log statement
> right before I do the invocation of the method. I don't do anything
> special when calling the method other than putting the first parameter as:
> ${body}
>
> Is this by design in Camel and I never noticed? If so, is there any way
> around it?
>
> Thanks for any help!
> Ryan
Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads
This bit is definitely a bug. Whether it's something in the component, or the SshClient library needs to be investigated. You should log this one in a Jira. On 12/11/15 16:42, codan84 wrote: Indeed the ClientSession of SshClient is being started, but never finished. Here are logs: 2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-1] INFO o.a.s.c.session.ClientSessionImpl - Client session created 2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-2] INFO o.a.s.c.session.ClientSessionImpl - Server version string: SSH-1.99-OpenSSH_3.9p1 2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-3] INFO o.a.s.c.session.ClientSessionImpl - Kex: server->client aes128-ctr hmac-sha1 none 2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-3] INFO o.a.s.c.session.ClientSessionImpl - Kex: client->server aes128-ctr hmac-sha1 none 2015-11-12 16:39:09 [sshd-SshClient[68c05218]-nio2-thread-5] WARN o.a.s.c.k.AcceptAllServerKeyVerifier - Server at *** presented unverified DSA key: *** Script output :) This just repeats for every call. -- View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773762.html Sent from the Camel - Users mailing list archive at Nabble.com.
Camel + SpringBoot + Endpoint Mocking
Hi,
I want to test a complex route that involves sending messages to ActiveMQ and
calling webservices with SpringWs.
The route itself is working fine using spring-boot (1.2.7.RELEASE) and the
camel-spring-boot plugin (2.16.0).
Here are the important parts of the code:
@Component
public class MyRoute extends SpringRouteBuilder {
@Override
public void configure() throws Exception {
from(direct:responseQueue)
.transacted()
.split(...)
.to(activemq:individual_persist_queue)
.end()
.to("spring-ws:http://localhost:8088/acknowledge_webservice";)
.log("DONE");
}
}
Now I want to test this route by mocking the activemq and spring-ws endpoints
so the test can be run without any dependency on the broker or the webserver.
My basic requirement is to verify that the right amount of messages are sent to
each endpoint.
In my current scenario, the original message is split into three parts which
should be sent to ActiveMQ, followed by a single acknowledge message to the
WebService.
The transaction is there to roll-back the JMS deliveries in case the
web-service call fails. None of that should be important for this test however.
My test looks as follows:
@RunWith(CamelSpringJUnit4ClassRunner.class)
@BootstrapWith(CamelTestContextBootstrapper.class)
@SpringApplicationConfiguration(classes = MyConfig.class)
@DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)
@MockEndpointsAndSkip
public class CamelSpringBootTest {
@Produce(uri = "direct:responseQueue ")
protected ProducerTemplate template;
@EndpointInject(uri = "mock:
spring-ws:http://localhost:8088/acknowledge_webservice";)
MockEndpoint webserviceMock;
@EndpointInject(uri = "mock:activemq:individual_persist_queue ")
MockEndpoint activemqMock;
@Test
public void test() throws Exception {
activemqMock.expectedMessageCount(3);
webserviceMock.expectedMessageCount(1);
template.sendBody(someXML);
MockEndpoint.assertIsSatisfied(10L, TimeUnit.SECONDS, toKcxMock);
}
}
When I run the test with the webservice and ActiveMQ available then everything
works as expected.
The assertions fail however as the mock endpoints don't register any messages.
If I disable the ActiveMQ broker, then I get 'Connection refused' exceptions
from the ActiveMQ component.
As far as I understand Camel shouldn't have tried to send the messages to
ActiveMQ though because of the @MockEndpointsAndSkip annotation.
What am I missing?
Thanks for any suggestion,
Kai
IMPORTANT NOTICE: This email is intended solely for the use of the individual
to whom it is addressed and may contain information that is privileged,
confidential or otherwise exempt from disclosure under applicable law. If the
reader of this email is not the intended recipient or the employee or agent
responsible for delivering the message to the intended recipient, you are
hereby notified that any dissemination, distribution, or copying of this
communication is strictly prohibited. If you have received this communication
in error, please immediately return the original message to the sender at the
listed email address. In accordance with Kewill policy, emails sent and
received may be monitored. Although Kewill takes reasonable precautions to
minimize the risk, Kewill accepts no responsibility for any loss or damage
should this email contain any virus, or similar destructive or mischievous code.
Pick latest file and ignore the remaining files for a given date
Hi All,
I have a camel route that is a file consumer and relies on the include
format with date and filename in it. The actual files arrive with an
additional timestamp. The camel route is started and stopped dynamically.
When the camel route starts, i need to pick the latest file which i can
achieve using sortBy=reverse:file:modified.
The problem is that I need to process only the latest file
Ex:
from("file:{srcDir}?include=.*[0-9]{8}_[0-9]{6}_fileName.*")
.bean("myBean", "myProcessMethod");
Sample file names
20141105_063012_fileName.txt
20141105_123012_fileName.txt
In this scenario, I wish to process only 20141105_123012_fileName.txt and
ignore the file 20141105_063012_fileName.txt.
Please suggest the best approach.
Thanks
Dwaraka
--
View this message in context:
http://camel.465427.n5.nabble.com/Pick-latest-file-and-ignore-the-remaining-files-for-a-given-date-tp5773806.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Get Socket Status of Netty4 Route
Hi there, i`m currently working on a project with several Netty4 routes to several servers. For an overview i want to show all connected sockets. Is there a possibility to get all socket connections for the routes ? Or capture connection loss events etc. Greetings, Leomar -- View this message in context: http://camel.465427.n5.nabble.com/Get-Socket-Status-of-Netty4-Route-tp5773807.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Get Socket Status of Netty4 Route
the producer definition is e.g. netty4:tcp://127.0.0.1:4021?clientMode=false&textline=false&?decoders=#plcDelimiterDecoder4&reuseAddress=false&sync=false -- View this message in context: http://camel.465427.n5.nabble.com/Get-Socket-Status-of-Netty4-Route-tp5773807p5773809.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Pick latest file and ignore the remaining files for a given date
You can use the filter eip, and then if the batch index == 1 then only
call the bean
from
filter batch index == 1
bean
end
On Fri, Nov 13, 2015 at 1:29 PM, dwarakart wrote:
> Hi All,
>
> I have a camel route that is a file consumer and relies on the include
> format with date and filename in it. The actual files arrive with an
> additional timestamp. The camel route is started and stopped dynamically.
> When the camel route starts, i need to pick the latest file which i can
> achieve using sortBy=reverse:file:modified.
>
> The problem is that I need to process only the latest file
> Ex:
> from("file:{srcDir}?include=.*[0-9]{8}_[0-9]{6}_fileName.*")
> .bean("myBean", "myProcessMethod");
>
> Sample file names
> 20141105_063012_fileName.txt
> 20141105_123012_fileName.txt
>
> In this scenario, I wish to process only 20141105_123012_fileName.txt and
> ignore the file 20141105_063012_fileName.txt.
>
> Please suggest the best approach.
>
> Thanks
> Dwaraka
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Pick-latest-file-and-ignore-the-remaining-files-for-a-given-date-tp5773806.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Pick latest file and ignore the remaining files for a given date
Awesome. Thanks a lot Claus for the quick response -- View this message in context: http://camel.465427.n5.nabble.com/Pick-latest-file-and-ignore-the-remaining-files-for-a-given-date-tp5773806p5773826.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: sftp endpoint is not as performant as expected
The 5 second check interval was something that I believe was introduced after 2.8.2. I don't see anything from the uri that would cause it either. We do explicitly set maxMessagesPerPoll however according to the documentation if not set it should default to unlimited, so that likely isn't the issue either but maybe worth a shot. -- View this message in context: http://camel.465427.n5.nabble.com/sftp-endpoint-is-not-as-performant-as-expected-tp5773654p5773862.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: sftp endpoint is not as performant as expected
Hi Yeah the current changed read-lock afair don't keep state between polls. So we could look into one that does that, and do a full scan of all the files, and then do a change detect on all files all together and make up which ones hasn't changed. And that way can react faster than currently. Now that may require doing more file directory listings to gather all those files and their timestamps / size to see which one has changed, instead of monitoring a single file one by one. Also it may mean that files can be processed out of order, if a file sort isn't must be strictly followed. On Thu, Nov 12, 2015 at 10:05 PM, David Hoffer wrote: > I'm rather new to Camel, I'm assuming that if the SFTP source gets 900 > files then the configured > GenericFileExclusiveReadLockStrategy#acquireExclusiveReadLock will get > called 900 times (once per file) every polling cycle. > > We too have a custom GenericFileExclusiveReadLockStrategy class that does > not block. What ours does is maintain a map/cache of of the > lastModifiedTime and fileSize for each of the 900 files. The call to > acquireExclusiveReadLock() will return true as soon as neither of those > values changed since the last poll cycle. > > We have our delay/poll cycle set to 60 seconds. I'm not sure why such a > large value but we are fine waiting 2 minutes to get each of the 900 > files...and by the time 2 minutes go by we will likely have more files in > the SFTP source. E.g. if we have a constant delay in receiving each file > of 2-3 minutes that is more than fine. The problem is that on each cycle > it only processes 3-5 files instead of the 900 that should have returned > true from acquireExclusiveReadLock() > > What seems to be happening is that Camel is not calling the > acquireExclusiveReadLock() method for each of the 900 files every 60 > seconds, rather it slows down and either calls just a few of those 900 > files or none. > > Any ideas? > > -Dave > > Btw, I'm using Camel 2.8.2 > > > > On Thu, Nov 12, 2015 at 12:43 PM, pmmerritt wrote: > >> We had a similar issue and traced it down to our usage of >> SftpChangedExclusiveReadLockStrategy which has a default check interval of >> 5 >> seconds which causes the polling to slow down waiting on files to finish >> being written, so max you can do is about 1 per 5 seconds. You can change >> the checkInterval so that it is faster, or we ended up writing our own read >> lock strategy that did not block the polling thread >> >> >> >> -- >> View this message in context: >> http://camel.465427.n5.nabble.com/sftp-endpoint-is-not-as-performant-as-expected-tp5773654p5773780.html >> Sent from the Camel - Users mailing list archive at Nabble.com. >> -- Claus Ibsen - http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
Re: Start/Stop CamelContext
Hi Yeah a full redeploy is safer. But if you only want to redeploy one or more routes. You would need to stop the route(s) then remove the route(s) and then add them again to the running camel context. There is API on CamelContext to do that from java code. On Fri, Nov 13, 2015 at 5:31 AM, Sashika wrote: > Hi, > I'm in need to load some camel route attributes like file paths etc from the > database and I'm initializing CamelContext from within spring. Also I use > java > DSL rather than XML DSL. I need camel to pickup the new values if I change > those > in the database. > 1. Do I have to restart the camel context in order for the new attributes to >take effect? > 2. If so how to start/stop CamelContext from within a Spring web application > > Any help is appreciated. > Regards Sashika. -- Claus Ibsen - http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
Re: Consume messages & rollback
hello, Okay even if I use spring transaction, i still don't get it if the processing is fail safe even if I set Ack to false For example I have a route as below and even a dead letter channel configured. In the error handler I am logging the message in a DB in case if it was not delivered. But what will happen in case the message was not logged in the DB due to some environment error etc. What happens to the message , will the Active MQ get the ACK . Because as far I have read in the documentation of camel , as soon as the Processor has finished execution the ack is sent back. (Hope I am right) even if there was a error or not after it . How can my scenario be handled in a effective way. http://camel.apache.org/schema/spring";> http://camel.apache.org/schema/spring";> regards, Felix T -- View this message in context: http://camel.465427.n5.nabble.com/Consume-messages-rollback-tp5773638p5773865.html Sent from the Camel - Users mailing list archive at Nabble.com.
Cannot install camel-swagger through Karaf
I'm not sure where to report this bug: Karaf or Camel. When I add the Camel repository and use "feature:install camel-swagger", I get the following error: Error executing command: Can't install feature camel-swagger/0.0.0: Could not start bundle mvn:com.fasterxml.jackson.module/jackson-module-scala_2.10/2.6.1 in feature(s) camel-swagger-2.17-SNAPSHOT: Unresolved constraint in bundle com.fasterxml.jackson.module.jackson.module.scala [99]: Unable to resolve 99.0: missing requirement [99.0] osgi.wiring.package; (&(osgi.wiring.package=com.fasterxml.jackson.module.paranamer)(version>=2.6.0)(!(version>=3.0.0))) I was able to fix this by manually installing the following bundles: mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.javax-inject/1_2 wrap:mvn:com.thoughtworks.paranamer/paranamer/2.6 mvn:com.fasterxml.jackson.module/jackson-module-paranamer/2.6.1 These seem to be missing dependencies from the camel feature file. -- Matt Sicker
Re: Cannot install camel-swagger through Karaf
Yeah use camel-swagger-java from Camel 2.16 onwards. The old scala based is deprecated and dont work really in OSGi. On Fri, Nov 13, 2015 at 6:17 PM, Matt Sicker wrote: > I'm not sure where to report this bug: Karaf or Camel. When I add the Camel > repository and use "feature:install camel-swagger", I get the following > error: > > Error executing command: Can't install feature camel-swagger/0.0.0: > Could not start bundle > mvn:com.fasterxml.jackson.module/jackson-module-scala_2.10/2.6.1 in > feature(s) camel-swagger-2.17-SNAPSHOT: Unresolved constraint in bundle > com.fasterxml.jackson.module.jackson.module.scala [99]: Unable to resolve > 99.0: missing requirement [99.0] osgi.wiring.package; > (&(osgi.wiring.package=com.fasterxml.jackson.module.paranamer)(version>=2.6.0)(!(version>=3.0.0))) > > I was able to fix this by manually installing the following bundles: > mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.javax-inject/1_2 > wrap:mvn:com.thoughtworks.paranamer/paranamer/2.6 > mvn:com.fasterxml.jackson.module/jackson-module-paranamer/2.6.1 > > These seem to be missing dependencies from the camel feature file. > > -- > Matt Sicker -- Claus Ibsen - http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
Possible issue with 2.16 SFTPConsumer startup
We ran into a scenario that I am not sure if the behavior is desired but wanted to mention it. The SFTPConsumer in 2.16 added support for the autoCreate option which is enabled by default. If for some reason the SFTP server is down or the directory can't be created and throwExceptionOnConnectFailed is true, then an exception is thrown which results in the CamelContext being shutdown. I wasn't sure if this is desired or not. I know for us we didn't want that behavior, but we also didn't need autoCreate so we just disabled it. -- View this message in context: http://camel.465427.n5.nabble.com/Possible-issue-with-2-16-SFTPConsumer-startup-tp5773870.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: sftp endpoint is not as performant as expected
+1 -- View this message in context: http://camel.465427.n5.nabble.com/sftp-endpoint-is-not-as-performant-as-expected-tp5773654p5773879.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Cannot install camel-swagger through Karaf
Thanks, that installs properly. On 13 November 2015 at 11:29, Claus Ibsen wrote: > Yeah use camel-swagger-java from Camel 2.16 onwards. The old scala > based is deprecated and dont work really in OSGi. > > On Fri, Nov 13, 2015 at 6:17 PM, Matt Sicker wrote: > > I'm not sure where to report this bug: Karaf or Camel. When I add the > Camel > > repository and use "feature:install camel-swagger", I get the following > > error: > > > > Error executing command: Can't install feature camel-swagger/0.0.0: > > Could not start bundle > > mvn:com.fasterxml.jackson.module/jackson-module-scala_2.10/2.6.1 in > > feature(s) camel-swagger-2.17-SNAPSHOT: Unresolved constraint in bundle > > com.fasterxml.jackson.module.jackson.module.scala [99]: Unable to resolve > > 99.0: missing requirement [99.0] osgi.wiring.package; > > > (&(osgi.wiring.package=com.fasterxml.jackson.module.paranamer)(version>=2.6.0)(!(version>=3.0.0))) > > > > I was able to fix this by manually installing the following bundles: > > > mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.javax-inject/1_2 > > wrap:mvn:com.thoughtworks.paranamer/paranamer/2.6 > > mvn:com.fasterxml.jackson.module/jackson-module-paranamer/2.6.1 > > > > These seem to be missing dependencies from the camel feature file. > > > > -- > > Matt Sicker > > > > -- > Claus Ibsen > - > http://davsclaus.com @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 > -- Matt Sicker
