Tracking Backlog on FTP server

Hi All, I have a camel application, configured with multiple routes, which poll ftp servers, download and process files. I am using specific values for maxMessagesPerPoll as a kind of rate limiting, to cap the number of files processed per poll. I am looking for a solution to be able to monitor th

Re: Distributing routes on multiple camel instances

multiple-camel-contexts/ > > As a sidenote: when you need more flexible and scalable solution you might > try to use message queueing like ActiveMQ (http://activemq.apache.org/) or > build a REST API with Camel ( > https://camel.apache.org/components/latest/rest-component.html). > > R

Distributing routes on multiple camel instances

Hi All, I am currently using the camel FTP component to download files and upload them to s3. The number of routes are growing at an alarming rate i.e. each route is reading from a separate FTP server and the servers are polled multiple times a day. I currently have only one camel instance running

Using RSA byte array in ftp camel route

HI, All I have an sftp consumer and I need to use privateKey as consumer option, the use case is I need to read this key from a vault and pass it as a byte[] in the option parameter. Besides using @BindToRegistry is there any other way i can pass privateKey=byte[] while building the route in java.

Re: Route setup to upload file to s3 and put message on kafka queue

nly thing I guess is to test what happens if there are no files do > download. Should you continue polling or stop immediately? > > On Sun, Jul 17, 2016 at 7:52 PM, Sherwin Pinto [via Camel] < > ml-node+s465427n578519...@n5.nabble.com > <mailto:ml-node+s465427n578519...@n5.nab

Re: Route setup to upload file to s3 and put message on kafka queue

> predicate is true do you send the metadata to kafka. > > If you are going to do batch processing, how do you know when the batch > starts and when the batch finishes? > > On Sun, Jul 17, 2016 at 7:18 PM, Sherwin Pinto [via Camel] < > ml-node+s465427n5785193...@n5.nabble.com

Re: Route setup to upload file to s3 and put message on kafka queue

Hi All, Can someone help me with this ? Thanks Sherwin > On Jul 14, 2016, at 1:18 PM, Sherwin Pinto wrote: > > HI All, > > My use case is as follows > > Use case 1: > 1. Pole ftp server for file. > 2. Upload file to S3 > 3. Put message on to kafka with fi

Route setup to upload file to s3 and put message on kafka queue

HI All, My use case is as follows Use case 1: 1. Pole ftp server for file. 2. Upload file to S3 3. Put message on to kafka with file meta info i. e filename, S3 bucket etc. This information is then used by a separate java process to download file and process it. I have a working prototype but

Help with Topology

Hi, I am using a Trident topology to process files, transform them from CSV, EDI, XML to a general JSON format. I have a working prototype, but wanted to make sure I am implementing this correctly. Here is the flow 1. Read message from kafka, this message is meta data of the file location on S3

Re: Running Same set of Camel routes (Camel Context) on multiple nodes

Thanks Karel … I had taken a look at it. Will research it in more detail > On May 17, 2016, at 11:25 AM, Karel Bernolet wrote: > > Have you looked into the leader election integration pattern? > I think camel-zookeeper and camel-hazelcast support it. > > On Tue, May 17, 2016 a

Re: Running Same set of Camel routes (Camel Context) on multiple nodes

Hi Yogesh, Thanks for your response. I am evaluating our options . Using quartz in a clustered environment is definitely an option . Are there any other configurations you would recommend ? Thanks Sherwin > On May 17, 2016, at 7:25 AM, yogu13 wrote: > > if you are using timer to trigger the

Running Same set of Camel routes (Camel Context) on multiple nodes

Hi, I am currently running a single instance of Camel. Routes are set up to pull files from ftp servers. Camel Version: 2.15.3 Running as a standalone application using spring I am using an Idempotent Consumer (JpaMessageIdRepository) to prevent downloading duplicate files. We are looking t