Hi All,
Recently I faced a problem in HDFS producer with files which have special
characters in the name. Apparently,when we create hadoop file system in
HdfsInfo.java and pass the hdfsPath(hdfs://ip:port/filename,file name is
abc}123.txt) it is unable to create the URI and it throws some excep
ce somehow.
Can you reproduce the issue without storm?
And how do you store in hdfs, do you use camel-hdfs or something else?
On Sun, Aug 31, 2014 at 10:25 AM, Chirag Dewan wrote:
>
>
> Hi All,
>
>
>
> I have a route which polls the files from FTP server and stores them in H
Hi All,
I have a route which polls the files from FTP server and stores them in HDFS.
Now when I shut down my datanodes in HDFS and try to poll a file,I get an
uncaught RuntimeException and my process dies(storm worker).
Is there any way I can catch the CamelRuntimeException,maybe log the e
something else?
On Wed, Aug 6, 2014 at 12:47 PM, Chirag Dewan wrote:
> Hi Claus,
>
> I have stopped my application,thus stopping CamelContext. But still I can see
> the Spool directory.
>
>
> Dont know if I am doing something wrong. Can this be a permissions issue?
> Wou
apache.org/stream-caching.html
You can turn this off with removeSpoolDirectoryWhenStopping
On Wed, Aug 6, 2014 at 10:25 AM, Chirag Dewan wrote:
> Hi Claus,
>
> Sorry for the delay to try the fix and revert.
>
> I actually upgraded the camel to 2.13.0. And switched from hdfs2 to fil
- can't
quite recall what the bug was, but just that it was in the .0 release.
On Wed, Jul 30, 2014 at 11:10 AM, Chirag Dewan wrote:
> Hi Camel Riders,
>
> I have a route like :
>
> from("ftp:someip").process("someprocessor").streamCaching().to("hdf
Hi Camel Riders,
I have a route like :
from("ftp:someip").process("someprocessor").streamCaching().to("hdfs2:someip");
Now I can see that there are a lot of temp files being created in the root
directory for caching which are not deleted even after a 1000's of files are
picked from FTP server.
Re: Using Stream Caching with FTP Consumer
The file metadata is stored as headers so you find them there.
On Mon, Jul 28, 2014 at 10:18 AM, Chirag Dewan wrote:
> Hi All,
>
> I have a route which consumes files from FTP server and distributes them to
> HDFS.
>
>
> Now,I h
Hi All,
I have a route which consumes files from FTP server and distributes them to
HDFS.
Now,I have a processor in which I want to rename a file and append properties
like file size and timestamp in the file name. Earlier,I was doing it by
getting the GenericFile instance from exchange body
ing Direct Component
I am not sure, maybe try setting the openedSuffix to an empty string
to see if that works?
On Thu, Jun 19, 2014 at 8:39 AM, Chirag Dewan wrote:
> Hi Claus,
>
> Thanks for response.
>
> I am using Camel 2.12.1 with camel-hdfs 2.
>
> Yes,that is the pro
Hi,
You want the exact same folder structure to be created on the destination
server?
If not,you can use recursive property of File component. It will poll the files
in sub folders too.
Chirag
From: skasish
To: users@camel.apache.org
Sent: Thursday, 19 Ju
: Claus Ibsen
To: "users@camel.apache.org"
Sent: Thursday, 19 June 2014 11:37 AM
Subject: Re: A basic query with using Direct Component
On Thu, Jun 19, 2014 at 7:11 AM, Chirag Dewan wrote:
> Hi All,
>
> I have a route with direct as following :
>
> from("direct:abc&q
Hi All,
I have a route with direct as following :
from("direct:abc").to("hdfs2:x?splitStrategy=MESSAGE");
I use the route write files in HDFS. I use the producer template to send a
OutputStream to direct:
producerTemplate,sendBody("direct:abc",OutputStream);
Now HDFS producer creates an .
Hello Camel Riders,
I have a route :
from("ftp:/user@host:port/path?streamdownload=true").process("File Renaming
Processor").to("file:/path);
Now this works fine for 10-15k files. After that no more files are consumed. I
used JMC to profile my application. I can see that the consumer thread is
Hi Claus,
Thanks for the quick reply.
I have to consume all the files upon a command,and then stop the route from
continuously polling the files untill the next command comes. So,can I also
stop the control bus once all my files are consumed? Maybe from within the
route?
BR,
Chirag Dewan
eateProducerTemplate().sendBody("seda:abc",null/"");
BR,
Chirag Dewan
Hi,
You can try the log component.
http://camel.apache.org/log.html
BR,
Chirag
From: Rural Hunter
To: users@camel.apache.org
Sent: Wednesday, 26 March 2014 7:20 AM
Subject: Error handler to both logger and dead letter queue?
Hi,
I configured the error
Java file reading here.
Thanks a lot for you help.
Chirag Dewan
From: Chirag Dewan
To: "users@camel.apache.org"
Sent: Thursday, 20 February 2014 4:06 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
Hi,
After setting the max cache fo
cache-size-for-producercache-or-producertemplate.html
You can set it to 1 or 0 to not have any pooling.
On Thu, Feb 20, 2014 at 10:53 AM, Chirag Dewan wrote:
> Hi Claus,
>
> Sorry for the late reply.
>
> I wrote a simple code to test this behaviour:
>
> CamelContex
And when I call template.stop(),all the threads are destroyed.
From: Chirag Dewan
To: "users@camel.apache.org"
Sent: Thursday, 20 February 2014 3:23 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
Hi Claus,
Sorry for the late reply
ate.doneUoW(exchange);
i++;
}
template.stop();
Even on profiling this I see a thread being created for every file consumed by
the template. And the memory continous to grow.
Thanks.
Chirag Dewan
From: Claus Ibsen
doneUoW the
inprogressRepository cache is cleared. Which is not getting cleared at
the moment(the reason I believe is causing the OOM) . Is there a possibility
that doneUoW is not working as it should?
Thanks .
Chirag
From: Chirag Dewan
To: "
users@camel.apache.org" ; Chirag Dewan
Sent: Monday, 17 February 2014 1:28 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
On Mon, Feb 17, 2014 at 8:49 AM, Chirag Dewan wrote:
> Hi Claus,
>
> So I dont need to start/stop the template for every file?
>
> Right now I create a co
t too
http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan wrote:
> Hi All,
>
> Any views on this?
>
> Actually my previous mail is not understandable. Let me rephrase the issue.
>
> I am using consumer
.
Is there something I am missing here?
Thanks in advance.
Chirag Dewan
From: Chirag Dewan
To: "users@camel.apache.org" ; Chirag Dewan
Sent: Friday, 14 February 2014 3:22 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
Hi All,
I
inProgressRepository? And when is it cleared from the memory?
Any help is much appreciated.
Thanks!
Chirag Dewan
From: Chirag Dewan
To: "users@camel.apache.org" ; Chirag Dewan
Sent: Wednesday, 12 February 2014 6:31 PM
Subject: Re: OOM is
Hi,
And I am using readLock=changed. So that can be a reason too?
Chirag Dewan
From: Chirag Dewan
To: "users@camel.apache.org"
Sent: Wednesday, 12 February 2014 6:07 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
Hi Claus,
Than
>org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
I maybe doing something wrong here. Is there anything else I need to do to make
sure that file is deleted after it has been processed and not stored in the
cache?
Thanks.
Chirag De
ory used,even if I am deleting the files and not storing
in memory? Is there a way I can avoid that?
I am testing this scenario under Storm as execution environment and with 2GB
max heap space.
Thanks in advance
Chirag Dewan
r details.
Regards,
Richard
On Wed, Feb 5, 2014 at 5:23 AM, Chirag Dewan wrote:
> Hi All,
>
> Any views on this?
>
> BR,
>
> Chirag
>
>
at causes threads to be spawned. You can
try to just do from ftp -> to file or from ftp -> to log and see if
there is some changes.
All the threads pool used by Camel are enlisted in JMX which you can
see. Maybe that can also give some pointers.
On Tue, Feb 4, 2014 at 12:33 PM, Chirag
Hi All,
Any views on this?
BR,
Chirag
Hi All,
I am using Camel 2.12.1. I have a route which consumes files from a FTP Server
and places them on the local disk.
from("ftp:admin@ip:port/source_dir&throwExceptionOnConnectFailed=true&stepwise=false&readLock=changed&fastExistsCheck=true&localWorkDirectory=destination_dir/.temp").to("fil
Hi All,
Does FTP Consumer support autoCreate and startingDirectoryMustExist options?
I was expecting FTP consumer to throw an exception if starting directory does
not exist(autoCreate=false and startingDirectoryMustExist=true) just like File
Consumer.
Thanks
Chirag Dewan
exception and thus stop polling and ?
Do I have to create Consumer template here? And if I do that how can I just
poll the FTP server without polling and downloading a file,since my endpoint
deletes the file.
Thanks
Chirag Dewan
there something else I need to do to achieve this.
Thanks
Chirag Dewan
Hi Claus,
So is there a language expression like date which I can use here or I have to
use a processor/bean?
Thanks for your help.
Sent from Yahoo Mail on Android
want the files with increasing sequence.
Thanks!
Chirag Dewan
want the files with increasing sequence.
Thanks!
Chirag Dewan
Hi all,
This may be a very straight forward thing,but I was not able to find it.
I have a route like:
public class MyRouteBuilder extends RouteBuilder
{
private MyObject myObject;
public MyRouteBuilder(MyObject myObject)
{
this.myObject = myObject;
}
@Override
Hi All,
I am using Apache Camel 2.12.1. I have a use case where one of my thread is
outputting data to a stream. I need a camel consumer which consumes this stream
from that thread and produces a file using camel-file or camel-ftp.
Looking at the components,I wonder what I can effectively use h
Hi All,
Any comments on the query?
Thanks
Chirag
Sent from Yahoo Mail on Android
Hi All,
I am working on an application where we intend to use camel Endpoints for most
of the components for eg. FTP,SMPP,HTTP,Servlet,Jetty and many more. The idea
is to create separate wrapper components for all the endpoints and then use
them as libraries in our project.
Now all my compo
Hi All,
I am working on an application where we intend to use camel Endpoints for most
of the components for eg. FTP,SMPP,HTTP,Servlet,Jetty and many more. The idea
is to create separate wrapper components for all the endpoints and then use
them as libraries in our project.
Now all my compon
44 matches
Mail list logo
