Re: FTP to HDFS - large gzipped files

2011-05-27 Thread Mond Raymond
OK. It's done https://issues.apache.org/jira/browse/CAMEL-4024 -- View this message in context: http://camel.465427.n5.nabble.com/FTP-to-HDFS-large-gzipped-files-tp3192431p4433386.html Sent from the Camel - Users mailing list archive at Nabble.com.

Re: FTP to HDFS - large gzipped files

2011-05-27 Thread Claus Ibsen
Nice. Do you mind creating a jira ticket and attach you source. Then we can use it as a contribution. On Thursday, May 26, 2011, Mond Raymond wrote: > Claus, > > Just thought I would follow up. > > I don't know if this counts as a contribution, but I ended up doing this as > a splitter as in: > >

Re: FTP to HDFS - large gzipped files

2011-05-26 Thread Mond Raymond
Claus, Just thought I would follow up. I don't know if this counts as a contribution, but I ended up doing this as a splitter as in: from(fromUri) .split().method(OuterZipFileDecompressingSplitter.class, "split") .streaming() .parallelPro

Re: FTP to HDFS - large gzipped files

2011-05-10 Thread Mond Raymond
Here is a Processor that works on ZIP files: import org.apache.camel.util.ExchangeHelper; import org.apache.camel.util.IOHelper; public class ZipFileSplittingProcessor implements Processor { public void process(Exchange exchange) throws Exception { InputStream is = ExchangeHelper.get

Re: FTP to HDFS - large gzipped files

2010-10-05 Thread Claus Ibsen
On Tue, Oct 5, 2010 at 9:28 AM, coolgold wrote: > > Thank you Claus.  Couldn't gzip data format be used for unzipping? > http://camel.apache.org/gzip-data-format.html > -- Ah yeah forgot about that one. However I think it works in memory only. However its java code so you can always look at the s

Re: FTP to HDFS - large gzipped files

2010-10-05 Thread coolgold
Thank you Claus. Couldn't gzip data format be used for unzipping? http://camel.apache.org/gzip-data-format.html -- View this message in context: http://camel.465427.n5.nabble.com/FTP-to-HDFS-large-gzipped-files-tp3192431p3198933.html Sent from the Camel - Users mailing list archive at Nabble.c

Re: FTP to HDFS - large gzipped files

2010-10-05 Thread Claus Ibsen
On Mon, Oct 4, 2010 at 7:14 PM, coolgold wrote: > > I'm considering using camel and activemq for moving gzipped (~1gb) files form > ftp server to hdfs. I'm aware of existence of ftp and hdfs components, but > I'm not sure about the support for splitting for gzipped files or streaming > method with