Hi Harish, Thanks for quick response..
No, Actually I did not try flume-ng. Flume-og was working good for me.. Just curious to know does flume-ng supports encrypted channel ? Thanks Chandrashekhar On Aug 7, 2012, at 1:11 PM, Hari Shreedharan wrote: > Chandrashekhar, > > Is there a specific reason you cannot use Flume NG. I believe that has > already been suggested on this thread. Flume OG is no longer actively > developed, and it is better to move to Flume NG - which had its second > release a few weeks back - Flume v 1.2.0. > > > Thanks > Hari > > -- > Hari Shreedharan > > > On Tuesday, August 7, 2012 at 12:16 AM, Chandrashekhar Shaw wrote: > >> Hi >> >> I was testing flume for log transfer in DFO mode. I was firing the single >> shot command as cron every 10th minute from the agent box. So the flume >> process starts every 10 minutes sends the logs to collector box and ends. >> >> It was pushing the logs very well. To test the reliability I stopped the >> collector and brought it back after 1 hour, what I see is >> >> * The flume process (started by the agent cron) kept on running in the agent >> machine and not able to transfer the rest of the file to collector. >> * The process did not end even after 5/6 hours. (I had to manually kill it). >> >> As I know flume agent should continue sending the log once the collector is >> re-started. ??? >> Is there something I am missing ?? >> >> Thanks >> Chandrashekhar Shaw >> >> >> >> >> On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote: >> >> Hello, >> >> Not too sure about Flume OG - I barely used it for a week before shifting to >> NG. Would there be any particular reason you are using OG? >> >> Yes, you can take a look at SyslogTcpSource (in NG), >> https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html >> >> This can be modified according to this Netty example (for asynchronous large >> file streaming) : >> http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html >> >> This is pretty much what I did when I needed a Flume HTTP source to match my >> requirements. >> >> Regards, >> Harish >> >> >> On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw >> <[email protected]<mailto:[email protected]>> wrote: >> I am using Flume - OG >> https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz >> . >> >> It there an available version which implements netty?? >> >> Thanks >> Chandrashekhar Shaw >> >> >> On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote: >> >> Hi, >> >> Which one of the sources are you using? Maybe a flume source that >> implements something like >> http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful. >> >> Regards, >> Harish >> >> >> On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw >> <[email protected]<mailto:[email protected]>>wrote: >> >> Hi, >> >> I am using flume for transferring logs from 20 agent boxes to 1 collector >> box. The file size which I am trying to transfer is of about 1 GB per hour. >> So every hour my agents run to transfer the log file generated during the >> last hour. >> I am using DFO sink for file transfer. >> >> But then I am facing following problems: >> >> 1) Some of the agents are not able to transfer the whole file even in one >> hour and the processes run continuously for hours. >> 2) Some of the agents stop within 5-6 minutes without sending the file >> completely. >> >> I want to know >> >> 1) if somebody else has faced the same issues? >> 2) What could be the problem? >> >> Thanks >> Chandrashekhar Shaw >> >> > >
