Re: Flume NG - preferred alternative to TailSource

Using tail command will not be useful if something goes wrong since it does not maintain the state.. Better to implemet your own parser or application that reads the file and sends the event to the flume over RPC client. Regards, Som On Wed, Jul 18, 2012 at 1:17 AM, Mohammad Tariq wrote: > Hi J

Re: flume-ng failure recovery

Moreover the reader part can also keep the list of hosts and act as an load balancer and also as failover mechanism... Regards, Som On Wed, Jul 18, 2012 at 12:13 PM, shekhar sharma wrote: > Dont use tail source since it does not maintain the state where it > left..so i would suggest something li

Re: flume-ng failure recovery

Dont use tail source since it does not maintain the state where it left..so i would suggest something like this:: (1)Implement a Reader part which reads the events from the file and also maintains the state if something goes wrong) (2) convert the events to the Flume Event type and using a RPC c

Re: flume-ng failure recovery

We use a tail -F -n0. This will result in the tail command starting at the beginning of the file and replaying all events. This will however result in duplicate events that you will need to deal with. Sent from my iPhone On Jul 17, 2012, at 11:53 PM, Jagadish Bihani wrote: > Hi > > We wan

flume-ng failure recovery

Hi We want to deploy flume-ng in the production environment in our organization. Here is the following scenario for which I am not able to find the answer: 1. We receive logs using 'tail -f' source. 2. Now the agent process gets killed. 3. We restart it. 4. How would the restarted agent will

Re: Flume NG - preferred alternative to TailSource

Hi Jon, Flume-NG provides an exec source that executes a given command and consumes the output. It should serve the purpose. Regards, Mohammad Tariq On Tue, Jul 17, 2012 at 10:52 PM, Jon Bender wrote: > Hi folks, > > I'm contemplating moving our current Flume 0.94 pipeline to Flum

Flume NG - preferred alternative to TailSource

Hi folks, I'm contemplating moving our current Flume 0.94 pipeline to Flume NG. At the moment we make heavy use of the "multitail" source with some custom plugins to open up files/archives as they arrive in a specified directory, do some parsing and then emit the parsed events. Ideally I would r

Re: Hbase Sink

Hi George, We need to add the payload column as well.Add the following line in your config file : hbase-agent.sinks.sink1.serializer.payloadColumn = col_name Regards, Mohammad Tariq On Tue, Jul 17, 2012 at 8:23 PM, George Forman wrote: > Hi all, > > Newbie here...I am trying to ge

Hbase Sink

Hi all, Newbie here...I am trying to get flume ng to do a simple hbase sink. I have downloaded/installed flume-ng and configured it based oninfo found on the net. I don't see flume connecting to hbase.I am wondering what I could be missing? - Thanks Here is my configuration: hbase-agent.sourc

would reverse SSH work with Flume?

Hi I'm trying to test if I could set up a secured SSH channel between 2 flume agents over a network, I have Agent1 with a AvroSink and Agent2 with a AvroSource which listens on localhost:10900. I've tried initiating a reverse SSH from where Agent2 resides with command ssh -NR 10900:agent2_server

Re: [VOTE] GIT as primary source control for Apache Flume

+1

Re: [VOTE] GIT as primary source control for Apache Flume

+1. Thanks, - Inder On Tue, Jul 17, 2012 at 12:15 PM, alo alt wrote: > +1 > > > On Jul 16, 2012, at 9:38 PM, Arvind Prabhakar wrote: > > > [X] +1 Accept git as the primary source control for Apache Flume. > > > > Regards, > > Arvind Prabhakar > > > > On Sun, Jul 15, 2012 at 12:22 PM, Hari Shree