Re: flume-ng syslogtcp issue

HI, try these for syslog tests: echo "<13>Jun 20 12:12:12 host foo[345]: a syslog message" > /tmp/foo nc -v FLUME SYSLOG AGENT 5140 < /tmp/foo > syslog-agent.sources.syslog.host = 172.17.1.92 Means, will only listen only on this interface, is the IP correct? cheers, Alex On Jul 10, 2012, at 1

Re: Flume - ng running issue

Hi, what exactly is the issue? Error messages, things like that. You posted only debugging and exec logs, thats pretty normal. For a first short overview about flume take a look into: https://cwiki.apache.org/FLUME/getting-started.html http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.

Re: Flume - ng running issue

FYI On Tue, Jul 10, 2012 at 6:39 PM, prabhu k wrote: > Hi, > > While i am executing flume-ng, i am getting following out put on my > terminal, it seems got struck. > > root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# > bin/flume-ng agent -n agent1 -c conf -f conf/agen

Re: Flume - ng running issue

can you please anyone respond on the below issue? On Tue, Jul 10, 2012 at 6:39 PM, prabhu k wrote: > Hi, > > While i am executing flume-ng, i am getting following out put on my > terminal, it seems got struck. > > root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# > bin

Re: File channel performance on a single disk is poor

Hi, thanks for clarifying. On 07/10/2012 06:36 PM, Arvind Prabhakar wrote: Hi, On Sun, Jul 8, 2012 at 11:14 PM, Juhani Connolly > wrote: Another matter that I'm curious of is whether or not we actually need separate files for the data and chec

flume-ng syslogtcp issue

Hey guys, I've been trying to use flume-ng (build in trunk) to read syslogs through tcp then write to a flat file, as seen in this configuration: # Source, sink, and channel name below syslog-agent.channels = rmc syslog-agent.sources = syslog syslog-agent.sinks = flat # Configure channels sysl

RE: accumulo sink

Sorry - this was not a flume issue. I had a leftover jar file from a previous version of accumulo. Got corrected. I think I am now standing up. Still testing. Again sorry. From: Martin, Ray [mailto:arma...@tycho.ncsc.mil] Sent: Tuesday, July 10, 2012 4:02 PM To: flume-u...@incubator.apac

RE: accumulo sink

I created a custom flume sink. Added appropriate jars into lib director of flume-ng. Set classpath in conf/flume-env.sh Configured conf/flume.conf as follows: # Sources, channels and sinks are defined per agent # Define a memory channel called ch1 on agent1 agent1.channels.ch1.type = memor

Re: Flume - ng running issue

Please suggest on the below issue On Tue, Jul 10, 2012 at 6:39 PM, prabhu k wrote: > Hi, > > While i am executing flume-ng, i am getting following out put on my > terminal, it seems got struck. > > root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# > bin/flume-ng agent

Re: Restarts without data loss

Yeah good point, the ExecSink does no batching and as such will be quite slow when interacting with any channel which guarantees no dataloss on a commit. On Tue, Jul 10, 2012 at 8:54 AM, Juhani Connolly < juhani_conno...@cyberagent.co.jp> wrote: > A further observation: > > When running our colle

Re: Restarts without data loss

Thanks Arvind. - Inder On Tue, Jul 10, 2012 at 2:53 PM, Arvind Prabhakar wrote: > One clarification - as Mubarak mentioned, there is already a Jira for this > FLUME-1318 . So instead > of filing a new issue, you can add your details and thoughts

Flume - ng running issue

Hi, While i am executing flume-ng, i am getting following out put on my terminal, it seems got struck. root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# bin/flume-ng agent -n agent1 -c conf -f conf/agent1.conf Info: Sourcing environment configuration script /usr/local/f

Re: File channel performance on a single disk is poor

Hi, On Sun, Jul 8, 2012 at 11:14 PM, Juhani Connolly < juhani_conno...@cyberagent.co.jp> wrote: > > Another matter that I'm curious of is whether or not we actually need > separate files for the data and checkpoints... > The data file and checkpoint files serve different purpose. Checkpoint resi

Re: Restarts without data loss

One clarification - as Mubarak mentioned, there is already a Jira for this FLUME-1318 . So instead of filing a new issue, you can add your details and thoughts to this. Regards, Arvind Prabhakar On Tue, Jul 10, 2012 at 2:20 AM, Arvind Prabhakar w

Re: Restarts without data loss

Hi Inder, On Mon, Jul 9, 2012 at 12:02 AM, Inder Pall wrote: > Arvind, > > to me this is an important use-case for frequent prod rollouts. How about > thinking in the direction of supporting graceful shutdown for agents. > I do believe that the Agent does shutdown gracefully on interrupt. Specif

Re: Restarts without data loss

A further observation: When running our collector node with avro source and hdfssink, I observed it keeping up with about 1400+ events per second. Upon looking at the exec sink I noticed it sends every item as a separate event to the processor. So I think I may have misunderstood the frequency