Try enabling debug logging. Also yes, use latest trunk. And make sure the
directory contents for the checkpoint and data dirs are empty… the format
changed on trunk last night. So if you need to keep data around that is in
existing file channels make sure to start up without a source and drain the
channel to another agent or file before shutting it down and deleting the file
channel data directories.
Side note: not sure putting the file channel data dirs on /tmp is safe for
production, but it depends on your setup. Some operating systems purge /tmp on
startup.
Regards,
Mike
On Tuesday, June 26, 2012 at 6:30 PM, Hari Shreedharan wrote:
> Is this on trunk? If you rebase on trunk, you should be ok.
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
>
> On Tuesday, June 26, 2012 at 6:08 PM, Kamal Bahadur wrote:
>
> > Hi,
> >
> > I am using the latest code from the HEAD. Following is my configuration: I
> > am using file channel and it looks like flume-ng gets stuck during startup
> > of file-channel. No errors found in the log as usual.
> >
> > I have already checked for permissions on the directories and they look ok.
> >
> > Thanks,
> >
> > Kamal
> >
> > agent.sources = avroSrc
> > agent.channels = fileChannel
> > agent.sinks = cassandraSink
> >
> > agent.sources.avroSrc.type = avro
> > agent.sources.avroSrc.bind = 0.0.0.0
> > agent.sources.avroSrc.port = 8800
> >
> > agent.sources.avroSrc.channels = fileChannel
> >
> > agent.sinks.cassandraSink.type = com.glsink.cassandra.CassandraSink
> > agent.sinks.cassandraSink.nodes = host1:9160,host2:9160,host3:9160
> > agent.sinks.cassandraSink.cluster = GL CLuster
> > agent.sinks.cassandraSink.keyspace = Global_Logging
> > agent.sinks.cassandraSink.columnFamily = Audit_Log
> > agent.sinks.cassandraSink.loadBalancingPolicy =
> > me.prettyprint.cassandra.connection.RoundRobinBalancingPolicy
> > agent.sinks.cassandraSink.username = USERDEV
> > agent.sinks.cassandraSink.password = pwd152==
> > agent.sinks.cassandraSink.batch-size = 100
> >
> > agent.sinks.cassandraSink.channel = fileChannel
> >
> > agent.channels.fileChannel.type = file
> >
> > agent.channels.fileChannel.checkpointDir = /tmp/checkpoint
> > agent.channels.fileChannel.dataDirs = /tmp/data
> >
> >
> >
> > Debug message:
> >
> > 2012-06-25 16:12:18,588 DEBUG conf.FlumeConfiguration: Starting validation
> > of configuration for agent: agent, initial-configuration:
> > AgentConfiguration[agent]
> > SOURCES: {avroSrc={ parameters:{port=8800, channels=fileChannel, type=avro,
> > bind=0.0.0.0} }}
> > CHANNELS: {fileChannel={ parameters:{checkpointDir=/tmp/checkpoint,
> > dataDirs=/tmp/data, type=FILE} }}
> > SINKS: {cassandraSink={
> > parameters:{loadBalancingPolicy=me.prettyprint.cassandra.connection.RoundRobinBalancingPolicy,
> > username=USERDEV , batch-size=100, nodes=
> > <http://10.136.239.61:9160><removed>,
> > keyspace=Global_Logging, cluster=GL CLuster, columnFamily=Audit_Log,
> > type=com.glsink.cassandra.CassandraSink, channel=fileChannel,
> > password=pwd152==} }}
> >
> >
> > 2012-06-25 16:12:18,594 DEBUG conf.FlumeConfiguration: Created channel
> > fileChannel
> > 2012-06-25 16:12:18,608 DEBUG conf.FlumeConfiguration: Creating sink:
> > cassandraSink using OTHER
> > 2012-06-25 16:12:18,610 DEBUG conf.FlumeConfiguration: Post validation
> > configuration for agent
> > AgentConfiguration created without Configuration stubs for which only basic
> > syntactical validation was performed[agent]
> > SOURCES: {avroSrc={ parameters:{port=8800, channels=fileChannel, type=avro,
> > bind=0.0.0.0} }}
> > CHANNELS: {fileChannel={ parameters:{checkpointDir=/tmp/checkpoint,
> > dataDirs=/tmp/data, type=FILE} }}
> > SINKS: {cassandraSink={
> > parameters:{loadBalancingPolicy=me.prettyprint.cassandra.connection.RoundRobinBalancingPolicy,
> > username=USERDEV , batch-size=100, nodes=
> > <http://10.136.239.61:9160><removed>,
> > keyspace=Global_Logging, cluster=GL CLuster, columnFamily=Audit_Log,
> > type=com.glsink.cassandra.CassandraSink, channel=fileChannel,
> > password=pwd152==} }}
> >
> > 2012-06-25 16:12:18,610 DEBUG conf.FlumeConfiguration: Channels:fileChannel
> >
> > 2012-06-25 16:12:18,610 DEBUG conf.FlumeConfiguration: Sinks cassandraSink
> >
> > 2012-06-25 16:12:18,610 DEBUG conf.FlumeConfiguration: Sources avroSrc
> >
> > 2012-06-25 16:12:18,610 INFO conf.FlumeConfiguration: Post-validation flume
> > configuration contains configuration for agents: [agent]
> > 2012-06-25 16:12:18,610 INFO
> > properties.PropertiesFileConfigurationProvider: Creating channels
> > 2012-06-25 16:12:18,610 DEBUG channel.DefaultChannelFactory: Creating
> > instance of channel fileChannel type FILE
> > 2012-06-25 16:12:18,624 INFO
> > properties.PropertiesFileConfigurationProvider: created channel fileChannel
> > 2012-06-25 16:12:18,624 DEBUG source.DefaultSourceFactory: Creating
> > instance of source avroSrc, type avro
> > 2012-06-25 16:12:18,646 INFO sink.DefaultSinkFactory: Creating instance of
> > sink cassandraSink
> > typecom.intuit.ifs.globallogging.glsink.cassandra.CassandraSink
> > 2012-06-25 16:12:18,647 DEBUG sink.DefaultSinkFactory: Sink type
> > com.intuit.ifs.globallogging.glsink.cassandra.CassandraSink is a custom
> > type
> > 2012-06-25 16:12:18,665 DEBUG cassandra.CassandraSink:
> > 2012-06-25 16:12:18,665 INFO cassandra.CassandraSink: Configiring Cassandra
> > sink
> > 2012-06-25 16:12:18,665 INFO cassandra.CassandraSink: Configured Cassandra
> > sink
> > 2012-06-25 16:12:18,667 INFO nodemanager.DefaultLogicalNodeManager:
> > Starting new configuration:{
> > sourceRunners:{avroSrc=EventDrivenSourceRunner: { source:AvroSource: {
> > bindAddress:0.0.0.0 port:8800 } }} sinkRunners:{cassandraSink=SinkRunner: {
> > policy:org.apache.flume.sink.DefaultSinkProcessor@7a148bd3 counterGroup:{
> > name:null counters:{} } }}
> > channels:{fileChannel=org.apache.flume.channel.file.FileChannel@5ae80842} }
> > 2012-06-25 16:12:18,667 INFO nodemanager.DefaultLogicalNodeManager:
> > Starting Channel fileChannel
> > 2012-06-25 16:12:18,667 INFO nodemanager.DefaultLogicalNodeManager: Waiting
> > for channel: fileChannel to start. Sleeping for 500 ms
> > 2012-06-25 16:12:18,668 INFO file.FileChannel: Starting FileChannel with
> > dataDir [/tmp/data]
> > 2012-06-25 16:12:19,170 INFO nodemanager.DefaultLogicalNodeManager: Waiting
> > for channel: fileChannel to start. Sleeping for 500 ms
> > 2012-06-25 16:12:19,672 INFO nodemanager.DefaultLogicalNodeManager: Waiting
> > for channel: fileChannel to start. Sleeping for 500 ms
> > 2012-06-25 16:12:20,175 INFO nodemanager.DefaultLogicalNodeManager: Waiting
> > for channel: fileChannel to start. Sleeping for 500 ms
> > 2012-06-25 16:12:20,678 INFO nodemanager.DefaultLogicalNodeManager: Waiting
> > for channel: fileChannel to start. Sleeping for 500 ms
> > 2012-06-25 16:12:21,180 INFO nodemanager.DefaultLogicalNodeManager: Waiting
> > for channel: fileChannel to start. Sleeping for 500 ms
>
