Logs ?
On Mon, Feb 17, 2014 at 5:51 AM, Kris Ogirri <[email protected]> wrote: > Dear Mailing Group, > > I am currently having issues with the Hbase sink function. I have developed > an agent with a fanout channel setup ( single source, multiple channels, > multiple sinks) sinking to a HDFS cluster and Hbase deployment. > > The issue is that although the HDFS is working well, the Hbase flow is > simply not working. There are no errors being reported by Flume for the > Hbase channel but there are never any records being written to the HBase > store. The Hbase table as stipulated in the config always remains empty. > Studying the Flume startup logs I observe that the session connection to > Zookeeper is always successfully established > > Are there any special configurations I am missing out? > > I am using the Async Event Serializer to persist the txns. > > Any assistance will be greatly appreciated. > > > Please see below for the flume configuration: > > [biadmin@bivm bin]$ cat flume-conf.properties.bigdemo > agent.sources=exec-source > agent.sinks=hdfs-sink hbase-sink > agent.channels=ch1 ch2 > > agent.sources.exec-source.type=exec > agent.sources.exec-source.command=tail -F > /home/biadmin/bigdemo/data/rec_telco.cdr > > agent.sinks.hdfs-sink.type=hdfs > agent.sinks.hdfs-sink.hdfs.path=hdfs://XXXX:9000/user/biadmin/bigdemo/ > agent.sinks.hdfs-sink.hdfs.filePrefix=telco_cdr_rec > # File size to trigger roll, in bytes (0: never roll based on file size) > agent.sinks.hdfs-sink.hdfs.rollSize = 134217728 > agent.sinks.hdfs-sink.hdfs.rollCount = 0 > # number of events written to file before it flushed to HDFS > agent.sinks.hdfs-sink.hdfs.batchSize = 10000 > agent.sinks.hdfs-sink.hdfs.txnEventMax = 40000 > > > agent.sinks.hbase-sink.type=org.apache.flume.sink.hbase.AsyncHBaseSink > agent.sinks.hbase-sink.serializer=org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer > agent.sinks.hbase-sink.table=telco_cdr_rec > agent.sinks.hbase-sink.columnFamily = colfam > agent.sinks.hbase-sink.channels = ch2 > #agent.sinks.hbase-sink.hdfs.batchSize = 10000 > #agent.sinks.hbase-sink.hdfs.txnEventMax = 40000 > > > agent.channels.ch1.type=file > agent.channels.ch1.checkpointInterval=3000 > agent.channels.ch1.transactionCapacity=10000 > agent.channels.ch1.checkpointDir=/home/BDadmin/.flume/file-channel/checkpoint > agent.channels.ch1.dataDirs=/home/BDadmin/.flume/file-channel/data > agent.channels.ch1.write-timeout=30 > agent.channels.ch1.keep-alive=30 > #agent.channels.ch1.capacity=1000 > > agent.channels.ch2.type=file > agent.channels.ch2.checkpointInterval=300 > agent.channels.ch2.transactionCapacity=10000 > agent.channels.ch2.checkpointDir=/home/BDadmin/.flume/file-channel2/checkpoint > agent.channels.ch2.dataDirs=/home/BDadmin/.flume/file-channel2/data > agent.channels.ch2.write-timeout=30 > agent.channels.ch2.keep-alive=30 > #agent.channels.ch2.capacity=1000 > > > agent.sources.exec-source.channels=ch1 ch2 > agent.sinks.hdfs-sink.channel=ch1 > agent.sinks.hbase-sink.channel=ch2 >
