Dear Mailing Group, I am currently having issues with the Hbase sink function. I have developed an agent with a fanout channel setup ( single source, multiple channels, multiple sinks) sinking to a HDFS cluster and Hbase deployment.
The issue is that although the HDFS is working well, the Hbase flow is simply not working. There are no errors being reported by Flume for the Hbase channel but there are never any records being written to the HBase store. The Hbase table as stipulated in the config always remains empty. Studying the Flume startup logs I observe that the session connection to Zookeeper is always successfully established Are there any special configurations I am missing out? I am using the Async Event Serializer to persist the txns. Any assistance will be greatly appreciated. Please see below for the flume configuration: [biadmin@bivm bin]$ cat flume-conf.properties.bigdemo agent.sources=exec-source agent.sinks=hdfs-sink hbase-sink agent.channels=ch1 ch2 agent.sources.exec-source.type=exec agent.sources.exec-source.command=tail -F /home/biadmin/bigdemo/data/rec_telco.cdr agent.sinks.hdfs-sink.type=hdfs agent.sinks.hdfs-sink.hdfs.path=hdfs://XXXX:9000/user/biadmin/bigdemo/ agent.sinks.hdfs-sink.hdfs.filePrefix=telco_cdr_rec # File size to trigger roll, in bytes (0: never roll based on file size) agent.sinks.hdfs-sink.hdfs.rollSize = 134217728 agent.sinks.hdfs-sink.hdfs.rollCount = 0 # number of events written to file before it flushed to HDFS agent.sinks.hdfs-sink.hdfs.batchSize = 10000 agent.sinks.hdfs-sink.hdfs.txnEventMax = 40000 agent.sinks.hbase-sink.type=org.apache.flume.sink.hbase.AsyncHBaseSink agent.sinks.hbase-sink.serializer=org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer agent.sinks.hbase-sink.table=telco_cdr_rec agent.sinks.hbase-sink.columnFamily = colfam agent.sinks.hbase-sink.channels = ch2 #agent.sinks.hbase-sink.hdfs.batchSize = 10000 #agent.sinks.hbase-sink.hdfs.txnEventMax = 40000 agent.channels.ch1.type=file agent.channels.ch1.checkpointInterval=3000 agent.channels.ch1.transactionCapacity=10000 agent.channels.ch1.checkpointDir=/home/BDadmin/.flume/file-channel/checkpoint agent.channels.ch1.dataDirs=/home/BDadmin/.flume/file-channel/data agent.channels.ch1.write-timeout=30 agent.channels.ch1.keep-alive=30 #agent.channels.ch1.capacity=1000 agent.channels.ch2.type=file agent.channels.ch2.checkpointInterval=300 agent.channels.ch2.transactionCapacity=10000 agent.channels.ch2.checkpointDir=/home/BDadmin/.flume/file-channel2/checkpoint agent.channels.ch2.dataDirs=/home/BDadmin/.flume/file-channel2/data agent.channels.ch2.write-timeout=30 agent.channels.ch2.keep-alive=30 #agent.channels.ch2.capacity=1000 agent.sources.exec-source.channels=ch1 ch2 agent.sinks.hdfs-sink.channel=ch1 agent.sinks.hbase-sink.channel=ch2
