Addon: The rollover count varies (it is not always done after 4 events, as written before).
Best, Thomas Von: thomas.b...@continental-corporation.com An: user@flume.apache.org, Datum: 21.09.2015 12:05 Betreff: HDFS sink - rollover Hi, I'm using the Kafka-Flume source and the Flume-HDFS sink for writing SequenceFiles. I would like to rollover a SequenceFile after a specific count of events/messages was written, e.g. after 50 messages (see rollCount parameter below) a new file should be written. My configuration seems to be incorrect as a rollover is done after 4 messages (instead of 50). I'm using the following "rollover" configuration: "a_cobepa_probe.sources = kafka-cobepa_probe a_cobepa_probe.channels = hdfs-channel-notused a_cobepa_probe.sinks = hdfs_cobepa_probe a_cobepa_probe.sources.kafka-cobepa_probe.type = org.apache.flume.source.kafka.KafkaSource a_cobepa_probe.sources.kafka-cobepa_probe.zookeeperConnect = <secret> a_cobepa_probe.sources.kafka-cobepa_probe.topic = cobepa_probe a_cobepa_probe.sources.kafka-cobepa_probe.batchSize = 1 a_cobepa_probe.sources.kafka-cobepa_probe.channels = hdfs-channel-notused a_cobepa_probe.channels.hdfs-channel-notused.type = memory a_cobepa_probe.sinks.hdfs_cobepa_probe.channel = hdfs-channel-notused a_cobepa_probe.sinks.hdfs_cobepa_probe.type = hdfs a_cobepa_probe.sinks.hdfs_cobepa_probe.hdfs.writeFormat = de.conti.backend.asw.flume.serializer.MyBuilder a_cobepa_probe.sinks.hdfs_cobepa_probe.hdfs.fileType = SequenceFile # a_cobepa_probe.sinks.hdfs_cobepa_probe.hdfs.fileType = DataStream a_cobepa_probe.sinks.hdfs_cobepa_probe.hdfs.filePrefix = %k%M a_cobepa_probe.sinks.hdfs_cobepa_probe.hdfs.fileSuffix = .cobr a_cobepa_probe.sinks.hdfs_cobepa_probe.hdfs.useLocalTimeStamp = true a_cobepa_probe.sinks.hdfs_cobepa_probe.hdfs.path = /etl/%{topic}/%y%m%d a_cobepa_probe.sinks.hdfs_cobepa_probe.hdfs.rollCount=50 a_cobepa_probe.sinks.hdfs_cobepa_probe.hdfs.rollSize=0 a_cobepa_probe.sinks.hdfs_cobepa_probe.hdfs.batchSize=50 a_cobepa_probe.channels.hdfs-channel-notused.capacity = 100 a_cobepa_probe.channels.hdfs-channel-notused.transactionCapacity = 100" Are there any dependencies to other configuration parameters (in addition to the rollCount and rollSize parameter)? Thank you very much and kind regards, Thomas