Cool, easy fix. What about the initial offset - does the consumer start from the oldest or the most recent offset?
Thanks! On Tue, Mar 10, 2015 at 12:31 PM, Gwen Shapira <[email protected]> wrote: > Just a small typo, I think. > > should be groupId (capital I. Flume standardized on camel case) > > > > On Tue, Mar 10, 2015 at 12:09 PM, Alex Bohr <[email protected]> wrote: > > After running some flafka tests I wanted to switch the name of the > consumer > > group to reset the offset to the most recent, instead of waiting for the > for > > the agents to plow through the backlog. > > So I changed the value in the "groupId" property. > > > > But I don't see the new consumer group in Kafka. It looks like the > agents > > are still consuming under the original consumer group name "flume." > > > > But I know the new config file is being used because I also changed the > > directory in the sink and they are writing to the new HDFS path. > > > > Here's my new config file: > > > > flume1.sources = kafka-source-1 > > flume1.channels = hdfs-channel-1 > > flume1.sinks = hdfs-sink-1 > > > > # For each source, channel, and sink, set > > # standard properties. > > flume1.sources.kafka-source-1.type = > > org.apache.flume.source.kafka.KafkaSource > > flume1.sources.kafka-source-1.zookeeperConnect = > > xxx.xx.xx.107:2181/kafkaCluster > > flume1.sources.kafka-source-1.topic = Events3 > > flume1.sources.kafka-source-1.groupid = newfloom > > flume1.sources.kafka-source-1.batchSize = 10000 > > flume1.sources.kafka-source-1.channels = hdfs-channel-1 > > > > flume1.channels.hdfs-channel-1.type = memory > > flume1.sinks.hdfs-sink-1.channel = hdfs-channel-1 > > flume1.sinks.hdfs-sink-1.type = hdfs > > flume1.sinks.hdfs-sink-1.hdfs.writeFormat = Writable > > flume1.sinks.hdfs-sink-1.hdfs.fileType = SequenceFile > > flume1.sinks.hdfs-sink-1.hdfs.filePrefix = > > %Y-%m-%d-%H-%M-%{host}-1-sequence-events > > flume1.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true > > flume1.sinks.hdfs-sink-1.hdfs.path = > > hdfs://xxx.xx.xxx.41:8020/user/gxetl/test_flume_2/%{topic} > > flume1.sinks.hdfs-sink-1.hdfs.rollCount=0 > > flume1.sinks.hdfs-sink-1.hdfs.rollSize=0 > > flume1.sinks.hdfs-sink-1.hdfs.rollInterval=120 > > > > # Other properties are specific to each type of > > # source, channel, or sink. In this case, we > > # specify the capacity of the memory channel. > > flume1.channels.hdfs-channel-1.capacity = 500000 > > flume1.channels.hdfs-channel-1.transactionCapacity = 100000 > > > > > > > > Any advice on how to change the consumer group name? > > > > Also, how does the kafka consumer set its initial offest when starting a > new > > source/consumer? > > I'm just hoping it sets it at the most recent but it probably sets it at > the > > oldest offset. And if so, how can I get it to start at the most recent? > > > > Thanks! >
