Kafka forum register

Hello, How can I register in Kafka forum? 童树山 中嘉仁和科技（北京）有限公司 北京市朝阳区东三环北路3号幸福大厦B座813室 邮编 100027 Room 813, Block B, Lucky Tower, 3 Dongsanhuan Beilu, Chaoyang District, Beijing 100027, China Tel: 010-64661846 Fax: 010-64661840 Mobile: 13818663262 E-mail:tong...@csbucn.com 发件人：

Re: Producer Properties

Chris, all of the keys are documented here: http://kafka.apache.org/documentation.html#producerconfigs Easier to read than ProducerConfig? -Peter > On Jun 27, 2016, at 6:34 AM, Chris Barlock wrote: > > Anybody? > > Chris > > > > > From: Chris

Re: console-consumer show offset

Vincent, you'll need to provide a custom Message Formatter implementation. I too found myself wishing for this for debugging purposes, but the default one doesn't support it. > ./bin/kafka-console-consumer --new-consumer --bootstrap.server localhost:9092 > --topic foo --formatter

Re: Brokers are crash due to __consumer_offsets folder are deleted

Dear 黄杰斌: I am guessing your operating system is configured to delete your /tmp directory when you restart the server. You will need to change the "log.dir" property in your broker's server.properties file to someplace permanent. Unfortunately, your data is lost unless you had a backup or

Re: Handling of uncommitted messages

http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/ From my understanding, the watermark will not advance until both failed follow-broker recovered and caught up. -Matthias On 07/01/2016 12:17 PM, Vikas Kumar wrote: > Hi, > > I have a question about

Re: Heartbeating during long processing times

Hi Elias, KIP-62 has a discussion of current options, and the improvements that are coming. Best, Shikhar On Thu, Jun 30, 2016 at 6:02 PM Elias Levy

Re: broker randomly shuts down

This is somewhat specific to your runtime environment, you can check out whatever script is getting used for bringing up Kafka, and where the stderr of the java command is being redirected (hopefully not /dev/null!). On Thu, Jun 30, 2016 at 5:24 PM allen chan wrote:

Re: ZkUtils creating parent path with different ACL

If i remember correctly it was because we wanted to allow non-secure client still get into child consumers node and create their zookeeper nodes to keep track of offset. If we add the acl at the parent path they won't be able to write to the child nodes. Thanks, Harsha On Fri, Jul 1, 2016, at

Re: Kafka without Storm/Spark

> Because i know that by using Storm, you can guarantee the messages (depending on the type of the Topology) > such as exactly once, at least once. If i simply use kafka consumer and another producer to forward the > messages, could the data tranfer completely be guaranteed as well? Addendum:

Re: Kafka without Storm/Spark

Numan, you may also want to take a look at Kafka Streams, which is a new stream processing library that's included in Apache Kafka since version 0.10. Kafka Streams is definitely more convenient and quicker to implement than the "normal" Kafka producer/consumer clients. Also, Kafka Streams does

Re: Kafka without Storm/Spark

Hi, Thanks for your answer. I meant actually if we can verify the data reliability without using Storm or Spark. Because i know that by using Storm, you can guarantee the messages (depending on the type of the Topology) such as exactly once, at least once. If i simply use kafka consumer and

Kafka without Storm/Spark

Hello everyone, I've a quick question:I'm using Apache Kafka producer to write the messages into a topic. My source at the moment a csv file but in the future i am supposed to read the messages from another kafka topic.My question is:Is it possible to consume messages from a Kafka topic in

Re: Kafka without Storm/Spark

Definitely. You can read off kafka using the samples shown in KafkaConsumer javadoc, transform if necessary and publish to the destination topic. On 01/07/2016 03:24, "numan goceri" wrote: >Hello everyone, >I've a quick question:I'm using Apache Kafka

Brokers are crash due to __consumer_offsets folder are deleted

Hi All, Do you encounter below issue when using kafka_2.11-0.10.0.0? All brokers are crash due to __consumer_offsets folder are deleted. sample log: [2016-06-30 12:46:32,579] FATAL [Replica Manager on Broker 2]: Halting due to unrecoverable I/O error while handling produce request:

Handling of uncommitted messages

Hi, I have a question about handling of uncommitted messages in Kafka. Lets say the partition has 4 replicas (1 leader, 3 followers) and all are currently in sync. min.insync.replicas is set to 3 and request.required.acks is set to all or -1. The producer send a message to the leader, the

ZkUtils creating parent path with different ACL

Hello Apache Kafka community, Is there a reason why acls are not passed through in https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/utils/ZkUtils.scala#L404 call ? If one overrides and uses custom acls for a node, parent if missing might get created with default ACL. Kind

Re: Coordinator lost for consumer groups

You might try increasing the log.cleaner.dedupe.buffer.size. This should increase the deduplication yield for each scan. If you haven’t seen them there are some notes on log compaction here: https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction

Re: Coordinator lost for consumer groups

The problem still persists. But note that I was running old consumer (Zk based) to describe consumers. Running ./kafka-consumer-groups.sh kafka-groups.sh --bootstrap-server 10.211.16.215 --group groupX --describe, I get the below error. So none of the consumer groups seem to have a coordinator

Re: Coordinator lost for consumer groups

And I am willing to suspend log compaction and restart the brokers, but am worried if that will leave the system in a recoverable state or If I just have to wait it out. On Fri, Jul 1, 2016 at 2:06 PM, Sathyakumar Seshachalam < sathyakumar_seshacha...@trimble.com> wrote: > Hi, > > I have 3 Kafka

Coordinator lost for consumer groups

Hi, I have 3 Kafka nodes (running 0.9.0) that all had active consumers and producers. Now all these had uncompacted __consumer_offsets group that grew to 1.8 TB. So I restarted these nodes with a log.cleaner.enabled to save some space. Since then consumers have stalled. When I do a