RE: is there a way to make sure two consumers receive the same message from the broker?

Excuse me this part was non-sense: if the latest update to a document in es always win in Cassandra's LWW, they will "eventually" "converge". From: AmirHossein Roozbahany Sent: ‎11/‎8/‎2016 8:16 AM To: users@kafka.apache.org

Re: is there a way to make sure two consumers receive the same message from the broker?

:) On Tue, Nov 8, 2016 at 1:37 AM, AmirHossein Roozbahany wrote: > Excuse me this part was non-sense: if the latest update to a document in > es always win in Cassandra's LWW, they will "eventually" "converge". > > From: AmirHossein Roozbahany

Re: consumer client pause/resume/rebalance

Hi Gwen - Makes sense. The way you explain it actually reminds me a little of the "worse is better" philosophy: https://www.jwz.org/doc/worse-is-better.html Perhaps a mention in the javadoc for pause() and/or ConsumerRebalanceListener would be sufficient. From:

Understanding zookeper and kafka server failures

Hi, We have following setup. 1. Three instances of zookeeper on three machines. 2. Three instances of kafka server on same three machines. 3. All the topics have replication factor 3. So when we create a topic on any node, i see that it gets replicated on all three instances. I also see that topic

Re: Understanding zookeper and kafka server failures

The question is what happens if one of the zookeeper crashes. Will the broker on that node also will crash? If 1/3 zookeeper nodes crashes, the other two will take over. Kafka broker will not crash. However, you should not run zookeeper and kafka on the same server in production. What happens i

Re: Understanding zookeper and kafka server failures

Hi, Thanks for the reply. From one obvious reason that is server crashes then both zookeeper and broker crashes, is there any other reason why we should not run broker and zookeeper on same server. If chances of server crash are extremely low can can be brought back up quickly, then can we keep bo

Re: Understanding zookeper and kafka server failures

It depends on the size and load of your cluster. Zookeeper is very I/O sensitive, so at least you have to make sure it doesn't share disk with the OS or Kafka. I assume you've read the documentation, but you might want to have a look at https://kafka.apache.org/documentation.html#zkops again,

Protecting kafka-producer against unavailability of all brokers (request.timeout.ms)

Hi, We are using kafka-producer 0.8.2 on our production. We configured it with retries to Integer.MAX_VALUE and buffer.memory to 1GB. Thanks to this setup we are protected from unavailability of all brokers for around one hour (taking into account our production traffic). For example, when all bro

Kafka streaming changelog topic max.message.bytes exception

Hi, We are using aggregation by key on a kstream to create a ktable. As I read from https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams%3A+Internal+Data+Management it creates an internal changelog topic. However over the time the streaming application is run message size increases and

[kafka] Errors during failover

Hello, We're planning to use kafka (0.10.1), so we tested it. I've done some fail-over tests, with unexpected results. We get 3 servers, each one is running a kafka broker. We created 3 messages queues (MSG01, MSG02, MSG03). Each message queue has only 1 partition, and has a replication factor

Re: Protecting kafka-producer against unavailability of all brokers (request.timeout.ms)

I agree that accumulator timeout should be independent from the other two you mentioned. We at LinkedIn have come up with a solution and I'll create a KIP for it soon. In essence, we want batch.expiry.ms configuration that directly specifies accumulator timeout separately from request.timeout. P

Re: Kafka streaming changelog topic max.message.bytes exception

Hi Sachin, Could you clarify what you mean by "message size increases"? Are messages going to the changelog topic increasing in size? Or is the changelog topic getting full? Thanks Eno > On 8 Nov 2016, at 16:49, Sachin Mittal wrote: > > Hi, > We are using aggregation by key on a kstream to

Re: Kafka streaming changelog topic max.message.bytes exception

Message size itself increases over the time. Message is something like key=[list on objects] This increases with time and then at a point kafka is not able to add any message to its topic because message size is greater than max.message.bytes. Since this is an internal topic based off a table I d

Re: Kafka streaming changelog topic max.message.bytes exception

Are you talking about total messages and therefore size or per message payload size. On Tue, Nov 8, 2016 at 10:00 AM, Sachin Mittal wrote: > Message size itself increases over the time. > > Message is something like > key=[list on objects] > > This increases with time and then at a point kafka i

Kafka ACL Groups/Wildcards

Hi everyone, I've been trying out Kafka security and was curious if there were plans/issues to add wildcards in resources or user group support to Kafka ACLs? If they are already implemented point me to the doc on how to use it? Specifically it would be nice to give group's of users access to th

Re: Kafka streaming changelog topic max.message.bytes exception

Per message payload size. The basic question is how can I control the internal change log topics parameters so as to avoid these errors. On Tue, Nov 8, 2016 at 11:37 PM, R Krishna wrote: > Are you talking about total messages and therefore size or per message > payload size. > > On Tue, Nov 8,

Re: Kafka streaming changelog topic max.message.bytes exception

Hi Sachin, One option right now would be to precreate all internal topics in Kafka, and only after that start the Kafka Streams application. This would require you knowing the internal name of the topics (in this case you probably already know it, but I agree that in general this is a bit cumbe

kafka topics failed UnknownHostException

Hi, Guys. Any idea about this error? I installed and add Kafka service through Cloudera Manager, which looks fine. But when I try to test it with list topics, it gave error like: ./kafka-topics.sh --list --zookeeper localhost:2181 java.net.UnknownHostException: opt at java.net.AbstractPlainS

Re: sliding ktable?

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Yes and no. Kafka allows you to set a retention time for compacted topics, too. Thus, if a key does not get an update for this retention time, it will be deleted, too. See here for details: https://cwiki.apache.org/confluence/display/KAFKA/KIP-71%3

Re: Kafka streaming changelog topic max.message.bytes exception

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 My two cents: Changelog topics are compacted topics, thus they do not have a retention time (there is an exception for windowed KTable changlog topics that are compacted and do have a retention time though). However, I do not understand how changin

Re: consumer client pause/resume/rebalance

Yeah, we should mention that in the javadoc. Want to send a PR? I like the "worse is better" philosophy - although we occasionally choose complex implementation for simpler APIs (especially when it comes to admin configurations). On Tue, Nov 8, 2016 at 2:34 AM, Paul Mackles wrote: > Hi Gwen - Ma

Re: Cleanup partition offsets that exist for consumer groups but not in broker

Hello Jeff, Generally speaking ZK's stored offset paths should not be used as the "source-of-truth" to determine which topic-partitions exist in the Kafka cluster, but instead the broker topics path should be treated as the "source-of-truth". More specifically, the common usage pattern would be:

Re: sliding ktable?

Yes, thanks.

Kafka performance on an ordinary machine

Hello We want to use Kafka in our production environment but we don't have any information about what's the best server configuration and the best benchmark can be achieve in our production environment (Based on our hardware and VMs) I'm really rookie in this area. For getting to the best configura