Re: [kafka-clients] [VOTE] 1.0.0 RC3

+1. passed kafka-python integration tests, and manually verified producer/consumer on both compressed and non-compressed data. -Dana On Mon, Oct 23, 2017 at 6:00 PM, Guozhang Wang wrote: > Hello Kafka users, developers and client-developers, > > This is the third candidate

Re: Process and punctuate contract

As mentioned, a commit can---from a contract point of view---happen anytime. Of course, we only commit offsets of records that are fully processed. As punctuations are independent of records, there is no guarantee when it will be called though. Currently, we do the following sequence (but this is

Re: Process and punctuate contract

Hi Matthias Does this mean that an offset can be committed before process or punctuate is called Or that it could be called up to 30s after process/punctuate is called? We would like to do batch writes in punctuate of X amount messages gathered in process. If the process or punctuate step fails

[VOTE] 1.0.0 RC3

Hello Kafka users, developers and client-developers, This is the third candidate for release of Apache Kafka 1.0.0. The main PRs that gets merged in after RC1 are the following: https://github.com/apache/kafka/commit/dc6bfa553e73ffccd1e604963e076c 78d8ddcd69 It's worth noting that starting in

Re: kafka broker loosing offsets?

unsubscribe *Eric Ho | Qventus* Awarded Top Innovation in Cost Savings On Fri, Oct 6, 2017 at 7:52 AM, Dmitriy Vsekhvalnov wrote: > Hi

Re: Process and punctuate contract

Committing is independent of process and/or punctuate. You can configure your Kafka Streams application commit interval to any value you like via `commit.interval.ms` parameter (default is 30 seconds). Thus, there is no guarantee when a commit exactly happens with regard to calling process and

Re: Get broker metrics timeout

Hello Yang, On the mailing list attachments are usually not allowed. Your attached file did not show up. Could you please paste it somewhere and send the link in this thread? As for your observed TCP backlog issue, have you tried to simply increase the backlog capacity and see if it helps?

Re: Kafka Streams 0.11.0.1 Rebalancing Delay

Hello Johan, Thanks for the information. Could you apply the patch as in https://github.com/apache/kafka/pull/4085 from a source release of Streams and see if it fixed your problem? Guozhang On Fri, Oct 20, 2017 at 12:25 PM, Johan Genberg wrote: > Thanks. I've

Re: Keyed messages distribution in partitions

wow, that’s probable. Excuse me but I’m a javascript noob. I understood I have to edit line 55: var producer = new HighLevelProducer(client); using the template: HighLevelProducer(client, [options], [customPartitioner]) where partitionerType is the option. but how to insert the

Re: Keyed messages distribution in partitions

looks like you need to use keyedPartitioner option(partitionerType =3). default is cyclic partitioning. https://www.npmjs.com/package/kafka-node#highlevelproducer On Mon, Oct 23, 2017 at 9:24 PM, wrote: > Please correct me if I've misunderstood something, but your keys

Re: Keyed messages distribution in partitions

No, I played a bit to check differences. I tried to send messages in both ways: But using just ONE keys (Subject0) for ALL the messages the situation was: bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic LCacc --time -1 LCacc:8:60699

Re: Keyed messages distribution in partitions

Please correct me if I've misunderstood something, but your keys being Subject0, Subject1, Subject2, etc, mean you don't actually have identical keys, right? If that's the case then you wouldn't expect to see all of the messages in a single partition. > On Oct 23, 2017, at 8:43 AM, Andrea

Re: reg Kafka Node num

There's a definite desire to separate the controller and normal traffic. That's evident in the comment linked below. I don't know the exact reasons why this is desired behavior. Also, I misspoke earlier. It's not just the logging. They want to separate the controller and normal traffic into

Re: Keyed messages distribution in partitions

The output of: ./kafka_2.11-0.11.0.0/bin/kafka-topics.sh --zookeeper localhost:2181 --list LCacc LCgyr LTacc LTgyr RCacc RCgyr RTacc RTgyr __consumer_offsets output sample for a topic (all equals) ./kafka_2.11-0.11.0.0/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic LCacc

Re: Keyed messages distribution in partitions

Can you post the sample code? On Mon, Oct 23, 2017 at 8:53 PM, Andrea Giordano < andrea.giordano@gmail.com> wrote: > Hi, > I set a Kafka broker with some topics and where each topic is divided into > 10 partitions. > As I understood on Kafka doc, if I send a keyed message to kafka the key >

Keyed messages distribution in partitions

Hi, I set a Kafka broker with some topics and where each topic is divided into 10 partitions. As I understood on Kafka doc, if I send a keyed message to kafka the key is used to distribute the message in a particular partition (using an hash value of the key): in this way messages with same key

Kafka topic monitoring

Hi All, I'm looking for a open source tool which can help in monitoring topic level, partition level, consumer group level historycal data. use case: 1. Finding when consumer lag started on which partition on yesterdays data. 2. messages in-rate in each partition.

Re: left join between PageViews(KStream) & UserProfile (KTable)

> *What key should the join on ? * The message key, on both cases, should contain the user ID in String format. > *There seems to be no common key (eg. user) between the 2 classes - PageView and UserProfile* The user ID is the common key, but the user ID is stored in the respective message

Re: reg Kafka Node num

i have one doubt in ur answer: why should we differentiate the kafka logs trafiic from controller node with the normal trafiic from the same node?? On Sat, Oct 21, 2017 at 1:52 AM, Eric Azama wrote: > Kafka logs traffic to the Controller node separately from the normal >

Process and punctuate contract

Hi What is the contract around Processor.process and punctuate. When will Kafka streams commit the offset After the process method is called successfully or not until punctuate is called? Regards Toby