Re: KTable aggregations send intermediate results downstream?

2016-08-24 Thread Mathieu Fenniak
Hi Guozhang, I've been working around this issue by dropping down to the Processor API, but, I was hoping you might be able to point out if there is a flaw is in this proposed change: https://github.com/apache/kafka/compare/trunk...mfenniak:suppress-duplicate-repartition-output This adjusts

Multiple Consumers From Same Group Assigned Same Partition

2016-08-24 Thread Nicholas Harezga
I am currently attempting to upgrade my software to use Kafka 0.9 from 0.8.2. I am trying to switch over to the new Consumer API to allow for rebalancing as machines are added or removed from our cluster. I am running into an issue where the same partition on a topic is being assigned to multipl

Re: KIP-33 Opt out from Time Based indexing

2016-08-24 Thread Jan Filipiak
Hey Jun, I go and try again :), wrote the first one in quite a stressful environment. The bottom line is that I, for our use cases, see a to small use/effort ratio in this time index. We do not bootstrap new consumers for key-less logs so frequently and when we do it, they usually want everyth

Re: Questions about Apache Kafka

2016-08-24 Thread Marko Bonaći
Hi Karin, regarding 5 (fsyncing to disk), take a look at the broker configuration parameters whose names start with log.flush. http://kafka.apache.org/documentation.html#brokerconfigs Marko Bonaći Monitoring | Alerting | Anomaly Detection | Centralized Log Management Solr & Elasticsearch Support

Re: Questions about Apache Kafka

2016-08-24 Thread Ian Wrigley
Actually, for 3 and 4 Kafka Streams is also a very good option — and comes as part of the standard Apache Kafka distribution. Ian. --- Ian Wrigley Director, Education Services Confluent, Inc > On Aug 24, 2016, at 2:37 PM, David Garcia wrote: > > Regarding 3 and 4: https://calcite.apache.org/d

Re: Questions about Apache Kafka

2016-08-24 Thread David Garcia
Regarding 3 and 4: https://calcite.apache.org/docs/stream.html (i.e. streaming SQL queries) On 8/24/16, 6:29 AM, "Herwerth Karin (CI/OSW3)" wrote: Dear Sir or Madam, I'm a beginner in Apache Kafka and have questions which I don't get it from the documentation. 1.

Re: Questions about Apache Kafka

2016-08-24 Thread Radoslaw Gruchalski
Hi Karin, 1. Kafka does not support TTL for individual messages, only the topic. From the documentation: The Kafka cluster retains all published messages—whether or not they have been consumed—for a configurable period of time. For example if the log retention is set to two days, then for the two

Re: How to contribute to the wiki's Ecosystem page?

2016-08-24 Thread Guozhang Wang
Hi Jeff, I have granted you (id: jklukas) the permission to edit wiki page. You can try again and it should work now. Guozhang On Wed, Aug 24, 2016 at 7:58 AM, Jeff Klukas wrote: > We've developed a Kafka client metrics reporter that registers Kafka > metrics with the Dropwizard Metrics frame

How to contribute to the wiki's Ecosystem page?

2016-08-24 Thread Jeff Klukas
We've developed a Kafka client metrics reporter that registers Kafka metrics with the Dropwizard Metrics framework: https://github.com/SimpleFinance/kafka-dropwizard-reporter I think this would be a useful contribution to the metrics section of https://cwiki.apache.org/confluence/display/KAFKA/Eco

Re: Re: KIP-33 Opt out from Time Based indexing

2016-08-24 Thread Jun Rao
Jan, Thanks for the reply. I actually wasn't sure what your main concern on time-based rolling is. Just a couple of clarifications. (1) Time-based rolling doesn't control how long a segment will be retained for. For retention, if you use time-based, it will now be based on the timestamp in the mes

Questions about Apache Kafka

2016-08-24 Thread Herwerth Karin (CI/OSW3)
Dear Sir or Madam, I'm a beginner in Apache Kafka and have questions which I don't get it from the documentation. 1.If you set a Time To Live to a topic and to a message, which Time To Live is prioritized? 2.Supports Apache Kafka a mechanism to stop the publisher if the subscriber is

Re: Re: KIP-33 Opt out from Time Based indexing

2016-08-24 Thread Jan Filipiak
Hi Jun, I copy pasted this mail from the archive, as I somehow didn't receive it per mail. I will sill make some comments in line, hopefully you can find them quick enough, my apologies. To make things more clear, you should also know, that all messages in our kafka setup have a common way to