Re: Producer request latency increase after client 0.10 upgrade

Hi Guozhang, batch-size-avg and request-size-avg looked similar after the upgrade. Yifan On Tue, Sep 6, 2016 at 2:19 PM, Guozhang Wang wrote: > Yifan, > > Did you look into the producer metrics and check the avg. batch size / > request size? Did they change significantly

Re: [ANNOUNCE] New committer: Jason Gustafson

congrats, Jason! On Wed, Sep 7, 2016 at 9:28 AM, Ashish Singh wrote: > Congrats, Jason! > > On Tuesday, September 6, 2016, Jason Gustafson wrote: > > > Thanks all! > > > > On Tue, Sep 6, 2016 at 5:13 PM, Becket Qin >

Re: [ANNOUNCE] New committer: Jason Gustafson

Congrats, Jason! On Tuesday, September 6, 2016, Jason Gustafson wrote: > Thanks all! > > On Tue, Sep 6, 2016 at 5:13 PM, Becket Qin > wrote: > > > Congrats, Jason! > > > > On Tue, Sep 6, 2016 at 5:09 PM, Onur Karaman >

Re: [ANNOUNCE] New committer: Jason Gustafson

Thanks all! On Tue, Sep 6, 2016 at 5:13 PM, Becket Qin wrote: > Congrats, Jason! > > On Tue, Sep 6, 2016 at 5:09 PM, Onur Karaman > > wrote: > > > congrats jason! > > > > On Tue, Sep 6, 2016 at 4:12 PM, Sriram Subramanian

Re: [ANNOUNCE] New committer: Jason Gustafson

Congrats, Jason! On Tue, Sep 6, 2016 at 5:09 PM, Onur Karaman wrote: > congrats jason! > > On Tue, Sep 6, 2016 at 4:12 PM, Sriram Subramanian > wrote: > > > Congratulations Jason! > > > > On Tue, Sep 6, 2016 at 3:40 PM, Vahid S Hashemian < > >

Re: [ANNOUNCE] New committer: Jason Gustafson

congrats jason! On Tue, Sep 6, 2016 at 4:12 PM, Sriram Subramanian wrote: > Congratulations Jason! > > On Tue, Sep 6, 2016 at 3:40 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com > > wrote: > > > Congratulations Jason on this very well deserved recognition. > > > >

Problem consuming message using custom serializer

Hi, I'm trying to send java object using kryo object serializer . The producer is able to send the payload to the queue, but I'm having issues reading the data in consumer. I'm using consumer group using KafkaStream. The consumer code is based out of the example in kafka documentation. Here's

Re: [ANNOUNCE] New committer: Jason Gustafson

Congratulations Jason! On Tue, Sep 6, 2016 at 3:40 PM, Vahid S Hashemian wrote: > Congratulations Jason on this very well deserved recognition. > > --Vahid > > > > From: Neha Narkhede > To: "d...@kafka.apache.org" , >

Re: [ANNOUNCE] New committer: Jason Gustafson

Congratulations Jason on this very well deserved recognition. --Vahid From: Neha Narkhede To: "d...@kafka.apache.org" , "users@kafka.apache.org" Cc: "priv...@kafka.apache.org" Date:

Re: KIP-33 Opt out from Time Based indexing

Jan, For the time rolling issue, Jiangjie has committed a fix ( https://issues.apache.org/jira/browse/KAFKA-4099) to trunk. Perhaps you can help test out trunk and see if there are any other issues related to time-based index? Thanks, Jun On Mon, Sep 5, 2016 at 11:52 PM, Jan Filipiak

Re: [ANNOUNCE] New committer: Jason Gustafson

Congratulations Jason! Well-deserved. :) Ismael On Tue, Sep 6, 2016 at 11:25 PM, Neha Narkhede wrote: > The PMC for Apache Kafka has invited Jason Gustafson to join as a > committer and > we are pleased to announce that he has accepted! > > Jason has contributed numerous

Re: [ANNOUNCE] New committer: Jason Gustafson

Well deserved, Jason. Looking forward to your future contributions :) On Tue, Sep 6, 2016 at 3:29 PM, Guozhang Wang wrote: > Welcome, and really happy to have you onboard Jason! > > > Guozhang > > On Tue, Sep 6, 2016 at 3:25 PM, Neha Narkhede wrote: > >>

Re: [ANNOUNCE] New committer: Jason Gustafson

Welcome, and really happy to have you onboard Jason! Guozhang On Tue, Sep 6, 2016 at 3:25 PM, Neha Narkhede wrote: > The PMC for Apache Kafka has invited Jason Gustafson to join as a > committer and > we are pleased to announce that he has accepted! > > Jason has

Re: Kafka 0.10.0 MetricsReporter implementation only reporting zeros

Michael, One thing to be aware is that if the producer stops sending messages, some of the metrics will be reset to 0 after the metric window elapses. Do you see metric values reported in the reporter different from that in jconsole? Thanks, Jun On Tue, Sep 6, 2016 at 3:03 PM, Michael Ross

[ANNOUNCE] New committer: Jason Gustafson

The PMC for Apache Kafka has invited Jason Gustafson to join as a committer and we are pleased to announce that he has accepted! Jason has contributed numerous patches to a wide range of areas, notably within the new consumer and the Kafka Connect layers. He has displayed great taste and

Kafka 0.10.0 MetricsReporter implementation only reporting zeros

I have a very simple Metrics Reporter that implements the Kafka 0.10.0 MetricsReporter interface, that takes in the metrics as a parameter and logs them. When a producer starts to send messages, I can see that it is correctly configured to use the Metrics Reporter I created because it starts to

Re: Producer request latency increase after client 0.10 upgrade

Yifan, Did you look into the producer metrics and check the avg. batch size / request size? Did they change significantly compared with 0.8.2.1 clients (there should be similar metrics but with different metrics name)? Guozhang On Mon, Sep 5, 2016 at 11:17 PM, Yifan Ying

Re: Authorization with Topic Wildcards

Hi Samuel, I am not aware of a JIRA for it. This has been discussed a few times, but I don't think anyone is actively working on it. Ismael On Tue, Sep 6, 2016 at 7:22 PM, Samuel Taylor wrote: > Is there a ticket to extend wildcard topic ACL support to include the use

Replica fetcher continually disconnecting after broker replacement

We had a hardware failure on broker 1 of a 3 broker cluster over the weekend. The broker was replaced, and when the replacement broker came up it started to replicate partitions from the other 2 brokers as you'd expect. But while broker 1 (the replacement) was able to fetch properly from

Re: Authorization with Topic Wildcards

Is there a ticket to extend wildcard topic ACL support to include the use case Derar originally mentioned? And/or are there plans to? On Mon, Sep 5, 2016 at 3:43 PM, Ismael Juma wrote: > Hi Derar, > > The support for wildcards is limited to `*` at this point. Sorry for the >

Re: Authorization with Topic Wildcards

Definitely worth putting it there. I will find some time soon to do it. This is the least I can do! Thanks guys for the quick feedback. Derar On Mon, Sep 5, 2016 at 1:43 PM, Ismael Juma wrote: > Hi Derar, > > The support for wildcards is limited to `*` at this point. Sorry

Re: KStreams regex-topic null pointer Exception with multiple Consumer processes

Done: https://issues.apache.org/jira/browse/KAFKA-4131 Thanks Guozhang for also looking at this. BTW, do you have any idea when 0.10.1 will be released? -David On 9/5/16, 12:14 AM, "Guozhang Wang" wrote: Hello David, Thanks for reporting this issue, and

Re: Kafka Streams: joins without windowing (KStream) and without being KTables

Also, another upcoming feature is (slightly simplified description follows) "global" state/KTable. Today, a KTable is always partitioned/sharded. One advantage of global state/tables will be for use cases where you need to perform non-key joins, similar to what Guillermo described previously in

Re: Using kafka as a "message bus" for an event store

Hey Tom, Thanks for your help :) In terms of the number of topics, there would be 1 topic per service. Each service would be running multiple instances, but they would be in the same consumer group consuming the same topic. I am expecting around 30ish microservices at the moment, so around

Re: Using kafka as a "message bus" for an event store

inline On Mon, Sep 5, 2016 at 11:58 PM, F21 wrote: > Hi Tom, > > Thank you so much for your response. I had a feeling that approach would > run into scalability problems, so thank you for confirming that. > > Another approach would be to have each service request a

Re: Producer/Consumer config for length Kafka broker maintenance

Hi there, If you're only shutting down a single broker of many and you have a replication factor more than 1, those consumer and producer configs should handle it. However, if you have only a single broker, I'd recommend getting some replication in before doing any maintenance work - Kafka is

Re: Kafka bootup exception while recovering log file

This sounds like Kafka not being entirely robust to disk corruption, which seems entirely possible and normal. I'd simply delete that log file and let a replica replay catch it up at broker bootup. Trying to guard against all possible disk corruption bugs sounds very difficult to me, it seems

Re: Partitioning at the edges

Hi Andy, One option would be to take the original transaction data and enrich it with the customer ID and then send the result to a new topic, partitioned appropriately. That would then allow you to do a join between that topic and the ledger data (otherwise I don't see how you can join given

Re: Kafka Streams: joins without windowing (KStream) and without being KTables

Hi, currently, in DSL only primary-key joins are supported. However in you case, you have a non-primary-key join. There is already a JIRA to add support for this: https://issues.apache.org/jira/browse/KAFKA-3705 Currently, you will need to use Processor API. For the non-time-based input (if not

Re: KIP-33 Opt out from Time Based indexing

Hi Jun, sorry for the late reply. Regarding B, my main concern was just complexity of understanding what's going on. As you can see it took me probably some 2 days or so, to fully grab all the details in the implementation and what the impacts are. Usually I prefer to turn things I don't use

Re: kafka broker is dropping the messages after acknowledging librdkafka

Hi Jun / Kafka Team, Do we have any solution for this issue ? "During zookeeper re-init kafka broker truncates messages and ends up in loosing records" I'm ok with with duplicate messages being stored instead of dropping. Is there any configuration in kafka where follower broker replicates

Re: Producer request latency increase after client 0.10 upgrade

Hi Ismael, Thanks for replying. Yes. It's the comparison between 0.8.2.1 and 0.10.0.0 clients connecting to 0.10.0.0 brokers(log.message.format.version=0.8.2.1). No compression is used and that's all the producer config. >From the Kafka code, I think the request-latency-avg metric seem not

Re: Kafka bootup exception while recovering log file

I'm not from the Kafka dev team so I won't be able to comment whether this is an expected way to fail or if this needs to be handled in a more cleaner/robust manner (at least very least probably a better exception message). Since you have put in efforts to write a test case and narrow it down