Re: [DISCUSS] KIP-219 - Improve Quota Communication

[Becket Qin]

Thanks Rajini.

1. Good point. We do need to bump up the protocol version so that the new
clients do not wait for another throttle time when they are talking to old
brokers. I'll update the KIP.

2. That is true. But the client was not supposed to send request to the
broker during that period anyways. So detecting the broker failure later
seems fine?

3. Wouldn't the CLOSE_WAIT handler number be the same as the current state?
Currently the broker will still mute the socket until it sends the response
back. If the clients disconnect while they are being throttled, the closed
socket will not be detected until the throttle time has passed.

Jun also suggested to bound the time by metric.sample.window.ms in the
ticket. I am not sure about the bound on throttle time. It seems a little
difficult to come up with a good bound. If the bound is too large, it does
not really help solve the various timeout issue we may face. If the bound
is too low, the quota is essentially not honored. We may potentially treat
different requests differently, but that seems too complicated and error
prone.

IMO, the key improvement we want to make is to tell the clients how long
they will be throttled so the clients knows what happened so they can act
accordingly instead of waiting naively. Muting the socket on the broker
side is just in case of non-cooperating clients. For the existing clients,
it seems this does not have much impact compare with what we have now.

Thanks,

Jiangjie (Becket) Qin



On Fri, Nov 3, 2017 at 3:09 PM, Rajini Sivaram 
wrote:

> Hi Becket,
>
> Thank you for the KIP. A few comments:
>
> 1.KIP says:  "*No public interface changes are needed. We only propose
> behavior change on the broker side.*"
>
> But from the proposed changes, it sounds like clients will be updated to
> wait for throttle-time before sending next response, and also not handle
> idle disconnections during that time. Doesn't that mean that clients need
> to know that the broker has sent the response before throttling, requiring
> protocol/version change?
>
>
> 2. At the moment, broker failures are detected by clients (and vice versa)
> within connections.max.idle.ms. By removing this check for an unlimited
> throttle time, failure detection could be delayed.
>
>
> 3. KIP says  "*Since this subsequent request is not actually handled until
> the broker unmutes the channel, the client can hit request.timeout.ms
>  and reconnect. However, this is no worse than
> the current state.*"
>
> I think this could be worse than the current state because broker doesn't
> detect and close the channel for an unlimited throttle time, while new
> connections will get accepted. As a result, lot of connections could be in
> CLOSE_WAIT state when throttle time is high.
>
>
> Perhaps it is better to combine this KIP with a bound on throttle time?
>
>
> Regards,
>
>
> Rajini
>
>
> On Fri, Nov 3, 2017 at 8:09 PM, Becket Qin  wrote:
>
> > Thanks for the comment, Jun,
> >
> > 1. Yes, you are right. This could also happen with the current quota
> > mechanism because we are essentially muting the socket during throttle
> > time. There might be two ways to solve this.
> > A) use another socket to send heartbeat.
> > B) let the GroupCoordinator know that the client will not heartbeat for
> > some time.
> > It seems the first solution is cleaner.
> >
> > 2. For consumer it seems returning an empty response is a better option.
> In
> > the producer case, if there is a spike in traffic. The brokers will see
> > queued up requests, but that is hard to avoid unless we have connection
> > level quota, which is a bigger change and may be easier to discuss it in
> a
> > separate KIP.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> > On Fri, Nov 3, 2017 at 10:28 AM, Jun Rao  wrote:
> >
> > > Hi, Jiangjie,
> > >
> > > Thanks for bringing this up. A couple of quick thoughts.
> > >
> > > 1. If the throttle time is large, what can happen is that a consumer
> > won't
> > > be able to heart beat to the group coordinator frequent enough. In that
> > > case, even with this KIP, it seems there could be frequent consumer
> group
> > > rebalances.
> > >
> > > 2. If we return a response immediately, for the consumer, do we return
> > the
> > > requested data or an empty response? If we do the former, it may not
> > > protect against the case when there are multiple consumer instances
> > > associated with the same user/clientid.
> > >
> > > Jun
> > >
> > > On Wed, Nov 1, 2017 at 9:53 AM, Becket Qin 
> wrote:
> > >
> > > > Hi,
> > > >
> > > > We would like to start the discussion on KIP-219.
> > > >
> > > > The KIP tries to improve quota throttling time communication between
> > > > brokers and clients to avoid clients timeout in case of long
> throttling
> > > > time.
> > > >
> > > > The KIP link is following:
> > > > 

Re: [DISCUSS] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

[Ted Yu]

Looks good overall.

bq. the creation within StreamsPartitionAssignor

Typo above: should be StreamPartitionAssignor

On Fri, Nov 3, 2017 at 4:49 PM, Guozhang Wang  wrote:

> Hello folks,
>
> I have filed a new KIP on adding AdminClient into Streams for internal
> topic management.
>
> Looking for feedback on
>
> *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier
>  220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier>*
>
> --
> -- Guozhang
>

[GitHub] kafka pull request #4176: KAFKA-6168 Connect Schema comparison is slow for l...

[tedyu]

GitHub user tedyu opened a pull request:

https://github.com/apache/kafka/pull/4176

KAFKA-6168 Connect Schema comparison is slow for large schemas

Re-arrange order of comparisons in equals() to evaluate non-composite 
fields first
Cache hash code

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tedyu/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4176.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4176


commit 4bae62c8ad64408b20e4925977beadaa42b54619
Author: tedyu 
Date:   2017-11-04T02:03:59Z

KAFKA-6168 Connect Schema comparison is slow for large schemas




---

Re: [VOTE] KIP-215: Add topic regex support for Connect sinks

[Stephane Maarek]

+1! The S3 or hdfs connector will now be super powerful !

On 4 Nov. 2017 11:27 am, "Konstantine Karantasis" 
wrote:

> Nice addition!
>
> +1 (non-binding)
>
> Konstantine
>
> On Fri, Nov 3, 2017 at 4:52 PM, Jeff Klukas  wrote:
>
> > So sorry for skirting the process there. I wasn't aware of the 72 hour
> > window and I don't see that mentioned in in
> > https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Voting
> >
> > Should I feel free to update that wiki page with a note about the window?
> >
> > On Fri, Nov 3, 2017 at 7:49 PM, Ewen Cheslack-Postava  >
> > wrote:
> >
> > > Jeff,
> > >
> > > Just FYI re: process, I think you're pretty much definitely in the
> clear
> > > hear since this one is a straightforward design I doubt anybody would
> > > object to, but voting normally stays open 72h to ensure everyone has a
> > > chance to weigh in.
> > >
> > > Again thanks for the KIP and we can move any final discussion over to
> the
> > > PR!
> > >
> > > -Ewen
> > >
> > > On Fri, Nov 3, 2017 at 4:43 PM, Jeff Klukas 
> wrote:
> > >
> > > > Looks like we've achieved lazy majority, so I'll move the KIP to
> > > approved.
> > > >
> > > > Thanks all for looking this over.
> > > >
> > > > On Fri, Nov 3, 2017 at 7:31 PM, Jason Gustafson 
> > > > wrote:
> > > >
> > > > > +1. Thanks for the KIP!
> > > > >
> > > > > On Fri, Nov 3, 2017 at 2:15 PM, Guozhang Wang 
> > > > wrote:
> > > > >
> > > > > > +1 binding
> > > > > >
> > > > > > On Fri, Nov 3, 2017 at 1:25 PM, Ewen Cheslack-Postava <
> > > > e...@confluent.io
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1 binding
> > > > > > >
> > > > > > > Thanks Jeff!
> > > > > > >
> > > > > > > On Wed, Nov 1, 2017 at 5:21 PM, Randall Hauch <
> rha...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > Thanks for pushing this through. Great work!
> > > > > > > >
> > > > > > > > Randall Hauch
> > > > > > > >
> > > > > > > > On Wed, Nov 1, 2017 at 9:40 AM, Jeff Klukas <
> > jklu...@simple.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > I haven't heard any additional concerns over the proposal,
> so
> > > I'd
> > > > > > like
> > > > > > > to
> > > > > > > > > get the voting process started for:
> > > > > > > > >
> > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > > 215%3A+Add+topic+regex+support+for+Connect+sinks
> > > > > > > > >
> > > > > > > > > It adds a topics.regex option for Kafka Connect sinks as an
> > > > > > alternative
> > > > > > > > to
> > > > > > > > > the existing required topics option.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

[Matt Farmer]

This seems like an A+ improvement to me.

On Fri, Nov 3, 2017 at 7:49 PM Guozhang Wang  wrote:

> Hello folks,
>
> I have filed a new KIP on adding AdminClient into Streams for internal
> topic management.
>
> Looking for feedback on
>
> *
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier
> >*
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-215: Add topic regex support for Connect sinks

[Konstantine Karantasis]

Nice addition!

+1 (non-binding)

Konstantine

On Fri, Nov 3, 2017 at 4:52 PM, Jeff Klukas  wrote:

> So sorry for skirting the process there. I wasn't aware of the 72 hour
> window and I don't see that mentioned in in
> https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Voting
>
> Should I feel free to update that wiki page with a note about the window?
>
> On Fri, Nov 3, 2017 at 7:49 PM, Ewen Cheslack-Postava 
> wrote:
>
> > Jeff,
> >
> > Just FYI re: process, I think you're pretty much definitely in the clear
> > hear since this one is a straightforward design I doubt anybody would
> > object to, but voting normally stays open 72h to ensure everyone has a
> > chance to weigh in.
> >
> > Again thanks for the KIP and we can move any final discussion over to the
> > PR!
> >
> > -Ewen
> >
> > On Fri, Nov 3, 2017 at 4:43 PM, Jeff Klukas  wrote:
> >
> > > Looks like we've achieved lazy majority, so I'll move the KIP to
> > approved.
> > >
> > > Thanks all for looking this over.
> > >
> > > On Fri, Nov 3, 2017 at 7:31 PM, Jason Gustafson 
> > > wrote:
> > >
> > > > +1. Thanks for the KIP!
> > > >
> > > > On Fri, Nov 3, 2017 at 2:15 PM, Guozhang Wang 
> > > wrote:
> > > >
> > > > > +1 binding
> > > > >
> > > > > On Fri, Nov 3, 2017 at 1:25 PM, Ewen Cheslack-Postava <
> > > e...@confluent.io
> > > > >
> > > > > wrote:
> > > > >
> > > > > > +1 binding
> > > > > >
> > > > > > Thanks Jeff!
> > > > > >
> > > > > > On Wed, Nov 1, 2017 at 5:21 PM, Randall Hauch 
> > > > wrote:
> > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > Thanks for pushing this through. Great work!
> > > > > > >
> > > > > > > Randall Hauch
> > > > > > >
> > > > > > > On Wed, Nov 1, 2017 at 9:40 AM, Jeff Klukas <
> jklu...@simple.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > I haven't heard any additional concerns over the proposal, so
> > I'd
> > > > > like
> > > > > > to
> > > > > > > > get the voting process started for:
> > > > > > > >
> > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > 215%3A+Add+topic+regex+support+for+Connect+sinks
> > > > > > > >
> > > > > > > > It adds a topics.regex option for Kafka Connect sinks as an
> > > > > alternative
> > > > > > > to
> > > > > > > > the existing required topics option.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-215: Add topic regex support for Connect sinks

[Jeff Klukas]

So sorry for skirting the process there. I wasn't aware of the 72 hour
window and I don't see that mentioned in in
https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Voting

Should I feel free to update that wiki page with a note about the window?

On Fri, Nov 3, 2017 at 7:49 PM, Ewen Cheslack-Postava 
wrote:

> Jeff,
>
> Just FYI re: process, I think you're pretty much definitely in the clear
> hear since this one is a straightforward design I doubt anybody would
> object to, but voting normally stays open 72h to ensure everyone has a
> chance to weigh in.
>
> Again thanks for the KIP and we can move any final discussion over to the
> PR!
>
> -Ewen
>
> On Fri, Nov 3, 2017 at 4:43 PM, Jeff Klukas  wrote:
>
> > Looks like we've achieved lazy majority, so I'll move the KIP to
> approved.
> >
> > Thanks all for looking this over.
> >
> > On Fri, Nov 3, 2017 at 7:31 PM, Jason Gustafson 
> > wrote:
> >
> > > +1. Thanks for the KIP!
> > >
> > > On Fri, Nov 3, 2017 at 2:15 PM, Guozhang Wang 
> > wrote:
> > >
> > > > +1 binding
> > > >
> > > > On Fri, Nov 3, 2017 at 1:25 PM, Ewen Cheslack-Postava <
> > e...@confluent.io
> > > >
> > > > wrote:
> > > >
> > > > > +1 binding
> > > > >
> > > > > Thanks Jeff!
> > > > >
> > > > > On Wed, Nov 1, 2017 at 5:21 PM, Randall Hauch 
> > > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Thanks for pushing this through. Great work!
> > > > > >
> > > > > > Randall Hauch
> > > > > >
> > > > > > On Wed, Nov 1, 2017 at 9:40 AM, Jeff Klukas 
> > > > wrote:
> > > > > >
> > > > > > > I haven't heard any additional concerns over the proposal, so
> I'd
> > > > like
> > > > > to
> > > > > > > get the voting process started for:
> > > > > > >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > 215%3A+Add+topic+regex+support+for+Connect+sinks
> > > > > > >
> > > > > > > It adds a topics.regex option for Kafka Connect sinks as an
> > > > alternative
> > > > > > to
> > > > > > > the existing required topics option.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
>

[DISCUSS] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

[Guozhang Wang]

Hello folks,

I have filed a new KIP on adding AdminClient into Streams for internal
topic management.

Looking for feedback on

*https://cwiki.apache.org/confluence/display/KAFKA/KIP-220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier
*

-- 
-- Guozhang

Re: [VOTE] KIP-215: Add topic regex support for Connect sinks

[Ewen Cheslack-Postava]

Jeff,

Just FYI re: process, I think you're pretty much definitely in the clear
hear since this one is a straightforward design I doubt anybody would
object to, but voting normally stays open 72h to ensure everyone has a
chance to weigh in.

Again thanks for the KIP and we can move any final discussion over to the
PR!

-Ewen

On Fri, Nov 3, 2017 at 4:43 PM, Jeff Klukas  wrote:

> Looks like we've achieved lazy majority, so I'll move the KIP to approved.
>
> Thanks all for looking this over.
>
> On Fri, Nov 3, 2017 at 7:31 PM, Jason Gustafson 
> wrote:
>
> > +1. Thanks for the KIP!
> >
> > On Fri, Nov 3, 2017 at 2:15 PM, Guozhang Wang 
> wrote:
> >
> > > +1 binding
> > >
> > > On Fri, Nov 3, 2017 at 1:25 PM, Ewen Cheslack-Postava <
> e...@confluent.io
> > >
> > > wrote:
> > >
> > > > +1 binding
> > > >
> > > > Thanks Jeff!
> > > >
> > > > On Wed, Nov 1, 2017 at 5:21 PM, Randall Hauch 
> > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Thanks for pushing this through. Great work!
> > > > >
> > > > > Randall Hauch
> > > > >
> > > > > On Wed, Nov 1, 2017 at 9:40 AM, Jeff Klukas 
> > > wrote:
> > > > >
> > > > > > I haven't heard any additional concerns over the proposal, so I'd
> > > like
> > > > to
> > > > > > get the voting process started for:
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 215%3A+Add+topic+regex+support+for+Connect+sinks
> > > > > >
> > > > > > It adds a topics.regex option for Kafka Connect sinks as an
> > > alternative
> > > > > to
> > > > > > the existing required topics option.
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: [VOTE] KIP-215: Add topic regex support for Connect sinks

[Jeff Klukas]

Looks like we've achieved lazy majority, so I'll move the KIP to approved.

Thanks all for looking this over.

On Fri, Nov 3, 2017 at 7:31 PM, Jason Gustafson  wrote:

> +1. Thanks for the KIP!
>
> On Fri, Nov 3, 2017 at 2:15 PM, Guozhang Wang  wrote:
>
> > +1 binding
> >
> > On Fri, Nov 3, 2017 at 1:25 PM, Ewen Cheslack-Postava  >
> > wrote:
> >
> > > +1 binding
> > >
> > > Thanks Jeff!
> > >
> > > On Wed, Nov 1, 2017 at 5:21 PM, Randall Hauch 
> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks for pushing this through. Great work!
> > > >
> > > > Randall Hauch
> > > >
> > > > On Wed, Nov 1, 2017 at 9:40 AM, Jeff Klukas 
> > wrote:
> > > >
> > > > > I haven't heard any additional concerns over the proposal, so I'd
> > like
> > > to
> > > > > get the voting process started for:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 215%3A+Add+topic+regex+support+for+Connect+sinks
> > > > >
> > > > > It adds a topics.regex option for Kafka Connect sinks as an
> > alternative
> > > > to
> > > > > the existing required topics option.
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

[jira] [Created] (KAFKA-6170) Add the AdminClient in Streams' KafkaClientSupplier

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-6170:


 Summary: Add the AdminClient in Streams' KafkaClientSupplier
 Key: KAFKA-6170
 URL: https://issues.apache.org/jira/browse/KAFKA-6170
 Project: Kafka
  Issue Type: New Feature
  Components: streams
Reporter: Guozhang Wang
Assignee: Guozhang Wang
Priority: Major


We will add Java AdminClient to Kafka Streams, in order to replace the internal 
StreamsKafkaClient. More details can be found in KIP-220 
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-220%3A+Add+AdminClient+into+Kafka+Streams%27+ClientSupplier)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-215: Add topic regex support for Connect sinks

[Jason Gustafson]

+1. Thanks for the KIP!

On Fri, Nov 3, 2017 at 2:15 PM, Guozhang Wang  wrote:

> +1 binding
>
> On Fri, Nov 3, 2017 at 1:25 PM, Ewen Cheslack-Postava 
> wrote:
>
> > +1 binding
> >
> > Thanks Jeff!
> >
> > On Wed, Nov 1, 2017 at 5:21 PM, Randall Hauch  wrote:
> >
> > > +1 (non-binding)
> > >
> > > Thanks for pushing this through. Great work!
> > >
> > > Randall Hauch
> > >
> > > On Wed, Nov 1, 2017 at 9:40 AM, Jeff Klukas 
> wrote:
> > >
> > > > I haven't heard any additional concerns over the proposal, so I'd
> like
> > to
> > > > get the voting process started for:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 215%3A+Add+topic+regex+support+for+Connect+sinks
> > > >
> > > > It adds a topics.regex option for Kafka Connect sinks as an
> alternative
> > > to
> > > > the existing required topics option.
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Guozhang Wang]

Thanks for writing up the KIP.

Vito, Matthias: one thing that I wanted to figure out first is what
categories of errors we want to notify the users, if we only wants to
distinguish fatal v.s. retriable then probably we should rename the
proposed StateStoreMigratedException / StateStoreClosedException classes.
And then from there we should list what are the possible internal
exceptions ever thrown in those APIs in the call trace, and which
exceptions should be wrapped to what others, and which ones should be
handled without re-throwing, and which ones should not be wrapped at all
but directly thrown to user's face.

Guozhang


On Wed, Nov 1, 2017 at 11:09 PM, vito jeng  wrote:

> Hi,
>
> I'd like to start discuss KIP-216:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
>
> Please have a look.
> Thanks!
>
> ---
> Vito
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-219 - Improve Quota Communication

[Rajini Sivaram]

Hi Becket,

Thank you for the KIP. A few comments:

1.KIP says:  "*No public interface changes are needed. We only propose
behavior change on the broker side.*"

But from the proposed changes, it sounds like clients will be updated to
wait for throttle-time before sending next response, and also not handle
idle disconnections during that time. Doesn't that mean that clients need
to know that the broker has sent the response before throttling, requiring
protocol/version change?


2. At the moment, broker failures are detected by clients (and vice versa)
within connections.max.idle.ms. By removing this check for an unlimited
throttle time, failure detection could be delayed.


3. KIP says  "*Since this subsequent request is not actually handled until
the broker unmutes the channel, the client can hit request.timeout.ms
 and reconnect. However, this is no worse than
the current state.*"

I think this could be worse than the current state because broker doesn't
detect and close the channel for an unlimited throttle time, while new
connections will get accepted. As a result, lot of connections could be in
CLOSE_WAIT state when throttle time is high.


Perhaps it is better to combine this KIP with a bound on throttle time?


Regards,


Rajini


On Fri, Nov 3, 2017 at 8:09 PM, Becket Qin  wrote:

> Thanks for the comment, Jun,
>
> 1. Yes, you are right. This could also happen with the current quota
> mechanism because we are essentially muting the socket during throttle
> time. There might be two ways to solve this.
> A) use another socket to send heartbeat.
> B) let the GroupCoordinator know that the client will not heartbeat for
> some time.
> It seems the first solution is cleaner.
>
> 2. For consumer it seems returning an empty response is a better option. In
> the producer case, if there is a spike in traffic. The brokers will see
> queued up requests, but that is hard to avoid unless we have connection
> level quota, which is a bigger change and may be easier to discuss it in a
> separate KIP.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On Fri, Nov 3, 2017 at 10:28 AM, Jun Rao  wrote:
>
> > Hi, Jiangjie,
> >
> > Thanks for bringing this up. A couple of quick thoughts.
> >
> > 1. If the throttle time is large, what can happen is that a consumer
> won't
> > be able to heart beat to the group coordinator frequent enough. In that
> > case, even with this KIP, it seems there could be frequent consumer group
> > rebalances.
> >
> > 2. If we return a response immediately, for the consumer, do we return
> the
> > requested data or an empty response? If we do the former, it may not
> > protect against the case when there are multiple consumer instances
> > associated with the same user/clientid.
> >
> > Jun
> >
> > On Wed, Nov 1, 2017 at 9:53 AM, Becket Qin  wrote:
> >
> > > Hi,
> > >
> > > We would like to start the discussion on KIP-219.
> > >
> > > The KIP tries to improve quota throttling time communication between
> > > brokers and clients to avoid clients timeout in case of long throttling
> > > time.
> > >
> > > The KIP link is following:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 219+-+Improve+quota+
> > > communication
> > >
> > > Comments are welcome.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> >
>

[jira] [Created] (KAFKA-6169) Kafka Log should reject negative timestamps

[Ryan P (JIRA)]

Ryan P created KAFKA-6169:
-

 Summary: Kafka Log should reject negative timestamps 
 Key: KAFKA-6169
 URL: https://issues.apache.org/jira/browse/KAFKA-6169
 Project: Kafka
  Issue Type: Bug
Reporter: Ryan P
Priority: Minor


Currently it would appear we rely on the Java Producer to prevent negative 
timestamps from being appended to the log. The broker should also include a 
validation which prevents poorly written third-party clients from doing the 
same. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6168) Connect Schema comparison is slow for large schemas

[Randall Hauch (JIRA)]

Randall Hauch created KAFKA-6168:


 Summary: Connect Schema comparison is slow for large schemas
 Key: KAFKA-6168
 URL: https://issues.apache.org/jira/browse/KAFKA-6168
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Affects Versions: 1.0.0
Reporter: Randall Hauch
Priority: Critical


The {{ConnectSchema}} implementation computes the hash code every time its 
needed, and {{equals(Object)}} is a deep equality check. This extra work can be 
expensive for large schemas, especially in code like the {{AvroConverter}} (or 
rather {{AvroData}} in the converter) that uses instances as keys in a hash map 
that then requires significant use of {{hashCode}} and {{equals}}.

The {{ConnectSchema}} is an immutable object and should at a minimum precompute 
the hash code. Also, the order that the fields are compared in {{equals(...)}} 
should use the cheapest comparisons first (e.g., the {{name}} field is one of 
the _last_ fields to be checked). Finally, it might be worth considering having 
each instance precompute and cache a string or byte[] representation of all 
fields that can be used for faster equality checking.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-215: Add topic regex support for Connect sinks

[Guozhang Wang]

+1 binding

On Fri, Nov 3, 2017 at 1:25 PM, Ewen Cheslack-Postava 
wrote:

> +1 binding
>
> Thanks Jeff!
>
> On Wed, Nov 1, 2017 at 5:21 PM, Randall Hauch  wrote:
>
> > +1 (non-binding)
> >
> > Thanks for pushing this through. Great work!
> >
> > Randall Hauch
> >
> > On Wed, Nov 1, 2017 at 9:40 AM, Jeff Klukas  wrote:
> >
> > > I haven't heard any additional concerns over the proposal, so I'd like
> to
> > > get the voting process started for:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 215%3A+Add+topic+regex+support+for+Connect+sinks
> > >
> > > It adds a topics.regex option for Kafka Connect sinks as an alternative
> > to
> > > the existing required topics option.
> > >
> >
>



-- 
-- Guozhang

Re: [VOTE] KIP-215: Add topic regex support for Connect sinks

[Ewen Cheslack-Postava]

+1 binding

Thanks Jeff!

On Wed, Nov 1, 2017 at 5:21 PM, Randall Hauch  wrote:

> +1 (non-binding)
>
> Thanks for pushing this through. Great work!
>
> Randall Hauch
>
> On Wed, Nov 1, 2017 at 9:40 AM, Jeff Klukas  wrote:
>
> > I haven't heard any additional concerns over the proposal, so I'd like to
> > get the voting process started for:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 215%3A+Add+topic+regex+support+for+Connect+sinks
> >
> > It adds a topics.regex option for Kafka Connect sinks as an alternative
> to
> > the existing required topics option.
> >
>

Re: [DISCUSS] KIP-219 - Improve Quota Communication

[Becket Qin]

Thanks for the comment, Jun,

1. Yes, you are right. This could also happen with the current quota
mechanism because we are essentially muting the socket during throttle
time. There might be two ways to solve this.
A) use another socket to send heartbeat.
B) let the GroupCoordinator know that the client will not heartbeat for
some time.
It seems the first solution is cleaner.

2. For consumer it seems returning an empty response is a better option. In
the producer case, if there is a spike in traffic. The brokers will see
queued up requests, but that is hard to avoid unless we have connection
level quota, which is a bigger change and may be easier to discuss it in a
separate KIP.

Thanks,

Jiangjie (Becket) Qin


On Fri, Nov 3, 2017 at 10:28 AM, Jun Rao  wrote:

> Hi, Jiangjie,
>
> Thanks for bringing this up. A couple of quick thoughts.
>
> 1. If the throttle time is large, what can happen is that a consumer won't
> be able to heart beat to the group coordinator frequent enough. In that
> case, even with this KIP, it seems there could be frequent consumer group
> rebalances.
>
> 2. If we return a response immediately, for the consumer, do we return the
> requested data or an empty response? If we do the former, it may not
> protect against the case when there are multiple consumer instances
> associated with the same user/clientid.
>
> Jun
>
> On Wed, Nov 1, 2017 at 9:53 AM, Becket Qin  wrote:
>
> > Hi,
> >
> > We would like to start the discussion on KIP-219.
> >
> > The KIP tries to improve quota throttling time communication between
> > brokers and clients to avoid clients timeout in case of long throttling
> > time.
> >
> > The KIP link is following:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 219+-+Improve+quota+
> > communication
> >
> > Comments are welcome.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
>

Jenkins build is back to normal : kafka-trunk-jdk8 #2188

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk7 #2946

[Apache Jenkins Server]

See 


Changes:

[wangguoz] HOTFIX: Add missing template ref in upgrade section

--
[...truncated 382.92 KB...]
kafka.api.TransactionsTest > testFencingOnSendOffsets STARTED

kafka.api.TransactionsTest > testFencingOnSendOffsets PASSED

kafka.api.TransactionsTest > testFencingOnAddPartitions STARTED

kafka.api.TransactionsTest > testFencingOnAddPartitions PASSED

kafka.api.TransactionsTest > testFencingOnTransactionExpiration STARTED

kafka.api.TransactionsTest > testFencingOnTransactionExpiration PASSED

kafka.api.TransactionsTest > testDelayedFetchIncludesAbortedTransaction STARTED

kafka.api.TransactionsTest > testDelayedFetchIncludesAbortedTransaction PASSED

kafka.api.TransactionsTest > testReadCommittedConsumerShouldNotSeeUndecidedData 
STARTED

kafka.api.TransactionsTest > testReadCommittedConsumerShouldNotSeeUndecidedData 
PASSED

kafka.api.TransactionsTest > testFencingOnSend STARTED

kafka.api.TransactionsTest > testFencingOnSend PASSED

kafka.api.TransactionsTest > testFencingOnCommit STARTED

kafka.api.TransactionsTest > testFencingOnCommit PASSED

kafka.api.TransactionsTest > testMultipleMarkersOneLeader STARTED

kafka.api.TransactionsTest > testMultipleMarkersOneLeader PASSED

kafka.api.TransactionsTest > testSendOffsets STARTED

kafka.api.TransactionsTest > testSendOffsets PASSED

kafka.api.GroupCoordinatorIntegrationTest > 
testGroupCoordinatorPropagatesOfffsetsTopicCompressionCodec STARTED

kafka.api.GroupCoordinatorIntegrationTest > 
testGroupCoordinatorPropagatesOfffsetsTopicCompressionCodec PASSED

kafka.api.AdminClientIntegrationTest > testDescribeReplicaLogDirs STARTED

kafka.api.AdminClientIntegrationTest > testDescribeReplicaLogDirs PASSED

kafka.api.AdminClientIntegrationTest > testInvalidAlterConfigs STARTED

kafka.api.AdminClientIntegrationTest > testInvalidAlterConfigs PASSED

kafka.api.AdminClientIntegrationTest > testClose STARTED

kafka.api.AdminClientIntegrationTest > testClose PASSED

kafka.api.AdminClientIntegrationTest > testMinimumRequestTimeouts STARTED

kafka.api.AdminClientIntegrationTest > testMinimumRequestTimeouts PASSED

kafka.api.AdminClientIntegrationTest > testForceClose STARTED

kafka.api.AdminClientIntegrationTest > testForceClose PASSED

kafka.api.AdminClientIntegrationTest > testListNodes STARTED

kafka.api.AdminClientIntegrationTest > testListNodes PASSED

kafka.api.AdminClientIntegrationTest > testDelayedClose STARTED

kafka.api.AdminClientIntegrationTest > testDelayedClose PASSED

kafka.api.AdminClientIntegrationTest > testCreateDeleteTopics STARTED

kafka.api.AdminClientIntegrationTest > testCreateDeleteTopics PASSED

kafka.api.AdminClientIntegrationTest > testDescribeLogDirs STARTED

kafka.api.AdminClientIntegrationTest > testDescribeLogDirs PASSED

kafka.api.AdminClientIntegrationTest > testAlterReplicaLogDirs STARTED

kafka.api.AdminClientIntegrationTest > testAlterReplicaLogDirs PASSED

kafka.api.AdminClientIntegrationTest > testAclOperations STARTED

kafka.api.AdminClientIntegrationTest > testAclOperations PASSED

kafka.api.AdminClientIntegrationTest > testDescribeCluster STARTED

kafka.api.AdminClientIntegrationTest > testDescribeCluster PASSED

kafka.api.AdminClientIntegrationTest > testCreatePartitions STARTED

kafka.api.AdminClientIntegrationTest > testCreatePartitions PASSED

kafka.api.AdminClientIntegrationTest > testDescribeNonExistingTopic STARTED

kafka.api.AdminClientIntegrationTest > testDescribeNonExistingTopic PASSED

kafka.api.AdminClientIntegrationTest > testDescribeAndAlterConfigs STARTED

kafka.api.AdminClientIntegrationTest > testDescribeAndAlterConfigs PASSED

kafka.api.AdminClientIntegrationTest > testCallInFlightTimeouts STARTED

kafka.api.AdminClientIntegrationTest > testCallInFlightTimeouts PASSED

kafka.api.LogAppendTimeTest > testProduceConsume STARTED

kafka.api.LogAppendTimeTest > testProduceConsume PASSED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testTransactionalProducerWithAuthenticationFailure STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testTransactionalProducerWithAuthenticationFailure PASSED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testManualAssignmentConsumerWithAuthenticationFailure STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testManualAssignmentConsumerWithAuthenticationFailure PASSED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testProducerWithAuthenticationFailure STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testProducerWithAuthenticationFailure PASSED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testConsumerGroupServiceWithAuthenticationFailure STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testConsumerGroupServiceWithAuthenticationFailure PASSED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testManualAssignmentConsumerWithAutoCommitDisabledWithAuthenticationFailure 
STARTED

Build failed in Jenkins: kafka-trunk-jdk9 #169

[Apache Jenkins Server]

See 


Changes:

[wangguoz] HOTFIX: Add missing template ref in upgrade section

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H25 (couchdbtest ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 520b3136280dc3dd427e773b88229051d59a 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 520b3136280dc3dd427e773b88229051d59a
Commit message: "HOTFIX: Add missing template ref in upgrade section"
 > git rev-list 487436b1a46b728904e543456c8bcc0d3ceea55a # timeout=10
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
[kafka-trunk-jdk9] $ /bin/bash -xe /tmp/jenkins7465879609551807470.sh
+ rm -rf 
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2/bin/gradle

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine java version from '9.0.1'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did tests run? 
For example, 

 is 7 days 2 hr old

Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
Not sending mail to unregistered user wangg...@gmail.com
Not sending mail to unregistered user rajinisiva...@googlemail.com

Re: [DISCUSS] KIP-219 - Improve Quota Communication

[Jun Rao]

Hi, Jiangjie,

Thanks for bringing this up. A couple of quick thoughts.

1. If the throttle time is large, what can happen is that a consumer won't
be able to heart beat to the group coordinator frequent enough. In that
case, even with this KIP, it seems there could be frequent consumer group
rebalances.

2. If we return a response immediately, for the consumer, do we return the
requested data or an empty response? If we do the former, it may not
protect against the case when there are multiple consumer instances
associated with the same user/clientid.

Jun

On Wed, Nov 1, 2017 at 9:53 AM, Becket Qin  wrote:

> Hi,
>
> We would like to start the discussion on KIP-219.
>
> The KIP tries to improve quota throttling time communication between
> brokers and clients to avoid clients timeout in case of long throttling
> time.
>
> The KIP link is following:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-219+-+Improve+quota+
> communication
>
> Comments are welcome.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>

Re: [VOTE] KIP-203: Add toLowerCase support to sasl.kerberos.principal.to.local rule

[Manikumar]

Bump up. waiting for few more binding votes.

On Wed, Oct 18, 2017 at 6:57 PM, Rajini Sivaram 
wrote:

> +1 (binding)
>
> On Mon, Oct 9, 2017 at 5:32 PM, Manikumar 
> wrote:
>
> > I'm bumping this up to get some attention :)
> >
> > On Wed, Sep 27, 2017 at 8:46 PM, Tom Bentley 
> > wrote:
> >
> > > +1 (nonbinding)
> > >
> > > On 27 September 2017 at 16:10, Manikumar 
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I'd like to start the vote on KIP-203. Details are here:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 203%3A+Add+toLowerCase+support+to+sasl.kerberos.
> > principal.to.local+rule
> > > >
> > > > Thanks,
> > > >
> > >
> >
>

Build failed in Jenkins: kafka-trunk-jdk9 #168

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Update docs for new version

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-4 (ubuntu trusty) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 487436b1a46b728904e543456c8bcc0d3ceea55a 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 487436b1a46b728904e543456c8bcc0d3ceea55a
Commit message: "MINOR: Update docs for new version"
 > git rev-list 4fac83ba1f80353e9544b15b95b8da9dc557041d # timeout=10
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
[kafka-trunk-jdk9] $ /bin/bash -xe /tmp/jenkins3177976388104819954.sh
+ rm -rf 
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2/bin/gradle

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine java version from '9.0.1'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did tests run? 
For example, 

 is 7 days 13 hr old

Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
Not sending mail to unregistered user wangg...@gmail.com
Not sending mail to unregistered user rajinisiva...@googlemail.com

Build failed in Jenkins: kafka-trunk-jdk8 #2187

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Update docs for new version

--
[...truncated 380.72 KB...]

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryTailIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnUnsupportedIfNoEpochRecorded PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 

[jira] [Created] (KAFKA-6167) Timestamp on streams directory contains a colon, which is an illegal character

[Justin Manchester (JIRA)]

Justin Manchester created KAFKA-6167:


 Summary: Timestamp on streams directory contains a colon, which is 
an illegal character
 Key: KAFKA-6167
 URL: https://issues.apache.org/jira/browse/KAFKA-6167
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.0
 Environment: AK 1.0.0
Kubernetes
CoreOS
JDK 1.8
Windows
Reporter: Justin Manchester
Priority: Normal


Problem:

Development on Windows, which is not fully supported, however still a bug that 
should be corrected.

It looks like a timestamp was added to the streams directory using a colon as 
separator. I believe this is an illegal character and potentially the cause for 
the exception below.

Error Stack:

2017-11-02 16:06:41 ERROR 
[StreamDeduplicatorAcceptanceTest1-a3ae0ac6-a024-4006-bcb1-01ff0f433f6e-StreamThread-1]
 org.apache.kafka.streams.processor.internals.AssignedTasks:301 - stream-thread 
[StreamDeduplicatorAcceptanceTest1-a3ae0ac6-a024-4006-bcb1-01ff0f433f6e-StreamThread-1]
 Failed to process stream task 0_0 due to the following error: 
org.apache.kafka.streams.errors.StreamsException: Exception caught in process. 
taskId=0_0, processor=KSTREAM-SOURCE-00, topic=input-a_1, partition=0, 
offset=0 
at 
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:232)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:403)
 [kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:317)
 [kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:942)
 [kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:822)
 [kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
 [kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
 [kafka-streams-1.0.0.jar:?] 
Caused by: org.apache.kafka.streams.errors.ProcessorStateException: Error 
opening store KSTREAM-JOINTHIS-04-store:150962400 at location 
C:\Users\ADRIAN~1.MCC\AppData\Local\Temp\kafka3548813472740086814\StreamDeduplicatorAcceptanceTest1\0_0\KSTREAM-JOINTHIS-04-store\KSTREAM-JOINTHIS-04-store:150962400
 
at 
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:204)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:174)
 ~[kafka-streams-1.0.0.jar:?] 
at org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:40) 
~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:89)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:81)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.state.internals.RocksDBWindowStore$RocksDBWindowBytesStore.put(RocksDBWindowStore.java:43)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.state.internals.RocksDBWindowStore$RocksDBWindowBytesStore.put(RocksDBWindowStore.java:34)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:67)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:33)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:96)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:89)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.kstream.internals.KStreamJoinWindow$KStreamJoinWindowProcessor.process(KStreamJoinWindow.java:64)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:46)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:208)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:124)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:85)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:80)
 ~[kafka-streams-1.0.0.jar:?] 
at 
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:216)
 ~[kafka-streams-1.0.0.jar:?] 
... 6 more 
Caused by: 

[GitHub] kafka pull request #4169: MINOR: Update docs for new version

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4169


---

[GitHub] kafka pull request #4175: KAFKA-6161 add base classes for (De)Serializers wi...

[evis]

GitHub user evis opened a pull request:

https://github.com/apache/kafka/pull/4175

KAFKA-6161 add base classes for (De)Serializers with empty conf methods

All (de)serializers, which have empty configure() and/or close() methods, 
are now inherit NoConf(De)Serializer. Also, such classes are created for 
extended (de)serializers.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/evis/kafka 
KAFKA-6161-serde-empty-conf-close-methods

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4175.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4175


commit 297115a53f6ea6c1bfd2383c0f73f011cb94c505
Author: Evgeny Veretennikov 
Date:   2017-11-03T15:38:18Z

KAFKA-6161 add base classes for (De)Serializers with empty conf methods

All (de)serializers, which have empty configure() and/or close()
methods, are now inherit NoConf(De)Serializer. Also, such classes are
created for extended (de)serializers.




---

[jira] [Resolved] (KAFKA-6159) Link to upgrade docs in 1.0.0 release notes is broken

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-6159.
--
Resolution: Fixed

> Link to upgrade docs in 1.0.0 release notes is broken
> -
>
> Key: KAFKA-6159
> URL: https://issues.apache.org/jira/browse/KAFKA-6159
> Project: Kafka
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 1.0.0
>Reporter: Martin Schröder
>Assignee: Guozhang Wang
>  Labels: release-notes
>
> The release notes for 1.0.0 
> (https://dist.apache.org/repos/dist/release/kafka/1.0.0/RELEASE_NOTES.html) 
> point to http://kafka.apache.org/100/documentation.html#upgrade for "upgrade 
> documentation", but that gives a 404.
> Maybe you mean http://kafka.apache.org/documentation.html#upgrade ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4174: MINOR: Add pull request template

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/4174

MINOR: Add pull request template



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka pull-request-template

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4174.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4174


commit d4334d8e1fd144ce08dae8767cb8181f16223157
Author: Ismael Juma 
Date:   2017-11-03T13:06:03Z

Add pull request template




---

Jenkins build is back to normal : kafka-trunk-jdk7 #2944

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-6166) Streams configuration requires consumer. and producer. in order to be read

[Justin Manchester (JIRA)]

Justin Manchester created KAFKA-6166:


 Summary: Streams configuration requires consumer. and producer. in 
order to be read
 Key: KAFKA-6166
 URL: https://issues.apache.org/jira/browse/KAFKA-6166
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.11.0.0
 Environment: Kafka 0.11.0.0
JDK 1.8
CoreOS
Reporter: Justin Manchester
Priority: Minor


Problem:

In previous release you could specify a custom metrics reporter like so:

Properties config = new Properties(); 
config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker1:9092"); 
config.put(StreamsConfig.METRIC_REPORTER_CLASSES_CONFIG, 
"com.mycompany.MetricReporter"); 
config.put("custom-key-for-metric-reporter", "value");

>From 0.11.0.0 onwards this is no longer possible, as you have to specify 
>consumer.custom-key-for-metric-reporter or 
>producer.custom-key-for-metric-reporter otherwise it's stripped out of the 
>configuration.

So, if you wish to use a metrics reporter and to collect producer and consumer 
metrics, as well as kafka-streams metrics, that you would need to specify 3 
distinct configs:

1) consumer.custom-key-for-metric-reporter 
2) producer.custom-key-for-metric-reporter 
3) custom-key-for-metric-reporter

This appears to be a regression.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-trunk-jdk7 #2943

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] KAFKA-6060; Add workload generation capabilities to Trogdor

--
[...truncated 382.47 KB...]

kafka.security.auth.ZkAuthorizationTest > testZkUtils STARTED

kafka.security.auth.ZkAuthorizationTest > testZkUtils PASSED

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testZkMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testChroot STARTED

kafka.security.auth.ZkAuthorizationTest > testChroot PASSED

kafka.security.auth.ZkAuthorizationTest > testDelete STARTED

kafka.security.auth.ZkAuthorizationTest > testDelete PASSED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive STARTED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete STARTED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
STARTED

Build failed in Jenkins: kafka-trunk-jdk9 #167

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] KAFKA-6060; Add workload generation capabilities to Trogdor

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H22 (couchdbtest ubuntu xenial) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 4fac83ba1f80353e9544b15b95b8da9dc557041d 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4fac83ba1f80353e9544b15b95b8da9dc557041d
Commit message: "KAFKA-6060; Add workload generation capabilities to Trogdor"
 > git rev-list e4208b1d5fa1c28ac7e64e2cb039404a14084dc0 # timeout=10
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
[kafka-trunk-jdk9] $ /bin/bash -xe /tmp/jenkins578210591197694.sh
+ rm -rf 
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2/bin/gradle

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine java version from '9.0.1'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
Not sending mail to unregistered user wangg...@gmail.com
Not sending mail to unregistered user rajinisiva...@googlemail.com

[jira] [Resolved] (KAFKA-6060) Add workload generation capabilities to Trogdor

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-6060.
---
   Resolution: Fixed
Fix Version/s: 1.1.0

Issue resolved by pull request 4073
https://github.com/apache/kafka/pull/4073

> Add workload generation capabilities to Trogdor
> ---
>
> Key: KAFKA-6060
> URL: https://issues.apache.org/jira/browse/KAFKA-6060
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>Priority: Major
> Fix For: 1.1.0
>
>
> Add workload generation capabilities to Trogdor



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4073: KAFKA-6060: Add workload generation capabilities t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4073


---

Re: kafka-pr-jdk9-scala2.12 keeps failing

[Ismael Juma]

This looks to be an issue in Jenkins, not in Kafka. Apache Infra updated
Java 9 to 9.0.1 and it seems to have broken some of the Jenkins code.

Ismael

On 3 Nov 2017 1:53 am, "Ted Yu"  wrote:

> Looking at earlier runs, e.g. :
> https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/2384/console
>
> FAILURE: Build failed with an exception.
>
> * What went wrong:
> Could not determine java version from '9.0.1'.
>
>
> This was the first build with 'out of range of int' exception:
>
>
> https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/2389/console
>
>
> However, I haven't found the commit which was at the tip of repo at that
> time.
>
>
> On Thu, Nov 2, 2017 at 6:40 PM, Guozhang Wang  wrote:
>
> > Noticed that as well, could we track down to which git commit / version
> > upgrade caused the issue?
> >
> >
> > Guozhang
> >
> > On Thu, Nov 2, 2017 at 6:25 PM, Ted Yu  wrote:
> >
> > > Hi,
> > > I took a look at recent runs under https://builds.apache.
> > > org/job/kafka-pr-jdk9-scala2.12
> > >
> > > All the recent runs failed with:
> > >
> > > Could not update commit status of the Pull Request on GitHub.
> > > org.kohsuke.github.HttpException: Server returned HTTP response code:
> > > 201, message: 'Created' for URL:
> > > https://api.github.com/repos/apache/kafka/statuses/
> > > 3d96c6f5b2edd3c1dbea11dab003c4ac78ee141a
> > > at org.kohsuke.github.Requester.parse(Requester.java:633)
> > > at org.kohsuke.github.Requester.parse(Requester.java:594)
> > > at org.kohsuke.github.Requester._to(Requester.java:272)
> > > at org.kohsuke.github.Requester.to(Requester.java:234)
> > > at org.kohsuke.github.GHRepository.createCommitStatus(
> > > GHRepository.java:1071)
> > >
> > > ...
> > >
> > > Caused by: com.fasterxml.jackson.databind.JsonMappingException:
> > > Numeric value (4298492118) out of range of int
> > >  at [Source: {"url":"https://api.github.com/repos/apache/kafka/
> statuses/
> > > 3d96c6f5b2edd3c1dbea11dab003c4ac78ee141a","id":4298492118,"
> > > state":"pending","description":"Build
> > > started sha1 is
> > > merged.","target_url":"https://builds.apache.org/job/kafka-
> > > pr-jdk9-scala2.12/2397/","context":"JDK
> > > 9 and Scala 2.12",
> > >
> > >
> > > Should we upgrade the version for jackson ?
> > >
> > >
> > > Cheers
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

[jira] [Created] (KAFKA-6165) Kafka Brokers goes down with outOfMemoryError.

[kaushik srinivas (JIRA)]

kaushik srinivas created KAFKA-6165:
---

 Summary: Kafka Brokers goes down with outOfMemoryError.
 Key: KAFKA-6165
 URL: https://issues.apache.org/jira/browse/KAFKA-6165
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.0
 Environment: DCOS cluster with 4 agent nodes and 3 masters.

agent machine config :
RAM : 384 GB
DISK : 4TB


Reporter: kaushik srinivas
Priority: Major
 Attachments: kafka_config.txt, stderr_broker1.txt, stderr_broker2.txt, 
stdout_broker1.txt, stdout_broker2.txt

Performance testing kafka with end to end pipe lines of,
Kafka Data Producer -> kafka -> spark streaming -> hdfs -- stream1
Kafka Data Producer -> kafka -> flume -> hdfs -- stream2

stream1 kafka configs :
No of topics : 10
No of partitions : 20 for all the topics

stream2 kafka configs :
No of topics : 10
No of partitions : 20 for all the topics

Some important Kafka Configuration :
"BROKER_MEM": "32768"(32GB)
"BROKER_JAVA_HEAP": "16384"(16GB)
"BROKER_COUNT": "3"
"KAFKA_MESSAGE_MAX_BYTES": "112"(1MB)
"KAFKA_REPLICA_FETCH_MAX_BYTES": "1048576"(1MB)
"KAFKA_NUM_PARTITIONS": "20"
"BROKER_DISK_SIZE": "5000" (5GB)
"KAFKA_LOG_SEGMENT_BYTES": "5000",(50MB)
"KAFKA_LOG_RETENTION_BYTES": "50"(5GB)

Data Producer to kafka Throughput:

message rate : 5 lakhs messages/sec approx across all the 3 brokers and 
topics/partitions.
message size : approx 300 to 400 bytes.

Issues observed with this configs:

Issue 1:

stack trace:

[2017-11-03 00:56:28,484] FATAL [Replica Manager on Broker 0]: Halting due to 
unrecoverable I/O error while handling produce request:  
(kafka.server.ReplicaManager)
kafka.common.KafkaStorageException: I/O exception in append to log 
'store_sales-16'
at kafka.log.Log.append(Log.scala:349)
at kafka.cluster.Partition$$anonfun$10.apply(Partition.scala:443)
at kafka.cluster.Partition$$anonfun$10.apply(Partition.scala:429)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:240)
at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:429)
at 
kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:407)
at 
kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:393)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at 
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at 
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at 
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at 
kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:393)
at kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:330)
at kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:425)
at kafka.server.KafkaApis.handle(KafkaApis.scala:78)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
at 
kafka.log.AbstractIndex$$anonfun$resize$1.apply(AbstractIndex.scala:116)
at 
kafka.log.AbstractIndex$$anonfun$resize$1.apply(AbstractIndex.scala:106)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.log.AbstractIndex.resize(AbstractIndex.scala:106)
at 
kafka.log.AbstractIndex$$anonfun$trimToValidSize$1.apply$mcV$sp(AbstractIndex.scala:160)
at 
kafka.log.AbstractIndex$$anonfun$trimToValidSize$1.apply(AbstractIndex.scala:160)
at 
kafka.log.AbstractIndex$$anonfun$trimToValidSize$1.apply(AbstractIndex.scala:160)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:159)
at kafka.log.Log.roll(Log.scala:771)
at kafka.log.Log.maybeRoll(Log.scala:742)
at kafka.log.Log.append(Log.scala:405)
... 22 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
... 34 more


Issue 2 :

stack trace :

[2017-11-02 23:55:49,602] FATAL [ReplicaFetcherThread-0-0], Disk error while 
replicating data for catalog_sales-3 

[GitHub] kafka pull request #4173: KAFKA-6156: Metric tag name should not contain col...

[huxihx]

GitHub user huxihx opened a pull request:

https://github.com/apache/kafka/pull/4173

KAFKA-6156: Metric tag name should not contain colons.

Windows directory paths often contain colons which are now allowed in 
yammer metrics. Should convert to its corresponding Unix style path before 
creating metrics.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/huxihx/kafka KAFKA-6156

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4173.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4173


commit 36a1188a3f9c93d21f68b591d4fd16fa90bd8bad
Author: huxihx 
Date:   2017-11-03T06:13:47Z

KAFKA-6156: Metric tag name should not contain colons.

Windows directory paths often contain colons which are now allowed in 
yammer metrics. Should convert to its corresponding Unix style path before 
creating metrics.




---