Jenkins build is back to normal : kafka-trunk-jdk8 #2776

See

[jira] [Created] (KAFKA-7113) AdminClient.create has an infinite retry loop bug

02tmqrmje9 created KAFKA-7113: - Summary: AdminClient.create has an infinite retry loop bug Key: KAFKA-7113 URL: https://issues.apache.org/jira/browse/KAFKA-7113 Project: Kafka Issue Type: Bug

[jira] [Resolved] (KAFKA-7104) ReplicaFetcher thread may die because of inconsistent log start offset in fetch response

[ https://issues.apache.org/jira/browse/KAFKA-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-7104. Resolution: Fixed > ReplicaFetcher thread may die because of inconsistent log start offset in >

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

The “Motivation” section of the KIP says that the starting offset will be useful but doesn’t say why. Can you add a use-case or two to describe how it will be useful? In our case (see https://github.com/wushujames/kafka-utilities/blob/master/ConsumerGroupLag/README.md), we found the starting

Re: [DISCUSS] KIP-308: Support dynamic update of max.connections.per.ip/max.connections.per.ip.overrides configs

Thanks for the responses. If there are no concerns, I will start the voting process soon. On Tue, Jun 26, 2018 at 10:14 PM Harsha wrote: > This is very useful. LGTM. > > Thanks, > Harsha > > On Mon, Jun 25th, 2018 at 10:20 AM, Dong Lin wrote: > > > > > > > > > Hey Manikumar, > > > > Thanks

Build failed in Jenkins: kafka-trunk-jdk8 #2775

See Changes: [jason] KAFKA-7076; Skip rebuilding producer state when using old message format [jason] KAFKA-7091; AdminClient should handle FindCoordinatorResponse errors

Jenkins build is back to normal : kafka-2.0-jdk8 #59

See

Re: [VOTE] KIP-330: Add retentionPeriod in SessionBytesStoreSupplier

+1 On Wed, Jun 27, 2018 at 4:40 PM, Bill Bejeck wrote: > +1 > > -Bill > > On Wed, Jun 27, 2018 at 7:39 PM Guozhang Wang wrote: > > > Hello folks, > > > > I'd like to start a voting thread on KIP-330. I've intentionally skipped > > the discuss phase since it is a pretty straight-forward public

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Hi Bill, Thanks for the review! Your question is very much applicable to the KIP and not at all an implementation detail. Thanks for bringing it up. I'm proposing not to change the existing caches and configurations at all (for now). Imagine you have a topology like this: commit.interval.ms =

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

It is incredibly weird, but it is also a repeating pattern. I figured we may as well address this. I definitely explained how Kafka is different from traditional queues and what lag means vs partition size. To the best of my understanding, they see size as a measure of a topic importance in terms

[jira] [Created] (KAFKA-7112) StreamThread does not check for state again after pollRequests()

Guozhang Wang created KAFKA-7112: Summary: StreamThread does not check for state again after pollRequests() Key: KAFKA-7112 URL: https://issues.apache.org/jira/browse/KAFKA-7112 Project: Kafka

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

Hey Gwen, Why do users want partition size? It seems like a weird thing to be concerned about. Perhaps they are trying to get a sense of the lag as a percentage of the total size of the partition or something like that? -Jason On Tue, Jun 26, 2018 at 9:12 PM, Gwen Shapira wrote: > Thank you!

Re: [VOTE] KIP-330: Add retentionPeriod in SessionBytesStoreSupplier

+1 -Bill On Wed, Jun 27, 2018 at 7:39 PM Guozhang Wang wrote: > Hello folks, > > I'd like to start a voting thread on KIP-330. I've intentionally skipped > the discuss phase since it is a pretty straight-forward public API change > and should actually be added since day one. The bug fix of

[VOTE] KIP-330: Add retentionPeriod in SessionBytesStoreSupplier

Hello folks, I'd like to start a voting thread on KIP-330. I've intentionally skipped the discuss phase since it is a pretty straight-forward public API change and should actually be added since day one. The bug fix of KAFKA-7071 helped us to discover this overlook. -- -- Guozhang

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

I see! Thanks. -Yishun On Wed, Jun 27, 2018, 4:10 PM Guozhang Wang wrote: > Hi Yishun, > > We need to wait at least 72 business hours with three binding votes, > although you already have enough votes (me, Matthias, Damian) we still need > to wait enough hours for people to take a look and see

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

Hi Yishun, We need to wait at least 72 business hours with three binding votes, although you already have enough votes (me, Matthias, Damian) we still need to wait enough hours for people to take a look and see if they have any different opinions. After 72 hours have passed since you started the

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Sounds good to me. On Wed, Jun 27, 2018 at 3:57 PM, Jason Gustafson wrote: > Hey Guozhang, > > That's fair. In fact, perhaps we do not need this API at all. We already > have the new seek() in this KIP which can do the lookup based on epoch for > this use case. I guess we should probably call

[jira] [Resolved] (KAFKA-7091) AdminClient should handle FindCoordinatorResponse errors

[ https://issues.apache.org/jira/browse/KAFKA-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-7091. Resolution: Fixed > AdminClient should handle FindCoordinatorResponse errors >

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Hey Guozhang, That's fair. In fact, perhaps we do not need this API at all. We already have the new seek() in this KIP which can do the lookup based on epoch for this use case. I guess we should probably call it seekToNearest() though to make it clear that the final position may be different from

[jira] [Created] (KAFKA-7111) Review the NetworkClient log level used

Luan Cestari created KAFKA-7111: --- Summary: Review the NetworkClient log level used Key: KAFKA-7111 URL: https://issues.apache.org/jira/browse/KAFKA-7111 Project: Kafka Issue Type: Improvement

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

Added! Thank you Colin. Do we now have enough votes? I read the bylaws and still a little bit confused. Thanks. - Yishun On Wed, Jun 27, 2018 at 3:24 PM, Colin McCabe wrote: > P.S. +1 (non-binding) once you add the info about it being thread-safe. > > best, > > > On Wed, Jun 27, 2018, at

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

P.S. +1 (non-binding) once you add the info about it being thread-safe. best, On Wed, Jun 27, 2018, at 15:23, Colin McCabe wrote: > On Tue, Jun 26, 2018, at 13:24, Yishun Guan wrote: > > Hi Colin, > > > > I agree with what Guozhang's opinion that because all the other clients > > have it

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

On Tue, Jun 26, 2018, at 13:24, Yishun Guan wrote: > Hi Colin, > > I agree with what Guozhang's opinion that because all the other clients > have it (producer, consumer..) and this will gain more visibility for those > application that use admin client. (Now I added this sentence to the KIP) I

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Hi Jason, I think it is less worthwhile to add KafkaConsumer#offsetsForLeaderEpochs, since probably only very advanced users are aware of the leaderEpoch, and hence ever care to use it anyways. It is more like an admin client operation than a consumer client operation: if the motivation is to

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Hi John, thanks for the KIP. Early on in the KIP, you mention the current approaches for controlling the rate of downstream records from a KTable, cache size configuration and commit time. Will these configuration parameters still be in effect for tables that don't use suppression? For tables

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Thanks for the feedback. I've updated the KIP. Specifically I removed the "closest" reset option and the proposal to reset by timestamp when the precise truncation point cannot be determined. Instead, I proposed that we always reset using the nearest epoch when a reset policy is defined (either

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

+1 (binding) Thanks, Harsha On Wed, Jun 27th, 2018 at 10:56 AM, Damian Guy wrote: > > > > +1 (binding) > > Thanks > > > > On Wed, 27 Jun 2018 at 18:50 Bill Bejeck < bbej...@gmail.com > wrote: > > > +1 > > > > -Bill > > > > On Wed, Jun 27, 2018 at 12:47 PM Manikumar <

[jira] [Created] (KAFKA-7110) Windowed changelog keys not deserialized properly by TimeWindowedSerde

Shawn Nguyen created KAFKA-7110: --- Summary: Windowed changelog keys not deserialized properly by TimeWindowedSerde Key: KAFKA-7110 URL: https://issues.apache.org/jira/browse/KAFKA-7110 Project: Kafka

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Thanks for the feedback, Matthias, It seems like in straightforward relational processing cases, it would not make sense to bound the lateness of KTables. In general, it seems better to have "guard rails" in place that make it easier to write sensible programs than insensible ones. But I'm still

Re: [VOTE] KIP-277 - Fine Grained ACL for CreateTopics API

Hello guys, Sorry for being late on this KIP, but while incorporating the docs of 277 and 290 I'm wondering if we should be extending the authorization with create topics on other operations with these two KIPs: Previously, in SimpleAclAuthorizer, "read, write, delete, or alter implies allowing

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Hello again all, I realized today that I neglected to include metrics in the proposal. I have added them just now. Thanks, -John On Tue, Jun 26, 2018 at 3:11 PM John Roesler wrote: > Hello devs and users, > > Please take some time to consider this proposal for Kafka Streams: > > KIP-328:

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

@Dong Those are fair points. Both approaches require some fuzziness to reset the offset in these pathological scenarios and we cannot guarantee at-least-once delivery either way unless we have the full history of leader epochs that were consumed. The KIP-101 logic may actually be more accurate

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

Thanks for the reply Jason and Dhruvil. Yeah we don't need config for the sanity check and thus we don't need a KIP. I think we are on the same page of just skipping the sanity check of segments before the recovery offset. I will close the KIP and submit a patch for this. On Wed, Jun 27, 2018 at

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

+1 (binding) Thanks On Wed, 27 Jun 2018 at 18:50 Bill Bejeck wrote: > +1 > > -Bill > > On Wed, Jun 27, 2018 at 12:47 PM Manikumar > wrote: > > > +1 (non-binding) > > > > Thanks. > > > > On Wed, Jun 27, 2018 at 10:15 PM Matthias J. Sax > > wrote: > > > > > +1 (binding) > > > > > > On 6/26/18

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

+1 to what Jason said. We need a better long-term strategy for dealing with corrupted log and index data, but the sanity checks we have do not guarantee much in this regard. For now, we could do away with these index sanity checks in my opinion. We could handle the missing index case at startup.

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Thanks for the KIP John. One initial comments about the last example "Bounded lateness": For a non-windowed KTable bounding the lateness does not really make sense, does it? Thus, I am wondering if we should allow `suppressLateEvents()` for this case? It seems to be better to only allow it for

RE: [VOTE] KIP-235 Add DNS alias support for secured connection

Hi, I've modified the PR last week following comments on unit tests, could it be reviewed ? https://github.com/apache/kafka/pull/4485 Jonathan Skrzypek -Original Message- From: Ismael Juma [mailto:ism...@juma.me.uk] Sent: 23 May 2018 01:29 To: dev Subject: Re: [VOTE] KIP-235 Add DNS

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

+1 -Bill On Wed, Jun 27, 2018 at 12:47 PM Manikumar wrote: > +1 (non-binding) > > Thanks. > > On Wed, Jun 27, 2018 at 10:15 PM Matthias J. Sax > wrote: > > > +1 (binding) > > > > On 6/26/18 2:33 PM, Guozhang Wang wrote: > > > +1. Thanks. > > > > > > On Tue, Jun 26, 2018 at 2:31 PM, Yishun

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

+1 (non-binding) Thanks. On Wed, Jun 27, 2018 at 10:15 PM Matthias J. Sax wrote: > +1 (binding) > > On 6/26/18 2:33 PM, Guozhang Wang wrote: > > +1. Thanks. > > > > On Tue, Jun 26, 2018 at 2:31 PM, Yishun Guan wrote: > > > >> Hi All, > >> > >> I am starting a vote on this KIP: > >> > >>

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

+1 (binding) On 6/26/18 2:33 PM, Guozhang Wang wrote: > +1. Thanks. > > On Tue, Jun 26, 2018 at 2:31 PM, Yishun Guan wrote: > >> Hi All, >> >> I am starting a vote on this KIP: >> >> https://cwiki.apache.org/confluence/x/lQg0BQ >> >> Thanks, >> Yishun >> > > > signature.asc Description:

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Jason, thanks for the KIP. A few comments: * I think Dong's question about whether to use timestamp-based approach v.s. start-offset-of-first-larger-epoch is valid; more specifically, with timestamp-based approach we may still be reseting to an offset falling into the truncated interval, and

[jira] [Created] (KAFKA-7109) KafkaConsumer should close its incremental fetch sessions on close

Colin P. McCabe created KAFKA-7109: -- Summary: KafkaConsumer should close its incremental fetch sessions on close Key: KAFKA-7109 URL: https://issues.apache.org/jira/browse/KAFKA-7109 Project: Kafka

Re: [VOTE] 2.0.0 RC0

+1 Checked signatures Ran test suite On Mon, Jun 25, 2018 at 11:12 AM, Thomas Crayford wrote: > +1 (non-binding) Heroku has run our usual set of upgrade and performance > tests, and we haven't found any notable issues through that. > > On Sat, Jun 23, 2018 at 12:30 AM, Vahid S Hashemian < >

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

I noticed this (lack of primary parameter) as well. What you gave as new example is semantically the same as what I suggested. So it is good by me. Thanks On Wed, Jun 27, 2018 at 7:31 AM, John Roesler wrote: > Thanks for taking look, Ted, > > I agree this is a departure from the conventions

Re: [kafka-clients] Re: [VOTE] 1.1.1 RC1

+1 (non-binding). Built from source and ran integration tests successfully. Gradle 4.8.1, Java 1.8.0_171 (Oracle Corporation 25.171-b11), on Ubuntu 16.04 LTS On Tue, Jun 26, 2018 at 4:55 AM, Jakub Scholz wrote: > +1 (non-binding) ... I ran my tests and verified the RC1 with my > applications.

Re: [VOTE] 2.0.0 RC0

+1 (non-binding). Built from source and ran integration tests successfully. On Tue, Jun 26, 2018 at 4:57 AM, Jakub Scholz wrote: > +1 (non-binding) ... I ran my tests and verified the RC0 against my > applications. > > On Mon, Jun 25, 2018 at 8:12 PM Thomas Crayford > wrote: > > > +1

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

Hey Dong, So the main concern with the above approach is that, if for any reason the > index files of inactive segment is deleted or corrupted, the broker will > halt if there is only one log directory. This is different from the > existing behavior where the broker will rebuild the index for

Re: Requesting Permission To Create KIP And Assign JIRAs

Hello Kevin, It's done. Guozhang On Tue, Jun 26, 2018 at 10:37 PM, Kevin Lu wrote: > Hi All, > > I would like to start contributing to Kafka but I do not have access to > create KIPs or assign JIRA to myself. > > Can someone set it up for me? > > Confluence id: lu.kevin > Jira username:

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Thanks for taking look, Ted, I agree this is a departure from the conventions of Streams DSL. Most of our config objects have one or two "required" parameters, which fit naturally with the static factory method approach. TimeWindow, for example, requires a size parameter, so we can naturally say

Re: [kafka-clients] Re: [VOTE] 1.1.1 RC1

+1 non-binding Ran quick start and checked signatures On Wed, Jun 27, 2018 at 8:07 AM, Dong Lin wrote: > Thank you all for your test and votes! > > It will be great to have more votes from PMC so that we can conclude > kafka-1.1.1 release :)

Jenkins build is back to normal : kafka-trunk-jdk10 #256

See

Re: [DISCUSS] KIP-322: Return new error code for DeleteTopics API when topic deletion disabled.

bq. set the serialVersionUID = 1L in Kafka exception classes Indeed. Should have checked earlier. On Wed, Jun 27, 2018 at 1:38 AM, Manikumar wrote: > Hi, > > Thanks for the review. > > > > nit: you are going to fill in an actual value below in your PR, right ? > > private static final

Re: [DISCUSS] KIP-322: Return new error code for DeleteTopics API when topic deletion disabled.

Hi, Thanks for the review. > nit: you are going to fill in an actual value below in your PR, right ? > private static final long serialVersionUID = 1L; > > We normally set the serialVersionUID = 1L in Kafka exception classes. In Motivation, please mention the IllegalStateException

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

So the main concern with the above approach is that, if for any reason the index files of inactive segment is deleted or corrupted, the broker will halt if there is only one log directory. This is different from the existing behavior where the broker will rebuild the index for this inactive

Re: [kafka-clients] Re: [VOTE] 1.1.1 RC1

Thank you all for your test and votes! It will be great to have more votes from PMC so that we can conclude kafka-1.1.1 release :)

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

Hey Jason, Thanks for the comment! Your comment reminded me to read through Jay's comments and my reply again. It seems that I probably have not captured idea of Jay's comment that says sanity check is not part of any formal guarantee we provide. I probably should have thought about this comment

[jira] [Created] (KAFKA-7108) "Exactly-once" stream breaks production exception handler contract

Anna O created KAFKA-7108: - Summary: "Exactly-once" stream breaks production exception handler contract Key: KAFKA-7108 URL: https://issues.apache.org/jira/browse/KAFKA-7108 Project: Kafka Issue

Re: [VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

+1. (binding) On Tue, Jun 26, 2018, 2:34 AM Damian Guy wrote: > Thanks Bill! +1 > > On Mon, 25 Jun 2018 at 18:57 Ted Yu wrote: > > > +1 > > > > On Mon, Jun 25, 2018 at 9:45 AM, Guozhang Wang > wrote: > > > > > +1. > > > > > > On Mon, Jun 25, 2018 at 8:12 AM, Matthias J. Sax < >