[jira] [Created] (KAFKA-10353) Trogdor - Fix RoundTripWorker to not fail when the topic it's trying to create already exists

Stanislav Kozlovski created KAFKA-10353: --- Summary: Trogdor - Fix RoundTripWorker to not fail when the topic it's trying to create already exists Key: KAFKA-10353 URL: https://issues.apache.org/jira/browse/KA

Re: [VOTE] KIP-648: Renaming getter method for Interactive Queries

Thank you for your votes everyone! The KIP has been accepted with 3 binding votes, 3 non-binding votes and no -1 votes. * 3 binding +1 : Matthias J. Sax, Boyang Chen, John Roesler * non-binding+1 (Navinder Brar, Jorge Esteban, Bruno Cadonna) - John From:

[jira] [Created] (KAFKA-10352) Error while reading checkpoint file /tmp/kafka-logs/cleaner-offset-checkpoint (kafka.server.LogDirFailureChannel)

Seongbae Chang created KAFKA-10352: -- Summary: Error while reading checkpoint file /tmp/kafka-logs/cleaner-offset-checkpoint (kafka.server.LogDirFailureChannel) Key: KAFKA-10352 URL: https://issues.apache.org/jira

Re: [VOTE] KIP-595: A Raft Protocol for the Metadata Quorum

Hi Colin, Thanks for the responses. > I have a few lingering questions. I still don't like the fact that the leader epoch / fetch epoch is 31 bits. What happens when this rolls over? Can we just make this 63 bits now so that we never have to worry about it again? ZK has some awful bugs surroun

[GitHub] [kafka-site] rhauch commented on a change in pull request #260: Add 2.5.0 release to website

rhauch commented on a change in pull request #260: URL: https://github.com/apache/kafka-site/pull/260#discussion_r464774655 ## File path: downloads.html ## @@ -5,12 +5,52 @@ Download -2.4.1 is the latest release. The current stable version is 2.4.1. +2.5

[GitHub] [kafka-site] rhauch opened a new pull request #277: 2.6.0 release

rhauch opened a new pull request #277: URL: https://github.com/apache/kafka-site/pull/277 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

Hi, The LeaseStartTimeMs is expected to be the broker's 'System.currentTimeMillis()' at the point of the request. The active controller will add its lease period to this in order to compute the LeaseEndTimeMs. I think this is a bit confusing. Monotonic clock on the active controller shoul

[jira] [Created] (KAFKA-10345) Redirect AlterPartitionReassignment to the controller

Boyang Chen created KAFKA-10345: --- Summary: Redirect AlterPartitionReassignment to the controller Key: KAFKA-10345 URL: https://issues.apache.org/jira/browse/KAFKA-10345 Project: Kafka Issue Ty

[jira] [Created] (KAFKA-10344) Redirect Create/Renew/ExpireDelegationToken to the controller

Boyang Chen created KAFKA-10344: --- Summary: Redirect Create/Renew/ExpireDelegationToken to the controller Key: KAFKA-10344 URL: https://issues.apache.org/jira/browse/KAFKA-10344 Project: Kafka

[jira] [Created] (KAFKA-10343) Redirect AlterClientQuotas to the controller

Boyang Chen created KAFKA-10343: --- Summary: Redirect AlterClientQuotas to the controller Key: KAFKA-10343 URL: https://issues.apache.org/jira/browse/KAFKA-10343 Project: Kafka Issue Type: Sub-ta

[jira] [Created] (KAFKA-10342) Redirect CreateAcls/DeleteAcls to the controller

Boyang Chen created KAFKA-10342: --- Summary: Redirect CreateAcls/DeleteAcls to the controller Key: KAFKA-10342 URL: https://issues.apache.org/jira/browse/KAFKA-10342 Project: Kafka Issue Type: Su

[jira] [Created] (KAFKA-10351) Add missing tests for IOExceptions for GlobalStateManagerImpl

Matthias J. Sax created KAFKA-10351: --- Summary: Add missing tests for IOExceptions for GlobalStateManagerImpl Key: KAFKA-10351 URL: https://issues.apache.org/jira/browse/KAFKA-10351 Project: Kafka

[jira] [Created] (KAFKA-10350) Add redirect request monitoring metrics

Boyang Chen created KAFKA-10350: --- Summary: Add redirect request monitoring metrics Key: KAFKA-10350 URL: https://issues.apache.org/jira/browse/KAFKA-10350 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-10349) Deprecate client side controller access

Boyang Chen created KAFKA-10349: --- Summary: Deprecate client side controller access Key: KAFKA-10349 URL: https://issues.apache.org/jira/browse/KAFKA-10349 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-10348) Redirect UpdateFeatures to the controller

Boyang Chen created KAFKA-10348: --- Summary: Redirect UpdateFeatures to the controller Key: KAFKA-10348 URL: https://issues.apache.org/jira/browse/KAFKA-10348 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-10347) Redirect Create/DeleteTopics to the controller

Boyang Chen created KAFKA-10347: --- Summary: Redirect Create/DeleteTopics to the controller Key: KAFKA-10347 URL: https://issues.apache.org/jira/browse/KAFKA-10347 Project: Kafka Issue Type: Sub-

[jira] [Created] (KAFKA-10346) Redirect CreatePartition to the controller

Boyang Chen created KAFKA-10346: --- Summary: Redirect CreatePartition to the controller Key: KAFKA-10346 URL: https://issues.apache.org/jira/browse/KAFKA-10346 Project: Kafka Issue Type: Sub-task

Re: [VOTE] KIP-554: Add Broker-side SCRAM Config API

Thanks, Ron. Sounds good. best, Colin On Tue, Jul 28, 2020, at 15:26, Ron Dagostino wrote: > Hi everyone. I just wanted to notify that while implementing this I > discovered that we had declared the ScramMechanism enum to have the values > HMAC_SHA_{256,512} instead of SCRAM_SHA_{256,512}. I b

Re: [DISCUSS] KIP-406: GlobalStreamThread should honor custom reset policy

Navinder, thanks for updating the KIP. I think the motivation section is not totally accurate (what is not your fault though, as the history of how we handle this case is intertwined...) For example, "auto.offset.reset" is hard-coded for the global consumer to "none" and using "global.consumer.aut

Re: [VOTE] KIP-595: A Raft Protocol for the Metadata Quorum

+1. Thanks for the detailed KIP! On Mon, Aug 3, 2020 at 11:03 AM Jason Gustafson wrote: > > Hi All, I'd like to start a vote on this proposal: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum. > The discussion has been active for a bit more t

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

On Mon, Aug 3, 2020, at 15:51, Jose Garcia Sancio wrote: > Thanks for the KIP Colin, > > Here is a partial review: > > > 1. Even when a broker and a controller are co-located in the same JVM, they > > must > > have different node IDs > > Why? What problem are you trying to solve? > Hi Jose,

Re: [VOTE] KIP-595: A Raft Protocol for the Metadata Quorum

Hi Jason, The KIP looks great. Thanks for all the work you've put into this. I have a few lingering questions. I still don't like the fact that the leader epoch / fetch epoch is 31 bits. What happens when this rolls over? Can we just make this 63 bits now so that we never have to worry abou

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

Thanks for the KIP Colin, Here is a partial review: 1. > Even when a broker and a controller are co-located in the same JVM, they must > have different node IDs Why? What problem are you trying to solve? 2. > Node IDs must be set in the configuration file for brokers and controllers. I unders

Re: [VOTE] KIP-648: Renaming getter method for Interactive Queries

Thanks for the KIP, +1 (binding) from me! On Sun, Aug 2, 2020 at 4:49 PM Matthias J. Sax wrote: > Thanks for the KIP, John! > > +1 (binding) > > > > On 8/2/20 11:04 AM, John Thomas wrote: > > Hello , > > > > The vote has been open for >72 hours, yet to get some binding votes. > Can everyone plea

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

Hi David, Here are some additional thoughts... > 1. Once dynamic configs have been loaded and resolved, how can a client > know what values are selected? A copy of the original user-provided configs is kept by the client. Currently these are used to revert to the user-provided config if a dynam

[jira] [Created] (KAFKA-10341) Add version 2.6 to streams and systems tests

Randall Hauch created KAFKA-10341: - Summary: Add version 2.6 to streams and systems tests Key: KAFKA-10341 URL: https://issues.apache.org/jira/browse/KAFKA-10341 Project: Kafka Issue Type: Ta

Re: [DISCUSS] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

Interesting proposal. However, it raises the question how the runtime would pass in the `systemTime` parameter? To be accurate, we would need to call `Time.milliseconds()` each time before we call the timestamp extractor. This sound expensive and maybe the extractor does not even use this value.

[jira] [Resolved] (KAFKA-10321) shouldDieOnInvalidOffsetExceptionDuringStartup would block forever on JDK11

[ https://issues.apache.org/jira/browse/KAFKA-10321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-10321. - Fix Version/s: 2.7.0 Resolution: Fixed > shouldDieOnInvalidOffsetExceptionDuringS

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Hi Jose, Thanks for the proposal. I think there are three main motivations for snapshotting over the existing compaction semantics. First we are arguing that compaction is a poor semantic fit for how we want to model the metadata in the cluster. We are trying to view the changes in the cluster as

Build failed in Jenkins: kafka-trunk-jdk11 #1686

See Changes: [github] MINOR: Add comments to constrainedAssign and generalAssign method -- [...truncated 3.21 MB...] org.apache.kafka.streams.TopologyTestDriverTest > s

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

On Mon, Aug 3, 2020, at 01:48, Unmesh Joshi wrote: > Just a quick comment. The description of the heartbeat in this KIP is more > about maintaining a 'time bound lease' with broker details, for group > membership and failure detection of the brokers. That way they are more > than just heartbeats, a

Re: New Website Layout

Hello Ben, I took a look at the corresponding PR and that looks great to me. Just one minor comment otherwise I'm all in for the new UX! * The "see full list" at the homepage just has a drop-down list, maybe better to re-direct to "powered by" if we click it. Guozhang On Fri, Jun 26, 2020 at 5:

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

On Mon, Jul 27, 2020, at 09:20, Jun Rao wrote: > Hi, Colin, > > Thanks for the KIP. A few comments below. > Hi Jun, Thanks for the review. Sorry that it took me a while to respond. I wanted to carefully re-read the latest version of KIP-595 first as well as the other discussions. > > 10. S

[jira] [Created] (KAFKA-10340) Source connectors should report error when trying to producer records to non-existent topics instead of hanging forever

Arjun Satish created KAFKA-10340: Summary: Source connectors should report error when trying to producer records to non-existent topics instead of hanging forever Key: KAFKA-10340 URL: https://issues.apache.org/ji

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

Hi David, Thank you for the comments, they were helpful. Here are the changes I made to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=12&selectedPageVersions=11 > 1. Once dynamic configs have been loaded and resolved, how can

Re: [VOTE] KIP-595: A Raft Protocol for the Metadata Quorum

+1 On Mon, 3 Aug 2020 at 19:03, Jason Gustafson wrote: > Hi All, I'd like to start a vote on this proposal: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum > . > The discussion has been active for a bit more than 3 months and I think the >

[VOTE] KIP-595: A Raft Protocol for the Metadata Quorum

Hi All, I'd like to start a vote on this proposal: https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum. The discussion has been active for a bit more than 3 months and I think the main points have been addressed. We have also moved some of the pieces

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

Hey Ryan, thanks for the KIP. This will be a really useful feature. Few questions 1. Once dynamic configs have been loaded and resolved, how can a client know what values are selected? Will we log the actual resolved values once they are loaded? 2. Do we want to support the case that we load dyna

[jira] [Created] (KAFKA-10339) MirrorMaker2 Exactly-once Semantics

Ning Zhang created KAFKA-10339: -- Summary: MirrorMaker2 Exactly-once Semantics Key: KAFKA-10339 URL: https://issues.apache.org/jira/browse/KAFKA-10339 Project: Kafka Issue Type: New Feature

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

Hey Jun, > Thanks for the response. Should we make clusterId a nullable field consistently in all new requests? Yes, makes sense. I updated the proposal. -Jason On Fri, Jul 31, 2020 at 4:17 PM Jun Rao wrote: > Hi, Jason, > > Thanks for the response. Should we make clusterId a nullable field >

[RESULTS] [VOTE] Release Kafka version 2.6.0

This vote passes with four +1 votes (3 bindings) and no 0 or -1 votes. +1 votes PMC Members (in voting order): * Rajini Sivaram * Gwen Shapira * Ismael Juma Committers (in voting order): * Bill Bejeck * Randall Hauch Community: * No votes 0 votes * No votes -1 votes * No votes Vote thread: ht

Re: [VOTE] 2.6.0 RC2

+1 (non-binding) I'm closing the vote since this has met the release criteria. Randall On Mon, Aug 3, 2020 at 2:57 AM Ismael Juma wrote: > +1 (binding), verified signatures, ran the tests on the source archive with > Scala 2.13 and Java 14 and verified the quickstart with the source archive >

Jenkins build is back to normal : kafka-trunk-jdk8 #4759

See

Re: [DISCUSS] KIP-651 - Support PEM format for SSL certificates and private key

Yes please! I had to build custom tooling to automate generation and distribution of java truststore and keystore files at Wikimedia for Kafka brokers and clients. PEM is much more standard outside of the Java world and it would be simpler to not have to maintain custom tooling for keystores. -

[DISCUSS] KIP-651 - Support PEM format for SSL certificates and private key

Hi all, I have submitted KIP-651 to support PEM format for SSL key and trust stores: - https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key This enables better integration with other Kafka features like dynamic config update a

Re: [VOTE] 2.5.1 RC0

+1 (binding) - built from source with Scala 2.12 and Java 8 - ran quickstart - verified signatures Thanks John for running this release On Fri, Jul 31, 2020 at 6:40 PM Manikumar wrote: > > Hi John, > > +1 (binding) > > - Verified signatures, artifacts, Release notes > - Built from sources, ran

[jira] [Created] (KAFKA-10338) Support PEM format for SSL certificates and private key

Rajini Sivaram created KAFKA-10338: -- Summary: Support PEM format for SSL certificates and private key Key: KAFKA-10338 URL: https://issues.apache.org/jira/browse/KAFKA-10338 Project: Kafka I

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

We could combine the heartbeat with the fetch request. It would basically mean moving all the heartbeat fields into the fetch request. As the KIP says, this would be pretty messy. Another reason why it would be messy is because of the timing. Fetch requests can get delayed when they're

Re: [VOTE] 2.6.0 RC2

+1 (binding), verified signatures, ran the tests on the source archive with Scala 2.13 and Java 14 and verified the quickstart with the source archive and Scala 2.13 binary archive. Thanks, Ismael On Tue, Jul 28, 2020, 2:52 PM Randall Hauch wrote: > Hello Kafka users, developers and client-deve