Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

On Thu, Sep 24, 2020, at 16:24, Jun Rao wrote: > Hi, Colin, > > Thanks for the reply and the updated KIP. A few more comments below. > Hi Jun, > > 53. It seems that you already incorporated the changes in KIP-516. With > topic ids, we don't need to wait for the topic's data to be deleted

Jenkins build is back to normal : Kafka » kafka-trunk-jdk8 #86

See

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

Hey Jason and Jun, thanks for the reply. Actually after some offline discussion, we have seen hassles around upgrading and downgrading RPCs during redirection, which is an error-prone approach to coordinate all parties to choose the correct version to handle. Alternatively, we propose to bring

[jira] [Created] (KAFKA-10527) Voters should always initialize as followers

Jason Gustafson created KAFKA-10527: --- Summary: Voters should always initialize as followers Key: KAFKA-10527 URL: https://issues.apache.org/jira/browse/KAFKA-10527 Project: Kafka Issue

[jira] [Created] (KAFKA-10526) Explore performance impact of leader fsync deferral

Jason Gustafson created KAFKA-10526: --- Summary: Explore performance impact of leader fsync deferral Key: KAFKA-10526 URL: https://issues.apache.org/jira/browse/KAFKA-10526 Project: Kafka

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

Hi, Jason, Yes, the most important thing is to be able to avoid two rolling restarts in the future. If we have a path to achieve that down the road, the changes here are fine. Thanks, Jun On Thu, Sep 24, 2020 at 3:20 PM Jason Gustafson wrote: > > One of the goals of KIP-584 (feature

Re: [VOTE] KIP-671: Add method to Shutdown entire Streams Application

Thanks for the KIP, Walker! I’m +1 (binding) -John On Mon, Sep 21, 2020, at 17:04, Guozhang Wang wrote: > Thanks for finalizing the KIP. +1 (binding) > > > Guozhang > > On Mon, Sep 21, 2020 at 1:38 PM Walker Carlson > wrote: > > > Hello all, > > > > I would like to start a thread to vote

[jira] [Resolved] (KAFKA-8836) Add inter-broker protocol to alter ISR

[ https://issues.apache.org/jira/browse/KAFKA-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-8836. Fix Version/s: 2.7.0 Assignee: David Arthur (was: Jason Gustafson)

Re: [DISCUSS] KIP-516: Topic Identifiers

> 2. Part of the usage of the file is to have persistent storage of the topic ID and use it to compare with the ID supplied in the LeaderAndIsr Request. There is some discussion in the KIP about changes to the directory structure, but I believe directory changes were considered to be out of scope

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

Hi, Colin, Thanks for the reply and the updated KIP. A few more comments below. 53. It seems that you already incorporated the changes in KIP-516. With topic ids, we don't need to wait for the topic's data to be deleted before removing the topic metadata. If the topic is recreated, we can still

[DISCUSS] KIP-673: Emit JSONs with new auto-generated schema

Hi all, I'd like to discuss KIP-673: https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema This is a proposal to change the format of request and response traces to JSON, which would be easier to load and parse, because the current format is only

Re: [DISCUSS] KIP-516: Topic Identifiers

Thanks Justine. Responses below: > 1. Yes, the directory will still be based on the topic names. LeaderAndIsrRequest is one of the few requests that will still contain the topic name. So I think we have this covered. Sorry for confusion. Ah, you're right. My eyes passed right over the field. >

Re: [DISCUSS] KIP-516: Topic Identifiers

Hi Jason, Thanks for your comments. 1. Yes, the directory will still be based on the topic names. LeaderAndIsrRequest is one of the few requests that will still contain the topic name. So I think we have this covered. Sorry for confusion. 2. Part of the usage of the file is to have persistent

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

> One of the goals of KIP-584 (feature versioning) is that we can get rid of IBP in the future. So does this change prevent us from removing IBP in the future? That is a good question. I think the problem here is that request forwarding puts an expectation on api version support which covers more

[jira] [Created] (KAFKA-10525) Emit JSONs with new auto-generated schema

Anastasia Vela created KAFKA-10525: -- Summary: Emit JSONs with new auto-generated schema Key: KAFKA-10525 URL: https://issues.apache.org/jira/browse/KAFKA-10525 Project: Kafka Issue Type:

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

Hi, Boyang, One of the goals of KIP-584 (feature versioning) is that we can get rid of IBP in the future. So does this change prevent us from removing IBP in the future? Thanks, Jun On Thu, Sep 24, 2020 at 12:46 PM Jason Gustafson wrote: > Hey Boyang, > > Thanks for the update. This seems

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Thanks Jose. Makes sense overall. A few specific responses below: > Generally the number of snapshots on disk will be one. I suspect that users will want some control over this. We can add a configuration option that doesn't delete, or advances the log begin offset past, the N latest snapshots.

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

Thanks for the feedback Jason. I have made the following changes to the KIP: 1. Better explanation of how followers will manage snapshots and the replicated log. This includes the necessary changes when granting or requesting votes. 2. How the Snapshot's epoch will be used for the LastFetchEpoch

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

On Mon, Sep 21, 2020, at 18:13, Jun Rao wrote: > Hi, Colin, > > Sorry for the late reply. A few more comments below. > Hi Jun, Thanks for taking another look. > > 50. Configurations > 50.1 controller.listeners: It seems that a controller just needs one > listener. Why do we need to have a

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

Hey Boyang, Thanks for the update. This seems like the best thing we can do. The alternative would be to always ensure that the forwarded APIs are safe for conversion between versions, but that would restrict the flexibility that the versioning is providing. It would also be a large effort to

Re: [DISCUSS] KIP-516: Topic Identifiers

Hi Justine, Thanks for picking up this work. I have a few questions/comments: 1. It sounds like the directory structure is still going to be based on topic names. Do I have that right? One complication is that the LeaderAndIsr request does not include the topic name any longer. This means that a

Re: [DISCUSS] KIP-516: Topic Identifiers

Hi all, Thanks for the discussion. I'm glad we are able to get our best ideas out there. David Jacot 1. I apologize for the incorrect information. I have fixed the KIP. 2. Yes. The difference between full and incremental is that on full we check for the two types of stale request—topics on the

Re: [DISCUSS] KIP-516: Topic Identifiers

Hi, Justine, Thanks for the updated KIP. A few more comments below. 30. {"name": "id", "type": "string", "doc": "version id"}}: The doc should say UUID. 31. LeaderAndIsrResponse v5 and StopReplicaResponse v4 : It seems there is no need to add topic_id at partitions level. 32. Regarding

Re: [DISCUSS] Apache Kafka 2.7.0 release

Hi All, Just a reminder that the KIP freeze is next Wednesday, September 30th. Any KIP aiming to go in the 2.7.0 release needs to be accepted by this date. Thanks, BIll On Tue, Sep 22, 2020 at 12:11 PM Bill Bejeck wrote: > Boyan, > > Done. Thanks for the heads up. > > -Bill > > On Mon, Sep

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

Interesting proposal. However, I am not totally convinced, because I see a fundamental difference between "data" and "metadata". Topic/partition/offset are "metadata" in the strong sense and they are immutable. On the other hand there is "primary" data like key and value, as well as "secondary"

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #85

See Changes: [github] KAFKA-9627: Replace ListOffset request/response with automated protocol (#8295) [github] MINOR: clarify variables for skipping idempotent source updates (#9316)

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

Hey there, we spotted a necessary case to handle the redirect request versioning, and proposed the following changes: 1. For redirection RPCs (AlterConfig, Acl, Token etc), the corresponding allowed versions in the ApiVersionResponse will be affected by the entire cluster's versioning, not just

[jira] [Created] (KAFKA-10524) Manpage help for kafka-configs.sh does not fully explain broker-loggers

Daniel Laing created KAFKA-10524: Summary: Manpage help for kafka-configs.sh does not fully explain broker-loggers Key: KAFKA-10524 URL: https://issues.apache.org/jira/browse/KAFKA-10524 Project:

[jira] [Created] (KAFKA-10523) Allow to provide producer ID

Luigi Berrettini created KAFKA-10523: Summary: Allow to provide producer ID Key: KAFKA-10523 URL: https://issues.apache.org/jira/browse/KAFKA-10523 Project: Kafka Issue Type: New Feature

[jira] [Created] (KAFKA-10522) Duplicate detection and max.in.flight.requests.per.connection details

Luigi Berrettini created KAFKA-10522: Summary: Duplicate detection and max.in.flight.requests.per.connection details Key: KAFKA-10522 URL: https://issues.apache.org/jira/browse/KAFKA-10522

Why should SASL principal be unchanged upon reauth

Hi. I was looking through Kafka code and found that SASL KafkaPrincipals are not supposed to change upon reauthentication, and if they do, the broker will kill the TCP connection. What is the reasoning behind this limitation? Thanks.

[VOTE] KIP-661: Expose task configurations in Connect REST API

Hi, I'd like to start a vote on KIP-661: https://cwiki.apache.org/confluence/display/KAFKA/KIP-661%3A+Expose+task+configurations+in+Connect+REST+API Thanks

Re: [DISCUSS] KIP-671: Shutdown Streams Application when appropriate exception is thrown

Hello all, Thanks for bringing this up, Bruno. It’s a really good point that a disconnected node would miss the signal and then resurrect a single-node “zombie cluster” when it reconnects. Offhand, I can’t think of a simple and reliable way to distinguish this case from one in which an

Jenkins build is back to normal : Kafka » kafka-trunk-jdk11 #85

See

Jenkins build is back to normal : Kafka » kafka-trunk-jdk15 #85

See

[jira] [Resolved] (KAFKA-9629) Replace Fetch request/response with automated protocol

[ https://issues.apache.org/jira/browse/KAFKA-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-9629. --- Fix Version/s: 2.7.0 Resolution: Fixed > Replace Fetch request/response with

[jira] [Created] (KAFKA-10521) Remove ZK watch for completing partition reassignment

David Arthur created KAFKA-10521: Summary: Remove ZK watch for completing partition reassignment Key: KAFKA-10521 URL: https://issues.apache.org/jira/browse/KAFKA-10521 Project: Kafka Issue

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #84

See Changes: [github] MINOR: Use JUnit 5 in raft module (#9331) [github] MINOR: Remove `zipWithIndex` to avoid tuple allocation in hot path in `LogValidator` (#9206)

[jira] [Created] (KAFKA-10520) InitProducerId may be blocked if least loaded node is not ready to send

Rajini Sivaram created KAFKA-10520: -- Summary: InitProducerId may be blocked if least loaded node is not ready to send Key: KAFKA-10520 URL: https://issues.apache.org/jira/browse/KAFKA-10520 Project:

Re: [DISCUSS] KIP-516: Topic Identifiers

Also, can we provide more details on how the Partition Metadata file will be used? Ismael On Thu, Sep 24, 2020 at 3:01 AM Ismael Juma wrote: > Hi Justine, > > I think we need to update the "Rejected Alternatives" section to take into > account that the proposal now removes the topic name from

Re: [DISCUSS] KIP-516: Topic Identifiers

Hi Justine, I think we need to update the "Rejected Alternatives" section to take into account that the proposal now removes the topic name from the fetch request. Also, if we are removing it from the Fetch request, does it make sense not to remove it from similar requests like ListOffsetRequest?

Re: [DISCUSS] KIP-516: Topic Identifiers

Hi Justine, Thanks for the KIP. I finally had time to read it :). It is a great improvement. I have a few comments/questions: 1. It seems that the schema of the StopReplicaRequest is slightly outdated. We did some changes as part of KIP-570. V3 is already organized by topics. 2. I just want to