[DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hi folks For https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer , It has not been updated for a long time. And I made some update, which has been pushed to https://github.com/apache/kafka/pull/8831 MetadataRequest has method Builder(Li

[jira] [Created] (KAFKA-10170) ReplicaManager should be responsible for checking delayed operations after appending to the log.

Chia-Ping Tsai created KAFKA-10170: -- Summary: ReplicaManager should be responsible for checking delayed operations after appending to the log. Key: KAFKA-10170 URL: https://issues.apache.org/jira/browse/KAFKA-101

Re: Permission to create KIP

Done. On 6/15/20 1:27 PM, Sam Pal wrote: > Hi, > > I'd like permission to create a KIP on the Apache Kafka Wiki. My WikiID is > spal. > > Best, > Sam > signature.asc Description: OpenPGP digital signature

Build failed in Jenkins: kafka-trunk-jdk11 #1570

See Changes: [github] HOTFIX: checkstyle error in ProcessorStateManager (#8874) -- [...truncated 1.89 MB...] kafka.utils.json.JsonValueTest > testDecodeDouble PASSED k

[VOTE] KIP-626: Rename StreamsConfig config variable name

Hi, I found a small inconsistency in our public API and propose a small KIP to fix it. As the change is trivial, I skip the discussion and call directly for a VOTE. https://cwiki.apache.org/confluence/display/KAFKA/KIP-626%3A+Rename+StreamsConfig+config+variable+name -Matthias signature.asc

[jira] [Created] (KAFKA-10169) KafkaException: Failing batch since transaction was aborted

Sophie Blee-Goldman created KAFKA-10169: --- Summary: KafkaException: Failing batch since transaction was aborted Key: KAFKA-10169 URL: https://issues.apache.org/jira/browse/KAFKA-10169 Project: Ka

[jira] [Created] (KAFKA-10168) Rename public StreamsConfig variable

Matthias J. Sax created KAFKA-10168: --- Summary: Rename public StreamsConfig variable Key: KAFKA-10168 URL: https://issues.apache.org/jira/browse/KAFKA-10168 Project: Kafka Issue Type: Improv

[jira] [Created] (KAFKA-10167) Streams EOS-Beta should not try to get end-offsets as read-committed

Guozhang Wang created KAFKA-10167: - Summary: Streams EOS-Beta should not try to get end-offsets as read-committed Key: KAFKA-10167 URL: https://issues.apache.org/jira/browse/KAFKA-10167 Project: Kafka

Re: [DISCUSS] KIP-617: Allow Kafka Streams State Stores to be iterated backwards

Hi everyone, sorry for the late reply. Thanks Matthias for your feedback. I think it makes sense to reconsider the current design based on your input. After digging deeper into the current implementation, I'd like to bring my current understanding to be double-checked as it might be redefining th

Re: First time patch submitter advice

Great, thanks Luke. I’ve undone the patch and added that comment. Cheers, Michael > On 15 Jun 2020, at 6:07 pm, Luke Chen wrote: > > Hi Michael, > The failed unit test has already handled here: > https://issues.apache.org/jira/browse/KAFKA-10155 > https://issues.apache.org/jira/browse/KAFKA-101

Build failed in Jenkins: kafka-trunk-jdk8 #4644

See Changes: [github] HOTFIX: checkstyle error in ProcessorStateManager (#8874) -- [...truncated 3.15 MB...] org.apache.kafka.streams.TopologyTestDriverTest > shouldPro

Build failed in Jenkins: kafka-2.6-jdk8 #45

See Changes: [wangguoz] HOTFIX: checkstyle error in ProcessorStateManager (#8874) -- [...truncated 3.14 MB...] org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shou

Build failed in Jenkins: kafka-trunk-jdk14 #220

See Changes: [github] MINOR: Fix log message for transition from standby to active (#8872) [github] HOTFIX: checkstyle error in ProcessorStateManager (#8874) -- [...trun

Re: Broker side round robin on topic partitions when receiving messages

I understand your point of view... My requirement is *exact* balancing - parts of our current flow have a consumption processing of around 5 minutes... (and this is an important/expensive part - because it's CPU and memory intensive and we'd like to avoid queueing) so we need EQUAL load balancing -

Permission to create KIP

Hi, I'd like permission to create a KIP on the Apache Kafka Wiki. My WikiID is spal. Best, Sam

Build failed in Jenkins: kafka-2.6-jdk8 #44

See Changes: [wangguoz] MINOR: Avoid WARN log message when re-init from checkpoint skipped [wangguoz] MINOR: Fix log message for transition from standby to active (#8872) --

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

Thanks for more feedback Colin! I have addressed them in the KIP. Boyang On Mon, Jun 15, 2020 at 11:29 AM Colin McCabe wrote: > On Fri, Jun 12, 2020, at 15:30, Boyang Chen wrote: > > Thanks Colin for the suggestions! > > > > On Fri, Jun 12, 2020 at 2:40 PM Colin McCabe wrote: > > > > > Hi Boya

Build failed in Jenkins: kafka-trunk-jdk11 #1569

See Changes: [github] MINOR: Avoid WARN log message when re-init from checkpoint skipped [github] MINOR: Fix log message for transition from standby to active (#8872) -

Build failed in Jenkins: kafka-trunk-jdk14 #219

See Changes: [github] MINOR: Avoid WARN log message when re-init from checkpoint skipped -- [...truncated 3.17 MB...] org.apache.kafka.streams.test.ConsumerRecordFactory

Build failed in Jenkins: kafka-trunk-jdk8 #4643

See Changes: [github] MINOR: Avoid WARN log message when re-init from checkpoint skipped [github] MINOR: Fix log message for transition from standby to active (#8872) --

Re: Broker side round robin on topic partitions when receiving messages

This is a bit frustrating since you keep saying that the load is not balanced, but the load actually is balanced, it's just balanced in an approximate fashion. If you need exact balancing (for example, because you're creating a job scheduler or something), then you need to use a different strat

[jira] [Created] (KAFKA-10166) Excessive TaskCorruptedException seen in soak

Sophie Blee-Goldman created KAFKA-10166: --- Summary: Excessive TaskCorruptedException seen in soak Key: KAFKA-10166 URL: https://issues.apache.org/jira/browse/KAFKA-10166 Project: Kafka I

Re: Broker side round robin on topic partitions when receiving messages

Hi Collin, One producer shouldn't need to know about the other to distribute the load equally, but what Kafka has now is roughly equal... If you have a single producer RounRobinPartitioner works fine, if you have 10 producers you can have 7/8 messages in one partition while another partition has n

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

On Fri, Jun 12, 2020, at 15:30, Boyang Chen wrote: > Thanks Colin for the suggestions! > > On Fri, Jun 12, 2020 at 2:40 PM Colin McCabe wrote: > > > Hi Boyang, > > > > Thanks for the KIP! I think it's getting close. > > > > > For older requests that need redirection, forwarding > > > broker w

Re: [DISCUSSION] KIP-619: Add internal topic creation support

Hi Cheng, The link from the main KIP page is an "edit link" meaning that it drops you into the editor for the wiki page. I think the link you meant to use is a "view link" that will just take you to view the page. In general I'm not sure what I'm supposed to take away from the large UML diagr

[jira] [Created] (KAFKA-10165) Percentiles metric leaking memory

Sophie Blee-Goldman created KAFKA-10165: --- Summary: Percentiles metric leaking memory Key: KAFKA-10165 URL: https://issues.apache.org/jira/browse/KAFKA-10165 Project: Kafka Issue Type: B

Re: Broker side round robin on topic partitions when receiving messages

Hi Vinicius, It's actually not necessary for one producer to know about the others to get an even distribution across partitions, right? All that's really required is that all producers produce a roughly equal amount of data to each partition, which is what RoundRobinPartitioner is designed to

Re: [DISCUSS]: KIP-625: Richer encodings for integral-typed protocol fields.

Hi Tom, It's an interesting idea. Obviously protocol buffers does this for all numeric fields. I have to admit I have some mixed feelings, since this is another thing that makes encoding more complex. And it's not a clear win in all cases, although it is in some. I assume that the performan

[DISCUSS]: KIP-625: Richer encodings for integral-typed protocol fields.

Hi, I'd like to start discussion on KIP-625: Richer encodings for integral-typed protocol fields. https://cwiki.apache.org/confluence/display/KAFKA/KIP-625%3A+Richer+encodings+for+integral-typed+protocol+fields It's about allowing regular/required fields of protocol messages to use variable leng

Re: First time patch submitter advice

Hi Michael, The failed unit test has already handled here: https://issues.apache.org/jira/browse/KAFKA-10155 https://issues.apache.org/jira/browse/KAFKA-10147 So, maybe you can ignore the test errors and mention the issue number in PR. Thanks. Luke On Mon, Jun 15, 2020 at 3:23 PM Michael Carter

[jira] [Created] (KAFKA-10164) Implement Admin side changes

David Jacot created KAFKA-10164: --- Summary: Implement Admin side changes Key: KAFKA-10164 URL: https://issues.apache.org/jira/browse/KAFKA-10164 Project: Kafka Issue Type: Sub-task R

[jira] [Created] (KAFKA-10162) Update Rate implementation to cope with spiky workload

David Jacot created KAFKA-10162: --- Summary: Update Rate implementation to cope with spiky workload Key: KAFKA-10162 URL: https://issues.apache.org/jira/browse/KAFKA-10162 Project: Kafka Issue Ty

[jira] [Created] (KAFKA-10163) Implement Broker side changes

David Jacot created KAFKA-10163: --- Summary: Implement Broker side changes Key: KAFKA-10163 URL: https://issues.apache.org/jira/browse/KAFKA-10163 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-10161) Update Documentation

David Jacot created KAFKA-10161: --- Summary: Update Documentation Key: KAFKA-10161 URL: https://issues.apache.org/jira/browse/KAFKA-10161 Project: Kafka Issue Type: Sub-task Reporter:

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

Hi all, The vote has passed with 5 binding votes (Gwen, Rajini, Mickael, Jun and Colin) and 2 non-binding votes (Tom, Anna). Thank you all for the fruitful discussion! I'd like to particularly thank Anna who has heavily contributed to the design of this KIP. Regards, David On Fri, Jun 12, 2020

Re: First time patch submitter advice

Thanks for the response Gwen, that clarifies things for me. Regarding the unit test (ReassignPartitionsUnitTest. testModifyBrokerThrottles), it appears to fail quite reliably on trunk as well (at least on my machine). It looks to me like a new override to MockAdminClient.describeConfigs(Collec

[jira] [Created] (KAFKA-10160) Kafka MM2 consumer configuration

Pavol Ipoth created KAFKA-10160: --- Summary: Kafka MM2 consumer configuration Key: KAFKA-10160 URL: https://issues.apache.org/jira/browse/KAFKA-10160 Project: Kafka Issue Type: Bug Affects Ve