Re: [DISCUSS] KIP-637 Include min.insync.replicas in MetadataResponse to make Producer smarter in partitioning events

Thanks for the comment. Yes, it's indeed a downside that this will increase the size of the metadata response, I was thinking if it's worth providing this information to the Producer conditionally, e.g. add a config to Producer to allow people choose whether to include this information in the

[jira] [Resolved] (KAFKA-10221) Backport fix for KAFKA-9603 to 2.5

[ https://issues.apache.org/jira/browse/KAFKA-10221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-10221. -- Resolution: Fixed > Backport fix for KAFKA-9603 to 2.5 > ---

[jira] [Created] (KAFKA-10246) AbstractProcessorContext topic() throws NullPointerException when modifying a state store within the DSL from a punctuator

Peter Pringle created KAFKA-10246: - Summary: AbstractProcessorContext topic() throws NullPointerException when modifying a state store within the DSL from a punctuator Key: KAFKA-10246 URL:

Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

Hi Badai, Thanks for picking this up. I've reviewed the KIP document and the threads you linked. I think we may want to make more improvements in the future to the printing of headers in particular, but this KIP seems like a clear benefit already. I think we can take it incrementally. I'm +1

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

HI Colin. Thanks for the KIP. Here is some feedback and various questions. "*Controller processes will listen on a separate port from brokers. This will be true even when the broker and controller are co-located in the same JVM*". I assume it is possible that the port numbers could be the same

Build failed in Jenkins: kafka-trunk-jdk11 #1626

See Changes: [github] KAFKA-10191 fix flaky StreamsOptimizedTest (#8913) -- [...truncated 6.38 MB...] org.apache.kafka.streams.test.OutputVerifierTest >

Build failed in Jenkins: kafka-trunk-jdk14 #275

See Changes: [github] KAFKA-10191 fix flaky StreamsOptimizedTest (#8913) -- [...truncated 6.38 MB...] org.apache.kafka.streams.TopologyTestDriverTest >

[DISCUSS] KIP-631: The Quorum-based Kafka Controller

Hi all, I posted a KIP about how the quorum-based controller envisioned in KIP-500 will work. Please take a look here: https://cwiki.apache.org/confluence/x/4RV4CQ best, Colin

Jenkins build is back to normal : kafka-trunk-jdk8 #4700

See

Re: [DISCUSS] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

Sure, I would appreciate help from Piotr creating an example. On Tue, Jul 7, 2020 at 12:03 PM Boyang Chen wrote: > Hey John, > > since ProcessorContext is a public API, I couldn't be sure that people > won't try to extend it. Without a default implementation, user code > compilation will break.

Jenkins build is back to normal : kafka-trunk-jdk14 #274

See

Re: [DISCUSS] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

Hey John, since ProcessorContext is a public API, I couldn't be sure that people won't try to extend it. Without a default implementation, user code compilation will break. William and Piotr, it seems that we haven't added any example usage of the new API, could we try to address that? It should

[VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

Hi everyone, I'd like to start a vote for adding two new time API's to ProcessorContext. Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext Thanks

[jira] [Resolved] (KAFKA-10222) Incorrect methods show up in 0.10 Kafka Streams docs

[ https://issues.apache.org/jira/browse/KAFKA-10222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-10222. - Resolution: Fixed > Incorrect methods show up in 0.10 Kafka Streams docs >

[jira] [Resolved] (KAFKA-10191) fix flaky StreamsOptimizedTest

[ https://issues.apache.org/jira/browse/KAFKA-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman resolved KAFKA-10191. - Resolution: Fixed > fix flaky StreamsOptimizedTest >

[jira] [Created] (KAFKA-10245) Using vulnerable log4j version

Pavel Kuznetsov created KAFKA-10245: --- Summary: Using vulnerable log4j version Key: KAFKA-10245 URL: https://issues.apache.org/jira/browse/KAFKA-10245 Project: Kafka Issue Type: Bug

Build failed in Jenkins: kafka-trunk-jdk8 #4699

See Changes: [github] KAFKA-10243; ConcurrentModificationException while processing connection -- [...truncated 3.16 MB...]

[GitHub] [kafka-site] mjsax commented on pull request #271: MINOR: Fix table of contents in protocol page

mjsax commented on pull request #271: URL: https://github.com/apache/kafka-site/pull/271#issuecomment-655024302 Thanks @wkodate This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [kafka-site] mjsax merged pull request #271: MINOR: Fix table of contents in protocol page

mjsax merged pull request #271: URL: https://github.com/apache/kafka-site/pull/271 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [kafka-site] mjsax commented on pull request #272: KAFKA-10222:Incorrect methods show up in 0.10 Kafka Streams docs

mjsax commented on pull request #272: URL: https://github.com/apache/kafka-site/pull/272#issuecomment-655021969 We can still merge this PR, because otherwise the web-page would not be updated. This is an automated message

[GitHub] [kafka-site] mjsax merged pull request #272: KAFKA-10222:Incorrect methods show up in 0.10 Kafka Streams docs

mjsax merged pull request #272: URL: https://github.com/apache/kafka-site/pull/272 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [DISCUSSION] KIP-619: Add internal topic creation support

Hi Colin, Thanks for the comments. I’ve modified the KIP accordingly. > I think we need to understand which of these limitations we will carry > forward and which we will not. We also have the option of putting > limitations just on consumer offsets, but not on other internal topics. In

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi, Ying, Thanks for the update. It's good to see the progress on this. Please let us know when you are done updating the KIP wiki. Jun On Tue, Jul 7, 2020 at 10:13 AM Ying Zheng wrote: > Hi Jun, > > Satish and I have added more design details in the KIP, including how to > keep consistency

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi Jun, Satish and I have added more design details in the KIP, including how to keep consistency between replicas (especially when there is leadership changes / log truncations) and new metrics. We also made some other minor changes in the doc. We will finish the KIP changes in the next couple

Re: [VOTE] KIP-632: Add DirectoryConfigProvider

+1 (non-binding). Thanks for the KIP! On Tue, Jul 7, 2020 at 12:54 PM Tom Bentley wrote: > Hi, > > I'd like to start a vote on KIP-632, which is about making the config > provider mechanism more ergonomic on Kubernetes: > > >

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi, Satish, Harsha, Any new updates on the KIP? This feature is one of the most important and most requested features in Apache Kafka right now. It would be helpful if we can make sustained progress on this. Could you share how far along is the design/implementation right now? Is there anything

Re: [VOTE] KIP-620 Deprecate ConsumerConfig#addDeserializerToConfig(Properties, Deserializer, Deserializer) and ProducerConfig#addSerializerToConfig(Properties, Serializer, Serializer)

Ok, after a second thought, keeping a function which still has production reference is ok. We probably should not make it public in the first place, but this is not high priority either. On Tue, Jul 7, 2020 at 9:03 AM Chia-Ping Tsai wrote: > > do we just suggest they no longer have any

Re: [VOTE] KIP-620 Deprecate ConsumerConfig#addDeserializerToConfig(Properties, Deserializer, Deserializer) and ProducerConfig#addSerializerToConfig(Properties, Serializer, Serializer)

> do we just suggest they no longer have any production use case? yep > KafkaProducer internal only. Do we also want to deprecate this public API as > well? We have to make sure users' code can keep working beyond recompilation when migrating to "next" release. Hence, deprecation cycle is

[DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

hi all, I would like to start the discussion for KIP-641. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569 Many thanks, Chia-Ping

[jira] [Resolved] (KAFKA-10243) ConcurrentModificationException while processing connection setup timeouts

[ https://issues.apache.org/jira/browse/KAFKA-10243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-10243. Reviewer: Rajini Sivaram Resolution: Fixed > ConcurrentModificationException while

[jira] [Created] (KAFKA-10244) An new java interface to replace 'kafka.common.MessageReader'

Chia-Ping Tsai created KAFKA-10244: -- Summary: An new java interface to replace 'kafka.common.MessageReader' Key: KAFKA-10244 URL: https://issues.apache.org/jira/browse/KAFKA-10244 Project: Kafka

Re: [DISCUSS] KIP-363

Hi Tom, Thanks for this. I think the tough part is probably the few messages that are still using manual serialization, which can't be easily converted to using this. So we will probably have to upgrade them to using automatic generation, or accept a little inconsistency for a while until

Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

Hi all After resurrecting the discussion thread [1] for KIP-431 and have not received any further feedback for 2 weeks, I would like to resurrect the voting thread [2] for KIP-431. I have updated KIP-431 wiki page

Re: [VOTE] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

Thanks, Tom. I tried to think of a better way to do this, but I think you're right that we probably just need different methods. +1 (binding). best, Colin On Mon, Jul 6, 2020, at 01:14, Tom Bentley wrote: > Hi, > > I'd like to start a vote on KIP-621 which is about deprecating methods in >

Re: [DISCUSS] Include min.insync.replicas in MetadataResponse to make Producer smarter in partitioning events

Hi Arvin, Thanks for the KIP. Unfortunately, I don't think this makes sense since it would increase the amount of data we send back in the metadata response, which is pretty bad for scalability. In general we probably want to avoid the case where we don't have the appropriate number of

Re: [DISCUSS] KIP-638: Deprecate DescribeLogDirsResponse.[LogDirInfo, ReplicaInfo]

Hi Dongjin, It looks like this KIP is addressing the same issue as KIP-621: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158862109 On Tue, Jul 7, 2020 at 2:29 PM Dongjin Lee wrote: > > Hi devs, > > I hope to start the discussion of KIP-638, which aims to fix a glitch in >

Re: Untrimmed Index Files resulting in data loss

Hi John, Thanks for reporting the issue. Let's continue the discussion in the PR. Ismael On Mon, Jul 6, 2020 at 7:13 PM John Malizia wrote: > Hi there, about a week ago I submitted an issue report and an associated PR > > https://issues.apache.org/jira/browse/KAFKA-10207 >

[DISCUSS] KIP-638: Deprecate DescribeLogDirsResponse.[LogDirInfo, ReplicaInfo]

Hi devs, I hope to start the discussion of KIP-638, which aims to fix a glitch in Admin#describeLogDirs method. - KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866169 - Jira: https://issues.apache.org/jira/browse/KAFKA-8794 All kinds of feedback will be greatly

Re: Feedback: Print schemaId using bin/kafka-dump-log.sh

Hi Mohanraj While I see the usefulness of your suggestion, the main issue is that you're using the Confluent schema registry's conventions and hardwiring them into Kafka core. Given that Confluent's standards are not part of Kafka's official standards, I do not think you will get approval to

[VOTE] KIP-632: Add DirectoryConfigProvider

Hi, I'd like to start a vote on KIP-632, which is about making the config provider mechanism more ergonomic on Kubernetes: https://cwiki.apache.org/confluence/display/KAFKA/KIP-632%3A+Add+DirectoryConfigProvider Please take a look if you have time. Many thanks, Tom

Build failed in Jenkins: kafka-2.5-jdk8 #163

See Changes: [boyang] KAFKA-10239: Make GroupInstanceId ignorable in DescribeGroups (#8989) -- [...truncated 5.92 MB...] org.apache.kafka.streams.TopologyTestDriverTest >

[jira] [Created] (KAFKA-10243) ConcurrentModificationException while processing connection setup timeouts

Rajini Sivaram created KAFKA-10243: -- Summary: ConcurrentModificationException while processing connection setup timeouts Key: KAFKA-10243 URL: https://issues.apache.org/jira/browse/KAFKA-10243

NotEnoughReplicasException: The size of the current ISR Set(2) is insufficient to satisfy the min.isr requirement of 3

I had the following setup Brokers : 3 - all are up and running with min.insync.replicas=3. I created a topic with the following configuration bin\windows\kafka-topics --zookeeper 127.0.0.1:2181 --topic topic-ack-all --create --partitions 4 --replication-factor 3 I triggered the producer with

Jenkins build is back to normal : kafka-trunk-jdk11 #1624

See

Re: KIP-560 Discuss

No worry! Could you address Matthias' comments in this mailing thread? Seems we still have some gaps. On Mon, Jul 6, 2020 at 7:14 AM Sang wn Lee wrote: > I'm sorry. > I just modified the KIP! > > On 2020/03/07 20:09:47, "Matthias J. Sax" wrote: > > -BEGIN PGP SIGNED MESSAGE- > > Hash:

Re: [VOTE] KIP-620 Deprecate ConsumerConfig#addDeserializerToConfig(Properties, Deserializer, Deserializer) and ProducerConfig#addSerializerToConfig(Properties, Serializer, Serializer)

Thanks for the KIP. One question I have is that when we refer to the two methods as useless, do we just suggest they no longer have any production use case? If this is the case, Producer#addSerializerToConfig(Map configs, keySerializer, valueSerializer) is only used in KafkaProducer internal only.

Re: [DISCUSS] KIP-616: Rename implicit Serdes instances in kafka-streams-scala

So, what's next? It's my first KIP and I'm not familiar with all processes. -Yuriy On Mon, Jul 6, 2020 at 1:32 AM John Roesler wrote: > Hi Yuriy, > > Thanks for the update! It looks good to me. > > Thanks, > John > > On Sun, Jul 5, 2020, at 03:27, Yuriy Badalyantc wrote: > > Hi John. > > > > I

[DISCUSS] KIP-637 Include min.insync.replicas in MetadataResponse to make Producer smarter in partitioning events

Updated the subject to add KIP number Arvin Zheng 于2020年7月6日周一 上午10:38写道： > Hi everyone, > > I would like to start the discussion for KIP-637 > >

Build failed in Jenkins: kafka-trunk-jdk14 #273

See Changes: [github] KAFKA-10166: checkpoint recycled standbys and ignore empty rocksdb base [github] MINOR: document timestamped state stores (#8920) [github] KAFKA-9930: Adjust ReplicaFetcherThread logging

Re: [VOTE] KIP-620 Deprecate ConsumerConfig#addDeserializerToConfig(Properties, Deserializer, Deserializer) and ProducerConfig#addSerializerToConfig(Properties, Serializer, Serializer)

+1 (binding) Thanks for the KIP. On Wed, Jun 10, 2020 at 11:43 PM Matthias J. Sax wrote: > Yes, it does. > > I guess many people are busy wrapping up 2.6 release. Today is code freeze. > > > -Matthias > > > On 6/10/20 12:11 AM, Chia-Ping Tsai wrote: > > hi Matthias, > > > > Does this

Build failed in Jenkins: kafka-trunk-jdk8 #4698

See Changes: [github] MINOR: document timestamped state stores (#8920) [github] KAFKA-9930: Adjust ReplicaFetcherThread logging when processing [github] KAFKA-10239: Make GroupInstanceId ignorable in