Re: Stream config in StreamPartitionAssignor

Hey Guozhang, thanks for the reply. Actually I'm not trying to pass config into StreamPartitionAssignor. My debugging was about some other configs that wasn't transferred properly (completely irrelevant), but the effort was delayed because the log would print "StreamsConfig values" for

[jira] [Created] (KAFKA-7144) Kafka Streams doesn't properly balance partition assignment

James Cheng created KAFKA-7144: -- Summary: Kafka Streams doesn't properly balance partition assignment Key: KAFKA-7144 URL: https://issues.apache.org/jira/browse/KAFKA-7144 Project: Kafka Issue

[jira] [Created] (KAFKA-7143) Cannot use KafkaConsumer with Kotlin coroutines due to Thread id check

Raman Gupta created KAFKA-7143: -- Summary: Cannot use KafkaConsumer with Kotlin coroutines due to Thread id check Key: KAFKA-7143 URL: https://issues.apache.org/jira/browse/KAFKA-7143 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk10 #281

See Changes: [wangguoz] MINOR: Add Scalafmt to Streams Scala API (#4965) -- [...truncated 1.53 MB...] kafka.coordinator.transaction.TransactionCoordinatorTest >

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Hi Dong, Thanks for considering my suggestions. Based on your comments, I realized that my suggestion was not complete with regard to KafkaConsumer API vs. consumer-broker protocol. While I propose to keep KafkaConsumer#seek() unchanged and take offset only, the underlying consumer will send

Re: [VOTE] 1.1.1 RC3

+1 (non-binding) - Ran testAll/releaseTarGzAll on 1.1.0-rc0 tag with jdk-8. - Ran through quickstart of core/streams on builds. - Ran few internal apps targeting to topics on 3 node cluster. Thanks, Satish. On Mon, Jul 9, 2018 at 11:39 PM, Harsha wrote: > +1. > > * Ran unit tests > *

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Hi Dong, The simulation of disk being slow is merely for me to easily construct a testing scenario with a backlog of produce requests. In production, other than the disk being slow, a backlog of produce requests may also be caused by high produce QPS. In that case, we may not want to kill the

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Let me clarify a bit on what I meant about moving `retentionPeriod` to WindowStoreBuilder: In another discussion we had around KIP-319 / 330, that the "retention period" should not really be a window spec, but only a window store spec, as it only affects how long to retain each window to be

[jira] [Created] (KAFKA-7142) Rebalancing large consumer group can block the coordinator broker for several seconds

Ying Zheng created KAFKA-7142: - Summary: Rebalancing large consumer group can block the coordinator broker for several seconds Key: KAFKA-7142 URL: https://issues.apache.org/jira/browse/KAFKA-7142

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Thanks for the reply, Guozhang, Good! I agree, that is also a good reason, and I actually made use of that in my tests. I'll update the KIP. By the way, I chose "allowedLateness" as I was trying to pick a better name than "close", but I think it's actually the wrong name. We don't want to bound

Re: [VOTE] KIP-231: Improve the Required ACL of ListGroups API

Late +1 (binding) from me too. Ismael On Mon, Jul 9, 2018 at 3:18 PM Vahid S Hashemian wrote: > KIP-231 is approved with 3 "+1" binding votes by Ewen, Jason, and Harsha. > > Thanks to everyone who provided feedback and/or voted. > > Regards. > --Vahid > > > > > From: Vahid S

Re: [VOTE] KIP-231: Improve the Required ACL of ListGroups API

KIP-231 is approved with 3 "+1" binding votes by Ewen, Jason, and Harsha. Thanks to everyone who provided feedback and/or voted. Regards. --Vahid From: Vahid S Hashemian/Silicon Valley/IBM To: dev Date: 12/19/2017 11:30 AM Subject:[VOTE] KIP-231: Improve the Required ACL of

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

John, Thanks for your replies. As for the two options of the API, I think I'm slightly inclined to the first option as well. My motivation is a bit different, as I think of the first one maybe more flexible, for example: KTable> table = ... count(); table.toStream().peek(..); // want to peek

Re: Stream config in StreamPartitionAssignor

Hello Boyang, Thanks for brining this up. Currently since the StreamsPartitionAssingor can only be initiated within the Consumer instance, I think letting users to pass in the config values with prefix is the preferrable way, i.e. we can improve our docs to educate users about that. BTW I'm

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Hey Matthias and Guozhang, Sorry for the slow reply. I was mulling about your feedback and weighing some ideas in a sketchbook PR: https://github.com/apache/kafka/pull/5337. Your thought about keeping suppression independent of business logic is a very good one. I agree that it would make more

Stream config in StreamPartitionAssignor

Hey there, over the weekend I was debugging the streams configuration not passed within threads. I noticed that one of the code path from KafkaConsumer (L743) was to initialize the StreamPartitionAssignor: this.assignors = config.getConfiguredInstances(

Recovering partition leadership outside ISR

Hi all, We had an outage last week that I think we could have prevented, and I'd like to get some feedback on the idea. tl;dr: When a partition leader writes an updated ISR, it should also record its current log-end-offset. On leader election, if there are no live replicas in the ISR, then a

Re: [VOTE] 1.1.1 RC3

+1. * Ran unit tests * Installed in a cluster and ran simple tests Thanks, Harsha On Mon, Jul 9th, 2018 at 6:38 AM, Ted Yu wrote: > > > > +1 > > Ran test suite. > > Checked signatures. > > > > On Sun, Jul 8, 2018 at 3:36 PM Dong Lin < lindon...@gmail.com > wrote: > > > Hello Kafka

[jira] [Created] (KAFKA-7141) kafka-consumer-group doesn't describe existing group

Bohdana Panchenko created KAFKA-7141: Summary: kafka-consumer-group doesn't describe existing group Key: KAFKA-7141 URL: https://issues.apache.org/jira/browse/KAFKA-7141 Project: Kafka

Register now for ApacheCon and save $250

Greetings, Apache software enthusiasts! (You’re getting this because you’re on one or more dev@ or users@ lists for some Apache Software Foundation project.) ApacheCon North America, in Montreal, is now just 80 days away, and early bird prices end in just two weeks - on July 21. Prices will

[jira] [Created] (KAFKA-7140) Remove deprecated poll usages

Viktor Somogyi created KAFKA-7140: - Summary: Remove deprecated poll usages Key: KAFKA-7140 URL: https://issues.apache.org/jira/browse/KAFKA-7140 Project: Kafka Issue Type: Improvement

Re: [VOTE] 1.1.1 RC3

+1 Ran test suite. Checked signatures. On Sun, Jul 8, 2018 at 3:36 PM Dong Lin wrote: > Hello Kafka users, developers and client-developers, > > This is the fourth candidate for release of Apache Kafka 1.1.1. > > Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first >

Re: [VOTE] KIP-280: Enhanced log compaction

Hi Luís, Can you please clarify how the header value has to be encoded in case log compaction strategy is 'header'. As I see current PR reads varLong in CleanerCache.extractVersion and read String and uses toLong in Cleaner.extractVersion while the KIP says no more than 'the header value (which

Re: [DISCUSSION] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

Thanks for the KIP. It would be helpful to understand the user experience for the case where the implementor uses the headers. It seems like it would require overriding two methods? Ismael On Mon, Jul 9, 2018 at 1:50 AM Viktor Somogyi wrote: > Hi folks, > > I've published KIP-336 which is

[jira] [Created] (KAFKA-7139) Support to exclude the internal topics in kafka-topics.sh command

Chia-Ping Tsai created KAFKA-7139: - Summary: Support to exclude the internal topics in kafka-topics.sh command Key: KAFKA-7139 URL: https://issues.apache.org/jira/browse/KAFKA-7139 Project: Kafka

[DISCUSSION] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

Hi folks, I've published KIP-336 which is about consolidating the Serializer/Deserializer interfaces. Basically the story here is when ExtendedSerializer and ExtendedDeserializer were added we still supported Java 7 and therefore had to use compatible constructs which now seem unnecessary since

Re: Old deprecated producer

Thx, Ismael. I mean "user batch". Not a background thread batching. So i guess that transactions allow me to do that. On 07/07/2018 17:34, Ismael Juma wrote: The old Scala producers were removed in 2.0.0. The Java producer supports batching. Ismael On Sat, 7 Jul 2018, 05:38 jna, wrote:

Build failed in Jenkins: kafka-trunk-jdk8 #2795

See Changes: [github] MINOR: Upgrade rocksdb to 5.14.2 (#5343) -- [...truncated 435.12 KB...] kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart

Build failed in Jenkins: kafka-trunk-jdk10 #280

See Changes: [github] MINOR: Upgrade rocksdb to 5.14.2 (#5343) -- [...truncated 1.98 MB...] org.apache.kafka.streams.StreamsConfigTest > shouldAcceptExactlyOnce PASSED