Re: [ANNOUNCE] New committer: John Roesler

Congrats John! On Wed, Nov 13, 2019 at 7:57 AM Dong Lin wrote: > Congratulations John! > > On Tue, Nov 12, 2019 at 1:56 PM Guozhang Wang wrote: > > > Hi Everyone, > > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer, > John > > Roesler. > > > > John has been

Re: [DISCUSS] Apache Kafka 2.4.0 release

Hi Manikumar, We have another potential blocker in 2.4, which affects Streams members that fall out of the group and may cause them to die upon rejoin. The ticket is here https://issues.apache.org/jira/browse/KAFKA-9178 The fix is very straightforward, and we have a small PR out for it here

Re: [DISCUSS] Apache Kafka 2.4.0 release

Hi Chris, Since this is existing issue, I am hesitant to include as blocker for 2.4 release. I am in the process of publishing RC0 for 2.4. We will try to include KAFKA-9046, if there any other blockers, which requires roll out of new RC. Thanks, On Wed, Nov 13, 2019 at 5:46 AM John Roesler

Re: [ANNOUNCE] New committer: John Roesler

Congratulations John! On Tue, Nov 12, 2019 at 1:56 PM Guozhang Wang wrote: > Hi Everyone, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer, John > Roesler. > > John has been contributing to Apache Kafka since early 2018. His main > contributions are primarily around

Build failed in Jenkins: kafka-trunk-jdk8 #4035

See Changes: [bbejeck] KAFKA-9011: Scala bindings for flatTransform and flatTransformValues in -- [...truncated 8.27 MB...]

Re: [DISCUSS] KIP-280: Enhanced log compaction

Hello Senthil, Let me try to re-iterate on Jun's comments with some context here: 51: today with the offset-only compaction strategy, the last record of the log (we call it the log-end-record, whose offset is log-end-offset) would always be preserved and not compacted. This is kinda important

Re: [ANNOUNCE] New committer: John Roesler

Congratulations John! Thanks, Mayuresh On Tue, Nov 12, 2019 at 4:54 PM Vahid Hashemian wrote: > Congratulations John! > > --Vahid > > On Tue, Nov 12, 2019 at 4:38 PM Adam Bellemare > wrote: > > > Congratulations John, and thanks for all your help on KIP-213! > > > > > On Nov 12, 2019, at

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

Sounds good to me, let's continue our voting process here. Guozhang On Tue, Nov 12, 2019 at 12:10 PM Ismael Juma wrote: > This is not a bug fix, in my opinion. The existing behavior may be > confusing, but it was documented, so I assume it was intended. > > Ismael > > On Mon, Nov 11, 2019,

Re: [ANNOUNCE] New committer: John Roesler

Congratulations John! --Vahid On Tue, Nov 12, 2019 at 4:38 PM Adam Bellemare wrote: > Congratulations John, and thanks for all your help on KIP-213! > > > On Nov 12, 2019, at 6:24 PM, Bill Bejeck wrote: > > > > Congratulations John! > > > > On Tue, Nov 12, 2019 at 6:20 PM Matthias J. Sax > >

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Thank you all for an interesting discussion. This is very enlightening. Thank you Matthias for your explanation. Your arguments are very true. It makes sense that if user specifies number of partitions he/she really cares that those specifications are applied to internal topics. Unfortunately,

Re: [ANNOUNCE] New committer: John Roesler

Congratulations John, and thanks for all your help on KIP-213! > On Nov 12, 2019, at 6:24 PM, Bill Bejeck wrote: > > Congratulations John! > > On Tue, Nov 12, 2019 at 6:20 PM Matthias J. Sax > wrote: > >> Congrats John! >> >> >>> On 11/12/19 2:52 PM, Boyang Chen wrote: >>> Great work John!

Re: Why standby tasks read from the StandbyTasks::checkpointedOffsets in assignStandbyPartitions()

Thanks for looking at it, Bill. I initially agreed with you, but Manikumar asked me to check if it's really a regression before calling it a blocker. I tested 2.3 and found the same (buggy) behavior, so I don't think we can call it a regression, and therefore, it's also not a blocker. I'm still

Re: [DISCUSS] Apache Kafka 2.4.0 release

Hi again Manikumar, Thanks for looking at the PR. It's not easy to pin this particular bug down to a specific commit, but I've ported my test to the 2.3 branch. It looks like this bug pre-dates 2.4, so it's not a regression. Therefore, after reflection, we don't need to block the 2.4 release on

[jira] [Created] (KAFKA-9178) Streams changelog topics are not cleaning up during lost task closing

Boyang Chen created KAFKA-9178: -- Summary: Streams changelog topics are not cleaning up during lost task closing Key: KAFKA-9178 URL: https://issues.apache.org/jira/browse/KAFKA-9178 Project: Kafka

Re: [DISCUSS] Apache Kafka 2.4.0 release

Hi Manikumar, I'd like to propose the following issue as a blocker: https://issues.apache.org/jira/browse/KAFKA-9046 with a fix PR available at: https://github.com/apache/kafka/pull/7525 Essentially, if a Connect cluster is configured to use the DLQ feature against a secured Kafka cluster, it

[jira] [Created] (KAFKA-9177) Pause completed partitions on restore consumer

Sophie Blee-Goldman created KAFKA-9177: -- Summary: Pause completed partitions on restore consumer Key: KAFKA-9177 URL: https://issues.apache.org/jira/browse/KAFKA-9177 Project: Kafka

Re: [ANNOUNCE] New committer: John Roesler

Congratulations John! On Tue, Nov 12, 2019 at 6:20 PM Matthias J. Sax wrote: > Congrats John! > > > On 11/12/19 2:52 PM, Boyang Chen wrote: > > Great work John! Well deserved > > > > On Tue, Nov 12, 2019 at 1:56 PM Guozhang Wang > wrote: > > > >> Hi Everyone, > >> > >> The PMC of Apache Kafka

Re: [ANNOUNCE] New committer: John Roesler

Congrats John! On 11/12/19 2:52 PM, Boyang Chen wrote: > Great work John! Well deserved > > On Tue, Nov 12, 2019 at 1:56 PM Guozhang Wang wrote: > >> Hi Everyone, >> >> The PMC of Apache Kafka is pleased to announce a new Kafka committer, John >> Roesler. >> >> John has been contributing to

Re: [ANNOUNCE] New committer: John Roesler

Great work John! Well deserved On Tue, Nov 12, 2019 at 1:56 PM Guozhang Wang wrote: > Hi Everyone, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer, John > Roesler. > > John has been contributing to Apache Kafka since early 2018. His main > contributions are primarily

[ANNOUNCE] New committer: John Roesler

Hi Everyone, The PMC of Apache Kafka is pleased to announce a new Kafka committer, John Roesler. John has been contributing to Apache Kafka since early 2018. His main contributions are primarily around Kafka Streams, but have also included improving our test coverage beyond Streams as well.

Build failed in Jenkins: kafka-2.4-jdk8 #70

See Changes: -- [...truncated 2.70 MB...] org.apache.kafka.streams.test.OutputVerifierTest > shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

Re: [VOTE] KIP-544: Make metrics exposed via JMX configurable

Thanks everyone, KIP-544 has been adopted with 4 binding, and 2 non-binding votes. A PR for the corresponding changes has been submitted https://github.com/apache/kafka/pull/7674 Xavier On Mon, Nov 11, 2019 at 7:56 AM Stanislav Kozlovski wrote: > +1 (non-binding). Thanks Xavier > > > On Sat,

[jira] [Reopened] (KAFKA-9011) Add KStream#flatTransform and KStream#flatTransformValues to Scala API

[ https://issues.apache.org/jira/browse/KAFKA-9011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck reopened KAFKA-9011: There's still something left to address on this PR

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

Hi Colin, Thanks for the feedback. I'm going to leave out some specifics in my response, since I'll go back to the KIP, revise it, and then post an update on the original discussion thread. I'll make two primary changes, (1) remove reference to expiry not being calculated appropriately, since

[jira] [Created] (KAFKA-9176) Flaky test failure: OptimizedKTableIntegrationTest.shouldApplyUpdatesToStandbyStore

Manikumar created KAFKA-9176: Summary: Flaky test failure: OptimizedKTableIntegrationTest.shouldApplyUpdatesToStandbyStore Key: KAFKA-9176 URL: https://issues.apache.org/jira/browse/KAFKA-9176 Project:

[jira] [Created] (KAFKA-9175) MirrorMaker 2 emits invalid topic partition metrics

Mickael Maison created KAFKA-9175: - Summary: MirrorMaker 2 emits invalid topic partition metrics Key: KAFKA-9175 URL: https://issues.apache.org/jira/browse/KAFKA-9175 Project: Kafka Issue

[jira] [Created] (KAFKA-9174) Improve documentation for State Restore lifecycle

Javier Holguera created KAFKA-9174: -- Summary: Improve documentation for State Restore lifecycle Key: KAFKA-9174 URL: https://issues.apache.org/jira/browse/KAFKA-9174 Project: Kafka Issue

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

In all, is everyone OK with - Dropping KeyQueryMetadata, and the allMetadataForKey() apis - Dropping allLagInfo() from KafkaStreams class, Drop StoreLagInfo class - Add offsetLag(store, key, serializer) -> Optional & offsetLag(store, key, partitioner) -> Optional to StreamsMetadata -

Re: [DISCUSS] Apache Kafka 2.4.0 release

Hi Manikumar, There is a new bug report that we think is a blocker: https://issues.apache.org/jira/browse/KAFKA-9169 https://github.com/apache/kafka/pull/7681 See also the dev@ thread entitled "Why standby tasks read from the StandbyTasks::checkpointedOffsets in assignStandbyPartitions()" for

Build failed in Jenkins: kafka-2.4-jdk8 #69

See Changes: -- [...truncated 2.69 MB...] org.apache.kafka.streams.test.OutputVerifierTest > shouldNotAllowNullExpectedRecordForCompareValue STARTED

Re: Why standby tasks read from the StandbyTasks::checkpointedOffsets in assignStandbyPartitions()

This could be a significant performance issue for some, so I think this fix needs to go into 2.4. Just my 2 cents. -Bill On Mon, Nov 11, 2019 at 5:57 PM John Roesler wrote: > Ok, created: https://github.com/apache/kafka/pull/7681 > > I'm on the fence about whether we should file this as a

KafkaChangeLog Segment Bytes Kafka version Error V2.0.0

hello Folks I create new topic with params Topic:testKafkachangelogPartitionCount:3ReplicationFactor:3 Configs:segment.bytes=5000,retention.ms=60,cleanup.policy=compact Topic: testKafkachangelog Partition: 0Leader: 0 Replicas: 0,2,1 Isr:

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

- Looking back, I agree that 2 calls to StreamsMetadata to fetch StreamsMetadata and then using something like ‘long StreamsMetadata#offsetLag(store, key)’ which Matthias suggested is better than introducing a new public API i.e. ‘KeyQueryMetadata’. I will change the KIP accordingly.

Build failed in Jenkins: kafka-2.4-jdk8 #68

See Changes: -- [...truncated 2.70 MB...] org.apache.kafka.streams.test.OutputVerifierTest > shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

[jira] [Created] (KAFKA-9173) StreamsPartitionAssignor assigns partitions to only one worker

Oleg Muravskiy created KAFKA-9173: - Summary: StreamsPartitionAssignor assigns partitions to only one worker Key: KAFKA-9173 URL: https://issues.apache.org/jira/browse/KAFKA-9173 Project: Kafka

[jira] [Created] (KAFKA-9172) Kafka Connect JMX : source & sink task metrics are missing in some cases after rebalancing of the tasks

Raj created KAFKA-9172: -- Summary: Kafka Connect JMX : source & sink task metrics are missing in some cases after rebalancing of the tasks Key: KAFKA-9172 URL: https://issues.apache.org/jira/browse/KAFKA-9172