Build failed in Jenkins: kafka-2.4-jdk8 #27

[Apache Jenkins Server]

See 


Changes:

[matthias] MINOR: fix typo in TestInputTopic.getTimestampAndAdvance (#7553)


--
[...truncated 5.11 MB...]
kafka.server.MetadataCacheTest > getTopicMetadataIsrNotAvailable PASSED

kafka.server.MetadataCacheTest > getTopicMetadata STARTED

kafka.server.MetadataCacheTest > getTopicMetadata PASSED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable STARTED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable PASSED

kafka.server.MetadataCacheTest > 
getTopicMetadataPartitionListenerNotAvailableOnLeaderOldMetadataVersion STARTED

kafka.server.MetadataCacheTest > 
getTopicMetadataPartitionListenerNotAvailableOnLeaderOldMetadataVersion PASSED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
STARTED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
PASSED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
STARTED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
PASSED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics STARTED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics PASSED

kafka.server.KafkaServerTest > testAlreadyRegisteredAdvertisedListeners STARTED

kafka.server.KafkaServerTest > testAlreadyRegisteredAdvertisedListeners PASSED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK STARTED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK PASSED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot STARTED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot PASSED

kafka.server.ServerStartupTest > testConflictBrokerStartupWithSamePort STARTED

kafka.server.ServerStartupTest > testConflictBrokerStartupWithSamePort PASSED

kafka.server.ServerStartupTest > testConflictBrokerRegistration STARTED

kafka.server.ServerStartupTest > testConflictBrokerRegistration PASSED

kafka.server.ServerStartupTest > testBrokerSelfAware STARTED

kafka.server.ServerStartupTest > testBrokerSelfAware PASSED

kafka.server.DeleteTopicsRequestTest > testValidDeleteTopicRequests STARTED

kafka.server.DeleteTopicsRequestTest > testValidDeleteTopicRequests PASSED

kafka.server.DeleteTopicsRequestTest > testErrorDeleteTopicRequests STARTED

kafka.server.DeleteTopicsRequestTest > testErrorDeleteTopicRequests PASSED

kafka.server.DeleteTopicsRequestTest > testNotController STARTED

kafka.server.DeleteTopicsRequestTest > testNotController PASSED

kafka.server.ReplicationQuotaManagerTest > shouldThrottleOnlyDefinedReplicas 
STARTED

kafka.server.ReplicationQuotaManagerTest > shouldThrottleOnlyDefinedReplicas 
PASSED

kafka.server.ReplicationQuotaManagerTest > 
shouldSupportWildcardThrottledReplicas STARTED

kafka.server.ReplicationQuotaManagerTest > 
shouldSupportWildcardThrottledReplicas PASSED

kafka.server.ReplicationQuotaManagerTest > 
shouldExceedQuotaThenReturnBackBelowBoundAsTimePasses STARTED

kafka.server.ReplicationQuotaManagerTest > 
shouldExceedQuotaThenReturnBackBelowBoundAsTimePasses PASSED

kafka.server.FetchRequestTest > testZStdCompressedRecords STARTED

kafka.server.FetchRequestTest > testZStdCompressedRecords PASSED

kafka.server.FetchRequestTest > testFetchRequestToNonReplica STARTED

kafka.server.FetchRequestTest > testFetchRequestToNonReplica PASSED

kafka.server.FetchRequestTest > testBrokerRespectsPartitionsOrderAndSizeLimits 
STARTED

kafka.server.FetchRequestTest > testBrokerRespectsPartitionsOrderAndSizeLimits 
PASSED

kafka.server.FetchRequestTest > testZStdCompressedTopic STARTED

kafka.server.FetchRequestTest > testZStdCompressedTopic PASSED

kafka.server.FetchRequestTest > 
testDownConversionFromBatchedToUnbatchedRespectsOffset STARTED

kafka.server.FetchRequestTest > 
testDownConversionFromBatchedToUnbatchedRespectsOffset PASSED

kafka.server.FetchRequestTest > testFetchRequestV2WithOversizedMessage STARTED

kafka.server.FetchRequestTest > testFetchRequestV2WithOversizedMessage PASSED

kafka.server.FetchRequestTest > testDownConversionWithConnectionFailure STARTED

kafka.server.FetchRequestTest > testDownConversionWithConnectionFailure PASSED

kafka.server.FetchRequestTest > testCurrentEpochValidation STARTED

kafka.server.FetchRequestTest > testCurrentEpochValidation PASSED

kafka.server.FetchRequestTest > testCreateIncrementalFetchWithPartitionsInError 
STARTED

kafka.server.FetchRequestTest > testCreateIncrementalFetchWithPartitionsInError 
PASSED

kafka.server.FetchRequestTest > testFetchRequestV4WithReadCommitted STARTED

kafka.server.FetchRequestTest > testFetchRequestV4WithReadCommitted PASSED

kafka.server.GssapiAuthenticationTest > testServerNotFoundInKerberosDatabase 
STARTED

kafka.server.GssapiAuthenticationTest > testServerNotFoundInKerberosDatabase 
PASSED

kafka.server.GssapiAuthenticationTest > testRequestIsAReplay STARTED

[jira] [Reopened] (KAFKA-4996) Fix findbugs multithreaded correctness warnings for streams

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-4996:


Thanks for pointing out [~ableegoldman] – we should address it – I was assuming 
that the build would fail if there is an issue. But maybe we can only enable a 
failing build after we did some cleanup to make it pass to begin with...

> Fix findbugs multithreaded correctness warnings for streams
> ---
>
> Key: KAFKA-4996
> URL: https://issues.apache.org/jira/browse/KAFKA-4996
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Colin McCabe
>Priority: Major
>  Labels: newbie
>
> Fix findbugs multithreaded correctness warnings for streams
> {code}
> Multithreaded correctness Warnings
>   
>   
> 
>   
>   
>   
> 
>Code Warning   
>   
>   
> 
>AT   Sequence of calls to java.util.concurrent.ConcurrentHashMap may not 
> be atomic in 
> org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(long, 
> ProcessorContext) 
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.KafkaStreams.stateListener; locked 66% of time   
>   
>   
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.processor.internals.StreamThread.stateListener; 
> locked 66% of time
>   
>  
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.processor.TopologyBuilder.applicationId; locked 50% 
> of time   
>   
>  
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingKeyValueStore.context; locked 
> 66% of time   
>   
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingWindowStore.cache; locked 60% 
> of time   
>   
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingWindowStore.context; locked 
> 66% of time   
>   
>   
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingWindowStore.name; locked 60% 
> of time   
>   
>  
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingWindowStore.serdes; locked 
> 70% of time   
>   
>
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.RocksDBStore.db; locked 63% of time  
>   
>   
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.RocksDBStore.serdes; locked 76% of 
> time  
> 

Build failed in Jenkins: kafka-trunk-jdk11 #893

[Apache Jenkins Server]

See 


Changes:

[matthias] MINOR: fix typo in TestInputTopic.getTimestampAndAdvance (#7553)


--
[...truncated 2.94 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 

Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in low volume suppression buffers

[Richard Yu]

Hi Bill,

Thanks for the input!
TBH, I am think that suppression buffers are not used *in response *to low
traffic conditions.
Rather, we are trying to fix the situation when low traffic conditions
occur in a suppression buffer (for example, previously, the same
suppression buffer had a decent volume of records entering it, thus
advancing the stream time).
In summary, when these conditions do hit, we want to advance the stream
time somehow to resolve this issue.
Reflecting on this though, I am not completely certain if this really stops
us from implementing per key stream time tracking because the problem
wouldn't be made *that *much worse.

Cheers,
Richard

On Fri, Oct 18, 2019 at 11:46 AM Bill Bejeck  wrote:

> Hi Richard,
>
> Thanks for the KIP proposal.  I understand the situation you are
> describing.
> But in my mind, if there is a low traffic condition and you need to keep
> records going downstream at regular intervals, I'm wondering if using
> suppression is the correct approach.
> IMHO it seems it would be better to use the PAPI or a Transform on the DSL
> with a scheduled punctuation call.
>
> Just my 2 cents.
>
> Thanks,
> Bill
>
> On Thu, Oct 17, 2019 at 7:42 PM Richard Yu 
> wrote:
>
> > Hi all,
> >
> > I wish to discuss this KIP which would help us in resolving some issues
> we
> > have with suppression buffers.
> > Below is the link:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-539%3A+Implement+mechanism+to+flush+out+records+in+low+volume+suppression+buffers
> >
> > @John Roesler if you have time, would be great if we could get your
> input.
> >
> > Cheers,
> > Richard
> >
>

[jira] [Resolved] (KAFKA-8994) Streams should expose standby replication information & allow stale reads of state store

[Vinoth Chandar (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinoth Chandar resolved KAFKA-8994.
---
Resolution: Duplicate

> Streams should expose standby replication information & allow stale reads of 
> state store
> 
>
> Key: KAFKA-8994
> URL: https://issues.apache.org/jira/browse/KAFKA-8994
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: Vinoth Chandar
>Assignee: Vinoth Chandar
>Priority: Major
>  Labels: needs-kip
>
> Currently Streams interactive queries (IQ) fail during the time period where 
> there is a rebalance in progress. 
> Consider the following scenario in a three node Streams cluster with node A, 
> node S and node R, executing a stateful sub-topology/topic group with 1 
> partition and `_num.standby.replicas=1_`  
>  * *t0*: A is the active instance owning the partition, B is the standby that 
> keeps replicating the A's state into its local disk, R just routes streams 
> IQs to active instance using StreamsMetadata
>  * *t1*: IQs pick node R as router, R forwards query to A, A responds back to 
> R which reverse forwards back the results.
>  * *t2:* Active A instance is killed and rebalance begins. IQs start failing 
> to A
>  * *t3*: Rebalance assignment happens and standby B is now promoted as active 
> instance. IQs continue to fail
>  * *t4*: B fully catches up to changelog tail and rewinds offsets to A's last 
> commit position, IQs continue to fail
>  * *t5*: IQs to R, get routed to B, which is now ready to serve results. IQs 
> start succeeding again
>  
> Depending on Kafka consumer group session/heartbeat timeouts, step t2,t3 can 
> take few seconds (~10 seconds based on defaults values). Depending on how 
> laggy the standby B was prior to A being killed, t4 can take few 
> seconds-minutes. 
> While this behavior favors consistency over availability at all times, the 
> long unavailability window might be undesirable for certain classes of 
> applications (e.g simple caches or dashboards). 
> This issue aims to also expose information about standby B to R, during each 
> rebalance such that the queries can be routed by an application to a standby 
> to serve stale reads, choosing availability over consistency. 
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (KAFKA-8994) Streams should expose standby replication information & allow stale reads of state store

[Vinoth Chandar (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinoth Chandar reopened KAFKA-8994:
---

> Streams should expose standby replication information & allow stale reads of 
> state store
> 
>
> Key: KAFKA-8994
> URL: https://issues.apache.org/jira/browse/KAFKA-8994
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: Vinoth Chandar
>Assignee: Vinoth Chandar
>Priority: Major
>  Labels: needs-kip
>
> Currently Streams interactive queries (IQ) fail during the time period where 
> there is a rebalance in progress. 
> Consider the following scenario in a three node Streams cluster with node A, 
> node S and node R, executing a stateful sub-topology/topic group with 1 
> partition and `_num.standby.replicas=1_`  
>  * *t0*: A is the active instance owning the partition, B is the standby that 
> keeps replicating the A's state into its local disk, R just routes streams 
> IQs to active instance using StreamsMetadata
>  * *t1*: IQs pick node R as router, R forwards query to A, A responds back to 
> R which reverse forwards back the results.
>  * *t2:* Active A instance is killed and rebalance begins. IQs start failing 
> to A
>  * *t3*: Rebalance assignment happens and standby B is now promoted as active 
> instance. IQs continue to fail
>  * *t4*: B fully catches up to changelog tail and rewinds offsets to A's last 
> commit position, IQs continue to fail
>  * *t5*: IQs to R, get routed to B, which is now ready to serve results. IQs 
> start succeeding again
>  
> Depending on Kafka consumer group session/heartbeat timeouts, step t2,t3 can 
> take few seconds (~10 seconds based on defaults values). Depending on how 
> laggy the standby B was prior to A being killed, t4 can take few 
> seconds-minutes. 
> While this behavior favors consistency over availability at all times, the 
> long unavailability window might be undesirable for certain classes of 
> applications (e.g simple caches or dashboards). 
> This issue aims to also expose information about standby B to R, during each 
> rebalance such that the queries can be routed by an application to a standby 
> to serve stale reads, choosing availability over consistency. 
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: KAFKA-8584: Support of ByteBuffer for bytes field implemented[Convert Kafka RPCs to use automatically generated code]

[Colin McCabe]

Hi Nikolay,

Sorry that I haven't had more bandwidth to review this recently.  I will take a 
look today.

In the future, can you please rebase your changes on top of trunk, rather than 
merging trunk into your branch?  It is difficult to follow which changes are 
yours and which come from the merge, when you do it the other way.

best,
Colin


On Thu, Oct 17, 2019, at 02:59, Nikolay Izhikov wrote:
> Hello.
> 
> Is there something wrong with the PR?
> Do we need this ticket to be done? [2]
> If no, let's close both PR [1] and ticket.
> 
> The design or implementation details were changed?
> If yes, can you, please, send a link where I can find details.
> 
> [1] https://github.com/apache/kafka/pull/7342
> [2] https://issues.apache.org/jira/browse/KAFKA-8885
> 
> пн, 7 окт. 2019 г. в 10:08, Nikolay Izhikov :
> 
> > Hello.
> >
> > Please, review my changes [1]
> > I fixed all conflicts after KAFKA-8885 [2] merge [3].
> >
> > [1] https://github.com/apache/kafka/pull/7342
> > [2] https://issues.apache.org/jira/browse/KAFKA-8885
> > [3]
> > https://github.com/apache/kafka/commit/0de61a4683b92bdee803c51211c3277578ab3edf
> >
> > В Пт, 20/09/2019 в 09:18 -0700, Colin McCabe пишет:
> > > Hi Nikolay,
> > >
> > > Thanks for working on this.  I think everyone agrees that we should have
> > byte buffer support in the generator.  We just haven't had a lot of time
> > for reviewing it lately.   I don't really mind which PR we use :)  I will
> > take a look at your PR today and see if we can get it into shape for what
> > we need.
> > >
> > > best,
> > > Colin
> > >
> > > On Fri, Sep 20, 2019, at 09:18, Nikolay Izhikov wrote:
> > > > Hello, all.
> > > >
> > > > Any feedback on this?
> > > > Do we need support of ByteBuffer in RPC generated code?
> > > >
> > > > Which PR should be reviwed and merged?
> > > >
> > > > В Чт, 19/09/2019 в 10:11 +0300, Nikolay Izhikov пишет:
> > > > > Hello, guys.
> > > > >
> > > > > Looks like we have duplicate tickets and PR's here.
> > > > >
> > > > > One from me:
> > > > >
> > > > > KAFKA-8584: Support of ByteBuffer for bytes field implemented.
> > > > > ticket - https://issues.apache.org/jira/browse/KAFKA-8584
> > > > > pr - https://github.com/apache/kafka/pull/7342
> > > > >
> > > > > and one from Colin McCabe:
> > > > >
> > > > > KAFKA-8628: Auto-generated Kafka RPC code should be able to use
> > zero-copy ByteBuffers
> > > > > ticket - https://issues.apache.org/jira/browse/KAFKA-8628
> > > > > pr - https://github.com/apache/kafka/pull/7032
> > > > >
> > > > > I want to continue work on my PR and got it merged.
> > > > > But, it up to community to decide which changes are best for the
> > product.
> > > > >
> > > > > Please, let me know, what do you think.
> > > > >
> > > > >
> > > > > В Вт, 17/09/2019 в 01:52 +0300, Nikolay Izhikov пишет:
> > > > > > Hello, Kafka team.
> > > > > >
> > > > > > I implemented KAFKA-8584 [1].
> > > > > > PR - [2]
> > > > > > Please, do the review.
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/KAFKA-8584
> > > > > > [2] https://github.com/apache/kafka/pull/7342
> > > >
> > > > Attachments:
> > > > * signature.asc
> >
>

Re: [VOTE] KIP-534: Retain tombstones for approximately delete.retention.ms milliseconds

[Richard Yu]

Hi all,

Seeing that we got all out votes needed with 3 binding votes and 0
nonbinding.
I consider this KIP passed.

Cheers,
Richard

On Fri, Oct 18, 2019 at 9:17 AM Guozhang Wang  wrote:

> Thanks Richard, I'm +1 on the KIP
>
> On Thu, Oct 17, 2019 at 3:51 PM Richard Yu 
> wrote:
>
> > Hi Guozhang, Jason,
> >
> > I've updated the KIP to include this warning as well.
> > If there is anything else that we need for it, let me know. :)
> > Otherwise, we should vote this KIP in.
> >
> > Cheers,
> > Richard
> >
> > On Thu, Oct 17, 2019 at 10:41 AM Jason Gustafson 
> > wrote:
> >
> > > Hi Guozhang,
> > >
> > > It's a fair point. For control records, I think it's a non-issue since
> > they
> > > are tiny and not batched. So the only case where this might matter is
> > large
> > > batch deletions. I think the size difference is not a major issue
> itself,
> > > but I think it's worth mentioning in the KIP the risk of exceeding the
> > max
> > > message size. I think the code should probably make this more of a soft
> > > limit when cleaning. We have run into scenarios in the past as well
> where
> > > recompression has actually increased message size. We may also want to
> be
> > > able to upconvert messages to the new format in the future in the
> > cleaner.
> > >
> > > -Jason
> > >
> > >
> > >
> > > On Thu, Oct 17, 2019 at 9:08 AM Guozhang Wang 
> > wrote:
> > >
> > > > Here's my understanding: when log compaction kicks in, the system
> time
> > at
> > > > the moment would be larger than the message timestamp to be
> compacted,
> > so
> > > > the modification on the batch timestamp would practically be
> increasing
> > > its
> > > > value, and hence the deltas for each inner message would be negative
> to
> > > > maintain their actual timestamp. Depending on the time diff between
> the
> > > > actual timestamp of the message and the time when log compaction
> > happens,
> > > > this negative delta can be large or small since it not long depends
> on
> > > the
> > > > cleaner thread wakeup frequency but also dirty ratio etc.
> > > >
> > > > With varInt encoding, the num.bytes needed for encode an int varies
> > from
> > > 1
> > > > to 5 bytes; before compaction, the deltas should be relatively small
> > > > positive values compared with the base timestamp, and hence most
> > likely 1
> > > > or 2 bytes needed to encode, after compaction, the deltas could be
> > > > relatively large negative values that may take more bytes to encode.
> > > With a
> > > > record batch of 512 in practice, and suppose after compaction each
> > record
> > > > would take 2 more byte for encoding deltas, that would be 1K more per
> > > > batch. Usually it would not be too big of an issue with reasonable
> > sized
> > > > message, but I just wanted to point out this as a potential
> regression.
> > > >
> > > >
> > > > Guozhang
> > > >
> > > > On Wed, Oct 16, 2019 at 9:36 PM Richard Yu <
> yohan.richard...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Guozhang,
> > > > >
> > > > > Your understanding basically is on point.
> > > > >
> > > > > I haven't looked into the details for what happens if we change the
> > > base
> > > > > timestamp and how its calculated, so I'm not clear on how small the
> > > delta
> > > > > (or big) is.
> > > > > To be fair, would the the delta size pose a big problem if it takes
> > up
> > > > more
> > > > > bytes to encode?
> > > > >
> > > > > Cheers,
> > > > > Richard
> > > > >
> > > > > On Wed, Oct 16, 2019 at 7:36 PM Guozhang Wang 
> > > > wrote:
> > > > >
> > > > > > Hello Richard,
> > > > > >
> > > > > > Thanks for the KIP, I just have one clarification regarding "So
> the
> > > > idea
> > > > > is
> > > > > > to set the base timestamp to the delete horizon and adjust the
> > deltas
> > > > > > accordingly." My understanding is that during compaction, for
> each
> > > > > > compacted new segment, we would set its base offset of each batch
> > as
> > > > the
> > > > > > delete horizon, which is the "current system time that cleaner
> has
> > > seen
> > > > > so
> > > > > > far", and adjust the delta timestamps of each of the inner
> records
> > of
> > > > the
> > > > > > batch (and practically the deltas will be all negative)?
> > > > > >
> > > > > > If that's case, could we do some back of the envelope calculation
> > on
> > > > > what's
> > > > > > the possible smallest case of deltas? Note that since we use
> varInt
> > > for
> > > > > > delta values for each record, the smaller the negative delta,
> that
> > > > would
> > > > > > take more bytes to encode.
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Wed, Oct 16, 2019 at 6:48 PM Jason Gustafson <
> > ja...@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > > +1. Thanks Richard.
> > > > > > >
> > > > > > > On Wed, Oct 16, 2019 at 10:04 AM Richard Yu <
> > > > > yohan.richard...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Want to try to get this KIP wrapped up. So it 

Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in low volume suppression buffers

[Bill Bejeck]

Hi Richard,

Thanks for the KIP proposal.  I understand the situation you are
describing.
But in my mind, if there is a low traffic condition and you need to keep
records going downstream at regular intervals, I'm wondering if using
suppression is the correct approach.
IMHO it seems it would be better to use the PAPI or a Transform on the DSL
with a scheduled punctuation call.

Just my 2 cents.

Thanks,
Bill

On Thu, Oct 17, 2019 at 7:42 PM Richard Yu 
wrote:

> Hi all,
>
> I wish to discuss this KIP which would help us in resolving some issues we
> have with suppression buffers.
> Below is the link:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-539%3A+Implement+mechanism+to+flush+out+records+in+low+volume+suppression+buffers
>
> @John Roesler if you have time, would be great if we could get your input.
>
> Cheers,
> Richard
>

[jira] [Resolved] (KAFKA-3534) Deserialize on demand when default time extractor used

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-3534.

Resolution: Fixed

> Deserialize on demand when default time extractor used
> --
>
> Key: KAFKA-3534
> URL: https://issues.apache.org/jira/browse/KAFKA-3534
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1, 0.10.2.0
>Reporter: Michael Coon
>Priority: Minor
>  Labels: performance
>
> When records are added to the RecordQueue, they are deserialized at that time 
> in order to extract the timestamp. But for some data flows where large 
> messages are consumed (particularly compressed messages), this can result in 
> large spikes in memory as all messages must be deserialized prior to 
> processing (and getting out of memory). An optimization might be to only 
> require deserialization at this stage if a non-default timestamp extractor is 
> being used.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-4996) Fix findbugs multithreaded correctness warnings for streams

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-4996.

Resolution: Fixed

The corresponding PR was merged. Closing this ticket.

> Fix findbugs multithreaded correctness warnings for streams
> ---
>
> Key: KAFKA-4996
> URL: https://issues.apache.org/jira/browse/KAFKA-4996
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Colin McCabe
>Priority: Major
>  Labels: newbie
>
> Fix findbugs multithreaded correctness warnings for streams
> {code}
> Multithreaded correctness Warnings
>   
>   
> 
>   
>   
>   
> 
>Code Warning   
>   
>   
> 
>AT   Sequence of calls to java.util.concurrent.ConcurrentHashMap may not 
> be atomic in 
> org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(long, 
> ProcessorContext) 
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.KafkaStreams.stateListener; locked 66% of time   
>   
>   
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.processor.internals.StreamThread.stateListener; 
> locked 66% of time
>   
>  
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.processor.TopologyBuilder.applicationId; locked 50% 
> of time   
>   
>  
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingKeyValueStore.context; locked 
> 66% of time   
>   
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingWindowStore.cache; locked 60% 
> of time   
>   
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingWindowStore.context; locked 
> 66% of time   
>   
>   
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingWindowStore.name; locked 60% 
> of time   
>   
>  
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingWindowStore.serdes; locked 
> 70% of time   
>   
>
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.RocksDBStore.db; locked 63% of time  
>   
>   
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.RocksDBStore.serdes; locked 76% of 
> time  
>   
>   
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-7424) State stores restoring from changelog topic not the source topic

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-7424.

Resolution: Fixed

[~jhay], [~vvcephei]: I am closing this ticket as it seems to be covered via 
KAFKA-6729.

If you disagree, feel free to reopen and provide more context on the issue.

> State stores restoring from changelog topic not the source topic
> 
>
> Key: KAFKA-7424
> URL: https://issues.apache.org/jira/browse/KAFKA-7424
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.1.1
>Reporter: James Hay
>Priority: Critical
>
> Hi,
>  
> I've recently attempted to upgrade a streams application form 1.1 to 1.1.1 
> and I noticed a drop in the number of messages being restored in our state 
> stores.
> It appears that there is a change in 1.1.1 which causes our state stores to 
> be restored from the changelog topic as opposed to version 1.1 where the 
> stores are restored from the source topic. In our application this causes an 
> issue as we switched to StreamsBuilder from KStreamBuilder in the middle of 
> the applications lifetime and so the changelog doesn't represent a full 
> history of the source topic.
> Has this switch been introduced intentionally? Is there a way to configure 
> our application to use 1.1.1 and still use the source stream to restore state 
> stores? Any recommendations on getting our changelog in sync with the source?
>  
> Thanks
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-8297) Kafka Streams ConsumerRecordFactory yields difficult compiler error about generics

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-8297.

Resolution: Fixed

Closing this ticket because the factory class was deprecated via 
https://issues.apache.org/jira/browse/KAFKA-8233 (KIP-470) in 2.4 release, and 
replaced with `TestInputTopic`.

> Kafka Streams ConsumerRecordFactory yields difficult compiler error about 
> generics
> --
>
> Key: KAFKA-8297
> URL: https://issues.apache.org/jira/browse/KAFKA-8297
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Michael Drogalis
>Priority: Minor
> Fix For: 2.4.0
>
>
> When using the ConsumerRecordFactory, it's convenient to specify a default 
> topic to create records with:
> {code:java}
> ConsumerRecordFactory inputFactory = new 
> ConsumerRecordFactory<>(inputTopic, keySerializer, valueSerializer);
> {code}
> However, when the factory is used to create a record with a String key:
> {code:java}
> inputFactory.create("any string", user)
> {code}
> Compilation fails with the following warning:
> {code:java}
> Ambiguous method call. Both:
> create(String, User) in ConsumerRecordFactory and
> create(String, User) in ConsumerRecordFactory match
> {code}
> At first glance, this is a really confusing error to see during compilation. 
> What's happening is that there are two clashing signatures for `create`: 
> create(K, V) and create(String, V). The latter signature represents a topic 
> name.
> It seems like fixing this would require breaking the existing interface. This 
> is a really opaque problem to hit though, and it would be great if we could 
> avoid having users encounter this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk8 #3979

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-8834; Add reassignment metrics and distinguish URPs (KIP-352)

[jason] KAFKA-8962; Use least loaded node for AdminClient#describeTopics (#7421)


--
[...truncated 7.77 MB...]
org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest
 > 
shouldFetchAllSegments[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 FAILED
java.lang.IllegalStateException: Shutdown in progress

org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest
 > 
shouldLogAndMeasureExpiredRecords[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 STARTED
org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest.shouldLogAndMeasureExpiredRecords[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 failed, log available in 


org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest
 > 
shouldLogAndMeasureExpiredRecords[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 FAILED
java.lang.IllegalStateException: Shutdown in progress

org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest
 > 
shouldCreateWriteBatches[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 STARTED
org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest.shouldCreateWriteBatches[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 failed, log available in 


org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest
 > 
shouldCreateWriteBatches[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 FAILED
java.lang.IllegalStateException: Shutdown in progress

org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest
 > 
shouldLoadSegmentsWithOldStyleColonFormattedName[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 STARTED
org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest.shouldLoadSegmentsWithOldStyleColonFormattedName[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 failed, log available in 


org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest
 > 
shouldLoadSegmentsWithOldStyleColonFormattedName[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 FAILED
java.lang.IllegalStateException: Shutdown in progress

org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest
 > 
shouldBeAbleToWriteToReInitializedStore[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 STARTED
org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest.shouldBeAbleToWriteToReInitializedStore[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 failed, log available in 


org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest
 > 
shouldBeAbleToWriteToReInitializedStore[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 FAILED
java.lang.IllegalStateException: Shutdown in progress

org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest
 > 
shouldRestoreToByteStore[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 STARTED
org.apache.kafka.streams.state.internals.RocksDBTimestampedSegmentedBytesStoreTest.shouldRestoreToByteStore[org.apache.kafka.streams.state.internals.WindowKeySchema@16553678]
 failed, log available in 

[jira] [Resolved] (KAFKA-8994) Streams should expose standby replication information & allow stale reads of state store

[Vinoth Chandar (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinoth Chandar resolved KAFKA-8994.
---
Resolution: Fixed

> Streams should expose standby replication information & allow stale reads of 
> state store
> 
>
> Key: KAFKA-8994
> URL: https://issues.apache.org/jira/browse/KAFKA-8994
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: Vinoth Chandar
>Assignee: Vinoth Chandar
>Priority: Major
>  Labels: needs-kip
>
> Currently Streams interactive queries (IQ) fail during the time period where 
> there is a rebalance in progress. 
> Consider the following scenario in a three node Streams cluster with node A, 
> node S and node R, executing a stateful sub-topology/topic group with 1 
> partition and `_num.standby.replicas=1_`  
>  * *t0*: A is the active instance owning the partition, B is the standby that 
> keeps replicating the A's state into its local disk, R just routes streams 
> IQs to active instance using StreamsMetadata
>  * *t1*: IQs pick node R as router, R forwards query to A, A responds back to 
> R which reverse forwards back the results.
>  * *t2:* Active A instance is killed and rebalance begins. IQs start failing 
> to A
>  * *t3*: Rebalance assignment happens and standby B is now promoted as active 
> instance. IQs continue to fail
>  * *t4*: B fully catches up to changelog tail and rewinds offsets to A's last 
> commit position, IQs continue to fail
>  * *t5*: IQs to R, get routed to B, which is now ready to serve results. IQs 
> start succeeding again
>  
> Depending on Kafka consumer group session/heartbeat timeouts, step t2,t3 can 
> take few seconds (~10 seconds based on defaults values). Depending on how 
> laggy the standby B was prior to A being killed, t4 can take few 
> seconds-minutes. 
> While this behavior favors consistency over availability at all times, the 
> long unavailability window might be undesirable for certain classes of 
> applications (e.g simple caches or dashboards). 
> This issue aims to also expose information about standby B to R, during each 
> rebalance such that the queries can be routed by an application to a standby 
> to serve stale reads, choosing availability over consistency. 
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-534: Retain tombstones for approximately delete.retention.ms milliseconds

[Guozhang Wang]

Thanks Richard, I'm +1 on the KIP

On Thu, Oct 17, 2019 at 3:51 PM Richard Yu 
wrote:

> Hi Guozhang, Jason,
>
> I've updated the KIP to include this warning as well.
> If there is anything else that we need for it, let me know. :)
> Otherwise, we should vote this KIP in.
>
> Cheers,
> Richard
>
> On Thu, Oct 17, 2019 at 10:41 AM Jason Gustafson 
> wrote:
>
> > Hi Guozhang,
> >
> > It's a fair point. For control records, I think it's a non-issue since
> they
> > are tiny and not batched. So the only case where this might matter is
> large
> > batch deletions. I think the size difference is not a major issue itself,
> > but I think it's worth mentioning in the KIP the risk of exceeding the
> max
> > message size. I think the code should probably make this more of a soft
> > limit when cleaning. We have run into scenarios in the past as well where
> > recompression has actually increased message size. We may also want to be
> > able to upconvert messages to the new format in the future in the
> cleaner.
> >
> > -Jason
> >
> >
> >
> > On Thu, Oct 17, 2019 at 9:08 AM Guozhang Wang 
> wrote:
> >
> > > Here's my understanding: when log compaction kicks in, the system time
> at
> > > the moment would be larger than the message timestamp to be compacted,
> so
> > > the modification on the batch timestamp would practically be increasing
> > its
> > > value, and hence the deltas for each inner message would be negative to
> > > maintain their actual timestamp. Depending on the time diff between the
> > > actual timestamp of the message and the time when log compaction
> happens,
> > > this negative delta can be large or small since it not long depends on
> > the
> > > cleaner thread wakeup frequency but also dirty ratio etc.
> > >
> > > With varInt encoding, the num.bytes needed for encode an int varies
> from
> > 1
> > > to 5 bytes; before compaction, the deltas should be relatively small
> > > positive values compared with the base timestamp, and hence most
> likely 1
> > > or 2 bytes needed to encode, after compaction, the deltas could be
> > > relatively large negative values that may take more bytes to encode.
> > With a
> > > record batch of 512 in practice, and suppose after compaction each
> record
> > > would take 2 more byte for encoding deltas, that would be 1K more per
> > > batch. Usually it would not be too big of an issue with reasonable
> sized
> > > message, but I just wanted to point out this as a potential regression.
> > >
> > >
> > > Guozhang
> > >
> > > On Wed, Oct 16, 2019 at 9:36 PM Richard Yu  >
> > > wrote:
> > >
> > > > Hi Guozhang,
> > > >
> > > > Your understanding basically is on point.
> > > >
> > > > I haven't looked into the details for what happens if we change the
> > base
> > > > timestamp and how its calculated, so I'm not clear on how small the
> > delta
> > > > (or big) is.
> > > > To be fair, would the the delta size pose a big problem if it takes
> up
> > > more
> > > > bytes to encode?
> > > >
> > > > Cheers,
> > > > Richard
> > > >
> > > > On Wed, Oct 16, 2019 at 7:36 PM Guozhang Wang 
> > > wrote:
> > > >
> > > > > Hello Richard,
> > > > >
> > > > > Thanks for the KIP, I just have one clarification regarding "So the
> > > idea
> > > > is
> > > > > to set the base timestamp to the delete horizon and adjust the
> deltas
> > > > > accordingly." My understanding is that during compaction, for each
> > > > > compacted new segment, we would set its base offset of each batch
> as
> > > the
> > > > > delete horizon, which is the "current system time that cleaner has
> > seen
> > > > so
> > > > > far", and adjust the delta timestamps of each of the inner records
> of
> > > the
> > > > > batch (and practically the deltas will be all negative)?
> > > > >
> > > > > If that's case, could we do some back of the envelope calculation
> on
> > > > what's
> > > > > the possible smallest case of deltas? Note that since we use varInt
> > for
> > > > > delta values for each record, the smaller the negative delta, that
> > > would
> > > > > take more bytes to encode.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Wed, Oct 16, 2019 at 6:48 PM Jason Gustafson <
> ja...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > +1. Thanks Richard.
> > > > > >
> > > > > > On Wed, Oct 16, 2019 at 10:04 AM Richard Yu <
> > > > yohan.richard...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Want to try to get this KIP wrapped up. So it would be great if
> > we
> > > > can
> > > > > > get
> > > > > > > some votes.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Richard
> > > > > > >
> > > > > > > On Tue, Oct 15, 2019 at 12:58 PM Jun Rao 
> > wrote:
> > > > > > >
> > > > > > > > Hi, Richard,
> > > > > > > >
> > > > > > > > Thanks for the updated KIP. +1 from me.
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Tue, Oct 15, 2019 at 12:46 PM Richard Yu <
> > > > > > yohan.richard...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > 

Re: [VOTE] KIP-534: Retain tombstones for approximately delete.retention.ms milliseconds

[Jun Rao]

It's true that the size of the tombstone or the control marker could
increase after a round of cleaning. This can already happen with
re-compression. We already have the logic in the log cleaner to auto grow
the read/write buffer during cleaning and can accommodate for an oversized
message. The only potential issue is that the estimation of the output
segment assumes that a batch doesn't grow in size after cleaning, which can
cause the output segment to overflow the 2GB size limit. However, since the
default log segment size is 1GB, this is unlikely going to be an issue in
practice.

Thanks,

Jun

On Thu, Oct 17, 2019 at 3:55 PM Guozhang Wang  wrote:

> Thanks for the reply Jason.
>
> On Thu, Oct 17, 2019 at 10:40 AM Jason Gustafson 
> wrote:
>
> > Hi Guozhang,
> >
> > It's a fair point. For control records, I think it's a non-issue since
> they
> > are tiny and not batched. So the only case where this might matter is
> large
> > batch deletions. I think the size difference is not a major issue itself,
> > but I think it's worth mentioning in the KIP the risk of exceeding the
> max
> > message size. I think the code should probably make this more of a soft
> > limit when cleaning. We have run into scenarios in the past as well where
> > recompression has actually increased message size. We may also want to be
> > able to upconvert messages to the new format in the future in the
> cleaner.
> >
> > -Jason
> >
> >
> >
> > On Thu, Oct 17, 2019 at 9:08 AM Guozhang Wang 
> wrote:
> >
> > > Here's my understanding: when log compaction kicks in, the system time
> at
> > > the moment would be larger than the message timestamp to be compacted,
> so
> > > the modification on the batch timestamp would practically be increasing
> > its
> > > value, and hence the deltas for each inner message would be negative to
> > > maintain their actual timestamp. Depending on the time diff between the
> > > actual timestamp of the message and the time when log compaction
> happens,
> > > this negative delta can be large or small since it not long depends on
> > the
> > > cleaner thread wakeup frequency but also dirty ratio etc.
> > >
> > > With varInt encoding, the num.bytes needed for encode an int varies
> from
> > 1
> > > to 5 bytes; before compaction, the deltas should be relatively small
> > > positive values compared with the base timestamp, and hence most
> likely 1
> > > or 2 bytes needed to encode, after compaction, the deltas could be
> > > relatively large negative values that may take more bytes to encode.
> > With a
> > > record batch of 512 in practice, and suppose after compaction each
> record
> > > would take 2 more byte for encoding deltas, that would be 1K more per
> > > batch. Usually it would not be too big of an issue with reasonable
> sized
> > > message, but I just wanted to point out this as a potential regression.
> > >
> > >
> > > Guozhang
> > >
> > > On Wed, Oct 16, 2019 at 9:36 PM Richard Yu  >
> > > wrote:
> > >
> > > > Hi Guozhang,
> > > >
> > > > Your understanding basically is on point.
> > > >
> > > > I haven't looked into the details for what happens if we change the
> > base
> > > > timestamp and how its calculated, so I'm not clear on how small the
> > delta
> > > > (or big) is.
> > > > To be fair, would the the delta size pose a big problem if it takes
> up
> > > more
> > > > bytes to encode?
> > > >
> > > > Cheers,
> > > > Richard
> > > >
> > > > On Wed, Oct 16, 2019 at 7:36 PM Guozhang Wang 
> > > wrote:
> > > >
> > > > > Hello Richard,
> > > > >
> > > > > Thanks for the KIP, I just have one clarification regarding "So the
> > > idea
> > > > is
> > > > > to set the base timestamp to the delete horizon and adjust the
> deltas
> > > > > accordingly." My understanding is that during compaction, for each
> > > > > compacted new segment, we would set its base offset of each batch
> as
> > > the
> > > > > delete horizon, which is the "current system time that cleaner has
> > seen
> > > > so
> > > > > far", and adjust the delta timestamps of each of the inner records
> of
> > > the
> > > > > batch (and practically the deltas will be all negative)?
> > > > >
> > > > > If that's case, could we do some back of the envelope calculation
> on
> > > > what's
> > > > > the possible smallest case of deltas? Note that since we use varInt
> > for
> > > > > delta values for each record, the smaller the negative delta, that
> > > would
> > > > > take more bytes to encode.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Wed, Oct 16, 2019 at 6:48 PM Jason Gustafson <
> ja...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > +1. Thanks Richard.
> > > > > >
> > > > > > On Wed, Oct 16, 2019 at 10:04 AM Richard Yu <
> > > > yohan.richard...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Want to try to get this KIP wrapped up. So it would be great if
> > we
> > > > can
> > > > > > get
> > > > > > > some votes.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Richard

Jenkins build is back to normal : kafka-2.3-jdk8 #131

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-9067) BigDecimal conversion unnecessarily enforces the scale

[Piotr Smolinski (Jira)]

Piotr Smolinski created KAFKA-9067:
--

 Summary: BigDecimal conversion unnecessarily enforces the scale 
 Key: KAFKA-9067
 URL: https://issues.apache.org/jira/browse/KAFKA-9067
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.3.0
Reporter: Piotr Smolinski


In Kafka Connect schema framework it is possible to use fixed point decimal 
numbers mapped as logical type Decimal. The type is related to Avro defined 
logical type. When the type is used, the scale value is stored in the schema 
definition (later it might end in Avro schema) and the unscaled value is stored 
as integer of unbounded size.

The problem arises when the decimal value to decode has different scale than 
the one declared in the schema. During conversion to Avro or JSON using 
standard converters the operation fails with DataException.

The proposed solution is to use setScale method to adapt the scale to the 
correct value and provide rounding mode as parameter to the schema:
https://docs.oracle.com/javase/8/docs/api/java/math/BigDecimal.html#setScale-int-java.math.RoundingMode-

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9066) Kafka Connect JMX : source & sink task metrics missing for tasks in failed state

[Jira]

Mikołaj Stefaniak created KAFKA-9066:


 Summary: Kafka Connect JMX : source & sink task metrics missing 
for tasks in failed state
 Key: KAFKA-9066
 URL: https://issues.apache.org/jira/browse/KAFKA-9066
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.1.1
Reporter: Mikołaj Stefaniak


h2. Overview

Kafka Connect exposes various metrics via JMX. Those metrics can be exported 
i.e. by _Prometheus JMX Exporter_ for further processing.
One of crucial attributes is connector's *task status.*

According to official Kafka docs, status is available as +status+ attribute of 
following MBean:
{quote}kafka.connect:type=connector-task-metrics,connector="\{connector}",task="\{task}"status
 - The status of the connector task. One of 'unassigned', 'running', 'paused', 
'failed', or 'destroyed'.
{quote}
h2. Issue

Generally +connector-task-metrics+ are exposed propery for tasks in +running+ 
status but not exposed at all if task is +failed+.

Failed Task *appears* properly with failed status when queried via *REST API*:

 
{code:java}
$ curl -X GET -u 'user:pass' 
http://kafka-connect.mydomain.com/connectors/customerconnector/status

{"name":"customerconnector","connector":{"state":"RUNNING","worker_id":"kafka-connect.mydomain.com:8080"},"tasks":[{"id":0,"state":"FAILED","worker_id":"kafka-connect.mydomain.com:8080","trace":"org.apache.kafka.connect.errors.ConnectException:
 Received DML 'DELETE FROM mysql.rds_sysinfo .."}],"type":"source"}
$ {code}
 

Failed Task *doesn't appear* as bean with +connector-task-metrics+ type when 
queried via *JMX*:

 
{code:java}
$ echo "beans -d kafka.connect" | java -jar 
target/jmxterm-1.1.0-SNAPSHOT-uber.jar -l localhost:8081 -n -v silent | grep 
connector=customerconnector
kafka.connect:connector=customerconnector,task=0,type=task-error-metricskafka.connect:connector=customerconnector,type=connector-metrics
$
{code}
h2. Expected result

It is expected, that bean with +connector-task-metrics+ type will appear also 
for tasks that failed.
Below is example of how beans are properly registered for tasks in Running 
state:

 
{code:java}
$ echo "get -b 
kafka.connect:connector=sinkConsentSubscription-1000,task=0,type=connector-task-metrics
 status" | java -jar target/jmxterm-1.1.0-SNAPSHOT-ube r.jar -l localhost:8081 
-n -v silent
status = running;
$
{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] 2.3.1 RC2

[David Arthur]

We found a few more critical issues and so have decided to do one more RC
for 2.3.1. Please review the release notes:
https://home.apache.org/~davidarthur/kafka-2.3.1-rc2/RELEASE_NOTES.html


*** Please download, test and vote by Tuesday, October 22, 9pm PDT


Kafka's KEYS file containing PGP keys we use to sign the release:

https://kafka.apache.org/KEYS


* Release artifacts to be voted upon (source and binary):

https://home.apache.org/~davidarthur/kafka-2.3.1-rc2/


* Maven artifacts to be voted upon:

https://repository.apache.org/content/groups/staging/org/apache/kafka/


* Javadoc:

https://home.apache.org/~davidarthur/kafka-2.3.1-rc2/javadoc/


* Tag to be voted upon (off 2.3 branch) is the 2.3.1 tag:

https://github.com/apache/kafka/releases/tag/2.3.1-rc2


* Documentation:

https://kafka.apache.org/23/documentation.html


* Protocol:

https://kafka.apache.org/23/protocol.html


* Successful Jenkins builds to follow


Thanks!

David

Build failed in Jenkins: kafka-2.4-jdk8 #26

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-8962; Use least loaded node for AdminClient#describeTopics (#7421)


--
[...truncated 2.69 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
STARTED

Build failed in Jenkins: kafka-trunk-jdk11 #892

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-8834; Add reassignment metrics and distinguish URPs (KIP-352)

[jason] KAFKA-8962; Use least loaded node for AdminClient#describeTopics (#7421)


--
[...truncated 2.70 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 

[jira] [Created] (KAFKA-9065) Loading offsets and group metadata loops forever

[David Jacot (Jira)]

David Jacot created KAFKA-9065:
--

 Summary: Loading offsets and group metadata loops forever
 Key: KAFKA-9065
 URL: https://issues.apache.org/jira/browse/KAFKA-9065
 Project: Kafka
  Issue Type: Bug
Reporter: David Jacot


When the metadata manager loads the groups and the offsets of a partition of 
the __consumer-offsets topic, `GroupMetadataManager.doLoadGroupsAndOffsets` 
could loop forever if the start offset of the partition is smaller than the end 
offset and no records are effectively read from the partition.

While the conditions leading to this issue are not clear, I've got the case 
where a partition was having two segments which were both empty in a cluster. 
This could theoretically happen when all the tombstones in the first are 
expired and the second is truncated.

 

As a side effect, the `doLoadGroupsAndOffsets` spins forever, blocks the single 
thread of the scheduler, blocks the loading of all the groups and offsets which 
are after in the queue, and blocks the expiration of the offsets.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk8 #3978

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] KAKFA-8950: Fix KafkaConsumer Fetcher breaking on concurrent 
disconnect

[github] MINOR: AbstractRequestResponse should be an interface (#7513)

[jason] KAFKA-9004; Prevent older clients from fetching from a follower (#7531)

[github] MINOR: Upgrade zk to 3.5.6 (#7544)

[matthias] MINOR: Add ability to wait for all instances in an application to be

[matthias] KAFKA-9058: Lift queriable and materialized restrictions on FK Join

[bbejeck] MINOR: Fix JavaDoc warning (#7546)


--
[...truncated 8.05 MB...]

org.apache.kafka.connect.util.ConnectUtilsTest > 
testLookupKafkaClusterIdTimeout PASSED

org.apache.kafka.connect.util.ShutdownableThreadTest > testGracefulShutdown 
STARTED

org.apache.kafka.connect.util.ShutdownableThreadTest > testGracefulShutdown 
PASSED

org.apache.kafka.connect.util.ShutdownableThreadTest > testForcibleShutdown 
STARTED

org.apache.kafka.connect.util.ShutdownableThreadTest > testForcibleShutdown 
PASSED

org.apache.kafka.connect.util.TopicAdminTest > 
shouldNotCreateTopicWhenItAlreadyExists STARTED

org.apache.kafka.connect.util.TopicAdminTest > 
shouldNotCreateTopicWhenItAlreadyExists PASSED

org.apache.kafka.connect.util.TopicAdminTest > 
returnNullWithTopicAuthorizationFailure STARTED

org.apache.kafka.connect.util.TopicAdminTest > 
returnNullWithTopicAuthorizationFailure PASSED

org.apache.kafka.connect.util.TopicAdminTest > 
shouldCreateTopicWhenItDoesNotExist STARTED

org.apache.kafka.connect.util.TopicAdminTest > 
shouldCreateTopicWhenItDoesNotExist PASSED

org.apache.kafka.connect.util.TopicAdminTest > 
shouldReturnFalseWhenSuppliedNullTopicDescription STARTED

org.apache.kafka.connect.util.TopicAdminTest > 
shouldReturnFalseWhenSuppliedNullTopicDescription PASSED

org.apache.kafka.connect.util.TopicAdminTest > returnNullWithApiVersionMismatch 
STARTED

org.apache.kafka.connect.util.TopicAdminTest > returnNullWithApiVersionMismatch 
PASSED

org.apache.kafka.connect.util.TopicAdminTest > 
shouldCreateOneTopicWhenProvidedMultipleDefinitionsWithSameTopicName STARTED

org.apache.kafka.connect.util.TopicAdminTest > 
shouldCreateOneTopicWhenProvidedMultipleDefinitionsWithSameTopicName PASSED

org.apache.kafka.connect.util.TopicAdminTest > 
returnNullWithClusterAuthorizationFailure STARTED

org.apache.kafka.connect.util.TopicAdminTest > 
returnNullWithClusterAuthorizationFailure PASSED

org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicyTest
 > testPrincipalPlusOtherConfigs STARTED

org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicyTest
 > testPrincipalPlusOtherConfigs PASSED

org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicyTest
 > testPrincipalOnly STARTED

org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicyTest
 > testPrincipalOnly PASSED

org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicyTest
 > testNoOverrides STARTED

org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicyTest
 > testNoOverrides PASSED

org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicyTest
 > testWithOverrides STARTED

org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicyTest
 > testWithOverrides PASSED

org.apache.kafka.connect.integration.SessionedProtocolIntegrationTest > 
ensureInternalEndpointIsSecured STARTED

org.apache.kafka.connect.integration.SessionedProtocolIntegrationTest > 
ensureInternalEndpointIsSecured SKIPPED

org.apache.kafka.connect.integration.ConnectWorkerIntegrationTest > 
testAddAndRemoveWorker STARTED

org.apache.kafka.connect.integration.ConnectWorkerIntegrationTest > 
testAddAndRemoveWorker PASSED

org.apache.kafka.connect.integration.RestExtensionIntegrationTest > 
testRestExtensionApi STARTED

org.apache.kafka.connect.integration.RestExtensionIntegrationTest > 
testRestExtensionApi PASSED

org.apache.kafka.connect.integration.StartAndStopLatchTest > 
shouldReturnFalseWhenAwaitingForDependentLatchToComplete STARTED

org.apache.kafka.connect.integration.StartAndStopLatchTest > 
shouldReturnFalseWhenAwaitingForDependentLatchToComplete PASSED

org.apache.kafka.connect.integration.StartAndStopLatchTest > 
shouldReturnTrueWhenAwaitingForStartAndStopAndDependentLatch STARTED

org.apache.kafka.connect.integration.StartAndStopLatchTest > 
shouldReturnTrueWhenAwaitingForStartAndStopAndDependentLatch PASSED

org.apache.kafka.connect.integration.StartAndStopLatchTest > 
shouldReturnFalseWhenAwaitingForStartToNeverComplete STARTED

org.apache.kafka.connect.integration.StartAndStopLatchTest > 
shouldReturnFalseWhenAwaitingForStartToNeverComplete PASSED

org.apache.kafka.connect.integration.StartAndStopLatchTest > 
shouldReturnFalseWhenAwaitingForStopToNeverComplete STARTED

[jira] [Resolved] (KAFKA-8962) KafkaAdminClient#describeTopics always goes through the controller

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8962.

Fix Version/s: 2.4.0
   Resolution: Fixed

> KafkaAdminClient#describeTopics always goes through the controller
> --
>
> Key: KAFKA-8962
> URL: https://issues.apache.org/jira/browse/KAFKA-8962
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dhruvil Shah
>Priority: Major
> Fix For: 2.4.0
>
>
> KafkaAdminClient#describeTopic makes a MetadataRequest against the 
> controller. We should consider routing the request to any broker in the 
> cluster using `LeastLoadedNodeProvider` instead, so that we don't overwhelm 
> the controller with these requests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)