Build failed in Jenkins: kafka-trunk-jdk11 #966

[Apache Jenkins Server]

See 


Changes:

[cmccabe] [MINOR] allow additional JVM args in KafkaService (#7297)

[jason] MINOR: Small logging fixes in AbstractCoordinator (#7230)

[github] KAFKA-9198; Complete purgatory operations on receiving StopReplica


--
[...truncated 4.62 MB...]

kafka.tools.CustomDeserializerTest > checkDeserializerTopicIsNotNull PASSED

kafka.tools.MirrorMakerTest > 
testDefaultMirrorMakerMessageHandlerWithNoTimestampInSourceMessage STARTED

kafka.tools.MirrorMakerTest > 
testDefaultMirrorMakerMessageHandlerWithNoTimestampInSourceMessage PASSED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandler STARTED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandler PASSED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandlerWithHeaders 
STARTED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandlerWithHeaders 
PASSED

kafka.tools.ConsumerPerformanceTest > testDetailedHeaderMatchBody STARTED

kafka.tools.ConsumerPerformanceTest > testDetailedHeaderMatchBody PASSED

kafka.tools.ConsumerPerformanceTest > testConfigWithUnrecognizedOption STARTED

kafka.tools.ConsumerPerformanceTest > testConfigWithUnrecognizedOption PASSED

kafka.tools.ConsumerPerformanceTest > testConfig STARTED

kafka.tools.ConsumerPerformanceTest > testConfig PASSED

kafka.tools.ConsumerPerformanceTest > testNonDetailedHeaderMatchBody STARTED

kafka.tools.ConsumerPerformanceTest > testNonDetailedHeaderMatchBody PASSED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit STARTED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseGroupIdFromBeginningGivenTogether 
STARTED

kafka.tools.ConsoleConsumerTest > shouldParseGroupIdFromBeginningGivenTogether 
PASSED

kafka.tools.ConsoleConsumerTest > shouldExitOnOffsetWithoutPartition STARTED

kafka.tools.ConsoleConsumerTest > shouldExitOnOffsetWithoutPartition PASSED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails STARTED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginning STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginning PASSED

kafka.tools.ConsoleConsumerTest > shouldResetUnConsumedOffsetsBeforeExit STARTED

kafka.tools.ConsoleConsumerTest > shouldResetUnConsumedOffsetsBeforeExit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile STARTED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidConsumerConfigWithAutoOffsetResetLatest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidConsumerConfigWithAutoOffsetResetLatest PASSED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
STARTED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning 
STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning PASSED

kafka.tools.ConsoleConsumerTest > shouldExitOnGroupIdAndPartitionGivenTogether 
STARTED

kafka.tools.ConsoleConsumerTest > shouldExitOnGroupIdAndPartitionGivenTogether 
PASSED

kafka.tools.ConsoleConsumerTest > shouldExitOnUnrecognizedNewConsumerOption 
STARTED

kafka.tools.ConsoleConsumerTest > shouldExitOnUnrecognizedNewConsumerOption 
PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidSimpleConsumerValidConfigWithNumericOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidSimpleConsumerValidConfigWithNumericOffset PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidConsumerConfigWithAutoOffsetResetEarliest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidConsumerConfigWithAutoOffsetResetEarliest PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidSimpleConsumerValidConfigWithStringOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidSimpleConsumerValidConfigWithStringOffset PASSED

kafka.tools.ConsoleConsumerTest > 
testCustomPropertyShouldBePassedToConfigureMethod STARTED

kafka.tools.ConsoleConsumerTest > 
testCustomPropertyShouldBePassedToConfigureMethod PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidConsumerValidConfig PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidConsumerConfigWithNoOffsetReset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidConsumerConfigWithNoOffsetReset PASSED

kafka.tools.ReplicaVerificationToolTest > testReplicaBufferVerifyChecksum 
STARTED

kafka.tools.ReplicaVerificationToolTest > 

[jira] [Created] (KAFKA-9210) kafka stream loss data

[panpan.liu (Jira)]

panpan.liu created KAFKA-9210:
-

 Summary: kafka stream loss data
 Key: KAFKA-9210
 URL: https://issues.apache.org/jira/browse/KAFKA-9210
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.0.1
Reporter: panpan.liu


```

Partitions [flash-app-xmc-worker-share-store-minute-repartition-1]Partitions 
[flash-app-xmc-worker-share-store-minute-repartition-1] for changelog 
flash-app-xmc-worker-share-store-minute-changelog-12019-11-19 05:50:49.816|WARN 
|flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|101|stream-thread
 [flash-client-xmc-StreamThread-3] Restoring StreamTasks failed. Deleting 
StreamTasks stores to recreate from 
scratch.org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets 
out of range with no configured reset policy for partitions: 
\{flash-app-xmc-worker-share-store-minute-changelog-1=6128684} at 
org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:987)2019-11-19
 05:50:49.817|INFO 
|flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|105|stream-thread
 [flash-client-xmc-StreamThread-3] Reinitializing StreamTask TaskId: 10_1 
ProcessorTopology: KSTREAM-SOURCE-70: topics: 
[flash-app-xmc-worker-share-store-minute-repartition] children: 
[KSTREAM-AGGREGATE-67] KSTREAM-AGGREGATE-67: states: 
[worker-share-store-minute] children: [KTABLE-TOSTREAM-71] 
KTABLE-TOSTREAM-71: children: [KSTREAM-SINK-72] 
KSTREAM-SINK-72: topic: 
StaticTopicNameExtractor(xmc-worker-share-minute)Partitions 
[flash-app-xmc-worker-share-store-minute-repartition-1] for changelog 
flash-app-xmc-worker-share-store-minute-changelog-12019-11-19 05:50:49.842|WARN 
|flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|101|stream-thread
 [flash-client-xmc-StreamThread-3] Restoring StreamTasks failed. Deleting 
StreamTasks stores to recreate from 
scratch.org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets 
out of range with no configured reset policy for partitions: 
\{flash-app-xmc-worker-share-store-minute-changelog-1=6128684} at 
org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:987)2019-11-19
 05:50:49.842|INFO 
|flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|105|stream-thread
 [flash-client-xmc-StreamThread-3] Reinitializing StreamTask TaskId: 10_1 
ProcessorTopology: KSTREAM-SOURCE-70: topics: 
[flash-app-xmc-worker-share-store-minute-repartition] children: 
[KSTREAM-AGGREGATE-67] KSTREAM-AGGREGATE-67: states: 
[worker-share-store-minute] children: [KTABLE-TOSTREAM-71] 
KTABLE-TOSTREAM-71: children: [KSTREAM-SINK-72] 
KSTREAM-SINK-72: topic: 
StaticTopicNameExtractor(xmc-worker-share-minute)Partitions 
[flash-app-xmc-worker-share-store-minute-repartition-1] for changelog 
flash-app-xmc-worker-share-store-minute-changelog-12019-11-19 05:50:49.905|WARN 
|flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|101|stream-thread
 [flash-client-xmc-StreamThread-3] Restoring StreamTasks failed. Deleting 
StreamTasks stores to recreate from 
scratch.org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets 
out of range with no configured reset policy for partitions: 
\{flash-app-xmc-worker-share-store-minute-changelog-1=6128684} at 
org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:987)2019-11-19
 05:50:49.906|INFO 
|flash-client-xmc-StreamThread-3|o.a.k.s.p.i.StoreChangelogReader|105|stream-thread
 [flash-client-xmc-StreamThread-3] Reinitializing StreamTask TaskId: 10_1 
ProcessorTopology: KSTREAM-SOURCE-70: topics: 
[flash-app-xmc-worker-share-store-minute-repartition] children: 
[KSTREAM-AGGREGATE-67] KSTREAM-AGGREGATE-67: states: 
[worker-share-store-minute]

```



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-2.4-jdk8 #78

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Fetch only from leader should be respected in purgatory (#7650)

[jason] KAFKA-9198; Complete purgatory operations on receiving StopReplica


--
[...truncated 5.45 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

Re: [VOTE] KIP-518: Allow listing consumer groups per state

[Kevin Lu]

+1 (non-binding)

I can see this being useful. Thanks for the KIP!

Regards,
Kevin

On Mon, Nov 18, 2019 at 4:09 AM Mickael Maison 
wrote:

> Hi all,
>
> I'd like to start the vote on KIP-518:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-518%3A+Allow+listing+consumer+groups+per+state
>
> Thanks
>

Build failed in Jenkins: kafka-trunk-jdk8 #4052

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Small logging fixes in AbstractCoordinator (#7230)

[github] KAFKA-9198; Complete purgatory operations on receiving StopReplica

[matthias] HOTFIX: Fix unit tests that failed when executed from IDE (#7707)

[github] MINOR: Remove explicit version checks in getErrorResponse methods

[github] KAFKA-8981 Add rate limiting to NetworkDegradeSpec (#7446)


--
[...truncated 2.74 MB...]
org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
PASSED

> Task :streams:upgrade-system-tests-0101:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:testClasses
> Task :streams:upgrade-system-tests-0101:checkstyleTest
> Task :streams:upgrade-system-tests-0101:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:test
> Task :streams:upgrade-system-tests-0102:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0102:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0102:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0102:checkstyleMain 

Re: ClientsMetricsTest.shouldAddCommitIdMetric() failed on RC0 ...

[Eric Lalonde]

Bruno,

I tested using the 2.4.0 release candidate 0 artifacts. These were uploaded
as part of seeking the open Kafka community feedback.

On Mon, Nov 18, 2019 at 12:24 PM Bruno Cadonna  wrote:

> Hi Vahid and Eric,
>
> Thank you for your input.
>
> I suppose you both used the archive of the release candidate and did
> not checkout the tag from the git repository.
>
> I found the issue. The archive misses the .git directory that is
> needed for the unit test to pass.
>
> Opened the following PR to fix it:
> https://github.com/apache/kafka/pull/7707
>
> Best,
> Bruno
>

Re: [DISCUSS] KIP-501 Avoid out-of-sync or offline partitions when follower fetch requests not processed in time

[Satish Duggana]

Hi Jason,
Thanks for looking into the KIP. Apologies for my late reply.
Increasing replica max lag to 30-45 secs did not help as we observed
that a few fetch requests took more than 1-2 minutes. We do not want
to increase further as it increases upper bound on commit latency. We
have strict SLAs on some of the clusters on end to end(producer to
consumer) latency. This proposal improves the availability of
partitions when followers are trying their best to be insync even when
leaders are slow in processing those requests.
I have updated the KIP to have a single config for giving backward
compatibility and I guess this config is more comprehensible than
earlier. But I believe there is no need to have config because the
suggested proposal in the KIP is an enhancement to the existing
behavior. Please let me know your comments.

Thanks,
Satish.

On Thu, Nov 14, 2019 at 10:57 AM Jason Gustafson  wrote:
>
> Hi Satish,
>
> Thanks for the KIP.  I'm wondering how much of this problem can be
> addressed just by increasing the replication max lag? That was one of the
> purposes of KIP-537 (the default increased from 10s to 30s). Also, the new
> configurations seem quite low level. I think they will be hard for users to
> understand (even reading through a couple times I'm not sure I understand
> them fully). I think if there's a way to improve this behavior without
> requiring any new configurations, it would be much more attractive.
>
> Best,
> Jason
>
> On Wed, Nov 6, 2019 at 8:14 AM Satish Duggana 
> wrote:
>
> > Hi Dhruvil,
> > Thanks for looking into the KIP.
> >
> > 10. I have an initial sketch of the KIP-500 in commit[a] which
> > discusses tracking the pending fetch requests. Tracking is not done in
> > Partition#readRecords because if it takes longer in reading any of the
> > partitions then we do not want any of the replicas of this fetch
> > request to go out of sync.
> >
> > 11. I think `Replica` class should be thread-safe to handle the remote
> > scenario of concurrent requests running for a follower replica. Or I
> > may be missing something here. This is a separate issue from KIP-500.
> > I will file a separate JIRA to discuss that issue.
> >
> > a -
> > https://github.com/satishd/kafka/commit/c69b525abe8f6aad5059236076a003cdec4c4eb7
> >
> > Thanks,
> > Satish.
> >
> > On Tue, Oct 29, 2019 at 10:57 AM Dhruvil Shah 
> > wrote:
> > >
> > > Hi Satish,
> > >
> > > Thanks for the KIP, those seems very useful. Could you elaborate on how
> > > pending fetch requests are tracked?
> > >
> > > Thanks,
> > > Dhruvil
> > >
> > > On Mon, Oct 28, 2019 at 9:43 PM Satish Duggana  > >
> > > wrote:
> > >
> > > > Hi All,
> > > > I wrote a short KIP about avoiding out-of-sync or offline partitions
> > > > when follower fetch requests are not processed in time by the leader
> > > > replica.
> > > > KIP-501 is located at https://s.apache.org/jhbpn
> > > >
> > > > Please take a look, I would like to hear your feedback and suggestions.
> > > >
> > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-8733
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> >

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

Hi Jun,
Thanks for your detailed review and comments.

>40. Local segment metadata storage: The KIP makes the assumption that the
metadata for the archived log segments are cached locally in every broker
and provides a specific implementation for the local storage in the
framework. We probably should discuss this more. For example, some tier
storage providers may not want to cache the metadata locally and just rely
upon a remote key/value store if such a store is already present. If a
local store is used, there could be different ways of implementing it
(e.g., based on customized local files, an embedded local store like
RocksDB, etc). An alternative of designing this is to just provide an
interface for retrieving the tier segment metadata and leave the details of
how to get the metadata outside of the framework.

I am fine with giving a way for RSM implementor to handle remote log
metadata. But we should give a default implementation if any RSM
implementers want to reuse that. The default implementation can be
storing them locally as it is mentioned in the KIP.

>41. RemoteStorageManager interface and the usage of the interface in the
framework: I am not sure if the interface is general enough.  For example,
it seems that RemoteLogIndexEntry is tied to a specific way of storing the
metadata in remote storage. The framework uses listRemoteSegments() api in
a pull-based approach. However, in some other implementations, a push-based
approach may be more preferred. I don’t have a concrete proposal yet. But,
it would be useful to give this area some more thoughts and see if we can
make the interface more general.

RemoteLogIndexEntry is aligned with record batch representation and it
also gives a generalized representation through RDI about the location
of that batch in the remote storage. If there are use cases to
represent them in a different way then we can have an interface and
refactor the current  RemoteLogIndexEntry as the default
implementation.
listRemoteSegments() API is to get metadata about a specific topic
partition’s remote log storage. We thought  RemoteLogManager should do
most of the heavy lifting as much as it can and it should use
RemoteStorageManager whenever it needs to retrieve remote log
metadata/data. We can start with this approach in the initial version.
If there are valid use cases to have push based mechanism we can add
them later.


>42. In the diagram, the RemoteLogManager is side by side with LogManager.
This KIP only discussed how the fetch request is handled between the two
layer. However, we should also consider how other requests that touch the
log can be handled. e.g., list offsets by timestamp, delete records, etc.
Also, in this model, it's not clear which component is responsible for
managing the log start offset. It seems that the log start offset could be
changed by both RemoteLogManager and LogManager.

Sure, we will add more details in the KIP about how different request
APIs which touch the log are handled.
With tiered storage, log will have local-log-start-offset,
remote-log-start-offset and effective-log-start-offset.
Existing log-start-offset field is effective-log-start-offset of the Log.
effective-log-start-offset = if(remote-log exists)
remote-log-start-offset else local-log-start-offset.
Log still manages log-start-offset but it can be updated by
RemoteLogManager if tiering is enabled.


>43. There are quite a few existing features not covered by the KIP. It
would be useful to discuss each of those.
>43.1 I won’t say that compacted topics are rarely used and always small.
For example, KStreams uses compacted topics for storing the states and
sometimes the size of the topic could be large. While it might be ok to not
support compacted topics initially, it would be useful to have a high level
idea on how this might be supported down the road so that we don’t have to
make incompatible API changes in the future.

As you know, any new APIs will evolve over the next couple of
versions, they may even be incompatible till stabilized. But we will
have the new APIs thinking through the possible usecases.
We can discuss a high level idea on how compact topics can be
supported but this is a lower priority for now.

>43.2 We need to discuss how EOS is supported. In particular, how is the
producer state integrated with the remote storage.

Right, EOS needs producer state snapshots of the log segments. These
snapshots can be maintained in remote storage like offset and time
indexes. I will update the KIP with the details.

>43.3 Now that KIP-392 (allow consumers to fetch from closest replica) is
implemented, we need to discuss how reading from a follower replica is
supported with tier storage.

We plan to support consumer fetch requests on follower replicas with
remote log segments. Remote log contains only the committed
records(till log-stable-offset), This constraint allows us to support
the ask here. I will update the KIP to make it clear that this is
supported.

>43.4 We 

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

[deng ziming]

hi, I reviewed the current code, the ProduceMetadata maintains an expiry
threshold for every topic, every time when we write to a topic we will set
the expiry time to -1 to indicate it should be updated, this does work to
reduce the size of the topic working set, but the producer will continue
fetching metadata for these topics in every metadata request for the full
expiry duration.

and we can improve the situation by 2 means:
1. we maintain a refresh threshold for every topic which is for example
0.8 * expiry_threshold, and when we send `MetadataRequest` to brokers we
just request unknownLeaderTopics + unknownPartitionTopics + topics
reach refresh threshold.
2. we don't invoke KafkaProducer#waitOnMetadata when we call
KafkaProducer#send because of we just send data to RecordAccumulator, and
before we send data to brokers we will invoke RecordAccumulator#ready(), so
we can only invoke waitOnMetadata to block when (number topics
reach refresh threshold)>(number of all known topics)*0.2.

I think the above 2 ways are enough to solve the current problem.

On Tue, Nov 19, 2019 at 3:20 AM Colin McCabe  wrote:

> On Mon, Nov 18, 2019, at 10:05, Brian Byrne wrote:
> > On Fri, Nov 15, 2019 at 5:08 PM Colin McCabe  wrote:
> >
> > > Two seconds doesn't seem like a reasonable amount of time to leave for
> the
> > > metadata fetch.  Fetching halfway through the expiration period seems
> more
> > > reasonable.  It also doesn't require us to create a new configuration
> key,
> > > which is nice.
> > >
> > > Another option is to just do the metadata fetch every
> metadata.max.age.ms,
> > > but not expire the topic until we can't fetch the metadata for 2 *
> > > metadata.max.age.ms.
> > >
> >
> > I'd expect two seconds to be reasonable in the common case. Keep in mind
> > that this doesn't affect correctness, and a control operation returning
> > cached metadata should be on the order of milliseconds.
> >
>
> Hi Brian,
>
> Thanks again for the KIP.
>
> I think the issue here is not the common case, but the uncommon case where
> the metadata fetch takes longer than expected.  In that case, we don't want
> to be in the position of having our metadata expire because we waited too
> long to renew it.
>
> This is one reason why I think that the metadata expiration time should be
> longer than the metadata refresh time.  In fact, it might be worth having
> two separate configuration keys for these two values.  I could imagine a
> user who is having trouble with metadata expiration wanting to increase the
> metadata expiration time, but without increasing the metadata refresh
> period.  In a sense, the metadata expiration time is like the ZK session
> expiration time.  You might want to turn it up if the cluster is
> experiencing load spikes.
>
> >
> > But to the general
> > point, defining the algorithm would mean enforcing it to fair accuracy,
> > whereas if the suggestion is that it'll be performed at a reasonable
> time,
> > it allows for batching and other optimizations. Perhaps I shouldn't be
> > regarding what's defined in a KIP to be contractual in these cases, but
> you
> > could consider a first implementation to collect topics whose metadata
> has
> > exceeded (metadata.max.age.ms / 2), and sending the batch once a
> > constituent topic's metadata is near the expiry, or a sufficient number
> of
> > topics have been collected (10? 100? 1000?).
> >
>
> I'm concerned that if we change the metadata caching strategy without
> discussing it first, it may improve certain workloads but make others
> worse.  We need to be concrete about what the proposed strategy is so that
> we can really evaluate it.
>
> >
> >
> > > We should be specific about what happens if the first few metadata
> fetches
> > > fail.  Do we use exponential backoff to decide when to resend?  It
> seems
> > > like we really should, for all the usual reasons (reduce the load on
> > > brokers, ride out temporary service disruptions, etc.)  Maybe we could
> have
> > > an exponential retry backoff for each broker (in other words, we
> should try
> > > to contact a different broker before applying the backoff.)  I think
> this
> > > already sort of happens with the disconnect timeout, but we might need
> a
> > > more general solution.
> > >
> >
> > I don't plan to change this behavior. Currently it retries after a fixed
> > value of 'retry.backoff.ms' (defaults to 100 ms). It's possible that
> > different brokers are attempted, but I haven't dug into it.
> >
>
> I think it's critical to understand what the current behavior is before we
> try to change it.  The difference between retrying the same broker and
> trying a different one has a large impact it has on cluster load and
> latency.  For what it's worth, I believe the behavior is the second one,
> but it has been a while since I checked.  Let's figure this out.
>
> >
> > > Thanks for the clarification.  Fully asynchronous is the way to go, I
> > > agree.  I'm having trouble understanding how 

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

[Gwen Shapira]

General mechanism for fetching metrics via RPC sounds like a good
idea. Especially since Kafka has clients in many languages, but
support for JMX is not wide-spread outside the Java ecosystem.
Command-line script seems like a bad fit for use-cases where you want
the clients themselves to use metric information about the broker to
drive their behavior. Or even for something like a Kubernetes
controller.

Gwen

On Mon, Nov 18, 2019 at 4:41 PM Colin McCabe  wrote:
>
> Hi,
>
> In general, metrics are a "broker status API" telling you important things 
> about the state of the broker, its performance, etc. etc., right?  What 
> argument is there for creating a separate API for this particular metric?  If 
> you argue that JMX is not convenient, it seems like that would also apply to 
> any other metric we expose.
>
> Perhaps we should just add an easy command-line script for querying the JMX 
> interface?  Or a more general mechanism for fetching metrics via Kafka RPC.
>
> best,
> Colin
>
>
> On Mon, Nov 18, 2019, at 10:54, Jason Gustafson wrote:
> > Hi Noa,
> >
> > Re; uptime. Sure, it was just a suggestion. However, we should be clear
> > that there are actually two timestamps at play. Your initial proposal
> > suggested using the timestamp from the registration znode. However, this
> > changes each time the broker loses its session. It does not reflect the
> > actual broker start time, which seems to be what you are interested in. So
> > possibly it is worth exposing both the true broker start time as well as
> > its session creation time.
> >
> > -Jason
> >
> >
> >
> > On Mon, Nov 18, 2019 at 7:48 AM Ismael Juma  wrote:
> >
> > > Hi Noa,
> > >
> > > I understand the desire for batching. However, once you do that, you 
> > > either
> > > need request forwarding or broker metadata propagation. It's worth
> > > exploring, but I suspect you will end up with most of the changes needed 
> > > by
> > > the original proposal, in that case.
> > >
> > > Ismael
> > >
> > > On Mon, Nov 18, 2019 at 7:07 AM Noa Resare  wrote:
> > >
> > > > Hi Jason & Gwen,
> > > >
> > > > Sure, this would solve our use case. I do have two questions, however:
> > > >
> > > > Moving from start time to uptime in theory would neatly side step the
> > > > clock skew problem,
> > > > but in practice I’m not sure it helps that much as the straightforward
> > > way
> > > > of going about
> > > > implementing this by returning (now - startTime) would break when the
> > > > clock on the broker
> > > > is adjusted. So, I think that startTime is more straight forward and
> > > > doesn’t hide the fact that
> > > > the system clock is sometimes off.
> > > >
> > > > Another thing I think would be useful is to build support for requesting
> > > > multiple (or all?)
> > > > brokers in a single request at the start. Having the request hold a set
> > > of
> > > > brokerIds
> > > > and return a set of brokers in the response, adding a brokerId field to
> > > > identify which
> > > > broker a specific broker status response is associated with seems
> > > straight
> > > > forward.
> > > >
> > > > I’ll start writing a proposal so that we have something concrete to
> > > > discuss.
> > > >
> > > > /noa
> > > >
> > > > > On 13 Nov 2019, at 16:43, Jason Gustafson  wrote:
> > > > >
> > > > > Ni Noa,
> > > > >
> > > > > Thanks for the KIP. I agree with the concerns about Metadata bloat. In
> > > > > fact, I have wanted a BrokerStatus API for a while now. Perhaps this
> > > is a
> > > > > good excuse to introduce it. I was thinking something like this:
> > > > >
> > > > > BrokerStatusRequest => BrokerId
> > > > >
> > > > > BrokerStatusResponse =>
> > > > >  Listeners => [ListenerName Host Port]
> > > > >  RackId
> > > > >  BrokerEpoch
> > > > >  BrokerState => {ACTIVE, RECOVERING, SHUTTING_DOWN}
> > > > >  UpTime
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > > On Tue, Nov 5, 2019 at 12:25 PM Noa Resare  wrote:
> > > > >
> > > > >> I agree with that. And looking at the MetadataResponse fields it 
> > > > >> seems
> > > > >> there has been some feature creep already here. Does the client use
> > > rack
> > > > >> information, for example?
> > > > >>
> > > > >> I guess one could do this either by introducing a new leaner message
> > > > pair,
> > > > >> used specifically enable client operation, and use
> > > > >> MetadataRequest/MetadataResponse for describeCluster() or one could
> > > > shrink
> > > > >> MetadataRequest/MetadataResponse and introduce a new more fully
> > > featured
> > > > >> message pair for the other stuff. I would be happy to spend some time
> > > > >> looking into implementing this if there is an interest.
> > > > >>
> > > > >> /noa
> > > > >>
> > > > >>> On 5 Nov 2019, at 15:43, Gwen Shapira  wrote:
> > > > >>>
> > > > >>> It isn't just about saving space. It increases complexity to default
> > > to
> > > > >>> always sharing a bit of information that is really only needed in a
> > > > >> 

Build failed in Jenkins: kafka-trunk-jdk8 #4051

[Apache Jenkins Server]

See 


Changes:

[cmccabe] [MINOR] allow additional JVM args in KafkaService (#7297)


--
[...truncated 5.54 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled 

[jira] [Created] (KAFKA-9209) Avoid sending unnecessary offset updates from consumer after KIP-211

[Michael Bingham (Jira)]

Michael Bingham created KAFKA-9209:
--

 Summary: Avoid sending unnecessary offset updates from consumer 
after KIP-211
 Key: KAFKA-9209
 URL: https://issues.apache.org/jira/browse/KAFKA-9209
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Affects Versions: 2.3.0
Reporter: Michael Bingham


With KIP-211 
([https://cwiki.apache.org/confluence/display/KAFKA/KIP-211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets]),
 offsets will no longer expire as long as the consumer group is active.

If the consumer has {{enable.auto.commit=true}}, and if no new events are 
arriving on subscribed partition(s), the consumer still sends offsets 
(unchanged) to the group coordinator just to keep them from expiring. This is 
no longer necessary, and an optimization could potentially be implemented to 
only send offsets with auto commit when there are actual updates to be made 
(i.e., when new events have been processed). 

This would require detecting whether the broker supports the new expiration 
semantics in KIP-211, and only apply the optimization when it does.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [kafka-clients] [VOTE] 2.4.0 RC0

[Matthias J. Sax]

Fix via https://github.com/apache/kafka/pull/7707

Thanks for the quick fix Bruno!



On 11/18/19 11:02 AM, Eric Lalonde wrote:
> This test has been failing when executed from the command line. I have not 
> run this test in an IDE. 
> 
>> On Nov 18, 2019, at 6:16 AM, Bruno Cadonna  wrote:
>>
>> Hi,
>>
>> ClientMetricsTest.shouldAddCommitIdMetric should only fail if executed
>> from an IDE. The test fails because the test expects a file on the
>> class path which is not there when the test is executed from the IDE,
>> but is there when the test is executed from gradle. I will try to fix
>> the test so that it can also be executed from the IDE.
>>
>> Best,
>> Bruno
>>
>> On Mon, Nov 18, 2019 at 6:51 AM Vahid Hashemian
>> mailto:vahid.hashem...@gmail.com>> wrote:
>>>
>>> Thanks Manikumar for managing this release. Looking forward to it.
>>>
>>> I built binary from the source and was able to successfully run the 
>>> quickstarts.
>>>
>>> However, this streams unit test also fails for me constantly:
>>>
>>> ClientMetricsTest. shouldAddCommitIdMetric
>>>
>>> java.lang.AssertionError:
>>>  Unexpected method call 
>>> StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The version 
>>> control commit ID of the Kafka Streams client", INFO, "unknown"):
>>>StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The 
>>> version control commit ID of the Kafka Streams client", INFO, 
>>> and(not("unknown"), notNull())): expected: 1, actual: 0
>>> at 
>>> org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:44)
>>> at 
>>> org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:101)
>>> at 
>>> org.easymock.internal.ClassProxyFactory$MockMethodInterceptor.intercept(ClassProxyFactory.java:97)
>>>...
>>>
>>> Thanks,
>>> --Vahid
>>>
>>> On Thu, Nov 14, 2019 at 10:21 AM Manikumar  
>>> wrote:

 Hello Kafka users, developers and client-developers,

 This is the first candidate for release of Apache Kafka 2.4.0.
 There is work in progress for couple blockers PRs. I am publishing RC0 to 
 avoid further delays in testing the release.

 This release includes many new features, including:
 - Allow consumers to fetch from closest replica
 - Support for incremental cooperative rebalancing to the consumer 
 rebalance protocol
 - MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication 
 engine
 - New Java authorizer Interface
 - Support for  non-key joining in KTable
 - Administrative API for replica reassignment
 - Sticky partitioner
 - Return topic metadata and configs in CreateTopics response
 - Securing Internal connect REST endpoints
 - API to delete consumer offsets and expose it via the AdminClient.

 Release notes for the 2.4.0 release:
 https://home.apache.org/~manikumar/kafka-2.4.0-rc0/RELEASE_NOTES.html

 *** Please download, test  by  Thursday, November 20, 9am PT

 Kafka's KEYS file containing PGP keys we use to sign the release:
 https://kafka.apache.org/KEYS

 * Release artifacts to be voted upon (source and binary):
 https://home.apache.org/~manikumar/kafka-2.4.0-rc0/

 * Maven artifacts to be voted upon:
 https://repository.apache.org/content/groups/staging/org/apache/kafka/

 * Javadoc:
 https://home.apache.org/~manikumar/kafka-2.4.0-rc0/javadoc/

 * Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
 https://github.com/apache/kafka/releases/tag/2.4.0-rc0

 * Documentation:
 https://kafka.apache.org/24/documentation.html

 * Protocol:
 https://kafka.apache.org/24/protocol.html

 Thanks,
 Manikumar

 --
 You received this message because you are subscribed to the Google Groups 
 "kafka-clients" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to kafka-clients+unsubscr...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/kafka-clients/CAMVt_Aw945uqcpisFjZHAR5m8Sidw6hW4ia%2B7%3DjxEfadmBPzcw%40mail.gmail.com.
>>>
>>>
>>>
>>> --
>>>
>>> Thanks!
>>> --Vahid
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "kafka-clients" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to kafka-clients+unsubscr...@googlegroups.com 
>>> .
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/kafka-clients/CAHR2v2mKJtHG6S9P%3Dmw08SxbWjQCowp8cpZNpzr9acW1EcdegQ%40mail.gmail.com
>>>  
>>> .
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

[Colin McCabe]

Hi,

In general, metrics are a "broker status API" telling you important things 
about the state of the broker, its performance, etc. etc., right?  What 
argument is there for creating a separate API for this particular metric?  If 
you argue that JMX is not convenient, it seems like that would also apply to 
any other metric we expose.

Perhaps we should just add an easy command-line script for querying the JMX 
interface?  Or a more general mechanism for fetching metrics via Kafka RPC.

best,
Colin


On Mon, Nov 18, 2019, at 10:54, Jason Gustafson wrote:
> Hi Noa,
> 
> Re; uptime. Sure, it was just a suggestion. However, we should be clear
> that there are actually two timestamps at play. Your initial proposal
> suggested using the timestamp from the registration znode. However, this
> changes each time the broker loses its session. It does not reflect the
> actual broker start time, which seems to be what you are interested in. So
> possibly it is worth exposing both the true broker start time as well as
> its session creation time.
> 
> -Jason
> 
> 
> 
> On Mon, Nov 18, 2019 at 7:48 AM Ismael Juma  wrote:
> 
> > Hi Noa,
> >
> > I understand the desire for batching. However, once you do that, you either
> > need request forwarding or broker metadata propagation. It's worth
> > exploring, but I suspect you will end up with most of the changes needed by
> > the original proposal, in that case.
> >
> > Ismael
> >
> > On Mon, Nov 18, 2019 at 7:07 AM Noa Resare  wrote:
> >
> > > Hi Jason & Gwen,
> > >
> > > Sure, this would solve our use case. I do have two questions, however:
> > >
> > > Moving from start time to uptime in theory would neatly side step the
> > > clock skew problem,
> > > but in practice I’m not sure it helps that much as the straightforward
> > way
> > > of going about
> > > implementing this by returning (now - startTime) would break when the
> > > clock on the broker
> > > is adjusted. So, I think that startTime is more straight forward and
> > > doesn’t hide the fact that
> > > the system clock is sometimes off.
> > >
> > > Another thing I think would be useful is to build support for requesting
> > > multiple (or all?)
> > > brokers in a single request at the start. Having the request hold a set
> > of
> > > brokerIds
> > > and return a set of brokers in the response, adding a brokerId field to
> > > identify which
> > > broker a specific broker status response is associated with seems
> > straight
> > > forward.
> > >
> > > I’ll start writing a proposal so that we have something concrete to
> > > discuss.
> > >
> > > /noa
> > >
> > > > On 13 Nov 2019, at 16:43, Jason Gustafson  wrote:
> > > >
> > > > Ni Noa,
> > > >
> > > > Thanks for the KIP. I agree with the concerns about Metadata bloat. In
> > > > fact, I have wanted a BrokerStatus API for a while now. Perhaps this
> > is a
> > > > good excuse to introduce it. I was thinking something like this:
> > > >
> > > > BrokerStatusRequest => BrokerId
> > > >
> > > > BrokerStatusResponse =>
> > > >  Listeners => [ListenerName Host Port]
> > > >  RackId
> > > >  BrokerEpoch
> > > >  BrokerState => {ACTIVE, RECOVERING, SHUTTING_DOWN}
> > > >  UpTime
> > > >
> > > > What do you think?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Tue, Nov 5, 2019 at 12:25 PM Noa Resare  wrote:
> > > >
> > > >> I agree with that. And looking at the MetadataResponse fields it seems
> > > >> there has been some feature creep already here. Does the client use
> > rack
> > > >> information, for example?
> > > >>
> > > >> I guess one could do this either by introducing a new leaner message
> > > pair,
> > > >> used specifically enable client operation, and use
> > > >> MetadataRequest/MetadataResponse for describeCluster() or one could
> > > shrink
> > > >> MetadataRequest/MetadataResponse and introduce a new more fully
> > featured
> > > >> message pair for the other stuff. I would be happy to spend some time
> > > >> looking into implementing this if there is an interest.
> > > >>
> > > >> /noa
> > > >>
> > > >>> On 5 Nov 2019, at 15:43, Gwen Shapira  wrote:
> > > >>>
> > > >>> It isn't just about saving space. It increases complexity to default
> > to
> > > >>> always sharing a bit of information that is really only needed in a
> > > >> single
> > > >>> use-case.
> > > >>> We avoid doing this as a matter of good protocol design.
> > > >>> Arguably, this should not really piggyback on cluster metadata at
> > all,
> > > >>> since the usage is so different.
> > > >>>
> > > >>> On Tue, Nov 5, 2019, 7:29 AM Noa Resare  wrote:
> > > >>>
> > >  It would certainly be possible to have the field be optional and
> > only
> > >  include it if some flag is set in the DescribeClusterOptions
> > instance
> > >  passed to Admin.describeCluster() that in turn would translate to a
> > > >> boolean
> > >  in MetadataRequest indicating that we are asking for this piece of
> > >  information.
> > > 
> > >  I’m not entirely sure that this extra 

[jira] [Resolved] (KAFKA-9198) StopReplica handler should complete pending purgatory operations

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-9198.

Fix Version/s: 2.4.0
   Resolution: Fixed

The PR has been merged. I meant to change the title to reference KAFKA-8571, 
but forgot about it. So instead I'll just resolve this one as fixed and mark 
the other as a dup. Sorry for the noise.

> StopReplica handler should complete pending purgatory operations
> 
>
> Key: KAFKA-9198
> URL: https://issues.apache.org/jira/browse/KAFKA-9198
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.4.0
>
>
> When a reassignment completes, the current leader may need to be shutdown 
> with a StopReplica request. It may still have fetch/produce requests in 
> purgatory when this happens. We do not have logic currently to force 
> completion of these requests which means they are doomed to eventually 
> timeout. This is mostly an issue for produce requests which use the default 
> request timeout of 30s.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-8571) Not complete delayed produce requests when processing StopReplicaRequest causing high produce latency for acks=all

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8571.

Resolution: Fixed

> Not complete delayed produce requests when processing StopReplicaRequest 
> causing high produce latency for acks=all
> --
>
> Key: KAFKA-8571
> URL: https://issues.apache.org/jira/browse/KAFKA-8571
> Project: Kafka
>  Issue Type: Bug
>Reporter: Zhanxiang (Patrick) Huang
>Assignee: Zhanxiang (Patrick) Huang
>Priority: Major
>
> Currently a broker will only attempt to complete delayed requests upon 
> highwater mark changes and receiving LeaderAndIsrRequest. When a broker 
> receives StopReplicaRequest, it will not try to complete delayed operations 
> including delayed produce for acks=all, which can cause the producer to 
> timeout even though the producer should have attempted to talk to the new 
> leader faster if a NotLeaderForPartition error is sent.
> This can happen during partition reassignment when controller is trying to 
> kick the previous leader out of the replica set. It this case, controller 
> will only send StopReplicaRequest (not LeaderAndIsrRequest) to the previous 
> leader in the replica set shrink phase. Here is an example:
> {noformat}
> During Reassign the replica set of partition A from [B1, B2] to [B2, B3]:
> t0: Controller expands the replica set to [B1, B2, B3]
> t1: B1 receives produce request PR on partition A with acks=all and timetout 
> T. B1 puts PR into the DelayedProducePurgatory with timeout T.
> t2: Controller elects B2 as the new leader and shrinks the replica set fo 
> [B2, B3]. LeaderAndIsrRequests are sent to B2 and B3. StopReplicaRequest is 
> sent to B!.
> t3: B1 receives StopReplicaRequest but doesn't try to comeplete PR.
> If PR cannot be fullfilled by t3, and t1 + T > t3, PR will eventually time 
> out in the purgatory and producer will eventually time out the produce 
> request.{noformat}
> Since it is possible for the leader to receive only a StopReplicaRequest 
> (without receiving any LeaderAndIsrRequest) to leave the replica set, a fix 
> for this issue is to also try to complete delay operations in processing 
> StopReplicaRequest.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (KAFKA-9198) StopReplica handler should complete pending purgatory operations

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson reopened KAFKA-9198:


> StopReplica handler should complete pending purgatory operations
> 
>
> Key: KAFKA-9198
> URL: https://issues.apache.org/jira/browse/KAFKA-9198
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
>
> When a reassignment completes, the current leader may need to be shutdown 
> with a StopReplica request. It may still have fetch/produce requests in 
> purgatory when this happens. We do not have logic currently to force 
> completion of these requests which means they are doomed to eventually 
> timeout. This is mostly an issue for produce requests which use the default 
> request timeout of 30s.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

ClientsMetricsTest.shouldAddCommitIdMetric() failed on RC0 ...

[Bruno Cadonna]

Hi Vahid and Eric,

Thank you for your input.

I suppose you both used the archive of the release candidate and did
not checkout the tag from the git repository.

I found the issue. The archive misses the .git directory that is
needed for the unit test to pass.

Opened the following PR to fix it: https://github.com/apache/kafka/pull/7707

Best,
Bruno

Build failed in Jenkins: kafka-trunk-jdk8 #4050

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Add missing parentheses in docs/streams/tutorial.html (#7696)

[github] KAFKA-9180: Introduce BrokerMetadataCheckpointTest (#7700)


--
[...truncated 2.74 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldStoreAndReturnStateStores STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldStoreAndReturnStateStores PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureOutputRecordsUsingTo STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureOutputRecordsUsingTo PASSED

org.apache.kafka.streams.MockProcessorContextTest > shouldCaptureOutputRecords 
STARTED

org.apache.kafka.streams.MockProcessorContextTest > shouldCaptureOutputRecords 
PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
fullConstructorShouldSetAllExpectedAttributes STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
fullConstructorShouldSetAllExpectedAttributes PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureCommitsAndAllowReset STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureCommitsAndAllowReset PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildName STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildName PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildIndex STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildIndex PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureApplicationAndRecordMetadata STARTED

org.apache.kafka.streams.MockProcessorContextTest > 

[jira] [Created] (KAFKA-9208) Flaky Test SslAdminClientIntegrationTest.testCreatePartitions

[Sophie Blee-Goldman (Jira)]

Sophie Blee-Goldman created KAFKA-9208:
--

 Summary: Flaky Test 
SslAdminClientIntegrationTest.testCreatePartitions
 Key: KAFKA-9208
 URL: https://issues.apache.org/jira/browse/KAFKA-9208
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 2.4.0
Reporter: Sophie Blee-Goldman


Java 8 build failed on 2.4-targeted PR
h3. Stacktrace

java.lang.AssertionError: validateOnly expected:<3> but was:<1> at 
org.junit.Assert.fail(Assert.java:89) at 
org.junit.Assert.failNotEquals(Assert.java:835) at 
org.junit.Assert.assertEquals(Assert.java:647) at 
kafka.api.AdminClientIntegrationTest$$anonfun$testCreatePartitions$6.apply(AdminClientIntegrationTest.scala:625)
 at 
kafka.api.AdminClientIntegrationTest$$anonfun$testCreatePartitions$6.apply(AdminClientIntegrationTest.scala:599)
 at scala.collection.immutable.List.foreach(List.scala:392) at 
kafka.api.AdminClientIntegrationTest.testCreatePartitions(AdminClientIntegrationTest.scala:599)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
 at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
java.lang.Thread.run(Thread.java:748)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9207) Replica Out of Sync as ReplicaFetcher thread is dead

[Xue Liu (Jira)]

Xue Liu created KAFKA-9207:
--

 Summary: Replica Out of Sync as ReplicaFetcher thread is dead
 Key: KAFKA-9207
 URL: https://issues.apache.org/jira/browse/KAFKA-9207
 Project: Kafka
  Issue Type: Bug
  Components: replication
Affects Versions: 2.3.0
Reporter: Xue Liu
 Attachments: Capture.PNG

We sometimes see a replica for a partition is out of sync. When the issue 
happens, it seems that we just lost that replica (would never catch-up), unless 
we restart that broker.

It appears that ReplicaFetcher thread for that partition is dead.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

[Colin McCabe]

On Mon, Nov 18, 2019, at 10:05, Brian Byrne wrote:
> On Fri, Nov 15, 2019 at 5:08 PM Colin McCabe  wrote:
> 
> > Two seconds doesn't seem like a reasonable amount of time to leave for the
> > metadata fetch.  Fetching halfway through the expiration period seems more
> > reasonable.  It also doesn't require us to create a new configuration key,
> > which is nice.
> >
> > Another option is to just do the metadata fetch every metadata.max.age.ms,
> > but not expire the topic until we can't fetch the metadata for 2 *
> > metadata.max.age.ms.
> >
> 
> I'd expect two seconds to be reasonable in the common case. Keep in mind
> that this doesn't affect correctness, and a control operation returning
> cached metadata should be on the order of milliseconds.
>

Hi Brian,

Thanks again for the KIP.

I think the issue here is not the common case, but the uncommon case where the 
metadata fetch takes longer than expected.  In that case, we don't want to be 
in the position of having our metadata expire because we waited too long to 
renew it.

This is one reason why I think that the metadata expiration time should be 
longer than the metadata refresh time.  In fact, it might be worth having two 
separate configuration keys for these two values.  I could imagine a user who 
is having trouble with metadata expiration wanting to increase the metadata 
expiration time, but without increasing the metadata refresh period.  In a 
sense, the metadata expiration time is like the ZK session expiration time.  
You might want to turn it up if the cluster is experiencing load spikes.

>
> But to the general
> point, defining the algorithm would mean enforcing it to fair accuracy,
> whereas if the suggestion is that it'll be performed at a reasonable time,
> it allows for batching and other optimizations. Perhaps I shouldn't be
> regarding what's defined in a KIP to be contractual in these cases, but you
> could consider a first implementation to collect topics whose metadata has
> exceeded (metadata.max.age.ms / 2), and sending the batch once a
> constituent topic's metadata is near the expiry, or a sufficient number of
> topics have been collected (10? 100? 1000?).
>

I'm concerned that if we change the metadata caching strategy without 
discussing it first, it may improve certain workloads but make others worse.  
We need to be concrete about what the proposed strategy is so that we can 
really evaluate it.

> 
> 
> > We should be specific about what happens if the first few metadata fetches
> > fail.  Do we use exponential backoff to decide when to resend?  It seems
> > like we really should, for all the usual reasons (reduce the load on
> > brokers, ride out temporary service disruptions, etc.)  Maybe we could have
> > an exponential retry backoff for each broker (in other words, we should try
> > to contact a different broker before applying the backoff.)  I think this
> > already sort of happens with the disconnect timeout, but we might need a
> > more general solution.
> >
> 
> I don't plan to change this behavior. Currently it retries after a fixed
> value of 'retry.backoff.ms' (defaults to 100 ms). It's possible that
> different brokers are attempted, but I haven't dug into it.
> 

I think it's critical to understand what the current behavior is before we try 
to change it.  The difference between retrying the same broker and trying a 
different one has a large impact it has on cluster load and latency.  For what 
it's worth, I believe the behavior is the second one, but it has been a while 
since I checked.  Let's figure this out.

> 
> > Thanks for the clarification.  Fully asynchronous is the way to go, I
> > agree.  I'm having trouble understanding how timeouts are handled in the
> > KIP.  It seems like if we can't fetch the metadata within the designated
> > metadata timeout, the future / callback should receive a TimeoutException
> > right?  We do not want the send call to be deferred forever if metadata
> > can't be fetched.  Eventually it should fail if it can't be performed.
> >
> > I do think this is something that will have to be mentioned in the
> > compatibility section.  There is some code out there that is probably
> > prepared to handle a timeout exception from the send function, which may
> > need to be updated to check for a timeout from the future or callback.
> >
> 
> Correct, a timeout exception would be delivered in the future. Sure, I can
> add that note to the KIP.
> 

Thanks.

best,
Colin

> 
> 
> > It seems like this is an existing problem.  You may fire off a lot of send
> > calls that get blocked because the broker that is the leader for a certain
> > partition is not responding.  I'm not sure that we need to do anything
> > special here.  On the other hand, we could make the case for a generic "max
> > number of outstanding sends" configuration to prevent surprise OOMs in the
> > existing cases, plus the new one we're adding.  But this feels like a bit
> > of a scope expansion.
> >
> 
> Right, 

[DISCUSS] KIP-547: Extend ConsumerInterceptor to allow modification of Consumer Commits

[Eric Azama]

Hi all,

I'd like to open discussion on KIP-547: Extend ConsumerInterceptor to allow
modification of Consumer Commits

This KIP hopes to enable better access to the Metadata included while
committing offsets.

LINK:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-547%3A+Extend+ConsumerInterceptor+to+allow+modification+of+Consumer+Commits


Thanks,
Eric A.

Re: [kafka-clients] [VOTE] 2.4.0 RC0

[Eric Lalonde]

This test has been failing when executed from the command line. I have not run 
this test in an IDE. 

> On Nov 18, 2019, at 6:16 AM, Bruno Cadonna  wrote:
> 
> Hi,
> 
> ClientMetricsTest.shouldAddCommitIdMetric should only fail if executed
> from an IDE. The test fails because the test expects a file on the
> class path which is not there when the test is executed from the IDE,
> but is there when the test is executed from gradle. I will try to fix
> the test so that it can also be executed from the IDE.
> 
> Best,
> Bruno
> 
> On Mon, Nov 18, 2019 at 6:51 AM Vahid Hashemian
> mailto:vahid.hashem...@gmail.com>> wrote:
>> 
>> Thanks Manikumar for managing this release. Looking forward to it.
>> 
>> I built binary from the source and was able to successfully run the 
>> quickstarts.
>> 
>> However, this streams unit test also fails for me constantly:
>> 
>> ClientMetricsTest. shouldAddCommitIdMetric
>> 
>> java.lang.AssertionError:
>>  Unexpected method call 
>> StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The version 
>> control commit ID of the Kafka Streams client", INFO, "unknown"):
>>StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The 
>> version control commit ID of the Kafka Streams client", INFO, 
>> and(not("unknown"), notNull())): expected: 1, actual: 0
>> at 
>> org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:44)
>> at 
>> org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:101)
>> at 
>> org.easymock.internal.ClassProxyFactory$MockMethodInterceptor.intercept(ClassProxyFactory.java:97)
>>...
>> 
>> Thanks,
>> --Vahid
>> 
>> On Thu, Nov 14, 2019 at 10:21 AM Manikumar  wrote:
>>> 
>>> Hello Kafka users, developers and client-developers,
>>> 
>>> This is the first candidate for release of Apache Kafka 2.4.0.
>>> There is work in progress for couple blockers PRs. I am publishing RC0 to 
>>> avoid further delays in testing the release.
>>> 
>>> This release includes many new features, including:
>>> - Allow consumers to fetch from closest replica
>>> - Support for incremental cooperative rebalancing to the consumer rebalance 
>>> protocol
>>> - MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication 
>>> engine
>>> - New Java authorizer Interface
>>> - Support for  non-key joining in KTable
>>> - Administrative API for replica reassignment
>>> - Sticky partitioner
>>> - Return topic metadata and configs in CreateTopics response
>>> - Securing Internal connect REST endpoints
>>> - API to delete consumer offsets and expose it via the AdminClient.
>>> 
>>> Release notes for the 2.4.0 release:
>>> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/RELEASE_NOTES.html
>>> 
>>> *** Please download, test  by  Thursday, November 20, 9am PT
>>> 
>>> Kafka's KEYS file containing PGP keys we use to sign the release:
>>> https://kafka.apache.org/KEYS
>>> 
>>> * Release artifacts to be voted upon (source and binary):
>>> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/
>>> 
>>> * Maven artifacts to be voted upon:
>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>> 
>>> * Javadoc:
>>> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/javadoc/
>>> 
>>> * Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
>>> https://github.com/apache/kafka/releases/tag/2.4.0-rc0
>>> 
>>> * Documentation:
>>> https://kafka.apache.org/24/documentation.html
>>> 
>>> * Protocol:
>>> https://kafka.apache.org/24/protocol.html
>>> 
>>> Thanks,
>>> Manikumar
>>> 
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "kafka-clients" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to kafka-clients+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/kafka-clients/CAMVt_Aw945uqcpisFjZHAR5m8Sidw6hW4ia%2B7%3DjxEfadmBPzcw%40mail.gmail.com.
>> 
>> 
>> 
>> --
>> 
>> Thanks!
>> --Vahid
>> 
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "kafka-clients" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to kafka-clients+unsubscr...@googlegroups.com 
>> .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/kafka-clients/CAHR2v2mKJtHG6S9P%3Dmw08SxbWjQCowp8cpZNpzr9acW1EcdegQ%40mail.gmail.com
>>  
>> .

[jira] [Created] (KAFKA-9206) Consumer should handle `CORRUPT_MESSAGE` error code in fetch response

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-9206:
--

 Summary: Consumer should handle `CORRUPT_MESSAGE` error code in 
fetch response
 Key: KAFKA-9206
 URL: https://issues.apache.org/jira/browse/KAFKA-9206
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson


This error code is possible, for example, when the broker scans the log to find 
the fetch offset after the index lookup. Currently this results in a slightly 
obscure message such as the following:
{code:java}
java.lang.IllegalStateException: Unexpected error code 2 while fetching 
data{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

[Jason Gustafson]

Hi Noa,

Re; uptime. Sure, it was just a suggestion. However, we should be clear
that there are actually two timestamps at play. Your initial proposal
suggested using the timestamp from the registration znode. However, this
changes each time the broker loses its session. It does not reflect the
actual broker start time, which seems to be what you are interested in. So
possibly it is worth exposing both the true broker start time as well as
its session creation time.

-Jason



On Mon, Nov 18, 2019 at 7:48 AM Ismael Juma  wrote:

> Hi Noa,
>
> I understand the desire for batching. However, once you do that, you either
> need request forwarding or broker metadata propagation. It's worth
> exploring, but I suspect you will end up with most of the changes needed by
> the original proposal, in that case.
>
> Ismael
>
> On Mon, Nov 18, 2019 at 7:07 AM Noa Resare  wrote:
>
> > Hi Jason & Gwen,
> >
> > Sure, this would solve our use case. I do have two questions, however:
> >
> > Moving from start time to uptime in theory would neatly side step the
> > clock skew problem,
> > but in practice I’m not sure it helps that much as the straightforward
> way
> > of going about
> > implementing this by returning (now - startTime) would break when the
> > clock on the broker
> > is adjusted. So, I think that startTime is more straight forward and
> > doesn’t hide the fact that
> > the system clock is sometimes off.
> >
> > Another thing I think would be useful is to build support for requesting
> > multiple (or all?)
> > brokers in a single request at the start. Having the request hold a set
> of
> > brokerIds
> > and return a set of brokers in the response, adding a brokerId field to
> > identify which
> > broker a specific broker status response is associated with seems
> straight
> > forward.
> >
> > I’ll start writing a proposal so that we have something concrete to
> > discuss.
> >
> > /noa
> >
> > > On 13 Nov 2019, at 16:43, Jason Gustafson  wrote:
> > >
> > > Ni Noa,
> > >
> > > Thanks for the KIP. I agree with the concerns about Metadata bloat. In
> > > fact, I have wanted a BrokerStatus API for a while now. Perhaps this
> is a
> > > good excuse to introduce it. I was thinking something like this:
> > >
> > > BrokerStatusRequest => BrokerId
> > >
> > > BrokerStatusResponse =>
> > >  Listeners => [ListenerName Host Port]
> > >  RackId
> > >  BrokerEpoch
> > >  BrokerState => {ACTIVE, RECOVERING, SHUTTING_DOWN}
> > >  UpTime
> > >
> > > What do you think?
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Tue, Nov 5, 2019 at 12:25 PM Noa Resare  wrote:
> > >
> > >> I agree with that. And looking at the MetadataResponse fields it seems
> > >> there has been some feature creep already here. Does the client use
> rack
> > >> information, for example?
> > >>
> > >> I guess one could do this either by introducing a new leaner message
> > pair,
> > >> used specifically enable client operation, and use
> > >> MetadataRequest/MetadataResponse for describeCluster() or one could
> > shrink
> > >> MetadataRequest/MetadataResponse and introduce a new more fully
> featured
> > >> message pair for the other stuff. I would be happy to spend some time
> > >> looking into implementing this if there is an interest.
> > >>
> > >> /noa
> > >>
> > >>> On 5 Nov 2019, at 15:43, Gwen Shapira  wrote:
> > >>>
> > >>> It isn't just about saving space. It increases complexity to default
> to
> > >>> always sharing a bit of information that is really only needed in a
> > >> single
> > >>> use-case.
> > >>> We avoid doing this as a matter of good protocol design.
> > >>> Arguably, this should not really piggyback on cluster metadata at
> all,
> > >>> since the usage is so different.
> > >>>
> > >>> On Tue, Nov 5, 2019, 7:29 AM Noa Resare  wrote:
> > >>>
> >  It would certainly be possible to have the field be optional and
> only
> >  include it if some flag is set in the DescribeClusterOptions
> instance
> >  passed to Admin.describeCluster() that in turn would translate to a
> > >> boolean
> >  in MetadataRequest indicating that we are asking for this piece of
> >  information.
> > 
> >  I’m not entirely sure that this extra complexity would be worth it
> for
> > >> the
> >  modestly smaller response size, in a message that already contains
> > >> things
> >  like the hostname and rack identifier per broker.
> > 
> >  /noa
> > 
> > > On 4 Nov 2019, at 14:45, Gwen Shapira  wrote:
> > >
> > > Cluster metadata is sent to clients on a very regular basis. Adding
> > > start-time there seems quite repetitive. Especially considering
> that
> > > this information is only useful in very specific cases.
> > >
> > > Can we add this capability to the Admin API in a way that won't
> > impact
> > > normal client workflow?
> > >
> > > On Mon, Nov 4, 2019 at 4:05 AM Noa Resare  wrote:
> > >>
> > >> Thank you for the feedback, Stanislav!
> > >>
> > >> I agree 

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

[Vinoth Chandar]

Thanks, everyone involved!

On Mon, Nov 18, 2019 at 7:51 AM John Roesler  wrote:

> Thanks to you, also, Navinder!
>
> Looking forward to getting this feature in.
> -John
>
> On Sun, Nov 17, 2019 at 11:34 PM Navinder Brar
>  wrote:
> >
> >  Hello all,
> >
> > With 4 binding +1 votes from Guozhang Wang, Matthias J. Sax, Bill Bejeck,
> > and John Roesler, the vote passes.
> > Thanks Guozhang, Matthias, Bill, John, Sophie for the healthy
> discussions and Vinoth for all the help on this KIP.
> > Best,
> > Navinder
> >
> > On Friday, 15 November, 2019, 11:32:31 pm IST, John Roesler <
> j...@confluent.io> wrote:
> >
> >  I'm +1 (binding) as well.
> >
> > Thanks,
> > -John
> >
> > On Fri, Nov 15, 2019 at 6:20 AM Bill Bejeck  wrote:
> > >
> > > +1 (binding)
> > >
> > > On Fri, Nov 15, 2019 at 1:11 AM Matthias J. Sax  >
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > >
> > > > On 11/14/19 3:48 PM, Guozhang Wang wrote:
> > > > > +1 (binding), thanks for the KIP!
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Fri, Nov 15, 2019 at 4:38 AM Navinder Brar
> > > > >  wrote:
> > > > >
> > > > >> Hello all,
> > > > >>
> > > > >> I'd like to propose a vote for serving interactive queries during
> > > > >> Rebalancing, as it is a big deal for applications looking for high
> > > > >> availability. With this change, users will have control over the
> > > > tradeoff
> > > > >> between consistency and availability during serving.
> > > > >> The full KIP is provided here:
> > > > >>
> > > > >>
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-535%3A+Allow+state+stores+to+serve+stale+reads+during+rebalance
> > > > >>
> > > > >>
> > > > >> Thanks,
> > > > >> Navinder
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> >
>

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

[Brian Byrne]

On Fri, Nov 15, 2019 at 5:08 PM Colin McCabe  wrote:

> Two seconds doesn't seem like a reasonable amount of time to leave for the
> metadata fetch.  Fetching halfway through the expiration period seems more
> reasonable.  It also doesn't require us to create a new configuration key,
> which is nice.
>
> Another option is to just do the metadata fetch every metadata.max.age.ms,
> but not expire the topic until we can't fetch the metadata for 2 *
> metadata.max.age.ms.
>

I'd expect two seconds to be reasonable in the common case. Keep in mind
that this doesn't affect correctness, and a control operation returning
cached metadata should be on the order of milliseconds. But to the general
point, defining the algorithm would mean enforcing it to fair accuracy,
whereas if the suggestion is that it'll be performed at a reasonable time,
it allows for batching and other optimizations. Perhaps I shouldn't be
regarding what's defined in a KIP to be contractual in these cases, but you
could consider a first implementation to collect topics whose metadata has
exceeded (metadata.max.age.ms / 2), and sending the batch once a
constituent topic's metadata is near the expiry, or a sufficient number of
topics have been collected (10? 100? 1000?).


We should be specific about what happens if the first few metadata fetches
> fail.  Do we use exponential backoff to decide when to resend?  It seems
> like we really should, for all the usual reasons (reduce the load on
> brokers, ride out temporary service disruptions, etc.)  Maybe we could have
> an exponential retry backoff for each broker (in other words, we should try
> to contact a different broker before applying the backoff.)  I think this
> already sort of happens with the disconnect timeout, but we might need a
> more general solution.
>

I don't plan to change this behavior. Currently it retries after a fixed
value of 'retry.backoff.ms' (defaults to 100 ms). It's possible that
different brokers are attempted, but I haven't dug into it.


Thanks for the clarification.  Fully asynchronous is the way to go, I
> agree.  I'm having trouble understanding how timeouts are handled in the
> KIP.  It seems like if we can't fetch the metadata within the designated
> metadata timeout, the future / callback should receive a TimeoutException
> right?  We do not want the send call to be deferred forever if metadata
> can't be fetched.  Eventually it should fail if it can't be performed.
>
> I do think this is something that will have to be mentioned in the
> compatibility section.  There is some code out there that is probably
> prepared to handle a timeout exception from the send function, which may
> need to be updated to check for a timeout from the future or callback.
>

Correct, a timeout exception would be delivered in the future. Sure, I can
add that note to the KIP.



> It seems like this is an existing problem.  You may fire off a lot of send
> calls that get blocked because the broker that is the leader for a certain
> partition is not responding.  I'm not sure that we need to do anything
> special here.  On the other hand, we could make the case for a generic "max
> number of outstanding sends" configuration to prevent surprise OOMs in the
> existing cases, plus the new one we're adding.  But this feels like a bit
> of a scope expansion.
>

Right, this is an existing problem, however the asynchronous send could
cause unexpected behavior. For example, if a client pinned
topics/partitions to individual send threads, then memory couldn't be
exhausted by a single topic since a blocking send would prevent further
records from being buffered on that topic. The compromise could be that we
only ever permit one outstanding record batch for a topic, which would keep
the code simple and wouldn't permit a single topic to consume all available
memory.



> They may be connected, but I'm not sure they should be the same.  Perhaps
> expiry should be 4x the typical fetch rate, for example.
>

That's true. You could also make the case for a faster expiry than refresh
as well. Makes sense to separate this out.



> Hmm are you sure this is an N^2 problem?  If you have T1 and T2, you
> fetch metadata for T1 and T2, right?  I guess you could argue that we often
> fetch metadata for partitions we don't care about, but that doesn't make it
> O(N^2).  Maybe there's something about the implementation that I'm missing.
>

My apologies, I left out the context. One issue the KIP is trying to
resolve is the metadata storm that's caused by producers starting up. In
the worst case, where the send call is only performed from a single thread
(i.e. no possible batching), fetching metadata for 1K topics will generate
1K RPCs, with payload 1+2+...+1K topics' metadata. Being smart about the
topics being refreshed would still generate 1K RPCs for 1 topic each, and
asynchronous behavior would permit batching (note steady-state refreshing
doesn't require the asynchronous behavior to batch).



> In 

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

[Sean Glover]

Here's a summary that can go under the section "What’s new in Kafka broker,
producer, and consumer" as an "improvement".  Feel free to rephrase as you
see fit.

When a partition is paused by the user in the consumer the partition is
> considered "unfetchable".  When the consumer has already fetched data for a
> partition, and then the partition is paused, then in the next consumer poll
> all data from "unfetchable" partitions will be discarded.  In use cases
> where pausing and resuming partitions is common during regular operation of
> the consumer this can result in discarding pre-fetched data when it's not
> necessary.  Once the partition is resumed then new fetch requests will be
> generated and sent to the broker to get the same partition data again.
> Depending on the frequency of pausing and resuming of partitions this can
> impact a number of different aspects of consumer polling including:
> broker/consumer throughput, number of consumer fetch requests, and
> NIO-related GC concerns for regularly dereferenced byte buffers of
> partition data.  This issue is now resolved by retaining completed fetch
> data for partitions that are paused so that it may be returned in a future
> consumer poll once the partition is resumed by the user.
>


See [KAFKA-7548](https://issues.apache.org/jira/browse/KAFKA-7548) for more
> details.


Regards,
Sean

On Mon, Nov 18, 2019 at 11:45 AM Ismael Juma  wrote:

> That makes sense to me.
>
> Ismael
>
> On Mon, Nov 18, 2019 at 8:40 AM Sean Glover 
> wrote:
>
> > Hi Manikumar,
> >
> > I'm putting together an akka.io blog post regarding [KAFKA-7548] -
> > KafkaConsumer should not throw away already fetched data for paused
> > partitions.  Since it doesn't change any user-facing APIs it has no KIP,
> > but it has a significant impact on consumer use cases that frequently
> pause
> > and resume partitions, such as in Alpakka Kafka.  I can provide a small
> > summary for you to include in your blog post if you think it's
> appropriate.
> >
> > Regards,
> > Sean
> >
> > On Mon, Nov 18, 2019 at 11:25 AM Manikumar 
> > wrote:
> >
> > > Thanks Chris. will update the blog content.
> > >
> > > On Fri, Nov 15, 2019 at 12:34 AM Chris Egerton 
> > > wrote:
> > >
> > > > Hi Manikumar,
> > > >
> > > > It looks like the header for KIP-440 is accurate ("KIP-440: Extend
> > > Connect
> > > > Converter to support headers") but the content appears to correspond
> to
> > > > KIP-481 ("SerDe Improvements for Connect Decimal type in JSON")
> > instead.
> > > > Could we double-check and make sure that the summary for KIP-440
> > matches
> > > > what was contributed for it (and it nothing was, alter the summary to
> > > more
> > > > closely reflect what KIP-440 accomplished)?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Thu, Nov 14, 2019 at 10:41 AM Manikumar <
> manikumar.re...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I've prepared a preliminary blog post about the upcoming Apache
> Kafka
> > > > 2.4.0
> > > > > release.
> > > > > Please take a look and let me know if you want to add/modify
> details.
> > > > > Thanks to all who contributed to this blog post.
> > > > >
> > > > >
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > >
> > >
> >

[jira] [Created] (KAFKA-9205) Add an option to enforce rack-aware partition reassignment

[Vahid Hashemian (Jira)]

Vahid Hashemian created KAFKA-9205:
--

 Summary: Add an option to enforce rack-aware partition reassignment
 Key: KAFKA-9205
 URL: https://issues.apache.org/jira/browse/KAFKA-9205
 Project: Kafka
  Issue Type: Improvement
  Components: admin, tools
Reporter: Vahid Hashemian


One regularly used healing operation on Kafka clusters is replica reassignments 
for topic partitions. For example, when there is a skew in inbound/outbound 
traffic of a broker replica reassignment can be used to move some 
leaders/followers from the broker; or if there is a skew in disk usage of 
brokers, replica reassignment can more some partitions to other brokers that 
have more disk space available.

In Kafka clusters that span across multiple data centers (or availability 
zones), high availability is a priority; in the sense that when a data center 
goes offline the cluster should be able to resume normal operation by 
guaranteeing partition replicas in all data centers.

This guarantee is currently the responsibility of the on-call engineer that 
performs the reassignment or the tool that automatically generates the 
reassignment plan for improving the cluster health (e.g. by considering the 
rack configuration value of each broker in the cluster). the former, is quite 
error-prone, and the latter, would lead to duplicate code in all such admin 
tools (which are not error free either).

It would be great for the built-in replica assignment API and tool provided by 
Kafka to support a rack aware verification option that would simply return an 
error when [some] brokers in any replica set share a common rack. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

[Mickael Maison]

+1 (binding)
Thanks for the KIP

On Mon, Nov 18, 2019 at 3:54 PM Sean Glover  wrote:
>
> +1 (non-binding)
>
> Good idea.  It will streamline the Streams Scala DSL nicely.  It will also
> affect 2.11 users of embedded-kafka.  Are there any other non-broker
> dependencies that could be affected?
>
> Sean
>
> On Mon, Nov 18, 2019 at 10:43 AM Ismael Juma  wrote:
>
> > Yes, everyone is encouraged to vote. Committer votes are binding, but we
> > are interested in what the wider community thinks too.
> >
> > Ismael
> >
> > On Mon, Nov 18, 2019 at 5:40 AM Ivan Yurchenko 
> > wrote:
> >
> > > Do I understand correctly, that non-commiters can also vote, despite
> > their
> > > votes don't decide?
> > >
> > > If so, then +1 from me.
> > >
> > > Ivan
> > >
> > >
> > > On Mon, 18 Nov 2019 at 15:19, Ismael Juma  wrote:
> > >
> > > > Hi all,
> > > >
> > > > People seemed supportive in general, so I'd like to start a vote on
> > > > KIP-531:
> > > >
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-531%3A+Drop+support+for+Scala+2.11+in+Kafka+2.5
> > > >
> > > > Ismael
> > > >
> > >
> >
>
>
> --
> Sean Glover
> Principal Engineer, Alpakka, Lightbend, Inc. 
> @seg1o , in/seanaglover
> 

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

[Ismael Juma]

That makes sense to me.

Ismael

On Mon, Nov 18, 2019 at 8:40 AM Sean Glover 
wrote:

> Hi Manikumar,
>
> I'm putting together an akka.io blog post regarding [KAFKA-7548] -
> KafkaConsumer should not throw away already fetched data for paused
> partitions.  Since it doesn't change any user-facing APIs it has no KIP,
> but it has a significant impact on consumer use cases that frequently pause
> and resume partitions, such as in Alpakka Kafka.  I can provide a small
> summary for you to include in your blog post if you think it's appropriate.
>
> Regards,
> Sean
>
> On Mon, Nov 18, 2019 at 11:25 AM Manikumar 
> wrote:
>
> > Thanks Chris. will update the blog content.
> >
> > On Fri, Nov 15, 2019 at 12:34 AM Chris Egerton 
> > wrote:
> >
> > > Hi Manikumar,
> > >
> > > It looks like the header for KIP-440 is accurate ("KIP-440: Extend
> > Connect
> > > Converter to support headers") but the content appears to correspond to
> > > KIP-481 ("SerDe Improvements for Connect Decimal type in JSON")
> instead.
> > > Could we double-check and make sure that the summary for KIP-440
> matches
> > > what was contributed for it (and it nothing was, alter the summary to
> > more
> > > closely reflect what KIP-440 accomplished)?
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Thu, Nov 14, 2019 at 10:41 AM Manikumar 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I've prepared a preliminary blog post about the upcoming Apache Kafka
> > > 2.4.0
> > > > release.
> > > > Please take a look and let me know if you want to add/modify details.
> > > > Thanks to all who contributed to this blog post.
> > > >
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > >
> >
>

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

[Sean Glover]

Hi Manikumar,

I'm putting together an akka.io blog post regarding [KAFKA-7548] -
KafkaConsumer should not throw away already fetched data for paused
partitions.  Since it doesn't change any user-facing APIs it has no KIP,
but it has a significant impact on consumer use cases that frequently pause
and resume partitions, such as in Alpakka Kafka.  I can provide a small
summary for you to include in your blog post if you think it's appropriate.

Regards,
Sean

On Mon, Nov 18, 2019 at 11:25 AM Manikumar 
wrote:

> Thanks Chris. will update the blog content.
>
> On Fri, Nov 15, 2019 at 12:34 AM Chris Egerton 
> wrote:
>
> > Hi Manikumar,
> >
> > It looks like the header for KIP-440 is accurate ("KIP-440: Extend
> Connect
> > Converter to support headers") but the content appears to correspond to
> > KIP-481 ("SerDe Improvements for Connect Decimal type in JSON") instead.
> > Could we double-check and make sure that the summary for KIP-440 matches
> > what was contributed for it (and it nothing was, alter the summary to
> more
> > closely reflect what KIP-440 accomplished)?
> >
> > Cheers,
> >
> > Chris
> >
> > On Thu, Nov 14, 2019 at 10:41 AM Manikumar 
> > wrote:
> >
> > > Hi all,
> > >
> > > I've prepared a preliminary blog post about the upcoming Apache Kafka
> > 2.4.0
> > > release.
> > > Please take a look and let me know if you want to add/modify details.
> > > Thanks to all who contributed to this blog post.
> > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>

Re: [kafka-clients] [VOTE] 2.4.0 RC0

[Vahid Hashemian]

Thanks Bruno,

Just to clarify, I ran the tests from the command line: ./gradlew
streams:test

Regards,
--Vahid

On Mon, Nov 18, 2019 at 6:16 AM Bruno Cadonna  wrote:

> Hi,
>
> ClientMetricsTest.shouldAddCommitIdMetric should only fail if executed
> from an IDE. The test fails because the test expects a file on the
> class path which is not there when the test is executed from the IDE,
> but is there when the test is executed from gradle. I will try to fix
> the test so that it can also be executed from the IDE.
>
> Best,
> Bruno
>
> On Mon, Nov 18, 2019 at 6:51 AM Vahid Hashemian
>  wrote:
> >
> > Thanks Manikumar for managing this release. Looking forward to it.
> >
> > I built binary from the source and was able to successfully run the
> quickstarts.
> >
> > However, this streams unit test also fails for me constantly:
> >
> > ClientMetricsTest. shouldAddCommitIdMetric
> >
> > java.lang.AssertionError:
> >   Unexpected method call
> StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The version
> control commit ID of the Kafka Streams client", INFO, "unknown"):
> > StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The
> version control commit ID of the Kafka Streams client", INFO,
> and(not("unknown"), notNull())): expected: 1, actual: 0
> > at
> org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:44)
> > at
> org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:101)
> > at
> org.easymock.internal.ClassProxyFactory$MockMethodInterceptor.intercept(ClassProxyFactory.java:97)
> > ...
> >
> > Thanks,
> > --Vahid
> >
> > On Thu, Nov 14, 2019 at 10:21 AM Manikumar 
> wrote:
> >>
> >> Hello Kafka users, developers and client-developers,
> >>
> >> This is the first candidate for release of Apache Kafka 2.4.0.
> >> There is work in progress for couple blockers PRs. I am publishing RC0
> to avoid further delays in testing the release.
> >>
> >> This release includes many new features, including:
> >> - Allow consumers to fetch from closest replica
> >> - Support for incremental cooperative rebalancing to the consumer
> rebalance protocol
> >> - MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter
> replication engine
> >> - New Java authorizer Interface
> >> - Support for  non-key joining in KTable
> >> - Administrative API for replica reassignment
> >> - Sticky partitioner
> >> - Return topic metadata and configs in CreateTopics response
> >> - Securing Internal connect REST endpoints
> >> - API to delete consumer offsets and expose it via the AdminClient.
> >>
> >> Release notes for the 2.4.0 release:
> >> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/RELEASE_NOTES.html
> >>
> >> *** Please download, test  by  Thursday, November 20, 9am PT
> >>
> >> Kafka's KEYS file containing PGP keys we use to sign the release:
> >> https://kafka.apache.org/KEYS
> >>
> >> * Release artifacts to be voted upon (source and binary):
> >> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/
> >>
> >> * Maven artifacts to be voted upon:
> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>
> >> * Javadoc:
> >> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/javadoc/
> >>
> >> * Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
> >> https://github.com/apache/kafka/releases/tag/2.4.0-rc0
> >>
> >> * Documentation:
> >> https://kafka.apache.org/24/documentation.html
> >>
> >> * Protocol:
> >> https://kafka.apache.org/24/protocol.html
> >>
> >> Thanks,
> >> Manikumar
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups "kafka-clients" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an email to kafka-clients+unsubscr...@googlegroups.com.
> >> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAMVt_Aw945uqcpisFjZHAR5m8Sidw6hW4ia%2B7%3DjxEfadmBPzcw%40mail.gmail.com
> .
> >
> >
> >
> > --
> >
> > Thanks!
> > --Vahid
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "kafka-clients" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to kafka-clients+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAHR2v2mKJtHG6S9P%3Dmw08SxbWjQCowp8cpZNpzr9acW1EcdegQ%40mail.gmail.com
> .
>


-- 

Thanks!
--Vahid

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

[Manikumar]

Thanks Chris. will update the blog content.

On Fri, Nov 15, 2019 at 12:34 AM Chris Egerton  wrote:

> Hi Manikumar,
>
> It looks like the header for KIP-440 is accurate ("KIP-440: Extend Connect
> Converter to support headers") but the content appears to correspond to
> KIP-481 ("SerDe Improvements for Connect Decimal type in JSON") instead.
> Could we double-check and make sure that the summary for KIP-440 matches
> what was contributed for it (and it nothing was, alter the summary to more
> closely reflect what KIP-440 accomplished)?
>
> Cheers,
>
> Chris
>
> On Thu, Nov 14, 2019 at 10:41 AM Manikumar 
> wrote:
>
> > Hi all,
> >
> > I've prepared a preliminary blog post about the upcoming Apache Kafka
> 2.4.0
> > release.
> > Please take a look and let me know if you want to add/modify details.
> > Thanks to all who contributed to this blog post.
> >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> >
> > Thanks,
> > Manikumar
> >
>

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

[Sean Glover]

+1 (non-binding)

Good idea.  It will streamline the Streams Scala DSL nicely.  It will also
affect 2.11 users of embedded-kafka.  Are there any other non-broker
dependencies that could be affected?

Sean

On Mon, Nov 18, 2019 at 10:43 AM Ismael Juma  wrote:

> Yes, everyone is encouraged to vote. Committer votes are binding, but we
> are interested in what the wider community thinks too.
>
> Ismael
>
> On Mon, Nov 18, 2019 at 5:40 AM Ivan Yurchenko 
> wrote:
>
> > Do I understand correctly, that non-commiters can also vote, despite
> their
> > votes don't decide?
> >
> > If so, then +1 from me.
> >
> > Ivan
> >
> >
> > On Mon, 18 Nov 2019 at 15:19, Ismael Juma  wrote:
> >
> > > Hi all,
> > >
> > > People seemed supportive in general, so I'd like to start a vote on
> > > KIP-531:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-531%3A+Drop+support+for+Scala+2.11+in+Kafka+2.5
> > >
> > > Ismael
> > >
> >
>


-- 
Sean Glover
Principal Engineer, Alpakka, Lightbend, Inc. 
@seg1o , in/seanaglover

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

[John Roesler]

Thanks to you, also, Navinder!

Looking forward to getting this feature in.
-John

On Sun, Nov 17, 2019 at 11:34 PM Navinder Brar
 wrote:
>
>  Hello all,
>
> With 4 binding +1 votes from Guozhang Wang, Matthias J. Sax, Bill Bejeck,
> and John Roesler, the vote passes.
> Thanks Guozhang, Matthias, Bill, John, Sophie for the healthy discussions and 
> Vinoth for all the help on this KIP.
> Best,
> Navinder
>
> On Friday, 15 November, 2019, 11:32:31 pm IST, John Roesler 
>  wrote:
>
>  I'm +1 (binding) as well.
>
> Thanks,
> -John
>
> On Fri, Nov 15, 2019 at 6:20 AM Bill Bejeck  wrote:
> >
> > +1 (binding)
> >
> > On Fri, Nov 15, 2019 at 1:11 AM Matthias J. Sax 
> > wrote:
> >
> > > +1 (binding)
> > >
> > >
> > > On 11/14/19 3:48 PM, Guozhang Wang wrote:
> > > > +1 (binding), thanks for the KIP!
> > > >
> > > > Guozhang
> > > >
> > > > On Fri, Nov 15, 2019 at 4:38 AM Navinder Brar
> > > >  wrote:
> > > >
> > > >> Hello all,
> > > >>
> > > >> I'd like to propose a vote for serving interactive queries during
> > > >> Rebalancing, as it is a big deal for applications looking for high
> > > >> availability. With this change, users will have control over the
> > > tradeoff
> > > >> between consistency and availability during serving.
> > > >> The full KIP is provided here:
> > > >>
> > > >>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-535%3A+Allow+state+stores+to+serve+stale+reads+during+rebalance
> > > >>
> > > >>
> > > >> Thanks,
> > > >> Navinder
> > > >
> > > >
> > > >
> > >
> > >
>

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

[John Roesler]

Thanks, Ismael.

+1 (binding)

John

On Mon, Nov 18, 2019 at 9:43 AM Ismael Juma  wrote:
>
> Yes, everyone is encouraged to vote. Committer votes are binding, but we
> are interested in what the wider community thinks too.
>
> Ismael
>
> On Mon, Nov 18, 2019 at 5:40 AM Ivan Yurchenko 
> wrote:
>
> > Do I understand correctly, that non-commiters can also vote, despite their
> > votes don't decide?
> >
> > If so, then +1 from me.
> >
> > Ivan
> >
> >
> > On Mon, 18 Nov 2019 at 15:19, Ismael Juma  wrote:
> >
> > > Hi all,
> > >
> > > People seemed supportive in general, so I'd like to start a vote on
> > > KIP-531:
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-531%3A+Drop+support+for+Scala+2.11+in+Kafka+2.5
> > >
> > > Ismael
> > >
> >

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

[Ismael Juma]

Hi Noa,

I understand the desire for batching. However, once you do that, you either
need request forwarding or broker metadata propagation. It's worth
exploring, but I suspect you will end up with most of the changes needed by
the original proposal, in that case.

Ismael

On Mon, Nov 18, 2019 at 7:07 AM Noa Resare  wrote:

> Hi Jason & Gwen,
>
> Sure, this would solve our use case. I do have two questions, however:
>
> Moving from start time to uptime in theory would neatly side step the
> clock skew problem,
> but in practice I’m not sure it helps that much as the straightforward way
> of going about
> implementing this by returning (now - startTime) would break when the
> clock on the broker
> is adjusted. So, I think that startTime is more straight forward and
> doesn’t hide the fact that
> the system clock is sometimes off.
>
> Another thing I think would be useful is to build support for requesting
> multiple (or all?)
> brokers in a single request at the start. Having the request hold a set of
> brokerIds
> and return a set of brokers in the response, adding a brokerId field to
> identify which
> broker a specific broker status response is associated with seems straight
> forward.
>
> I’ll start writing a proposal so that we have something concrete to
> discuss.
>
> /noa
>
> > On 13 Nov 2019, at 16:43, Jason Gustafson  wrote:
> >
> > Ni Noa,
> >
> > Thanks for the KIP. I agree with the concerns about Metadata bloat. In
> > fact, I have wanted a BrokerStatus API for a while now. Perhaps this is a
> > good excuse to introduce it. I was thinking something like this:
> >
> > BrokerStatusRequest => BrokerId
> >
> > BrokerStatusResponse =>
> >  Listeners => [ListenerName Host Port]
> >  RackId
> >  BrokerEpoch
> >  BrokerState => {ACTIVE, RECOVERING, SHUTTING_DOWN}
> >  UpTime
> >
> > What do you think?
> >
> > Thanks,
> > Jason
> >
> > On Tue, Nov 5, 2019 at 12:25 PM Noa Resare  wrote:
> >
> >> I agree with that. And looking at the MetadataResponse fields it seems
> >> there has been some feature creep already here. Does the client use rack
> >> information, for example?
> >>
> >> I guess one could do this either by introducing a new leaner message
> pair,
> >> used specifically enable client operation, and use
> >> MetadataRequest/MetadataResponse for describeCluster() or one could
> shrink
> >> MetadataRequest/MetadataResponse and introduce a new more fully featured
> >> message pair for the other stuff. I would be happy to spend some time
> >> looking into implementing this if there is an interest.
> >>
> >> /noa
> >>
> >>> On 5 Nov 2019, at 15:43, Gwen Shapira  wrote:
> >>>
> >>> It isn't just about saving space. It increases complexity to default to
> >>> always sharing a bit of information that is really only needed in a
> >> single
> >>> use-case.
> >>> We avoid doing this as a matter of good protocol design.
> >>> Arguably, this should not really piggyback on cluster metadata at all,
> >>> since the usage is so different.
> >>>
> >>> On Tue, Nov 5, 2019, 7:29 AM Noa Resare  wrote:
> >>>
>  It would certainly be possible to have the field be optional and only
>  include it if some flag is set in the DescribeClusterOptions instance
>  passed to Admin.describeCluster() that in turn would translate to a
> >> boolean
>  in MetadataRequest indicating that we are asking for this piece of
>  information.
> 
>  I’m not entirely sure that this extra complexity would be worth it for
> >> the
>  modestly smaller response size, in a message that already contains
> >> things
>  like the hostname and rack identifier per broker.
> 
>  /noa
> 
> > On 4 Nov 2019, at 14:45, Gwen Shapira  wrote:
> >
> > Cluster metadata is sent to clients on a very regular basis. Adding
> > start-time there seems quite repetitive. Especially considering that
> > this information is only useful in very specific cases.
> >
> > Can we add this capability to the Admin API in a way that won't
> impact
> > normal client workflow?
> >
> > On Mon, Nov 4, 2019 at 4:05 AM Noa Resare  wrote:
> >>
> >> Thank you for the feedback, Stanislav!
> >>
> >> I agree that it would be good to harmonise the naming, and
>  start-time-ms definitely more descriptive.
> >>
> >> I have updated the proposal to reflect this, and also added the
> >> updated
>  json RPC changes. Please have a look.
> >>
> >> /noa
> >>
> >>> On 1 Nov 2019, at 09:13, Stanislav Kozlovski <
> stanis...@confluent.io
> >>>
>  wrote:
> >>>
> >>> Hey Noa,
> >>>
> >>> KIP-436 added a JMX metric in Kafka for this exact use-case, called
> >>> `start-time-ms`. Perhaps it would be useful to name this public
>  interface
> >>> in the same way for consistency.
> >>>
> >>> Could you update the KIP to include the specific RPC changes
> >> regarding
>  the
> >>> metadata request/responses? Here is a recent example of 

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

[Ismael Juma]

Yes, everyone is encouraged to vote. Committer votes are binding, but we
are interested in what the wider community thinks too.

Ismael

On Mon, Nov 18, 2019 at 5:40 AM Ivan Yurchenko 
wrote:

> Do I understand correctly, that non-commiters can also vote, despite their
> votes don't decide?
>
> If so, then +1 from me.
>
> Ivan
>
>
> On Mon, 18 Nov 2019 at 15:19, Ismael Juma  wrote:
>
> > Hi all,
> >
> > People seemed supportive in general, so I'd like to start a vote on
> > KIP-531:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-531%3A+Drop+support+for+Scala+2.11+in+Kafka+2.5
> >
> > Ismael
> >
>

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

[Noa Resare]

Hi Jason & Gwen,

Sure, this would solve our use case. I do have two questions, however:

Moving from start time to uptime in theory would neatly side step the clock 
skew problem,
but in practice I’m not sure it helps that much as the straightforward way of 
going about
implementing this by returning (now - startTime) would break when the clock on 
the broker
is adjusted. So, I think that startTime is more straight forward and doesn’t 
hide the fact that
the system clock is sometimes off.

Another thing I think would be useful is to build support for requesting 
multiple (or all?) 
brokers in a single request at the start. Having the request hold a set of 
brokerIds
and return a set of brokers in the response, adding a brokerId field to 
identify which
broker a specific broker status response is associated with seems straight 
forward.

I’ll start writing a proposal so that we have something concrete to discuss.

/noa

> On 13 Nov 2019, at 16:43, Jason Gustafson  wrote:
> 
> Ni Noa,
> 
> Thanks for the KIP. I agree with the concerns about Metadata bloat. In
> fact, I have wanted a BrokerStatus API for a while now. Perhaps this is a
> good excuse to introduce it. I was thinking something like this:
> 
> BrokerStatusRequest => BrokerId
> 
> BrokerStatusResponse =>
>  Listeners => [ListenerName Host Port]
>  RackId
>  BrokerEpoch
>  BrokerState => {ACTIVE, RECOVERING, SHUTTING_DOWN}
>  UpTime
> 
> What do you think?
> 
> Thanks,
> Jason
> 
> On Tue, Nov 5, 2019 at 12:25 PM Noa Resare  wrote:
> 
>> I agree with that. And looking at the MetadataResponse fields it seems
>> there has been some feature creep already here. Does the client use rack
>> information, for example?
>> 
>> I guess one could do this either by introducing a new leaner message pair,
>> used specifically enable client operation, and use
>> MetadataRequest/MetadataResponse for describeCluster() or one could shrink
>> MetadataRequest/MetadataResponse and introduce a new more fully featured
>> message pair for the other stuff. I would be happy to spend some time
>> looking into implementing this if there is an interest.
>> 
>> /noa
>> 
>>> On 5 Nov 2019, at 15:43, Gwen Shapira  wrote:
>>> 
>>> It isn't just about saving space. It increases complexity to default to
>>> always sharing a bit of information that is really only needed in a
>> single
>>> use-case.
>>> We avoid doing this as a matter of good protocol design.
>>> Arguably, this should not really piggyback on cluster metadata at all,
>>> since the usage is so different.
>>> 
>>> On Tue, Nov 5, 2019, 7:29 AM Noa Resare  wrote:
>>> 
 It would certainly be possible to have the field be optional and only
 include it if some flag is set in the DescribeClusterOptions instance
 passed to Admin.describeCluster() that in turn would translate to a
>> boolean
 in MetadataRequest indicating that we are asking for this piece of
 information.
 
 I’m not entirely sure that this extra complexity would be worth it for
>> the
 modestly smaller response size, in a message that already contains
>> things
 like the hostname and rack identifier per broker.
 
 /noa
 
> On 4 Nov 2019, at 14:45, Gwen Shapira  wrote:
> 
> Cluster metadata is sent to clients on a very regular basis. Adding
> start-time there seems quite repetitive. Especially considering that
> this information is only useful in very specific cases.
> 
> Can we add this capability to the Admin API in a way that won't impact
> normal client workflow?
> 
> On Mon, Nov 4, 2019 at 4:05 AM Noa Resare  wrote:
>> 
>> Thank you for the feedback, Stanislav!
>> 
>> I agree that it would be good to harmonise the naming, and
 start-time-ms definitely more descriptive.
>> 
>> I have updated the proposal to reflect this, and also added the
>> updated
 json RPC changes. Please have a look.
>> 
>> /noa
>> 
>>> On 1 Nov 2019, at 09:13, Stanislav Kozlovski >> 
 wrote:
>>> 
>>> Hey Noa,
>>> 
>>> KIP-436 added a JMX metric in Kafka for this exact use-case, called
>>> `start-time-ms`. Perhaps it would be useful to name this public
 interface
>>> in the same way for consistency.
>>> 
>>> Could you update the KIP to include the specific RPC changes
>> regarding
 the
>>> metadata request/responses? Here is a recent example of how to
>> portray
 the
>>> changes -
>>> 
 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-525+-+Return+topic+metadata+and+configs+in+CreateTopics+response
>>> 
>>> Thanks,
>>> Stanislav!
>>> 
>>> On Mon, Oct 14, 2019 at 2:46 PM Noa Resare  wrote:
>>> 
 We are in the process of migrating the pieces of automation that
 currently
 reads and modifies zookeeper state to use the Admin API.
 
 One of the things that we miss doing this is access to the start
>> time
 of
 

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

[Bill Bejeck]

+1 (binding)

On Mon, Nov 18, 2019 at 9:30 AM Satish Duggana 
wrote:

> +1 (non-binding), Thanks Ismael for the KIP. I believe it is time to
> drop Scala 2.11.
>
>
> On Mon, Nov 18, 2019 at 7:10 PM Ivan Yurchenko 
> wrote:
> >
> > Do I understand correctly, that non-commiters can also vote, despite
> their
> > votes don't decide?
> >
> > If so, then +1 from me.
> >
> > Ivan
> >
> >
> > On Mon, 18 Nov 2019 at 15:19, Ismael Juma  wrote:
> >
> > > Hi all,
> > >
> > > People seemed supportive in general, so I'd like to start a vote on
> > > KIP-531:
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-531%3A+Drop+support+for+Scala+2.11+in+Kafka+2.5
> > >
> > > Ismael
> > >
>

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

[Satish Duggana]

+1 (non-binding), Thanks Ismael for the KIP. I believe it is time to
drop Scala 2.11.


On Mon, Nov 18, 2019 at 7:10 PM Ivan Yurchenko  wrote:
>
> Do I understand correctly, that non-commiters can also vote, despite their
> votes don't decide?
>
> If so, then +1 from me.
>
> Ivan
>
>
> On Mon, 18 Nov 2019 at 15:19, Ismael Juma  wrote:
>
> > Hi all,
> >
> > People seemed supportive in general, so I'd like to start a vote on
> > KIP-531:
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-531%3A+Drop+support+for+Scala+2.11+in+Kafka+2.5
> >
> > Ismael
> >

Re: [kafka-clients] [VOTE] 2.4.0 RC0

[Bruno Cadonna]

Hi,

ClientMetricsTest.shouldAddCommitIdMetric should only fail if executed
from an IDE. The test fails because the test expects a file on the
class path which is not there when the test is executed from the IDE,
but is there when the test is executed from gradle. I will try to fix
the test so that it can also be executed from the IDE.

Best,
Bruno

On Mon, Nov 18, 2019 at 6:51 AM Vahid Hashemian
 wrote:
>
> Thanks Manikumar for managing this release. Looking forward to it.
>
> I built binary from the source and was able to successfully run the 
> quickstarts.
>
> However, this streams unit test also fails for me constantly:
>
> ClientMetricsTest. shouldAddCommitIdMetric
>
> java.lang.AssertionError:
>   Unexpected method call 
> StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The version 
> control commit ID of the Kafka Streams client", INFO, "unknown"):
> StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The 
> version control commit ID of the Kafka Streams client", INFO, 
> and(not("unknown"), notNull())): expected: 1, actual: 0
> at 
> org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:44)
> at 
> org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:101)
> at 
> org.easymock.internal.ClassProxyFactory$MockMethodInterceptor.intercept(ClassProxyFactory.java:97)
> ...
>
> Thanks,
> --Vahid
>
> On Thu, Nov 14, 2019 at 10:21 AM Manikumar  wrote:
>>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the first candidate for release of Apache Kafka 2.4.0.
>> There is work in progress for couple blockers PRs. I am publishing RC0 to 
>> avoid further delays in testing the release.
>>
>> This release includes many new features, including:
>> - Allow consumers to fetch from closest replica
>> - Support for incremental cooperative rebalancing to the consumer rebalance 
>> protocol
>> - MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication 
>> engine
>> - New Java authorizer Interface
>> - Support for  non-key joining in KTable
>> - Administrative API for replica reassignment
>> - Sticky partitioner
>> - Return topic metadata and configs in CreateTopics response
>> - Securing Internal connect REST endpoints
>> - API to delete consumer offsets and expose it via the AdminClient.
>>
>> Release notes for the 2.4.0 release:
>> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/RELEASE_NOTES.html
>>
>> *** Please download, test  by  Thursday, November 20, 9am PT
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> https://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>
>> * Javadoc:
>> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/javadoc/
>>
>> * Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
>> https://github.com/apache/kafka/releases/tag/2.4.0-rc0
>>
>> * Documentation:
>> https://kafka.apache.org/24/documentation.html
>>
>> * Protocol:
>> https://kafka.apache.org/24/protocol.html
>>
>> Thanks,
>> Manikumar
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "kafka-clients" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to kafka-clients+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/kafka-clients/CAMVt_Aw945uqcpisFjZHAR5m8Sidw6hW4ia%2B7%3DjxEfadmBPzcw%40mail.gmail.com.
>
>
>
> --
>
> Thanks!
> --Vahid
>
> --
> You received this message because you are subscribed to the Google Groups 
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to kafka-clients+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/kafka-clients/CAHR2v2mKJtHG6S9P%3Dmw08SxbWjQCowp8cpZNpzr9acW1EcdegQ%40mail.gmail.com.

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

[Ivan Yurchenko]

Do I understand correctly, that non-commiters can also vote, despite their
votes don't decide?

If so, then +1 from me.

Ivan


On Mon, 18 Nov 2019 at 15:19, Ismael Juma  wrote:

> Hi all,
>
> People seemed supportive in general, so I'd like to start a vote on
> KIP-531:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-531%3A+Drop+support+for+Scala+2.11+in+Kafka+2.5
>
> Ismael
>

[VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

[Ismael Juma]

Hi all,

People seemed supportive in general, so I'd like to start a vote on KIP-531:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-531%3A+Drop+support+for+Scala+2.11+in+Kafka+2.5

Ismael

[VOTE] KIP-518: Allow listing consumer groups per state

[Mickael Maison]

Hi all,

I'd like to start the vote on KIP-518:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-518%3A+Allow+listing+consumer+groups+per+state

Thanks

[jira] [Created] (KAFKA-9204) ReplaceField transformation fails when encountering tombstone event

[Georgios Kalogiros (Jira)]

Georgios Kalogiros created KAFKA-9204:
-

 Summary: ReplaceField transformation fails when encountering 
tombstone event
 Key: KAFKA-9204
 URL: https://issues.apache.org/jira/browse/KAFKA-9204
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Georgios Kalogiros
 Fix For: 2.3.0


When applying the {{ReplaceField}} transformation to a tombstone event, an 
exception is raised:

 
{code:java}
org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error 
handler
at 
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)
at 
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
at 
org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:50)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:506)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:464)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:320)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at 
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.connect.errors.DataException: Only Map objects 
supported in absence of schema for [field replacement], found: null
at 
org.apache.kafka.connect.transforms.util.Requirements.requireMap(Requirements.java:38)
at 
org.apache.kafka.connect.transforms.ReplaceField.applySchemaless(ReplaceField.java:134)
at 
org.apache.kafka.connect.transforms.ReplaceField.apply(ReplaceField.java:127)
at 
org.apache.kafka.connect.runtime.TransformationChain.lambda$apply$0(TransformationChain.java:50)
at 
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)
at 
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)
... 14 more
{code}
There was a similar bug for the InsertField transformation that got merged in 
recently:
https://issues.apache.org/jira/browse/KAFKA-8523

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-2.4-jdk8 #77

[Apache Jenkins Server]

See 


Changes:

[ismael] KAFKA-9200: ListOffsetRequest missing error response for v5 (#7704)


--
[...truncated 2.69 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 

[jira] [Created] (KAFKA-9203) kafka-client 2.3.1 fails to consume lz4 compressed topic in kafka 2.1.1

[David Watzke (Jira)]

David Watzke created KAFKA-9203:
---

 Summary: kafka-client 2.3.1 fails to consume lz4 compressed topic 
in kafka 2.1.1
 Key: KAFKA-9203
 URL: https://issues.apache.org/jira/browse/KAFKA-9203
 Project: Kafka
  Issue Type: Bug
  Components: compression, consumer
Affects Versions: 2.3.0, 2.3.1
Reporter: David Watzke


I run kafka cluster 2.1.1

when I upgraded a client app to use kafka-client 2.3.0 (or 2.3.1) instead of 
2.2.0, I immediately started getting the following exceptions in a loop:

{noformat}

2019-11-18 11:59:16.888 ERROR [afka-consumer-thread] 
com.avast.utils2.kafka.consumer.ConsumerThread: Unexpected exception occurred 
while polling and processing messages: org.apache.kafka.common.KafkaExce
ption: Received exception when fetching the next record from 
FILEREP_LONER_REQUEST-4. If needed, please seek past the record to continue 
consumption. 
org.apache.kafka.common.KafkaException: Received exception when fetching the 
next record from FILEREP_LONER_REQUEST-4. If needed, please seek past the 
record to continue consumption. 
    at 
org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.fetchRecords(Fetcher.java:1473)
 
    at 
org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.access$1600(Fetcher.java:1332)
 
    at 
org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:645)
 
    at 
org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:606)
 
    at 
org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1263)
 
    at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225) 
    at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201) 
    at 
com.avast.utils2.kafka.consumer.ConsumerThread.pollAndProcessMessagesFromKafka(ConsumerThread.java:180)
 
    at 
com.avast.utils2.kafka.consumer.ConsumerThread.run(ConsumerThread.java:149) 
    at 
com.avast.filerep.saver.RequestSaver$.$anonfun$main$8(RequestSaver.scala:28) 
    at 
com.avast.filerep.saver.RequestSaver$.$anonfun$main$8$adapted(RequestSaver.scala:19)
 
    at 
resource.AbstractManagedResource.$anonfun$acquireFor$1(AbstractManagedResource.scala:88)
 
    at 
scala.util.control.Exception$Catch.$anonfun$either$1(Exception.scala:252) 
    at scala.util.control.Exception$Catch.apply(Exception.scala:228) 
    at scala.util.control.Exception$Catch.either(Exception.scala:252) 
    at 
resource.AbstractManagedResource.acquireFor(AbstractManagedResource.scala:88) 
    at 
resource.ManagedResourceOperations.apply(ManagedResourceOperations.scala:26) 
    at 
resource.ManagedResourceOperations.apply$(ManagedResourceOperations.scala:26) 
    at 
resource.AbstractManagedResource.apply(AbstractManagedResource.scala:50) 
    at 
resource.ManagedResourceOperations.acquireAndGet(ManagedResourceOperations.scala:25)
 
    at 
resource.ManagedResourceOperations.acquireAndGet$(ManagedResourceOperations.scala:25)
 
    at 
resource.AbstractManagedResource.acquireAndGet(AbstractManagedResource.scala:50)
 
    at 
resource.ManagedResourceOperations.foreach(ManagedResourceOperations.scala:53) 
    at 
resource.ManagedResourceOperations.foreach$(ManagedResourceOperations.scala:53) 
    at 
resource.AbstractManagedResource.foreach(AbstractManagedResource.scala:50) 
    at 
com.avast.filerep.saver.RequestSaver$.$anonfun$main$6(RequestSaver.scala:19) 
    at 
com.avast.filerep.saver.RequestSaver$.$anonfun$main$6$adapted(RequestSaver.scala:18)
 
    at 
resource.AbstractManagedResource.$anonfun$acquireFor$1(AbstractManagedResource.scala:88)
 
    at 
scala.util.control.Exception$Catch.$anonfun$either$1(Exception.scala:252) 
    at scala.util.control.Exception$Catch.apply(Exception.scala:228) 
    at scala.util.control.Exception$Catch.either(Exception.scala:252) 
    at 
resource.AbstractManagedResource.acquireFor(AbstractManagedResource.scala:88) 
    at 
resource.ManagedResourceOperations.apply(ManagedResourceOperations.scala:26) 
    at 
resource.ManagedResourceOperations.apply$(ManagedResourceOperations.scala:26) 
    at 
resource.AbstractManagedResource.apply(AbstractManagedResource.scala:50) 
    at 
resource.ManagedResourceOperations.acquireAndGet(ManagedResourceOperations.scala:25)
 
    at 
resource.ManagedResourceOperations.acquireAndGet$(ManagedResourceOperations.scala:25)
 
    at 
resource.AbstractManagedResource.acquireAndGet(AbstractManagedResource.scala:50)
 
    at 
resource.ManagedResourceOperations.foreach(ManagedResourceOperations.scala:53) 
    at 
resource.ManagedResourceOperations.foreach$(ManagedResourceOperations.scala:53) 
    at 

Build failed in Jenkins: kafka-trunk-jdk8 #4049

[Apache Jenkins Server]

See 


Changes:

[ismael] KAFKA-9200: ListOffsetRequest missing error response for v5 (#7704)


--
[...truncated 2.74 MB...]

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest >