date:20200225

Build failed in Jenkins: kafka-trunk-jdk11 #1193

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[github] HOTFIX: compile error in EmbeddedKafkaCluster (#8170)

[manikumar] KAFKA-9594: Add a separate lock to pause the follower log append 
while


--
[...truncated 2.89 MB...]
org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task

Build failed in Jenkins: kafka-trunk-jdk8 #4272

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-9602: Close the stream internal producer only in EOS (#8166)


--
[...truncated 2.10 MB...]
org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:compileTestJava
> Task :streams:upgrade-system-tests-0101:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:testClasses
> Task :streams:upgrade-system-tests-0101:checkstyleTest
> Task :streams:upgrade-system-tests-0101:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:test
> Task :streams:upgrade-system-tests-0102:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0102:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0102:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0102:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0102:compileTestJava
> Task

Build failed in Jenkins: kafka-trunk-jdk11 #1192

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-9602: Close the stream internal producer only in EOS (#8166)


--
[...truncated 2.12 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest >

[jira] [Resolved] (KAFKA-9594) speed up the processing of LeaderAndIsrRequest

2020-02-25 Thread Manikumar (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-9594.
--
Resolution: Fixed

Issue resolved by pull request 8153
[https://github.com/apache/kafka/pull/8153]

> speed up the processing of LeaderAndIsrRequest
> --
>
> Key: KAFKA-9594
> URL: https://issues.apache.org/jira/browse/KAFKA-9594
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Rao
>Assignee: Manikumar
>Priority: Minor
> Fix For: 2.6.0
>
>
> Observations from [~junrao]
> Currently, Partition.makerFollower() holds a write lock on 
> leaderIsrUpdateLock. Partition.doAppendRecordsToFollowerOrFutureReplica() 
> holds a read lock on leaderIsrUpdateLock. So, if there is an ongoing log 
> append on the follower, the makeFollower() call will be delayed. This path is 
> a bit different when serving the Partition.makeLeader() call. Before we make 
> a call on Partition.makerLeader(), we first remove the follower from the 
> replicaFetcherThread. So, the makerLeader() call won't be delayed because of 
> log append. This means that when we change one follower to become leader and 
> another follower to follow the new leader during a controlled shutdown, the 
> makerLeader() call typically completes faster than the makeFollower() call, 
> which can delay the follower fetching from the new leader and cause ISR to 
> shrink.
> This only reason that Partition.doAppendRecordsToFollowerOrFutureReplica() 
> needs to hold a read lock on leaderIsrUpdateLock is for 
> Partiiton.maybeReplaceCurrentWithFutureReplica() to pause the log append 
> while checking if the log dir could be replaced. We could potentially add a 
> separate lock (sth like futureLogLock) that's synced between 
> maybeReplaceCurrentWithFutureReplica() and 
> doAppendRecordsToFollowerOrFutureReplica(). Then, 
> doAppendRecordsToFollowerOrFutureReplica() doesn't need to hold the lock on 
> leaderIsrUpdateLock.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-2.4-jdk8 #155

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[bill] Revert "KAFKA-9533: ValueTransform forwards `null` values (#8108)"


--
[...truncated 5.58 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

[jira] [Resolved] (KAFKA-9602) Incorrect close of producer instance during partition assignment

2020-02-25 Thread Guozhang Wang (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-9602.
--
Resolution: Fixed

> Incorrect close of producer instance during partition assignment
> 
>
> Key: KAFKA-9602
> URL: https://issues.apache.org/jira/browse/KAFKA-9602
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
>
> The new StreamProducer instance close doesn't distinguish between an 
> EOS/non-EOS shutdown. The StreamProducer should take care of that.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [KAFKA-557] Add emit on change support for Kafka Streams

2020-02-25 Thread Richard Yu

Hi John,

Sounds goods. It looks like we are close to wrapping things up. If there
isn't any other revisions which needs to be made. (If so, please comment in
the thread)
I will start the voting process this Thursday (Pacific Standard Time).

Cheers,
Richard

On Tue, Feb 25, 2020 at 11:59 AM John Roesler  wrote:

> Hi Richard,
>
> Sorry for the slow reply. I actually think we should avoid checking
> equals() for now. Your reasoning is good, but the truth is that
> depending on the implementation of equals() is non-trivial,
> semantically, and (though I proposed it before), I'm not convinced
> it's worth the risk. Much better to start with exactly one kind of
> "idempotence detection".
>
> Even if someone does update their serdes, we know that the new
> serde would still be able to _de_serialize the old format, or the whole
> app would break. The situation is that the new result gets encoded
> in the new binary format, which means we don't detect an idempotent
> update for what it is. In this case, we'd write the new binary format to
> disk and the changelog, and forward it downstream. However, we only
> do this once. Now that the binary format for that record has been updated,
> we would correctly detect idempotence of any subsequent updates.
>
> Plus, we would still be able to filter out idempotent updates in
> repartition
> sinks, since for those, we use the new serde to serialize both the "old"
> and
> "new" result.
>
> It's certainly a good observation, but I think we can just make a note of
> it
> in "rejected alternatives" for now, and plan to refine it later, if it does
> pose a big performance problem.
>
> Thanks!
> -John
>
> On Sat, Feb 22, 2020, at 18:14, Richard Yu wrote:
> > Hi all,
> >
> > Updated the KIP.
> >
> > Just a question: do you think it would be a good idea if we check for
> both
> > Object#equals() and binary equality?
> > Because there might be some subtle changes in the serialization (for
> > example, if the user decides to upgrade their serialization procedure to
> a
> > new one), but the underlying values of the result might be the same.
> > (therefore equals() might return true)
> >
> > Do you think this would be plausible?
> >
> > Cheers,
> > Richard
> >
> > On Fri, Feb 21, 2020 at 2:37 PM Richard Yu 
> > wrote:
> >
> > > Hello,
> > >
> > > Just to make some updates. I changed the name of the metric so that it
> was
> > > more in line with usual Kafka naming conventions for metrics / sensors.
> > > Below is the updated description of the metric:
> > >
> > > dropped-idempotent-updates : (Level 2 - Per Task) DEBUG (rate | total)
> > >
> > > Description: This metric will record the number of updates that have
> been
> > > dropped since they are essentially re-performing an earlier operation.
> > >
> > > Note:
> > >
> > >- The rate option indicates the ratio of records dropped to actual
> > >volume of records passing through the task.
> > >- The total option will just give a raw count of the number of
> records
> > >dropped.
> > >
> > >
> > > I hope that this is more on point.
> > >
> > > Best,
> > > Richard
> > >
> > > On Fri, Feb 21, 2020 at 2:20 PM Richard Yu  >
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> Thanks for the clarification. I was just confused a little on what was
> > >> going on.
> > >>
> > >> So I guess then that for the actual proposal. We got the following:
> > >>
> > >> 1. We check for binary equality, and perform no extra look ups.
> > >> 2. Emphasize that this applies only to materialized tables.
> > >> 3. We drop aggregation updates if key, value and timestamp is the
> same.
> > >>
> > >> Then that settles the behavior changes. So it looks like the Metric
> that
> > >> is the only thing that is left. In this case, I think the metric
> would be
> > >> named the following: IdempotentUpdateMetric. This is mostly off the
> top of
> > >> my head. So if you think that we change it, feel free to say so.
> > >> The metric will report the number of dropped operations inherently.
> > >>
> > >> It will probably be added as a Sensor, similar to the dropped records
> > >> sensor we already have.
> > >>
> > >> If there isn't anything else, I will probably start the voting process
> > >> next week!
> > >>
> > >> Cheers,
> > >> Richard
> > >>
> > >>
> > >> On Fri, Feb 21, 2020 at 11:23 AM John Roesler 
> > >> wrote:
> > >>
> > >>> Hi Bruno,
> > >>>
> > >>> Thanks for the clarification. Indeed, I was thinking two things:
> > >>> 1. For the initial implementation, we can just avoid adding any extra
> > >>> lookups, but only do the comparison when we already happen to have
> > >>> the prior value.
> > >>> 2. I think, as a result of the timestamp semantics, we actually _do_
> look
> > >>> up the prior value approximately all the time, so the idempotence
> check
> > >>> should be quite effective.
> > >>>
> > >>> I think that second point is the same thing you're referring to
> > >>> potentially
> > >>> being unnecessary. It does mean that we do fetch the

Build failed in Jenkins: kafka-trunk-jdk11 #1191

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-9498; Topic validation during the topic creation triggers


--
[...truncated 2.12 MB...]

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:compileTestJava
> Task

Build failed in Jenkins: kafka-trunk-jdk8 #4271

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-9498; Topic validation during the topic creation triggers


--
[...truncated 2.10 MB...]
org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:compileTestJava
> Task :streams:upgrade-system-tests-0101:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:testClasses
> Task :streams:upgrade-system-tests-0101:checkstyleTest
> Task :streams:upgrade-system-tests-0101:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:test
> Task :streams:upgrade-system-tests-0102:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0102:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0102:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0102:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0102:compileTestJava
> Task

[jira] [Resolved] (KAFKA-9498) Topic validation during the creation trigger unnecessary TopicChange events

2020-02-25 Thread Jun Rao (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-9498.

Fix Version/s: 2.6.0
   Resolution: Fixed

Merged the PR to trunk.

> Topic validation during the creation trigger unnecessary TopicChange events 
> 
>
> Key: KAFKA-9498
> URL: https://issues.apache.org/jira/browse/KAFKA-9498
> Project: Kafka
>  Issue Type: Bug
>Reporter: David Jacot
>Assignee: David Jacot
>Priority: Major
> Fix For: 2.6.0
>
>
> I have found out that the topic validation logic, which is executed when 
> CreateTopicPolicy or when validateOnly is set, triggers unnecessary 
> ChangeTopic events in the controller. In the worst case, it can trigger up to 
> one event per created topic and leads to overloading the controller.
> This happens because the validation logic reads all the topics from ZK using 
> the method getAllTopicsInCluster provided by the KafkaZKClient. This method 
> registers a watch every time the topics are read from Zookeeper.
> I think that we should make the watch registration optional for this call in 
> oder to avoid this unwanted behaviour.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk8 #4270

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: don't assign standby tasks with no logged state (#8147)


--
[...truncated 2.87 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task

Build failed in Jenkins: kafka-trunk-jdk11 #1190

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: don't assign standby tasks with no logged state (#8147)


--
[...truncated 2.89 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest >

[jira] [Resolved] (KAFKA-9572) Sum Computation with Exactly-Once Enabled and Injected Failures Misses Some Records

2020-02-25 Thread Guozhang Wang (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-9572.
--
Fix Version/s: (was: 2.5.0)
   2.6.0
   Resolution: Fixed

> Sum Computation with Exactly-Once Enabled and Injected Failures Misses Some 
> Records
> ---
>
> Key: KAFKA-9572
> URL: https://issues.apache.org/jira/browse/KAFKA-9572
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 2.4.0
>Reporter: Bruno Cadonna
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 2.6.0
>
> Attachments: 7-changelog-1.txt, data-1.txt, streams22.log, 
> streams23.log, streams30.log, sum-1.txt
>
>
> System test {{StreamsEosTest.test_failure_and_recovery}} failed due to a 
> wrongly computed aggregation under exactly-once (EOS). The specific error is:
> {code:java}
> Exception in thread "main" java.lang.RuntimeException: Result verification 
> failed for ConsumerRecord(topic = sum, partition = 1, leaderEpoch = 0, offset 
> = 2805, CreateTime = 1580719595164, serialized key size = 4, serialized value 
> size = 8, headers = RecordHeaders(headers = [], isReadOnly = false), key = 
> [B@6c779568, value = [B@f381794) expected <6069,17269> but was <6069,10698>
>   at 
> org.apache.kafka.streams.tests.EosTestDriver.verifySum(EosTestDriver.java:444)
>   at 
> org.apache.kafka.streams.tests.EosTestDriver.verify(EosTestDriver.java:196)
>   at 
> org.apache.kafka.streams.tests.StreamsEosTest.main(StreamsEosTest.java:69)
> {code} 
> That means, the sum computed by the Streams app seems to be wrong for key 
> 6069. I checked the dumps of the log segments of the input topic partition 
> (attached: data-1.txt) and indeed two input records are not considered in the 
> sum. With those two missed records the sum would be correct. More concretely, 
> the input values for key 6069 are:
> # 147
> # 9250
> # 5340 
> # 1231
> # 1301
> The sum of this values is 17269 as stated in the exception above as expected 
> sum. If you subtract values 3 and 4, i.e., 5340 and 1231 from 17269, you get 
> 10698 , which is the actual sum in the exception above. Somehow those two 
> values are missing.
> In the log dump of the output topic partition (attached: sum-1.txt), the sum 
> is correct until the 4th value 1231 , i.e. 15968, then it is overwritten with 
> 10698.
> In the log dump of the changelog topic of the state store that stores the sum 
> (attached: 7-changelog-1.txt), the sum is also overwritten as in the output 
> topic.
> I attached the logs of the three Streams instances involved.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9610) Should not throw illegal state exception during task revocation

2020-02-25 Thread Boyang Chen (Jira)

Boyang Chen created KAFKA-9610:
--

 Summary: Should not throw illegal state exception during task 
revocation
 Key: KAFKA-9610
 URL: https://issues.apache.org/jira/browse/KAFKA-9610
 Project: Kafka
  Issue Type: Bug
Reporter: Boyang Chen
Assignee: Boyang Chen






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9609) Memory Leak in Kafka Producer

2020-02-25 Thread Satish (Jira)

Satish created KAFKA-9609:
-

 Summary: Memory Leak in Kafka Producer
 Key: KAFKA-9609
 URL: https://issues.apache.org/jira/browse/KAFKA-9609
 Project: Kafka
  Issue Type: Bug
  Components: clients, producer 
Affects Versions: 2.4.0
Reporter: Satish


org.apache.kafka.clients.producer.internals.Sender adds Topic Metrics for every 
topic that we are writing messages to but it never been cleaned up until we 
close the producer.

This is an issue if we use single producer and have more number of Dynamic 
topics (eg: ~ 500 topics per hour) and writing messages to them.  As this 
Metrics map is getting accumulated for every topic, over a period of time we 
notice the memory usage gets increased gradually. 

It can be easily reproducible by writing messages to the more # of dynamic 
topics using the same KafkaProducer from apache kafka client libraries or 
KafkaTemplate from Spring.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk8 #4269

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-8147: Add changelog topic configuration to KTable suppress 
(#8029)


--
[...truncated 2.87 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain

Build failed in Jenkins: kafka-trunk-jdk11 #1189

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[github] Revert "KAFKA-9533: ValueTransform forwards `null` values (#8108)"

[github] KAFKA-8147: Add changelog topic configuration to KTable suppress 
(#8029)


--
[...truncated 2.89 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled =

Build failed in Jenkins: kafka-2.4-jdk8 #154

2020-02-25 Thread Apache Jenkins Server

See 

Changes:


--
[...truncated 5.57 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

[jira] [Created] (KAFKA-9608) An EOS model simulation test

2020-02-25 Thread Boyang Chen (Jira)

Boyang Chen created KAFKA-9608:
--

 Summary: An EOS model simulation test
 Key: KAFKA-9608
 URL: https://issues.apache.org/jira/browse/KAFKA-9608
 Project: Kafka
  Issue Type: Improvement
Reporter: Boyang Chen
Assignee: Boyang Chen






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-2.5-jdk8 #44

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[bill] Revert "KAFKA-9533: ValueTransform forwards `null` values (#8108)"


--
[...truncated 2.91 MB...]
org.apache.kafka.streams.MockProcessorContextTest > shouldCapturePunctuator 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task

[jira] [Created] (KAFKA-9607) Should not clear partition group if the task will be revived again

2020-02-25 Thread Boyang Chen (Jira)

Boyang Chen created KAFKA-9607:
--

 Summary: Should not clear partition group if the task will be 
revived again
 Key: KAFKA-9607
 URL: https://issues.apache.org/jira/browse/KAFKA-9607
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Boyang Chen
Assignee: Boyang Chen


We detected an issue with a corrupted task failed to revive:
{code:java}
[2020-02-25T08:23:38-08:00] 
(streams-soak-trunk_soak_i-06305ad57801079ae_streamslog) [2020-02-25 
16:23:38,137] INFO 
[stream-soak-test-8f2124ec-8bd0-410a-8e3d-f202a32ab774-StreamThread-1] 
stream-thread 
[stream-soak-test-8f2124ec-8bd0-410a-8e3d-f202a32ab774-StreamThread-1] Handle 
new assignment with:
        New active tasks: [0_0, 3_1]
        New standby tasks: []
        Existing active tasks: [0_0]
        Existing standby tasks: [] 
(org.apache.kafka.streams.processor.internals.TaskManager)
[2020-02-25T08:23:38-08:00] 
(streams-soak-trunk_soak_i-06305ad57801079ae_streamslog) [2020-02-25 
16:23:38,138] INFO 
[stream-soak-test-8f2124ec-8bd0-410a-8e3d-f202a32ab774-StreamThread-1] 
[Consumer 
clientId=stream-soak-test-8f2124ec-8bd0-410a-8e3d-f202a32ab774-StreamThread-1-consumer,
 groupId=stream-soak-test] Adding newly assigned partitions: 
k8sName-id-repartition-1 
(org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2020-02-25T08:23:38-08:00] 
(streams-soak-trunk_soak_i-06305ad57801079ae_streamslog) [2020-02-25 
16:23:38,138] INFO 
[stream-soak-test-8f2124ec-8bd0-410a-8e3d-f202a32ab774-StreamThread-1] 
stream-thread 
[stream-soak-test-8f2124ec-8bd0-410a-8e3d-f202a32ab774-StreamThread-1] State 
transition from RUNNING to PARTITIONS_ASSIGNED 
(org.apache.kafka.streams.processor.internals.StreamThread)
[2020-02-25T08:23:39-08:00] 
(streams-soak-trunk_soak_i-06305ad57801079ae_streamslog) [2020-02-25 
16:23:38,419] WARN 
[stream-soak-test-8f2124ec-8bd0-410a-8e3d-f202a32ab774-StreamThread-1] 
stream-thread 
[stream-soak-test-8f2124ec-8bd0-410a-8e3d-f202a32ab774-StreamThread-1] 
Encountered org.apache.kafka.clients.consumer.OffsetOutOfRangeException 
fetching records from restore consumer for partitions 
[stream-soak-test-KSTREAM-AGGREGATE-STATE-STORE-49-changelog-1], it is 
likely that the consumer's position has fallen out of the topic partition 
offset range because the topic was truncated or compacted on the broker, 
marking the corresponding tasks as corrupted and re-initializingit later. 
(org.apache.kafka.streams.processor.internals.StoreChangelogReader)
[2020-02-25T08:23:38-08:00] 
(streams-soak-trunk_soak_i-06305ad57801079ae_streamslog) [2020-02-25 
16:23:38,139] INFO 
[stream-soak-test-8f2124ec-8bd0-410a-8e3d-f202a32ab774-StreamThread-1] 
[Consumer 
clientId=stream-soak-test-8f2124ec-8bd0-410a-8e3d-f202a32ab774-StreamThread-1-consumer,
 groupId=stream-soak-test] Setting offset for partition 
k8sName-id-repartition-1 to the committed offset FetchPosition{offset=3592242, 
offsetEpoch=Optional.empty, 
currentLeader=LeaderAndEpoch{leader=Optional[ip-172-31-25-115.us-west-2.compute.internal:9092
 (id: 1003 rack: null)], epoch=absent}} 
(org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2020-02-25T08:23:39-08:00] 
(streams-soak-trunk_soak_i-06305ad57801079ae_streamslog) [2020-02-25 
16:23:38,463] ERROR 
[stream-soak-test-8f2124ec-8bd0-410a-8e3d-f202a32ab774-StreamThread-1] 
stream-thread 
[stream-soak-test-8f2124ec-8bd0-410a-8e3d-f202a32ab774-StreamThread-1] 
Encountered the following exception during processing and the thread is going 
to shut down:  (org.apache.kafka.streams.processor.internals.StreamThread)
[2020-02-25T08:23:39-08:00] 
(streams-soak-trunk_soak_i-06305ad57801079ae_streamslog) 
java.lang.IllegalStateException: Partition k8sName-id-repartition-1 not found.
        at 
org.apache.kafka.streams.processor.internals.PartitionGroup.setPartitionTime(PartitionGroup.java:99)
        at 
org.apache.kafka.streams.processor.internals.StreamTask.initializeTaskTime(StreamTask.java:651)
        at 
org.apache.kafka.streams.processor.internals.StreamTask.initializeMetadata(StreamTask.java:631)
        at 
org.apache.kafka.streams.processor.internals.StreamTask.completeRestoration(StreamTask.java:209)
        at 
org.apache.kafka.streams.processor.internals.TaskManager.tryToCompleteRestoration(TaskManager.java:270)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:834)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:725)
{code}
The root cause is that we accidentally cleanup the partition group map so that 
next time we reboot the task it would miss input partitions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk8 #4268

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-9308: Reworded the ssl part of the security documentation  
(#8009)

[github] MINOR: Consistent terminal period in Errors.defaultExceptionMessage

[github] Revert "KAFKA-9533: ValueTransform forwards `null` values (#8108)"


--
[...truncated 2.87 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

2020-02-25 Thread Jun Rao

Hi, Satish,

Thanks for the updated doc. The new API seems to be an improvement overall.
A few more comments below.

100. For each of the operations related to tiering, it would be useful to
provide a description on how it works with the new API. These include
things like consumer fetch, replica fetch, offsetForTimestamp, retention
(remote and local) by size, time and logStartOffset, topic deletion, etc.
This will tell us if the proposed APIs are sufficient.

101. For the default implementation based on internal topic, is it meant as
a proof of concept or for production usage? I assume that it's the former.
However, if it's the latter, then the KIP needs to describe the design in
more detail.

102. When tiering a segment, the segment is first written to the object
store and then its metadata is written to RLMM using the api "void
putRemoteLogSegmentData()".
One potential issue with this approach is that if the system fails after
the first operation, it leaves a garbage in the object store that's never
reclaimed. One way to improve this is to have two separate APIs, sth like
preparePutRemoteLogSegmentData() and commitPutRemoteLogSegmentData().

103. It seems that the transactional support and the ability to read from
follower are missing.

104. It would be useful to provide a testing plan for this KIP.

Thanks,

Jun

On Mon, Feb 24, 2020 at 8:10 AM Satish Duggana 
wrote:

> Hi Jun,
> Please look at the earlier reply and let us know your comments.
>
> Thanks,
> Satish.
>
> On Wed, Feb 12, 2020 at 4:06 PM Satish Duggana 
> wrote:
> >
> > Hi Jun,
> > Thanks for your comments on the separation of remote log metadata
> > storage and remote log storage.
> > We had a few discussions since early Jan on how to support eventually
> > consistent stores like S3 by uncoupling remote log segment metadata
> > and remote log storage. It is written with details in the doc here(1).
> > Below is the brief summary of the discussion from that doc.
> >
> > The current approach consists of pulling the remote log segment
> > metadata from remote log storage APIs. It worked fine for storages
> > like HDFS. But one of the problems of relying on the remote storage to
> > maintain metadata is that tiered-storage needs to be strongly
> > consistent, with an impact not only on the metadata(e.g. LIST in S3)
> > but also on the segment data(e.g. GET after a DELETE in S3). The cost
> > of maintaining metadata in remote storage needs to be factored in.
> > This is true in the case of S3, LIST APIs incur huge costs as you
> > raised earlier.
> > So, it is good to separate the remote storage from the remote log
> > metadata store. We refactored the existing RemoteStorageManager and
> > introduced RemoteLogMetadataManager. Remote log metadata store should
> > give strong consistency semantics but remote log storage can be
> > eventually consistent.
> > We can have a default implementation for RemoteLogMetadataManager
> > which uses an internal topic(as mentioned in one of our earlier
> > emails) as storage. But users can always plugin their own
> > RemoteLogMetadataManager implementation based on their environment.
> >
> > Please go through the updated KIP and let us know your comments. We
> > have started refactoring for the changes mentioned in the KIP and
> > there may be a few more updates to the APIs.
> >
> > [1]
> https://docs.google.com/document/d/1qfkBCWL1e7ZWkHU7brxKDBebq4ie9yK20XJnKbgAlew/edit?ts=5e208ec7#
> >
> > On Fri, Dec 27, 2019 at 5:43 PM Ivan Yurchenko 
> wrote:
> > >
> > > Hi all,
> > >
> > >
> > > Jun:
> > > > (a) Cost: S3 list object requests cost $0.005 per 1000 requests. If
> you
> > > > have 100,000 partitions and want to pull the metadata for each
> partition
> > > at
> > > > the rate of 1/sec. It can cost $0.5/sec, which is roughly $40K per
> day.
> > >
> > > I want to note here, that no reasonably durable storage will be cheap
> > > at 100k RPS. For example, DynamoDB might give the same ballpark
> figures.
> > > If we want to keep the pull-based approach, we can try to reduce this
> number
> > > in several ways: doing listings less frequently (as Satish mentioned,
> > > with the current defaults it's ~3.33k RPS for your example),
> > > batching listing operations in some way (depending on the storage;
> > > it might require the change of RSM's interface).
> > >
> > >
> > > > There are different ways for doing push based metadata propagation.
> Some
> > > > object stores may support that already. For example, S3 supports
> events
> > > > notification
> > > This sounds interesting. However, I see a couple of issues using it:
> > >   1. As I understand the documentation, notification delivery is not
> > > guaranteed
> > > and it's recommended to periodically do LIST to fill the gaps.
> > > Which brings us back to the same LIST consistency guarantees issue.
> > >   2. The same goes for the broker start: to get the current state, we
> need
> > > to LIST.
> > >   3. The dynamic set of multiple consumers (RSMs):

Build failed in Jenkins: kafka-2.2-jdk8-old #213

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[bill] Revert "KAFKA-9533: ValueTransform forwards `null` values (#8108)"


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H22 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/2.2^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/2.2^{commit} # timeout=10
Checking out Revision 7bab36dc97c30043fbbd09174e056ec7ed8e4f43 
(refs/remotes/origin/2.2)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7bab36dc97c30043fbbd09174e056ec7ed8e4f43
Commit message: "Revert "KAFKA-9533: ValueTransform forwards `null` values 
(#8108)""
 > git rev-list --no-walk 5fbe4472bd42ed2b3b6e35b6d0bdc5238a6d0868 # timeout=10
ERROR: No tool found matching GRADLE_4_8_1_HOME
[kafka-2.2-jdk8-old] $ /bin/bash -xe /tmp/jenkins900754750247808.sh
+ rm -rf 
+ /bin/gradle
/tmp/jenkins900754750247808.sh: line 4: /bin/gradle: No such file or 
directory
Build step 'Execute shell' marked build as failure
[FINDBUGS] Collecting findbugs analysis files...
ERROR: No tool found matching GRADLE_4_8_1_HOME
[FINDBUGS] Searching for all files in 
 that match the pattern 
**/build/reports/findbugs/*.xml
[FINDBUGS] No files found. Configuration error?
ERROR: No tool found matching GRADLE_4_8_1_HOME
No credentials specified
ERROR: No tool found matching GRADLE_4_8_1_HOME
 Using GitBlamer to create author and commit information for all 
warnings.
 GIT_COMMIT=7bab36dc97c30043fbbd09174e056ec7ed8e4f43, 
workspace=
[FINDBUGS] Computing warning deltas based on reference build #175
Recording test results
ERROR: No tool found matching GRADLE_4_8_1_HOME
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
ERROR: No tool found matching GRADLE_4_8_1_HOME
Not sending mail to unregistered user nore...@github.com
Not sending mail to unregistered user ism...@juma.me.uk
Not sending mail to unregistered user b...@confluent.io
Not sending mail to unregistered user wangg...@gmail.com
Not sending mail to unregistered user j...@confluent.io

Re: [KAFKA-557] Add emit on change support for Kafka Streams

2020-02-25 Thread John Roesler

Hi Richard,

Sorry for the slow reply. I actually think we should avoid checking
equals() for now. Your reasoning is good, but the truth is that
depending on the implementation of equals() is non-trivial,
semantically, and (though I proposed it before), I'm not convinced
it's worth the risk. Much better to start with exactly one kind of
"idempotence detection".

Even if someone does update their serdes, we know that the new
serde would still be able to _de_serialize the old format, or the whole
app would break. The situation is that the new result gets encoded
in the new binary format, which means we don't detect an idempotent
update for what it is. In this case, we'd write the new binary format to
disk and the changelog, and forward it downstream. However, we only
do this once. Now that the binary format for that record has been updated,
we would correctly detect idempotence of any subsequent updates.

Plus, we would still be able to filter out idempotent updates in repartition
sinks, since for those, we use the new serde to serialize both the "old" and
"new" result.

It's certainly a good observation, but I think we can just make a note of it
in "rejected alternatives" for now, and plan to refine it later, if it does
pose a big performance problem.

Thanks!
-John

On Sat, Feb 22, 2020, at 18:14, Richard Yu wrote:
> Hi all,
> 
> Updated the KIP.
> 
> Just a question: do you think it would be a good idea if we check for both
> Object#equals() and binary equality?
> Because there might be some subtle changes in the serialization (for
> example, if the user decides to upgrade their serialization procedure to a
> new one), but the underlying values of the result might be the same.
> (therefore equals() might return true)
> 
> Do you think this would be plausible?
> 
> Cheers,
> Richard
> 
> On Fri, Feb 21, 2020 at 2:37 PM Richard Yu 
> wrote:
> 
> > Hello,
> >
> > Just to make some updates. I changed the name of the metric so that it was
> > more in line with usual Kafka naming conventions for metrics / sensors.
> > Below is the updated description of the metric:
> >
> > dropped-idempotent-updates : (Level 2 - Per Task) DEBUG (rate | total)
> >
> > Description: This metric will record the number of updates that have been
> > dropped since they are essentially re-performing an earlier operation.
> >
> > Note:
> >
> >- The rate option indicates the ratio of records dropped to actual
> >volume of records passing through the task.
> >- The total option will just give a raw count of the number of records
> >dropped.
> >
> >
> > I hope that this is more on point.
> >
> > Best,
> > Richard
> >
> > On Fri, Feb 21, 2020 at 2:20 PM Richard Yu 
> > wrote:
> >
> >> Hi all,
> >>
> >> Thanks for the clarification. I was just confused a little on what was
> >> going on.
> >>
> >> So I guess then that for the actual proposal. We got the following:
> >>
> >> 1. We check for binary equality, and perform no extra look ups.
> >> 2. Emphasize that this applies only to materialized tables.
> >> 3. We drop aggregation updates if key, value and timestamp is the same.
> >>
> >> Then that settles the behavior changes. So it looks like the Metric that
> >> is the only thing that is left. In this case, I think the metric would be
> >> named the following: IdempotentUpdateMetric. This is mostly off the top of
> >> my head. So if you think that we change it, feel free to say so.
> >> The metric will report the number of dropped operations inherently.
> >>
> >> It will probably be added as a Sensor, similar to the dropped records
> >> sensor we already have.
> >>
> >> If there isn't anything else, I will probably start the voting process
> >> next week!
> >>
> >> Cheers,
> >> Richard
> >>
> >>
> >> On Fri, Feb 21, 2020 at 11:23 AM John Roesler 
> >> wrote:
> >>
> >>> Hi Bruno,
> >>>
> >>> Thanks for the clarification. Indeed, I was thinking two things:
> >>> 1. For the initial implementation, we can just avoid adding any extra
> >>> lookups, but only do the comparison when we already happen to have
> >>> the prior value.
> >>> 2. I think, as a result of the timestamp semantics, we actually _do_ look
> >>> up the prior value approximately all the time, so the idempotence check
> >>> should be quite effective.
> >>>
> >>> I think that second point is the same thing you're referring to
> >>> potentially
> >>> being unnecessary. It does mean that we do fetch the whole value in a
> >>> lot of cases where we really only need the timestamp, so it could
> >>> certainly
> >>> be optimized in the future. In that future, we would need to weigh that
> >>> optimization against losing the idempotence check. But, that's a problem
> >>> for tomorrow :)
> >>>
> >>> I'm 100% on board with scrutinizing the performance as we implement
> >>> this feature.
> >>>
> >>> Thanks again,
> >>> -John
> >>>
> >>> On Thu, Feb 20, 2020, at 03:25, Bruno Cadonna wrote:
> >>> > Hi John,
> >>> >
> >>> > I am glad to help you with your imagination. With

[jira] [Resolved] (KAFKA-8147) Add changelog topic configuration to KTable suppress

2020-02-25 Thread John Roesler (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-8147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Roesler resolved KAFKA-8147.
-
Resolution: Fixed

Merged as 
https://github.com/apache/kafka/commit/90640266393b530107db8256d38ec5aeba4805e1

> Add changelog topic configuration to KTable suppress
> 
>
> Key: KAFKA-8147
> URL: https://issues.apache.org/jira/browse/KAFKA-8147
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 2.1.1
>Reporter: Maarten
>Assignee: highluck
>Priority: Minor
>  Labels: kip
> Fix For: 2.6.0
>
>
> The streams DSL does not provide a way to configure the changelog topic 
> created by KTable.suppress.
> From the perspective of an external user this could be implemented similar to 
> the configuration of aggregate + materialized, i.e.,
> {code:java}
> changelogTopicConfigs = // Configs
> materialized = Materialized.as(..).withLoggingEnabled(changelogTopicConfigs)
> ..
> KGroupedStream.aggregate(..,materialized)
> {code}
> [KIP-446: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-446%3A+Add+changelog+topic+configuration+to+KTable+suppress|https://cwiki.apache.org/confluence/display/KAFKA/KIP-446%3A+Add+changelog+topic+configuration+to+KTable+suppress]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1188

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-9308: Reworded the ssl part of the security documentation  
(#8009)

[github] MINOR: Consistent terminal period in Errors.defaultExceptionMessage


--
[...truncated 2.89 MB...]
org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
>

Build failed in Jenkins: kafka-trunk-jdk8 #4267

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[manikumar] KAFKA-9567: Docs, system tests for ZooKeeper 3.5.7


--
[...truncated 5.82 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task

[jira] [Created] (KAFKA-9606) Document Metrics Changes from KIP-444

2020-02-25 Thread Bruno Cadonna (Jira)

Bruno Cadonna created KAFKA-9606:


 Summary: Document Metrics Changes from KIP-444
 Key: KAFKA-9606
 URL: https://issues.apache.org/jira/browse/KAFKA-9606
 Project: Kafka
  Issue Type: Task
  Components: docs, streams
Affects Versions: 2.5.0
Reporter: Bruno Cadonna
 Fix For: 2.5.0


Changes introduced in KIP-444 shall be documented. See 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams|https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%253A+Augment+metrics+for+Kafka+Streams]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-2.5-jdk8 #43

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[manikumar] KAFKA-9567: Docs, system tests for ZooKeeper 3.5.7


--
[...truncated 2.90 MB...]

org.apache.kafka.streams.MockProcessorContextTest > shouldCapturePunctuator 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task

[jira] [Created] (KAFKA-9605) EOS Producer could throw illegal state if trying to complete a failed batch after fatal error

2020-02-25 Thread Boyang Chen (Jira)

Boyang Chen created KAFKA-9605:
--

 Summary: EOS Producer could throw illegal state if trying to 
complete a failed batch after fatal error
 Key: KAFKA-9605
 URL: https://issues.apache.org/jira/browse/KAFKA-9605
 Project: Kafka
  Issue Type: Bug
Reporter: Boyang Chen
Assignee: Boyang Chen


```

[2020-02-24T13:23:29-08:00] 
(streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 
21:23:28,673] ERROR [kafka-producer-network-thread | 
stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer]
 [Producer 
clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer,
 transactionalId=stream-soak-test-1_0] Aborting producer batches due to fatal 
error (org.apache.kafka.clients.producer.internals.Sender)

[2020-02-24T13:23:29-08:00] 
(streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) 
org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an 
operation with an old epoch. Either there is a newer producer with the same 
transactionalId, or the producer's transaction has been expired by the broker.

[2020-02-24T13:23:29-08:00] 
(streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 
21:23:28,674] INFO 
[stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3] 
[Producer 
clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-0_0-producer,
 transactionalId=stream-soak-test-0_0] Closing the Kafka producer with 
timeoutMillis = 9223372036854775807 ms. 
(org.apache.kafka.clients.producer.KafkaProducer)

[2020-02-24T13:23:29-08:00] 
(streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 
21:23:28,684] INFO [kafka-producer-network-thread | 
stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer]
 [Producer 
clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer,
 transactionalId=stream-soak-test-1_0] Resetting sequence number of batch with 
current sequence 354277 for partition windowed-node-counts-0 to 354276 
(org.apache.kafka.clients.producer.internals.TransactionManager)

[2020-02-24T13:23:29-08:00] 
(streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 
21:23:28,684] INFO [kafka-producer-network-thread | 
stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer]
 Resetting sequence number of batch with current sequence 354277 for partition 
windowed-node-counts-0 to 354276 
(org.apache.kafka.clients.producer.internals.ProducerBatch)

[2020-02-24T13:23:29-08:00] 
(streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 
21:23:28,685] ERROR [kafka-producer-network-thread | 
stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer]
 [Producer 
clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer,
 transactionalId=stream-soak-test-1_0] Uncaught error in request completion: 
(org.apache.kafka.clients.NetworkClient)

[2020-02-24T13:23:29-08:00] 
(streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) 
java.lang.IllegalStateException: Should not reopen a batch which is already 
aborted.

        at 
org.apache.kafka.common.record.MemoryRecordsBuilder.reopenAndRewriteProducerState(MemoryRecordsBuilder.java:295)

        at 
org.apache.kafka.clients.producer.internals.ProducerBatch.resetProducerState(ProducerBatch.java:395)

        at 
org.apache.kafka.clients.producer.internals.TransactionManager.lambda$adjustSequencesDueToFailedBatch$4(TransactionManager.java:770)

        at 
org.apache.kafka.clients.producer.internals.TransactionManager$TopicPartitionEntry.resetSequenceNumbers(TransactionManager.java:180)

        at 
org.apache.kafka.clients.producer.internals.TransactionManager.adjustSequencesDueToFailedBatch(TransactionManager.java:760)

        at 
org.apache.kafka.clients.producer.internals.TransactionManager.handleFailedBatch(TransactionManager.java:735)

        at 
org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:671)

        at 
org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:662)

        at 
org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:620)

        at 
org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:554)

        at 
org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:69)

        at 
org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:745)

        at 
org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)

        at 
org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:571)

        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:563)

        at

Build failed in Jenkins: kafka-trunk-jdk11 #1187

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[manikumar] KAFKA-9567: Docs, system tests for ZooKeeper 3.5.7


--
[...truncated 2.89 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

[jira] [Resolved] (KAFKA-9308) Misses SAN after certificate creation

2020-02-25 Thread Mickael Maison (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-9308.
---
Fix Version/s: 2.6.0
   Resolution: Fixed

> Misses SAN after certificate creation
> -
>
> Key: KAFKA-9308
> URL: https://issues.apache.org/jira/browse/KAFKA-9308
> Project: Kafka
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.3.1
>Reporter: Agostino Sarubbo
>Assignee: Sönke Liebau
>Priority: Minor
> Fix For: 2.6.0
>
>
> Hello,
> I followed the documentation to use kafka with ssl, however the entire 
> 'procedure' loses at the end the specified SAN.
> To test, run (after the first keytool command and after the latest):
>  
> {code:java}
> keytool -list -v -keystore server.keystore.jks
> {code}
> Reference:
>  [http://kafka.apache.org/documentation.html#security_ssl]
>  
> {code:java}
> #!/bin/bash
> #Step 1
> keytool -keystore server.keystore.jks -alias localhost -validity 365 -keyalg 
> RSA -genkey -ext SAN=DNS:test.test.com
> #Step 2
> openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
> keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
> keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
> #Step 3
> keytool -keystore server.keystore.jks -alias localhost -certreq -file 
> cert-file 
> openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed 
> -days 365 -CAcreateserial -passin pass:test1234 
> keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert 
> keytool -keystore server.keystore.jks -alias localhost -import -file 
> cert-signed
> {code}
>  
> In the detail, the SAN is losed after:
> {code:java}
> keytool -keystore server.keystore.jks -alias localhost -import -file 
> cert-signed
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (KAFKA-9533) ValueTransform forwards `null` values

2020-02-25 Thread John Roesler (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Roesler reopened KAFKA-9533:
-

> ValueTransform forwards `null` values
> -
>
> Key: KAFKA-9533
> URL: https://issues.apache.org/jira/browse/KAFKA-9533
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0, 0.10.2.2, 0.11.0.3, 1.1.1, 2.0.1, 2.2.2, 
> 2.4.0, 2.3.1
>Reporter: Michael Viamari
>Assignee: Michael Viamari
>Priority: Major
> Fix For: 2.2.3, 2.5.0, 2.3.2, 2.4.1
>
>
> According to the documentation for `KStream#transformValues`, nulls returned 
> from `ValueTransformer#transform` are not forwarded. (see 
> [KStream#transformValues|https://kafka.apache.org/24/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-]
> However, this does not appear to be the case. In 
> `KStreamTransformValuesProcessor#process` the result of the transform is 
> forwarded directly.
> {code:java}
>  @Override
>  public void process(final K key, final V value) {
>  context.forward(key, valueTransformer.transform(key, value));
>  }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9567) Docs and system tests for ZooKeeper 3.5.7 and KIP-515

2020-02-25 Thread Manikumar (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-9567.
--
Fix Version/s: 2.5.0
   Resolution: Fixed

Issue resolved by pull request 8132
[https://github.com/apache/kafka/pull/8132]

> Docs and system tests for ZooKeeper 3.5.7 and KIP-515
> -
>
> Key: KAFKA-9567
> URL: https://issues.apache.org/jira/browse/KAFKA-9567
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 2.5.0
>Reporter: Ron Dagostino
>Priority: Blocker
> Fix For: 2.5.0
>
>
> These changes depend on [KIP-515: Enable ZK client to use the new TLS 
> supported 
> authentication|https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication],
>  which was only added to 2.5.0.  The upgrade to ZooKeeper 3.5.7 was merged to 
> both 2.5.0 and 2.4.1 via https://issues.apache.org/jira/browse/KAFKA-9515, 
> but this change must only be merged to 2.5.0 (it will break the system tests 
> if merged to 2.4.1).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-5554) Hilight config settings for particular common use cases

2020-02-25 Thread Tom Bentley (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-5554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Bentley resolved KAFKA-5554.

Resolution: Abandoned

> Hilight config settings for particular common use cases
> ---
>
> Key: KAFKA-5554
> URL: https://issues.apache.org/jira/browse/KAFKA-5554
> Project: Kafka
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Tom Bentley
>Assignee: Tom Bentley
>Priority: Minor
>
> Judging by the sorts of questions seen on the mailling list, stack overflow 
> etc it seems common for users to assume that Kafka will default to settings 
> which won't lose messages. They start using Kafka and at some later time find 
> messages have been lost.
> While it's not our fault if users don't read the documentation, there's a lot 
> of configuration documentation to digest and it's easy for people to miss an 
> important setting.
> Therefore, I'd like to suggest that in addition to the current configuration 
> docs we add a short section highlighting those settings which pertain to 
> common use cases, such as:
> * configs to avoid lost messages
> * configs for low latency
> I'm sure some users will continue to not read the documentation, but when 
> they inevitably start asking questions it means people can respond with "have 
> you configured everything as described here?"



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-6359) Work for KIP-236

2020-02-25 Thread Tom Bentley (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Bentley resolved KAFKA-6359.

Resolution: Implemented

This was addressed by KIP-455 instead.

> Work for KIP-236
> 
>
> Key: KAFKA-6359
> URL: https://issues.apache.org/jira/browse/KAFKA-6359
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Tom Bentley
>Assignee: GEORGE LI
>Priority: Minor
>
> This issue is for the work described in KIP-236.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9604) Падение кластера

2020-02-25 Thread Maksim Larionov (Jira)

Maksim Larionov created KAFKA-9604:
--

 Summary: Падение кластера
 Key: KAFKA-9604
 URL: https://issues.apache.org/jira/browse/KAFKA-9604
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.2.1
Reporter: Maksim Larionov


Добрый день!

На одном из серверов в кластере произошло переполнение дискового пространства. 
При очистке по ошибке были удалены некоторые файлы *.log некоторых реплик в 
log.dirs. При достижении retention time сработала очистка и физический файл 
07607076.log не был найден. Брокер аварийно остановился.
[2020-02-06 13:32:48,965] INFO [Log partition=ocs.account-balances-12, 
dir=/data/ocswf/kafka_broker/kafka-data] Found deletable segments with base 
offsets [7607076] due to retention time 60480ms breach (kafka.log.Log)
[2020-02-06 13:32:48,966] INFO [Log partition=ocs.account-balances-12, 
dir=/data/ocswf/kafka_broker/kafka-data] Scheduling log segment [baseOffset 
7607076, size 131228281] for deletion. (kafka.log.Log)
[2020-02-06 13:32:48,979] ERROR Error while deleting segments for 
ocs.account-balances-12 in dir /data/ocswf/kafka_broker/kafka-data 
(kafka.server.LogDirFailureChannel)
java.nio.file.NoSuchFileException: 
/data/ocswf/kafka_broker/kafka-data/ocs.account-balances-12/07607076.log
Suppressed: java.nio.file.NoSuchFileException: 
/data/ocswf/kafka_broker/kafka-data/ocs.account-balances-12/07607076.log
 -> 
/data/ocswf/kafka_broker/kafka-data/ocs.account-balances-12/07607076.log.deleted
[2020-02-06 13:32:48,982] INFO [ReplicaManager broker=3] Stopping serving 
replicas in dir /data/ocswf/kafka_broker/kafka-data 
(kafka.server.ReplicaManager)
[2020-02-06 13:32:48,983] ERROR Uncaught exception in scheduled task 
'kafka-log-retention' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting 
segments for ocs.account-balances-12 in dir /data/ocswf/kafka_broker/kafka-data
Caused by: java.nio.file.NoSuchFileException: 
/data/ocswf/kafka_broker/kafka-data/ocs.account-balances-12/07607076.log
Suppressed: java.nio.file.NoSuchFileException: 
/data/ocswf/kafka_broker/kafka-data/ocs.account-balances-12/07607076.log
 -> 
/data/ocswf/kafka_broker/kafka-data/ocs.account-balances-12/07607076.log.deleted
...
[2020-02-06 13:32:49,058] INFO Stopping serving logs in dir 
/data/ocswf/kafka_broker/kafka-data (kafka.log.LogManager)
[2020-02-06 13:32:49,078] ERROR Shutdown broker because all log dirs in 
/data/ocswf/kafka_broker/kafka-data have failed (kafka.log.LogManager)
Затем аварийно остановились все остальные ноды кластера на выборах лидеров 
партиций:
[2020-02-06 13:32:53,620] ERROR [ReplicaManager broker=1] Error while making 
broker the leader for partition Topic: ocs.counter-balances; Partition: 40; 
Leader: Some(3); AllReplicas: 1,2,3,4; InSyncReplicas: 1,2,4 in dir 
Some(/data/ocswf/kafka_broker/kafka-data) (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.KafkaStorageException: Error while writing to 
checkpoint file 
/data/ocswf/kafka_broker/kafka-data/ocs.counter-balances-40/leader-epoch-checkpoint
Caused by: java.io.FileNotFoundException: 
/data/ocswf/kafka_broker/kafka-data/ocs.counter-balances-40/leader-epoch-checkpoint.tmp
 (No such file or directory)
не смогли переписать leader-epoch-checkpoint и остановились по этой причине
[2020-02-06 13:32:53,687] INFO Stopping serving logs in dir 
/data/ocswf/kafka_broker/kafka-data (kafka.log.LogManager)
[2020-02-06 13:32:53,698] ERROR Shutdown broker because all log dirs in 
/data/ocswf/kafka_broker/kafka-data have failed (kafka.log.LogManager)
Является ли эта ситуация нормой?

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Apache Kafka 2.5.0 release

2020-02-25 Thread Viktor Somogyi-Vass

Hi David,

There are two short JIRAs related to KIP-352 that documents the newly added
metrics. Is it possible to merge them in?
https://github.com/apache/kafka/pull/7434 (trunk)
https://github.com/apache/kafka/pull/8127 (2.5 cherry-pick)

Thanks,
Viktor


On Mon, Feb 24, 2020 at 7:22 PM David Arthur  wrote:

> Thanks, Tu. I've moved KIP-467 out of the release plan.
>
> -David
>
> On Thu, Feb 20, 2020 at 6:00 PM Tu Tran  wrote:
>
> > Hi David,
> >
> > Thanks for being the release main driver. Since the implementation for
> the
> > last part of KIP-467 wasn't finalized prior to Feb 12th, could you remove
> > KIP-467 from the list?
> >
> > Thanks,
> > Tu
> >
> > On Thu, Feb 20, 2020 at 7:18 AM David Arthur  wrote:
> >
> > > Randall / Konstantine,
> > >
> > > Sorry for the late reply. Thanks for the fix and for the update! I see
> > this
> > > change on the 2.5 branch (@b403c66). Consider this a retroactive
> approval
> > > for this bugfix :)
> > >
> > > -David
> > >
> > > On Fri, Feb 14, 2020 at 2:21 PM Konstantine Karantasis <
> > > konstant...@confluent.io> wrote:
> > >
> > > > Hi David,
> > > >
> > > > I want to confirm what Randall mentions above. The code fixes for
> > > > KAFKA-9556 were in place before code freeze on Wed, but we spent a
> bit
> > > more
> > > > time hardening the conditions of the integration tests and fixing
> some
> > > > jenkins branch builders to run the test on repeat.
> > > >
> > > > Best,
> > > > Konstantine
> > > >
> > > >
> > > > On Fri, Feb 14, 2020 at 7:42 AM Randall Hauch 
> > wrote:
> > > >
> > > > > Hi, David.
> > > > >
> > > > > I just filed https://issues.apache.org/jira/browse/KAFKA-9556 that
> > > > > identifies two pretty minor issues with the new KIP-558 that adds
> new
> > > > > Connect REST API endpoints to get the list of topics used by a
> > > connector.
> > > > > The impact is high: the feature cannot be fully disabled, and
> Connect
> > > > does
> > > > > not automatically reset the topic set when a connector is deleted.
> > > > > https://github.com/apache/kafka/pull/8085 includes the two fixes,
> > and
> > > > also
> > > > > adds more unit and integration tests for this feature. Although I
> > just
> > > > > created the blocker this AM, Konstantine has actually be working on
> > the
> > > > fix
> > > > > for four days. Risk of merging this PR is low, since a) the new
> > > > integration
> > > > > tests add significant coverage and we've run the new tests numerous
> > > > times,
> > > > > and b) the fixes help gate the new feature even more and allow the
> > > > feature
> > > > > to be completely disabled.
> > > > >
> > > > > I'd like approve to merge
> https://github.com/apache/kafka/pull/8085
> > > > >
> > > > > Thanks!
> > > > > Randall
> > > > >
> > > > > On Mon, Feb 10, 2020 at 11:31 AM David Arthur 
> > > wrote:
> > > > >
> > > > > > Just a friendly reminder that this Wednesday, February 12th, is
> the
> > > > code
> > > > > > freeze for the 2.5.0 release. After this time we will only accept
> > > > blocker
> > > > > > bugs onto the release branch.
> > > > > >
> > > > > > Thanks!
> > > > > > David
> > > > > >
> > > > > > On Fri, Jan 31, 2020 at 5:13 PM David Arthur 
> > > wrote:
> > > > > >
> > > > > > > Thanks! I've updated the list.
> > > > > > >
> > > > > > > On Thu, Jan 30, 2020 at 5:48 PM Konstantine Karantasis <
> > > > > > > konstant...@confluent.io> wrote:
> > > > > > >
> > > > > > >> Hi David,
> > > > > > >>
> > > > > > >> thanks for driving the release.
> > > > > > >>
> > > > > > >> Please also remove KIP-158 from the list of KIPs that you plan
> > to
> > > > > > include
> > > > > > >> in 2.5
> > > > > > >> KIP-158 has been accepted, but the implementation is not yet
> > > final.
> > > > It
> > > > > > >> will be included in the release that follows 2.5.
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> Konstantine
> > > > > > >>
> > > > > > >> On 1/30/20, Matthias J. Sax  wrote:
> > > > > > >> > Hi David,
> > > > > > >> >
> > > > > > >> > the following KIP from the list did not make it:
> > > > > > >> >
> > > > > > >> >  - KIP-216 (no PR yet)
> > > > > > >> >  - KIP-399 (no PR yet)
> > > > > > >> >  - KIP-401 (PR not merged yet)
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > KIP-444 should be included as we did make progress, but it
> is
> > > > still
> > > > > > not
> > > > > > >> > fully implement and we need to finish in in 2.6 release.
> > > > > > >> >
> > > > > > >> > KIP-447 is partially implemented in 2.5 (ie, broker and
> > > > > > >> > consumer/producer changes -- the Kafka Streams parts slip)
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > -Matthias
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On 1/29/20 9:05 AM, David Arthur wrote:
> > > > > > >> >> Hey everyone, just a quick update on the 2.5 release.
> > > > > > >> >>
> > > > > > >> >> I have updated the list of planned KIPs on the release wiki
> > > page
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Build failed in Jenkins: kafka-2.5-jdk8 #42

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[wangguoz] KAFKA-9599 create unique sensor to record group rebalance (#8159)


--
[...truncated 1.83 MB...]
> Task :streams:upgrade-system-tests-11:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-20:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-20:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-20:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-20:checkstyleMain NO-SOURCE

> Task :streams:upgrade-system-tests-20:compileTestJava FAILED
Fatal Error: Unable to find package java.lang in classpath or bootclasspath

> Task :streams:upgrade-system-tests-20:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-20:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-21:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-21:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-21:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-21:checkstyleMain NO-SOURCE

> Task :streams:upgrade-system-tests-21:compileTestJava FAILED
Fatal Error: Unable to find package java.lang in classpath or bootclasspath

> Task :streams:upgrade-system-tests-21:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-21:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-22:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-22:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-22:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-22:checkstyleMain NO-SOURCE

> Task :streams:upgrade-system-tests-22:compileTestJava FAILED
Fatal Error: Unable to find package java.lang in classpath or bootclasspath

> Task :streams:upgrade-system-tests-22:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-22:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-23:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-23:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-23:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-23:checkstyleMain NO-SOURCE

> Task :streams:upgrade-system-tests-23:compileTestJava FAILED
Fatal Error: Unable to find package java.lang in classpath or bootclasspath

> Task :streams:upgrade-system-tests-23:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-23:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-24:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-24:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-24:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-24:checkstyleMain NO-SOURCE

> Task :streams:upgrade-system-tests-24:compileTestJava FAILED
Fatal Error: Unable to find package java.lang in classpath or bootclasspath

> Task :streams:upgrade-system-tests-24:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-24:spotbugsMain NO-SOURCE

FAILURE: Build completed with 18 failures.

1: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':examples:compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

2: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':generator:compileTestJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

3: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':generator:spotbugsMain'.
> Failed to run Gradle SpotBugs Worker
   > A problem occurred starting process 'Gradle SpotBugs Worker 6'

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

4: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':streams:processMessages'.
> A problem occurred starting process 'command 
> '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java''

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

5: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':connect:api:compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack

Build failed in Jenkins: kafka-trunk-jdk11 #1186

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-9581: Remove rebalance exception withholding (#8145)

[github] KAFKA-9599 create unique sensor to record group rebalance (#8159)

[github] MINOR: Improve AuthorizerIntegrationTest (#7926)


--
[...truncated 2.89 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest >

[jira] [Created] (KAFKA-9603) Number of open files keeps increasing in Streams application

2020-02-25 Thread Bruno Iljazovic (Jira)

Bruno Iljazovic created KAFKA-9603:
--

 Summary: Number of open files keeps increasing in Streams 
application
 Key: KAFKA-9603
 URL: https://issues.apache.org/jira/browse/KAFKA-9603
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.3.1, 2.4.0
 Environment: Spring Boot 2.2.4, OpenJDK 13, Centos image
Reporter: Bruno Iljazovic


Problem appeared when upgrading from *2.0.1* to *2.3.1*. 

Relevant Kafka Streams code:
{code:java}
KStream events1 =
builder.stream(FIRST_TOPIC_NAME, Consumed.with(stringSerde, event1Serde, 
event1TimestampExtractor(), null))
   .mapValues(...);

KStream events2 =
builder.stream(SECOND_TOPIC_NAME, Consumed.with(stringSerde, event2Serde, 
event2TimestampExtractor(), null))
   .mapValues(...);

var joinWindows = JoinWindows.of(Duration.of(1, MINUTES).toMillis())
 .until(Duration.of(1, HOURS).toMillis());

events2.join(events1, this::join, joinWindows, Joined.with(stringSerde, 
event2Serde, event1Serde))
   .foreach(...);
{code}
Number of open *.sst files keeps increasing until eventually it hits the os 
limit (65536) and causes this exception:
{code:java}
Caused by: org.rocksdb.RocksDBException: While open a file for appending: 
/.../0_8/KSTREAM-JOINOTHER-10-store/KSTREAM-JOINOTHER-10-store.157943520/001354.sst:
 Too many open files
at org.rocksdb.RocksDB.flush(Native Method)
at org.rocksdb.RocksDB.flush(RocksDB.java:2394)
{code}
Here are example files that are opened and never closed:
{code:java}
/.../0_27/KSTREAM-JOINTHIS-09-store/KSTREAM-JOINTHIS-09-store.158245920/000114.sst
/.../0_27/KSTREAM-JOINOTHER-10-store/KSTREAM-JOINOTHER-10-store.158245920/65.sst
/.../0_29/KSTREAM-JOINTHIS-09-store/KSTREAM-JOINTHIS-09-store.158215680/000115.sst
/.../0_29/KSTREAM-JOINTHIS-09-store/KSTREAM-JOINTHIS-09-store.158245920/000112.sst
/.../0_31/KSTREAM-JOINTHIS-09-store/KSTREAM-JOINTHIS-09-store.158185440/51.sst
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk8 #4266

2020-02-25 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-9581: Remove rebalance exception withholding (#8145)

[github] KAFKA-9599 create unique sensor to record group rebalance (#8159)

[github] MINOR: Improve AuthorizerIntegrationTest (#7926)


--
[...truncated 2.60 MB...]
4: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':connect:api:spotbugsMain'.
> Failed to run Gradle SpotBugs Worker
   > A problem occurred starting process 'Gradle SpotBugs Worker 18'

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

5: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':connect:basic-auth-extension:compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

6: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':connect:file:compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

7: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':connect:json:compileTestJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

8: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':connect:json:spotbugsMain'.
> Failed to run Gradle SpotBugs Worker
   > A problem occurred starting process 'Gradle SpotBugs Worker 19'

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

9: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':connect:mirror-client:compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

10: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':connect:transforms:compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

11: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':streams:examples:compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

12: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':streams:streams-scala:compileScala'.
> object java.lang.Object in compiler mirror not found.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

13: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':streams:test-utils:compileTestJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.
==

14: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':streams:test-utils:spotbugsMain'.
> Failed to run Gradle SpotBugs Worker
   > A problem occurred starting process 'Gradle SpotBugs Worker 20'

* Try:
Run with

43 matches

Mail list logo