date:20200726

Build failed in Jenkins: kafka-trunk-jdk8 #4745

2020-07-26 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-10306: GlobalThread should fail on InvalidOffsetException (#9075)


--
[...truncated 2.76 MB...]

org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[exactly_once] STARTED

org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[exactly_once] PASSED

org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[exactly_once_beta] STARTED

org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[exactly_once_beta] PASSED

org.apache.kafka.streams.integration.RocksDBMetricsIntegrationTest > 
shouldExposeRocksDBMetricsBeforeAndAfterFailureWithEmptyStateDir[at_least_once] 
STARTED

org.apache.kafka.streams.integration.RocksDBMetricsIntegrationTest > 
shouldExposeRocksDBMetricsBeforeAndAfterFailureWithEmptyStateDir[at_least_once] 
PASSED

org.apache.kafka.streams.integration.RocksDBMetricsIntegrationTest > 
shouldExposeRocksDBMetricsBeforeAndAfterFailureWithEmptyStateDir[exactly_once] 
STARTED

org.apache.kafka.streams.integration.RocksDBMetricsIntegrationTest > 
shouldExposeRocksDBMetricsBeforeAndAfterFailureWithEmptyStateDir[exactly_once] 
PASSED

org.apache.kafka.streams.integration.RocksDBMetricsIntegrationTest > 
shouldExposeRocksDBMetricsBeforeAndAfterFailureWithEmptyStateDir[exactly_once_beta]
 STARTED

org.apache.kafka.streams.integration.RocksDBMetricsIntegrationTest > 
shouldExposeRocksDBMetricsBeforeAndAfterFailureWithEmptyStateDir[exactly_once_beta]
 PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceSessionWindows STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceSessionWindows PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduce STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduce PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldAggregate STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldAggregate PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCount STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCount PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldGroupByKey STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldGroupByKey PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountWithInternalStore STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountWithInternalStore PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountUnlimitedWindows STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountUnlimitedWindows PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceWindowed STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduceWindowed PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountSessionWindows STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldCountSessionWindows PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldAggregateWindowed STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldAggregateWindowed PASSED

org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldGetToRunningWithOnlyGlobalTopology STARTED

org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldGetToRunningWithOnlyGlobalTopology PASSED

org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableLeftJoin STARTED

org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableLeftJoin PASSED

org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldRestoreGlobalInMemoryKTableOnRestart STARTED

org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldRestoreGlobalInMemoryKTableOnRestart PASSED

org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableJoin STARTED

org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableJoin PASSED

org.apache.kafka.streams.integration.RestoreIntegrationTest > 
shouldRecycleStateFromStandbyTaskPromotedToActiveTaskAndNotRestore STARTED

org.apache.kafka.streams.integration.RestoreIntegrationTest >

[jira] [Created] (KAFKA-10312) MetadataCache.getPartitionMetadata may return incorrect error code when partition's leader is not present at the MetadataCache

2020-07-26 Thread Raman Verma (Jira)

Raman Verma created KAFKA-10312:
---

 Summary: MetadataCache.getPartitionMetadata may return incorrect 
error code when partition's leader is not present at the MetadataCache
 Key: KAFKA-10312
 URL: https://issues.apache.org/jira/browse/KAFKA-10312
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: Raman Verma


`MetadataCache.getPartitionMetadata` returns an error code if the partition's 
leader is not present at the MetadataCache, or if the Listener endpoint is not 
present for the leader. Newer versions for metadata returns 
LEADER_NOT_AVAILABLE and LISTENER_NOT_FOUND for the above two cases 
respectively.

 There is a small bug in this logic. Current code checks the MetadataCache's 
host broker Id instead of the partition's leader id (in the list of alive 
brokers) when determining what error code to return. This may result in this 
call returning LISTENER_NOT_FOUND error code even when partition's leader is 
not present at MetadataCache and it should have returned LEADER_NOT_AVAILABLE.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-07-26 Thread Unmesh Joshi

>>Could you clarify on this question? Which part of the raft group doesn't
>>know about leader dis-connection?
The leader of the controller quorum is partitioned from the controller
cluster, and a different leader is elected for the remaining controller
cluster.
I think there are two things here,
1.  The old leader will not know if it's disconnected from the rest of the
controller quorum cluster unless it receives BeginQuorumEpoch from the new
leader. So it will keep on serving stale metadata to the clients (Brokers,
Producers and Consumers)
2. I assume, the Broker Leases will be managed on the controller quorum
leader. This partitioned leader will keep on tracking broker leases it has,
while the new leader of the quorum will also start managing broker leases.
So while the quorum leader is partitioned, there will be two membership
views of the kafka brokers managed on two leaders.
Unless broker heartbeats are also replicated as part of the Raft log, there
is no way to solve this?
I know LogCabin implementation does replicate client heartbeats. I suspect
that the same issue is there in Zookeeper, which does not replicate client
Ping requests..

Thanks,
Unmesh



On Mon, Jul 27, 2020 at 6:23 AM Boyang Chen 
wrote:

> Thanks for the questions Unmesh!
>
> On Sun, Jul 26, 2020 at 6:18 AM Unmesh Joshi 
> wrote:
>
> > Hi,
> >
> > In the FetchRequest Handling, how to make sure we handle scenarios where
> > the leader might have been disconnected from the cluster, but doesn't
> know
> > yet?
> >
> Could you clarify on this question? Which part of the raft group doesn't
> know about leader
> dis-connection?
>
>
> > As discussed in the Raft Thesis section 6.4, the linearizable semantics
> of
> > read requests is implemented in LogCabin by sending heartbeat to
> followers
> > and waiting till the heartbeats are successful to make sure that the
> leader
> > is still the leader.
> > I think for the controller quorum to make sure none of the consumers get
> > stale data, it's important to have linearizable semantics? In the pull
> > based model, the leader will need to wait for heartbeats from the
> followers
> > before returning each fetch request from the consumer then? Or do we need
> > to introduce some other request?
> > (Zookeeper does not have linearizable semantics for read requests, but as
> > of now all the kafka interactions are through writes and watches).
> >
> > This is a very good question. For our v1 implementation we are not aiming
> to guarantee linearizable read, which
> would be considered as a follow-up effort. Note that today in Kafka there
> is no guarantee on the metadata freshness either,
> so no regression is introduced.
>
>
> > Thanks,
> > Unmesh
> >
> > On Fri, Jul 24, 2020 at 11:36 PM Jun Rao  wrote:
> >
> > > Hi, Jason,
> > >
> > > Thanks for the reply.
> > >
> > > 101. Sounds good. Regarding clusterId, I am not sure storing it in the
> > > metadata log is enough. For example, the vote request includes
> clusterId.
> > > So, no one can vote until they know the clusterId. Also, it would be
> > useful
> > > to support the case when a voter completely loses its disk and needs to
> > > recover.
> > >
> > > 210. There is no longer a FindQuorum request. When a follower restarts,
> > how
> > > does it discover the leader? Is that based on DescribeQuorum? It would
> be
> > > useful to document this.
> > >
> > > Jun
> > >
> > > On Fri, Jul 17, 2020 at 2:15 PM Jason Gustafson 
> > > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Thanks for the questions.
> > > >
> > > > 101. I am treating some of the bootstrapping problems as out of the
> > scope
> > > > of this KIP. I am working on a separate proposal which addresses
> > > > bootstrapping security credentials specifically. Here is a rough
> sketch
> > > of
> > > > how I am seeing it:
> > > >
> > > > 1. Dynamic broker configurations including encrypted passwords will
> be
> > > > persisted in the metadata log and cached in the broker's
> > > `meta.properties`
> > > > file.
> > > > 2. We will provide a tool which allows users to directly override the
> > > > values in `meta.properties` without requiring access to the quorum.
> > This
> > > > can be used to bootstrap the credentials of the voter set itself
> before
> > > the
> > > > cluster has been started.
> > > > 3. Some dynamic config changes will only be allowed when a broker is
> > > > online. For example, changing a truststore password dynamically would
> > > > prevent that broker from being able to start if it were offline when
> > the
> > > > change was made.
> > > > 4. I am still thinking a little bit about SCRAM credentials, but most
> > > > likely they will be handled with an approach similar to
> > > `meta.properties`.
> > > >
> > > > 101.3 As for the question about `clusterId`, I think the way we would
> > do
> > > > this is to have the first elected leader generate a UUID and write it
> > to
> > > > the metadata log. Let me add some detail to the proposal about this.
> > > >
> > > > A few

Build failed in Jenkins: kafka-trunk-jdk14 #320

2020-07-26 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-10306: GlobalThread should fail on InvalidOffsetException (#9075)

[github] MINOR: Recommend Java 11 (#9080)


--
[...truncated 2.80 MB...]

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldAssignEvenlyAcrossConsumersOneClientMultipleThreads[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldGetTime[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldGetTime[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigPortIsNotAnInteger[task assignor = 
class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigPortIsNotAnInteger[task assignor = 
class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldNotFailOnBranchedMultiLevelRepartitionConnectedTopology[task assignor = 
class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldNotFailOnBranchedMultiLevelRepartitionConnectedTopology[task assignor = 
class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowKafkaExceptionIfVersionProbingFlagConfigIsNotAtomicInteger[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowKafkaExceptionIfVersionProbingFlagConfigIsNotAtomicInteger[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowKafkaExceptionIfTaskMangerConfigIsNotTaskManagerInstance[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowKafkaExceptionIfTaskMangerConfigIsNotTaskManagerInstance[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldTriggerImmediateRebalanceOnHostInfoChange[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldTriggerImmediateRebalanceOnHostInfoChange[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldProduceMaximallyStickyAssignmentWhenMemberLeaves[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldProduceMaximallyStickyAssignmentWhenMemberLeaves[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldGetAssignmentConfigs[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldGetAssignmentConfigs[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowIfV2SubscriptionAndFutureSubscriptionIsMixed[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowIfV2SubscriptionAndFutureSubscriptionIsMixed[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

[jira] [Created] (KAFKA-10311) Flaky test KafkaAdminClientTest#testMetadataRetries

2020-07-26 Thread Boyang Chen (Jira)

Boyang Chen created KAFKA-10311:
---

 Summary: Flaky test KafkaAdminClientTest#testMetadataRetries
 Key: KAFKA-10311
 URL: https://issues.apache.org/jira/browse/KAFKA-10311
 Project: Kafka
  Issue Type: Bug
Reporter: Boyang Chen


[https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3545/testReport/junit/org.apache.kafka.clients.admin/KafkaAdminClientTest/testMetadataRetries/]

 
h3. Error Message

java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: Call(callName=describeTopics, 
deadlineMs=1595694629113, tries=1, nextAllowedTryMs=1595694629217) timed out at 
1595694629117 after 1 attempt(s)
h3. Stacktrace

java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: Call(callName=describeTopics, 
deadlineMs=1595694629113, tries=1, nextAllowedTryMs=1595694629217) timed out at 
1595694629117 after 1 attempt(s) at 
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
 at 
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
 at 
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
 at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) 
at 
org.apache.kafka.clients.admin.KafkaAdminClientTest.testMetadataRetries(KafkaAdminClientTest.java:995)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
 at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
java.lang.Thread.run(Thread.java:748) Caused by: 
org.apache.kafka.common.errors.TimeoutException: Call(callName=describeTopics, 
deadlineMs=1595694629113, tries=1, nextAllowedTryMs=1595694629217) timed out at 
1595694629117 after 1 attempt(s) Caused by: 
org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node 
assignment.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1671

2020-07-26 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: INFO log4j when request re-join (#9068)


--
[...truncated 3.20 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp

Build failed in Jenkins: kafka-2.6-jdk8 #98

2020-07-26 Thread Apache Jenkins Server

See 


Changes:

[matthias] KAFKA-10306: GlobalThread should fail on InvalidOffsetException 
(#9075)


--
[...truncated 3.15 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-07-26 Thread Boyang Chen

Thanks for the questions Unmesh!

On Sun, Jul 26, 2020 at 6:18 AM Unmesh Joshi  wrote:

> Hi,
>
> In the FetchRequest Handling, how to make sure we handle scenarios where
> the leader might have been disconnected from the cluster, but doesn't know
> yet?
>
Could you clarify on this question? Which part of the raft group doesn't
know about leader
dis-connection?


> As discussed in the Raft Thesis section 6.4, the linearizable semantics of
> read requests is implemented in LogCabin by sending heartbeat to followers
> and waiting till the heartbeats are successful to make sure that the leader
> is still the leader.
> I think for the controller quorum to make sure none of the consumers get
> stale data, it's important to have linearizable semantics? In the pull
> based model, the leader will need to wait for heartbeats from the followers
> before returning each fetch request from the consumer then? Or do we need
> to introduce some other request?
> (Zookeeper does not have linearizable semantics for read requests, but as
> of now all the kafka interactions are through writes and watches).
>
> This is a very good question. For our v1 implementation we are not aiming
to guarantee linearizable read, which
would be considered as a follow-up effort. Note that today in Kafka there
is no guarantee on the metadata freshness either,
so no regression is introduced.


> Thanks,
> Unmesh
>
> On Fri, Jul 24, 2020 at 11:36 PM Jun Rao  wrote:
>
> > Hi, Jason,
> >
> > Thanks for the reply.
> >
> > 101. Sounds good. Regarding clusterId, I am not sure storing it in the
> > metadata log is enough. For example, the vote request includes clusterId.
> > So, no one can vote until they know the clusterId. Also, it would be
> useful
> > to support the case when a voter completely loses its disk and needs to
> > recover.
> >
> > 210. There is no longer a FindQuorum request. When a follower restarts,
> how
> > does it discover the leader? Is that based on DescribeQuorum? It would be
> > useful to document this.
> >
> > Jun
> >
> > On Fri, Jul 17, 2020 at 2:15 PM Jason Gustafson 
> > wrote:
> >
> > > Hi Jun,
> > >
> > > Thanks for the questions.
> > >
> > > 101. I am treating some of the bootstrapping problems as out of the
> scope
> > > of this KIP. I am working on a separate proposal which addresses
> > > bootstrapping security credentials specifically. Here is a rough sketch
> > of
> > > how I am seeing it:
> > >
> > > 1. Dynamic broker configurations including encrypted passwords will be
> > > persisted in the metadata log and cached in the broker's
> > `meta.properties`
> > > file.
> > > 2. We will provide a tool which allows users to directly override the
> > > values in `meta.properties` without requiring access to the quorum.
> This
> > > can be used to bootstrap the credentials of the voter set itself before
> > the
> > > cluster has been started.
> > > 3. Some dynamic config changes will only be allowed when a broker is
> > > online. For example, changing a truststore password dynamically would
> > > prevent that broker from being able to start if it were offline when
> the
> > > change was made.
> > > 4. I am still thinking a little bit about SCRAM credentials, but most
> > > likely they will be handled with an approach similar to
> > `meta.properties`.
> > >
> > > 101.3 As for the question about `clusterId`, I think the way we would
> do
> > > this is to have the first elected leader generate a UUID and write it
> to
> > > the metadata log. Let me add some detail to the proposal about this.
> > >
> > > A few additional answers below:
> > >
> > > 203. Yes, that is correct.
> > >
> > > 204. That is a good question. What happens in this case is that all
> > voters
> > > advance their epoch to the one designated by the candidate even if they
> > > reject its vote request. Assuming the candidate fails to be elected,
> the
> > > election will be retried until a leader emerges.
> > >
> > > 205. I had some discussion with Colin offline about this problem. I
> think
> > > the answer should be "yes," but it probably needs a little more
> thought.
> > > Handling JBOD failures is tricky. For an observer, we can replicate the
> > > metadata log from scratch safely in a new log dir. But if the log dir
> of
> > a
> > > voter fails, I do not think it is generally safe to start from an empty
> > > state.
> > >
> > > 206. Yes, that is discussed in KIP-631 I believe.
> > >
> > > 207. Good suggestion. I will work on this.
> > >
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Jul 16, 2020 at 3:44 PM Jun Rao  wrote:
> > >
> > > > Hi, Jason,
> > > >
> > > > Thanks for the updated KIP. Looks good overall. A few more comments
> > > below.
> > > >
> > > > 101. I still don't see a section on bootstrapping related issues. It
> > > would
> > > > be useful to document if/how the following is supported.
> > > > 101.1 Currently, we support auto broker id generation. Is this
> > supported
> > > > for bootstrap brokers?
> > > >

[DISCUSS] KIP-630: Kafka Raft Snapshot

2020-07-26 Thread Jose Garcia Sancio

Hi All,

I would like to start a discussion on KIP-630:
https://cwiki.apache.org/confluence/x/exV4CQ

This proposal specifies extensions to KIP-595 to support generating
snapshots for the replicated log. Please let me know if you have any
comments and suggestions.

Thanks!
-- 
-Jose

[jira] [Created] (KAFKA-10310) Kafka Raft Snapshot

2020-07-26 Thread Jose Armando Garcia Sancio (Jira)

Jose Armando Garcia Sancio created KAFKA-10310:
--

 Summary: Kafka Raft Snapshot
 Key: KAFKA-10310
 URL: https://issues.apache.org/jira/browse/KAFKA-10310
 Project: Kafka
  Issue Type: New Feature
Reporter: Jose Armando Garcia Sancio
Assignee: Jose Armando Garcia Sancio


Tracking issue for KIP-630: Kafka Raft Snapshot



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1670

2020-07-26 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-10306: GlobalThread should fail on InvalidOffsetException (#9075)

[github] MINOR: Recommend Java 11 (#9080)


--
[...truncated 3.21 MB...]
org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders

Re: [DISCUSS] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

2020-07-26 Thread Matthias J. Sax

Hi,

I just had one more thought about an additional improvement we might
want to include in this KIP.

Kafka Streams ships with a `WallclockTimestampExtractor` that just
returns `System.currentTimeMillis()` and thus, cannot be mocked. And it
seems that there is no way for a user to build a custom timestamps
extractor that returns the TTD's mocked system time.

Thus, it might be nice, to add a `MockTimeExtractor` (only in the
test-util package) that users could set and this `MockTimeExtractor`
returns the mocked system time.

Thoughts?


-Matthias

On 7/7/20 11:11 PM, Matthias J. Sax wrote:
> I think, we don't need a default implementation for the new methods.
> 
> What would be the use-case to implement the  `ProcessorContext`
> interface? In contract to, for example, `KeyValueStore`,
> `ProcessorContext` is a use-only interface because it's never passed
> into Kafka Streams, but only handed out to the user.
> 
> 
> -Matthias
> 
> 
> On 7/7/20 1:28 PM, William Bottrell wrote:
>> Sure, I would appreciate help from Piotr creating an example.
>>
>> On Tue, Jul 7, 2020 at 12:03 PM Boyang Chen 
>> wrote:
>>
>>> Hey John,
>>>
>>> since ProcessorContext is a public API, I couldn't be sure that people
>>> won't try to extend it. Without a default implementation, user code
>>> compilation will break.
>>>
>>> William and Piotr, it seems that we haven't added any example usage of the
>>> new API, could we try to address that? It should help with the motivation
>>> and follow-up meta comments as John proposed.
>>>
>>> Boyang
>>>
>>> On Mon, Jul 6, 2020 at 12:04 PM Matthias J. Sax  wrote:
>>>
 William,

 thanks for the KIP. LGMT. Feel free to start a vote.


 -Matthias


 On 7/4/20 10:14 AM, John Roesler wrote:
> Hi Richard,
>
> It’s good to hear from you!
>
> Thanks for bringing up the wall-clock suppression feature. IIRC,
>>> someone
 actually started a KIP discussion for it already, but I don’t think it
>>> went
 to a vote. I don’t recall any technical impediment, just the lack of
 availability to finish it up. Although there is some association, it
>>> would
 be good to keep the KIPs separate.
>
> Thanks,
> John
>
> On Sat, Jul 4, 2020, at 10:05, Richard Yu wrote:
>> Hi all,
>>
>> This reminds me of a previous issue I think that we were discussing.
>> @John Roesler  I think you should
>>> remember
 this one.
>>
>> A while back, we were talking about having suppress operator emit
>> records by wall-clock time instead of stream time.
>> If we are adding this, wouldn't that make it more feasible for us to
>> implement that feature for suppression?
>>
>> If I recall correctly, there actually had been quite a bit of user
>> demand for such a feature.
>> Might be good to include it in this KIP (or maybe use one of the prior
>> KIPs for this feature).
>>
>> Best,
>> Richard
>>
>> On Sat, Jul 4, 2020 at 6:58 AM John Roesler 
 wrote:
>>> Hi all,
>>>
>>>  1. Thanks, Boyang, it is nice to see usage examples in KIPs like
 this. It helps during the discussion, and it’s also good documentation
 later on.
>>>
>>>  2. Yeah, this is a subtle point. The motivation mentions being able
 to control the time during tests, but to be able to make it work, the
 processor implementation needs a public method on ProcessorContext to get
 the time. Otherwise, processors would have to check the type of the
>>> context
 and cast, depending on whether they’re running inside a test or not. In
 retrospect, if we’d had a usage example, this probably would have been
 clear.
>>>
>>>  3. I don’t think we expect people to have their own implementations
 of ProcessorContext. Since all implementations are internal, it’s really
>>> an
 implementation detail whether we use a default method, abstract methods,
>>> or
 concrete methods. I can’t think of an implementation that really wants to
 just look up the system time. In the production code path, we cache the
 time for performance, and in testing, we use a mock time.
>>>
>>>  Thanks,
>>>  John
>>>
>>>
>>>  On Fri, Jul 3, 2020, at 06:41, Piotr Smoliński wrote:
>>>  > 1. Makes sense; let me propose something
>>>  >
>>>  > 2. That's not testing-only. The goal is to use the same API to
 access
>>>  > the time
>>>  > in deployment and testing environments. The major driver is
>>>  > System.currentTimeMillis(),
>>>  > which a) cannot be used in tests b) could go in specific cases
>>> back
 c)
>>>  > is not compatible
>>>  > with punctuator call. The idea is that we could access clock using
>>>  > uniform API.
>>>  > For completness we should have same API for system and stream
>>> time.
>>>  >
>>>  > 3. There aren't that many subclasses. Two

Re: [DISCUSS] KIP-645: Replace abstract class Windows with a proper interface

2020-07-26 Thread John Roesler

Thanks Sophie and Boyang for asking for specifics.

As far as I can tell, if we were to _remove_ all the non-public-method
members from Windows, including any constructors, we are left with
effectively an interface. I think this was my plan before. I don't think
I realized at the time that it's possible to replace the entire class with
an interface. Now I realize it is possible, hence the motivation to do it.

We can choose either to maintain forever the tech debt of having to
enforce that Windows look, sound, and act just like an interface, or we
can just replace it with an interface and be done with it. I.e., the
motivation is less maintenence burden for us and for users.

Coincidentally, I had an interesting conversation with Matthias about
this interface, and he made me realize that "fixed size" isn't the essential
nature of this entity. Instead being enumerable is. Reframing the interface
in this way will enable us to implement variable sized windows like
MonthlyWindows.

So, now there are two good reasons to vote for this KIP :)

Anyway, I've updated the proposed interface and the motivation.

To Sophie latter question, all of the necessary deprecation is listed
in the KIP. We do not have to deprecate any windowBy methods.

Thanks,
-John

On Sat, Jul 25, 2020, at 00:52, Boyang Chen wrote:
> Thanks for the KIP John. I share a similar concern with the motivation, it
> would be good if you could cast light on the actual downside of using a
> base class vs the interface, is it making the code fragile, or requiring
> redundant implementation, etc.
> 
> Boyang
> 
> On Tue, Jul 21, 2020 at 2:19 PM Sophie Blee-Goldman 
> wrote:
> 
> > Hey John,
> >
> > Thanks for the KIP. I know this has been bugging you :)
> >
> > That said, I think the KIP is missing some elaboration in the Motivation
> > section.
> > You mention a number of problems we've had and lived with in the past --
> > could
> > you give an example of one, and how it would be solved by your proposal?
> >
> > By the way, I assume we would also need to deprecate all APIs that accept a
> > Windows
> > parameter in favor of new ones that accept a FixedSizeWindowDefinition? Off
> > the
> > top of my head that would be the windowedBy methods in KGroupedStream and
> > CogroupedKStream
> >
> > On Tue, Jul 21, 2020 at 1:46 PM John Roesler  wrote:
> >
> > > Hello all,
> > >
> > > I'd like to propose KIP-645, to correct a small API mistake in Streams.
> > > Fixing this now allows us to avoid perpetuating the mistake in new work.
> > > For example, it will allow us to implement KIP-450 cleanly.
> > >
> > > The change itself should be seamless for users.
> > >
> > > Please see https://cwiki.apache.org/confluence/x/6SN4CQ for details.
> > >
> > > Thanks,
> > > -John
> > >
> >
>

Jenkins build is back to normal : kafka-trunk-jdk11 #1669

2020-07-26 Thread Apache Jenkins Server

See

Build failed in Jenkins: kafka-2.5-jdk8 #171

2020-07-26 Thread Apache Jenkins Server

See 


Changes:

[ismael] KAFKA-10158: Fix flaky


--
[...truncated 2.93 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

Build failed in Jenkins: kafka-trunk-jdk14 #319

2020-07-26 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-10158: Fix flaky


--
[...truncated 6.41 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCleanUpPersistentStateStoresOnClose[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCleanUpPersistentStateStoresOnClose[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest >

Build failed in Jenkins: kafka-trunk-jdk8 #4744

2020-07-26 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-10158: Fix flaky


--
[...truncated 3.18 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest >

[jira] [Created] (KAFKA-10309) KafkaProducer's sendOffsetsToTransaction should not block infinitively

2020-07-26 Thread Sasaki Toru (Jira)

Sasaki Toru created KAFKA-10309:
---

 Summary: KafkaProducer's sendOffsetsToTransaction should not block 
infinitively
 Key: KAFKA-10309
 URL: https://issues.apache.org/jira/browse/KAFKA-10309
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Affects Versions: 2.5.0
Reporter: Sasaki Toru
Assignee: Sasaki Toru


Currently, KafkaProducer's sendOffsetsToTransaction maybe block infinitively 
when Kafka cluster go down.

It's considered that this method should be affected with max.block.ms to avoid 
block infinitively, like other transaction-related methods such as 
initTransaction and commitTransaction, etc.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

2020-07-26 Thread John Roesler

Thanks William,

I’m +1 (binding)

Thanks,
John

On Fri, Jul 24, 2020, at 20:22, Sophie Blee-Goldman wrote:
> Thanks all, +1 (non-binding)
> 
> Cheers,
> Sophie
> 
> On Wed, Jul 8, 2020 at 4:02 AM Bruno Cadonna  wrote:
> 
> > Thanks Will and Piotr,
> >
> > +1 (non-binding)
> >
> > Best,
> > Bruno
> >
> > On Wed, Jul 8, 2020 at 8:12 AM Matthias J. Sax  wrote:
> > >
> > > Thanks for the KIP.
> > >
> > > +1 (binding)
> > >
> > >
> > > -Matthias
> > >
> > > On 7/7/20 11:48 AM, William Bottrell wrote:
> > > > Hi everyone,
> > > >
> > > > I'd like to start a vote for adding two new time API's to
> > ProcessorContext.
> > > >
> > > > Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
> > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
> > >
> > > >
> > > >  Thanks everyone for the initial feedback and thanks for your time.
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-406: GlobalStreamThread should honor custom reset policy

2020-07-26 Thread Navinder Brar

Hi,

Sorry, it took some time to respond back.

“but I thought we would pass the config through to the client.”

>> @John, sure we can use the config in GloablStreamThread, that could be one 
>> of the way to solve it.

@Matthias, sure cleaning the store and recreating is one way but since we are 
giving an option to reset in StreamThread why the implementation should be 
different in GlobalStreamThread. I think we should use the 
global.consumer.auto.offset.reset config to accept the reset strategy opted by 
the user although I would be ok with just cleaning and resetting to the latest 
as well for now. Currently, we throw a StreamsException in case of 
InvalidOffsetException in GlobalStreamThread so just resetting would still be 
better than what happens currently. 

Matthias, I found this comment in StreamBuilder for GlobalKTable ‘* Note that 
{@link GlobalKTable} always applies {@code "auto.offset.reset"} strategy {@code 
"earliest"} regardless of the specified value in {@link StreamsConfig} or 
{@link Consumed}.’ 
So, I guess we are already cleaning up and recreating for GlobalKTable from 
earliest offset.

@Guozhan while looking at the code, I also noticed a TODO: pending in 
GlobalStateManagerImpl, when InvalidOffsetException is thrown. Earlier, we were 
directly clearing the store here and recreating from scratch but that code 
piece is removed now. Are you working on a follow-up PR for this or just 
handling the reset in GlobalStreamThread should be sufficient?

Regards,
Navinder

On Tuesday, 7 July, 2020, 12:53:36 am IST, Matthias J. Sax 
 wrote:  

 Atm, the config should be ignored and the global-consumer should use
"none" in a hard-coded way.

However, if am still wondering if we actually want/need to allow users
to specify the reset policy? It might be worth to consider, to just
change the behavior: catch the exception, log an ERROR (for information
purpose), wipe the store, seekToBeginning(), and recreate the store?

Btw: if we want to allow users to set the reset policy, this should be
possible via the config, or via overwriting the config in the method
itself. Thus, we would need to add the new overloaded method to
`Topology` and `StreamsBuilder`.

Another question to ask: what about GlobalKTables? Should they behave
the same? An alternative design could be, to allow users to specify a
flexible reset policy for global-stores, but not for GlobalKTables and
use the strategy suggested above for this case.

Thoughts?

-Matthias

On 7/2/20 2:14 PM, John Roesler wrote:
> Hi Navinder,
> 
> Thanks for the response. I’m sorry if I’m being dense... You said we are not 
> currently using the config, but I thought we would pass the config through to 
> the client.  Can you confirm whether or not the existing config works for 
> your use case?
> 
> Thanks,
> John
> 
> On Sun, Jun 28, 2020, at 14:09, Navinder Brar wrote:
>> Sorry my bad. Found it.
>>
>>
>>
>> Prefix used to override {@link KafkaConsumer consumer} configs for the 
>> global consumer client from
>>
>> * the general consumer client configs. The override precedence is the 
>> following (from highest to lowest precedence):
>> * 1. global.consumer.[config-name]..
>> public static final String GLOBAL_CONSUMER_PREFIX = "global.consumer.";
>>
>>
>>
>> So, that's great. We already have a config exposed to reset offsets for 
>> global topics via global.consumer.auto.offset.reset just that we are 
>> not actually using it inside GlobalStreamThread to reset.
>>
>> -Navinder
>>    On Monday, 29 June, 2020, 12:24:21 am IST, Navinder Brar 
>>  wrote:  
>>  
>>  Hi John,
>>
>> Thanks for your feedback. 
>> 1. I think there is some confusion on my first point, the enum I am 
>> sure we can use the same one but the external config which controls the 
>> resetting in global stream thread either we can the same one which 
>> users use for source topics(StreamThread) or we can provide a new one 
>> which specifically controls global topics. For e.g. currently if I get 
>> an InvalidOffsetException in any of my source topics, I can choose 
>> whether to reset from Earliest or Latest(with auto.offset.reset). Now 
>> either we can use the same option and say if I get the same exception 
>> for global topics I will follow same resetting. Or some users might 
>> want to have totally different setting for both source and global 
>> topics, like for source topic I want resetting from Latest but for 
>> global topics I want resetting from Earliest so in that case adding a 
>> new config might be better.
>>
>> 2. I couldn't find this config currently 
>> "global.consumer.auto.offset.reset". Infact in GlobalStreamThread.java 
>> we are throwing a StreamsException for InvalidOffsetException and there 
>> is a test as 
>> well GlobalStreamThreadTest#shouldDieOnInvalidOffsetException(), so I 
>> think this is the config we are trying to introduce with this KIP.
>>
>> -Navinder  On Saturday, 27 June, 2020, 07:03:04 pm IST, John Roesler 
>>

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-07-26 Thread Unmesh Joshi

Hi,

In the FetchRequest Handling, how to make sure we handle scenarios where
the leader might have been disconnected from the cluster, but doesn't know
yet?
As discussed in the Raft Thesis section 6.4, the linearizable semantics of
read requests is implemented in LogCabin by sending heartbeat to followers
and waiting till the heartbeats are successful to make sure that the leader
is still the leader.
I think for the controller quorum to make sure none of the consumers get
stale data, it's important to have linearizable semantics? In the pull
based model, the leader will need to wait for heartbeats from the followers
before returning each fetch request from the consumer then? Or do we need
to introduce some other request?
(Zookeeper does not have linearizable semantics for read requests, but as
of now all the kafka interactions are through writes and watches).

Thanks,
Unmesh

On Fri, Jul 24, 2020 at 11:36 PM Jun Rao  wrote:

> Hi, Jason,
>
> Thanks for the reply.
>
> 101. Sounds good. Regarding clusterId, I am not sure storing it in the
> metadata log is enough. For example, the vote request includes clusterId.
> So, no one can vote until they know the clusterId. Also, it would be useful
> to support the case when a voter completely loses its disk and needs to
> recover.
>
> 210. There is no longer a FindQuorum request. When a follower restarts, how
> does it discover the leader? Is that based on DescribeQuorum? It would be
> useful to document this.
>
> Jun
>
> On Fri, Jul 17, 2020 at 2:15 PM Jason Gustafson 
> wrote:
>
> > Hi Jun,
> >
> > Thanks for the questions.
> >
> > 101. I am treating some of the bootstrapping problems as out of the scope
> > of this KIP. I am working on a separate proposal which addresses
> > bootstrapping security credentials specifically. Here is a rough sketch
> of
> > how I am seeing it:
> >
> > 1. Dynamic broker configurations including encrypted passwords will be
> > persisted in the metadata log and cached in the broker's
> `meta.properties`
> > file.
> > 2. We will provide a tool which allows users to directly override the
> > values in `meta.properties` without requiring access to the quorum. This
> > can be used to bootstrap the credentials of the voter set itself before
> the
> > cluster has been started.
> > 3. Some dynamic config changes will only be allowed when a broker is
> > online. For example, changing a truststore password dynamically would
> > prevent that broker from being able to start if it were offline when the
> > change was made.
> > 4. I am still thinking a little bit about SCRAM credentials, but most
> > likely they will be handled with an approach similar to
> `meta.properties`.
> >
> > 101.3 As for the question about `clusterId`, I think the way we would do
> > this is to have the first elected leader generate a UUID and write it to
> > the metadata log. Let me add some detail to the proposal about this.
> >
> > A few additional answers below:
> >
> > 203. Yes, that is correct.
> >
> > 204. That is a good question. What happens in this case is that all
> voters
> > advance their epoch to the one designated by the candidate even if they
> > reject its vote request. Assuming the candidate fails to be elected, the
> > election will be retried until a leader emerges.
> >
> > 205. I had some discussion with Colin offline about this problem. I think
> > the answer should be "yes," but it probably needs a little more thought.
> > Handling JBOD failures is tricky. For an observer, we can replicate the
> > metadata log from scratch safely in a new log dir. But if the log dir of
> a
> > voter fails, I do not think it is generally safe to start from an empty
> > state.
> >
> > 206. Yes, that is discussed in KIP-631 I believe.
> >
> > 207. Good suggestion. I will work on this.
> >
> >
> > Thanks,
> > Jason
> >
> >
> >
> >
> >
> > On Thu, Jul 16, 2020 at 3:44 PM Jun Rao  wrote:
> >
> > > Hi, Jason,
> > >
> > > Thanks for the updated KIP. Looks good overall. A few more comments
> > below.
> > >
> > > 101. I still don't see a section on bootstrapping related issues. It
> > would
> > > be useful to document if/how the following is supported.
> > > 101.1 Currently, we support auto broker id generation. Is this
> supported
> > > for bootstrap brokers?
> > > 101.2 As Colin mentioned, sometimes we may need to load the security
> > > credentials to be broker before it can be connected to. Could you
> > provide a
> > > bit more detail on how this will work?
> > > 101.3 Currently, we use ZK to generate clusterId on a new cluster. With
> > > Raft, how does every broker generate the same clusterId in a
> distributed
> > > way?
> > >
> > > 200. It would be useful to document if the various special offsets (log
> > > start offset, recovery point, HWM, etc) for the Raft log are stored in
> > the
> > > same existing checkpoint files or not.
> > > 200.1 Since the Raft log flushes every append, does that allow us to
> > > recover from a recovery point within the active segment

Build failed in Jenkins: kafka-trunk-jdk8 #4745

[jira] [Created] (KAFKA-10312) MetadataCache.getPartitionMetadata may return incorrect error code when partition's leader is not present at the MetadataCache

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

Build failed in Jenkins: kafka-trunk-jdk14 #320

[jira] [Created] (KAFKA-10311) Flaky test KafkaAdminClientTest#testMetadataRetries

Build failed in Jenkins: kafka-trunk-jdk11 #1671

Build failed in Jenkins: kafka-2.6-jdk8 #98

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

[DISCUSS] KIP-630: Kafka Raft Snapshot

[jira] [Created] (KAFKA-10310) Kafka Raft Snapshot

Build failed in Jenkins: kafka-trunk-jdk11 #1670

Re: [DISCUSS] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

Re: [DISCUSS] KIP-645: Replace abstract class Windows with a proper interface

Jenkins build is back to normal : kafka-trunk-jdk11 #1669

Build failed in Jenkins: kafka-2.5-jdk8 #171

Build failed in Jenkins: kafka-trunk-jdk14 #319

Build failed in Jenkins: kafka-trunk-jdk8 #4744

[jira] [Created] (KAFKA-10309) KafkaProducer's sendOffsetsToTransaction should not block infinitively

Re: [VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

Re: [DISCUSS] KIP-406: GlobalStreamThread should honor custom reset policy

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

21 matches

Site Navigation

Mail list logo

Footer information