Re: Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Chia-Ping Tsai]

Fair enough. I will isolate KIP-706 later.

On 2021/01/30 20:18:42 Ismael Juma wrote:
> I agree with Jason and would rather tackle this as one KIP and the error
> handling in another. The error handling can follow this one once we agree
> on the method signatures. Furthermore, we should align with the longer term
> plan for the AdminClient that returns KafkaFuture today.
> 
> Ismael
> 
> On Sat, Jan 30, 2021 at 11:40 AM Chia-Ping Tsai  wrote:
> 
> > I'd like to merge KIP-706 to KIP-691 as it can bring a comprehensive
> > design for both new API and exception. The new exception should be included
> > by the new API also.
> >
> > On 2021/01/30 19:30:40, Jason Gustafson  wrote:
> > > I think this still makes sense as a separate KIP. For KIP-691, we are
> > just
> > > looking to help define the error contract for the new API.
> > >
> > > -Jason
> > >
> > > On Sat, Jan 30, 2021 at 8:39 AM Ismael Juma  wrote:
> > >
> > > > Are we saying that we won't pursue this KIP in favor of the other one?
> > > >
> > > > Ismael
> > > >
> > > > On Sat, Jan 30, 2021, 4:15 AM Chia-Ping Tsai 
> > wrote:
> > > >
> > > > > hi Jason
> > > > >
> > > > > Thanks for your response. "transmit" is good to me.
> > > > >
> > > > > As we discussed by email, KIP-706 is going to be merged to KIP-691(
> > > > > https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please feel
> > free
> > > > to
> > > > > replace "produce" by "transmit" in KIP-691.
> > > > >
> > > > > Best,
> > > > > Chia-Ping
> > > > >
> > > > > On 2021/01/30 00:48:38, Jason Gustafson  wrote:
> > > > > > Hi Chia-Ping,
> > > > > >
> > > > > > I think this is a great idea. It is a pity that we cannot continue
> > to
> > > > use
> > > > > > the `send` verb, but I don't see how we can. I know we considered
> > > > > > `transmit` as another option which is closer to `send`. That would
> > > > avoid
> > > > > > the redundancy when people choose the common "producer" variable
> > name.
> > > > > >
> > > > > > producer.transmit
> > > > > >
> > > > > > instead of
> > > > > >
> > > > > > producer.produce
> > > > > >
> > > > > > A couple alternatives might be `write` or `append`. I'm happy with
> > > > > > `produce` as well, but curious if others have thoughts.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > >
> > > > > > On Wed, Jan 20, 2021 at 9:37 AM Chia-Ping Tsai <
> > chia7...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Dear all,
> > > > > > >
> > > > > > > I'd like to start the discussion thread for KIP-706:
> > > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459
> > > > > > >
> > > > > > > KIP-706 is proposing to introduce new API "CompletionStage
> > > > > > > produce(record)" to Producer. Kafka users can leverage
> > > > CompletionStage
> > > > > to
> > > > > > > write asynchronous non-blocking code. CompletionStage is more
> > > > powerful
> > > > > than
> > > > > > > Future and callback. Also, the code using Future and callback
> > can be
> > > > > easily
> > > > > > > re-written by CompletionStage.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Chia-Ping
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Chia-Ping Tsai]

It seems to me changing the input type might make complicate the migration from 
deprecated send method to new API.

Personally, I prefer to introduce a interface called “SendRecord” to replace 
ProducerRecord. Hence, the new API/classes is shown below.

1. CompletionStage send(SendRecord)
2. class ProducerRecord implement SendRecord
3. Introduce builder pattern for SendRecord

That includes following benefit.

1. Kafka users who don’t use both return type and callback do not need to 
change code even though we remove deprecated send methods. (of course, they 
still need to compile code with new Kafka)

2. Kafka users who need Future can easily migrate to new API by regex 
replacement. (cast ProduceRecord to SendCast and add toCompletableFuture)

3. It is easy to support topic id in the future. We can add new method to 
SendRecord builder. For example:

Builder topicName(String)
Builder topicId(UUID)

4. builder pattern can make code more readable. Especially, Produce record has 
a lot of fields which can be defined by users.
—
Chia-Ping

On 2021/01/30 22:50:36 Ismael Juma wrote:
> Another thing to think about: the consumer api currently has
> `subscribe(String|Pattern)` and a number of methods that accept
> `TopicPartition`. A similar approach could be used for the Consumer to work
> with topic ids or topic names. The consumer side also has to support
> regexes so it probably makes sense to have a separate interface.
> 
> Ismael
> 
> On Sat, Jan 30, 2021 at 2:40 PM Ismael Juma  wrote:
> 
> > I think this is a promising idea. I'd personally avoid the overload and
> > simply have a `Topic` type that implements `SendTarget`. It's a mix of both
> > proposals: strongly typed, no overloads and general class names that
> > implement `SendTarget`.
> >
> > Ismael
> >
> > On Sat, Jan 30, 2021 at 2:22 PM Jason Gustafson 
> > wrote:
> >
> >> Giving this a little more thought, I imagine sending to a topic is the
> >> most
> >> common case, so maybe it's an overload worth having. Also, if `SendTarget`
> >> is just a marker interface, we could let `TopicPartition` implement it
> >> directly. Then we have:
> >>
> >> interface SendTarget;
> >> class TopicPartition implements SendTarget;
> >>
> >> CompletionStage send(String topic, Record record);
> >> CompletionStage send(SendTarget target, Record record);
> >>
> >> The `SendTarget` would give us a lot of flexibility in the future. It
> >> would
> >> give us a couple options for topic ids. We could either have an overload
> >> of
> >> `send` which accepts `Uuid`, or we could add a `TopicId` type which
> >> implements `SendTarget`.
> >>
> >> -Jason
> >>
> >>
> >> On Sat, Jan 30, 2021 at 1:11 PM Jason Gustafson 
> >> wrote:
> >>
> >> > Yeah, good question. I guess we always tend to regret using lower-level
> >> > types in these APIs. Perhaps there should be some kind of interface:
> >> >
> >> > interface SendTarget
> >> > class TopicIdTarget implements SendTarget
> >> > class TopicTarget implements SendTarget
> >> > class TopicPartitionTarget implements SendTarget
> >> >
> >> > Then we just have:
> >> >
> >> > CompletionStage send(SendTarget target, Record record);
> >> >
> >> > Not sure if we could reuse `Record` in the consumer though. We do have
> >> > some state in `ConsumerRecord` which is not present in `ProducerRecord`
> >> > (e.g. offset). Perhaps we could provide a `Record` view from
> >> > `ConsumerRecord` for convenience. That would be useful for use cases
> >> which
> >> > involve reading from one topic and writing to another.
> >> >
> >> > -Jason
> >> >
> >> > On Sat, Jan 30, 2021 at 12:29 PM Ismael Juma  wrote:
> >> >
> >> >> Interesting idea. A couple of things to consider:
> >> >>
> >> >> 1. Would we introduce the Message concept to the Consumer too? I think
> >> >> that's what .NET does.
> >> >> 2. If we eventually allow a send to a topic id instead of topic name,
> >> >> would
> >> >> that result in two additional overloads?
> >> >>
> >> >> Ismael
> >> >>
> >> >> On Sat, Jan 30, 2021 at 11:38 AM Jason Gustafson 
> >> >> wrote:
> >> >>
> >> >> > For the sake of having another option to shoot down, we could take a
> >> >> page
> >> >> > from the .net client and separate the message data from the
> >> destination
> >> >> > (i.e. topic or partition). This would get around the need to use a
> >> new
> >> >> > verb. For example:
> >> >> >
> >> >> > CompletionStage send(String topic, Message message);
> >> >> > CompletionStage send(TopicPartition topicPartition,
> >> >> Message
> >> >> > message);
> >> >> >
> >> >> > -Jason
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Sat, Jan 30, 2021 at 11:30 AM Jason Gustafson  >> >
> >> >> > wrote:
> >> >> >
> >> >> > > I think this still makes sense as a separate KIP. For KIP-691, we
> >> are
> >> >> > just
> >> >> > > looking to help define the error contract for the new API.
> >> >> > >
> >> >> > > -Jason
> >> >> > >
> >> >> > > On Sat, Jan 30, 2021 at 8:39 AM Ismael Juma 
> >> >> wrote:
> >> >> > >
> >> >> > >> Are we sayin

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #476

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-12236; New meta.properties logic for KIP-500 (#9967)


--
[...truncated 3.58 MB...]

ZooKeeperClientTest > testPipelinedGetData() PASSED

ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChange() STARTED

ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChange() PASSED

ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren() STARTED

ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren() PASSED

ZooKeeperClientTest > testSetDataExistingZNode() STARTED

ZooKeeperClientTest > testSetDataExistingZNode() PASSED

ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChangeNotTriggered() 
STARTED

ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChangeNotTriggered() 
PASSED

ZooKeeperClientTest > testMixedPipeline() STARTED

ZooKeeperClientTest > testMixedPipeline() PASSED

ZooKeeperClientTest > testGetDataExistingZNode() STARTED

ZooKeeperClientTest > testGetDataExistingZNode() PASSED

ZooKeeperClientTest > testDeleteExistingZNode() STARTED

ZooKeeperClientTest > testDeleteExistingZNode() PASSED

ZooKeeperClientTest > testSessionExpiry() STARTED

ZooKeeperClientTest > testSessionExpiry() PASSED

ZooKeeperClientTest > testSetDataNonExistentZNode() STARTED

ZooKeeperClientTest > testSetDataNonExistentZNode() PASSED

ZooKeeperClientTest > testConnectionViaNettyClient() STARTED

ZooKeeperClientTest > testConnectionViaNettyClient() PASSED

ZooKeeperClientTest > testDeleteNonExistentZNode() STARTED

ZooKeeperClientTest > testDeleteNonExistentZNode() PASSED

ZooKeeperClientTest > testExistsExistingZNode() STARTED

ZooKeeperClientTest > testExistsExistingZNode() PASSED

ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics() STARTED

ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics() PASSED

ZooKeeperClientTest > testZNodeChangeHandlerForDeletion() STARTED

ZooKeeperClientTest > testZNodeChangeHandlerForDeletion() PASSED

ZooKeeperClientTest > testGetAclNonExistentZNode() STARTED

ZooKeeperClientTest > testGetAclNonExistentZNode() PASSED

ZooKeeperClientTest > testStateChangeHandlerForAuthFailure() STARTED

ZooKeeperClientTest > testStateChangeHandlerForAuthFailure() PASSED

ResourceTypeTest > testJavaConversions() STARTED

ResourceTypeTest > testJavaConversions() PASSED

ResourceTypeTest > testFromString() STARTED

ResourceTypeTest > testFromString() PASSED

OperationTest > testJavaConversions() STARTED

OperationTest > testJavaConversions() PASSED

DelegationTokenManagerTest > testPeriodicTokenExpiry() STARTED

DelegationTokenManagerTest > testPeriodicTokenExpiry() PASSED

DelegationTokenManagerTest > testTokenRequestsWithDelegationTokenDisabled() 
STARTED

DelegationTokenManagerTest > testTokenRequestsWithDelegationTokenDisabled() 
PASSED

DelegationTokenManagerTest > testDescribeToken() STARTED

DelegationTokenManagerTest > testDescribeToken() PASSED

DelegationTokenManagerTest > testCreateToken() STARTED

DelegationTokenManagerTest > testCreateToken() PASSED

DelegationTokenManagerTest > testExpireToken() STARTED

DelegationTokenManagerTest > testExpireToken() PASSED

DelegationTokenManagerTest > testRenewToken() STARTED

DelegationTokenManagerTest > testRenewToken() PASSED

DelegationTokenManagerTest > testRemoveTokenHmac() STARTED

DelegationTokenManagerTest > testRemoveTokenHmac() PASSED

AclAuthorizerWithZkSaslTest > testAclUpdateWithSessionExpiration() STARTED

AclAuthorizerWithZkSaslTest > testAclUpdateWithSessionExpiration() PASSED

AclAuthorizerWithZkSaslTest > testAclUpdateWithAuthFailure() STARTED

AclAuthorizerWithZkSaslTest > testAclUpdateWithAuthFailure() PASSED

DefaultMessageFormatterTest > [1] name=print nothing, 
record=ConsumerRecord(topic = someTopic, partition = 9, leaderEpoch = null, 
offset = 9876, CreateTime = 1234, serialized key size = 0, serialized value 
size = 0, headers = RecordHeaders(headers = [RecordHeader(key = h1, value = 
[118, 49]), RecordHeader(key = h2, value = [118, 50])], isReadOnly = false), 
key = [B@5be2d170, value = [B@1a693eb6), properties=Map(print.value -> false), 
expected= STARTED

DefaultMessageFormatterTest > [1] name=print nothing, 
record=ConsumerRecord(topic = someTopic, partition = 9, leaderEpoch = null, 
offset = 9876, CreateTime = 1234, serialized key size = 0, serialized value 
size = 0, headers = RecordHeaders(headers = [RecordHeader(key = h1, value = 
[118, 49]), RecordHeader(key = h2, value = [118, 50])], isReadOnly = false), 
key = [B@5be2d170, value = [B@1a693eb6), properties=Map(print.value -> false), 
expected= PASSED

DefaultMessageFormatterTest > [2] name=print key, record=ConsumerRecord(topic = 
someTopic, partition = 9, leaderEpoch = null, offset = 9876, CreateTime = 1234, 
serialized key size = 0, serialized value size = 0, headers = 
RecordHeaders(headers = [RecordHeader(key = h1, value = [118, 49]), 
RecordHeader(key = h

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #427

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-12236; New meta.properties logic for KIP-500 (#9967)


--
[...truncated 3.57 MB...]

LogValidatorTest > testUncompressedBatchWithoutRecordsNotAllowed() STARTED

LogValidatorTest > testUncompressedBatchWithoutRecordsNotAllowed() PASSED

LogValidatorTest > testOffsetAssignmentAfterDownConversionV1ToV0NonCompressed() 
STARTED

LogValidatorTest > testOffsetAssignmentAfterDownConversionV1ToV0NonCompressed() 
PASSED

LogValidatorTest > testAbsoluteOffsetAssignmentNonCompressed() STARTED

LogValidatorTest > testAbsoluteOffsetAssignmentNonCompressed() PASSED

LogValidatorTest > testOffsetAssignmentAfterDownConversionV2ToV1Compressed() 
STARTED

LogValidatorTest > testOffsetAssignmentAfterDownConversionV2ToV1Compressed() 
PASSED

LogValidatorTest > testOffsetAssignmentAfterDownConversionV1ToV0Compressed() 
STARTED

LogValidatorTest > testOffsetAssignmentAfterDownConversionV1ToV0Compressed() 
PASSED

LogValidatorTest > testOffsetAssignmentAfterUpConversionV0ToV2Compressed() 
STARTED

LogValidatorTest > testOffsetAssignmentAfterUpConversionV0ToV2Compressed() 
PASSED

LogValidatorTest > testNonCompressedV1() STARTED

LogValidatorTest > testNonCompressedV1() PASSED

LogValidatorTest > testNonCompressedV2() STARTED

LogValidatorTest > testNonCompressedV2() PASSED

LogValidatorTest > testOffsetAssignmentAfterUpConversionV1ToV2NonCompressed() 
STARTED

LogValidatorTest > testOffsetAssignmentAfterUpConversionV1ToV2NonCompressed() 
PASSED

LogValidatorTest > testInvalidCreateTimeCompressedV1() STARTED

LogValidatorTest > testInvalidCreateTimeCompressedV1() PASSED

LogValidatorTest > testInvalidCreateTimeCompressedV2() STARTED

LogValidatorTest > testInvalidCreateTimeCompressedV2() PASSED

LogValidatorTest > testNonIncreasingOffsetRecordBatchHasMetricsLogged() STARTED

LogValidatorTest > testNonIncreasingOffsetRecordBatchHasMetricsLogged() PASSED

LogValidatorTest > testRecompressionV1() STARTED

LogValidatorTest > testRecompressionV1() PASSED

LogValidatorTest > testRecompressionV2() STARTED

LogValidatorTest > testRecompressionV2() PASSED

ProducerStateManagerTest > testSkipEmptyTransactions() STARTED

ProducerStateManagerTest > testSkipEmptyTransactions() PASSED

ProducerStateManagerTest > testControlRecordBumpsProducerEpoch() STARTED

ProducerStateManagerTest > testControlRecordBumpsProducerEpoch() PASSED

ProducerStateManagerTest > testProducerSequenceWithWrapAroundBatchRecord() 
STARTED

ProducerStateManagerTest > testProducerSequenceWithWrapAroundBatchRecord() 
PASSED

ProducerStateManagerTest > testCoordinatorFencing() STARTED

ProducerStateManagerTest > testCoordinatorFencing() PASSED

ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile() STARTED

ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile() PASSED

ProducerStateManagerTest > testTruncateFullyAndStartAt() STARTED

ProducerStateManagerTest > testTruncateFullyAndStartAt() PASSED

ProducerStateManagerTest > testRemoveExpiredPidsOnReload() STARTED

ProducerStateManagerTest > testRemoveExpiredPidsOnReload() PASSED

ProducerStateManagerTest > testRecoverFromSnapshotFinishedTransaction() STARTED

ProducerStateManagerTest > testRecoverFromSnapshotFinishedTransaction() PASSED

ProducerStateManagerTest > testOutOfSequenceAfterControlRecordEpochBump() 
STARTED

ProducerStateManagerTest > testOutOfSequenceAfterControlRecordEpochBump() PASSED

ProducerStateManagerTest > testFirstUnstableOffsetAfterTruncation() STARTED

ProducerStateManagerTest > testFirstUnstableOffsetAfterTruncation() PASSED

ProducerStateManagerTest > testTakeSnapshot() STARTED

ProducerStateManagerTest > testTakeSnapshot() PASSED

ProducerStateManagerTest > testRecoverFromSnapshotUnfinishedTransaction() 
STARTED

ProducerStateManagerTest > testRecoverFromSnapshotUnfinishedTransaction() PASSED

ProducerStateManagerTest > testDeleteSnapshotsBefore() STARTED

ProducerStateManagerTest > testDeleteSnapshotsBefore() PASSED

ProducerStateManagerTest > testAppendEmptyControlBatch() STARTED

ProducerStateManagerTest > testAppendEmptyControlBatch() PASSED

ProducerStateManagerTest > testNoValidationOnFirstEntryWhenLoadingLog() STARTED

ProducerStateManagerTest > testNoValidationOnFirstEntryWhenLoadingLog() PASSED

ProducerStateManagerTest > testRemoveStraySnapshotsKeepCleanShutdownSnapshot() 
STARTED

ProducerStateManagerTest > testRemoveStraySnapshotsKeepCleanShutdownSnapshot() 
PASSED

ProducerStateManagerTest > testRemoveAllStraySnapshots() STARTED

ProducerStateManagerTest > testRemoveAllStraySnapshots() PASSED

ProducerStateManagerTest > testLoadFromEmptySnapshotFile() STARTED

ProducerStateManagerTest > testLoadFromEmptySnapshotFile() PASSED

ProducerStateManagerTest > testProducersWithOngoingTransactionsDontExpire() 
STARTED

ProducerStateManagerTest > testProducersWithOngoingTransactionsDontExpire() 
PASSED

ProducerStat

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #460

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-12236; New meta.properties logic for KIP-500 (#9967)


--
[...truncated 3.60 MB...]
KafkaZkClientTest > testSetTopicPartitionStatesRaw() STARTED

KafkaZkClientTest > testSetTopicPartitionStatesRaw() PASSED

KafkaZkClientTest > testAclManagementMethods() STARTED

KafkaZkClientTest > testAclManagementMethods() PASSED

KafkaZkClientTest > testPreferredReplicaElectionMethods() STARTED

KafkaZkClientTest > testPreferredReplicaElectionMethods() PASSED

KafkaZkClientTest > testPropagateLogDir() STARTED

KafkaZkClientTest > testPropagateLogDir() PASSED

KafkaZkClientTest > testGetDataAndStat() STARTED

KafkaZkClientTest > testGetDataAndStat() PASSED

KafkaZkClientTest > testReassignPartitionsInProgress() STARTED

KafkaZkClientTest > testReassignPartitionsInProgress() PASSED

KafkaZkClientTest > testCreateTopLevelPaths() STARTED

KafkaZkClientTest > testCreateTopLevelPaths() PASSED

KafkaZkClientTest > testGetAllTopicsInClusterDoesNotTriggerWatch() STARTED

KafkaZkClientTest > testGetAllTopicsInClusterDoesNotTriggerWatch() PASSED

KafkaZkClientTest > testIsrChangeNotificationGetters() STARTED

KafkaZkClientTest > testIsrChangeNotificationGetters() PASSED

KafkaZkClientTest > testLogDirEventNotificationsDeletion() STARTED

KafkaZkClientTest > testLogDirEventNotificationsDeletion() PASSED

KafkaZkClientTest > testGetLogConfigs() STARTED

KafkaZkClientTest > testGetLogConfigs() PASSED

KafkaZkClientTest > testBrokerSequenceIdMethods() STARTED

KafkaZkClientTest > testBrokerSequenceIdMethods() PASSED

KafkaZkClientTest > testAclMethods() STARTED

KafkaZkClientTest > testAclMethods() PASSED

KafkaZkClientTest > testCreateSequentialPersistentPath() STARTED

KafkaZkClientTest > testCreateSequentialPersistentPath() PASSED

KafkaZkClientTest > testConditionalUpdatePath() STARTED

KafkaZkClientTest > testConditionalUpdatePath() PASSED

KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() STARTED

KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() PASSED

KafkaZkClientTest > testDeleteTopicZNode() STARTED

KafkaZkClientTest > testDeleteTopicZNode() PASSED

KafkaZkClientTest > testDeletePath() STARTED

KafkaZkClientTest > testDeletePath() PASSED

KafkaZkClientTest > testGetBrokerMethods() STARTED

KafkaZkClientTest > testGetBrokerMethods() PASSED

KafkaZkClientTest > testCreateTokenChangeNotification() STARTED

KafkaZkClientTest > testCreateTokenChangeNotification() PASSED

KafkaZkClientTest > testGetTopicsAndPartitions() STARTED

KafkaZkClientTest > testGetTopicsAndPartitions() PASSED

KafkaZkClientTest > testRegisterBrokerInfo() STARTED

KafkaZkClientTest > testRegisterBrokerInfo() PASSED

KafkaZkClientTest > testRetryRegisterBrokerInfo() STARTED

KafkaZkClientTest > testRetryRegisterBrokerInfo() PASSED

KafkaZkClientTest > testConsumerOffsetPath() STARTED

KafkaZkClientTest > testConsumerOffsetPath() PASSED

KafkaZkClientTest > testDeleteRecursiveWithControllerEpochVersionCheck() STARTED

KafkaZkClientTest > testDeleteRecursiveWithControllerEpochVersionCheck() PASSED

KafkaZkClientTest > testTopicAssignments() STARTED

KafkaZkClientTest > testTopicAssignments() PASSED

KafkaZkClientTest > testControllerManagementMethods() STARTED

KafkaZkClientTest > testControllerManagementMethods() PASSED

KafkaZkClientTest > testTopicAssignmentMethods() STARTED

KafkaZkClientTest > testTopicAssignmentMethods() PASSED

KafkaZkClientTest > testConnectionViaNettyClient() STARTED

KafkaZkClientTest > testConnectionViaNettyClient() PASSED

KafkaZkClientTest > testPropagateIsrChanges() STARTED

KafkaZkClientTest > testPropagateIsrChanges() PASSED

KafkaZkClientTest > testControllerEpochMethods() STARTED

KafkaZkClientTest > testControllerEpochMethods() PASSED

KafkaZkClientTest > testDeleteRecursive() STARTED

KafkaZkClientTest > testDeleteRecursive() PASSED

KafkaZkClientTest > testGetTopicPartitionStates() STARTED

KafkaZkClientTest > testGetTopicPartitionStates() PASSED

KafkaZkClientTest > testCreateConfigChangeNotification() STARTED

KafkaZkClientTest > testCreateConfigChangeNotification() PASSED

KafkaZkClientTest > testDelegationTokenMethods() STARTED

KafkaZkClientTest > testDelegationTokenMethods() PASSED

LiteralAclStoreTest > shouldHaveCorrectPaths() STARTED

LiteralAclStoreTest > shouldHaveCorrectPaths() PASSED

LiteralAclStoreTest > shouldRoundTripChangeNode() STARTED

LiteralAclStoreTest > shouldRoundTripChangeNode() PASSED

LiteralAclStoreTest > shouldThrowFromEncodeOnNoneLiteral() STARTED

LiteralAclStoreTest > shouldThrowFromEncodeOnNoneLiteral() PASSED

LiteralAclStoreTest > shouldWriteChangesToTheWritePath() STARTED

LiteralAclStoreTest > shouldWriteChangesToTheWritePath() PASSED

LiteralAclStoreTest > shouldHaveCorrectPatternType() STARTED

LiteralAclStoreTest > shouldHaveCorrectPatternType() PASSED

LiteralAclStoreTest > shouldDecodeResourc

[jira] [Resolved] (KAFKA-12236) Add version 1 of meta.properties for KIP-500

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-12236.
-
Resolution: Fixed

> Add version 1 of meta.properties for KIP-500
> 
>
> Key: KAFKA-12236
> URL: https://issues.apache.org/jira/browse/KAFKA-12236
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
>
> KIP-631 dictates a new meta.properties format with version=1. Unlike the old 
> meta.properties, this is required for initialization of the KIP-500 
> broker/controller. Unlike with ZK, we use `meta.properties` for the 
> initialization of the clusterId.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Ismael Juma]

Another thing to think about: the consumer api currently has
`subscribe(String|Pattern)` and a number of methods that accept
`TopicPartition`. A similar approach could be used for the Consumer to work
with topic ids or topic names. The consumer side also has to support
regexes so it probably makes sense to have a separate interface.

Ismael

On Sat, Jan 30, 2021 at 2:40 PM Ismael Juma  wrote:

> I think this is a promising idea. I'd personally avoid the overload and
> simply have a `Topic` type that implements `SendTarget`. It's a mix of both
> proposals: strongly typed, no overloads and general class names that
> implement `SendTarget`.
>
> Ismael
>
> On Sat, Jan 30, 2021 at 2:22 PM Jason Gustafson 
> wrote:
>
>> Giving this a little more thought, I imagine sending to a topic is the
>> most
>> common case, so maybe it's an overload worth having. Also, if `SendTarget`
>> is just a marker interface, we could let `TopicPartition` implement it
>> directly. Then we have:
>>
>> interface SendTarget;
>> class TopicPartition implements SendTarget;
>>
>> CompletionStage send(String topic, Record record);
>> CompletionStage send(SendTarget target, Record record);
>>
>> The `SendTarget` would give us a lot of flexibility in the future. It
>> would
>> give us a couple options for topic ids. We could either have an overload
>> of
>> `send` which accepts `Uuid`, or we could add a `TopicId` type which
>> implements `SendTarget`.
>>
>> -Jason
>>
>>
>> On Sat, Jan 30, 2021 at 1:11 PM Jason Gustafson 
>> wrote:
>>
>> > Yeah, good question. I guess we always tend to regret using lower-level
>> > types in these APIs. Perhaps there should be some kind of interface:
>> >
>> > interface SendTarget
>> > class TopicIdTarget implements SendTarget
>> > class TopicTarget implements SendTarget
>> > class TopicPartitionTarget implements SendTarget
>> >
>> > Then we just have:
>> >
>> > CompletionStage send(SendTarget target, Record record);
>> >
>> > Not sure if we could reuse `Record` in the consumer though. We do have
>> > some state in `ConsumerRecord` which is not present in `ProducerRecord`
>> > (e.g. offset). Perhaps we could provide a `Record` view from
>> > `ConsumerRecord` for convenience. That would be useful for use cases
>> which
>> > involve reading from one topic and writing to another.
>> >
>> > -Jason
>> >
>> > On Sat, Jan 30, 2021 at 12:29 PM Ismael Juma  wrote:
>> >
>> >> Interesting idea. A couple of things to consider:
>> >>
>> >> 1. Would we introduce the Message concept to the Consumer too? I think
>> >> that's what .NET does.
>> >> 2. If we eventually allow a send to a topic id instead of topic name,
>> >> would
>> >> that result in two additional overloads?
>> >>
>> >> Ismael
>> >>
>> >> On Sat, Jan 30, 2021 at 11:38 AM Jason Gustafson 
>> >> wrote:
>> >>
>> >> > For the sake of having another option to shoot down, we could take a
>> >> page
>> >> > from the .net client and separate the message data from the
>> destination
>> >> > (i.e. topic or partition). This would get around the need to use a
>> new
>> >> > verb. For example:
>> >> >
>> >> > CompletionStage send(String topic, Message message);
>> >> > CompletionStage send(TopicPartition topicPartition,
>> >> Message
>> >> > message);
>> >> >
>> >> > -Jason
>> >> >
>> >> >
>> >> >
>> >> > On Sat, Jan 30, 2021 at 11:30 AM Jason Gustafson > >
>> >> > wrote:
>> >> >
>> >> > > I think this still makes sense as a separate KIP. For KIP-691, we
>> are
>> >> > just
>> >> > > looking to help define the error contract for the new API.
>> >> > >
>> >> > > -Jason
>> >> > >
>> >> > > On Sat, Jan 30, 2021 at 8:39 AM Ismael Juma 
>> >> wrote:
>> >> > >
>> >> > >> Are we saying that we won't pursue this KIP in favor of the other
>> >> one?
>> >> > >>
>> >> > >> Ismael
>> >> > >>
>> >> > >> On Sat, Jan 30, 2021, 4:15 AM Chia-Ping Tsai > >
>> >> > wrote:
>> >> > >>
>> >> > >> > hi Jason
>> >> > >> >
>> >> > >> > Thanks for your response. "transmit" is good to me.
>> >> > >> >
>> >> > >> > As we discussed by email, KIP-706 is going to be merged to
>> KIP-691(
>> >> > >> > https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please
>> feel
>> >> > free
>> >> > >> to
>> >> > >> > replace "produce" by "transmit" in KIP-691.
>> >> > >> >
>> >> > >> > Best,
>> >> > >> > Chia-Ping
>> >> > >> >
>> >> > >> > On 2021/01/30 00:48:38, Jason Gustafson 
>> >> wrote:
>> >> > >> > > Hi Chia-Ping,
>> >> > >> > >
>> >> > >> > > I think this is a great idea. It is a pity that we cannot
>> >> continue
>> >> > to
>> >> > >> use
>> >> > >> > > the `send` verb, but I don't see how we can. I know we
>> considered
>> >> > >> > > `transmit` as another option which is closer to `send`. That
>> >> would
>> >> > >> avoid
>> >> > >> > > the redundancy when people choose the common "producer"
>> variable
>> >> > name.
>> >> > >> > >
>> >> > >> > > producer.transmit
>> >> > >> > >
>> >> > >> > > instead of
>> >> > >> > >
>> >> > >> > > producer.produce
>> >> > >> > >
>> >> > >> > > A couple alternatives mi

Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Ismael Juma]

I think this is a promising idea. I'd personally avoid the overload and
simply have a `Topic` type that implements `SendTarget`. It's a mix of both
proposals: strongly typed, no overloads and general class names that
implement `SendTarget`.

Ismael

On Sat, Jan 30, 2021 at 2:22 PM Jason Gustafson  wrote:

> Giving this a little more thought, I imagine sending to a topic is the most
> common case, so maybe it's an overload worth having. Also, if `SendTarget`
> is just a marker interface, we could let `TopicPartition` implement it
> directly. Then we have:
>
> interface SendTarget;
> class TopicPartition implements SendTarget;
>
> CompletionStage send(String topic, Record record);
> CompletionStage send(SendTarget target, Record record);
>
> The `SendTarget` would give us a lot of flexibility in the future. It would
> give us a couple options for topic ids. We could either have an overload of
> `send` which accepts `Uuid`, or we could add a `TopicId` type which
> implements `SendTarget`.
>
> -Jason
>
>
> On Sat, Jan 30, 2021 at 1:11 PM Jason Gustafson 
> wrote:
>
> > Yeah, good question. I guess we always tend to regret using lower-level
> > types in these APIs. Perhaps there should be some kind of interface:
> >
> > interface SendTarget
> > class TopicIdTarget implements SendTarget
> > class TopicTarget implements SendTarget
> > class TopicPartitionTarget implements SendTarget
> >
> > Then we just have:
> >
> > CompletionStage send(SendTarget target, Record record);
> >
> > Not sure if we could reuse `Record` in the consumer though. We do have
> > some state in `ConsumerRecord` which is not present in `ProducerRecord`
> > (e.g. offset). Perhaps we could provide a `Record` view from
> > `ConsumerRecord` for convenience. That would be useful for use cases
> which
> > involve reading from one topic and writing to another.
> >
> > -Jason
> >
> > On Sat, Jan 30, 2021 at 12:29 PM Ismael Juma  wrote:
> >
> >> Interesting idea. A couple of things to consider:
> >>
> >> 1. Would we introduce the Message concept to the Consumer too? I think
> >> that's what .NET does.
> >> 2. If we eventually allow a send to a topic id instead of topic name,
> >> would
> >> that result in two additional overloads?
> >>
> >> Ismael
> >>
> >> On Sat, Jan 30, 2021 at 11:38 AM Jason Gustafson 
> >> wrote:
> >>
> >> > For the sake of having another option to shoot down, we could take a
> >> page
> >> > from the .net client and separate the message data from the
> destination
> >> > (i.e. topic or partition). This would get around the need to use a new
> >> > verb. For example:
> >> >
> >> > CompletionStage send(String topic, Message message);
> >> > CompletionStage send(TopicPartition topicPartition,
> >> Message
> >> > message);
> >> >
> >> > -Jason
> >> >
> >> >
> >> >
> >> > On Sat, Jan 30, 2021 at 11:30 AM Jason Gustafson 
> >> > wrote:
> >> >
> >> > > I think this still makes sense as a separate KIP. For KIP-691, we
> are
> >> > just
> >> > > looking to help define the error contract for the new API.
> >> > >
> >> > > -Jason
> >> > >
> >> > > On Sat, Jan 30, 2021 at 8:39 AM Ismael Juma 
> >> wrote:
> >> > >
> >> > >> Are we saying that we won't pursue this KIP in favor of the other
> >> one?
> >> > >>
> >> > >> Ismael
> >> > >>
> >> > >> On Sat, Jan 30, 2021, 4:15 AM Chia-Ping Tsai 
> >> > wrote:
> >> > >>
> >> > >> > hi Jason
> >> > >> >
> >> > >> > Thanks for your response. "transmit" is good to me.
> >> > >> >
> >> > >> > As we discussed by email, KIP-706 is going to be merged to
> KIP-691(
> >> > >> > https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please
> feel
> >> > free
> >> > >> to
> >> > >> > replace "produce" by "transmit" in KIP-691.
> >> > >> >
> >> > >> > Best,
> >> > >> > Chia-Ping
> >> > >> >
> >> > >> > On 2021/01/30 00:48:38, Jason Gustafson 
> >> wrote:
> >> > >> > > Hi Chia-Ping,
> >> > >> > >
> >> > >> > > I think this is a great idea. It is a pity that we cannot
> >> continue
> >> > to
> >> > >> use
> >> > >> > > the `send` verb, but I don't see how we can. I know we
> considered
> >> > >> > > `transmit` as another option which is closer to `send`. That
> >> would
> >> > >> avoid
> >> > >> > > the redundancy when people choose the common "producer"
> variable
> >> > name.
> >> > >> > >
> >> > >> > > producer.transmit
> >> > >> > >
> >> > >> > > instead of
> >> > >> > >
> >> > >> > > producer.produce
> >> > >> > >
> >> > >> > > A couple alternatives might be `write` or `append`. I'm happy
> >> with
> >> > >> > > `produce` as well, but curious if others have thoughts.
> >> > >> > >
> >> > >> > > -Jason
> >> > >> > >
> >> > >> > >
> >> > >> > > On Wed, Jan 20, 2021 at 9:37 AM Chia-Ping Tsai <
> >> chia7...@apache.org
> >> > >
> >> > >> > wrote:
> >> > >> > >
> >> > >> > > > Dear all,
> >> > >> > > >
> >> > >> > > > I'd like to start the discussion thread for KIP-706:
> >> > >> > > >
> >> > >> >
> >> > >>
> >> >
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459
> >> > >> > > >
>

Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Jason Gustafson]

Giving this a little more thought, I imagine sending to a topic is the most
common case, so maybe it's an overload worth having. Also, if `SendTarget`
is just a marker interface, we could let `TopicPartition` implement it
directly. Then we have:

interface SendTarget;
class TopicPartition implements SendTarget;

CompletionStage send(String topic, Record record);
CompletionStage send(SendTarget target, Record record);

The `SendTarget` would give us a lot of flexibility in the future. It would
give us a couple options for topic ids. We could either have an overload of
`send` which accepts `Uuid`, or we could add a `TopicId` type which
implements `SendTarget`.

-Jason


On Sat, Jan 30, 2021 at 1:11 PM Jason Gustafson  wrote:

> Yeah, good question. I guess we always tend to regret using lower-level
> types in these APIs. Perhaps there should be some kind of interface:
>
> interface SendTarget
> class TopicIdTarget implements SendTarget
> class TopicTarget implements SendTarget
> class TopicPartitionTarget implements SendTarget
>
> Then we just have:
>
> CompletionStage send(SendTarget target, Record record);
>
> Not sure if we could reuse `Record` in the consumer though. We do have
> some state in `ConsumerRecord` which is not present in `ProducerRecord`
> (e.g. offset). Perhaps we could provide a `Record` view from
> `ConsumerRecord` for convenience. That would be useful for use cases which
> involve reading from one topic and writing to another.
>
> -Jason
>
> On Sat, Jan 30, 2021 at 12:29 PM Ismael Juma  wrote:
>
>> Interesting idea. A couple of things to consider:
>>
>> 1. Would we introduce the Message concept to the Consumer too? I think
>> that's what .NET does.
>> 2. If we eventually allow a send to a topic id instead of topic name,
>> would
>> that result in two additional overloads?
>>
>> Ismael
>>
>> On Sat, Jan 30, 2021 at 11:38 AM Jason Gustafson 
>> wrote:
>>
>> > For the sake of having another option to shoot down, we could take a
>> page
>> > from the .net client and separate the message data from the destination
>> > (i.e. topic or partition). This would get around the need to use a new
>> > verb. For example:
>> >
>> > CompletionStage send(String topic, Message message);
>> > CompletionStage send(TopicPartition topicPartition,
>> Message
>> > message);
>> >
>> > -Jason
>> >
>> >
>> >
>> > On Sat, Jan 30, 2021 at 11:30 AM Jason Gustafson 
>> > wrote:
>> >
>> > > I think this still makes sense as a separate KIP. For KIP-691, we are
>> > just
>> > > looking to help define the error contract for the new API.
>> > >
>> > > -Jason
>> > >
>> > > On Sat, Jan 30, 2021 at 8:39 AM Ismael Juma 
>> wrote:
>> > >
>> > >> Are we saying that we won't pursue this KIP in favor of the other
>> one?
>> > >>
>> > >> Ismael
>> > >>
>> > >> On Sat, Jan 30, 2021, 4:15 AM Chia-Ping Tsai 
>> > wrote:
>> > >>
>> > >> > hi Jason
>> > >> >
>> > >> > Thanks for your response. "transmit" is good to me.
>> > >> >
>> > >> > As we discussed by email, KIP-706 is going to be merged to KIP-691(
>> > >> > https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please feel
>> > free
>> > >> to
>> > >> > replace "produce" by "transmit" in KIP-691.
>> > >> >
>> > >> > Best,
>> > >> > Chia-Ping
>> > >> >
>> > >> > On 2021/01/30 00:48:38, Jason Gustafson 
>> wrote:
>> > >> > > Hi Chia-Ping,
>> > >> > >
>> > >> > > I think this is a great idea. It is a pity that we cannot
>> continue
>> > to
>> > >> use
>> > >> > > the `send` verb, but I don't see how we can. I know we considered
>> > >> > > `transmit` as another option which is closer to `send`. That
>> would
>> > >> avoid
>> > >> > > the redundancy when people choose the common "producer" variable
>> > name.
>> > >> > >
>> > >> > > producer.transmit
>> > >> > >
>> > >> > > instead of
>> > >> > >
>> > >> > > producer.produce
>> > >> > >
>> > >> > > A couple alternatives might be `write` or `append`. I'm happy
>> with
>> > >> > > `produce` as well, but curious if others have thoughts.
>> > >> > >
>> > >> > > -Jason
>> > >> > >
>> > >> > >
>> > >> > > On Wed, Jan 20, 2021 at 9:37 AM Chia-Ping Tsai <
>> chia7...@apache.org
>> > >
>> > >> > wrote:
>> > >> > >
>> > >> > > > Dear all,
>> > >> > > >
>> > >> > > > I'd like to start the discussion thread for KIP-706:
>> > >> > > >
>> > >> >
>> > >>
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459
>> > >> > > >
>> > >> > > > KIP-706 is proposing to introduce new API "CompletionStage
>> > >> > > > produce(record)" to Producer. Kafka users can leverage
>> > >> CompletionStage
>> > >> > to
>> > >> > > > write asynchronous non-blocking code. CompletionStage is more
>> > >> powerful
>> > >> > than
>> > >> > > > Future and callback. Also, the code using Future and callback
>> can
>> > be
>> > >> > easily
>> > >> > > > re-written by CompletionStage.
>> > >> > > >
>> > >> > > > Cheers,
>> > >> > > > Chia-Ping
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>

Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Jason Gustafson]

Yeah, good question. I guess we always tend to regret using lower-level
types in these APIs. Perhaps there should be some kind of interface:

interface SendTarget
class TopicIdTarget implements SendTarget
class TopicTarget implements SendTarget
class TopicPartitionTarget implements SendTarget

Then we just have:

CompletionStage send(SendTarget target, Record record);

Not sure if we could reuse `Record` in the consumer though. We do have some
state in `ConsumerRecord` which is not present in `ProducerRecord` (e.g.
offset). Perhaps we could provide a `Record` view from `ConsumerRecord` for
convenience. That would be useful for use cases which involve reading from
one topic and writing to another.

-Jason

On Sat, Jan 30, 2021 at 12:29 PM Ismael Juma  wrote:

> Interesting idea. A couple of things to consider:
>
> 1. Would we introduce the Message concept to the Consumer too? I think
> that's what .NET does.
> 2. If we eventually allow a send to a topic id instead of topic name, would
> that result in two additional overloads?
>
> Ismael
>
> On Sat, Jan 30, 2021 at 11:38 AM Jason Gustafson 
> wrote:
>
> > For the sake of having another option to shoot down, we could take a page
> > from the .net client and separate the message data from the destination
> > (i.e. topic or partition). This would get around the need to use a new
> > verb. For example:
> >
> > CompletionStage send(String topic, Message message);
> > CompletionStage send(TopicPartition topicPartition,
> Message
> > message);
> >
> > -Jason
> >
> >
> >
> > On Sat, Jan 30, 2021 at 11:30 AM Jason Gustafson 
> > wrote:
> >
> > > I think this still makes sense as a separate KIP. For KIP-691, we are
> > just
> > > looking to help define the error contract for the new API.
> > >
> > > -Jason
> > >
> > > On Sat, Jan 30, 2021 at 8:39 AM Ismael Juma  wrote:
> > >
> > >> Are we saying that we won't pursue this KIP in favor of the other one?
> > >>
> > >> Ismael
> > >>
> > >> On Sat, Jan 30, 2021, 4:15 AM Chia-Ping Tsai 
> > wrote:
> > >>
> > >> > hi Jason
> > >> >
> > >> > Thanks for your response. "transmit" is good to me.
> > >> >
> > >> > As we discussed by email, KIP-706 is going to be merged to KIP-691(
> > >> > https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please feel
> > free
> > >> to
> > >> > replace "produce" by "transmit" in KIP-691.
> > >> >
> > >> > Best,
> > >> > Chia-Ping
> > >> >
> > >> > On 2021/01/30 00:48:38, Jason Gustafson  wrote:
> > >> > > Hi Chia-Ping,
> > >> > >
> > >> > > I think this is a great idea. It is a pity that we cannot continue
> > to
> > >> use
> > >> > > the `send` verb, but I don't see how we can. I know we considered
> > >> > > `transmit` as another option which is closer to `send`. That would
> > >> avoid
> > >> > > the redundancy when people choose the common "producer" variable
> > name.
> > >> > >
> > >> > > producer.transmit
> > >> > >
> > >> > > instead of
> > >> > >
> > >> > > producer.produce
> > >> > >
> > >> > > A couple alternatives might be `write` or `append`. I'm happy with
> > >> > > `produce` as well, but curious if others have thoughts.
> > >> > >
> > >> > > -Jason
> > >> > >
> > >> > >
> > >> > > On Wed, Jan 20, 2021 at 9:37 AM Chia-Ping Tsai <
> chia7...@apache.org
> > >
> > >> > wrote:
> > >> > >
> > >> > > > Dear all,
> > >> > > >
> > >> > > > I'd like to start the discussion thread for KIP-706:
> > >> > > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459
> > >> > > >
> > >> > > > KIP-706 is proposing to introduce new API "CompletionStage
> > >> > > > produce(record)" to Producer. Kafka users can leverage
> > >> CompletionStage
> > >> > to
> > >> > > > write asynchronous non-blocking code. CompletionStage is more
> > >> powerful
> > >> > than
> > >> > > > Future and callback. Also, the code using Future and callback
> can
> > be
> > >> > easily
> > >> > > > re-written by CompletionStage.
> > >> > > >
> > >> > > > Cheers,
> > >> > > > Chia-Ping
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

[jira] [Created] (KAFKA-12260) PartitionsFor should not return null value

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-12260:
---

 Summary: PartitionsFor should not return null value
 Key: KAFKA-12260
 URL: https://issues.apache.org/jira/browse/KAFKA-12260
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Reporter: Boyang Chen


consumer.partitionsFor() could return null value when topic was not found. This 
was not properly documented and was error-prone when the return type was a 
list. We should fix the logic internally to prevent partitionsFor returning 
null result.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Ismael Juma]

Interesting idea. A couple of things to consider:

1. Would we introduce the Message concept to the Consumer too? I think
that's what .NET does.
2. If we eventually allow a send to a topic id instead of topic name, would
that result in two additional overloads?

Ismael

On Sat, Jan 30, 2021 at 11:38 AM Jason Gustafson  wrote:

> For the sake of having another option to shoot down, we could take a page
> from the .net client and separate the message data from the destination
> (i.e. topic or partition). This would get around the need to use a new
> verb. For example:
>
> CompletionStage send(String topic, Message message);
> CompletionStage send(TopicPartition topicPartition, Message
> message);
>
> -Jason
>
>
>
> On Sat, Jan 30, 2021 at 11:30 AM Jason Gustafson 
> wrote:
>
> > I think this still makes sense as a separate KIP. For KIP-691, we are
> just
> > looking to help define the error contract for the new API.
> >
> > -Jason
> >
> > On Sat, Jan 30, 2021 at 8:39 AM Ismael Juma  wrote:
> >
> >> Are we saying that we won't pursue this KIP in favor of the other one?
> >>
> >> Ismael
> >>
> >> On Sat, Jan 30, 2021, 4:15 AM Chia-Ping Tsai 
> wrote:
> >>
> >> > hi Jason
> >> >
> >> > Thanks for your response. "transmit" is good to me.
> >> >
> >> > As we discussed by email, KIP-706 is going to be merged to KIP-691(
> >> > https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please feel
> free
> >> to
> >> > replace "produce" by "transmit" in KIP-691.
> >> >
> >> > Best,
> >> > Chia-Ping
> >> >
> >> > On 2021/01/30 00:48:38, Jason Gustafson  wrote:
> >> > > Hi Chia-Ping,
> >> > >
> >> > > I think this is a great idea. It is a pity that we cannot continue
> to
> >> use
> >> > > the `send` verb, but I don't see how we can. I know we considered
> >> > > `transmit` as another option which is closer to `send`. That would
> >> avoid
> >> > > the redundancy when people choose the common "producer" variable
> name.
> >> > >
> >> > > producer.transmit
> >> > >
> >> > > instead of
> >> > >
> >> > > producer.produce
> >> > >
> >> > > A couple alternatives might be `write` or `append`. I'm happy with
> >> > > `produce` as well, but curious if others have thoughts.
> >> > >
> >> > > -Jason
> >> > >
> >> > >
> >> > > On Wed, Jan 20, 2021 at 9:37 AM Chia-Ping Tsai  >
> >> > wrote:
> >> > >
> >> > > > Dear all,
> >> > > >
> >> > > > I'd like to start the discussion thread for KIP-706:
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459
> >> > > >
> >> > > > KIP-706 is proposing to introduce new API "CompletionStage
> >> > > > produce(record)" to Producer. Kafka users can leverage
> >> CompletionStage
> >> > to
> >> > > > write asynchronous non-blocking code. CompletionStage is more
> >> powerful
> >> > than
> >> > > > Future and callback. Also, the code using Future and callback can
> be
> >> > easily
> >> > > > re-written by CompletionStage.
> >> > > >
> >> > > > Cheers,
> >> > > > Chia-Ping
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Ismael Juma]

I agree with Jason and would rather tackle this as one KIP and the error
handling in another. The error handling can follow this one once we agree
on the method signatures. Furthermore, we should align with the longer term
plan for the AdminClient that returns KafkaFuture today.

Ismael

On Sat, Jan 30, 2021 at 11:40 AM Chia-Ping Tsai  wrote:

> I'd like to merge KIP-706 to KIP-691 as it can bring a comprehensive
> design for both new API and exception. The new exception should be included
> by the new API also.
>
> On 2021/01/30 19:30:40, Jason Gustafson  wrote:
> > I think this still makes sense as a separate KIP. For KIP-691, we are
> just
> > looking to help define the error contract for the new API.
> >
> > -Jason
> >
> > On Sat, Jan 30, 2021 at 8:39 AM Ismael Juma  wrote:
> >
> > > Are we saying that we won't pursue this KIP in favor of the other one?
> > >
> > > Ismael
> > >
> > > On Sat, Jan 30, 2021, 4:15 AM Chia-Ping Tsai 
> wrote:
> > >
> > > > hi Jason
> > > >
> > > > Thanks for your response. "transmit" is good to me.
> > > >
> > > > As we discussed by email, KIP-706 is going to be merged to KIP-691(
> > > > https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please feel
> free
> > > to
> > > > replace "produce" by "transmit" in KIP-691.
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > > On 2021/01/30 00:48:38, Jason Gustafson  wrote:
> > > > > Hi Chia-Ping,
> > > > >
> > > > > I think this is a great idea. It is a pity that we cannot continue
> to
> > > use
> > > > > the `send` verb, but I don't see how we can. I know we considered
> > > > > `transmit` as another option which is closer to `send`. That would
> > > avoid
> > > > > the redundancy when people choose the common "producer" variable
> name.
> > > > >
> > > > > producer.transmit
> > > > >
> > > > > instead of
> > > > >
> > > > > producer.produce
> > > > >
> > > > > A couple alternatives might be `write` or `append`. I'm happy with
> > > > > `produce` as well, but curious if others have thoughts.
> > > > >
> > > > > -Jason
> > > > >
> > > > >
> > > > > On Wed, Jan 20, 2021 at 9:37 AM Chia-Ping Tsai <
> chia7...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Dear all,
> > > > > >
> > > > > > I'd like to start the discussion thread for KIP-706:
> > > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459
> > > > > >
> > > > > > KIP-706 is proposing to introduce new API "CompletionStage
> > > > > > produce(record)" to Producer. Kafka users can leverage
> > > CompletionStage
> > > > to
> > > > > > write asynchronous non-blocking code. CompletionStage is more
> > > powerful
> > > > than
> > > > > > Future and callback. Also, the code using Future and callback
> can be
> > > > easily
> > > > > > re-written by CompletionStage.
> > > > > >
> > > > > > Cheers,
> > > > > > Chia-Ping
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-691: Transactional Producer Exception Handling

[Chia-Ping Tsai]

> a richer return type. Let me know if you are good with this, and whether
> Chia-Ping is also happy here :)

sure. I'd like to join this bigger party :)

On 2021/01/30 01:03:57, Boyang Chen  wrote: 
> This is a great proposal Jason, I already integrated KIP-691 with KIP-706
> template to provide a new ProduceFailedException as well as a new
> Producer#produce API with CompletionStage as old send API replacement with
> a richer return type. Let me know if you are good with this, and whether
> Chia-Ping is also happy here :)
> 
> Best,
> Boyang
> 
> On Fri, Jan 29, 2021 at 4:23 PM Jason Gustafson  wrote:
> 
> > Hi Boyang,
> >
> > Ok, if we are going to ask all users to update their code anyway, maybe we
> > can get more bang out of this.
> >
> > One of the remaining problems is that a user will only get the richer
> > information if they are supplying a Callback. If they wait on the future,
> > then they will get the usual exception. Our hands are somewhat tied here
> > because of compatibility. One thing we have wanted to do for a long time is
> > introduce a new send method which returns
> > `CompletableFuture` instead. If we did so, then we wouldn't
> > need to change `Callback` at all. Instead we could make the contract of the
> > new api that it will always return `SendFailedException`, which could
> > provide a method to get the failure type.
> >
> > I think Chia-Ping has been thinking about this:
> > https://issues.apache.org/jira/browse/KAFKA-12227. Is there a way to
> > consolidate our efforts in order to avoid unnecessary noise for users? It
> > would be awesome to see a more modern API which also addresses the
> > confusing error handling.
> >
> > -Jason
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Jan 29, 2021 at 3:48 PM Boyang Chen 
> > wrote:
> >
> > > Thanks Jason, my hope is that we could streamline the callback function
> > > signature, so that in the next major release (3.0),
> > > onCompletion(RecordMetadata metadata, SendFailure sendFailure) will
> > become
> > > the only API to be implemented.
> > >
> > > On Fri, Jan 29, 2021 at 3:42 PM Jason Gustafson 
> > > wrote:
> > >
> > > > Hi Boyang,
> > > >
> > > > Do you think it's necessary to deprecate the old `onCompletion`
> > > callback? I
> > > > was thinking there's probably no harm leaving it around. Users might
> > not
> > > > care about the failure type. Other than that, it looks good to me.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Thu, Jan 28, 2021 at 6:58 PM Boyang Chen <
> > reluctanthero...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hey Guozhang,
> > > > >
> > > > > the CommitFailedException wrapping would cover all the non-fatal
> > > > exceptions
> > > > > as we listed out in the KIP, generally speaking any exception that
> > > could
> > > > > recover safely by calling abortTxn should be wrapped.
> > > > >
> > > > > Best,
> > > > > Boyang
> > > > >
> > > > > On Thu, Jan 28, 2021 at 5:22 PM Guozhang Wang 
> > > > wrote:
> > > > >
> > > > > > Hey Boyang,
> > > > > >
> > > > > > I think maybe there's something cracking here :) I'm just asking
> > for
> > > > > > clarifications that as of today, which non-fatal exceptions the
> > newly
> > > > > > introduced CommitFailedException would cover, and it seems to be
> > only
> > > > 1)
> > > > > > unknown pid, 2) invalid pid mapping, and 3) concurrent
> > transactions.
> > > Is
> > > > > > that correct?
> > > > > >
> > > > > > On Thu, Jan 28, 2021 at 5:06 PM Boyang Chen <
> > > > reluctanthero...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Guozhang, I think TimeoutException would not be covered here
> > as
> > > > it
> > > > > > > potentially has a risk of hitting an illegal state on the broker
> > > side
> > > > > > when
> > > > > > > the previous commit was actually successful. Users should try to
> > > > > increase
> > > > > > > their max.block.ms to avoid hitting the timeout as a base
> > > > suggestion,
> > > > > > > which
> > > > > > > is discussed in the KIP.
> > > > > > >
> > > > > > > As for the umbrella exception, I agree it has values to some
> > > extent,
> > > > > but
> > > > > > in
> > > > > > > terms of adoption difficulty, additional exception types usually
> > > make
> > > > > the
> > > > > > > error handling more complex than simplified, and we are doing our
> > > > best
> > > > > to
> > > > > > > avoid compatibility issues caused by wrapping previously thrown
> > raw
> > > > > > > exceptions that invalidates user error handling. And I'm also not
> > > in
> > > > > > favor
> > > > > > > of our exception hierarchy today where all exceptions are
> > > subclasses
> > > > of
> > > > > > > KafkaException, and we already mixed its handling with other
> > > specific
> > > > > > > exception types that could be caught as superclass
> > KafkaException.
> > > > That
> > > > > > > could be addressed in a separate proposal if we see necessary.
> > > > > > >
> > > > > > > Boyang
> > > > > > >
> > > > > > > On Thu, Jan 28, 2021 at 3:28 PM Guozhang Wang <
> 

Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Chia-Ping Tsai]

I'd like to merge KIP-706 to KIP-691 as it can bring a comprehensive design for 
both new API and exception. The new exception should be included by the new API 
also.

On 2021/01/30 19:30:40, Jason Gustafson  wrote: 
> I think this still makes sense as a separate KIP. For KIP-691, we are just
> looking to help define the error contract for the new API.
> 
> -Jason
> 
> On Sat, Jan 30, 2021 at 8:39 AM Ismael Juma  wrote:
> 
> > Are we saying that we won't pursue this KIP in favor of the other one?
> >
> > Ismael
> >
> > On Sat, Jan 30, 2021, 4:15 AM Chia-Ping Tsai  wrote:
> >
> > > hi Jason
> > >
> > > Thanks for your response. "transmit" is good to me.
> > >
> > > As we discussed by email, KIP-706 is going to be merged to KIP-691(
> > > https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please feel free
> > to
> > > replace "produce" by "transmit" in KIP-691.
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > On 2021/01/30 00:48:38, Jason Gustafson  wrote:
> > > > Hi Chia-Ping,
> > > >
> > > > I think this is a great idea. It is a pity that we cannot continue to
> > use
> > > > the `send` verb, but I don't see how we can. I know we considered
> > > > `transmit` as another option which is closer to `send`. That would
> > avoid
> > > > the redundancy when people choose the common "producer" variable name.
> > > >
> > > > producer.transmit
> > > >
> > > > instead of
> > > >
> > > > producer.produce
> > > >
> > > > A couple alternatives might be `write` or `append`. I'm happy with
> > > > `produce` as well, but curious if others have thoughts.
> > > >
> > > > -Jason
> > > >
> > > >
> > > > On Wed, Jan 20, 2021 at 9:37 AM Chia-Ping Tsai 
> > > wrote:
> > > >
> > > > > Dear all,
> > > > >
> > > > > I'd like to start the discussion thread for KIP-706:
> > > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459
> > > > >
> > > > > KIP-706 is proposing to introduce new API "CompletionStage
> > > > > produce(record)" to Producer. Kafka users can leverage
> > CompletionStage
> > > to
> > > > > write asynchronous non-blocking code. CompletionStage is more
> > powerful
> > > than
> > > > > Future and callback. Also, the code using Future and callback can be
> > > easily
> > > > > re-written by CompletionStage.
> > > > >
> > > > > Cheers,
> > > > > Chia-Ping
> > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Jason Gustafson]

For the sake of having another option to shoot down, we could take a page
from the .net client and separate the message data from the destination
(i.e. topic or partition). This would get around the need to use a new
verb. For example:

CompletionStage send(String topic, Message message);
CompletionStage send(TopicPartition topicPartition, Message
message);

-Jason



On Sat, Jan 30, 2021 at 11:30 AM Jason Gustafson  wrote:

> I think this still makes sense as a separate KIP. For KIP-691, we are just
> looking to help define the error contract for the new API.
>
> -Jason
>
> On Sat, Jan 30, 2021 at 8:39 AM Ismael Juma  wrote:
>
>> Are we saying that we won't pursue this KIP in favor of the other one?
>>
>> Ismael
>>
>> On Sat, Jan 30, 2021, 4:15 AM Chia-Ping Tsai  wrote:
>>
>> > hi Jason
>> >
>> > Thanks for your response. "transmit" is good to me.
>> >
>> > As we discussed by email, KIP-706 is going to be merged to KIP-691(
>> > https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please feel free
>> to
>> > replace "produce" by "transmit" in KIP-691.
>> >
>> > Best,
>> > Chia-Ping
>> >
>> > On 2021/01/30 00:48:38, Jason Gustafson  wrote:
>> > > Hi Chia-Ping,
>> > >
>> > > I think this is a great idea. It is a pity that we cannot continue to
>> use
>> > > the `send` verb, but I don't see how we can. I know we considered
>> > > `transmit` as another option which is closer to `send`. That would
>> avoid
>> > > the redundancy when people choose the common "producer" variable name.
>> > >
>> > > producer.transmit
>> > >
>> > > instead of
>> > >
>> > > producer.produce
>> > >
>> > > A couple alternatives might be `write` or `append`. I'm happy with
>> > > `produce` as well, but curious if others have thoughts.
>> > >
>> > > -Jason
>> > >
>> > >
>> > > On Wed, Jan 20, 2021 at 9:37 AM Chia-Ping Tsai 
>> > wrote:
>> > >
>> > > > Dear all,
>> > > >
>> > > > I'd like to start the discussion thread for KIP-706:
>> > > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459
>> > > >
>> > > > KIP-706 is proposing to introduce new API "CompletionStage
>> > > > produce(record)" to Producer. Kafka users can leverage
>> CompletionStage
>> > to
>> > > > write asynchronous non-blocking code. CompletionStage is more
>> powerful
>> > than
>> > > > Future and callback. Also, the code using Future and callback can be
>> > easily
>> > > > re-written by CompletionStage.
>> > > >
>> > > > Cheers,
>> > > > Chia-Ping
>> > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Jason Gustafson]

I think this still makes sense as a separate KIP. For KIP-691, we are just
looking to help define the error contract for the new API.

-Jason

On Sat, Jan 30, 2021 at 8:39 AM Ismael Juma  wrote:

> Are we saying that we won't pursue this KIP in favor of the other one?
>
> Ismael
>
> On Sat, Jan 30, 2021, 4:15 AM Chia-Ping Tsai  wrote:
>
> > hi Jason
> >
> > Thanks for your response. "transmit" is good to me.
> >
> > As we discussed by email, KIP-706 is going to be merged to KIP-691(
> > https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please feel free
> to
> > replace "produce" by "transmit" in KIP-691.
> >
> > Best,
> > Chia-Ping
> >
> > On 2021/01/30 00:48:38, Jason Gustafson  wrote:
> > > Hi Chia-Ping,
> > >
> > > I think this is a great idea. It is a pity that we cannot continue to
> use
> > > the `send` verb, but I don't see how we can. I know we considered
> > > `transmit` as another option which is closer to `send`. That would
> avoid
> > > the redundancy when people choose the common "producer" variable name.
> > >
> > > producer.transmit
> > >
> > > instead of
> > >
> > > producer.produce
> > >
> > > A couple alternatives might be `write` or `append`. I'm happy with
> > > `produce` as well, but curious if others have thoughts.
> > >
> > > -Jason
> > >
> > >
> > > On Wed, Jan 20, 2021 at 9:37 AM Chia-Ping Tsai 
> > wrote:
> > >
> > > > Dear all,
> > > >
> > > > I'd like to start the discussion thread for KIP-706:
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459
> > > >
> > > > KIP-706 is proposing to introduce new API "CompletionStage
> > > > produce(record)" to Producer. Kafka users can leverage
> CompletionStage
> > to
> > > > write asynchronous non-blocking code. CompletionStage is more
> powerful
> > than
> > > > Future and callback. Also, the code using Future and callback can be
> > easily
> > > > re-written by CompletionStage.
> > > >
> > > > Cheers,
> > > > Chia-Ping
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Ismael Juma]

Are we saying that we won't pursue this KIP in favor of the other one?

Ismael

On Sat, Jan 30, 2021, 4:15 AM Chia-Ping Tsai  wrote:

> hi Jason
>
> Thanks for your response. "transmit" is good to me.
>
> As we discussed by email, KIP-706 is going to be merged to KIP-691(
> https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please feel free to
> replace "produce" by "transmit" in KIP-691.
>
> Best,
> Chia-Ping
>
> On 2021/01/30 00:48:38, Jason Gustafson  wrote:
> > Hi Chia-Ping,
> >
> > I think this is a great idea. It is a pity that we cannot continue to use
> > the `send` verb, but I don't see how we can. I know we considered
> > `transmit` as another option which is closer to `send`. That would avoid
> > the redundancy when people choose the common "producer" variable name.
> >
> > producer.transmit
> >
> > instead of
> >
> > producer.produce
> >
> > A couple alternatives might be `write` or `append`. I'm happy with
> > `produce` as well, but curious if others have thoughts.
> >
> > -Jason
> >
> >
> > On Wed, Jan 20, 2021 at 9:37 AM Chia-Ping Tsai 
> wrote:
> >
> > > Dear all,
> > >
> > > I'd like to start the discussion thread for KIP-706:
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459
> > >
> > > KIP-706 is proposing to introduce new API "CompletionStage
> > > produce(record)" to Producer. Kafka users can leverage CompletionStage
> to
> > > write asynchronous non-blocking code. CompletionStage is more powerful
> than
> > > Future and callback. Also, the code using Future and callback can be
> easily
> > > re-written by CompletionStage.
> > >
> > > Cheers,
> > > Chia-Ping
> > >
> > >
> >
>

Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Jorge Esteban Quilcate Otoya]

Hi all,


My 2c:


Pros of "Produce":

- Same as Kafka Protocol "ProduceRequest" and simple enough 
"producer#produce()"; though consumer also differ between "FetchRequest" 
and "consumer#poll".


- As reference, other libraries also use "produce", e.g.: 
librdkafka-derived libraries: 
[confluent-kafka-python](https://github.com/confluentinc/confluent-kafka-python), 
[confluent-kafka-go](https://github.com/confluentinc/confluent-kafka-go)



Pros of "Publish":

- Known from "pub/sub"; though consumer does not "Subscribe" neither.


I personally find that "transmit" adds a bit more complexity than the 
alternatives, as it makes me compare it with "send" and if the semantics 
are the same.



I also like "append", as it represents the operation more precisely: 
append to the batch/buffer to flush async.


If we want to "change" the semantics I'd consider between "append" and 
"write" alternatives.



Jorge.


On 30/01/2021 12:21, Govinda Sakhare wrote:

Hi,

Indeed `transmit(msg)` looks good. Another option could be `publish(msg)`

Best Regards,
Govinda


On Sat, Jan 30, 2021, 17:45 Chia-Ping Tsai  wrote:


hi Jason

Thanks for your response. "transmit" is good to me.

As we discussed by email, KIP-706 is going to be merged to KIP-691(
https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please feel free to
replace "produce" by "transmit" in KIP-691.

Best,
Chia-Ping

On 2021/01/30 00:48:38, Jason Gustafson  wrote:

Hi Chia-Ping,

I think this is a great idea. It is a pity that we cannot continue to use
the `send` verb, but I don't see how we can. I know we considered
`transmit` as another option which is closer to `send`. That would avoid
the redundancy when people choose the common "producer" variable name.

producer.transmit

instead of

producer.produce

A couple alternatives might be `write` or `append`. I'm happy with
`produce` as well, but curious if others have thoughts.

-Jason


On Wed, Jan 20, 2021 at 9:37 AM Chia-Ping Tsai 

wrote:

Dear all,

I'd like to start the discussion thread for KIP-706:


https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459

KIP-706 is proposing to introduce new API "CompletionStage
produce(record)" to Producer. Kafka users can leverage CompletionStage

to

write asynchronous non-blocking code. CompletionStage is more powerful

than

Future and callback. Also, the code using Future and callback can be

easily

re-written by CompletionStage.

Cheers,
Chia-Ping

Jenkins build is back to normal : Kafka » kafka-trunk-jdk8 #426

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #475

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: AbstractCoordinatorTest should close coordinator explicitly 
(#10001)


--
[...truncated 3.53 MB...]

ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren() STARTED

ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren() PASSED

ZooKeeperClientTest > testSetDataExistingZNode() STARTED

ZooKeeperClientTest > testSetDataExistingZNode() PASSED

ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChangeNotTriggered() 
STARTED

ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChangeNotTriggered() 
PASSED

ZooKeeperClientTest > testMixedPipeline() STARTED

ZooKeeperClientTest > testMixedPipeline() PASSED

ZooKeeperClientTest > testGetDataExistingZNode() STARTED

ZooKeeperClientTest > testGetDataExistingZNode() PASSED

ZooKeeperClientTest > testDeleteExistingZNode() STARTED

ZooKeeperClientTest > testDeleteExistingZNode() PASSED

ZooKeeperClientTest > testSessionExpiry() STARTED

ZooKeeperClientTest > testSessionExpiry() PASSED

ZooKeeperClientTest > testSetDataNonExistentZNode() STARTED

ZooKeeperClientTest > testSetDataNonExistentZNode() PASSED

ZooKeeperClientTest > testConnectionViaNettyClient() STARTED

ZooKeeperClientTest > testConnectionViaNettyClient() PASSED

ZooKeeperClientTest > testDeleteNonExistentZNode() STARTED

ZooKeeperClientTest > testDeleteNonExistentZNode() PASSED

ZooKeeperClientTest > testExistsExistingZNode() STARTED

ZooKeeperClientTest > testExistsExistingZNode() PASSED

ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics() STARTED

ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics() PASSED

ZooKeeperClientTest > testZNodeChangeHandlerForDeletion() STARTED

ZooKeeperClientTest > testZNodeChangeHandlerForDeletion() PASSED

ZooKeeperClientTest > testGetAclNonExistentZNode() STARTED

ZooKeeperClientTest > testGetAclNonExistentZNode() PASSED

ZooKeeperClientTest > testStateChangeHandlerForAuthFailure() STARTED

ZooKeeperClientTest > testStateChangeHandlerForAuthFailure() PASSED

ResourceTypeTest > testJavaConversions() STARTED

ResourceTypeTest > testJavaConversions() PASSED

ResourceTypeTest > testFromString() STARTED

ResourceTypeTest > testFromString() PASSED

OperationTest > testJavaConversions() STARTED

OperationTest > testJavaConversions() PASSED

DelegationTokenManagerTest > testPeriodicTokenExpiry() STARTED

DelegationTokenManagerTest > testPeriodicTokenExpiry() PASSED

DelegationTokenManagerTest > testTokenRequestsWithDelegationTokenDisabled() 
STARTED

DelegationTokenManagerTest > testTokenRequestsWithDelegationTokenDisabled() 
PASSED

DelegationTokenManagerTest > testDescribeToken() STARTED

DelegationTokenManagerTest > testDescribeToken() PASSED

DelegationTokenManagerTest > testCreateToken() STARTED

DelegationTokenManagerTest > testCreateToken() PASSED

DelegationTokenManagerTest > testExpireToken() STARTED

DelegationTokenManagerTest > testExpireToken() PASSED

DelegationTokenManagerTest > testRenewToken() STARTED

DelegationTokenManagerTest > testRenewToken() PASSED

DelegationTokenManagerTest > testRemoveTokenHmac() STARTED

DelegationTokenManagerTest > testRemoveTokenHmac() PASSED

AclAuthorizerWithZkSaslTest > testAclUpdateWithSessionExpiration() STARTED

AclAuthorizerWithZkSaslTest > testAclUpdateWithSessionExpiration() PASSED

AclAuthorizerWithZkSaslTest > testAclUpdateWithAuthFailure() STARTED

AclAuthorizerWithZkSaslTest > testAclUpdateWithAuthFailure() PASSED

DefaultMessageFormatterTest > [1] name=print nothing, 
record=ConsumerRecord(topic = someTopic, partition = 9, leaderEpoch = null, 
offset = 9876, CreateTime = 1234, serialized key size = 0, serialized value 
size = 0, headers = RecordHeaders(headers = [RecordHeader(key = h1, value = 
[118, 49]), RecordHeader(key = h2, value = [118, 50])], isReadOnly = false), 
key = [B@768c569d, value = [B@799d8e6), properties=Map(print.value -> false), 
expected= STARTED

DefaultMessageFormatterTest > [1] name=print nothing, 
record=ConsumerRecord(topic = someTopic, partition = 9, leaderEpoch = null, 
offset = 9876, CreateTime = 1234, serialized key size = 0, serialized value 
size = 0, headers = RecordHeaders(headers = [RecordHeader(key = h1, value = 
[118, 49]), RecordHeader(key = h2, value = [118, 50])], isReadOnly = false), 
key = [B@768c569d, value = [B@799d8e6), properties=Map(print.value -> false), 
expected= PASSED

DefaultMessageFormatterTest > [2] name=print key, record=ConsumerRecord(topic = 
someTopic, partition = 9, leaderEpoch = null, offset = 9876, CreateTime = 1234, 
serialized key size = 0, serialized value size = 0, headers = 
RecordHeaders(headers = [RecordHeader(key = h1, value = [118, 49]), 
RecordHeader(key = h2, value = [118, 50])], isReadOnly = false), key = 
[B@833a683, value = [B@6935f62f), properties=Map(print.key -> true, print.value 
-> false), expected=someKey
 STARTED

DefaultMessageF

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #459

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: AbstractCoordinatorTest should close coordinator explicitly 
(#10001)


--
[...truncated 3.57 MB...]

ControllerEventManagerTest > testMetricsCleanedOnClose() PASSED

ControllerChannelManagerTest > 
testStopReplicaRequestWithAlreadyDefinedDeletedPartition() STARTED

ControllerChannelManagerTest > 
testStopReplicaRequestWithAlreadyDefinedDeletedPartition() PASSED

ControllerChannelManagerTest > testUpdateMetadataInterBrokerProtocolVersion() 
STARTED

ControllerChannelManagerTest > testUpdateMetadataInterBrokerProtocolVersion() 
PASSED

ControllerChannelManagerTest > testLeaderAndIsrRequestIsNew() STARTED

ControllerChannelManagerTest > testLeaderAndIsrRequestIsNew() PASSED

ControllerChannelManagerTest > 
testStopReplicaRequestsWhileTopicQueuedForDeletion() STARTED

ControllerChannelManagerTest > 
testStopReplicaRequestsWhileTopicQueuedForDeletion() PASSED

ControllerChannelManagerTest > 
testLeaderAndIsrRequestSentToLiveOrShuttingDownBrokers() STARTED

ControllerChannelManagerTest > 
testLeaderAndIsrRequestSentToLiveOrShuttingDownBrokers() PASSED

ControllerChannelManagerTest > testStopReplicaInterBrokerProtocolVersion() 
STARTED

ControllerChannelManagerTest > testStopReplicaInterBrokerProtocolVersion() 
PASSED

ControllerChannelManagerTest > 
testStopReplicaSentOnlyToLiveAndShuttingDownBrokers() STARTED

ControllerChannelManagerTest > 
testStopReplicaSentOnlyToLiveAndShuttingDownBrokers() PASSED

ControllerChannelManagerTest > testStopReplicaGroupsByBroker() STARTED

ControllerChannelManagerTest > testStopReplicaGroupsByBroker() PASSED

ControllerChannelManagerTest > 
testUpdateMetadataDoesNotIncludePartitionsWithoutLeaderAndIsr() STARTED

ControllerChannelManagerTest > 
testUpdateMetadataDoesNotIncludePartitionsWithoutLeaderAndIsr() PASSED

ControllerChannelManagerTest > testMixedDeleteAndNotDeleteStopReplicaRequests() 
STARTED

ControllerChannelManagerTest > testMixedDeleteAndNotDeleteStopReplicaRequests() 
PASSED

ControllerChannelManagerTest > testLeaderAndIsrInterBrokerProtocolVersion() 
STARTED

ControllerChannelManagerTest > testLeaderAndIsrInterBrokerProtocolVersion() 
PASSED

ControllerChannelManagerTest > testUpdateMetadataRequestSent() STARTED

ControllerChannelManagerTest > testUpdateMetadataRequestSent() PASSED

ControllerChannelManagerTest > testUpdateMetadataRequestDuringTopicDeletion() 
STARTED

ControllerChannelManagerTest > testUpdateMetadataRequestDuringTopicDeletion() 
PASSED

ControllerChannelManagerTest > 
testUpdateMetadataIncludesLiveOrShuttingDownBrokers() STARTED

ControllerChannelManagerTest > 
testUpdateMetadataIncludesLiveOrShuttingDownBrokers() PASSED

ControllerChannelManagerTest > testStopReplicaRequestSent() STARTED

ControllerChannelManagerTest > testStopReplicaRequestSent() PASSED

ControllerChannelManagerTest > 
testStopReplicaRequestsWhileTopicDeletionStarted() STARTED

ControllerChannelManagerTest > 
testStopReplicaRequestsWhileTopicDeletionStarted() PASSED

ControllerChannelManagerTest > testLeaderAndIsrRequestSent() STARTED

ControllerChannelManagerTest > testLeaderAndIsrRequestSent() PASSED

ControllerChannelManagerTest > 
testStopReplicaRequestWithoutDeletePartitionWhileTopicDeletionStarted() STARTED

ControllerChannelManagerTest > 
testStopReplicaRequestWithoutDeletePartitionWhileTopicDeletionStarted() PASSED

FeatureZNodeTest > testDecodeFailOnInvalidFeatures() STARTED

FeatureZNodeTest > testDecodeFailOnInvalidFeatures() PASSED

FeatureZNodeTest > testEncodeDecode() STARTED

FeatureZNodeTest > testEncodeDecode() PASSED

FeatureZNodeTest > testDecodeSuccess() STARTED

FeatureZNodeTest > testDecodeSuccess() PASSED

FeatureZNodeTest > testDecodeFailOnInvalidVersionAndStatus() STARTED

FeatureZNodeTest > testDecodeFailOnInvalidVersionAndStatus() PASSED

ExtendedAclStoreTest > shouldHaveCorrectPaths() STARTED

ExtendedAclStoreTest > shouldHaveCorrectPaths() PASSED

ExtendedAclStoreTest > shouldRoundTripChangeNode() STARTED

ExtendedAclStoreTest > shouldRoundTripChangeNode() PASSED

ExtendedAclStoreTest > shouldThrowFromEncodeOnLiteral() STARTED

ExtendedAclStoreTest > shouldThrowFromEncodeOnLiteral() PASSED

ExtendedAclStoreTest > shouldThrowIfConstructedWithLiteral() STARTED

ExtendedAclStoreTest > shouldThrowIfConstructedWithLiteral() PASSED

ExtendedAclStoreTest > shouldWriteChangesToTheWritePath() STARTED

ExtendedAclStoreTest > shouldWriteChangesToTheWritePath() PASSED

ExtendedAclStoreTest > shouldHaveCorrectPatternType() STARTED

ExtendedAclStoreTest > shouldHaveCorrectPatternType() PASSED

DefaultMessageFormatterTest > [1] name=print nothing, 
record=ConsumerRecord(topic = someTopic, partition = 9, leaderEpoch = null, 
offset = 9876, CreateTime = 1234, serialized key size = 0, serialized value 
size = 0, headers = RecordHeaders(headers = [RecordHeader(key = h1, value = 
[118, 49]), RecordHea

Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Govinda Sakhare]

Hi,

Indeed `transmit(msg)` looks good. Another option could be `publish(msg)`

Best Regards,
Govinda


On Sat, Jan 30, 2021, 17:45 Chia-Ping Tsai  wrote:

> hi Jason
>
> Thanks for your response. "transmit" is good to me.
>
> As we discussed by email, KIP-706 is going to be merged to KIP-691(
> https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please feel free to
> replace "produce" by "transmit" in KIP-691.
>
> Best,
> Chia-Ping
>
> On 2021/01/30 00:48:38, Jason Gustafson  wrote:
> > Hi Chia-Ping,
> >
> > I think this is a great idea. It is a pity that we cannot continue to use
> > the `send` verb, but I don't see how we can. I know we considered
> > `transmit` as another option which is closer to `send`. That would avoid
> > the redundancy when people choose the common "producer" variable name.
> >
> > producer.transmit
> >
> > instead of
> >
> > producer.produce
> >
> > A couple alternatives might be `write` or `append`. I'm happy with
> > `produce` as well, but curious if others have thoughts.
> >
> > -Jason
> >
> >
> > On Wed, Jan 20, 2021 at 9:37 AM Chia-Ping Tsai 
> wrote:
> >
> > > Dear all,
> > >
> > > I'd like to start the discussion thread for KIP-706:
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459
> > >
> > > KIP-706 is proposing to introduce new API "CompletionStage
> > > produce(record)" to Producer. Kafka users can leverage CompletionStage
> to
> > > write asynchronous non-blocking code. CompletionStage is more powerful
> than
> > > Future and callback. Also, the code using Future and callback can be
> easily
> > > re-written by CompletionStage.
> > >
> > > Cheers,
> > > Chia-Ping
> > >
> > >
> >
>

Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

[Chia-Ping Tsai]

hi Jason

Thanks for your response. "transmit" is good to me. 

As we discussed by email, KIP-706 is going to be merged to 
KIP-691(https://cwiki.apache.org/confluence/x/PSfZCQ). Hence, please feel free 
to replace "produce" by "transmit" in KIP-691.

Best,
Chia-Ping

On 2021/01/30 00:48:38, Jason Gustafson  wrote: 
> Hi Chia-Ping,
> 
> I think this is a great idea. It is a pity that we cannot continue to use
> the `send` verb, but I don't see how we can. I know we considered
> `transmit` as another option which is closer to `send`. That would avoid
> the redundancy when people choose the common "producer" variable name.
> 
> producer.transmit
> 
> instead of
> 
> producer.produce
> 
> A couple alternatives might be `write` or `append`. I'm happy with
> `produce` as well, but curious if others have thoughts.
> 
> -Jason
> 
> 
> On Wed, Jan 20, 2021 at 9:37 AM Chia-Ping Tsai  wrote:
> 
> > Dear all,
> >
> > I'd like to start the discussion thread for KIP-706:
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459
> >
> > KIP-706 is proposing to introduce new API "CompletionStage
> > produce(record)" to Producer. Kafka users can leverage CompletionStage to
> > write asynchronous non-blocking code. CompletionStage is more powerful than
> > Future and callback. Also, the code using Future and callback can be easily
> > re-written by CompletionStage.
> >
> > Cheers,
> > Chia-Ping
> >
> >
>