Build failed in Jenkins: kafka-2.1-jdk8 #68

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: improve QueryableStateIntegrationTest (#5987)

--
[...truncated 459.40 KB...]

kafka.server.KafkaConfigTest > testListenerAndAdvertisedListenerNames STARTED

kafka.server.KafkaConfigTest > testListenerAndAdvertisedListenerNames PASSED

kafka.server.KafkaConfigTest > testNonroutableAdvertisedListeners STARTED

kafka.server.KafkaConfigTest > testNonroutableAdvertisedListeners PASSED

kafka.server.KafkaConfigTest > 
testInterBrokerListenerNameAndSecurityProtocolSet STARTED

kafka.server.KafkaConfigTest > 
testInterBrokerListenerNameAndSecurityProtocolSet PASSED

kafka.server.KafkaConfigTest > testFromPropsInvalid STARTED

kafka.server.KafkaConfigTest > testFromPropsInvalid PASSED

kafka.server.KafkaConfigTest > testInvalidCompressionType STARTED

kafka.server.KafkaConfigTest > testInvalidCompressionType PASSED

kafka.server.KafkaConfigTest > testAdvertiseHostNameDefault STARTED

kafka.server.KafkaConfigTest > testAdvertiseHostNameDefault PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeMinutesProvided STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeMinutesProvided PASSED

kafka.server.KafkaConfigTest > testValidCompressionType STARTED

kafka.server.KafkaConfigTest > testValidCompressionType PASSED

kafka.server.KafkaConfigTest > testUncleanElectionInvalid STARTED

kafka.server.KafkaConfigTest > testUncleanElectionInvalid PASSED

kafka.server.KafkaConfigTest > testListenerNamesWithAdvertisedListenerUnset 
STARTED

kafka.server.KafkaConfigTest > testListenerNamesWithAdvertisedListenerUnset 
PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
PASSED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided STARTED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided PASSED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault STARTED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault PASSED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol PASSED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled STARTED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled PASSED

kafka.server.KafkaConfigTest > testInterBrokerVersionMessageFormatCompatibility 
STARTED

kafka.server.KafkaConfigTest > testInterBrokerVersionMessageFormatCompatibility 
PASSED

kafka.server.KafkaConfigTest > testAdvertisePortDefault STARTED

kafka.server.KafkaConfigTest > testAdvertisePortDefault PASSED

kafka.server.KafkaConfigTest > testVersionConfiguration STARTED

kafka.server.KafkaConfigTest > testVersionConfiguration PASSED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol PASSED

kafka.server.ListOffsetsRequestTest > testListOffsetsErrorCodes STARTED

kafka.server.ListOffsetsRequestTest > testListOffsetsErrorCodes PASSED

kafka.server.ListOffsetsRequestTest > testCurrentEpochValidation STARTED

kafka.server.ListOffsetsRequestTest > testCurrentEpochValidation PASSED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testNotController STARTED

kafka.server.CreateTopicsRequestTest > testNotController PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithDownConversionDisabled STARTED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithDownConversionDisabled PASSED

kafka.server.FetchRequestDownConversionConfigTest > testV1FetchFromReplica 
STARTED

kafka.server.FetchRequestDownConversionConfigTest > testV1FetchFromReplica 
PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testLatestFetchWithDownConversionDisabled STARTED

kafka.server.FetchRequestDownConversionConfigTest > 
testLatestFetchWithDownConversionDisabled PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithTopicLevelOverrides STARTED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1

Build failed in Jenkins: kafka-0.10.2-jdk7 #240

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: improve QueryableStateIntegrationTest (#5988)

--
[...truncated 326.13 KB...]

kafka.server.ClientQuotaManagerTest > testExpireThrottleTimeSensor STARTED

kafka.server.ClientQuotaManagerTest > testExpireThrottleTimeSensor PASSED

kafka.server.ClientQuotaManagerTest > testUserClientIdQuotaParsing STARTED

kafka.server.ClientQuotaManagerTest > testUserClientIdQuotaParsing PASSED

kafka.server.ClientQuotaManagerTest > 
testUserClientQuotaParsingIdWithDefaultClientIdQuota STARTED

kafka.server.ClientQuotaManagerTest > 
testUserClientQuotaParsingIdWithDefaultClientIdQuota PASSED

kafka.server.OffsetCommitTest > testUpdateOffsets STARTED

kafka.server.OffsetCommitTest > testUpdateOffsets PASSED

kafka.server.OffsetCommitTest > testLargeMetadataPayload STARTED

kafka.server.OffsetCommitTest > testLargeMetadataPayload PASSED

kafka.server.OffsetCommitTest > testOffsetsDeleteAfterTopicDeletion STARTED

kafka.server.OffsetCommitTest > testOffsetsDeleteAfterTopicDeletion PASSED

kafka.server.OffsetCommitTest > testCommitAndFetchOffsets STARTED

kafka.server.OffsetCommitTest > testCommitAndFetchOffsets PASSED

kafka.server.OffsetCommitTest > testOffsetExpiration STARTED

kafka.server.OffsetCommitTest > testOffsetExpiration PASSED

kafka.server.OffsetCommitTest > testNonExistingTopicOffsetCommit STARTED

kafka.server.OffsetCommitTest > testNonExistingTopicOffsetCommit PASSED

kafka.server.MetadataCacheTest > 
getTopicMetadataWithNonSupportedSecurityProtocol STARTED

kafka.server.MetadataCacheTest > 
getTopicMetadataWithNonSupportedSecurityProtocol PASSED

kafka.server.MetadataCacheTest > getTopicMetadataIsrNotAvailable STARTED

kafka.server.MetadataCacheTest > getTopicMetadataIsrNotAvailable PASSED

kafka.server.MetadataCacheTest > getTopicMetadata STARTED

kafka.server.MetadataCacheTest > getTopicMetadata PASSED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable STARTED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable PASSED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
STARTED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
PASSED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
STARTED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
PASSED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics STARTED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics PASSED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping STARTED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping PASSED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse STARTED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse PASSED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks STARTED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks PASSED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower STARTED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower PASSED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
STARTED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
PASSED

kafka.server.MetadataRequestTest > testReplicaDownResponse STARTED

kafka.server.MetadataRequestTest > testReplicaDownResponse PASSED

kafka.server.MetadataRequestTest > testRack STARTED

kafka.server.MetadataRequestTest > testRack PASSED

kafka.server.MetadataRequestTest > testIsInternal STARTED

kafka.server.MetadataRequestTest > testIsInternal PASSED

kafka.server.MetadataRequestTest > testControllerId STARTED

kafka.server.MetadataRequestTest > testControllerId PASSED

kafka.server.MetadataRequestTest > testAllTopicsRequest STARTED

kafka.server.MetadataRequestTest > testAllTopicsRequest PASSED

kafka.server.MetadataRequestTest > testClusterIdIsValid STARTED

kafka.server.MetadataRequestTest > testClusterIdIsValid PASSED

kafka.server.MetadataRequestTest > testNoTopicsRequest STARTED

kafka.server.MetadataRequestTest > testNoTopicsRequest PASSED

kafka.server.MetadataRequestTest > testClusterIdWithRequestVersion1 STARTED

kafka.server.MetadataRequestTest > testClusterIdWithRequestVersion1 PASSED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testNotController STARTED

Build failed in Jenkins: kafka-trunk-jdk11 #131

[Apache Jenkins Server]

See 


Changes:

[manikumar.reddy] MINOR: hygene cleanup in TransactionManagerTest (#5951)

[manikumar.reddy] KAFKA-4544: Add system tests for delegation token based 
authentication

[junrao] KAFKA-7235: Detect outdated control requests and bounced brokers using

[wangguoz] MINOR: improve QueryableStateIntegrationTest (#5987)

--
[...truncated 2.24 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTi

Build failed in Jenkins: kafka-trunk-jdk8 #3234

[Apache Jenkins Server]

See 


Changes:

[manikumar.reddy] MINOR: hygene cleanup in TransactionManagerTest (#5951)

[manikumar.reddy] KAFKA-4544: Add system tests for delegation token based 
authentication

[junrao] KAFKA-7235: Detect outdated control requests and bounced brokers using

[wangguoz] MINOR: improve QueryableStateIntegrationTest (#5987)

--
[...truncated 2.61 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyVal

Jenkins build is back to normal : kafka-2.0-jdk8 #195

[Apache Jenkins Server]

See 

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

[Jan Filipiak]

On 29.11.2018 15:14, John Roesler wrote:
> Hi all,
>
> Sorry that this discussion petered out... I think the 2.1 release caused an
> extended distraction that pushed it off everyone's radar (which was
> precisely Adam's concern). Personally, I've also had some extend
> distractions of my own that kept (and continue to keep) me preoccupied.
>
> However, calling for a vote did wake me up, so I guess Jan was on the right
> track!
>
> I've gone back and reviewed the whole KIP document and the prior
> discussion, and I'd like to offer a few thoughts:
>
> API Thoughts:
>
> 1. If I read the KIP right, you are proposing a many-to-one join. Could we
> consider naming it manyToOneJoin? Or, if you prefer, flip the design around
> and make it a oneToManyJoin?
>
> The proposed name "joinOnForeignKey" disguises the join type, and it seems
> like it might trick some people into using it for a one-to-one join. This
> would work, of course, but it would be super inefficient compared to a
> simple rekey-and-join.
>
> 2. I might have missed it, but I don't think it's specified whether it's an
> inner, outer, or left join. I'm guessing an outer join, as (neglecting IQ),
> the rest can be achieved by filtering or by handling it in the ValueJoiner.
>
> 3. The arg list to joinOnForeignKey doesn't look quite right.
> 3a. Regarding Serialized: There are a few different paradigms in play in
> the Streams API, so it's confusing, but instead of three Serialized args, I
> think it would be better to have one that allows (optionally) setting the 4
> incoming serdes. The result serde is defined by the Materialized. The
> incoming serdes can be optional because they might already be available on
> the source KTables, or the default serdes from the config might be
> applicable.
>
> 3b. Is the StreamPartitioner necessary? The other joins don't allow setting
> one, and it seems like it might actually be harmful, since the rekey
> operation needs to produce results that are co-partitioned with the "other"
> KTable.
>
> 4. I'm fine with the "reserved word" header, but I didn't actually follow
> what Matthias meant about namespacing requiring "deserializing" the record
> header. The headers are already Strings, so I don't think that
> deserialization is required. If we applied the namespace at source nodes
> and stripped it at sink nodes, this would be practically no overhead. The
> advantage of the namespace idea is that no public API change wrt headers
> needs to happen, and no restrictions need to be placed on users' headers.
>
> (Although I'm wondering if we can get away without the header at all...
> stay tuned)
>
> 5. I also didn't follow the discussion about the HWM table growing without
> bound. As I read it, the HWM table is effectively implementing OCC to
> resolve the problem you noted with disordering when the rekey is
> reversed... particularly notable when the FK changes. As such, it only
> needs to track the most recent "version" (the offset in the source
> partition) of each key. Therefore, it should have the same number of keys
> as the source table at all times.
>
> I see that you are aware of KIP-258, which I think might be relevant in a
> couple of ways. One: it's just about storing the timestamp in the state
> store, but the ultimate idea is to effectively use the timestamp as an OCC
> "version" to drop disordered updates. You wouldn't want to use the
> timestamp for this operation, but if you were to use a similar mechanism to
> store the source offset in the store alongside the re-keyed values, then
> you could avoid a separate table.
>
> 6. You and Jan have been thinking about this for a long time, so I've
> probably missed something here, but I'm wondering if we can avoid the HWM
> tracking at all and resolve out-of-order during a final join instead...
>
> Let's say we're joining a left table (Integer K: Letter FK, (other data))
> to a right table (Letter K: (some data)).
>
> Left table:
> 1: (A, xyz)
> 2: (B, asd)
>
> Right table:
> A: EntityA
> B: EntityB
>
> We could do a rekey as you proposed with a combined key, but not
> propagating the value at all..
> Rekey table:
> A-1: (dummy value)
> B-2: (dummy value)
>
> Which we then join with the right table to produce:
> A-1: EntityA
> B-2: EntityB
>
> Which gets rekeyed back:
> 1: A, EntityA
> 2: B, EntityB
>
> And finally we do the actual join:
> Result table:
> 1: ((A, xyz), EntityA)
> 2: ((B, asd), EntityB)
>
> The thing is that in that last join, we have the opportunity to compare the
> current FK in the left table with the incoming PK of the right table. If
> they don't match, we just drop the event, since it must be outdated.
>

> In your KIP, you gave an example in which (1: A, xyz) gets updated to (1:
> B, xyz), ultimately yielding a conundrum about whether the final state
> should be (1: null) or (1: joined-on-B). With the algorithm above, you
> would be considering (1: (B, xyz), (A, null)) vs (1: (B, xyz), (B,
> EntityB)). It seems like this does give you e

[jira] [Created] (KAFKA-7695) Cannot override StreamsPartitionAssignor in configuration

[Dmitry Buykin (JIRA)]

Dmitry Buykin created KAFKA-7695:


 Summary: Cannot override StreamsPartitionAssignor in configuration 
 Key: KAFKA-7695
 URL: https://issues.apache.org/jira/browse/KAFKA-7695
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.0.1, 2.0.0
Reporter: Dmitry Buykin


Cannot override StreamsPartitionAssignor by changing property 
partition.assignment.strategy in KStreams 2.0.1 because the streams are 
crashing inside KafkaClientSupplier.getGlobalConsumer. This GlobalConsumer 
works only with RangeAssignor which configured by default.

Could be reproduced by setting up
`props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, 
StreamsPartitionAssignor.class.getName());`
For me it looks like a bug.

Opened a discussion here 

https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1543395977453700

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

[Mickael Maison]

Hi Tom,

This is a very interesting KIP. If you are not going to continue
working on it, would it be ok for us to grab it and complete it?
Thanks
On Thu, Jun 14, 2018 at 7:06 PM Anna Povzner  wrote:
>
> Hi Tom,
>
> Just wanted to check what you think about the comments I made in my last
> message. I think this KIP is a big improvement to our current policy
> interfaces, and really hope we can get this KIP in.
>
> Thanks,
> Anna
>
> On Thu, May 31, 2018 at 3:29 PM, Anna Povzner  wrote:
>
> > Hi Tom,
> >
> >
> > Thanks for the KIP. I am aware that the voting thread was started, but
> > wanted to discuss couple of concerns here first.
> >
> >
> > I think the coupling of RequestedTopicState#generatedReplicaAssignment()
> > and TopicState#replicasAssignments() does not work well in case where the
> > request deals only with a subset of partitions (e.g., add partitions) or no
> > assignment at all (alter topic config). In particular:
> >
> > 1) Alter topic config use case: There is no replica assignment in the
> > request, and generatedReplicaAssignment()  returning either true or false
> > is both misleading. The user can interpret this as assignment being
> > generated or provided by the user originally (e.g., on topic create), while
> > I don’t think we track such thing.
> >
> > 2) On add partitions, we may have manual assignment for new partitions.
> > What I understood from the KIP,  generatedReplicaAssignment() will return
> > true or false based on whether new partitions were manually assigned or
> > not, while TopicState#replicasAssignments() will return replica
> > assignments for all partitions. I think it is confusing in a way that
> > assignment of old partitions could be auto-generated but new partitions are
> > manually assigned.
> >
> > 3) Generalizing #2, suppose in a future, a user can re-assign replicas for
> > a set of partitions.
> >
> >
> > One way to address this with minimal changes to proposed API is to rename
> > RequestedTopicState#generatedReplicaAssignment() to 
> > RequestedTopicState#manualReplicaAssignment()
> > and change the API behavior and description to : “True if the client
> > explicitly provided replica assignments in this request, which means that
> > some or all assignments returned by TopicState#replicasAssignments() are
> > explicitly requested by the user”. The user then will have to diff
> > TopicState#replicasAssignments() from clusterState and TopicState#
> > replicasAssignments()  from RequestedTopicState, and assume that
> > assignments that are different are manually assigned (if
> > RequestedTopicState#manualReplicaAssignment()  returns true). We will
> > need to clearly document this and it still seems awkward.
> >
> >
> > I think a cleaner way is to make RequestedTopicState to provide replica
> > assignments only for partitions that were manually assigned replicas in the
> > request that is being validated. Similarly, for alter topic validation, it
> > would be nice to make it more clear for the user what has been changed. I
> > remember that you already raised that point earlier by comparing current
> > proposed API with having separate methods for each specific command.
> > However, I agree that it will make it harder to change the interface in the
> > future.
> >
> >
> > Could we explore the option of pushing methods that are currently in
> > TopicState to CreateTopicRequest and AlterTopicRequest? TopicState will
> > still be used for requesting current topic state via ClusterState.
> >
> > Something like:
> >
> > interface CreateTopicRequest extends AbstractRequestMetadata {
> >
> >   // requested number of partitions or if manual assignment is given,
> > number of partitions in the assignment
> >
> >   int numPartitions();
> >
> >   // requested replication factor, or if manual assignment is given,
> > number of replicas in assignment for partition 0
> >
> >   short replicationFactor();
> >
> >  // replica assignment requested by the client, or null if assignment is
> > auto-generated
> >
> >  map> manualReplicaAssignment();
> >
> >  map configs();
> >
> > }
> >
> >
> > interface AlterTopicRequest extends AbstractRequestMetadata {
> >
> >   // updated topic configs, or null if not changed
> >
> >   map updatedConfigs();
> >
> >   // proposed replica assignment in this request, or null. For adding new
> > partitions request, this is proposed replica assignment for new partitions.
> > For replica re-assignment case, this is proposed new assignment.
> >
> >   map> proposedReplicaAssignment();
> >
> >   // new number of partitions (due to increase/decrease), or null if
> > number of partitions not changed
> >
> >   Integer updatedNumPartitions()
> >
> > }
> >
> >
> > I did not spend much time on my AlterTopicRequest interface proposal, but
> > the idea is basically to return only the parts which were changed. The
> > advantage of this approach over having separate methods for each specific
> > alter topic request is that it is more flexible for future mixing 

[jira] [Created] (KAFKA-7696) kafka-delegation-tokens.sh using a config file that contains security.protocol=SASL_PLAINTEXT throws OutOfMemoryError if it tries to connect to an SSL-enabled secured bro

[Attila Sasvari (JIRA)]

Attila Sasvari created KAFKA-7696:
-

 Summary: kafka-delegation-tokens.sh using a config file that 
contains security.protocol=SASL_PLAINTEXT throws OutOfMemoryError if it tries 
to connect to an SSL-enabled secured broker
 Key: KAFKA-7696
 URL: https://issues.apache.org/jira/browse/KAFKA-7696
 Project: Kafka
  Issue Type: Bug
  Components: tools
Affects Versions: 2.0.0, 1.1.0
Reporter: Attila Sasvari


When the command-config file of kafka-delegation-tokens contain 
security.protocol=SASL_PLAINTEXT instead of SASL_SSL (i.e. due to a user 
error), the process throws a java.lang.OutOfMemoryError upon connection attempt 
to a secured (i.e. Kerberized, SSL-enabled) Kafka broker.

{code}
[2018-12-03 11:27:13,221] ERROR Uncaught exception in thread 
'kafka-admin-client-thread | adminclient-1': 
(org.apache.kafka.common.utils.KafkaThread)
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at 
org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
at 
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:112)
at 
org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.receiveResponseOrToken(SaslClientAuthenticator.java:407)
at 
org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.receiveKafkaResponse(SaslClientAuthenticator.java:497)
at 
org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:207)
at 
org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)
at 
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:533)
at org.apache.kafka.common.network.Selector.poll(Selector.java:468)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
at 
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1125)
at java.lang.Thread.run(Thread.java:745)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-392: Allow consumers to fetch from the closest replica

[Stanislav Kozlovski]

Hey Jason,

This is certainly a very exciting KIP.
I assume that no changes will be made to the offset commits and they will
continue to be sent to the group coordinator?

I also wanted to address metrics - have we considered any changes there? I
imagine that it would be valuable for users to be able to differentiate
between which consumers' partitions are fetched from replicas and which
aren't. I guess that would need to be addressed both in the server's
fetcher lag metrics and in the consumers.

Thanks,
Stanislav

On Wed, Nov 28, 2018 at 10:08 PM Jun Rao  wrote:

> Hi, Jason,
>
> Thanks for the KIP. Looks good overall. A few minor comments below.
>
> 1. The section on handling FETCH_OFFSET_TOO_LARGE error says "Use the
> OffsetForLeaderEpoch API to verify the current position with the leader".
> The OffsetForLeaderEpoch request returns log end offset if the request
> leader epoch is the latest. So, we won't know the true high watermark from
> that request. It seems that the consumer still needs to send ListOffset
> request to the leader to obtain high watermark?
>
> 2. If a non in-sync replica receives a fetch request from a consumer,
> should it return a new type of error like ReplicaNotInSync?
>
> 3. Could ReplicaSelector be closable?
>
> 4. Currently, the ISR propagation from the leader to the controller can be
> delayed up to 60 secs through ReplicaManager.IsrChangePropagationInterval.
> In that window, the consumer could still be consuming from a non in-sync
> replica. The relatively large delay is mostly for reducing the ZK writes
> and the watcher overhead. Not sure what's the best way to address this. We
> could potentially make this configurable.
>
> 5. It may be worth mentioning that, to take advantage of affinity, one may
> also want to have a customized PartitionAssignor to have an affinity aware
> assignment in addition to a customized ReplicaSelector.
>
> Thanks,
>
> Jun
>
> On Wed, Nov 21, 2018 at 12:54 PM Jason Gustafson 
> wrote:
>
> > Hi All,
> >
> > I've posted a KIP to add the often-requested support for fetching from
> > followers:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> > .
> > Please take a look and let me know what you think.
> >
> > Thanks,
> > Jason
> >
>


-- 
Best,
Stanislav

Re: [DISCUSS] KIP-392: Allow consumers to fetch from the closest replica

[Mickael Maison]

Hi Jason,

Very cool KIP!
A couple of questions:
- I'm guessing the selector will be invoke after each rebalance so
every time the consumer is assigned a partition it will be able to
select it. Is that true?

- From the selector API, I'm not sure how the consumer will be able to
address some of the choices mentioned in "Finding the preferred
follower". Especially the available bandwidth and the load balancing.
By only having the list of Nodes, a consumer can pick the nereast
replica (assuming the rack field means anything to users) or balance
its own bandwidth but that might not necessarily mean improved
performance or a balanced load on the brokers.

Thanks
On Mon, Dec 3, 2018 at 11:35 AM Stanislav Kozlovski
 wrote:
>
> Hey Jason,
>
> This is certainly a very exciting KIP.
> I assume that no changes will be made to the offset commits and they will
> continue to be sent to the group coordinator?
>
> I also wanted to address metrics - have we considered any changes there? I
> imagine that it would be valuable for users to be able to differentiate
> between which consumers' partitions are fetched from replicas and which
> aren't. I guess that would need to be addressed both in the server's
> fetcher lag metrics and in the consumers.
>
> Thanks,
> Stanislav
>
> On Wed, Nov 28, 2018 at 10:08 PM Jun Rao  wrote:
>
> > Hi, Jason,
> >
> > Thanks for the KIP. Looks good overall. A few minor comments below.
> >
> > 1. The section on handling FETCH_OFFSET_TOO_LARGE error says "Use the
> > OffsetForLeaderEpoch API to verify the current position with the leader".
> > The OffsetForLeaderEpoch request returns log end offset if the request
> > leader epoch is the latest. So, we won't know the true high watermark from
> > that request. It seems that the consumer still needs to send ListOffset
> > request to the leader to obtain high watermark?
> >
> > 2. If a non in-sync replica receives a fetch request from a consumer,
> > should it return a new type of error like ReplicaNotInSync?
> >
> > 3. Could ReplicaSelector be closable?
> >
> > 4. Currently, the ISR propagation from the leader to the controller can be
> > delayed up to 60 secs through ReplicaManager.IsrChangePropagationInterval.
> > In that window, the consumer could still be consuming from a non in-sync
> > replica. The relatively large delay is mostly for reducing the ZK writes
> > and the watcher overhead. Not sure what's the best way to address this. We
> > could potentially make this configurable.
> >
> > 5. It may be worth mentioning that, to take advantage of affinity, one may
> > also want to have a customized PartitionAssignor to have an affinity aware
> > assignment in addition to a customized ReplicaSelector.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Nov 21, 2018 at 12:54 PM Jason Gustafson 
> > wrote:
> >
> > > Hi All,
> > >
> > > I've posted a KIP to add the often-requested support for fetching from
> > > followers:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> > > .
> > > Please take a look and let me know what you think.
> > >
> > > Thanks,
> > > Jason
> > >
> >
>
>
> --
> Best,
> Stanislav

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

[Stanislav Kozlovski]

Hi Jason,

> 2. Do you think we should make this a dynamic config?
I'm not sure. Looking at the config from the perspective of a prescriptive
config, we may get away with not updating it dynamically.
But in my opinion, it always makes sense to have a config be dynamically
configurable. As long as we limit it to being a cluster-wide config, we
should be fine.

> 1. I think it would be helpful to clarify the details on how the
coordinator will shrink the group. It will need to choose which members to
remove. Are we going to give current members an opportunity to commit
offsets before kicking them from the group?

This turns out to be somewhat tricky. I think that we may not be able to
guarantee that consumers don't process a message twice.
My initial approach was to do as much as we could to let consumers commit
offsets.

I was thinking that we mark a group to be shrunk, we could keep a map of
consumer_id->boolean indicating whether they have committed offsets. I then
thought we could delay the rebalance until every consumer commits (or some
time passes).
In the meantime, we would block all incoming fetch calls (by either
returning empty records or a retriable error) and we would continue to
accept offset commits (even twice for a single consumer)

I see two problems with this approach:
* We have async offset commits, which implies that we can receive fetch
requests before the offset commit req has been handled. i.e consmer sends
fetchReq A, offsetCommit B, fetchReq C - we may receive A,C,B in the
broker. Meaning we could have saved the offsets for B but rebalance before
the offsetCommit for the offsets processed in C come in.
* KIP-392 Allow consumers to fetch from closest replica

would
make it significantly harder to block poll() calls on consumers whose
groups are being shrunk. Even if we implemented a solution, the same race
condition noted above seems to apply and probably others


Given those constraints, I think that we can simply mark the group as
`PreparingRebalance` with a rebalanceTimeout of the server setting `
group.max.session.timeout.ms`. That's a bit long by default (5 minutes) but
I can't seem to come up with a better alternative

I'm interested in hearing your thoughts.

Thanks,
Stanislav

On Fri, Nov 30, 2018 at 8:38 AM Jason Gustafson  wrote:

> Hey Stanislav,
>
> What do you think about the use case I mentioned in my previous reply about
> > a more resilient self-service Kafka? I believe the benefit there is
> bigger.
>
>
> I see this config as analogous to the open file limit. Probably this limit
> was intended to be prescriptive at some point about what was deemed a
> reasonable number of open files for an application. But mostly people treat
> it as an annoyance which they have to work around. If it happens to be hit,
> usually you just increase it because it is not tied to an actual resource
> constraint. However, occasionally hitting the limit does indicate an
> application bug such as a leak, so I wouldn't say it is useless. Similarly,
> the issue in KAFKA-7610 was a consumer leak and having this limit would
> have allowed the problem to be detected before it impacted the cluster. To
> me, that's the main benefit. It's possible that it could be used
> prescriptively to prevent poor usage of groups, but like the open file
> limit, I suspect administrators will just set it large enough that users
> are unlikely to complain.
>
> Anyway, just a couple additional questions:
>
> 1. I think it would be helpful to clarify the details on how the
> coordinator will shrink the group. It will need to choose which members to
> remove. Are we going to give current members an opportunity to commit
> offsets before kicking them from the group?
>
> 2. Do you think we should make this a dynamic config?
>
> Thanks,
> Jason
>
>
>
>
> On Wed, Nov 28, 2018 at 2:42 AM Stanislav Kozlovski <
> stanis...@confluent.io>
> wrote:
>
> > Hi Jason,
> >
> > You raise some very valid points.
> >
> > > The benefit of this KIP is probably limited to preventing "runaway"
> > consumer groups due to leaks or some other application bug
> > What do you think about the use case I mentioned in my previous reply
> about
> > a more resilient self-service Kafka? I believe the benefit there is
> bigger
> >
> > * Default value
> > You're right, we probably do need to be conservative. Big consumer groups
> > are considered an anti-pattern and my goal was to also hint at this
> through
> > the config's default. Regardless, it is better to not have the potential
> to
> > break applications with an upgrade.
> > Choosing between the default of something big like 5000 or an opt-in
> > option, I think we should go with the *disabled default option*  (-1).
> > The only benefit we would get from a big default of 5000 is default
> > protection against buggy/malicious applications that hit the KAFKA-7610
> > issue.
> > While this KI

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

[Stanislav Kozlovski]

Everything sounds good to me.

On Sun, Dec 2, 2018 at 1:24 PM Boyang Chen  wrote:

> In fact, it's probably better to move KIP-394<
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-394%3A+Require+member.id+for+initial+join+group+request>
> to the vote stage first, so that it's easier to finalize the timeline and
> smooth the rollout plan for KIP-345. Jason and Stanislav, since you two
> involve most in this KIP, could you let me know if there is still any
> unclarity we want to resolve before moving to vote?
>
> Best,
> Boyang
> 
> From: Boyang Chen 
> Sent: Saturday, December 1, 2018 10:53 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-394: Require member.id for initial join group
> request
>
> Thanks Jason for the reply! Since the overall motivation and design is
> pretty clear, I will go ahead to start implementation and we could discuss
> the underlying details in the PR.
>
> Best,
> Boyang
> 
> From: Matthias J. Sax 
> Sent: Saturday, December 1, 2018 3:12 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-394: Require member.id for initial join group
> request
>
> SGTM.
>
> On 11/30/18 10:17 AM, Jason Gustafson wrote:
> > Using the session expiration logic we already have seems like the
> simplest
> > option (this is probably a one or two line change). The rejoin should be
> > quick anyway, so I don't think it's worth optimizing for unjoined new
> > members. Just my two cents. This is more of an implementation detail, so
> > need not necessarily be resolved here.
> >
> > -Jason
> >
> > On Fri, Nov 30, 2018 at 12:56 AM Boyang Chen 
> wrote:
> >
> >> Thanks Matthias for the question. I'm thinking of having a separate hash
> >> set called `registeredMemberIds` which
> >> will be cleared out every time a group finishes one round of rebalance.
> >> Since storing one id is pretty trivial, using
> >> purgatory to track the id removal is a bit wasteful in my opinion.
> >> 
> >> From: Matthias J. Sax 
> >> Sent: Friday, November 30, 2018 10:26 AM
> >> To: dev@kafka.apache.org
> >> Subject: Re: [DISCUSS] KIP-394: Require member.id for initial join
> group
> >> request
> >>
> >> Thanks! Makes sense.
> >>
> >> I missed that fact, that the `member.id` is added on the second
> >> joinGroup request that contains the `member.id`.
> >>
> >> However, it seems there is another race condition for this design:
> >>
> >> If two consumers join at the same time, it it possible that the broker
> >> assigns the same `member.id` to both (because none of them have joined
> >> the group yet--ie, second joinGroup request not sent yet--, the
> >> `member.id` is not store broker side yes and broker cannot check for
> >> duplicates when creating a new `member.id`.
> >>
> >> The probability might be fairly low thought. However, what Stanislav
> >> proposed, to add the `member.id` directly, and remove it after
> >> `session.timeout.ms` sound like a save option that avoids this issue.
> >>
> >> Thoughts?
> >>
> >>
> >> -Matthias
> >>
> >> On 11/28/18 8:15 PM, Boyang Chen wrote:
> >>> Thanks Matthias for the question, and Stanislav for the explanation!
> >>>
> >>> For the scenario described, we will never let a member join the
> >> GroupMetadata map
> >>> if it uses UNKNOWN_MEMBER_ID. So the workflow will be like this:
> >>>
> >>>   1.  Group is empty. Consumer c1 started. Join with UNKNOWN_MEMBER_ID;
> >>>   2.  Broker rejects while allocating a member.id to c1 in response
> (c1
> >> protocol version is current);
> >>>   3.  c1 handles the error and rejoins with assigned member.id;
> >>>   4.  Broker stores c1 in its group metadata;
> >>>   5.  Consumer c2 started. Join with UNKNOWN_MEMBER_ID;
> >>>   6.  Broker rejects while allocating a member.id to c2 in response
> (c2
> >> protocol version is current);
> >>>   7.  c2 fails to get the response/crashes in the middle;
> >>>   8.  After certain time, c2 restarts a join request with
> >> UNKNOWN_MEMBER_ID;
> >>>
> >>> As you could see, c2 will repeat step 6~8 until successfully send back
> a
> >> join group request with allocated id.
> >>> By then broker will include c2 within the broker metadata map.
> >>>
> >>> Does this sound clear to you?
> >>>
> >>> Best,
> >>> Boyang
> >>> 
> >>> From: Stanislav Kozlovski 
> >>> Sent: Wednesday, November 28, 2018 7:39 PM
> >>> To: dev@kafka.apache.org
> >>> Subject: Re: [DISCUSS] KIP-394: Require member.id for initial join
> >> group request
> >>>
> >>> Hey Matthias,
> >>>
> >>> I think the notion is to have the `session.timeout.ms` to start
> ticking
> >>> when the broker responds with the member.id. Then, the broker would
> >>> properly expire consumers and not hold too many stale ones.
> >>> This isn't mentioned in the KIP though so it is worth to wait for
> Boyang
> >> to
> >>> confirm
> >>>
> >>> On Wed, Nov 28, 2018 at 3:10 AM Matthias J. Sax  >
> >>> wrote:
> >>>
>  Thanks for the KIP

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Hey community friends,

after another month of polishing, 
KIP-345
 design is ready for vote. Feel free to add your comment on the discussion
thread or here.

Thanks for your time!

Boyang

From: Boyang Chen 
Sent: Friday, November 9, 2018 6:35 AM
To: dev@kafka.apache.org
Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

Hey all,


thanks so much for all the inputs on KIP-345 so far. The original proposal has 
enhanced a lot with your help. To make sure the implementation go smoothly 
without back and forth, I would like to start a vote on the final design 
agreement now:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-

345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances

KIP-345: Introduce static membership protocol to reduce 
...
cwiki.apache.org
For stateful applications, one of the biggest performance bottleneck is the 
state shuffling. In Kafka consumer, there is a concept called "rebalance" which 
means that for given M partitions and N consumers in one consumer group, Kafka 
will try to balance the load between consumers and ideally have ...


Let me know if you have any questions.


Best,

Boyang

Re: [DISCUSS] KIP-392: Allow consumers to fetch from the closest replica

[Eno Thereska]

Hi Jason,

This is an interesting KIP. This will have massive implications for
consistency and serialization, since currently the leader for a partition
serializes requests. A few questions for now:

- before we deal with the complexity, it'd be great to see a crisp example
in the motivation as to when this will have the most benefit for a
customer. In particular, although the customer might have a multi-DC
deployment, the DCs could still be close by in a region, so what is the
expected best-case scenario for a performance gain? E.g., if all DCs are on
the east-cost, say. Right now it's not clear to me.
- perhaps performance is not the right metric. Is the metric you are
optimizing for latency, throughput or cross-DC cost? (I believe it is
cross-DC cost from the KIP). Just wanted to double-check since I'm not sure
latency would improve. Throughput could really improve from parallelism
(especially in cases when there is mostly consuming going on). So it could
be throughput as well.
- the proposal would probably lead to choosing a more complex consistency.
I tend to like the description Doug Terry has in his paper "Replicated Data
Consistency Explained Through Baseball"
https://www.microsoft.com/en-us/research/wp-content/uploads/2011/10/ConsistencyAndBaseballReport.pdf.
To start with, could we get in scenarios where a client that has both a
producer and a consumer (e.g., Kafka streams) produces a record, then
attempts to consume it back and the consume() comes back with "record does
not exist"? That's fine, but could complicate application handling of such
scenarios.

Thanks,
Eno

On Mon, Dec 3, 2018 at 12:24 PM Mickael Maison 
wrote:

> Hi Jason,
>
> Very cool KIP!
> A couple of questions:
> - I'm guessing the selector will be invoke after each rebalance so
> every time the consumer is assigned a partition it will be able to
> select it. Is that true?
>
> - From the selector API, I'm not sure how the consumer will be able to
> address some of the choices mentioned in "Finding the preferred
> follower". Especially the available bandwidth and the load balancing.
> By only having the list of Nodes, a consumer can pick the nereast
> replica (assuming the rack field means anything to users) or balance
> its own bandwidth but that might not necessarily mean improved
> performance or a balanced load on the brokers.
>
> Thanks
> On Mon, Dec 3, 2018 at 11:35 AM Stanislav Kozlovski
>  wrote:
> >
> > Hey Jason,
> >
> > This is certainly a very exciting KIP.
> > I assume that no changes will be made to the offset commits and they will
> > continue to be sent to the group coordinator?
> >
> > I also wanted to address metrics - have we considered any changes there?
> I
> > imagine that it would be valuable for users to be able to differentiate
> > between which consumers' partitions are fetched from replicas and which
> > aren't. I guess that would need to be addressed both in the server's
> > fetcher lag metrics and in the consumers.
> >
> > Thanks,
> > Stanislav
> >
> > On Wed, Nov 28, 2018 at 10:08 PM Jun Rao  wrote:
> >
> > > Hi, Jason,
> > >
> > > Thanks for the KIP. Looks good overall. A few minor comments below.
> > >
> > > 1. The section on handling FETCH_OFFSET_TOO_LARGE error says "Use the
> > > OffsetForLeaderEpoch API to verify the current position with the
> leader".
> > > The OffsetForLeaderEpoch request returns log end offset if the request
> > > leader epoch is the latest. So, we won't know the true high watermark
> from
> > > that request. It seems that the consumer still needs to send ListOffset
> > > request to the leader to obtain high watermark?
> > >
> > > 2. If a non in-sync replica receives a fetch request from a consumer,
> > > should it return a new type of error like ReplicaNotInSync?
> > >
> > > 3. Could ReplicaSelector be closable?
> > >
> > > 4. Currently, the ISR propagation from the leader to the controller
> can be
> > > delayed up to 60 secs through
> ReplicaManager.IsrChangePropagationInterval.
> > > In that window, the consumer could still be consuming from a non
> in-sync
> > > replica. The relatively large delay is mostly for reducing the ZK
> writes
> > > and the watcher overhead. Not sure what's the best way to address
> this. We
> > > could potentially make this configurable.
> > >
> > > 5. It may be worth mentioning that, to take advantage of affinity, one
> may
> > > also want to have a customized PartitionAssignor to have an affinity
> aware
> > > assignment in addition to a customized ReplicaSelector.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Nov 21, 2018 at 12:54 PM Jason Gustafson 
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I've posted a KIP to add the often-requested support for fetching
> from
> > > > followers:
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> > > > .
> > > > Please take a look and let me know what you think.
> > > >
> > > > Thanks,
> > > > Jason
> > >

[jira] [Created] (KAFKA-7697) Possible deadlock in kafka.cluster.Partition

[Gian Merlino (JIRA)]

Gian Merlino created KAFKA-7697:
---

 Summary: Possible deadlock in kafka.cluster.Partition
 Key: KAFKA-7697
 URL: https://issues.apache.org/jira/browse/KAFKA-7697
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Gian Merlino
 Attachments: threaddump.txt

After upgrading a fairly busy broker from 0.10.2.0 to 2.1.0, it locked up 
within a few minutes (by "locked up" I mean that all request handler threads 
were busy, and other brokers reported that they couldn't communicate with it). 
I restarted it a few times and it did the same thing each time. After 
downgrading to 0.10.2.0, the broker was stable. I attached a thread dump from 
the last attempt on 2.1.0 that shows lots of kafka-request-handler- threads 
trying to acquire the leaderIsrUpdateLock lock in kafka.cluster.Partition.

It jumps out that there are two threads that already have some read lock (can't 
tell which one) and are trying to acquire a second one (on two different read 
locks: 0x000708184b88 and 0x00070821f188): kafka-request-handler-1 and 
kafka-request-handler-4. Both are handling a produce request, and in the 
process of doing so, are calling Partition.fetchOffsetSnapshot while trying to 
complete a DelayedFetch. At the same time, both of those locks have writers 
from other threads waiting on them (kafka-request-handler-2 and 
kafka-scheduler-6). Neither of those locks appear to have writers that hold 
them (if only because no threads in the dump are deep enough in inWriteLock to 
indicate that).

ReentrantReadWriteLock in nonfair mode prioritizes waiting writers over 
readers. Is it possible that kafka-request-handler-1 and 
kafka-request-handler-4 are each trying to read-lock the partition that is 
currently locked by the other one, and they're both parked waiting for 
kafka-request-handler-2 and kafka-scheduler-6 to get write locks, which they 
never will, because the former two threads own read locks and aren't giving 
them up?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Guozhang Wang]

Hello Boyang,

I've browsed through the new wiki and there are still a couple of minor
things to notice:

1. RemoveMemberFromGroupOptions seems not defined anywhere.

2. LeaveGroupRequest added a list of group instance id, but still keep the
member id as a singleton; is that intentional? I think to make the protocol
consistent both member id and instance ids could be plural.

3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
can defer adding this while just add the corresponding calls of the
LeaveGroupRequest inside Streams until we have used it in production and
hence have a better understanding on how flexible or extensible if we want
to add any cmd tools. The rationale is that if we do not necessarily need
it now, we can always add it later with a more think-through API design,
but if we add the tool in a rush, we may need to extend or modify it soon
after we realize its limits in operations.

Otherwise, I'm +1 on the proposal.

Guozhang


On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen  wrote:

> Hey community friends,
>
> after another month of polishing, KIP-345<
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
> design is ready for vote. Feel free to add your comment on the discussion
> thread or here.
>
> Thanks for your time!
>
> Boyang
> 
> From: Boyang Chen 
> Sent: Friday, November 9, 2018 6:35 AM
> To: dev@kafka.apache.org
> Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce
> consumer rebalances
>
> Hey all,
>
>
> thanks so much for all the inputs on KIP-345 so far. The original proposal
> has enhanced a lot with your help. To make sure the implementation go
> smoothly without back and forth, I would like to start a vote on the final
> design agreement now:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-<
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
>
> 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances<
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
>
> KIP-345: Introduce static membership protocol to reduce ...<
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
> cwiki.apache.org
> For stateful applications, one of the biggest performance bottleneck is
> the state shuffling. In Kafka consumer, there is a concept called
> "rebalance" which means that for given M partitions and N consumers in one
> consumer group, Kafka will try to balance the load between consumers and
> ideally have ...
>
>
> Let me know if you have any questions.
>
>
> Best,
>
> Boyang
>
>

-- 
-- Guozhang

[jira] [Created] (KAFKA-7698) Kafka Broker fail to start: ProducerFencedException thrown from producerstatemanager.scala!checkProducerEpoch

[Ming Liu (JIRA)]

Ming Liu created KAFKA-7698:
---

 Summary: Kafka Broker fail to start: ProducerFencedException 
thrown from producerstatemanager.scala!checkProducerEpoch 
 Key: KAFKA-7698
 URL: https://issues.apache.org/jira/browse/KAFKA-7698
 Project: Kafka
  Issue Type: Bug
Reporter: Ming Liu


During our operation of Kafka, we frequently saw this failure: 

   There was an error in one of the threads during logs loading: 
org.apache.kafka.common.errors.ProducerFencedException:

{code:java}

[06:57:09,697] INFO [ProducerStateManager partition=interaction_events-127] 
Loading producer state from snapshot file 
'/data/disk5/kafka/interaction_events-127/092130764817.snapshot' 
(kafka.log.ProducerStateManager)
[06:57:09,698] INFO [Log partition=interaction_events-127, 
dir=/data/disk5/kafka] Completed load of log with 11 segments, log start offset 
91975003024 and log end offset 92130764817 in 12701 ms (kafka.log.Log)
[06:57:09,701] ERROR There was an error in one of the threads during logs 
loading: org.apache.kafka.common.errors.ProducerFencedException: Producer's 
epoch is no longer valid. There is probably another producer with a newer 
epoch. 63 (request epoch), 66 (server epoch) (kafka.log.LogManager)
[06:57:09,705] INFO [ProducerStateManager 
partition=client_interaction_events_authorid_enrichment-20] Writing producer 
snapshot at offset 92418754384 (kafka.log.ProducerStateManager)
[06:57:09,707] ERROR [KafkaServer id=2] Fatal error during KafkaServer startup. 
Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.errors.ProducerFencedException: Producer's epoch is no 
longer valid. There is probably another producer with a newer epoch. 63 
(request epoch), 66 (server epoch)
{code:java}
 {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7699) Improve wall-clock time punctuations

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-7699:
--

 Summary: Improve wall-clock time punctuations
 Key: KAFKA-7699
 URL: https://issues.apache.org/jira/browse/KAFKA-7699
 Project: Kafka
  Issue Type: New Feature
  Components: streams
Reporter: Matthias J. Sax


Currently, wall-clock time punctuation allow to schedule periodic call backs 
based on wall-clock time progress. The punctuation time starts, when the 
punctuation is scheduled, thus, it's non-deterministic what is desired for many 
use cases (I want a call-back in 5 minutes from "now").

It would be a nice improvement, to allow users to "anchor" wall-clock 
punctation, too, similar to a cron job: Thus, a punctuation would be triggered 
at "fixed" times like the beginning of the next hour, independent when the 
punctuation was registered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7700) AbstractConfig does not honor Properties defaults

[Tommy Becker (JIRA)]

Tommy Becker created KAFKA-7700:
---

 Summary: AbstractConfig does not honor Properties defaults
 Key: KAFKA-7700
 URL: https://issues.apache.org/jira/browse/KAFKA-7700
 Project: Kafka
  Issue Type: Bug
  Components: config
Affects Versions: 2.1.0
Reporter: Tommy Becker
Assignee: Tommy Becker


Kafka clients such as the Consumer and Producer require various configuration 
parameters to work. In the case of the Consumer and Producer, these parameters 
are provided by passing either a {{Map}} or {{Properties}} instance 
to the respective constructor.

{{Properties}} is a legacy class (akin to {{Vector)}} that adds no value above 
{{Map}} other than the ability to wrap another {{Properties}} 
instance that provides defaults. But Kafka negates this benefit by treating the 
{{Properties}} instance as a {{Map}}, which only works due to an unfortunate 
decision to have {{Properties}} extend {{Hashtable}}.  Such treatment bypasses 
the defaults because they are only consulted by {{Properties.getProperty()}}. 
The net result is that when creating Kafka clients via {{Properties}}, any 
configuration from its defaults is ignored.

This has been reported several times over the years as KAFKA-1909, KAFKA-2184, 
KAFKA-3049, and KAFKA-5514. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk8 #3235

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-7701) Revert kafka-trunk-jdk8 CI change to re-enable testAll-based builds

[Colin Hicks (JIRA)]

Colin Hicks created KAFKA-7701:
--

 Summary: Revert kafka-trunk-jdk8 CI change to re-enable 
testAll-based builds
 Key: KAFKA-7701
 URL: https://issues.apache.org/jira/browse/KAFKA-7701
 Project: Kafka
  Issue Type: Task
Reporter: Colin Hicks


As of Apache Jenkins build #3235, the kafka-trunk-jdk8 job was configured to 
build and test independently against Scala 2.12, followed by building and 
testing against Scala 2.11. Previously, the configuration leveraged the testAll 
Gradle task.

Build #3235 completed successfully. The previous series of failures failures of 
the kafka-trunk-jdk8 job correspond to the introduction of Gradle 5.0 in 
[https://github.com/apache/kafka/commit/4a0fd4c41b3255a6df932eb22bd4f45d38717641.]
 The consistent failure symptoms have been instances of 
java.lang.NoClassDefFoundError in Kafka Streams tests.

After addressing the issues assumed to be caused by the Gradle change, the CI 
configuration should be reverted to its previous state.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7702) Prefixed ACLs don't work with single character prefix

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-7702:
-

 Summary: Prefixed ACLs don't work with single character prefix
 Key: KAFKA-7702
 URL: https://issues.apache.org/jira/browse/KAFKA-7702
 Project: Kafka
  Issue Type: Bug
  Components: security
Affects Versions: 2.1.0, 2.0.1
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 2.2.0, 2.1.1, 2.0.2


Prefixed ACLs with a single character are not matched correctly against 
resource names. ALLOW rule with single character prefix doesn't grant access to 
any resource and DENY rule with single character prefix doesn't deny access to 
any resource since the prefix is not matched correctly.

This is not an exploitable security vulnerability since only authenticated 
users with authorization to create ACLs can create the prefixed ACLs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7703) KafkaConsumer.position may return a wrong offset after "seekToEnd" is called

[Shixiong Zhu (JIRA)]

Shixiong Zhu created KAFKA-7703:
---

 Summary: KafkaConsumer.position may return a wrong offset after 
"seekToEnd" is called
 Key: KAFKA-7703
 URL: https://issues.apache.org/jira/browse/KAFKA-7703
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 2.1.0
Reporter: Shixiong Zhu


After "seekToEnd" is called, "KafkaConsumer.position" may return a wrong offset 
set by another reset request.

Here is a reproducer: 
https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246

In this reproducer, "poll(0)" will send an "earliest" request in background. 
However, after "seekToEnd" is called, due to a race condition in 
"Fetcher.resetOffsetIfNeeded" (It's not atomic, "seekToEnd" could happen 
between 
https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246#diff-b45245913eaae46aa847d2615d62cde0R585
 and 
https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246#diff-b45245913eaae46aa847d2615d62cde0R605),
 "KafkaConsumer.position" may return an "earliest" offset.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk11 #132

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-7704) kafka.server.ReplicaFetechManager.MaxLag.Replica metric is reported incorrectly

[Yu Yang (JIRA)]

Yu Yang created KAFKA-7704:
--

 Summary: kafka.server.ReplicaFetechManager.MaxLag.Replica metric 
is reported incorrectly
 Key: KAFKA-7704
 URL: https://issues.apache.org/jira/browse/KAFKA-7704
 Project: Kafka
  Issue Type: Bug
  Components: metrics
Affects Versions: 2.1.0
Reporter: Yu Yang
 Attachments: Screen Shot 2018-12-03 at 4.33.35 PM.png

We deployed kafka 2.1, and noticed a jump in 
kafka.server.ReplicaFetcherManager.MaxLag.Replica metric. At the same time, 
there is no under-replicated partitions. 

The initial analysis showed that kafka 2.1.0 does not report metric correctly 
for topics that have no incoming traffic right now, but had traffic earlier. 
For those topics, ReplicaFetcherManager will consider the maxLag be the latest 
offset. 

For instance, we have a topic *test_topic*: 

{code}
[root@kafkabroker03002:/mnt/kafka/test_topic-0]# ls -l
total 8
-rw-rw-r-- 1 kafka kafka 10485760 Dec  4 00:13 099043947579.index
-rw-rw-r-- 1 kafka kafka0 Sep 23 03:01 099043947579.log
-rw-rw-r-- 1 kafka kafka   10 Dec  4 00:13 099043947579.snapshot
-rw-rw-r-- 1 kafka kafka 10485756 Dec  4 00:13 099043947579.timeindex
-rw-rw-r-- 1 kafka kafka4 Dec  4 00:13 leader-epoch-checkpoint
{code}

kafka reports ReplicaFetcherManager.MaxLag.Replica be 99043947579

 !Screen Shot 2018-12-03 at 4.33.35 PM.png|width=720px! 





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

[John Roesler]

Hi Jan and Adam,

Wow, thanks for doing that test, Adam. Those results are encouraging.

Thanks for your performance experience as well, Jan. I agree that avoiding
unnecessary join outputs is especially important when the fan-out is so
high. I suppose this could also be built into the implementation we're
discussing, but it wouldn't have to be specified in the KIP (since it's an
API-transparent optimization).

As far as whether or not to re-repartition the data, I didn't bring it up
because it sounded like the two of you agreed to leave the KIP as-is,
despite the disagreement.

If you want my opinion, I feel like both approaches are reasonable.
It sounds like Jan values more the potential for developers to optimize
their topologies to re-use the intermediate nodes, whereas Adam places more
value on having a single operator that people can use without extra steps
at the end.

Personally, although I do find it exceptionally annoying when a framework
gets in my way when I'm trying to optimize something, it seems better to go
for a single operation.
* Encapsulating the internal transitions gives us significant latitude in
the implementation (for example, joining only at the end, not in the middle
to avoid extra data copying and out-of-order resolution; how we represent
the first repartition keys (combined keys vs. value vectors), etc.). If we
publish something like a KScatteredTable with the right-partitioned joined
data, then the API pretty much locks in the implementation as well.
* The API seems simpler to understand and use. I do mean "seems"; if anyone
wants to make the case that KScatteredTable is actually simpler, I think
hypothetical usage code would help. From a relational algebra perspective,
it seems like KTable.join(KTable) should produce a new KTable in all cases.
* That said, there might still be room in the API for a different operation
like what Jan has proposed to scatter a KTable, and then do things like
join, re-group, etc from there... I'm not sure; I haven't thought through
all the consequences yet.

This is all just my opinion after thinking over the discussion so far...
-John

On Mon, Dec 3, 2018 at 2:56 PM Adam Bellemare 
wrote:

> Updated the PR to take into account John's feedback.
>
> I did some preliminary testing for the performance of the prefixScan. I
> have attached the file, but I will also include the text in the body here
> for archival purposes (I am not sure what happens to attached files). I
> also updated the PR and the KIP accordingly.
>
> Summary: It scales exceptionally well for scanning large values of
> records. As Jan mentioned previously, the real issue would be more around
> processing the resulting records after obtaining them. For instance, it
> takes approximately ~80-120 mS to flush the buffer and a further ~35-85mS
> to scan 27.5M records, obtaining matches for 2.5M of them. Iterating
> through the records just to generate a simple count takes ~ 40 times longer
> than the flush + scan combined.
>
> 
> Setup:
>
> 
> Java 9 with default settings aside from a 512 MB heap (Xmx512m, Xms512m)
> CPU: i7 2.2 Ghz.
>
> Note: I am using a slightly-modified, directly-accessible Kafka Streams
> RocksDB
> implementation (RocksDB.java, basically just avoiding the
> ProcessorContext).
> There are no modifications to the default RocksDB values provided in the
> 2.1/trunk release.
>
>
> keysize = 128 bytes
> valsize = 512 bytes
>
> Step 1:
> Write X positive matching events: (key = prefix + left-padded
> auto-incrementing integer)
> Step 2:
> Write 10X negative matching events (key = left-padded auto-incrementing
> integer)
> Step 3:
> Perform flush
> Step 4:
> Perform prefixScan
> Step 5:
> Iterate through return Iterator and validate the count of expected events.
>
>
> 
> Results:
>
> 
> X = 1k (11k events total)
> Flush Time = 39 mS
> Scan Time = 7 mS
> 6.9 MB disk
>
> 
> X = 10k (110k events total)
> Flush Time = 45 mS
> Scan Time = 8 mS
> 127 MB
>
> 
> X = 100k (1.1M events total)
> Test1:
> Flush Time = 60 mS
> Scan Time = 12 mS
> 678 MB
>
> Test2:
> Flush Time = 45 mS
> Scan Time = 7 mS
> 576 MB
>
> 
> X = 1MB (11M events total)
> Test1:
> Flush Time = 52 mS
> Scan Time = 19 mS
> 7.2 GB
>
> Test2:
> Flush Time = 84 mS
> Scan Time = 34 mS
> 9.1 GB
>
> -

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Thanks Guozhang for the reply!

1. RemoveMemberFromGroupOptions seems not defined anywhere.
Added the definition.
2. LeaveGroupRequest added a list of group instance id, but still keep the
member id as a singleton; is that intentional? I think to make the protocol
consistent both member id and instance ids could be plural.
Since a dynamic member would send LeaveGroupRequest with its member.id,
I feel it's ok to keep the existing API instead of expanding singleton to a 
list. Haven't
been able to define a scenario where we need to pass a list of `member.id`.
What do you think?

3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
can defer adding this while just add the corresponding calls of the
LeaveGroupRequest inside Streams until we have used it in production and
hence have a better understanding on how flexible or extensible if we want
to add any cmd tools. The rationale is that if we do not necessarily need
it now, we can always add it later with a more think-through API design,
but if we add the tool in a rush, we may need to extend or modify it soon
after we realize its limits in operations.
Totally agree. I moved this part to the future work, because tooling options 
could be addressed
in a separate KIP and a universally favorable solution could be discussed 
independently (for different
company setup)

Best,
Boyang


From: Guozhang Wang 
Sent: Tuesday, December 4, 2018 1:27 AM
To: dev
Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

Hello Boyang,

I've browsed through the new wiki and there are still a couple of minor
things to notice:

1. RemoveMemberFromGroupOptions seems not defined anywhere.

2. LeaveGroupRequest added a list of group instance id, but still keep the
member id as a singleton; is that intentional? I think to make the protocol
consistent both member id and instance ids could be plural.

3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
can defer adding this while just add the corresponding calls of the
LeaveGroupRequest inside Streams until we have used it in production and
hence have a better understanding on how flexible or extensible if we want
to add any cmd tools. The rationale is that if we do not necessarily need
it now, we can always add it later with a more think-through API design,
but if we add the tool in a rush, we may need to extend or modify it soon
after we realize its limits in operations.

Otherwise, I'm +1 on the proposal.

Guozhang


On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen  wrote:

> Hey community friends,
>
> after another month of polishing, KIP-345<
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008&sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&reserved=0>
> design is ready for vote. Feel free to add your comment on the discussion
> thread or here.
>
> Thanks for your time!
>
> Boyang
> 
> From: Boyang Chen 
> Sent: Friday, November 9, 2018 6:35 AM
> To: dev@kafka.apache.org
> Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce
> consumer rebalances
>
> Hey all,
>
>
> thanks so much for all the inputs on KIP-345 so far. The original proposal
> has enhanced a lot with your help. To make sure the implementation go
> smoothly without back and forth, I would like to start a vote on the final
> design agreement now:
>
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008&sdata=GwbfkDFkY2m38V2e%2B6bEWU7PKWPoia5Hw6KmdOXrdcs%3D&reserved=0<
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008&sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&reserved=0
> >
>
> 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances<
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008&sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&reserved=0
> >
>
> KIP-345: Introduce static membership protocol to reduce ...<
> https://nam05.safelinks.protection.outlook.com/?url=https%

[jira] [Created] (KAFKA-7705) Update javadoc for default value of delivery.timeout.ms

[huxihx (JIRA)]

huxihx created KAFKA-7705:
-

 Summary: Update javadoc for default value of delivery.timeout.ms
 Key: KAFKA-7705
 URL: https://issues.apache.org/jira/browse/KAFKA-7705
 Project: Kafka
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.1.0
Reporter: huxihx


In 
[https://kafka.apache.org/21/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html,]

the sample producer code fails to run due to the ConfigException thrown: 
delivery.timeout.ms should be equal to or larger than linger.ms + 
request.timeout.ms

The default value for delivery.timeout.ms or linger.ms should be updated 
accordingly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Mayuresh Gharat]

Hi Folks,

Would it be good to move this to the DISCUSS thread and keep this thread
only for voting purposes, else it will be hard to coordinate responses
between 2 threads.

Thanks,

Mayuresh



On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen  wrote:

> Thanks Guozhang for the reply!
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> Added the definition.
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
> Since a dynamic member would send LeaveGroupRequest with its member.id,
> I feel it's ok to keep the existing API instead of expanding singleton to
> a list. Haven't
> been able to define a scenario where we need to pass a list of `member.id
> `.
> What do you think?
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
> Totally agree. I moved this part to the future work, because tooling
> options could be addressed
> in a separate KIP and a universally favorable solution could be discussed
> independently (for different
> company setup)
>
> Best,
> Boyang
>
> 
> From: Guozhang Wang 
> Sent: Tuesday, December 4, 2018 1:27 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hello Boyang,
>
> I've browsed through the new wiki and there are still a couple of minor
> things to notice:
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
>
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
>
> Otherwise, I'm +1 on the proposal.
>
> Guozhang
>
>
> On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen  wrote:
>
> > Hey community friends,
> >
> > after another month of polishing, KIP-345<
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008&sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&reserved=0
> >
> > design is ready for vote. Feel free to add your comment on the discussion
> > thread or here.
> >
> > Thanks for your time!
> >
> > Boyang
> > 
> > From: Boyang Chen 
> > Sent: Friday, November 9, 2018 6:35 AM
> > To: dev@kafka.apache.org
> > Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce
> > consumer rebalances
> >
> > Hey all,
> >
> >
> > thanks so much for all the inputs on KIP-345 so far. The original
> proposal
> > has enhanced a lot with your help. To make sure the implementation go
> > smoothly without back and forth, I would like to start a vote on the
> final
> > design agreement now:
> >
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008&sdata=GwbfkDFkY2m38V2e%2B6bEWU7PKWPoia5Hw6KmdOXrdcs%3D&reserved=0
> <
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008&sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&reserved=0
> > >
> >
> >
> 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances<
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Boyang Chen]

Oops, sure thing Mayuresh :)

I have only one open question for Guozhang. Will definitely move the discussion 
back.

Boyang

From: Mayuresh Gharat 
Sent: Tuesday, December 4, 2018 9:52 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce 
consumer rebalances

Hi Folks,

Would it be good to move this to the DISCUSS thread and keep this thread
only for voting purposes, else it will be hard to coordinate responses
between 2 threads.

Thanks,

Mayuresh



On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen  wrote:

> Thanks Guozhang for the reply!
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> Added the definition.
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
> Since a dynamic member would send LeaveGroupRequest with its member.id,
> I feel it's ok to keep the existing API instead of expanding singleton to
> a list. Haven't
> been able to define a scenario where we need to pass a list of `member.id
> `.
> What do you think?
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
> Totally agree. I moved this part to the future work, because tooling
> options could be addressed
> in a separate KIP and a universally favorable solution could be discussed
> independently (for different
> company setup)
>
> Best,
> Boyang
>
> 
> From: Guozhang Wang 
> Sent: Tuesday, December 4, 2018 1:27 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hello Boyang,
>
> I've browsed through the new wiki and there are still a couple of minor
> things to notice:
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
>
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
>
> Otherwise, I'm +1 on the proposal.
>
> Guozhang
>
>
> On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen  wrote:
>
> > Hey community friends,
> >
> > after another month of polishing, KIP-345<
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C879b92d6c3fe4da92c1908d6598d7b3d%7C84df9e7fe9f640afb435%7C1%7C0%7C636794861519727292&sdata=s6MOIBWvgDrGPXXBu4vXcsI1Z2dcJIteJQT4zdo%2FSMY%3D&reserved=0
> >
> > design is ready for vote. Feel free to add your comment on the discussion
> > thread or here.
> >
> > Thanks for your time!
> >
> > Boyang
> > 
> > From: Boyang Chen 
> > Sent: Friday, November 9, 2018 6:35 AM
> > To: dev@kafka.apache.org
> > Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce
> > consumer rebalances
> >
> > Hey all,
> >
> >
> > thanks so much for all the inputs on KIP-345 so far. The original
> proposal
> > has enhanced a lot with your help. To make sure the implementation go
> > smoothly without back and forth, I would like to start a vote on the
> final
> > design agreement now:
> >
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-&data=02%7C01%7C%7C879b92d6c3fe4da92c1908d6598d7b3d%7C84df9e7fe9f640afb435%7C1%7C0%7C636794861519727292&sdata=9QF8gdy%2FGD2NSOjNDsfZ3J3FPlk4ShZ8nL%2BW%2FVU5mpc%3D&reserved=0
> <
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Guozhang Wang]

2. That's a good point. I think I'm convinced.


Guozhang

On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen  wrote:

> Thanks Guozhang for the reply!
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> Added the definition.
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
> Since a dynamic member would send LeaveGroupRequest with its member.id,
> I feel it's ok to keep the existing API instead of expanding singleton to
> a list. Haven't
> been able to define a scenario where we need to pass a list of `member.id
> `.
> What do you think?
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
> Totally agree. I moved this part to the future work, because tooling
> options could be addressed
> in a separate KIP and a universally favorable solution could be discussed
> independently (for different
> company setup)
>
> Best,
> Boyang
>
> 
> From: Guozhang Wang 
> Sent: Tuesday, December 4, 2018 1:27 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hello Boyang,
>
> I've browsed through the new wiki and there are still a couple of minor
> things to notice:
>
> 1. RemoveMemberFromGroupOptions seems not defined anywhere.
>
> 2. LeaveGroupRequest added a list of group instance id, but still keep the
> member id as a singleton; is that intentional? I think to make the protocol
> consistent both member id and instance ids could be plural.
>
> 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we
> can defer adding this while just add the corresponding calls of the
> LeaveGroupRequest inside Streams until we have used it in production and
> hence have a better understanding on how flexible or extensible if we want
> to add any cmd tools. The rationale is that if we do not necessarily need
> it now, we can always add it later with a more think-through API design,
> but if we add the tool in a rush, we may need to extend or modify it soon
> after we realize its limits in operations.
>
> Otherwise, I'm +1 on the proposal.
>
> Guozhang
>
>
> On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen  wrote:
>
> > Hey community friends,
> >
> > after another month of polishing, KIP-345<
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008&sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&reserved=0
> >
> > design is ready for vote. Feel free to add your comment on the discussion
> > thread or here.
> >
> > Thanks for your time!
> >
> > Boyang
> > 
> > From: Boyang Chen 
> > Sent: Friday, November 9, 2018 6:35 AM
> > To: dev@kafka.apache.org
> > Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce
> > consumer rebalances
> >
> > Hey all,
> >
> >
> > thanks so much for all the inputs on KIP-345 so far. The original
> proposal
> > has enhanced a lot with your help. To make sure the implementation go
> > smoothly without back and forth, I would like to start a vote on the
> final
> > design agreement now:
> >
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008&sdata=GwbfkDFkY2m38V2e%2B6bEWU7PKWPoia5Hw6KmdOXrdcs%3D&reserved=0
> <
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C94b56c977f3647e1141908d659458c8c%7C84df9e7fe9f640afb435%7C1%7C0%7C636794552568572008&sdata=LiNnhFJm8Avri26aEBa3q4%2Fr4aRKVIrZzKHzn71U3Xk%3D&reserved=0
> > >
> >
> >
> 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances<
> >
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data

KAFKA-6144(or other ticket to resolve)

[Nikolay Izhikov]

Hello, Kafka developers.

I found the link to a KAFKA-6144 in youtube video [1]
Gwen, Matthias, great video by the way :)

I want to know - is this ticket actual for the Kafka product?

Are commiters and other community members are interested in it's resolve?

It unassigned for now and I want to work on it.
Can I expect that some commiters help me with it?

May be you can offer some other ticket to me, that has to be resolved?

I'm experienced java programmer with 1 KIP done and a couple commits in Kafka 
trunk :)

[1] https://www.youtube.com/watch?v=TG77-nNR1yw


signature.asc
Description: This is a digitally signed message part

Re: [DISCUSS] KIP-378: Enable Dependency Injection for Kafka Streams handlers

[Guozhang Wang]

Hello Wladimir,

I've thought about the two options and I think I'm sold on the second
option and actually I think it is better generalize it to be potentially
used for other clients (producer, consumer) as while since they also have
similar dependency injection requests for metrics reporter, partitioner,
partition assignor etc.

So I'd suggest we add the following to AbstractConfig directly (note I
intentionally renamed the class to ConfiguredInstanceFactory to be used for
other clients as well):

```
AbstractConfig(ConfigDef definition, Map originals,
ConfiguredInstanceFactory, boolean doLog)
```

And then in StreamsConfig add:

```
StreamsConfig(Map props, ConfiguredInstanceFactory)
```

which would call the above AbstractConfig constructor (we can leave to core
team to decide when they want to add for producer and consumer);

And in KafkaStreams / TopologyTestDriver we can add one overloaded
constructor each that includes all the parameters including the
ConfiguredInstanceFactory --- for those who only want `factory` but not
`client-suppliers` for example, they can set it to `null` and the streams
library will just use the default one.


Guozhang


On Sun, Dec 2, 2018 at 12:13 PM Wladimir Schmidt  wrote:

> Hello Guozhang,
> sure, the first approach is very straight-forward and allows minimal
> changes to the Kafka Streams API.
> On the other hand, second approach with the interface implementation looks
> more cleaner to me.
> I totally agree that this should be first discussed before will be
> implemented.
>
> Thanks, Wladimir
>
>
> On 17-Nov-18 23:37, Guozhang Wang wrote:
>
> Hello folks,
>
> I'd like to revive this thread for discussion. After reading the previous
> emails I think I'm still a bit leaning towards re-enabling to pass in
> StreamsConfig to Kafka Streams constructors compared with a
> ConfiguredStreamsFactory as additional parameters to overloaded
> KafkaStreams constructors: although the former seems less cleaner as it
> requires users to read through the usage of AbstractConfig to know how to
> use it in their frameworks, this to me is a solvable problem through
> documentations, plus AbstractConfig is a public interface already and hence
> the additional ConfiguredStreamsFactory to me is really a bit overlapping
> in functionality.
>
>
> Guozhang
>
>
>
> On Sun, Oct 21, 2018 at 1:41 PM Wladimir Schmidt  
>  wrote:
>
>
> Hi Damian,
>
> The first approach was added only because it had been initially proposed
> in my pull request,
> which started a discussion and thus, the KIP-378 was born.
>
> Yes, I would like to have something "injectable". In this regard, a
> `ConfiguredStreamsFactory` (name is a subject to discussion)
> is a good option to be introduced into `KafkaStreams` constructor.
>
> Even though, I consider the second approach to be cleaner, it involves a
> certain amount of refactoring of the streams library.
> The first approach, on the contrary, adds (or removes deprecated
> annotation, if the method has not been removed yet) only additional
> constructors with
> considerably less intervention into a streams library (no changes, which
> would break an API. Please see a pull 
> request:https://github.com/apache/kafka/pull/5344).
>
> Thanks
> Wladimir
>
> On 10-Oct-18 15:51, Damian Guy wrote:
>
> Hi Wladimir,
>
> Of the two approaches in the KIP - i feel the second approach is cleaner.
> However, am i correct in assuming that you want to have the
> `ConfiguredStreamsFactory` as a ctor arg in `StreamsConfig` so that
>
> Spring
>
> can inject this for you?
>
> Otherwise you could just put the ApplicationContext as a property in the
> config and then use that via the configure method of the appropriate
> handler to get your actual handler.
>
> Thanks,
> Damian
>
> On Tue, 9 Oct 2018 at 01:55, Guozhang Wang  
>  wrote:
>
>
> John, thanks for the explanation, now it makes much more sense to me.
>
> As for the concrete approach, to me it seems the first option requires
>
> less
>
> changes than the second (ConfiguredStreamsFactory based) approach,
>
> whereas
>
> the second one requires an additional interface that is overlapping with
> the AbstractConfig.
>
> I'm aware that in KafkaProducer / KafkaConsumer we do not have public
> constructors for taking a ProducerConfig or ConsumerConfig directly, and
> anyone using Spring can share how you've worked around it by far? If it
>
> is
>
> very awkward I'm not against just adding the XXXConfigs to the
>
> constructors
>
> directly.
>
> Guozhang
>
> On Fri, Oct 5, 2018 at 1:48 PM, John Roesler  
>  wrote:
>
>
> Hi Wladimir,
>
> Thanks for the KIP!
>
> As I mentioned in the PR discussion, I personally prefer not to
>
> recommend
>
> overriding StreamsConfig for this purpose.
>
> It seems like a person wishing to create a DI shim would have to
>
> acquire
>
> quite a deep understanding of the class and its usage to figure out
>
> what
>
> exactly to override to accomplish their goals without breaking
>
> everything.
>
> I'm honestly impr