date:20200407

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-04-07 Thread Kowshik Prakasam

Hi Colin,

Thanks for the feedback, and suggestions! It is a great idea to provide a
`--finalize-latest` flag. I agree it's a burden to ask the user to manually
upgrade each feature to the latest version, after a release.

I have now updated the KIP adding this idea.

> What about a simple solution to problem this where we add a flag to the
command-line tool like --enable-latest?  The command-line tool could query
what the highest possible versions for
> each feature were (using the API) and then make another RPC to enable the
latest features.

(Kowshik): I've updated the KIP with the above idea, please look at this
section (point #3 and the tooling example later):
https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP-584:Versioningschemeforfeatures-Toolingsupport


> I think this is actually much easier than the version number solution.
The version string solution requires us to maintain a complicated mapping
table between version strings and features.  > In practice, we also have
"internal versions" in ApiVersion.scala like 2.4IV0, 2.4IV1, and so on.
This isn't simple for users to understand or use.

> It's also hard to know what the difference is between different version
strings.  For example, there's actually no difference between 2.5IV0 and
2.4IV1, but you wouldn't know that unless you > read the comments in
ApiVersion.scala.  A system administrator who didn't know this might end up
doing a cluster roll to upgrade the IBP that turned out to be unnecessary.

(Kowshik): Yes, I can see the disadvantages!


Cheers,
Kowshik



On Mon, Apr 6, 2020 at 3:46 PM Colin McCabe  wrote:

> Hi Jun,
>
> I agree that asking the user to manually upgrade all features to the
> latest version is a burden.  Then the user has to know what the latest
> version of every feature is when upgrading.
>
> What about a simple solution to problem this where we add a flag to the
> command-line tool like --enable-latest?  The command-line tool could query
> what the highest possible versions for each feature were (using the API)
> and then make another RPC to enable the latest features.
>
> I think this is actually much easier than the version number solution.
> The version string solution requires us to maintain a complicated mapping
> table between version strings and features.  In practice, we also have
> "internal versions" in ApiVersion.scala like 2.4IV0, 2.4IV1, and so on.
> This isn't simple for users to understand or use.
>
> It's also hard to know what the difference is between different version
> strings.  For example, there's actually no difference between 2.5IV0 and
> 2.4IV1, but you wouldn't know that unless you read the comments in
> ApiVersion.scala.  A system administrator who didn't know this might end up
> doing a cluster roll to upgrade the IBP that turned out to be unnecessary.
>
> best,
> Colin
>
>
> On Mon, Apr 6, 2020, at 12:06, Jun Rao wrote:
> > Hi, Kowshik,
> >
> > Thanks for the reply. A few more replies below.
> >
> > 100.6 You can look for the sentence "This operation requires ALTER on
> > CLUSTER." in KIP-455. Also, you can check its usage in
> > KafkaApis.authorize().
> >
> > 110. From the external client/tooling perspective, it's more natural to
> use
> > the release version for features. If we can use the same release version
> > for internal representation, it seems simpler (easier to understand, no
> > mapping overhead, etc). Is there a benefit with separate external and
> > internal versioning schemes?
> >
> > 111. To put this in context, when we had IBP, the default value is the
> > current released version. So, if you are a brand new user, you don't need
> > to configure IBP and all new features will be immediately available in
> the
> > new cluster. If you are upgrading from an old version, you do need to
> > understand and configure IBP. I see a similar pattern here for
> > features. From the ease of use perspective, ideally, we shouldn't
> require a
> > new user to have an extra step such as running a bootstrap script unless
> > it's truly necessary. If someone has a special need (all the cases you
> > mentioned seem special cases?), they can configure a mode such that
> > features are enabled/disabled manually.
> >
> > Jun
> >
> > On Fri, Apr 3, 2020 at 5:54 PM Kowshik Prakasam 
> > wrote:
> >
> > > Hi Jun,
> > >
> > > Thanks for the feedback and suggestions. Please find my response below.
> > >
> > > > 100.6 For every new request, the admin needs to control who is
> allowed to
> > > > issue that request if security is enabled. So, we need to assign the
> new
> > > > request a ResourceType and possible AclOperations. See
> > > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-455%3A+Create+an+Administrative+API+for+Replica+Reassignment
> > > > as an example.
> > >
> > > (Kowshik): I don't see any reference to the words ResourceType or
> > > AclOperations
> > > in the KIP. Please let me know how I can use the KIP that you linked to

Re: [DISCUSS] KIP-584: Versioning scheme for features

2020-04-07 Thread Kowshik Prakasam

Hi Jun,

I have updated the KIP for the item 111.
I'm in the process of addressing 100.6, and will provide an update soon.
I think item 110 is still under discussion given we are now providing a way
to finalize
all features to their latest version levels. In any case, please let us know
how you feel in response to Colin's comments on this topic.

> 111. To put this in context, when we had IBP, the default value is the
> current released version. So, if you are a brand new user, you don't need
> to configure IBP and all new features will be immediately available in the
> new cluster. If you are upgrading from an old version, you do need to
> understand and configure IBP. I see a similar pattern here for
> features. From the ease of use perspective, ideally, we shouldn't require
a
> new user to have an extra step such as running a bootstrap script unless
> it's truly necessary. If someone has a special need (all the cases you
> mentioned seem special cases?), they can configure a mode such that
> features are enabled/disabled manually.

(Kowshik): That makes sense, thanks for the idea! Sorry if I didn't
understand
this need earlier. I have updated the KIP with the approach that whenever
the '/features' node is absent, the controller by default will bootstrap
the node
to contain the latest feature levels. Here is the new section in the KIP
describing
the same:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP-584:Versioningschemeforfeatures-Controller:ZKnodebootstrapwithdefaultvalues

Next, as I explained in my response to Colin's suggestions, we are now
providing a `--finalize-latest-features` flag with the tooling. This lets
the sysadmin finalize all features known to the controller to their latest
version
levels. Please look at this section (point #3 and the tooling example
later):
https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP-584:Versioningschemeforfeatures-Toolingsupport


Do you feel this addresses your comment/concern?


Cheers,
Kowshik

On Mon, Apr 6, 2020 at 12:06 PM Jun Rao  wrote:

> Hi, Kowshik,
>
> Thanks for the reply. A few more replies below.
>
> 100.6 You can look for the sentence "This operation requires ALTER on
> CLUSTER." in KIP-455. Also, you can check its usage in
> KafkaApis.authorize().
>
> 110. From the external client/tooling perspective, it's more natural to use
> the release version for features. If we can use the same release version
> for internal representation, it seems simpler (easier to understand, no
> mapping overhead, etc). Is there a benefit with separate external and
> internal versioning schemes?
>
> 111. To put this in context, when we had IBP, the default value is the
> current released version. So, if you are a brand new user, you don't need
> to configure IBP and all new features will be immediately available in the
> new cluster. If you are upgrading from an old version, you do need to
> understand and configure IBP. I see a similar pattern here for
> features. From the ease of use perspective, ideally, we shouldn't require a
> new user to have an extra step such as running a bootstrap script unless
> it's truly necessary. If someone has a special need (all the cases you
> mentioned seem special cases?), they can configure a mode such that
> features are enabled/disabled manually.
>
> Jun
>
> On Fri, Apr 3, 2020 at 5:54 PM Kowshik Prakasam 
> wrote:
>
> > Hi Jun,
> >
> > Thanks for the feedback and suggestions. Please find my response below.
> >
> > > 100.6 For every new request, the admin needs to control who is allowed
> to
> > > issue that request if security is enabled. So, we need to assign the
> new
> > > request a ResourceType and possible AclOperations. See
> > >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-455%3A+Create+an+Administrative+API+for+Replica+Reassignment
> > > as an example.
> >
> > (Kowshik): I don't see any reference to the words ResourceType or
> > AclOperations
> > in the KIP. Please let me know how I can use the KIP that you linked to
> > know how to
> > setup the appropriate ResourceType and/or ClusterOperation?
> >
> > > 105. If we change delete to disable, it's better to do this
> consistently
> > in
> > > request protocol and admin api as well.
> >
> > (Kowshik): The API shouldn't be called 'disable' when it is deleting a
> > feature.
> > I've just changed the KIP to use 'delete'. I don't have a strong
> > preference.
> >
> > > 110. The minVersion/maxVersion for features use int64. Currently, our
> > > release version schema is major.minor.bugfix (e.g. 2.5.0). It's
> possible
> > > for new features to be included in minor releases too. Should we make
> the
> > > feature versioning match the release versioning?
> >
> > (Kowshik): The release version can be mapped to a set of feature
> versions,
> > and this can be done, for example in the tool (or even external to the
> > tool).
> > Can you please clarify what I'm missing?

Build failed in Jenkins: kafka-trunk-jdk11 #1329

2020-04-07 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: Should cleanup the tasks after dirty close (#8433)

[github] KAFKA-9818: improve error message to debug test (#8423)


--
[...truncated 3.01 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRe

[jira] [Created] (KAFKA-9828) Add partition to TestRecord in streams test-utils

2020-04-07 Thread Levani Kokhreidze (Jira)

Levani Kokhreidze created KAFKA-9828:


 Summary: Add partition to TestRecord in streams test-utils
 Key: KAFKA-9828
 URL: https://issues.apache.org/jira/browse/KAFKA-9828
 Project: Kafka
  Issue Type: Improvement
  Components: streams-test-utils
Reporter: Levani Kokhreidze


TopologyTestDriver creates `TestRecord` for consumed events. In order to test 
partitioning, when one uses custom partitioner, would be useful if `TestRecord` 
had `partition` field as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk8 #4407

2020-04-07 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: Should cleanup the tasks after dirty close (#8433)

[github] KAFKA-9818: improve error message to debug test (#8423)


--
[...truncated 2.99 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNot

Build failed in Jenkins: kafka-2.4-jdk8 #184

2020-04-07 Thread Apache Jenkins Server

See 


Changes:

[jason] KAFKA-9815; Ensure consumer always re-joins if JoinGroup fails (#8420)


--
[...truncated 5.58 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > shouldCloseProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldCloseProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFeedStoreFromGlobalKTable[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFeedStoreFromGlobalKTable[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCleanUpPersistentStateStoresOnClose[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCleanUpPersistentStateStoresOnClose[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForSta

Re: [VOTE] KIP-444: Refactor and Augment Metrics for Kafka Streams

2020-04-07 Thread Bruno Cadonna

Hi Guozhang,

Thank you for the additions.

Best,
Bruno

On Fri, Apr 3, 2020 at 2:08 AM Guozhang Wang  wrote:
>
> A correction for the previous email:
>
>
>- number of alive global threads, INFO
>- number of alive restore threads, INFO
>
>
> These two metrics are not going to be added in this KIP, since we do not
> have restore threads in Kafka Streams yet.
>
>
> Guozhang
>
>
> On Thu, Apr 2, 2020 at 2:02 PM Guozhang Wang  wrote:
>
> > Hello all,
> >
> > While implementing the last piece of this KIP for the coming 2.6 release,
> > I realized that it is important to cover the following monitoring metrics
> > as well so I'd propose adding them as part of KIP-444 too:
> >
> > Instance-level:
> >
> >- number of alive stream threads, INFO
> >- number of alive cleanup threads, INFO
> >- number of alive global threads, INFO
> >- number of alive restore threads, INFO
> >
> > Monitoring these numbers can help if any threads died unexpectedly while
> > the instance is still proceeding.
> >
> > Thread-level:
> >
> >- avg / max number of records polled from the consumer per thread
> >iteration, INFO
> >- avg / max number of records processed by the task manager (i.e.
> >across all tasks) per thread iteration, INFO
> >
> > Ideally the all polled records can be processed as well within one
> > iteration --- if one observed either we polled too few records such that
> > thread is mostly idling, or polled too many records that the thread cannot
> > keep up, she should go ahead and tune the consumer configs.
> >
> > Task-level:
> >
> >- number of current buffered records at the moment (i.e. it is just a
> >dynamic gauge), DEBUG.
> >
> > This is a finer grained metric indicating which task's processing cannot
> > keep up with the fetching throughput.
> >
> >
> > Please let me know if anyone has any concerns about the proposed metrics.
> >
> >
> > Guozhang
> >
> >
> >
> > On Mon, Sep 9, 2019 at 5:17 PM Matthias J. Sax 
> > wrote:
> >
> >> +1 (binding)
> >>
> >>
> >> -Matthias
> >>
> >> On 9/5/19 11:47 AM, Guozhang Wang wrote:
> >> > +1 from myself.
> >> >
> >> > I'm now officially closing this voting thread with the following tally:
> >> >
> >> > binding +1: 3 (Guozhang, Bill, Matthias voted on the DISCUSS thread).
> >> > non-binding +1: 2 (Bruno, John).
> >> >
> >> >
> >> > Guozhang
> >> >
> >> >
> >> > On Thu, Aug 22, 2019 at 8:16 AM Bill Bejeck  wrote:
> >> >
> >> >> +1 (binding)
> >> >>
> >> >> -Bill
> >> >>
> >> >> On Thu, Aug 22, 2019 at 10:55 AM John Roesler 
> >> wrote:
> >> >>
> >> >>> Hi Guozhang, thanks for cleaning this up.
> >> >>>
> >> >>> I'm +1 (non-binding)
> >> >>>
> >> >>> Thanks,
> >> >>> -John
> >> >>>
> >> >>> On Thu, Aug 22, 2019 at 2:26 AM Bruno Cadonna 
> >> >> wrote:
> >> >>>
> >>  Hi Guozhang,
> >> 
> >>  +1 (non-binding)
> >> 
> >>  Thank you for driving this!
> >>  Bruno
> >> 
> >>  On Tue, Aug 20, 2019 at 8:29 PM Guozhang Wang 
> >> >>> wrote:
> >> >
> >> > Hello folks,
> >> >
> >> > I'd like to start a voting thread the following KIP to improve the
> >> >>> Kafka
> >> > Streams metrics mechanism to users. This includes 1) renaming
> >> changes
> >> >>> in
> >> > the public StreamsMetrics utils API, and 2) a major cleanup on the
> >>  Streams'
> >> > own built-in metrics hierarchy.
> >> >
> >> > Details can be found here:
> >> >
> >> >
> >> 
> >> >>>
> >> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams
> >> >
> >> > I'd love to hear your thoughts and feedbacks. Thanks!
> >> >
> >> > --
> >> > -- Guozhang
> >> 
> >> >>>
> >> >>
> >> >
> >> >
> >>
> >>
> >
> > --
> > -- Guozhang
> >
>
>
> --
> -- Guozhang

Shutdown Broker Because Log Dirs Failed

2020-04-07 Thread Qarl

Hi, my name is Qarl. I am a newbie in software development, and I am using 
Kafka in this software. I have been facing this issue for weeks now. I’m 
running Kafka on Windows platform. The issue is when I run the Kafka server, 
and the retention period has reached, it will clear the logs file but there is 
always problem, such as ‘The process cannot access the file because it is being 
used by another process.’  How do I solve this? I’m using Kafka in a way where 
if the Kafka server stops running, our software will face a serious issue. 


Sent from Mail for Windows 10

Re: Shutdown Broker Because Log Dirs Failed

2020-04-07 Thread Sönke Liebau

Hi Qarl,

there probably (and sadly) is no good answer to your question. Windows is
not an officially supported platform to run Kafka on and there is a large
number of unresolved tickets in Jira on this topic.
Most notable is probably KAFKA-1194 [1].

I haven't closely followed this, I thought we were getting close to a
solution a little while ago, but the ticket is still unresolved .. maybe
someone else knows more on the current state than is documented in the
ticket?

Best regards,
Sönke

[1] https://issues.apache.org/jira/browse/KAFKA-1194

On Tue, 7 Apr 2020 at 11:00, Qarl  wrote:

> Hi, my name is Qarl. I am a newbie in software development, and I am using
> Kafka in this software. I have been facing this issue for weeks now. I’m
> running Kafka on Windows platform. The issue is when I run the Kafka
> server, and the retention period has reached, it will clear the logs file
> but there is always problem, such as ‘The process cannot access the file
> because it is being used by another process.’  How do I solve this? I’m
> using Kafka in a way where if the Kafka server stops running, our software
> will face a serious issue.
>
>
> Sent from Mail for Windows 10
>

RE: Shutdown Broker Because Log Dirs Failed

2020-04-07 Thread Qarl

Thank you so much for the response! 

If this is unresolved, could you suggest any alternative for me to make this 
work? If that is possible. 

Any other solution is highly appreciated! Thank you.

From: Sönke Liebau
Sent: Tuesday, 7 April, 2020 6:56 PM
To: dev
Subject: Re: Shutdown Broker Because Log Dirs Failed

Hi Qarl,

there probably (and sadly) is no good answer to your question. Windows is
not an officially supported platform to run Kafka on and there is a large
number of unresolved tickets in Jira on this topic.
Most notable is probably KAFKA-1194 [1].

I haven't closely followed this, I thought we were getting close to a
solution a little while ago, but the ticket is still unresolved .. maybe
someone else knows more on the current state than is documented in the
ticket?

Best regards,
Sönke

[1] https://issues.apache.org/jira/browse/KAFKA-1194

On Tue, 7 Apr 2020 at 11:00, Qarl  wrote:

> Hi, my name is Qarl. I am a newbie in software development, and I am using
> Kafka in this software. I have been facing this issue for weeks now. I’m
> running Kafka on Windows platform. The issue is when I run the Kafka
> server, and the retention period has reached, it will clear the logs file
> but there is always problem, such as ‘The process cannot access the file
> because it is being used by another process.’  How do I solve this? I’m
> using Kafka in a way where if the Kafka server stops running, our software
> will face a serious issue.
>
>
> Sent from Mail for Windows 10
>

回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter

2020-04-07 Thread feyman2009

Hi, Matthias
Thanks a lot!
So you do not plan so support removing a _single static_ member via 
`StreamsResetter`? 
=> 
Would you mind to elaborate why we still need that if we are able to 
batch remove active members with adminClient?

Thanks! 

Feyman
 --
发件人：Matthias J. Sax 
发送时间：2020年4月7日(星期二) 08:25
收件人：dev 
主　题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in 
StreamsResetter

Overall LGTM.

+1 (binding)

So you do not plan so support removing a _single static_ member via
`StreamsResetter`? We can of course still add this as a follow up but it
might be nice to just add it to this KIP right away. Up to you if you
want to include it or not.


-Matthias



On 3/30/20 8:16 AM, feyman2009 wrote:
> Hi, Boyang
> Thanks a lot, that make sense, we should not expose member.id in the 
> MemberToRemove anymore, I have fixed it in the KIP.
> 
> 
> Feyman
> --
> 发件人：Boyang Chen 
> 发送时间：2020年3月29日(星期日) 01:45
> 收件人：dev ; feyman2009 
> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in 
> StreamsResetter
> 
> Hey Feyman,
> 
> thanks for the update. I assume we would rely entirely on the internal 
> changes for `removeMemberFromGroup` by sending a DescribeGroup request first. 
> With that in mind, I don't think we need to add member.id to MemberToRemove 
> anymore, as it is only facing public where users will only configure 
> group.instance.id? 
> On Fri, Mar 27, 2020 at 5:04 PM feyman2009  
> wrote:
> Bump, can anyone kindly take a look at the updated KIP-571? Thanks!
> 
> 
>  --
>  发件人：feyman2009 
>  发送时间：2020年3月23日(星期一) 08:51
>  收件人：dev 
>  主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in 
> StreamsResetter
> 
>  Hi, team
>  I have updated the KIP-571 according to our latest discussion results, 
> would you mind to take a look? Thanks!
> 
>  Feyman
> 
> 
>  --
>  发件人：Boyang Chen 
>  发送时间：2020年3月19日(星期四) 13:41
>  收件人：dev ; feyman2009 
>  主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in 
> StreamsResetter
> 
>  Thanks for the insight Feyman. I personally feel adding another admin client 
> command is redundant, so picking option 1). The memberInfos struct is 
> internal and just used for result reference purposes. I think it could still 
> work even we overload with `removeAll` option, if that makes sense.
> 
>  Boyang
>  On Wed, Mar 18, 2020 at 8:51 PM feyman2009  
> wrote:
>  Hi, team
>   Before going too far on the KIP update, I would like to hear your 
> opinions about how we would change the interface of AdminClient, the two 
> alternatives I could think of are:
>   1) Extend adminClient.removeMembersFromConsumerGroup to support remove 
> all
>   As Guochang suggested, we could add some flag param in 
> RemoveMembersFromConsumerGroupOptions to indicated the "remove all" logic.  
>   2) Add a new API like 
> adminClient.removeAllMembersFromConsumerGroup(groupId, options) 
> 
>   I think 1) will be more compact from the API perspective, but looking 
> at the code, I found that the if we are going to remove all, then the 
> RemoveMembersFromConsumerGroupResult#memberInfos/memberResult()/all() should 
> be changed accordingly, and they seem not that meaningful under the "remove 
> all" scenario.
> 
>   A minor thought about the adminClient.removeMembersFromConsumerGroup 
> API is:
>   Looking at some other deleteXX APIs, like deleteTopics, deleteRecords, 
> the results contains only a Map>, I think it's enough to 
> describe the related results, is it make sense that we may remove memberInfos 
> in RemoveMembersFromConsumerGroupResult ? This KIP has no dependency on this 
> if we choose alternative 2)
> 
>   Could you advise? Thanks!
> 
>   Feyman
> 
> 
>   送时间：2020年3月15日(星期日) 10:11
>   收件人：dev 
>   主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in 
> StreamsResetter
> 
>   Hi, all
>   Thanks a lot for your feedback!
>   According to the discussion, it seems we don't have some valid use 
> cases for removing specific dynamic members, I think it makes sense to 
> encapsulate the "get and delete" logic in adminClient. I will update the KIP 
> shortly!
> 
>   Thanks!
> 
>   Feyman
> 
> 
>   --
>   发件人：Boyang Chen 
>   发送时间：2020年3月14日(星期六) 00:39
>   收件人：dev 
>   主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in 
> StreamsResetter
> 
>   Thanks Matthias and Guozhang for the feedback. I'm not worrying too much
>   about the member.id exposure as we have done so in a couple of areas. As
>   for the recommended admin client change, I think it makes sense in an
>   encapsulation perspective. Maybe I'm still a bit

Re: Shutdown Broker Because Log Dirs Failed

2020-04-07 Thread Sönke Liebau

Hi Qarl,

ideally you'd run Kafka on a linux host. If all you have is a Windows
machine then Docker might be an option, Confluent has images available that
you could use. Alternatively some form of virtual machine running linux
would work as well.

Best regards,
Sönke

Qarl  schrieb am Di., 7. Apr. 2020, 16:02:

> Thank you so much for the response!
>
> If this is unresolved, could you suggest any alternative for me to make
> this work? If that is possible.
>
> Any other solution is highly appreciated! Thank you.
>
> From: Sönke Liebau
> Sent: Tuesday, 7 April, 2020 6:56 PM
> To: dev
> Subject: Re: Shutdown Broker Because Log Dirs Failed
>
> Hi Qarl,
>
> there probably (and sadly) is no good answer to your question. Windows is
> not an officially supported platform to run Kafka on and there is a large
> number of unresolved tickets in Jira on this topic.
> Most notable is probably KAFKA-1194 [1].
>
> I haven't closely followed this, I thought we were getting close to a
> solution a little while ago, but the ticket is still unresolved .. maybe
> someone else knows more on the current state than is documented in the
> ticket?
>
> Best regards,
> Sönke
>
> [1] https://issues.apache.org/jira/browse/KAFKA-1194
>
> On Tue, 7 Apr 2020 at 11:00, Qarl  wrote:
>
> > Hi, my name is Qarl. I am a newbie in software development, and I am
> using
> > Kafka in this software. I have been facing this issue for weeks now. I’m
> > running Kafka on Windows platform. The issue is when I run the Kafka
> > server, and the retention period has reached, it will clear the logs file
> > but there is always problem, such as ‘The process cannot access the file
> > because it is being used by another process.’  How do I solve this? I’m
> > using Kafka in a way where if the Kafka server stops running, our
> software
> > will face a serious issue.
> >
> >
> > Sent from Mail for Windows 10
> >
>
>

Re: 2.5.0 Producer.sendOffsetsToTransaction() and Backwards Compatibility

2020-04-07 Thread Gary Russell

Thanks; Matthias,

Very useful.

On Mon, Apr 6, 2020 at 8:05 PM Matthias J. Sax  wrote:

> I guess one important point to mention is why Kafka Streams needs the
> internal config though: it's about a save upgrade path.
>
> Even if the user tells us that they are on old brokers, we call the new
> `sendOffsetsToTransaction()` API blindly and let the producer downgrade
> the request. If the user upgrades their brokers, the producer can
> automatically switch to the new request type. During this phase, the old
> `transaction.id` fencing _and_ the new fetch-offset-request fencing are
> both active at the same time.
>
> Because both fencing mechanism are active, a user can now switch the
> config to enable eos-beta and do a single round of rolling bounces.
>
> It is not save to have only `transaction.id` fencing (but not
> fetch-offset-request fencing) and switch to fetch-offset-request fencing
> in a single rolling bounce. If an error occurs during the upgrade, EOS
> might be violated.
>
> Thus, in the end the internal config is a way to simplify the upgrade
> mechanism. We could also have chosen to add a new public config to Kafka
> Streams instead, but we wanted to improve the user experience.
>
> Thus, for your use case Gary, you would need to have one more config
> value for the upgrade mode in which you still use the old model but
> enable fetch-offset-request fencing in parallel. An upgrade would
> require two rolling bounce, first to enable upgrade mode, second to
> enable the new feature.
>
>
> -Matthias
>
> On 4/6/20 12:10 PM, Gary Russell wrote:
> > Thanks, all,
> >
> >> Just to clarify, even for Streams client it cannot detect automatically
> the
> > broker's version and hence as KIP-447 proposed, the customer needs to
> set a
> > config value indicating that she is assured the broker version is newer
> and
> > hence the new API can be used.
> >
> > Yes, I noticed that; Ok, I'll go the same way, then and make it a
> > configuration option.
> >
> > Yes, KIP-584 looks interesting to me.
> >
> > Thanks again.
> >
> >
> > On Sun, Apr 5, 2020 at 5:30 PM Guozhang Wang  wrote:
> >
> >> Hello Gary,
> >>
> >> Just to clarify, even for Streams client it cannot detect automatically
> the
> >> broker's version and hence as KIP-447 proposed, the customer needs to
> set a
> >> config value indicating that she is assured the broker version is newer
> and
> >> hence the new API can be used. On the other hand, if the config is not
> set,
> >> even if broker is on newer version we would still use the old behavior
> (one
> >> producer per task) and the although overloaded sendOffsetsToTransaction
> is
> >> used, it would not be much helpful in the scope of KIP-447.
> >>
> >> As for your case, currently there's no good ways to determine if the new
> >> overloaded function can be safely used except the users need to indicate
> >> (via configs, for example) that she's assured the broker version is
> newer.
> >> I think the ultimate solution for you would be KIP-584, in which client
> can
> >> dynamically ask the broker cluster which version(s) they would support
> and
> >> decides which function to trigger accordingly.
> >>
> >>
> >> Guozhang
> >>
> >>
> >> On Sat, Apr 4, 2020 at 2:46 PM Boyang Chen 
> >> wrote:
> >>
> >>> That’s a fair point Ismael. After a second thought, I feel that if Gary
> >> is
> >>> building frameworks for general purpose usage, relying on private flag
> >>> seems not a good idea.
> >>>
> >>> On Sat, Apr 4, 2020 at 10:01 AM Ismael Juma  wrote:
> >>>
>  The internal config was meant to be internal, right? That is, no
>  compatibility guarantees are offered? The current discussion implies
> we
> >>> are
>  considering it a public config.
> 
>  Ismael
> 
>  On Sat, Apr 4, 2020 at 9:31 AM Boyang Chen 
> >> wrote:
> 
> > For Gary's case, I think the internal config should be a sort of
> >> help,
>  and
> > not violating our public agreement.
> >
> > On Fri, Apr 3, 2020 at 7:44 PM Matthias J. Sax 
> >>> wrote:
> >
> >> I guess you would need to catch the exception and retry?
> >>
> >> It's a little unfortunate. Not sure if we could back-port the
> >>> internal
> >> producer config that we add in 2.6 for auto-downgrade to a 2.5 bug
> >>> fix
> >> release?
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 4/2/20 7:25 PM, Gary Russell wrote:
> >>> Thanks Mattias
> >>>
>  Hence, why do you want/need to switch to the newer overload that
>  only
> >>> works for 2.5+ brokers?
> >>>
> >>> So I can choose to use the producer per consumer thread Vs. the
> > producer
> >>> per group/topic/partition threading model for zombie fencing,
> >> based
>  on
> >> the
> >>> broker version.
> >>>
> >>> I don't have the same luxury as kafka streams (i.e. don't use
> >>> streams
> > 2.6
> >>> unless you have 2.5+ brokers).
> >>>
> >>> I add new features with e

Re: [DISCUSS] (KAFKA-9806) authorize cluster operation when creating internal topics

2020-04-07 Thread Paolo Moriello

Hi Colin,

Thanks for your interest in this. I agree with you, this change could break
compatibility. However, changing the source principal is non trivial in
this case. In fact, here the problem is not in the internal topic creation
- which succeeds - but in the two subsequent LeaderAndIsr and
UpdateMetadata requests.

When a consumer tries to consume for the first time, the creation of
internal topic completes, zk-nodes are filled with the necessary metadata,
and this triggers a ZkPartitionStateMachine (PartitionStateMachine.scala)
update which, in turn, makes the ControllerChannelManager
(ControllerChannelManager.scala) send LeaderAndIsr and UpdateMetadata
requests to the brokers; (I can be wrong, but I believe that this requests
are already being executed with broker principal). These requests fail
because we authorize the cluster operation there, so the __consumer_offsets
topic remains in a bad state.

Is there a reason to not authorize the operation for find coordinator
requests as well?

Thanks,
Paolo

On Mon, 6 Apr 2020 at 23:58, Colin McCabe  wrote:

> Hi Paolo,
>
> Thanks for finding this issue.
>
> Unfortunately, you certainly can't add a new permission requirement to an
> existing RPC without breaking compatibility.  So the current solution in
> the PR will not work.  However, you should be able to have the broker
> create the topic using its own principal rather than the caller's.
> Basically the equivalent of a doAs block (I forget how we do this exactly,
> but we do have some way of doing it).
>
> best,
> Colin
>
>
> On Mon, Apr 6, 2020, at 02:56, Paolo Moriello wrote:
> > Hello everybody,
> >
> > I've opened a Jira to fix a bug on creation of internal topics. This
> > happens when the topics are created under insufficient ACLs: eg.
> > __consumer_offset is created but subsequent updateMetadata and leaderIsr
> > requests fail; the topic is than in an inconsistent state and it is
> > impossible to consume.
> >
> > Jira: https://issues.apache.org/jira/browse/KAFKA-9806
> >
> > A simple fix to solve this problem is to authorize the cluster operation
> > before creating these topics. I've submitted a PR with the fix:
> > https://github.com/apache/kafka/pull/8415
> >
> > Please take a look and let me know if you have any feedback.
> > Thanks,
> > Paolo
> >
>

[jira] [Created] (KAFKA-9829) Kafka brokers are un-registered on Zookeeper node replacement

2020-04-07 Thread Pradeep (Jira)

Pradeep created KAFKA-9829:
--

 Summary: Kafka brokers are un-registered on Zookeeper node 
replacement
 Key: KAFKA-9829
 URL: https://issues.apache.org/jira/browse/KAFKA-9829
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.2.1
Reporter: Pradeep


We have a Kafka cluster with 3 nodes connected to a Zookeeper (3.4.14) cluster 
of 3 nodes in AWS. We make use of the auto-scaling group to provision nodes 
upon failures. We are seeing an issue where the Kafka brokers are getting 
un-registered when all the Zookeeper nodes are replaced one after the other. 
Every Zookeeper node is terminated from AWS console and we wait for a 
replacement node to be provisioned with Zookeeper initialized before 
terminating the other node.

On every Zookeeper node replacement, the /broker/ids path show all the Kafka 
broker ids in the cluster. But only on the final Zookeeper node replacement, 
the content in /broker/ids become empty.

We are seeing below logs in one of the Zookeeper nodes when all of the original 
nodes are replaced.

{{2020-03-26 20:29:20,303 [myid:3] - INFO 
[[SessionTracker:ZooKeeperServer@355|sessiontracker:ZooKeeperServer@355]] - 
Expiring session 0x10003b973b50016, timeout of 6000ms exceeded}}

{{2020-03-26 20:29:20,303 [myid:3] - INFO 
[[SessionTracker:ZooKeeperServer@355|sessiontracker:ZooKeeperServer@355]] - 
Expiring session 0x10003b973b5000e, timeout of 6000ms exceeded}}

{{2020-03-26 20:29:20,303 [myid:3] - INFO 
[[SessionTracker:ZooKeeperServer@355|sessiontracker:ZooKeeperServer@355]] - 
Expiring session 0x30003a126690002, timeout of 6000ms exceeded}}

{{2020-03-26 20:29:20,307 [myid:3] - DEBUG [CommitProcessor:3:DataTree@893] - 
Deleting ephemeral node /brokers/ids/1002 for session 0x10003b973b50016}}

{{2020-03-26 20:29:20,307 [myid:3] - DEBUG [CommitProcessor:3:DataTree@893] - 
Deleting ephemeral node /brokers/ids/1003 for session 0x10003b973b5000e}}

{{2020-03-26 20:29:20,307 [myid:3] - DEBUG [CommitProcessor:3:DataTree@893] - 
Deleting ephemeral node /controller for session 0x30003a126690002}}

{{2020-03-26 20:29:20,307 [myid:3] - DEBUG [CommitProcessor:3:DataTree@893] - 
Deleting ephemeral node /brokers/ids/1001 for session 0x30003a126690002}}

 

I am not sure if the issue is related to KAFKA-5473.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [jira] [Created] (KAFKA-9829) Kafka brokers are un-registered on Zookeeper node replacement

2020-04-07 Thread nitin agarwal

What value of zookeeper.connect is specified in server.properties of the
broker. If it is having IPs rather than hostname then it is expected
behaviour as all three Zookeeper nodes are destroyed and broker will not be
able to communicate with Zookeeper.

Thanks,
Nitin

On Tue, Apr 7, 2020 at 8:44 PM Pradeep (Jira)  wrote:

> Pradeep created KAFKA-9829:
> --
>
>  Summary: Kafka brokers are un-registered on Zookeeper node
> replacement
>  Key: KAFKA-9829
>  URL: https://issues.apache.org/jira/browse/KAFKA-9829
>  Project: Kafka
>   Issue Type: Bug
> Affects Versions: 0.10.2.1
> Reporter: Pradeep
>
>
> We have a Kafka cluster with 3 nodes connected to a Zookeeper (3.4.14)
> cluster of 3 nodes in AWS. We make use of the auto-scaling group to
> provision nodes upon failures. We are seeing an issue where the Kafka
> brokers are getting un-registered when all the Zookeeper nodes are replaced
> one after the other. Every Zookeeper node is terminated from AWS console
> and we wait for a replacement node to be provisioned with Zookeeper
> initialized before terminating the other node.
>
> On every Zookeeper node replacement, the /broker/ids path show all the
> Kafka broker ids in the cluster. But only on the final Zookeeper node
> replacement, the content in /broker/ids become empty.
>
> We are seeing below logs in one of the Zookeeper nodes when all of the
> original nodes are replaced.
>
> {{2020-03-26 20:29:20,303 [myid:3] - INFO
> [[SessionTracker:ZooKeeperServer@355|sessiontracker:ZooKeeperServer@355]]
> - Expiring session 0x10003b973b50016, timeout of 6000ms exceeded}}
>
> {{2020-03-26 20:29:20,303 [myid:3] - INFO
> [[SessionTracker:ZooKeeperServer@355|sessiontracker:ZooKeeperServer@355]]
> - Expiring session 0x10003b973b5000e, timeout of 6000ms exceeded}}
>
> {{2020-03-26 20:29:20,303 [myid:3] - INFO
> [[SessionTracker:ZooKeeperServer@355|sessiontracker:ZooKeeperServer@355]]
> - Expiring session 0x30003a126690002, timeout of 6000ms exceeded}}
>
> {{2020-03-26 20:29:20,307 [myid:3] - DEBUG [CommitProcessor:3:DataTree@893]
> - Deleting ephemeral node /brokers/ids/1002 for session 0x10003b973b50016}}
>
> {{2020-03-26 20:29:20,307 [myid:3] - DEBUG [CommitProcessor:3:DataTree@893]
> - Deleting ephemeral node /brokers/ids/1003 for session 0x10003b973b5000e}}
>
> {{2020-03-26 20:29:20,307 [myid:3] - DEBUG [CommitProcessor:3:DataTree@893]
> - Deleting ephemeral node /controller for session 0x30003a126690002}}
>
> {{2020-03-26 20:29:20,307 [myid:3] - DEBUG [CommitProcessor:3:DataTree@893]
> - Deleting ephemeral node /brokers/ids/1001 for session 0x30003a126690002}}
>
>
>
> I am not sure if the issue is related to KAFKA-5473.
>
>
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>

Re: [DISCUSS] (KAFKA-9806) authorize cluster operation when creating internal topics

2020-04-07 Thread Colin McCabe

On Tue, Apr 7, 2020, at 08:08, Paolo Moriello wrote:
> Hi Colin,
> 
> Thanks for your interest in this. I agree with you, this change could break
> compatibility. However, changing the source principal is non trivial in
> this case. In fact, here the problem is not in the internal topic creation
> - which succeeds - but in the two subsequent LeaderAndIsr and
> UpdateMetadata requests.
> 
> When a consumer tries to consume for the first time, the creation of
> internal topic completes, zk-nodes are filled with the necessary metadata,
> and this triggers a ZkPartitionStateMachine (PartitionStateMachine.scala)
> update which, in turn, makes the ControllerChannelManager
> (ControllerChannelManager.scala) send LeaderAndIsr and UpdateMetadata
> requests to the brokers; (I can be wrong, but I believe that this requests
> are already being executed with broker principal). These requests fail
> because we authorize the cluster operation there, so the __consumer_offsets
> topic remains in a bad state.

I might be misunderstanding something here, but it seems to me that if 
LeaderAndIsrRequest or UpdateMetadataRequest are failing with authorization 
errors, then there is a configuration problem on the cluster which doesn't have 
anything to do with the __consumer_offsets topic.

> 
> Is there a reason to not authorize the operation for find coordinator
> requests as well?

To be clear, we can't change the authorization for FindCoordinatorRequest.

best,
Colin

> 
> Thanks,
> Paolo
> 
> On Mon, 6 Apr 2020 at 23:58, Colin McCabe  wrote:
> 
> > Hi Paolo,
> >
> > Thanks for finding this issue.
> >
> > Unfortunately, you certainly can't add a new permission requirement to an
> > existing RPC without breaking compatibility.  So the current solution in
> > the PR will not work.  However, you should be able to have the broker
> > create the topic using its own principal rather than the caller's.
> > Basically the equivalent of a doAs block (I forget how we do this exactly,
> > but we do have some way of doing it).
> >
> > best,
> > Colin
> >
> >
> > On Mon, Apr 6, 2020, at 02:56, Paolo Moriello wrote:
> > > Hello everybody,
> > >
> > > I've opened a Jira to fix a bug on creation of internal topics. This
> > > happens when the topics are created under insufficient ACLs: eg.
> > > __consumer_offset is created but subsequent updateMetadata and leaderIsr
> > > requests fail; the topic is than in an inconsistent state and it is
> > > impossible to consume.
> > >
> > > Jira: https://issues.apache.org/jira/browse/KAFKA-9806
> > >
> > > A simple fix to solve this problem is to authorize the cluster operation
> > > before creating these topics. I've submitted a PR with the fix:
> > > https://github.com/apache/kafka/pull/8415
> > >
> > > Please take a look and let me know if you have any feedback.
> > > Thanks,
> > > Paolo
> > >
> >
>

Build failed in Jenkins: kafka-trunk-jdk11 #1330

2020-04-07 Thread Apache Jenkins Server

See 


Changes:

[github] MINOR: Pass one action per unique resource name in


--
[...truncated 3.01 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.str

[jira] [Created] (KAFKA-9830) DeadLetterQueueReporter leaks KafkaProducer instance

2020-04-07 Thread Greg Harris (Jira)

Greg Harris created KAFKA-9830:
--

 Summary: DeadLetterQueueReporter leaks KafkaProducer instance
 Key: KAFKA-9830
 URL: https://issues.apache.org/jira/browse/KAFKA-9830
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.0.0
Reporter: Greg Harris


The DeadLetterQueueReporter (introduced by KAFKA-6738) creates a KafkaProducer 
to report errors to Kafka, but does not clean up the producer, leaving many 
idle network threads open after tasks are stopped.

Reproduction steps:
Start a task that has a non-empty DLQ topic name
Stop the task
Observe the list of running threads

Expected result:
There is no thread related to the stopped task's DLQ left running

Actual result:
There is a thread named something like kafka-producer-network-thread | 
connector-dlq-producer-task-0" #1234 left running



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-trunk-jdk8 #4408

2020-04-07 Thread Apache Jenkins Server

See

Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter

2020-04-07 Thread Matthias J. Sax

> Would you mind to elaborate why we still need that

Sure.

For static memership, the session timeout it usually set quite high.
This make scaling in an application tricky: if you shut down one
instance, no rebalance would happen until `session.timeout.ms` hits.
This is specific to Kafka Streams, because when a Kafka Stream client is
closed, it does _not_ send a `LeaveGroupRequest`. Hence, the
corresponding partitions would not be processed for a long time and
thus, fall back.

Given that each instance will have a unique `instance.id` provided by
the user, we could allow users to remove the instance they want to
decommission from the consumer group without the need to wait for
`session.timeout.ms`.

Hence, it's not an application reset scenario for which one wants to
remove all members, but a scaling-in scenario. For dynamic membership,
this issue usually does not occur because the `session.timeout.ms` is
set to a fairly low value and a rebalance would happen quickly after an
instance is decommissioned.

Does this make sense?

As said before, we may or may not include this in this KIP. It's up to
you if you want to address it or not.


-Matthias



On 4/7/20 7:12 AM, feyman2009 wrote:
> Hi, Matthias
> Thanks a lot!
> So you do not plan so support removing a _single static_ member via 
> `StreamsResetter`? 
> => 
> Would you mind to elaborate why we still need that if we are able to 
> batch remove active members with adminClient?
> 
> Thanks! 
> 
> Feyman
>  --
> 发件人：Matthias J. Sax 
> 发送时间：2020年4月7日(星期二) 08:25
> 收件人：dev 
> 主　题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in 
> StreamsResetter
> 
> Overall LGTM.
> 
> +1 (binding)
> 
> So you do not plan so support removing a _single static_ member via
> `StreamsResetter`? We can of course still add this as a follow up but it
> might be nice to just add it to this KIP right away. Up to you if you
> want to include it or not.
> 
> 
> -Matthias
> 
> 
> 
> On 3/30/20 8:16 AM, feyman2009 wrote:
>> Hi, Boyang
>> Thanks a lot, that make sense, we should not expose member.id in the 
>> MemberToRemove anymore, I have fixed it in the KIP.
>>
>>
>> Feyman
>> --
>> 发件人：Boyang Chen 
>> 发送时间：2020年3月29日(星期日) 01:45
>> 收件人：dev ; feyman2009 
>> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in 
>> StreamsResetter
>>
>> Hey Feyman,
>>
>> thanks for the update. I assume we would rely entirely on the internal 
>> changes for `removeMemberFromGroup` by sending a DescribeGroup request 
>> first. With that in mind, I don't think we need to add member.id to 
>> MemberToRemove anymore, as it is only facing public where users will only 
>> configure group.instance.id? 
>> On Fri, Mar 27, 2020 at 5:04 PM feyman2009  
>> wrote:
>> Bump, can anyone kindly take a look at the updated KIP-571? Thanks!
>>
>>
>>  --
>>  发件人：feyman2009 
>>  发送时间：2020年3月23日(星期一) 08:51
>>  收件人：dev 
>>  主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in 
>> StreamsResetter
>>
>>  Hi, team
>>  I have updated the KIP-571 according to our latest discussion results, 
>> would you mind to take a look? Thanks!
>>
>>  Feyman
>>
>>
>>  --
>>  发件人：Boyang Chen 
>>  发送时间：2020年3月19日(星期四) 13:41
>>  收件人：dev ; feyman2009 
>>  主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in 
>> StreamsResetter
>>
>>  Thanks for the insight Feyman. I personally feel adding another admin 
>> client command is redundant, so picking option 1). The memberInfos struct is 
>> internal and just used for result reference purposes. I think it could still 
>> work even we overload with `removeAll` option, if that makes sense.
>>
>>  Boyang
>>  On Wed, Mar 18, 2020 at 8:51 PM feyman2009  
>> wrote:
>>  Hi, team
>>   Before going too far on the KIP update, I would like to hear your 
>> opinions about how we would change the interface of AdminClient, the two 
>> alternatives I could think of are:
>>   1) Extend adminClient.removeMembersFromConsumerGroup to support remove 
>> all
>>   As Guochang suggested, we could add some flag param in 
>> RemoveMembersFromConsumerGroupOptions to indicated the "remove all" logic.  
>>   2) Add a new API like 
>> adminClient.removeAllMembersFromConsumerGroup(groupId, options) 
>>
>>   I think 1) will be more compact from the API perspective, but looking 
>> at the code, I found that the if we are going to remove all, then the 
>> RemoveMembersFromConsumerGroupResult#memberInfos/memberResult()/all() should 
>> be changed accordingly, and they seem not that meaningful under the "remove 
>> all" scenario.
>>
>>   A minor thought about the adminClient.removeMembersFromConsumerGroup 
>> API is:
>>   Looking at some other deleteXX API

[jira] [Created] (KAFKA-9831) Failing test: EosIntegrationTest.shouldNotViolateEosIfOneTaskFailsWithState[exactly_once_beta]

2020-04-07 Thread John Roesler (Jira)

John Roesler created KAFKA-9831:
---

 Summary: Failing test: 
EosIntegrationTest.shouldNotViolateEosIfOneTaskFailsWithState[exactly_once_beta]
 Key: KAFKA-9831
 URL: https://issues.apache.org/jira/browse/KAFKA-9831
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: John Roesler
 Attachments: one.stdout.txt, two.stdout.txt

I've seen this fail twice in a row on the same build, but with different 
errors. Stacktraces follow; stdout is attached.

One:
{noformat}
java.lang.AssertionError: Did not receive all 40 records from topic 
singlePartitionOutputTopic within 6 ms
Expected: is a value equal to or greater than <40>
 but: <39> was less than <40>
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.lambda$waitUntilMinKeyValueRecordsReceived$1(IntegrationTestUtils.java:517)
at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:415)
at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:383)
at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:513)
at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:491)
at 
org.apache.kafka.streams.integration.EosIntegrationTest.readResult(EosIntegrationTest.java:766)
at 
org.apache.kafka.streams.integration.EosIntegrationTest.shouldNotViolateEosIfOneTaskFailsWithState(EosIntegrationTest.java:473)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at

Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter

2020-04-07 Thread Boyang Chen

Hey Feyman,

I think Matthias' suggestion is optional, and we could just use admin tool
to remove single static members as well.

Boyang

On Tue, Apr 7, 2020 at 11:00 AM Matthias J. Sax  wrote:

> > Would you mind to elaborate why we still need that
>
> Sure.
>
> For static memership, the session timeout it usually set quite high.
> This make scaling in an application tricky: if you shut down one
> instance, no rebalance would happen until `session.timeout.ms` hits.
> This is specific to Kafka Streams, because when a Kafka Stream client is
> closed, it does _not_ send a `LeaveGroupRequest`. Hence, the
> corresponding partitions would not be processed for a long time and
> thus, fall back.
>
> Given that each instance will have a unique `instance.id` provided by
> the user, we could allow users to remove the instance they want to
> decommission from the consumer group without the need to wait for
> `session.timeout.ms`.
>
> Hence, it's not an application reset scenario for which one wants to
> remove all members, but a scaling-in scenario. For dynamic membership,
> this issue usually does not occur because the `session.timeout.ms` is
> set to a fairly low value and a rebalance would happen quickly after an
> instance is decommissioned.
>
> Does this make sense?
>
> As said before, we may or may not include this in this KIP. It's up to
> you if you want to address it or not.
>
>
> -Matthias
>
>
>
> On 4/7/20 7:12 AM, feyman2009 wrote:
> > Hi, Matthias
> > Thanks a lot!
> > So you do not plan so support removing a _single static_ member via
> `StreamsResetter`?
> > =>
> > Would you mind to elaborate why we still need that if we are
> able to batch remove active members with adminClient?
> >
> > Thanks!
> >
> > Feyman
> >  --
> > 发件人：Matthias J. Sax 
> > 发送时间：2020年4月7日(星期二) 08:25
> > 收件人：dev 
> > 主 题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> in StreamsResetter
> >
> > Overall LGTM.
> >
> > +1 (binding)
> >
> > So you do not plan so support removing a _single static_ member via
> > `StreamsResetter`? We can of course still add this as a follow up but it
> > might be nice to just add it to this KIP right away. Up to you if you
> > want to include it or not.
> >
> >
> > -Matthias
> >
> >
> >
> > On 3/30/20 8:16 AM, feyman2009 wrote:
> >> Hi, Boyang
> >> Thanks a lot, that make sense, we should not expose member.id in
> the MemberToRemove anymore, I have fixed it in the KIP.
> >>
> >>
> >> Feyman
> >> --
> >> 发件人：Boyang Chen 
> >> 发送时间：2020年3月29日(星期日) 01:45
> >> 收件人：dev ; feyman2009 
> >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Hey Feyman,
> >>
> >> thanks for the update. I assume we would rely entirely on the internal
> changes for `removeMemberFromGroup` by sending a DescribeGroup request
> first. With that in mind, I don't think we need to add member.id to
> MemberToRemove anymore, as it is only facing public where users will only
> configure group.instance.id?
> >> On Fri, Mar 27, 2020 at 5:04 PM feyman2009
>  wrote:
> >> Bump, can anyone kindly take a look at the updated KIP-571? Thanks!
> >>
> >>
> >>  --
> >>  发件人：feyman2009 
> >>  发送时间：2020年3月23日(星期一) 08:51
> >>  收件人：dev 
> >>  主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >>  Hi, team
> >>  I have updated the KIP-571 according to our latest discussion
> results, would you mind to take a look? Thanks!
> >>
> >>  Feyman
> >>
> >>
> >>  --
> >>  发件人：Boyang Chen 
> >>  发送时间：2020年3月19日(星期四) 13:41
> >>  收件人：dev ; feyman2009 
> >>  主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >>  Thanks for the insight Feyman. I personally feel adding another admin
> client command is redundant, so picking option 1). The memberInfos struct
> is internal and just used for result reference purposes. I think it could
> still work even we overload with `removeAll` option, if that makes sense.
> >>
> >>  Boyang
> >>  On Wed, Mar 18, 2020 at 8:51 PM feyman2009
>  wrote:
> >>  Hi, team
> >>   Before going too far on the KIP update, I would like to hear your
> opinions about how we would change the interface of AdminClient, the two
> alternatives I could think of are:
> >>   1) Extend adminClient.removeMembersFromConsumerGroup to support
> remove all
> >>   As Guochang suggested, we could add some flag param in
> RemoveMembersFromConsumerGroupOptions to indicated the "remove all" logic.
> >>   2) Add a new API like
> adminClient.removeAllMembersFromConsumerGroup(groupId, options)
> >>
> >>   I think 1) will be more compact from the API perspective, but
> looking at the code, I found that the if we are

[DISCUSS] KIP-589 Add API to Update Replica State in Controller

2020-04-07 Thread David Arthur

Hey everyone,

I'd like to start the discussion for KIP-589, part of the KIP-500 effort

https://cwiki.apache.org/confluence/display/KAFKA/KIP-589+Add+API+to+update+Replica+state+in+Controller

This is a proposal to use a new RPC instead of ZooKeeper for notifying the
controller of an offline replica. Please give a read and let me know your
thoughts.

Thanks!
David

[jira] [Created] (KAFKA-9832) Extend EOS system tests for EOS-beta

2020-04-07 Thread Matthias J. Sax (Jira)

Matthias J. Sax created KAFKA-9832:
--

 Summary: Extend EOS system tests for EOS-beta
 Key: KAFKA-9832
 URL: https://issues.apache.org/jira/browse/KAFKA-9832
 Project: Kafka
  Issue Type: Sub-task
Reporter: Matthias J. Sax






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-trunk-jdk11 #1331

2020-04-07 Thread Apache Jenkins Server

See

Build failed in Jenkins: kafka-trunk-jdk11 #1332

2020-04-07 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-8107; Flaky Test


--
[...truncated 3.01 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apac

回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter

2020-04-07 Thread feyman2009

Hi, Boyang&Matthias
I think Matthias's proposal makes sense, but we can use the admin tool for 
this scenario as Boyang mentioned or follow up later, so I prefer to keep this 
KIP unchanged to minimize the scope.
Calling for vote ~

Thanks!
Feyman

--
发件人：Boyang Chen 
发送时间：2020年4月8日(星期三) 02:15
收件人：dev 
主　题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in 
StreamsResetter

Hey Feyman,

I think Matthias' suggestion is optional, and we could just use admin tool
to remove single static members as well.

Boyang

On Tue, Apr 7, 2020 at 11:00 AM Matthias J. Sax  wrote:

> > Would you mind to elaborate why we still need that
>
> Sure.
>
> For static memership, the session timeout it usually set quite high.
> This make scaling in an application tricky: if you shut down one
> instance, no rebalance would happen until `session.timeout.ms` hits.
> This is specific to Kafka Streams, because when a Kafka Stream client is
> closed, it does _not_ send a `LeaveGroupRequest`. Hence, the
> corresponding partitions would not be processed for a long time and
> thus, fall back.
>
> Given that each instance will have a unique `instance.id` provided by
> the user, we could allow users to remove the instance they want to
> decommission from the consumer group without the need to wait for
> `session.timeout.ms`.
>
> Hence, it's not an application reset scenario for which one wants to
> remove all members, but a scaling-in scenario. For dynamic membership,
> this issue usually does not occur because the `session.timeout.ms` is
> set to a fairly low value and a rebalance would happen quickly after an
> instance is decommissioned.
>
> Does this make sense?
>
> As said before, we may or may not include this in this KIP. It's up to
> you if you want to address it or not.
>
>
> -Matthias
>
>
>
> On 4/7/20 7:12 AM, feyman2009 wrote:
> > Hi, Matthias
> > Thanks a lot!
> > So you do not plan so support removing a _single static_ member via
> `StreamsResetter`?
> > =>
> > Would you mind to elaborate why we still need that if we are
> able to batch remove active members with adminClient?
> >
> > Thanks!
> >
> > Feyman
> >  --
> > 发件人：Matthias J. Sax 
> > 发送时间：2020年4月7日(星期二) 08:25
> > 收件人：dev 
> > 主 题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> in StreamsResetter
> >
> > Overall LGTM.
> >
> > +1 (binding)
> >
> > So you do not plan so support removing a _single static_ member via
> > `StreamsResetter`? We can of course still add this as a follow up but it
> > might be nice to just add it to this KIP right away. Up to you if you
> > want to include it or not.
> >
> >
> > -Matthias
> >
> >
> >
> > On 3/30/20 8:16 AM, feyman2009 wrote:
> >> Hi, Boyang
> >> Thanks a lot, that make sense, we should not expose member.id in
> the MemberToRemove anymore, I have fixed it in the KIP.
> >>
> >>
> >> Feyman
> >> --
> >> 发件人：Boyang Chen 
> >> 发送时间：2020年3月29日(星期日) 01:45
> >> 收件人：dev ; feyman2009 
> >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >> Hey Feyman,
> >>
> >> thanks for the update. I assume we would rely entirely on the internal
> changes for `removeMemberFromGroup` by sending a DescribeGroup request
> first. With that in mind, I don't think we need to add member.id to
> MemberToRemove anymore, as it is only facing public where users will only
> configure group.instance.id?
> >> On Fri, Mar 27, 2020 at 5:04 PM feyman2009
>  wrote:
> >> Bump, can anyone kindly take a look at the updated KIP-571? Thanks!
> >>
> >>
> >>  --
> >>  发件人：feyman2009 
> >>  发送时间：2020年3月23日(星期一) 08:51
> >>  收件人：dev 
> >>  主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >>  Hi, team
> >>  I have updated the KIP-571 according to our latest discussion
> results, would you mind to take a look? Thanks!
> >>
> >>  Feyman
> >>
> >>
> >>  --
> >>  发件人：Boyang Chen 
> >>  发送时间：2020年3月19日(星期四) 13:41
> >>  收件人：dev ; feyman2009 
> >>  主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> StreamsResetter
> >>
> >>  Thanks for the insight Feyman. I personally feel adding another admin
> client command is redundant, so picking option 1). The memberInfos struct
> is internal and just used for result reference purposes. I think it could
> still work even we overload with `removeAll` option, if that makes sense.
> >>
> >>  Boyang
> >>  On Wed, Mar 18, 2020 at 8:51 PM feyman2009
>  wrote:
> >>  Hi, team
> >>   Before going too far on the KIP update, I would like to hear your
> opinions about how we would change the interface of AdminClient, the two
> alternatives I could think

Build failed in Jenkins: kafka-trunk-jdk8 #4410

2020-04-07 Thread Apache Jenkins Server

See 


Changes:

[github] KAFKA-8107; Flaky Test


--
[...truncated 5.99 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTe

Issue with stale consumer groups

2020-04-07 Thread Suraj D

Hi Dev,

At my company we run Kafka version 0.10.1.1 in production. I noticed a
strange behavior around stale consumers and wanted some help. So there are
some consumer group ids which are no longer being used, but kafka seems to
think that the group is rebalancing.

root@kafka32:/opt/kafka/bin# ./kafka-consumer-groups.sh --describe --group
consumer-abcd-defg-ads-9ffbcd955-8fdwn --bootstrap-server kafka1:9092
Consumer group `consumer-abcd-defg-ads-9ffbcd955-8fdwn` is rebalancing.

I see a message like the above. Since the kafka version I am on stores
consumer groups in _consumer_offsets (and not in ZK) I am not able to
delete these stale consumer groups.

1) Is this something I should be worried about? We might have ephemeral
consumer groups in our system and there could be stale consumer groups. I'm
worried that this might cause _consumer_offsets to blow up somehow? I saw
some bugs reported about this online
https://github.com/apache/kafka/pull/1622, so Im not sure if this actually
an issue or just a problem with ./kafka-consumer-groups.sh.

2) Is there a way I can cleanup these consumer groups?

Any help is appreciated!

Regards,
Suraj

[jira] [Created] (KAFKA-9833) Allow environment variable to set output directory of Kafka Connect logs

2020-04-07 Thread Brandon Moe (Jira)

Brandon Moe created KAFKA-9833:
--

 Summary: Allow environment variable to set output directory of 
Kafka Connect logs
 Key: KAFKA-9833
 URL: https://issues.apache.org/jira/browse/KAFKA-9833
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: Brandon Moe


Currently in Kafka Connect I believe that if you want to change what file logs 
are written to you need to include a new log4j .properties file and then 
override the *KAFKA_LOG4J_OPTS* environment variable. This can be [witnessed 
here|[https://github.com/apache/kafka/blob/trunk/bin/connect-distributed.sh]]. 
I believe there is a valid use case for allowing a separate environment 
variable such as *CONNECT_LOG4J_DIR* or similar to allow a user simply to chose 
where the logs are written. This makes sense to me in a case where the default 
.properties file works great but we simply want to change the output location.  
An example could be if we are deploying to Kubernetes or similar and want to 
write logs to a persistent volume. With an environment variable dedicated to 
this use it saves busy work and extra steps while providing the desired 
functionality succinctly. In the case when the variable is not set we simply 
revert to the default settings. I do not believe I am alone on wanting this use 
case as I have come across several other Stack Overflow and similar posts. 

 

I believe this change would be pretty simple to implement and I would like to 
take a shot at it if this seems legitimate, would love some feedback

 

Best,

Brandon



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9833) Kafka Connect: Allow environment variable to set output directory of Kafka Connect logs

2020-04-07 Thread Brandon Moe (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Moe resolved KAFKA-9833.

Resolution: Abandoned

> Kafka Connect: Allow environment variable to set output directory of Kafka 
> Connect logs
> ---
>
> Key: KAFKA-9833
> URL: https://issues.apache.org/jira/browse/KAFKA-9833
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Brandon Moe
>Priority: Minor
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Currently in Kafka Connect I believe that if you want to change what file 
> logs are written to you need to include a new log4j .properties file and then 
> override the *KAFKA_LOG4J_OPTS* environment variable. This can be seen here 
> [https://github.com/apache/kafka/blob/trunk/bin/connect-distributed.sh] I 
> believe there is a valid use case for allowing a separate environment 
> variable such as *CONNECT_LOG4J_DIR* or similar to allow a user simply to 
> chose where the logs are written. This makes sense to me in a case where the 
> default .properties file works great but we simply want to change the output 
> location.  An example could be if we are deploying to Kubernetes or similar 
> and want to write logs to a persistent volume. With an environment variable 
> dedicated to this use it saves busy work and extra steps while providing the 
> desired functionality succinctly. In the case when the variable is not set we 
> simply revert to the default settings. I do not believe I am alone on wanting 
> this use case as I have come across several other Stack Overflow and similar 
> posts. 
>  
> I believe this change would be pretty simple to implement and I would like to 
> take a shot at it if this seems legitimate, would love some feedback
>  
> Best,
> Brandon



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] 2.5.0 RC3

2020-04-07 Thread David Arthur

Hello Kafka users, developers and client-developers,

This is the forth candidate for release of Apache Kafka 2.5.0.

* TLS 1.3 support (1.2 is now the default)
* Co-groups for Kafka Streams
* Incremental rebalance for Kafka Consumer
* New metrics for better operational insight
* Upgrade Zookeeper to 3.5.7
* Deprecate support for Scala 2.11

Release notes for the 2.5.0 release:
https://home.apache.org/~davidarthur/kafka-2.5.0-rc3/RELEASE_NOTES.html

*** Please download, test and vote by Friday April 10th 5pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~davidarthur/kafka-2.5.0-rc3/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~davidarthur/kafka-2.5.0-rc3/javadoc/

* Tag to be voted upon (off 2.5 branch) is the 2.5.0 tag:
https://github.com/apache/kafka/releases/tag/2.5.0-rc3

* Documentation:
https://kafka.apache.org/25/documentation.html

* Protocol:
https://kafka.apache.org/25/protocol.html

Successful Jenkins builds to follow

Thanks!
David

Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter

2020-04-07 Thread Boyang Chen

You should already get enough votes if I'm counting correctly (Guozhang,
John, Matthias)

On Tue, Apr 7, 2020 at 6:59 PM feyman2009 
wrote:

> Hi, Boyang&Matthias
> I think Matthias's proposal makes sense, but we can use the admin tool
> for this scenario as Boyang mentioned or follow up later, so I prefer to
> keep this KIP unchanged to minimize the scope.
> Calling for vote ~
>
> Thanks!
> Feyman
>
> --
> 发件人：Boyang Chen 
> 发送时间：2020年4月8日(星期三) 02:15
> 收件人：dev 
> 主 题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> in StreamsResetter
>
> Hey Feyman,
>
> I think Matthias' suggestion is optional, and we could just use admin tool
> to remove single static members as well.
>
> Boyang
>
> On Tue, Apr 7, 2020 at 11:00 AM Matthias J. Sax  wrote:
>
> > > Would you mind to elaborate why we still need that
> >
> > Sure.
> >
> > For static memership, the session timeout it usually set quite high.
> > This make scaling in an application tricky: if you shut down one
> > instance, no rebalance would happen until `session.timeout.ms` hits.
> > This is specific to Kafka Streams, because when a Kafka Stream client is
> > closed, it does _not_ send a `LeaveGroupRequest`. Hence, the
> > corresponding partitions would not be processed for a long time and
> > thus, fall back.
> >
> > Given that each instance will have a unique `instance.id` provided by
> > the user, we could allow users to remove the instance they want to
> > decommission from the consumer group without the need to wait for
> > `session.timeout.ms`.
> >
> > Hence, it's not an application reset scenario for which one wants to
> > remove all members, but a scaling-in scenario. For dynamic membership,
> > this issue usually does not occur because the `session.timeout.ms` is
> > set to a fairly low value and a rebalance would happen quickly after an
> > instance is decommissioned.
> >
> > Does this make sense?
> >
> > As said before, we may or may not include this in this KIP. It's up to
> > you if you want to address it or not.
> >
> >
> > -Matthias
> >
> >
> >
> > On 4/7/20 7:12 AM, feyman2009 wrote:
> > > Hi, Matthias
> > > Thanks a lot!
> > > So you do not plan so support removing a _single static_ member via
> > `StreamsResetter`?
> > > =>
> > > Would you mind to elaborate why we still need that if we are
> > able to batch remove active members with adminClient?
> > >
> > > Thanks!
> > >
> > > Feyman
> > >  --
> > > 发件人：Matthias J. Sax 
> > > 发送时间：2020年4月7日(星期二) 08:25
> > > 收件人：dev 
> > > 主 题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> > in StreamsResetter
> > >
> > > Overall LGTM.
> > >
> > > +1 (binding)
> > >
> > > So you do not plan so support removing a _single static_ member via
> > > `StreamsResetter`? We can of course still add this as a follow up but
> it
> > > might be nice to just add it to this KIP right away. Up to you if you
> > > want to include it or not.
> > >
> > >
> > > -Matthias
> > >
> > >
> > >
> > > On 3/30/20 8:16 AM, feyman2009 wrote:
> > >> Hi, Boyang
> > >> Thanks a lot, that make sense, we should not expose member.id in
> > the MemberToRemove anymore, I have fixed it in the KIP.
> > >>
> > >>
> > >> Feyman
> > >> --
> > >> 发件人：Boyang Chen 
> > >> 发送时间：2020年3月29日(星期日) 01:45
> > >> 收件人：dev ; feyman2009 
> > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> > StreamsResetter
> > >>
> > >> Hey Feyman,
> > >>
> > >> thanks for the update. I assume we would rely entirely on the internal
> > changes for `removeMemberFromGroup` by sending a DescribeGroup request
> > first. With that in mind, I don't think we need to add member.id to
> > MemberToRemove anymore, as it is only facing public where users will only
> > configure group.instance.id?
> > >> On Fri, Mar 27, 2020 at 5:04 PM feyman2009
> >  wrote:
> > >> Bump, can anyone kindly take a look at the updated KIP-571? Thanks!
> > >>
> > >>
> > >>  --
> > >>  发件人：feyman2009 
> > >>  发送时间：2020年3月23日(星期一) 08:51
> > >>  收件人：dev 
> > >>  主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
> > StreamsResetter
> > >>
> > >>  Hi, team
> > >>  I have updated the KIP-571 according to our latest discussion
> > results, would you mind to take a look? Thanks!
> > >>
> > >>  Feyman
> > >>
> > >>
> > >>  --
> > >>  发件人：Boyang Chen 
> > >>  发送时间：2020年3月19日(星期四) 13:41
> > >>  收件人：dev ; feyman2009 
> > >>  主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members
> in
> > StreamsResetter
> > >>
> > >>  Thanks for the insight Feyman. I personally feel adding another admin
> > client command is redundant, so picking option 1). The memberInfos struct
> > is in

Jenkins build is back to normal : kafka-2.5-jdk8 #90

2020-04-07 Thread Apache Jenkins Server

See

[jira] [Created] (KAFKA-9834) Add config to set ZSTD compresson level

2020-04-07 Thread jiamei xie (Jira)

jiamei xie created KAFKA-9834:
-

 Summary: Add config to set ZSTD compresson level
 Key: KAFKA-9834
 URL: https://issues.apache.org/jira/browse/KAFKA-9834
 Project: Kafka
  Issue Type: Bug
Reporter: jiamei xie
Assignee: jiamei xie


It seems kafka use zstd default compression level 3 and doesn't have support 
for setting zstd compression level.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

回复：回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter

2020-04-07 Thread feyman2009

Hi Boyang,
Thanks for reminding me of that!
I'm not sure about the convention, I thought it would need to re-collect 
votes if the KIP has changed~
Let's leave the vote thread here for 2 days, if no objection, I will take 
it as approved and update the PR accordingly.

Thanks!
Feyman



--
发件人：Boyang Chen 
发送时间：2020年4月8日(星期三) 12:42
收件人：dev ; feyman2009 
主　题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in 
StreamsResetter

You should already get enough votes if I'm counting correctly (Guozhang, John, 
Matthias)
On Tue, Apr 7, 2020 at 6:59 PM feyman2009  wrote:
Hi, Boyang&Matthias
 I think Matthias's proposal makes sense, but we can use the admin tool for 
this scenario as Boyang mentioned or follow up later, so I prefer to keep this 
KIP unchanged to minimize the scope.
 Calling for vote ~

 Thanks!
 Feyman

 --
 发件人：Boyang Chen 
 发送时间：2020年4月8日(星期三) 02:15
 收件人：dev 
 主　题：Re: 回复：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in 
StreamsResetter

 Hey Feyman,

 I think Matthias' suggestion is optional, and we could just use admin tool
 to remove single static members as well.

 Boyang

 On Tue, Apr 7, 2020 at 11:00 AM Matthias J. Sax  wrote:

 > > Would you mind to elaborate why we still need that
 >
 > Sure.
 >
 > For static memership, the session timeout it usually set quite high.
 > This make scaling in an application tricky: if you shut down one
 > instance, no rebalance would happen until `session.timeout.ms` hits.
 > This is specific to Kafka Streams, because when a Kafka Stream client is
 > closed, it does _not_ send a `LeaveGroupRequest`. Hence, the
 > corresponding partitions would not be processed for a long time and
 > thus, fall back.
 >
 > Given that each instance will have a unique `instance.id` provided by
 > the user, we could allow users to remove the instance they want to
 > decommission from the consumer group without the need to wait for
 > `session.timeout.ms`.
 >
 > Hence, it's not an application reset scenario for which one wants to
 > remove all members, but a scaling-in scenario. For dynamic membership,
 > this issue usually does not occur because the `session.timeout.ms` is
 > set to a fairly low value and a rebalance would happen quickly after an
 > instance is decommissioned.
 >
 > Does this make sense?
 >
 > As said before, we may or may not include this in this KIP. It's up to
 > you if you want to address it or not.
 >
 >
 > -Matthias
 >
 >
 >
 > On 4/7/20 7:12 AM, feyman2009 wrote:
 > > Hi, Matthias
 > > Thanks a lot!
 > > So you do not plan so support removing a _single static_ member via
 > `StreamsResetter`?
 > > =>
 > > Would you mind to elaborate why we still need that if we are
 > able to batch remove active members with adminClient?
 > >
 > > Thanks!
 > >
 > > Feyman
 > >  --
 > > 发件人：Matthias J. Sax 
 > > 发送时间：2020年4月7日(星期二) 08:25
 > > 收件人：dev 
 > > 主 题：Re: 回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members
 > in StreamsResetter
 > >
 > > Overall LGTM.
 > >
 > > +1 (binding)
 > >
 > > So you do not plan so support removing a _single static_ member via
 > > `StreamsResetter`? We can of course still add this as a follow up but it
 > > might be nice to just add it to this KIP right away. Up to you if you
 > > want to include it or not.
 > >
 > >
 > > -Matthias
 > >
 > >
 > >
 > > On 3/30/20 8:16 AM, feyman2009 wrote:
 > >> Hi, Boyang
 > >> Thanks a lot, that make sense, we should not expose member.id in
 > the MemberToRemove anymore, I have fixed it in the KIP.
 > >>
 > >>
 > >> Feyman
 > >> --
 > >> 发件人：Boyang Chen 
 > >> 发送时间：2020年3月29日(星期日) 01:45
 > >> 收件人：dev ; feyman2009 
 > >> 主 题：Re: 回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
 > StreamsResetter
 > >>
 > >> Hey Feyman,
 > >>
 > >> thanks for the update. I assume we would rely entirely on the internal
 > changes for `removeMemberFromGroup` by sending a DescribeGroup request
 > first. With that in mind, I don't think we need to add member.id to
 > MemberToRemove anymore, as it is only facing public where users will only
 > configure group.instance.id?
 > >> On Fri, Mar 27, 2020 at 5:04 PM feyman2009
 >  wrote:
 > >> Bump, can anyone kindly take a look at the updated KIP-571? Thanks!
 > >>
 > >>
 > >>  --
 > >>  发件人：feyman2009 
 > >>  发送时间：2020年3月23日(星期一) 08:51
 > >>  收件人：dev 
 > >>  主 题：回复：回复：回复：回复：[Vote] KIP-571: Add option to force remove members in
 > StreamsResetter
 > >>
 > >>  Hi, team
 > >>  I have updated the KIP-571 according to our latest discussion
 > results, would you mind to take a look? Thanks!
 > >>
 > >>  Feyman
 > >>
 > >>
 > >>  ---

Build failed in Jenkins: kafka-trunk-jdk8 #4411

2020-04-07 Thread Apache Jenkins Server

See 


Changes:

[wangguoz] KAFKA-9801: Still trigger rebalance when static member joins in


--
[...truncated 1.62 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

> Task :streams:upgrade-system-tests-0100:com

Build failed in Jenkins: kafka-trunk-jdk11 #1333

2020-04-07 Thread Apache Jenkins Server

See 


Changes:

[wangguoz] KAFKA-9801: Still trigger rebalance when static member joins in


--
[...truncated 1.62 MB...]

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTime

[jira] [Created] (KAFKA-9835) Race condition with concurrent write allows reads above high watermark

2020-04-07 Thread Jason Gustafson (Jira)

Jason Gustafson created KAFKA-9835:
--

 Summary: Race condition with concurrent write allows reads above 
high watermark
 Key: KAFKA-9835
 URL: https://issues.apache.org/jira/browse/KAFKA-9835
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson
Assignee: Jason Gustafson


Kafka's log implementation serializes all writes using a lock, but allows 
multiple concurrent reads while that lock is held. The `FileRecords` class 
contains the core implementation. Reads to the log create logical slices of 
`FileRecords` which are then passed to the network layer for sending. An 
abridged version of the implementation of `slice` is provided below:

{code}
public FileRecords slice(int position, int size) throws IOException {
int end = this.start + position + size;
// handle integer overflow or if end is beyond the end of the file
if (end < 0 || end >= start + sizeInBytes())
end = start + sizeInBytes();
return new FileRecords(file, channel, this.start + position, end, true);
}
{code}

The `size` parameter here is typically derived from the fetch size, but is 
upper-bounded with respect to the high watermark. The two calls to 
`sizeInBytes` here are problematic because the size of the file may change in 
between them. Specifically a concurrent write may increase the size of the file 
after the first call to `sizeInBytes` but before the second one. In the worst 
case, when `size` defines the limit of the high watermark, this can lead to a 
slice containing uncommitted data.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

40 matches

Mail list logo