[DISCUSS] 2.2.1 Bug Fix Release

[Vahid Hashemian]

Hi all,

I'd like to volunteer for the release manager of the 2.2.1 bug fix release.
Kafka 2.2.0 was released on March 22, 2019.

At this point, there are 29 resolved JIRA issues scheduled for inclusion in
2.2.1:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%202.2.1

The release plan is documented here:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.2.1

Thanks!
--Vahid

[jira] [Resolved] (KAFKA-8237) Untangle TopicDeletionManager and add test cases

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8237.

Resolution: Fixed

> Untangle TopicDeletionManager and add test cases
> 
>
> Key: KAFKA-8237
> URL: https://issues.apache.org/jira/browse/KAFKA-8237
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
>
> There are a few circular dependencies involving `TopicDeletionManager`. For 
> example, both `PartitionStateMachine` and `ReplicaStateMachine` depend on 
> `TopicDeletionManager` while it also depends on them. This makes testing 
> difficult and so there are no unit tests. We should fix this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-460: Admin Leader Election RPC

[Jose Armando Garcia Sancio]

Thanks for the reply. Comments below.

On Wed, Apr 24, 2019 at 6:07 PM Colin McCabe  wrote:

> Hi Jose,
>
> Thanks for the KIP, looks valuable.
>
> If I use a PreferredLeaderElection RPC to specifically request an unclean
> leader election, will this take effect even if unclean leader elections are
> disabled on the topic involved?  I assume that the answer is yes, but it
> would be good to clarify this in the KIP.
>

Yes. One of the motivation for this change is to allow the user to attempt
unclean leader election without having to change the topic configuration. I
will update the motivation and design section.

What ACLs will be required to perform this action?  WRITE on the topic
> resource?  Or ALTER on KafkaCluster?  Or perhaps ALTER on the topic would
> be most appropriate, since we probably don't want ordinary producers
> triggering unclean leader elections.
>

I am not sure. Let me investigate what the current RPC requires and get
back to you. This is not a new RPC. We are instead updating an existing RPC
that already performs authorization. The RPC has API key 32.

What's the rationale for using an int8 rather than just having a boolean
> that is either true or false for "unclean"?  We only have two values now,
> and it seems like we could modify the RPC schema in the future if needed.
> Or is the intention to add more flags later?
>

There are two reason:

   1. The controller supports 4 (5 technically) different election
   algorithms. We are only exposing "preferred" and "unclean" through the
   admin client because we only have use cases for those two election types.
   It is possible that in the future we may want to support more algorithms.
   This would allow us to make that change easier.
   2. I believe that an enum is more descriptive than a boolean flag as it
   is not a matter of "unclean" vs "clean" or "preferred" vs "non-preferred".
  1. Preferred means that the controller will attempt to elect only the
  fist replica describe in the partition assignment if it is
online and it is
  in-sync.
  2. Unclean means that the controller will attempt to elect the first
  in-sync and alive replica given the order of the partition assignment. If
  this is not satisfied it will attempt to elect the first replica in the
  assignment that is alive.

Build failed in Jenkins: kafka-2.1-jdk8 #172

[Apache Jenkins Server]

See 


Changes:

[bill] KAFKA-8227 DOCS Fixed missing links duality of streams tables (#6625)

--
[...truncated 919.08 KB...]

kafka.log.ProducerStateManagerTest > 
testSequenceNotValidatedForGroupMetadataTopic PASSED

kafka.log.ProducerStateManagerTest > testLastStableOffsetCompletedTxn STARTED

kafka.log.ProducerStateManagerTest > testLastStableOffsetCompletedTxn PASSED

kafka.log.ProducerStateManagerTest > 
testLoadFromSnapshotRemovesNonRetainedProducers STARTED

kafka.log.ProducerStateManagerTest > 
testLoadFromSnapshotRemovesNonRetainedProducers PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffset STARTED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffset PASSED

kafka.log.ProducerStateManagerTest > testTxnFirstOffsetMetadataCached STARTED

kafka.log.ProducerStateManagerTest > testTxnFirstOffsetMetadataCached PASSED

kafka.log.ProducerStateManagerTest > testCoordinatorFencedAfterReload STARTED

kafka.log.ProducerStateManagerTest > testCoordinatorFencedAfterReload PASSED

kafka.log.ProducerStateManagerTest > testControlRecordBumpsEpoch STARTED

kafka.log.ProducerStateManagerTest > testControlRecordBumpsEpoch PASSED

kafka.log.ProducerStateManagerTest > 
testAcceptAppendWithoutProducerStateOnReplica STARTED

kafka.log.ProducerStateManagerTest > 
testAcceptAppendWithoutProducerStateOnReplica PASSED

kafka.log.ProducerStateManagerTest > testLoadFromCorruptSnapshotFile STARTED

kafka.log.ProducerStateManagerTest > testLoadFromCorruptSnapshotFile PASSED

kafka.log.ProducerStateManagerTest > testProducerSequenceWrapAround STARTED

kafka.log.ProducerStateManagerTest > testProducerSequenceWrapAround PASSED

kafka.log.ProducerStateManagerTest > testPidExpirationTimeout STARTED

kafka.log.ProducerStateManagerTest > testPidExpirationTimeout PASSED

kafka.log.ProducerStateManagerTest > testAcceptAppendWithSequenceGapsOnReplica 
STARTED

kafka.log.ProducerStateManagerTest > testAcceptAppendWithSequenceGapsOnReplica 
PASSED

kafka.log.ProducerStateManagerTest > testAppendTxnMarkerWithNoProducerState 
STARTED

kafka.log.ProducerStateManagerTest > testAppendTxnMarkerWithNoProducerState 
PASSED

kafka.log.ProducerStateManagerTest > testOldEpochForControlRecord STARTED

kafka.log.ProducerStateManagerTest > testOldEpochForControlRecord PASSED

kafka.log.ProducerStateManagerTest > 
testTruncateAndReloadRemovesOutOfRangeSnapshots STARTED

kafka.log.ProducerStateManagerTest > 
testTruncateAndReloadRemovesOutOfRangeSnapshots PASSED

kafka.log.ProducerStateManagerTest > testStartOffset STARTED

kafka.log.ProducerStateManagerTest > testStartOffset PASSED

kafka.log.ProducerStateManagerTest > testProducerSequenceInvalidWrapAround 
STARTED

kafka.log.ProducerStateManagerTest > testProducerSequenceInvalidWrapAround 
PASSED

kafka.log.ProducerStateManagerTest > testTruncateHead STARTED

kafka.log.ProducerStateManagerTest > testTruncateHead PASSED

kafka.log.ProducerStateManagerTest > 
testNonTransactionalAppendWithOngoingTransaction STARTED

kafka.log.ProducerStateManagerTest > 
testNonTransactionalAppendWithOngoingTransaction PASSED

kafka.log.ProducerStateManagerTest > testSkipSnapshotIfOffsetUnchanged STARTED

kafka.log.ProducerStateManagerTest > testSkipSnapshotIfOffsetUnchanged PASSED

kafka.log.LogCleanerLagIntegrationTest > cleanerTest[0] STARTED

kafka.log.LogCleanerLagIntegrationTest > cleanerTest[0] PASSED

kafka.log.LogCleanerLagIntegrationTest > cleanerTest[1] STARTED

kafka.log.LogCleanerLagIntegrationTest > cleanerTest[1] PASSED

kafka.log.LogCleanerLagIntegrationTest > cleanerTest[2] STARTED

kafka.log.LogCleanerLagIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerLagIntegrationTest > cleanerTest[3] STARTED

kafka.log.LogCleanerLagIntegrationTest > cleanerTest[3] PASSED

kafka.log.LogCleanerLagIntegrationTest > cleanerTest[4] STARTED

kafka.log.LogCleanerLagIntegrationTest > cleanerTest[4] PASSED

kafka.log.LogCleanerParameterizedIntegrationTest > cleanerConfigUpdateTest[0] 
STARTED

kafka.log.LogCleanerParameterizedIntegrationTest > cleanerConfigUpdateTest[0] 
PASSED

kafka.log.LogCleanerParameterizedIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[0] STARTED

kafka.log.LogCleanerParameterizedIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[0] PASSED

kafka.log.LogCleanerParameterizedIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] STARTED

kafka.log.LogCleanerParameterizedIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] PASSED

kafka.log.LogCleanerParameterizedIntegrationTest > cleanerTest[0] STARTED

kafka.log.LogCleanerParameterizedIntegrationTest > cleanerTest[0] PASSED

kafka.log.LogCleanerParameterizedIntegrationTest > 
testCleanerWithMessageFormatV0[0] STARTED

kafka.log.LogCleanerParameterizedIntegrationTest > 
testCleanerWithMessageFormatV0[0] PASSED

Re: Apply for contributor permission

[Lifei Chen]

Thanks, Matthias.  And congratulations for you becoming a new PMC member !

Matthias J. Sax  于2019年4月25日周四 上午5:54写道：

> Done.
>
> Welcome to the community! Looking forward to your first PR!
>
>
> -Matthias
>
>
> On 4/24/19 3:50 AM, Lifei Chen wrote:
> > Hi, all:
> >
> > I am very interested in  kafka streams,  and willing to learn more about
> it
> > and  contribute to it when there is appropriate issue for me.
> >
> > Could someone help to give me the contributor permission?  My jira id is
> > hustclf .
> > Thanks a lot.
> >
> >
> >
> > Best.
> > Lifei
> >
>
>

Re: [DISCUSS] KIP-460: Admin Leader Election RPC

[Colin McCabe]

Hi Jose,

Thanks for the KIP, looks valuable.

If I use a PreferredLeaderElection RPC to specifically request an unclean 
leader election, will this take effect even if unclean leader elections are 
disabled on the topic involved?  I assume that the answer is yes, but it would 
be good to clarify this in the KIP.

What ACLs will be required to perform this action?  WRITE on the topic 
resource?  Or ALTER on KafkaCluster?  Or perhaps ALTER on the topic would be 
most appropriate, since we probably don't want ordinary producers triggering 
unclean leader elections.

What's the rationale for using an int8 rather than just having a boolean that 
is either true or false for "unclean"?  We only have two values now, and it 
seems like we could modify the RPC schema in the future if needed.  Or is the 
intention to add more flags later?

best,
Colin


On Wed, Apr 24, 2019, at 15:45, Jose Armando Garcia Sancio wrote:
> Hi all,
> 
> We would like to extend the "preferred leader election" RPC for the admin
> client to also support unclean leader elections.
> 
> The KIP can be found here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-460%3A+Admin+Leader+Election+RPC
> 
> Thanks!
> -Jose
>

[jira] [Created] (KAFKA-8287) JVM global map to fence duplicate client id

[Boyang Chen (JIRA)]

Boyang Chen created KAFKA-8287:
--

 Summary: JVM global map to fence duplicate client id
 Key: KAFKA-8287
 URL: https://issues.apache.org/jira/browse/KAFKA-8287
 Project: Kafka
  Issue Type: Sub-task
Reporter: Boyang Chen
Assignee: Boyang Chen


After change in https://issues.apache.org/jira/browse/KAFKA-8285, the two 
stream instances scheduled on same JVM will be mutually affected if they 
accidentally assign same client.id, since the thread-id becomes local now. The 
solution is to build a global concurrent map for solving conflict if two 
threads happen to be having the same client.id.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[VOTE] KIP-401: TransformerSupplier/ProcessorSupplier StateStore connecting

[Paul Whalen]

Hi all,

After some good discussion on and adjustments to KIP-401 (which I renamed
slightly for clarity), chatter has died down so I figured I may as well
start a vote.

KIP:
TransformerSupplier/ProcessorSupplier StateStore connecting

Discussion:
https://lists.apache.org/thread.html/600996d83d485f2b8daf45037de64a60cebdfac9b234bf3449b6b753@%3Cdev.kafka.apache.org%3E

Thanks!
Paul

Build failed in Jenkins: kafka-2.0-jdk8 #255

[Apache Jenkins Server]

See 


Changes:

[bill] KAFKA-8227 DOCS Fixed missing links duality of streams tables (#6625)

--
[...truncated 442.58 KB...]
kafka.zk.LiteralAclStoreTest > shouldRoundTripChangeNode PASSED

kafka.zk.LiteralAclStoreTest > shouldThrowFromEncodeOnNoneLiteral STARTED

kafka.zk.LiteralAclStoreTest > shouldThrowFromEncodeOnNoneLiteral PASSED

kafka.zk.LiteralAclStoreTest > shouldWriteChangesToTheWritePath STARTED

kafka.zk.LiteralAclStoreTest > shouldWriteChangesToTheWritePath PASSED

kafka.zk.LiteralAclStoreTest > shouldHaveCorrectPatternType STARTED

kafka.zk.LiteralAclStoreTest > shouldHaveCorrectPatternType PASSED

kafka.zk.LiteralAclStoreTest > shouldDecodeResourceUsingTwoPartLogic STARTED

kafka.zk.LiteralAclStoreTest > shouldDecodeResourceUsingTwoPartLogic PASSED

kafka.zk.KafkaZkClientTest > testZNodeChangeHandlerForDataChange STARTED

kafka.zk.KafkaZkClientTest > testZNodeChangeHandlerForDataChange PASSED

kafka.zk.KafkaZkClientTest > testCreateAndGetTopicPartitionStatesRaw STARTED

kafka.zk.KafkaZkClientTest > testCreateAndGetTopicPartitionStatesRaw PASSED

kafka.zk.KafkaZkClientTest > testLogDirGetters STARTED

kafka.zk.KafkaZkClientTest > testLogDirGetters PASSED

kafka.zk.KafkaZkClientTest > testSetGetAndDeletePartitionReassignment STARTED

kafka.zk.KafkaZkClientTest > testSetGetAndDeletePartitionReassignment PASSED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationsDeletion STARTED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationsDeletion PASSED

kafka.zk.KafkaZkClientTest > testGetDataAndVersion STARTED

kafka.zk.KafkaZkClientTest > testGetDataAndVersion PASSED

kafka.zk.KafkaZkClientTest > testGetChildren STARTED

kafka.zk.KafkaZkClientTest > testGetChildren PASSED

kafka.zk.KafkaZkClientTest > testSetAndGetConsumerOffset STARTED

kafka.zk.KafkaZkClientTest > testSetAndGetConsumerOffset PASSED

kafka.zk.KafkaZkClientTest > testClusterIdMethods STARTED

kafka.zk.KafkaZkClientTest > testClusterIdMethods PASSED

kafka.zk.KafkaZkClientTest > testEntityConfigManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testEntityConfigManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testUpdateLeaderAndIsr STARTED

kafka.zk.KafkaZkClientTest > testUpdateLeaderAndIsr PASSED

kafka.zk.KafkaZkClientTest > testUpdateBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testUpdateBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testCreateRecursive STARTED

kafka.zk.KafkaZkClientTest > testCreateRecursive PASSED

kafka.zk.KafkaZkClientTest > testGetConsumerOffsetNoData STARTED

kafka.zk.KafkaZkClientTest > testGetConsumerOffsetNoData PASSED

kafka.zk.KafkaZkClientTest > testDeleteTopicPathMethods STARTED

kafka.zk.KafkaZkClientTest > testDeleteTopicPathMethods PASSED

kafka.zk.KafkaZkClientTest > testSetTopicPartitionStatesRaw STARTED

kafka.zk.KafkaZkClientTest > testSetTopicPartitionStatesRaw PASSED

kafka.zk.KafkaZkClientTest > testAclManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testAclManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testPreferredReplicaElectionMethods STARTED

kafka.zk.KafkaZkClientTest > testPreferredReplicaElectionMethods PASSED

kafka.zk.KafkaZkClientTest > testPropagateLogDir STARTED

kafka.zk.KafkaZkClientTest > testPropagateLogDir PASSED

kafka.zk.KafkaZkClientTest > testGetDataAndStat STARTED

kafka.zk.KafkaZkClientTest > testGetDataAndStat PASSED

kafka.zk.KafkaZkClientTest > testReassignPartitionsInProgress STARTED

kafka.zk.KafkaZkClientTest > testReassignPartitionsInProgress PASSED

kafka.zk.KafkaZkClientTest > testCreateTopLevelPaths STARTED

kafka.zk.KafkaZkClientTest > testCreateTopLevelPaths PASSED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationGetters STARTED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationGetters PASSED

kafka.zk.KafkaZkClientTest > testLogDirEventNotificationsDeletion STARTED

kafka.zk.KafkaZkClientTest > testLogDirEventNotificationsDeletion PASSED

kafka.zk.KafkaZkClientTest > testGetLogConfigs STARTED

kafka.zk.KafkaZkClientTest > testGetLogConfigs PASSED

kafka.zk.KafkaZkClientTest > testBrokerSequenceIdMethods STARTED

kafka.zk.KafkaZkClientTest > testBrokerSequenceIdMethods PASSED

kafka.zk.KafkaZkClientTest > testCreateSequentialPersistentPath STARTED

kafka.zk.KafkaZkClientTest > testCreateSequentialPersistentPath PASSED

kafka.zk.KafkaZkClientTest > testConditionalUpdatePath STARTED

kafka.zk.KafkaZkClientTest > testConditionalUpdatePath PASSED

kafka.zk.KafkaZkClientTest > testDeleteTopicZNode STARTED

kafka.zk.KafkaZkClientTest > testDeleteTopicZNode PASSED

kafka.zk.KafkaZkClientTest > testDeletePath STARTED

kafka.zk.KafkaZkClientTest > testDeletePath PASSED

kafka.zk.KafkaZkClientTest > testGetBrokerMethods STARTED

kafka.zk.KafkaZkClientTest > testGetBrokerMethods PASSED

kafka.zk.KafkaZkClientTest > testCreateTokenChangeNotification STARTED

kafka.zk.KafkaZkClientTest > 

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

[Magesh Nandakumar]

Hi Chris,

Instead of calling it ConnectClusterId, perhaps call it
ConnectClusterDetails which can include things like groupid, underlying
kafkaclusterId, standalone or distributed, etc. This will help expose any
cluster related information in the future.
Let me know if that would work?

Thanks,
Magesh

On Wed, Apr 24, 2019 at 4:26 PM Chris Egerton  wrote:

> Hi Magesh,
>
> 1. After ruminating for a little while on the inclusion of a method to
> retrieve task configurations I've tentatively decided to remove it from the
> proposal and place it in the rejected alternatives section. If anyone
> presents a reasonable use case for it I'll be happy to discuss further but
> right now I think this is the way to go. Thanks for your suggestion!
>
> 2. The idea of a Connect cluster ID method is certainly fascinating, but
> there are a few questions it raises. First off, what would the group.id be
> for a standalone cluster? Second, why return a formatted string there
> instead of a new class such as a ConnectClusterId that provides the two in
> separate methods? And lastly, since REST extensions are configured with all
> of the properties available to the worker, wouldn't it be possible to just
> get the group ID of the Connect cluster from there? The reason I'd like to
> see the Kafka cluster ID made available to REST extensions is that
> retrieving it isn't as simple as reading a configuration from a properties
> map and instead involves creating an admin client from those properties and
> using it to perform a `describe cluster` call, which comes with its own
> pitfalls as far as error handling, interruptions, and timeouts go. Since
> this information is available to the herder already, it seems like a good
> tradeoff to expose that information to REST extensions so that developers
> don't have to duplicate that logic themselves. I'm unsure that the same
> arguments would apply to exposing a group.id to REST extensions through
> the
> ConnectClusterInterface. What do you think?
>
> Thanks again for your thoughts!
>
> Cheers,
>
> Chris
>
> On Tue, Apr 23, 2019 at 4:18 PM Magesh Nandakumar 
> wrote:
>
> > Chris,
> >
> > I certainly would love to hear others thoughts on #1 but IMO, it might
> not
> > be as useful as ConnectorConfigs and as you mentioned, we could always
> add
> > it when the need arises.
> > Thanks for clarifying the details on my concern #2 regarding the
> > kafkaClusterId. While not a perfect fit in the interface, I'm not
> > completely opposed to having it in the interface. The other option, I can
> > think is to expose a connectClusterId() returning group.id +
> > kafkaClusterId
> > (with some delimiter) rather than returning the kafkaClusterId. If we
> > choose to go this route, we can even make this a first-class citizen of
> the
> > Herder interface. Let me know what you think.
> >
> > Thanks
> > Magesh
> >
> > On Thu, Apr 18, 2019 at 2:45 PM Chris Egerton 
> wrote:
> >
> > > Hi Magesh,
> > >
> > > Thanks for your comments. I'll address them in the order you provided
> > them:
> > >
> > > 1 - Reason for exposing task configurations to REST extensions:
> > > Yes, the motivation is a little thin for exposing task configs to REST
> > > extensions. I can think of a few uses for this functionality, such as
> > > attempting to infer problematic configurations by examining failed
> tasks
> > > and comparing their configurations to the configurations of running
> > tasks,
> > > but like you've indicated it's dubious that the best place for anything
> > > like that belongs in a REST extension.
> > > I'd be interested to hear others' thoughts, but right now I'm not too
> > > opposed to erring on the side of caution and leaving it out. Worst
> case,
> > it
> > > takes another KIP to add this later on down the road, but that's a
> small
> > > price to pay to avoid adding support for a feature that nobody needs.
> > >
> > > 2. Usefulness of exposing Kafka cluster ID to REST extensions:
> > > As the KIP states, "the Kafka cluster ID may be useful for the purpose
> of
> > > uniquely identifying a Connect cluster from within a REST extension,
> > since
> > > users may be running multiple Kafka clusters and the group.id for a
> > > distributed Connect cluster may not be sufficient to identify a
> cluster."
> > > Even though there may be producer or consumer overrides for
> > > bootstrap.servers present in the configuration for the worker, these
> will
> > > not affect which Kafka cluster is used as a backing store for connector
> > > configurations, offsets, and statuses, so the Kafka cluster ID for the
> > > worker in conjunction with the Connect group ID should be sufficient to
> > > uniquely identify a Connect cluster.
> > > We can and should document that the Connect cluster with overridden
> > > producer.bootstrap.servers or consumer.bootstrap.servers may be writing
> > > to/reading from a different Kafka cluster. However, REST extensions are
> > > already passed the configs for their worker through 

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

[Chris Egerton]

Hi Magesh,

1. After ruminating for a little while on the inclusion of a method to
retrieve task configurations I've tentatively decided to remove it from the
proposal and place it in the rejected alternatives section. If anyone
presents a reasonable use case for it I'll be happy to discuss further but
right now I think this is the way to go. Thanks for your suggestion!

2. The idea of a Connect cluster ID method is certainly fascinating, but
there are a few questions it raises. First off, what would the group.id be
for a standalone cluster? Second, why return a formatted string there
instead of a new class such as a ConnectClusterId that provides the two in
separate methods? And lastly, since REST extensions are configured with all
of the properties available to the worker, wouldn't it be possible to just
get the group ID of the Connect cluster from there? The reason I'd like to
see the Kafka cluster ID made available to REST extensions is that
retrieving it isn't as simple as reading a configuration from a properties
map and instead involves creating an admin client from those properties and
using it to perform a `describe cluster` call, which comes with its own
pitfalls as far as error handling, interruptions, and timeouts go. Since
this information is available to the herder already, it seems like a good
tradeoff to expose that information to REST extensions so that developers
don't have to duplicate that logic themselves. I'm unsure that the same
arguments would apply to exposing a group.id to REST extensions through the
ConnectClusterInterface. What do you think?

Thanks again for your thoughts!

Cheers,

Chris

On Tue, Apr 23, 2019 at 4:18 PM Magesh Nandakumar 
wrote:

> Chris,
>
> I certainly would love to hear others thoughts on #1 but IMO, it might not
> be as useful as ConnectorConfigs and as you mentioned, we could always add
> it when the need arises.
> Thanks for clarifying the details on my concern #2 regarding the
> kafkaClusterId. While not a perfect fit in the interface, I'm not
> completely opposed to having it in the interface. The other option, I can
> think is to expose a connectClusterId() returning group.id +
> kafkaClusterId
> (with some delimiter) rather than returning the kafkaClusterId. If we
> choose to go this route, we can even make this a first-class citizen of the
> Herder interface. Let me know what you think.
>
> Thanks
> Magesh
>
> On Thu, Apr 18, 2019 at 2:45 PM Chris Egerton  wrote:
>
> > Hi Magesh,
> >
> > Thanks for your comments. I'll address them in the order you provided
> them:
> >
> > 1 - Reason for exposing task configurations to REST extensions:
> > Yes, the motivation is a little thin for exposing task configs to REST
> > extensions. I can think of a few uses for this functionality, such as
> > attempting to infer problematic configurations by examining failed tasks
> > and comparing their configurations to the configurations of running
> tasks,
> > but like you've indicated it's dubious that the best place for anything
> > like that belongs in a REST extension.
> > I'd be interested to hear others' thoughts, but right now I'm not too
> > opposed to erring on the side of caution and leaving it out. Worst case,
> it
> > takes another KIP to add this later on down the road, but that's a small
> > price to pay to avoid adding support for a feature that nobody needs.
> >
> > 2. Usefulness of exposing Kafka cluster ID to REST extensions:
> > As the KIP states, "the Kafka cluster ID may be useful for the purpose of
> > uniquely identifying a Connect cluster from within a REST extension,
> since
> > users may be running multiple Kafka clusters and the group.id for a
> > distributed Connect cluster may not be sufficient to identify a cluster."
> > Even though there may be producer or consumer overrides for
> > bootstrap.servers present in the configuration for the worker, these will
> > not affect which Kafka cluster is used as a backing store for connector
> > configurations, offsets, and statuses, so the Kafka cluster ID for the
> > worker in conjunction with the Connect group ID should be sufficient to
> > uniquely identify a Connect cluster.
> > We can and should document that the Connect cluster with overridden
> > producer.bootstrap.servers or consumer.bootstrap.servers may be writing
> > to/reading from a different Kafka cluster. However, REST extensions are
> > already passed the configs for their worker through their configure(...)
> > method, so they'd be able to detect any such overrides and act
> accordingly.
> >
> > Thanks again for your thoughts!
> >
> > Cheers,
> >
> > Chris
> >
> > On Thu, Apr 18, 2019 at 11:08 AM Magesh Nandakumar  >
> > wrote:
> >
> > > Hi Chris,
> > >
> > > Thanks for the KIP. Overall, it looks good and straightforward to me.
> > >
> > > I had a few questions on the new methods
> > >
> > > 1. I'm not sure if an extension would ever require the task configs. An
> > > extension generally should only require the health and current 

[DISCUSS] KIP-460: Admin Leader Election RPC

[Jose Armando Garcia Sancio]

Hi all,

We would like to extend the "preferred leader election" RPC for the admin
client to also support unclean leader elections.

The KIP can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-460%3A+Admin+Leader+Election+RPC

Thanks!
-Jose

[jira] [Created] (KAFKA-8286) KIP-460 Admin Leader Election RPC

[Jose Armando Garcia Sancio (JIRA)]

Jose Armando Garcia Sancio created KAFKA-8286:
-

 Summary: KIP-460 Admin Leader Election RPC
 Key: KAFKA-8286
 URL: https://issues.apache.org/jira/browse/KAFKA-8286
 Project: Kafka
  Issue Type: New Feature
  Components: admin, clients, core
Reporter: Jose Armando Garcia Sancio
Assignee: Jose Armando Garcia Sancio


Tracking issue for implementing KIP-460.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-8227) Missing Links in Duality of Streams and Tables section

[Bill Bejeck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Bejeck resolved KAFKA-8227.

Resolution: Fixed

> Missing Links in Duality of Streams and Tables section
> --
>
> Key: KAFKA-8227
> URL: https://issues.apache.org/jira/browse/KAFKA-8227
> Project: Kafka
>  Issue Type: Bug
>  Components: documentation, streams
>Reporter: Bill Bejeck
>Assignee: Victoria Bialas
>Priority: Minor
> Fix For: 2.0.2, 2.3.0, 2.1.2, 2.2.1
>
>
> [https://kafka.apache.org/21/documentation/streams/core-concepts#streams-concepts-duality]
>  section there several sections that are emphasized but should be links
>  # streams 
>  # tables 
>  # elastic 
>  # fault-tolerant stateful processing 
> 
>  # interactive queries 
>  # aggregations 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Permission to create a KIP

[Jose Armando Garcia Sancio]

I got permission now. Thanks!

On Wed, Apr 24, 2019 at 12:02 PM Jose Armando Garcia Sancio <
jsan...@confluent.io> wrote:

> Hello,
>
> Can I get permission to create a KIP in the Apache Kafka wiki page?
>
> Thanks!
>
> --
> -Jose
>


-- 
-Jose

Re: Apply for contributor permission

[Matthias J. Sax]

Done.

Welcome to the community! Looking forward to your first PR!


-Matthias


On 4/24/19 3:50 AM, Lifei Chen wrote:
> Hi, all:
> 
> I am very interested in  kafka streams,  and willing to learn more about it
> and  contribute to it when there is appropriate issue for me.
> 
> Could someone help to give me the contributor permission?  My jira id is
> hustclf .
> Thanks a lot.
> 
> 
> 
> Best.
> Lifei
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Reopened] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-3729:


>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Assignee: Ted Yu
>Priority: Major
>  Labels: api, newbie
> Fix For: 2.3.0
>
> Attachments: 3729.txt, 3729.v6.txt
>
>
> From Guozhang Wang:
>  "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."
> After the PR was merged, we realized that the current approach to implement 
> this features is actually not backward compatible. Thus, we need to revert 
> the commit for now to not break backward compatibility in 2.3 release. After 
> some more thinking, it seems that this feature is actually more complicated 
> to get right as it seem on the surface and hence it would required a proper 
> KIP.
> The following issues are identified:
>  * in the new code, configure() would be called twice, one in user code (if 
> people don't rewrite existing applications) and later via Kafka Streams – the 
> second call could "reconfigure" the Serde and overwrite the correct 
> configuration from the first call done by the user
>  * if there are multiple Serdes using the same configuration parameters 
> names, it's only possible to specify this parameter name once in the global 
> StreamsConfig; hence, it's not possible for users to configure both Serdes 
> differently
>  * basically, the global StreamsConfig needs to contain all configuration 
> parameters over all used Serdes to make a potential second call to 
> `configure()` idempotant
> To address the issues, some ideas would be:
>  * pass in the configuration via the constructor and deprecate `configure()` 
> method
>  * add a new method `isConfigured()` that would allow to skip the second 
> configuration call within Kafka Streams runtime
> There might be other ways to address this, and the different options should 
> be discussed on the KIP.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8285) Handle thread-id random switch on JVM for KStream

[Boyang Chen (JIRA)]

Boyang Chen created KAFKA-8285:
--

 Summary: Handle thread-id random switch on JVM for KStream
 Key: KAFKA-8285
 URL: https://issues.apache.org/jira/browse/KAFKA-8285
 Project: Kafka
  Issue Type: Sub-task
Reporter: Boyang Chen
Assignee: Boyang Chen






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8284) Enable static membership on KStream

[Boyang Chen (JIRA)]

Boyang Chen created KAFKA-8284:
--

 Summary: Enable static membership on KStream
 Key: KAFKA-8284
 URL: https://issues.apache.org/jira/browse/KAFKA-8284
 Project: Kafka
  Issue Type: Sub-task
Reporter: Boyang Chen
Assignee: Boyang Chen






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Permission to create a KIP

[Jose Armando Garcia Sancio]

Hello,

Can I get permission to create a KIP in the Apache Kafka wiki page?

Thanks!

-- 
-Jose

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

[Magesh Nandakumar]

HI Chrise,

You are right about the "admin." prefix creating conflicts. Here are few
options that I can think of

1. Use `dlq.admin` since admin client is used only for DLQ. But this might
not really be the case in the future. So, we should possibly drop this idea
:)
2.  Use `connector.producer`, `connector.consumer` and `connector.admin` -
provides better context that its connector specific property
3.  Use `producer.override`, '`consumer.override` and `admin.override` -
provides better clarity that these are overrides.

I don't have a strong opinion in choosing between #2 and #3. Let me
know what you think.

Thanks
Magesh

On Wed, Apr 24, 2019 at 10:25 AM Chris Egerton  wrote:

> Hi Magesh,
>
> Next round :)
>
> 1. It looks like MM2 will also support "admin." properties that affect
> AdminClients it creates and uses, which IIUC is the same prefix name to be
> used for managing the DLQ for sink connectors in this KIP. Doesn't that
> still leave room for conflict? I'm imagining a scenario like this: a
> Connect worker is configured to use the
> PrincipalConnectorClientConfigPolicy, someone tries to start an instance of
> an MM2 sink with "admin." properties beyond just "admin.sasl.jaas.config",
> and gets rejected because those properties are then interpreted by the
> worker as overrides for the AdminClient it uses to manage the DLQ.
> 2. (LGTM)
> 3. I'm convinced by this, as long as nobody else identifies a common use
> case that would involve a similar client config policy implementation that
> would be limited to a small set of whitelisted configs. For now keeping the
> PrincipalConnectorClientConfigPolicy sounds fine to me.
>
> Cheers,
>
> Chris
>
> On Tue, Apr 23, 2019 at 10:30 PM Magesh Nandakumar 
> wrote:
>
> > Hi all,
> >
> > I also have a draft implementation of the KIP
> > https://github.com/apache/kafka/pull/6624. I would still need to include
> > more tests and docs but I thought it would be useful to have this for the
> > KIP discussion. Looking forward to all of your valuable feedback.
> >
> > Thanks
> > Magesh
> >
> > On Tue, Apr 23, 2019 at 10:27 PM Magesh Nandakumar  >
> > wrote:
> >
> > > Chrise,
> > >
> > > Thanks a lot for your feedback. I will address them in order of your
> > > questions/comments.
> > >
> > > 1. Thanks for bringing this to my attention about KIP-382. I had a
> closer
> > > look at the KIP and IIUC, the KIP allows `consumer.` prefix for
> > SourceConnector
> > > and producer. prefix for SinkConnector since those are additional
> > > connector properties to help resolve the Kafka cluster other than the
> one
> > > Connect framework knows about. Whereas, the proposal in KIP-458 applies
> > > producer policies for SinkConnectors and consumer policies
> > > SourceConnectors.  So, from what I understand this new policy should
> work
> > > without any issues even for Mirror Maker 2.0.
> > > 2. I have updated the KIP to use a default value of null and use that
> to
> > > determine if we need to ignore overrides.
> > > 3. I would still prefer to keep the special
> > PrincipalConnectorClientConfigPolicy
> > > since that is one of the most common use cases one would choose to use
> > this
> > > feature. If we make it a general case, that would involve users
> requiring
> > > to add additional configuration and they might require well more than
> > just
> > > the list of configs but might also want some restriction on values. If
> > the
> > > concern is about users wanting principal and also other configs, it
> would
> > > still be possible by means of a custom implementation. As is, I would
> > > prefer to keep the proposal to be the same for this. Let me know your
> > > thoughts.
> > >
> > > Thanks,
> > > Magesh
> > >
> > >
> > > On Mon, Apr 22, 2019 at 3:44 PM Chris Egerton 
> > wrote:
> > >
> > >> Hi Magesh,
> > >>
> > >> This is an exciting KIP! I have a few questions/comments but overall I
> > >> like
> > >> the direction it's headed in and hope to see it included in the
> Connect
> > >> framework soon.
> > >>
> > >> 1. With the proposed "consumer.", "producer.", and "admin." prefixes,
> > how
> > >> will this interact with connectors such as the upcoming Mirror Maker
> 2.0
> > >> (KIP-382) that already support properties with those prefixes? Would
> it
> > be
> > >> possible for a user to configure MM2 with those properties without
> them
> > >> being interpreted as Connect client overrides, without isolating MM2
> > onto
> > >> its own cluster and using the IgnoreConnectorClientConfigPolicy
> policy?
> > >> 2. Is the IgnoreConnectorClientConfigPolicy class necessary? The
> default
> > >> for the connector.client.config.policy property could simply be null
> > >> instead of a new policy that, as far as I can tell, isn't an actual
> > policy
> > >> in that its validate(...) method is never invoked and instead
> > represents a
> > >> special case to the Connect framework that says "Drop all overrides
> and
> > >> never use me".
> > >> 3. The 

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

[Jason Gustafson]

Hi George,

I think focusing on the admin API for cancellation in this KIP is
reasonable. Colin wrote up KIP-455 which adds APIs to submit a reassignment
and to list reassignments in progress. Probably as long as we make these
APIs all consistent with each other, it is fine to do them separately.

The major point we need to reach agreement on is how an active reassignment
will be stored. In the proposal here, we continue to
use /admin/reassign_partitions and we further extend it to differentiate
between the original replicas and the target replicas of a reassignment. In
KIP-455, Colin proposes to store the same information, but do it at the
partition level. Obviously we would only do one of these. I think the main
difference is how we think about reassignment in general. Currently we see
it as basically a batch job which is submitted atomically and not completed
until all the partitions in the reassignment have been moved. The other
perspective is to treat reassignments more fluidly. At any point, we just
have some partitions which are undergoing reassignment. Then there is no
longer a question about supporting multiple reassignments: we can reassign
new partitions at any time and we can cancel individual partition
reassignments also at any time. This is a more ambitious approach, but it
seems much more flexible in the long term.

Thanks,
Jason



On Fri, Apr 5, 2019 at 7:25 PM George Li 
wrote:

>  Hi Jun,
>
> Thanks for the feedback!
>
> for 40,  I agree.  It makes sense to do it via RPC request to the
> controller.  Maybe for KIP-236,  I will just implement RPC for reassignment
> cancellation only?  otherwise if KIP-236 includes other previous
> Reassignment related operations such as Submitting Reassignments,
> --generate,  --verify, Change/Remove Throttle, I think KIP-236 will incur a
> lot of changes.
>
> For 41, the current reassignment logic, after /admin/reassign_partitions
> has the "new_replicas" for the topic/partition,  the controller will set
> the AR (Assigned Replicas) to be the Set(original_replicas) +
> Set(new_replicas),  thus when reassignment begins, the "original_replicas"
> info is lost. It's not possible to use the AR and new_replicas in
> /admin/reassign_partitions to derive back the "original_replicas".
>
> To cancel the current pending reassignments, the "original_replicas" info
> is required to restore/rollback.
>
> As a matter of fact, if we have "original_replicas" info when doing
> reassignment, some features can be accomplished.  e.g. To name a few:
> separating the Reassignment Replication Traffic with the normal Follower's
> ReplicaFetcher thread pool, the dedicated Reassignment ReplicaFetcher
> thread pool can have throttle enforced, while existing Follower
> ReplicaFetcher not throttled. some networking parameters can be tuned for
> Reassignment ReplicaFetcher thread pool.  The Reassignment can have its own
> MaxLag, TotalLag metrics.  and URP (Under Replicated Partition) can be
> reported differently for existing followers Vs. the URP caused by
> Reassignment.  Anyway, these might be some future KIPs that we might
> submit.
>
> Thanks,
> George
>
>
> On Thursday, April 4, 2019, 4:32:26 PM PDT, Jun Rao 
> wrote:
>
>  Hi, George,
>
> Thanks for the KIP. Sorry for the late reply. A couple of comments below.
>
> 40. I agree that it's better to issue an RPC request to the controller for
> reassignment cancellation. If we do that, it would be useful to decide
> whether that call blocks on cancellation completion or not.
>
> 41. Is it necessary to add the new "original replicas" field in
> /admin/reassign_partitions?
> The original replicas are already in the topic path in ZK.
>
> Jun
>
> On Tue, Mar 26, 2019 at 5:24 PM George Li  .invalid>
> wrote:
>
> >  Hi Ismael,
> >
> > Thanks,  I understand your points. I will add the RPC mechanism for
> > reassignments to KIP-236.  I can think of a few Requests/Responses
> > corresponding to the old Scala AdminClient using ZK:
> >
> > SubmitReassignments (--execute)
> > StatusReassignments (--verify)
> > ChangeOrRemoveReassignmentsThrottle (--verify)
> > GenerateReassignments (--generate)
> > CancelPendingReassignments (new in KIP-236)
> >
> > To clarify the "structure" change of KIP-236, the ZK public interface
> > remains the same:  {topic, partition, new_replicas},  the user client
> > generate/submit the reassignment plan the same way as before.  the new
> > "original_replicas" in the ZK node is added by admin client before
> writing
> > to the ZK node.  A bit similar to the "log_dirs".  User's direct
> > modification of ZK /admin/reassign_partitions is strongly discouraged.
> >
> > The original_replicas info is essential for cancellation/rollback of the
> > reassignments still pending.
> >
> > Thanks,
> > George
> >
> >On Monday, March 25, 2019, 4:43:30 PM PDT, Ismael Juma <
> > ism...@juma.me.uk> wrote:
> >
> >  Hi George,
> >
> > The goal is not to prevent people from updating ZK directly. The goal is
> to
> > offer a 

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

[Chris Egerton]

Hi Magesh,

Next round :)

1. It looks like MM2 will also support "admin." properties that affect
AdminClients it creates and uses, which IIUC is the same prefix name to be
used for managing the DLQ for sink connectors in this KIP. Doesn't that
still leave room for conflict? I'm imagining a scenario like this: a
Connect worker is configured to use the
PrincipalConnectorClientConfigPolicy, someone tries to start an instance of
an MM2 sink with "admin." properties beyond just "admin.sasl.jaas.config",
and gets rejected because those properties are then interpreted by the
worker as overrides for the AdminClient it uses to manage the DLQ.
2. (LGTM)
3. I'm convinced by this, as long as nobody else identifies a common use
case that would involve a similar client config policy implementation that
would be limited to a small set of whitelisted configs. For now keeping the
PrincipalConnectorClientConfigPolicy sounds fine to me.

Cheers,

Chris

On Tue, Apr 23, 2019 at 10:30 PM Magesh Nandakumar 
wrote:

> Hi all,
>
> I also have a draft implementation of the KIP
> https://github.com/apache/kafka/pull/6624. I would still need to include
> more tests and docs but I thought it would be useful to have this for the
> KIP discussion. Looking forward to all of your valuable feedback.
>
> Thanks
> Magesh
>
> On Tue, Apr 23, 2019 at 10:27 PM Magesh Nandakumar 
> wrote:
>
> > Chrise,
> >
> > Thanks a lot for your feedback. I will address them in order of your
> > questions/comments.
> >
> > 1. Thanks for bringing this to my attention about KIP-382. I had a closer
> > look at the KIP and IIUC, the KIP allows `consumer.` prefix for
> SourceConnector
> > and producer. prefix for SinkConnector since those are additional
> > connector properties to help resolve the Kafka cluster other than the one
> > Connect framework knows about. Whereas, the proposal in KIP-458 applies
> > producer policies for SinkConnectors and consumer policies
> > SourceConnectors.  So, from what I understand this new policy should work
> > without any issues even for Mirror Maker 2.0.
> > 2. I have updated the KIP to use a default value of null and use that to
> > determine if we need to ignore overrides.
> > 3. I would still prefer to keep the special
> PrincipalConnectorClientConfigPolicy
> > since that is one of the most common use cases one would choose to use
> this
> > feature. If we make it a general case, that would involve users requiring
> > to add additional configuration and they might require well more than
> just
> > the list of configs but might also want some restriction on values. If
> the
> > concern is about users wanting principal and also other configs, it would
> > still be possible by means of a custom implementation. As is, I would
> > prefer to keep the proposal to be the same for this. Let me know your
> > thoughts.
> >
> > Thanks,
> > Magesh
> >
> >
> > On Mon, Apr 22, 2019 at 3:44 PM Chris Egerton 
> wrote:
> >
> >> Hi Magesh,
> >>
> >> This is an exciting KIP! I have a few questions/comments but overall I
> >> like
> >> the direction it's headed in and hope to see it included in the Connect
> >> framework soon.
> >>
> >> 1. With the proposed "consumer.", "producer.", and "admin." prefixes,
> how
> >> will this interact with connectors such as the upcoming Mirror Maker 2.0
> >> (KIP-382) that already support properties with those prefixes? Would it
> be
> >> possible for a user to configure MM2 with those properties without them
> >> being interpreted as Connect client overrides, without isolating MM2
> onto
> >> its own cluster and using the IgnoreConnectorClientConfigPolicy policy?
> >> 2. Is the IgnoreConnectorClientConfigPolicy class necessary? The default
> >> for the connector.client.config.policy property could simply be null
> >> instead of a new policy that, as far as I can tell, isn't an actual
> policy
> >> in that its validate(...) method is never invoked and instead
> represents a
> >> special case to the Connect framework that says "Drop all overrides and
> >> never use me".
> >> 3. The PrincipalConnectorClientConfigPolicy seems like a specific
> instance
> >> of a more general use case: allow exactly a small set of overrides and
> no
> >> others. Why not generalize here and create a policy that accepts a list
> of
> >> allowed overrides during configuration?
> >>
> >> Thanks again for the KIP.
> >>
> >> Cheers,
> >>
> >> Chris
> >>
> >> On Fri, Apr 19, 2019 at 2:53 PM Magesh Nandakumar  >
> >> wrote:
> >>
> >> > Hi all,
> >> >
> >> > I've posted "KIP-458: Connector Client Config Override Policy", which
> >> > allows users to override the connector client configurations based on
> a
> >> > policy defined by the administrator.
> >> >
> >> > The KIP can be found at
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy
> >> > .
> >> >
> >> > Looking forward for the discussion on the KIP and all of your
> thoughts &
> >> > feedback on this 

Re: [VOTE] KIP-421: Automatically resolve external configurations.

[TEJAL ADSUL]

Thanks Dongjin for taking time to review the KIP and vote 

Thanks,
Tejal
On 2019/04/24 01:19:36, Dongjin Lee  wrote: 
> @Colin I see. Thanks for the correction!
> @Tejal Thanks for the excellent proposal! +1k!
> 
> Thanks,
> Dongjin
> 
> On Mon, Apr 22, 2019 at 10:26 PM Colin McCabe  wrote:
> 
> > Hi Dongjin,
> >
> > Thanks for checking out the KIP.
> >
> > Randall is a committer, actually.  He was added two months ago.  It looks
> > like he isn't on the committers page yet, though.
> >
> > Randall, can you add your name to https://kafka.apache.org/committers.html
> > to help prevent confusion in the future?
> >
> > best,
> > Colin
> >
> >
> > On Mon, Apr 22, 2019, at 16:44, Dongjin Lee wrote:
> > > I just read the KIP. +1.
> > >
> > > Currently:
> > >
> > > Binding: +2 (Colin, Gwen)
> > > Non-binding: +2 (Randall, Dongjin)
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > On Mon, Apr 22, 2019 at 10:26 AM TEJAL ADSUL  wrote:
> > >
> > > > Hi Folks,
> > > >
> > > > Just a reminder that I will be closing the vote for KIP-421 by today
> > EOD,
> > > > please cast your votes by today.
> > > >
> > > > Thanks Colin, Randall, Gwen for the votes.
> > > >
> > > > Thanks,
> > > > Tejal
> > > >
> > > >
> > > >
> > > > On 2019/04/19 22:43:25, "Colin McCabe"  wrote:
> > > > > +1.  Thanks, Tejal.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Thu, Apr 18, 2019, at 15:02, TEJAL ADSUL wrote:
> > > > > > Hi All,
> > > > > >
> > > > > > As we have reached a consensus on the design, I would like to
> > start a
> > > > > > vote for KIP-421. Below are the links for this proposal:
> > > > > >
> > > > > > KIP Link:
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829515
> > > > > > DiscussionThread:
> > > > > >
> > > >
> > https://lists.apache.org/thread.html/a2f834d876e9f8fb3977db794bf161818c97f7f481edd1b10449d89f@%3Cdev.kafka.apache.org%3E
> > > > > >
> > > > > > Thanks,
> > > > > > Tejal
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > > *github:  github.com/dongjinleekr
> > > linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > speakerdeck:
> > speakerdeck.com/dongjin
> > > *
> > >
> >
> 
> 
> -- 
> *Dongjin Lee*
> 
> *A hitchhiker in the mathematical world.*
> *github:  github.com/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck: speakerdeck.com/dongjin
> *
> 

Re: [VOTE] KIP-433: Block old clients on brokers

[Ismael Juma]

Hi Harsha,

Comments inline.

On Fri, Apr 12, 2019 at 8:22 PM Harsha  wrote:

> Hi Ismael,
> I meant to say blocking clients based on their API version
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48
> But If I understand what you are saying, since each client release can
> support different versions for each of fetch, produce, offset commit etc..
> and it's harder to block just based on single min.api.version setting
> across different clients.
>

That's what I mean, yes.


> The idea I had in my mind was to do this via ApiVersionRequest, when a
> client makes api request to broker in response we return min and max
> version supported for each Api. When min.api.version enabled on broker, it
> returns the maxVersion it supports for each of the requests in that release
> as min versions to the clients.
>

Yes, my concern is not the implementation, that's straightforward, it's how
it's meant to be used.

Irrespective of the above approach I understand your point still stands
> which is sarama might not choose to implement all the higher version
> protocols for Kafka 1.1 release and they might introduce higher version of
> produce request in a subsequent minor release and it will be harder for
> users to figure out which release of sarama client they can use.
>

Yes, that's exactly right. The current KIP is not treating the Kafka
protocol as a specification that is implemented by multiple clients in
multiple languages. During KIP-35, it was agreed that the protocol would
not be tied to a particular Kafka version. The current KIP proposal doesn't
take that into account.

I looked at the latest updates and I don't think the core issue is
addressed:

https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=103091409=15=13

It would be useful to hear feedback from other people.

Ismael

Re: [VOTE] KIP-433: Block old clients on brokers

[Harsha]

Hi Gwen & Ismael,
   Do you have any feedback on with the proposed approach, 
min.api.version allowing users to specify versions for every request.

Thanks,
Harsha

On Fri, Apr 19, 2019, at 10:24 AM, Harsha wrote:
> Thanks Ying for updating the KIP. 
> Hi Ismael,
>  Given min.api.version allows admin/users to specifiy 
> min.version for each request this should address your concerns right?
> 
> Thanks,
> Harsha
> 
> On Wed, Apr 17, 2019, at 2:29 PM, Ying Zheng wrote:
> > I have updated the config description in the KIP, made the example more
> > clear
> > 
> > The proposed change allows setting different min versions for different
> > APIs, and the ApiVersionRequest change is already in the KIP.
> > 
> > On Fri, Apr 12, 2019 at 8:22 PM Harsha  wrote:
> > 
> > > Hi Ismael,
> > > I meant to say blocking clients based on their API version
> > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48
> > > But If I understand what you are saying, since each client release can
> > > support different versions for each of fetch, produce, offset commit etc..
> > > and it's harder to block just based on single min.api.version setting
> > > across different clients.
> > > The idea I had in my mind was to do this via ApiVersionRequest, when a
> > > client makes api request to broker in response we return min and max
> > > version supported for each Api. When min.api.version enabled on broker, it
> > > returns the maxVersion it supports for each of the requests in that 
> > > release
> > > as min versions to the clients.
> > >
> > > Example:
> > > Kafka 1.1.1 broker and min.api.verson set to
> > > https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/api/ApiVersion.scala#L79
> > > (KAFKA_1_1_IV0) and client makes a ApiVersionsRequest and in response for
> > > example produce request
> > >
> > > https://github.com/apache/kafka/blob/1.1/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java#L112
> > > Instead of returning all of the supported versions it will return
> > > PRODUCE_REQUEST_V5 as the only supported version.
> > >
> > > Irrespective of the above approach I understand your point still stands
> > > which is sarama might not choose to implement all the higher version
> > > protocols for Kafka 1.1 release and they might introduce higher version of
> > > produce request in a subsequent minor release and it will be harder for
> > > users to figure out which release of sarama client they can use.
> > >
> > >
> > > Ying, if you have a different apporach which might address this issue
> > > please add.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Apr 12, 2019, at 7:23 PM, Ismael Juma wrote:
> > > > Hi Harsha,
> > > >
> > > > There is no such thing as 1.1 protocol. I encourage you to describe an
> > > > example config that achieves what you are suggesting here. It's pretty
> > > > complicated because the versions are per API and each client evolves
> > > > independently.
> > > >
> > > > Ismael
> > > >
> > > > On Sat, Apr 13, 2019 at 4:09 AM Harsha  wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > "Relying on min.version seems like a pretty clunky way to achieve the
> > > above
> > > > > > list. The challenge is that it's pretty difficult to do it in a way
> > > that
> > > > > > works for clients across languages. They each add support for new
> > > > > protocol
> > > > > > versions independently (it could even happen in a bug fix release).
> > > So,
> > > > > if
> > > > > > you tried to block Sarama in #2, you may block Java clients too."
> > > > >
> > > > > That's the intended effect, right?  if you as the admin/operator
> > > > > configures the broker to have min.api.version to be 1.1
> > > > > it should block java , sarama clients etc.. which are below the 1.1
> > > > > protocol.  As mentioned this is not just related to log.format upgrade
> > > > > problem but in general a forcing cause to get the users to upgrade
> > > their
> > > > > client version in a multi-tenant environment.
> > > > >
> > > > > "> For #3, it seems simplest to have a config that requires clients to
> > > > > support
> > > > > > a given message format version (or higher). For #2, it seems like
> > > you'd
> > > > > > want clients to advertise their versions. That would be useful for
> > > > > multiple
> > > > > > reasons."
> > > > > This kip offers the ability to block clients based on the protocol 
> > > > > they
> > > > > support. This should be independent of the message format upgrade. Not
> > > all
> > > > > of the features or bugs are dependent on a message format and having a
> > > > > message format dependency to block clients means we have to upgrade to
> > > > > message.format and we cannot just say we've 1.1 brokers with 0.8.2
> > > message
> > > > > format and now we want to block all 0.8.x clients.
> > > > >
> > > > > min.api.version helps at the cluster level to say that all users
> > > required
> > > > 

Apply for contributor permission

[Lifei Chen]

Hi, all:

I am very interested in  kafka streams,  and willing to learn more about it
and  contribute to it when there is appropriate issue for me.

Could someone help to give me the contributor permission?  My jira id is
hustclf .
Thanks a lot.



Best.
Lifei

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Kamal Chandraprakash]

Stanislav,

Any updates on this KIP? We have internal users who want to skip the
corrupted message while consuming the records.


On Fri, Oct 19, 2018 at 11:34 PM Matthias J. Sax 
wrote:

> I am not 100% familiar with the details of the consumer code, however I
> tend to disagree with:
>
> > There's no difference between the two cases -- if (and only if) the
> message is corrupt, it can't be deserialized.  If (and only if) it can't be
> deserialized, it is corrupt.
>
> Assume that a user configures a JSON deserializer but a faulty upstream
> producer writes an Avro message. For this case, the message is not
> corrupted, but still can't be deserialized. And I can imaging that users
> want to handle both cases differently.
>
> Thus, I think it makes sense to have two different exceptions
> `RecordDeserializationException` and `CorruptedRecordException` that can
> both extend `FaultyRecordException` (don't like this name too much
> honestly, but don't have a better idea for it anyway).
>
> Side remark. If we introduce class `RecordDeserializationException` and
> `CorruptedRecordException`, we can also add an interface that both
> implement to return partition/offset information and let both extend
> `SerializationException` directly without an intermediate class in the
> exception hierarchy.
>
>
> -Matthias
>
> On 8/8/18 2:57 AM, Stanislav Kozlovski wrote:
> >> If you are inheriting from SerializationException, your derived class
> > should also be a kind of serialization exception.  Not something more
> > general.
> > Yeah, the reason for inheriting it would be for backwards-compatibility.
> >
> >> Hmm.  Can you think of any new scenarios that would make Kafka force the
> > user need to skip a specific record?  Perhaps one scenario is if records
> > are lost but we don't know how many.
> > Not on the spot, but I do wonder how likely a new scenario is to surface
> in
> > the future and how we'd handle the exceptions' class hierarchy then.
> >
> >> Which offset were we planning to use in the
> > exception?
> > The offset of the record which caused the exception. In the case of
> > batches, we use the last offset of the batch. In both cases, users should
> > have to seek +1 from the given offset. You can review the PR to ensure
> its
> > accurate
> >
> >
> > If both of you prefer `RecordDeserializationException`, we can go with
> > that. Please do confirm that is okay
> >
> > On Tue, Aug 7, 2018 at 11:35 PM Jason Gustafson 
> wrote:
> >
> >> One difference between the two cases is that we can't generally trust
> the
> >> offset of a corrupt message. Which offset were we planning to use in the
> >> exception? Maybe it should be either the fetch offset or one plus the
> last
> >> consumed offset? I think I'm with Colin in preferring
> >> RecordDeserializationException for both cases if possible. For one
> thing,
> >> that makes the behavior consistent whether or not `check.crs` is
> enabled.
> >>
> >> -Jason
> >>
> >> On Tue, Aug 7, 2018 at 11:17 AM, Colin McCabe 
> wrote:
> >>
> >>> Hi Stanislav,
> >>>
> >>> On Sat, Aug 4, 2018, at 10:44, Stanislav Kozlovski wrote:
>  Hey Colin,
> 
>  It may be a bit vague but keep in mind we also raise the exception in
> >> the
>  case where the record is corrupted.
>  We discussed with Jason offline that message corruption most often
> >>> prevents
>  deserialization itself and that may be enough of an argument to raise
>  `RecordDeserializationException` in the case of a corrupt record. I
>  personally find that misleading.
> >>>
> >>> Hmm.  I think that by definition, corrupt records are records that
> can't
> >>> be deserialized.  There's no difference between the two cases -- if
> (and
> >>> only if) the message is corrupt, it can't be deserialized.  If (and
> only
> >>> if) it can't be deserialized, it is corrupt.
> >>>
> 
>  In the end, I think it might be worth it to have a bit of a
>  wider-encompassing `FaultyRecordException` (or even
>  `UnconsumableRecordException`) which would allow users to skip over
> the
>  record.
> >>>
> >>> If you are inheriting from SerializationException, your derived class
> >>> should also be a kind of serialization exception.  Not something more
> >>> general.
> >>>
>  We could then potentially have more specific exceptions (e.g
>  deserialization) inherit that but I'm not sure if we should.
>  A case for a more general exception which provides access to the
>  partition/offset is future backwards-compatibility. If there is ever a
> >>> new
>  scenario that would make the user need to skip a specific record -
> >> there
>  would already be such an exception and this will provide some
>  backward-compatibility with older clients.
> >>>
> >>> Hmm.  Can you think of any new scenarios that would make Kafka force
> the
> >>> user need to skip a specific record?  Perhaps one scenario is if
> records
> >>> are lost but we don't know how many.
> >>>
> >>> If we 

Build failed in Jenkins: kafka-2.1-jdk8 #171

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-8204: fix Streams store flush order (#6555) (#6623)

--
[...truncated 919.79 KB...]
kafka.server.KafkaConfigTest > testFromPropsInvalid STARTED

kafka.server.KafkaConfigTest > testFromPropsInvalid PASSED

kafka.server.KafkaConfigTest > testInvalidCompressionType STARTED

kafka.server.KafkaConfigTest > testInvalidCompressionType PASSED

kafka.server.KafkaConfigTest > testAdvertiseHostNameDefault STARTED

kafka.server.KafkaConfigTest > testAdvertiseHostNameDefault PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeMinutesProvided STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeMinutesProvided PASSED

kafka.server.KafkaConfigTest > testValidCompressionType STARTED

kafka.server.KafkaConfigTest > testValidCompressionType PASSED

kafka.server.KafkaConfigTest > testUncleanElectionInvalid STARTED

kafka.server.KafkaConfigTest > testUncleanElectionInvalid PASSED

kafka.server.KafkaConfigTest > testListenerNamesWithAdvertisedListenerUnset 
STARTED

kafka.server.KafkaConfigTest > testListenerNamesWithAdvertisedListenerUnset 
PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
PASSED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided STARTED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided PASSED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault STARTED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault PASSED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol PASSED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled STARTED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled PASSED

kafka.server.KafkaConfigTest > testInterBrokerVersionMessageFormatCompatibility 
STARTED

kafka.server.KafkaConfigTest > testInterBrokerVersionMessageFormatCompatibility 
PASSED

kafka.server.KafkaConfigTest > testAdvertisePortDefault STARTED

kafka.server.KafkaConfigTest > testAdvertisePortDefault PASSED

kafka.server.KafkaConfigTest > testVersionConfiguration STARTED

kafka.server.KafkaConfigTest > testVersionConfiguration PASSED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol PASSED

kafka.server.ListOffsetsRequestTest > testListOffsetsErrorCodes STARTED

kafka.server.ListOffsetsRequestTest > testListOffsetsErrorCodes PASSED

kafka.server.ListOffsetsRequestTest > testCurrentEpochValidation STARTED

kafka.server.ListOffsetsRequestTest > testCurrentEpochValidation PASSED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testNotController STARTED

kafka.server.CreateTopicsRequestTest > testNotController PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithDownConversionDisabled STARTED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithDownConversionDisabled PASSED

kafka.server.FetchRequestDownConversionConfigTest > testV1FetchFromReplica 
STARTED

kafka.server.FetchRequestDownConversionConfigTest > testV1FetchFromReplica 
PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testLatestFetchWithDownConversionDisabled STARTED

kafka.server.FetchRequestDownConversionConfigTest > 
testLatestFetchWithDownConversionDisabled PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithTopicLevelOverrides STARTED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithTopicLevelOverrides PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest PASSED

kafka.server.SaslApiVersionsRequestTest >