[jira] [Created] (KAFKA-8801) electLeaderForPartitions redundancy for some success elect partitions

[shilin Lu (JIRA)]

shilin Lu created KAFKA-8801:


 Summary: electLeaderForPartitions redundancy for some success 
elect partitions
 Key: KAFKA-8801
 URL: https://issues.apache.org/jira/browse/KAFKA-8801
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.3.0
Reporter: shilin Lu
 Attachments: code.png

!code.png!

This is the code of electLeaderForPartitions. The logic of code is if update 
leaderAndIsr to zk failed,will add the partitions to remaining retry seq. so i 
think the parameters in the red box should change to remainings



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

[Mickael Maison]

Hi Colin,

Thanks for raising these 2 valid points. I've updated the KIP accordingly.

On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang  wrote:
>
> Hi Mickael,
>
> Thanks for the KIP!
>
> Just some minor comments.
>
> 1. Java class names are stale, e.g. "CommitOffsetsOptions.java" should be
> "AlterOffsetsOptions".
>
> 2. I'd suggest we change the future structure of "AlterOffsetsResult" to
>
> *KafkaFuture>>*
>
> This is because we will have a hierarchy of two-layers of errors since we
> need to find out the group coordinator first and then issue the commit
> offset request (see e.g. the ListConsumerGroupOffsetsResult which exclude
> partitions that have errors, or the DeleteMembersResult as part of KIP-345).
>
> If the discover-coordinator returns non-triable error, we would set it on
> the first layer of the KafkaFuture, and the per-partition error would be
> set on the second layer of the KafkaFuture.
>
>
> Guozhang
>
>
> On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe  wrote:
>
> > Hi Mickael,
> >
> > Considering that KIP-496, which adds a way of deleting consumer offsets
> > from AdminClient, looks like it is going to get in, this seems like
> > functionality we should definitely have.
> >
> > For alterConsumerGroupOffsets, is the intention to ignore partitions that
> > are not specified in the map?  If so, we should specify that in the JavaDoc.
> >
> > isolationLevel seems like it should be an enum rather than a string.  The
> > existing enum is in org.apache.kafka.common.requests, so we should probably
> > create a new one which is public in org.apache.kafka.clients.admin.
> >
> > best,
> > Colin
> >
> >
> > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > > Bumping this thread once again
> > >
> > > Ismael, have I answered your questions?
> > > While this has received a few non-binding +1s, no committers have
> > > voted yet. If you have concerns or questions, please let me know.
> > >
> > > Thanks
> > >
> > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > >  wrote:
> > > >
> > > > Bumping this thread as it's been a couple of weeks.
> > > >
> > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > mickael.mai...@gmail.com> wrote:
> > > > >
> > > > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > > > - Having the reset functionality in the AdminClient:
> > > > > The fact we have a command line tool illustrate that this operation
> > is
> > > > > relatively common. I seems valuable to be able to perform this
> > > > > operation directly via a proper API in addition of the CLI tool.
> > > > >
> > > > > - Sending an OffsetCommit directly instead of relying on
> > KafkaConsumer:
> > > > > The KafkaConsumer requires a lot of stuff to commit offsets. Its
> > group
> > > > > cannot change so you need to start a new Consumer every time, that
> > > > > creates new connections and overal sends more requests. Also there
> > are
> > > > > already  a bunch of AdminClient APIs that have logic very close to
> > > > > what needs to be done to send a commit request, keeping the code
> > small
> > > > > and consistent.
> > > > >
> > > > > I've updated the KIP with these details and moved the 2nd part to
> > > > > "Proposed changes" as it's more an implementation detail.
> > > > >
> > > > > I hope this answers your question
> > > > >
> > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma 
> > wrote:
> > > > > >
> > > > > > The KIP doesn't discuss the option of using KafkaConsumer directly
> > as far
> > > > > > as I can tell. We have tried to avoid having the same
> > functionality in
> > > > > > multiple clients so it would be good to explain why this is
> > necessary here
> > > > > > (not saying it isn't).
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > mickael.mai...@gmail.com
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Ryanne for the feedback, all suggestions sounded good,
> > I've
> > > > > > > updated the KIP accordingly.
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > ryannedo...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > But I suggest:
> > > > > > > >
> > > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > > >
> > > > > > > > - add to the motivation section why this is better than
> > constructing a
> > > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > > >
> > > > > > > > - add some rejected alternatives.
> > > > > > > >
> > > > > > > > Ryanne
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee  > wrote:
> > > > > > > >
> > > > > > > > > We have +4 non-binding for this vote. Is there any committer
> > who is
> > > > > > > > > interested in this issue?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Dongjin
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > > 

Build failed in Jenkins: kafka-trunk-jdk11 #752

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: remove unnecessary #remove overrides (#7178)

[github] KAFKA-8791: RocksDBTimestampedStore should open in regular mode by

[ismael] MINOR: Avoid unnecessary leaderFor calls when ProducerBatch queue empty

[github] KAFKA-8765: Remove interface annotations in Streams API (#7174)

[gwen] KAFKA-8792; Default ZK configuration to disable AdminServer

--
[...truncated 2.63 MB...]

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testThrowExceptionInKafkaConsume STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testThrowExceptionInKafkaConsume PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testThrowExceptionInKafkaProduce STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testThrowExceptionInKafkaProduce PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testExecAndHandleRetriableErrorOnce STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testExecAndHandleRetriableErrorOnce PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testExecAndHandleRetriableErrorThrice STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testExecAndHandleRetriableErrorThrice PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testExecAndHandleNonRetriableErrorOnce STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testExecAndHandleNonRetriableErrorOnce PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testExecAndHandleNonRetriableErrorThrice STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testExecAndHandleNonRetriableErrorThrice PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testCheckRetryLimit STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testCheckRetryLimit PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testBackoffLimit STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testBackoffLimit PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testToleranceLimit STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testToleranceLimit PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testDefaultConfigs STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testDefaultConfigs PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testSetConfigs STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testSetConfigs PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testHandleExceptionInTransformations STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testHandleExceptionInTransformations PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testHandleExceptionInHeaderConverter STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testHandleExceptionInHeaderConverter PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testHandleExceptionInValueConverter STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testHandleExceptionInValueConverter PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testHandleExceptionInTaskPut STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testHandleExceptionInTaskPut PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testHandleExceptionInTaskPoll STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testHandleExceptionInTaskPoll PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testThrowExceptionInTaskPut STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testThrowExceptionInTaskPut PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testThrowExceptionInTaskPoll STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testThrowExceptionInTaskPoll PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
initializeDLQWithNullMetrics STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
initializeDLQWithNullMetrics PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testDLQConfigWithEmptyTopicName STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testDLQConfigWithEmptyTopicName PASSED

org.apache.kafka.conne

Re: [DISCUSS] KIP-373: Allow users to create delegation tokens for other users

[Manikumar]

Hi,

Why do we need  new ACL operation  "CreateUsers"?
I think,  "CreateTokens" Operation is sufficient to create "UserA can
create tokens for UserB, UserC" association.

Thanks,

On Tue, Aug 13, 2019 at 3:37 PM Viktor Somogyi-Vass 
wrote:

> Hi Manikumar,
>
> Yea, I just brought up superuser for the sake of simplicity :).
> Anyway, your proposition makes sense to me, I'll modify the KIP for this.
>
> The changes summarized:
> 1. We'll need a new ACL operation as well (say "CreateUsers") to create the
> "UserA can create tokens for UserB, UserC" association. This can be used
> via the createAcls API of the AdminClient.
> 2. CreateToken will be a User level operation (instead of a Cluster level
> as in previous drafts). So that means any user who wants to create a
> delegation token for other users will have to have an ACL set up by a
> higher level user to authorize this.
> 3. DescribeToken will also be a User level operation. In this case tokenT
> owned by userB will be described if userA has a Describe ACL on tokenT or
> has a DescribeToken ACL on userB. Note that in the latter case userA will
> be able to describe all other tokens belonging to userB.
>
> Would this work for you?
>
> Viktor
>
> On Mon, Aug 12, 2019 at 5:45 PM Colin McCabe  wrote:
>
> > +1 for better access control here. In general, impersonating another user
> > seems like it’s equivalent to super user access.
> >
> > Colin
> >
> > On Mon, Aug 12, 2019, at 05:43, Manikumar wrote:
> > > Hi Viktor,
> > >
> > > As per the KIP, It's not only superuser, any user with required
> > permissions
> > > (CreateTokens on Cluster Resource), can create the tokens for other
> > users.
> > > Current proposed permissions defined like, "UserA can create tokens for
> > any
> > > user".
> > > I am thinking, can we change the permissions like "UserA can create
> > tokens
> > > for UserB, UserC"?
> > >
> > > Thanks,
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Aug 9, 2019 at 6:39 PM Viktor Somogyi-Vass <
> > viktorsomo...@gmail.com>
> > > wrote:
> > >
> > > > Hey Manikumar,
> > > >
> > > > Thanks for the feedback.
> > > > I'm not sure I fully grasp the use-case. Would this be a quota? Do we
> > say
> > > > something like "there can be 10 active delegation tokens at a time
> > that is
> > > > created by superuserA for other users"?
> > > > I think such a feature could be useful to limit the responsibility of
> > said
> > > > superuser (and blast radius in case of a faulty/malicious superuser)
> > and
> > > > also to limit potential programming errors. Do you have other use
> cases
> > > > too?
> > > >
> > > > Thanks,
> > > > Viktor
> > > >
> > > >
> > > > On Tue, Aug 6, 2019 at 1:28 PM Manikumar 
> > > > wrote:
> > > >
> > > > > Hi Viktor,
> > > > >
> > > > > Thanks for taking over this KP.
> > > > >
> > > > > Current proposed ACL changes allows users to create tokens for any
> > user.
> > > > > Thinking again about this, admins may want to configure a user to
> > > > > impersonate limited number of other users.
> > > > > This allows us to configure fine-grained permissions. But this
> > requires
> > > > a
> > > > > new resourceType "User". What do you think?
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > > >
> > > > > On Wed, Jul 31, 2019 at 2:26 PM Viktor Somogyi-Vass <
> > > > > viktorsomo...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Folks,
> > > > > >
> > > > > > I'm starting a vote on this.
> > > > > >
> > > > > > Viktor
> > > > > >
> > > > > > On Thu, Jun 27, 2019 at 12:02 PM Viktor Somogyi-Vass <
> > > > > > viktorsomo...@gmail.com> wrote:
> > > > > >
> > > > > > > Hi Folks,
> > > > > > >
> > > > > > > I took over this issue from Manikumar. Recently another
> > motivation
> > > > have
> > > > > > > been raised in Spark for this (SPARK-28173) and I think it'd be
> > great
> > > > > to
> > > > > > > continue this task.
> > > > > > > I updated the KIP and will wait for a few days to get some
> > feedback
> > > > > then
> > > > > > > proceed for the vote.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Viktor
> > > > > > >
> > > > > > > On Tue, Dec 11, 2018 at 8:29 AM Manikumar <
> > manikumar.re...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Harsha,
> > > > > > >>
> > > > > > >> Thanks for the review.
> > > > > > >>
> > > > > > >> With this KIP a designated superuser can create tokens without
> > > > > requiring
> > > > > > >> individual user credentials.
> > > > > > >> Any client can authenticate brokers using the created tokens.
> > We may
> > > > > not
> > > > > > >> call this as impersonation,
> > > > > > >> since the clients API calls are executing on their own
> > authenticated
> > > > > > >> connections.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Manikumar
> > > > > > >>
> > > > > > >> On Fri, Dec 7, 2018 at 11:56 PM Harsha 
> wrote:
> > > > > > >>
> > > > > > >> > Hi Mani,
> > > > > > >> > Overall KIP looks good to me. Can we call this
> > > > > > >> Impersonation
>

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

[Mickael Maison]

Hi Guozhang,

Thanks for taking a look.

1. Right, I updated the titles of the code blocks

2. Yes that's a good idea. I've updated the KIP

Thank you

On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
 wrote:
>
> Hi Colin,
>
> Thanks for raising these 2 valid points. I've updated the KIP accordingly.
>
> On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang  wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the KIP!
> >
> > Just some minor comments.
> >
> > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java" should be
> > "AlterOffsetsOptions".
> >
> > 2. I'd suggest we change the future structure of "AlterOffsetsResult" to
> >
> > *KafkaFuture>>*
> >
> > This is because we will have a hierarchy of two-layers of errors since we
> > need to find out the group coordinator first and then issue the commit
> > offset request (see e.g. the ListConsumerGroupOffsetsResult which exclude
> > partitions that have errors, or the DeleteMembersResult as part of KIP-345).
> >
> > If the discover-coordinator returns non-triable error, we would set it on
> > the first layer of the KafkaFuture, and the per-partition error would be
> > set on the second layer of the KafkaFuture.
> >
> >
> > Guozhang
> >
> >
> > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe  wrote:
> >
> > > Hi Mickael,
> > >
> > > Considering that KIP-496, which adds a way of deleting consumer offsets
> > > from AdminClient, looks like it is going to get in, this seems like
> > > functionality we should definitely have.
> > >
> > > For alterConsumerGroupOffsets, is the intention to ignore partitions that
> > > are not specified in the map?  If so, we should specify that in the 
> > > JavaDoc.
> > >
> > > isolationLevel seems like it should be an enum rather than a string.  The
> > > existing enum is in org.apache.kafka.common.requests, so we should 
> > > probably
> > > create a new one which is public in org.apache.kafka.clients.admin.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > > > Bumping this thread once again
> > > >
> > > > Ismael, have I answered your questions?
> > > > While this has received a few non-binding +1s, no committers have
> > > > voted yet. If you have concerns or questions, please let me know.
> > > >
> > > > Thanks
> > > >
> > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > >  wrote:
> > > > >
> > > > > Bumping this thread as it's been a couple of weeks.
> > > > >
> > > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > > mickael.mai...@gmail.com> wrote:
> > > > > >
> > > > > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > > > > - Having the reset functionality in the AdminClient:
> > > > > > The fact we have a command line tool illustrate that this operation
> > > is
> > > > > > relatively common. I seems valuable to be able to perform this
> > > > > > operation directly via a proper API in addition of the CLI tool.
> > > > > >
> > > > > > - Sending an OffsetCommit directly instead of relying on
> > > KafkaConsumer:
> > > > > > The KafkaConsumer requires a lot of stuff to commit offsets. Its
> > > group
> > > > > > cannot change so you need to start a new Consumer every time, that
> > > > > > creates new connections and overal sends more requests. Also there
> > > are
> > > > > > already  a bunch of AdminClient APIs that have logic very close to
> > > > > > what needs to be done to send a commit request, keeping the code
> > > small
> > > > > > and consistent.
> > > > > >
> > > > > > I've updated the KIP with these details and moved the 2nd part to
> > > > > > "Proposed changes" as it's more an implementation detail.
> > > > > >
> > > > > > I hope this answers your question
> > > > > >
> > > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma 
> > > wrote:
> > > > > > >
> > > > > > > The KIP doesn't discuss the option of using KafkaConsumer directly
> > > as far
> > > > > > > as I can tell. We have tried to avoid having the same
> > > functionality in
> > > > > > > multiple clients so it would be good to explain why this is
> > > necessary here
> > > > > > > (not saying it isn't).
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > > mickael.mai...@gmail.com
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Ryanne for the feedback, all suggestions sounded good,
> > > I've
> > > > > > > > updated the KIP accordingly.
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > > ryannedo...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > +1 (non-binding)
> > > > > > > > >
> > > > > > > > > But I suggest:
> > > > > > > > >
> > > > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > > > >
> > > > > > > > > - add to the motivation section why this is better than
> > > constructing a
> > > > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > > > >
> > > > > > > > > - add some rejected alternatives.
> >

Build failed in Jenkins: kafka-trunk-jdk8 #3850

[Apache Jenkins Server]

See 


Changes:

[bbejeck] KAFKA-8736: Track size in InMemoryKeyValueStore (#7177)

[wangguoz] MINOR: remove unnecessary #remove overrides (#7178)

[github] KAFKA-8791: RocksDBTimestampedStore should open in regular mode by

--
[...truncated 9.02 MB...]
kafka.api.GroupAuthorizerIntegrationTest > 
testTransactionalProducerInitTransactionsNoWriteTransactionalIdAcl STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testTransactionalProducerInitTransactionsNoWriteTransactionalIdAcl PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testIncrementalAlterConfigsRequestRequiresClusterPermissionForBrokerLogger 
STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testIncrementalAlterConfigsRequestRequiresClusterPermissionForBrokerLogger 
PASSED

kafka.api.GroupAuthorizerIntegrationTest > testOffsetFetchWithNoGroupAccess 
STARTED

kafka.api.GroupAuthorizerIntegrationTest > testOffsetFetchWithNoGroupAccess 
PASSED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithNoAccess STARTED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithNoAccess PASSED

kafka.api.GroupAuthorizerIntegrationTest > testUnauthorizedCreatePartitions 
STARTED

kafka.api.GroupAuthorizerIntegrationTest > testUnauthorizedCreatePartitions 
PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testConsumeWithoutTopicDescribeAccess STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testConsumeWithoutTopicDescribeAccess PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
shouldThrowTransactionalIdAuthorizationExceptionWhenNoTransactionAccessOnEndTransaction
 STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
shouldThrowTransactionalIdAuthorizationExceptionWhenNoTransactionAccessOnEndTransaction
 PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
shouldThrowTransactionalIdAuthorizationExceptionWhenNoTransactionAccessOnSendOffsetsToTxn
 STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
shouldThrowTransactionalIdAuthorizationExceptionWhenNoTransactionAccessOnSendOffsetsToTxn
 PASSED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithNoGroupAccess STARTED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithNoGroupAccess PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testTransactionalProducerInitTransactionsNoDescribeTransactionalIdAcl STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testTransactionalProducerInitTransactionsNoDescribeTransactionalIdAcl PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testUnauthorizedDeleteRecordsWithDescribe STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testUnauthorizedDeleteRecordsWithDescribe PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testCreateTopicAuthorizationWithClusterCreate STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testCreateTopicAuthorizationWithClusterCreate PASSED

kafka.api.GroupAuthorizerIntegrationTest > testOffsetFetchWithTopicAndGroupRead 
STARTED

kafka.api.GroupAuthorizerIntegrationTest > testOffsetFetchWithTopicAndGroupRead 
PASSED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithTopicDescribe STARTED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithTopicDescribe PASSED

kafka.api.GroupAuthorizerIntegrationTest > testAuthorizationWithTopicExisting 
STARTED

kafka.api.GroupAuthorizerIntegrationTest > testAuthorizationWithTopicExisting 
PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testUnauthorizedDeleteRecordsWithoutDescribe STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testUnauthorizedDeleteRecordsWithoutDescribe PASSED

kafka.api.GroupAuthorizerIntegrationTest > testMetadataWithTopicDescribe STARTED

kafka.api.GroupAuthorizerIntegrationTest > testMetadataWithTopicDescribe PASSED

kafka.api.GroupAuthorizerIntegrationTest > testProduceWithTopicDescribe STARTED

kafka.api.GroupAuthorizerIntegrationTest > testProduceWithTopicDescribe PASSED

kafka.api.GroupAuthorizerIntegrationTest > testDescribeGroupApiWithNoGroupAcl 
STARTED

kafka.api.GroupAuthorizerIntegrationTest > testDescribeGroupApiWithNoGroupAcl 
PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testPatternSubscriptionMatchingInternalTopic STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testPatternSubscriptionMatchingInternalTopic PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testSendOffsetsWithNoConsumerGroupDescribeAccess STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testSendOffsetsWithNoConsumerGroupDescribeAccess PASSED

kafka.api.GroupAuthorizerIntegrationTest > testOffsetFetchTopicDescribe STARTED

kafka.api.GroupAuthorizerIntegrationTest > testOffsetFetchTopicDescribe PASSED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithTopicAndGroupRead 
STARTED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithTopicAndGroupRead 
PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testIdempotentProducerNoIdempotentWriteAclInInitProducerId STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
tes

Re: [DISCUSS] KIP-373: Allow users to create delegation tokens for other users

[Viktor Somogyi-Vass]

Sorry, reading my email the second time I probably wasn't clear.
So basically the concept is that there is a user who can add other users as
resources (such as userB and userC) prior to creating the "userA can create
delegation token for userB and userC" association with CreateTokens. To
limit who can add new users as resources I thought we can introduce a
CreateUser operation. It's true though that we could also say that a Create
operation permission on the Cluster resource would be enough to create new
users but I think from a generic security perspective it's better if we
don't extend already existing operations.
So a classic flow would be that prior to creating the delegation token for
userB, userB itself has to be added by another user who has CreateUser
permissions. After this a CreateToken permission has to be created that
says "userA can create delegation tokens for userB" and after this userA
can actually create the token.
Let me know what you think.

Viktor

On Wed, Aug 14, 2019 at 1:30 PM Manikumar  wrote:

> Hi,
>
> Why do we need  new ACL operation  "CreateUsers"?
> I think,  "CreateTokens" Operation is sufficient to create "UserA can
> create tokens for UserB, UserC" association.
>
> Thanks,
>
> On Tue, Aug 13, 2019 at 3:37 PM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com>
> wrote:
>
> > Hi Manikumar,
> >
> > Yea, I just brought up superuser for the sake of simplicity :).
> > Anyway, your proposition makes sense to me, I'll modify the KIP for this.
> >
> > The changes summarized:
> > 1. We'll need a new ACL operation as well (say "CreateUsers") to create
> the
> > "UserA can create tokens for UserB, UserC" association. This can be used
> > via the createAcls API of the AdminClient.
> > 2. CreateToken will be a User level operation (instead of a Cluster level
> > as in previous drafts). So that means any user who wants to create a
> > delegation token for other users will have to have an ACL set up by a
> > higher level user to authorize this.
> > 3. DescribeToken will also be a User level operation. In this case tokenT
> > owned by userB will be described if userA has a Describe ACL on tokenT or
> > has a DescribeToken ACL on userB. Note that in the latter case userA will
> > be able to describe all other tokens belonging to userB.
> >
> > Would this work for you?
> >
> > Viktor
> >
> > On Mon, Aug 12, 2019 at 5:45 PM Colin McCabe  wrote:
> >
> > > +1 for better access control here. In general, impersonating another
> user
> > > seems like it’s equivalent to super user access.
> > >
> > > Colin
> > >
> > > On Mon, Aug 12, 2019, at 05:43, Manikumar wrote:
> > > > Hi Viktor,
> > > >
> > > > As per the KIP, It's not only superuser, any user with required
> > > permissions
> > > > (CreateTokens on Cluster Resource), can create the tokens for other
> > > users.
> > > > Current proposed permissions defined like, "UserA can create tokens
> for
> > > any
> > > > user".
> > > > I am thinking, can we change the permissions like "UserA can create
> > > tokens
> > > > for UserB, UserC"?
> > > >
> > > > Thanks,
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Aug 9, 2019 at 6:39 PM Viktor Somogyi-Vass <
> > > viktorsomo...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hey Manikumar,
> > > > >
> > > > > Thanks for the feedback.
> > > > > I'm not sure I fully grasp the use-case. Would this be a quota? Do
> we
> > > say
> > > > > something like "there can be 10 active delegation tokens at a time
> > > that is
> > > > > created by superuserA for other users"?
> > > > > I think such a feature could be useful to limit the responsibility
> of
> > > said
> > > > > superuser (and blast radius in case of a faulty/malicious
> superuser)
> > > and
> > > > > also to limit potential programming errors. Do you have other use
> > cases
> > > > > too?
> > > > >
> > > > > Thanks,
> > > > > Viktor
> > > > >
> > > > >
> > > > > On Tue, Aug 6, 2019 at 1:28 PM Manikumar <
> manikumar.re...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Viktor,
> > > > > >
> > > > > > Thanks for taking over this KP.
> > > > > >
> > > > > > Current proposed ACL changes allows users to create tokens for
> any
> > > user.
> > > > > > Thinking again about this, admins may want to configure a user to
> > > > > > impersonate limited number of other users.
> > > > > > This allows us to configure fine-grained permissions. But this
> > > requires
> > > > > a
> > > > > > new resourceType "User". What do you think?
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Manikumar
> > > > > >
> > > > > >
> > > > > > On Wed, Jul 31, 2019 at 2:26 PM Viktor Somogyi-Vass <
> > > > > > viktorsomo...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Folks,
> > > > > > >
> > > > > > > I'm starting a vote on this.
> > > > > > >
> > > > > > > Viktor
> > > > > > >
> > > > > > > On Thu, Jun 27, 2019 at 12:02 PM Viktor Somogyi-Vass <
> > > > > > > viktorsomo...@gmail.com> wrote:
> > > > > > >
> > > > > > > > Hi Folks,
> > > > > > > >
> > 

Re: [VOTE] KIP-503: deleted topics metric

[Mickael Maison]

+1 non binding
Thank you!

On Tue, Aug 13, 2019 at 9:07 PM Stanislav Kozlovski
 wrote:
>
> +1 (non-binding)
>
> Thanks for the simple but very useful KIP!
> Best,
> Stanislav
>
> On Tue, Aug 13, 2019 at 8:32 PM Harsha Chintalapani  wrote:
>
> > +1 (binding)
> >
> > Thanks,
> > Harsha
> >
> >
> > On Tue, Aug 13, 2019 at 12:08 PM, David Arthur 
> > wrote:
> >
> > > Hello all,
> > >
> > > I'd like to start the vote on KIP-503
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion
> > >
> > > Thanks!
> > > David
> > >
> >
>
>
> --
> Best,
> Stanislav

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Mickael Maison]

Hi Rajini,

Thanks for the KIP!
I really like that authorize() will be able to take a batch of
requests, this will speed up many implementations!

On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram  wrote:
>
> Thanks David! I have fixed the typo.
>
> Also made a couple of changes to make the context interfaces more generic.
> KafkaRequestContext now returns the 16-bit API key as Colin suggested as
> well as the friendly name used in metrics which are useful in audit logs.
> `Authorizer#start` is now provided a generic `BrokerInfo` interface that
> gives cluster id, broker id and endpoint information. The generic interface
> can potentially be used in other broker plugins in future and provides
> dynamically generated configs like broker id and ports which are currently
> not available to plugins unless these configs are statically configured.
> Please let me know if there are any concerns.
>
> Regards,
>
> Rajini
>
> On Tue, Aug 13, 2019 at 4:30 PM David Jacot  wrote:
>
> > Hi Rajini,
> >
> > Thank you for the update! It looks good to me. There is a typo in the
> > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` -> `MANDATORY_AUTHORIZE`.
> >
> > Regards,
> > David
> >
> > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram 
> > wrote:
> >
> > > Hi David,
> > >
> > > Thanks for reviewing the KIP! Since questions about `authorization mode`
> > > and `count` have come up multiple times, I have renamed both.
> > >
> > > 1) Renamed `count` to `resourceReferenceCount`. It is the number of times
> > > the resource being authorized is referenced within the request.
> > >
> > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided to improve
> > > audit logging in the authorizer. The enum values have javadoc which
> > > indicate how the authorization result is used in each of the modes to
> > > enable authorizers to log audit messages at the right severity level.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot  wrote:
> > >
> > > > Hi Rajini,
> > > >
> > > > Thank you for the KIP. Overall, it looks good to me. I have few
> > > > questions/suggestions:
> > > >
> > > > 1. It is hard to grasp what `Action#count` is for. I guess I understand
> > > > where you want to go with it but it took me a while to figure it out.
> > > > Perhaps, we could come up with a better name than `count`?
> > > >
> > > > 2. I had a hard time trying to understand the `AuthorizationMode`
> > > concept,
> > > > especially wrt. the OPTIONAL one. My understanding is that an ACL is
> > > either
> > > > defined or not. Could you elaborate a bit more on that?
> > > >
> > > > Thanks,
> > > > David
> > > >
> > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai 
> > > wrote:
> > > >
> > > > > Hi Rajini
> > > > >
> > > > > 3.2 - This makes sense. Thanks for clarifying.
> > > > >
> > > > > Rest looks fine. Once the implementations are done, it will be more
> > > clear
> > > > > on the different values RequestType and Mode.
> > > > >
> > > > > Thanks
> > > > >
> > > > > Bosco
> > > > >
> > > > >
> > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" 
> > wrote:
> > > > >
> > > > > Hi Don,
> > > > >
> > > > > Thanks for the suggestions. A few responses below:
> > > > >
> > > > > 3.1 Can rename and improve docs if we keep this. Let's finish the
> > > > > discussion on Colin's suggestions regarding this first.
> > > > > 3.2 No, I was thinking of some requests that have an old way of
> > > > > authorizing
> > > > > and a new way where we have retained the old way for backward
> > > > > compatibility. One example is Cluster:Create permission to create
> > > > > topics.
> > > > > We have replaced this with fine-grained topic create access using
> > > > > Topic:Create
> > > > > for topic patterns. But we still check if user has Cluster:Create
> > > > > first. If
> > > > > Denied, the deny is ignored and we check Topic:Create. We dont
> > want
> > > > to
> > > > > log
> > > > > DENY for Cluster:Create at INFO level for this, since this is
> > not a
> > > > > mandatory ACL for creating topics. We will get appropriate logs
> > > from
> > > > > the
> > > > > subsequent Topic:Create in this case.
> > > > > 3.3 They are not quite the same. FILTER implies that user
> > actually
> > > > > requested access to and performed some operation on the filtered
> > > > > resources.
> > > > > LIST_AUTHORZED did not result in any actual access. User simply
> > > > wanted
> > > > > to
> > > > > know what they are allowed to access.
> > > > > 3.4 Request types are Produce, JoinGroup, OffsetCommit etc. So
> > that
> > > > is
> > > > > different from authorization mode, operation etc.
> > > > >
> > > > >
> > > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <
> > bo...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Rajini
> > > > > >
> > > > > > Thanks for clarifying. This is very helpful...
> > > > > >
> > > > > > #1 - On the Ranger side, we

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

[Konstantine Karantasis]

Thanks Arjun for tackling the need to support this very useful feature.

One thing I noticed while reading the KIP is that I would have loved to see
more info regarding how this proposal depends on the underlying logging
APIs and implementations. For instance, my understanding is that slf4j can
not be leveraged and that the logging framework needs to be pegged to log4j
explicitly (or another logging implementation). Correct me if I'm wrong,
but if such a dependency is introduced I believe it's worth mentioning.

Additionally, if the above is correct, there are differences in log4j's
APIs between version 1 and version 2. In version 2, Logger#setLevel method
has been removed from the Logger interface and in order to set the log
level programmatically the Configurator class needs to used, which as
stated in the FAQ (
https://logging.apache.org/log4j/2.x/faq.html#reconfig_level_from_code)
it's not part of log4j2's public API. Is this a concern? I believe that
even if these are implementation specific details for the wrappers
introduced by this KIP (which to a certain extent they are), a mention in
the KIP text and a few references would be useful to understand the changes
and the dependencies introduced by this proposal.

And a few minor comments:
- Is there any specific reason that object types were preferred in the
proposed interface compared to primitive types? My understanding is that
`null` is not expected as a return value.
- Related to the above, I think it'd be nice for the javadoc to mention
when a parameter is not expected to be `null` with an appropriate comment
(e.g. foo bar etc; may not be null)

Cheers,
Konstantine

On Tue, Aug 6, 2019 at 9:34 AM Cyrus Vafadari  wrote:

> This looks like a useful feature, the strategy makes sense, and the KIP is
> thorough and nicely written. Thanks!
>
> Cyrus
>
> On Thu, Aug 1, 2019, 12:40 PM Chris Egerton  wrote:
>
> > Thanks Arjun! Looks good to me.
> >
> > On Thu, Aug 1, 2019 at 12:33 PM Arjun Satish 
> > wrote:
> >
> > > Thanks for the feedback, Chris!
> > >
> > > Yes, the example is pretty much how Connect will use the new feature.
> > > Tweaked the section to make this more clear.
> > >
> > > Best,
> > >
> > > On Fri, Jul 26, 2019 at 11:52 AM Chris Egerton 
> > > wrote:
> > >
> > > > Hi Arjun,
> > > >
> > > > This looks great. The changes to public interface are pretty small
> and
> > > > moving the Log4jController class into the clients package seems like
> > the
> > > > right way to go. One question I have--it looks like the purpose of
> this
> > > KIP
> > > > is to enable dynamic setting of log levels in the Connect framework,
> > but
> > > > it's not clear how the Connect framework will use that new utility.
> Is
> > > the
> > > > "Example Usage" section (which involves invoking the utility with a
> > > > namespace of "kafka.connect") actually meant to be part of the
> proposed
> > > > changes to public interface?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Mon, Jul 22, 2019 at 11:03 PM Arjun Satish <
> arjun.sat...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi everyone.
> > > > >
> > > > > I'd like to propose the following KIP to implement changing log
> > levels
> > > on
> > > > > the fly in Connect workers:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect
> > > > >
> > > > > Would like to hear your thoughts on this.
> > > > >
> > > > > Thanks very much,
> > > > > Arjun
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

[Konstantine Karantasis]

And one thing I forgot is also related to Chris's comment above. I agree
that an example on how a user is expected to set the log level (for
instance to DEBUG) would be nice, even if it's showing only one out of the
many possible ways to achieve that.

- Konstantine

On Wed, Aug 14, 2019 at 4:38 PM Konstantine Karantasis <
konstant...@confluent.io> wrote:

>
> Thanks Arjun for tackling the need to support this very useful feature.
>
> One thing I noticed while reading the KIP is that I would have loved to
> see more info regarding how this proposal depends on the underlying logging
> APIs and implementations. For instance, my understanding is that slf4j can
> not be leveraged and that the logging framework needs to be pegged to log4j
> explicitly (or another logging implementation). Correct me if I'm wrong,
> but if such a dependency is introduced I believe it's worth mentioning.
>
> Additionally, if the above is correct, there are differences in log4j's
> APIs between version 1 and version 2. In version 2, Logger#setLevel method
> has been removed from the Logger interface and in order to set the log
> level programmatically the Configurator class needs to used, which as
> stated in the FAQ (
> https://logging.apache.org/log4j/2.x/faq.html#reconfig_level_from_code)
> it's not part of log4j2's public API. Is this a concern? I believe that
> even if these are implementation specific details for the wrappers
> introduced by this KIP (which to a certain extent they are), a mention in
> the KIP text and a few references would be useful to understand the changes
> and the dependencies introduced by this proposal.
>
> And a few minor comments:
> - Is there any specific reason that object types were preferred in the
> proposed interface compared to primitive types? My understanding is that
> `null` is not expected as a return value.
> - Related to the above, I think it'd be nice for the javadoc to mention
> when a parameter is not expected to be `null` with an appropriate comment
> (e.g. foo bar etc; may not be null)
>
> Cheers,
> Konstantine
>
> On Tue, Aug 6, 2019 at 9:34 AM Cyrus Vafadari  wrote:
>
>> This looks like a useful feature, the strategy makes sense, and the KIP is
>> thorough and nicely written. Thanks!
>>
>> Cyrus
>>
>> On Thu, Aug 1, 2019, 12:40 PM Chris Egerton  wrote:
>>
>> > Thanks Arjun! Looks good to me.
>> >
>> > On Thu, Aug 1, 2019 at 12:33 PM Arjun Satish 
>> > wrote:
>> >
>> > > Thanks for the feedback, Chris!
>> > >
>> > > Yes, the example is pretty much how Connect will use the new feature.
>> > > Tweaked the section to make this more clear.
>> > >
>> > > Best,
>> > >
>> > > On Fri, Jul 26, 2019 at 11:52 AM Chris Egerton 
>> > > wrote:
>> > >
>> > > > Hi Arjun,
>> > > >
>> > > > This looks great. The changes to public interface are pretty small
>> and
>> > > > moving the Log4jController class into the clients package seems like
>> > the
>> > > > right way to go. One question I have--it looks like the purpose of
>> this
>> > > KIP
>> > > > is to enable dynamic setting of log levels in the Connect framework,
>> > but
>> > > > it's not clear how the Connect framework will use that new utility.
>> Is
>> > > the
>> > > > "Example Usage" section (which involves invoking the utility with a
>> > > > namespace of "kafka.connect") actually meant to be part of the
>> proposed
>> > > > changes to public interface?
>> > > >
>> > > > Cheers,
>> > > >
>> > > > Chris
>> > > >
>> > > > On Mon, Jul 22, 2019 at 11:03 PM Arjun Satish <
>> arjun.sat...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi everyone.
>> > > > >
>> > > > > I'd like to propose the following KIP to implement changing log
>> > levels
>> > > on
>> > > > > the fly in Connect workers:
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect
>> > > > >
>> > > > > Would like to hear your thoughts on this.
>> > > > >
>> > > > > Thanks very much,
>> > > > > Arjun
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [VOTE] KIP-495: Dynamically Adjust Log Levels in Connect

[Konstantine Karantasis]

Thanks for the KIP Arjun.

FYI, I left a few comments on the discussion thread, but mentioning here
too since I noticed that the vote started a few hours ago.

Konstantine


On Wed, Aug 14, 2019 at 12:13 AM Cyrus Vafadari  wrote:

> I am excited to see this implemented +1 nonbinding
>
> On Tue, Aug 13, 2019 at 2:01 PM Chris Egerton  wrote:
>
> > Nice stuff, Arjun! +1 (non-binding)
> >
> > On Tue, Aug 13, 2019 at 1:55 PM Arjun Satish 
> > wrote:
> >
> > > Hey everyone,
> > >
> > > I'd like to start a vote for KIP-495 (
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect
> > > ).
> > > This change will make Connect easier to debug in production
> environment.
> > >
> > > Based on the discussion, I updated the KIP to reflect how Connect will
> > use
> > > the changes to the log4j controller to initialize its mBean.
> > >
> > > For your reference, this is the discussion thread
> > > https://www.mail-archive.com/dev@kafka.apache.org/msg99656.html
> > >
> > > Thanks in advance,
> > > Arjun
> > >
> >
>

Build failed in Jenkins: kafka-trunk-jdk8 #3851

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Avoid unnecessary leaderFor calls when ProducerBatch queue empty

[github] KAFKA-8765: Remove interface annotations in Streams API (#7174)

[gwen] KAFKA-8792; Default ZK configuration to disable AdminServer

--
[...truncated 2.59 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

Re: [VOTE] KIP-497: Add inter-broker API to alter ISR

[David Arthur]

+1 binding, this looks great!

-David

On Tue, Aug 13, 2019 at 4:55 PM Guozhang Wang  wrote:

> +1 (binding). This is a great KIP, thanks Jason!
>
> Regarding the naming of the zkVersion, I'm actually fine to name it more
> generally and leave a note that at the moment its value is defined as the
> zk version.
>
>
> Guozhang
>
>
> On Mon, Aug 12, 2019 at 2:22 PM Jason Gustafson 
> wrote:
>
> > Hi Viktor,
> >
> > I originally named the field `CurrentVersion`. I didn't have 'Zk' in the
> > name in anticipation of KIP-500. I thought about it and decided it makes
> > sense to keep naming consistent with other APIs. Even if KIP-500 passes,
> > there will be some time during which it only refers to the zk version.
> > Eventually we'll have to decide whether it makes sense to change the name
> > or just introduce a new field.
> >
> > Thanks,
> > Jason
> >
> > On Fri, Aug 9, 2019 at 9:19 AM Viktor Somogyi-Vass <
> > viktorsomo...@gmail.com>
> > wrote:
> >
> > > Hey Jason,
> > >
> > > +1 from me too.
> > > One note though: since it's a new protocol we could perhaps rename
> > > CurrentZkVersion to something like "IsrEpoch" or "IsrVersion". I think
> > > that'd reflect its purpose better.
> > >
> > > Best,
> > > Viktor
> > >
> > > On Wed, Aug 7, 2019 at 8:37 PM Jason Gustafson 
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I'd like to start a vote on KIP-497:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-497%3A+Add+inter-broker+API+to+alter+ISR
> > > > .
> > > > +1
> > > > from me.
> > > >
> > > > -Jason
> > > >
> > >
> >
>
>
> --
> -- Guozhang
>


-- 
David Arthur

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

[Jungtaek Lim]

+1 (non-binding)

I found it very useful for Spark's case. (Discussion on KIP-505 described
it.)

Thanks for driving the effort!

2019년 8월 14일 (수) 오후 8:49, Mickael Maison 님이 작성:

> Hi Guozhang,
>
> Thanks for taking a look.
>
> 1. Right, I updated the titles of the code blocks
>
> 2. Yes that's a good idea. I've updated the KIP
>
> Thank you
>
> On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
>  wrote:
> >
> > Hi Colin,
> >
> > Thanks for raising these 2 valid points. I've updated the KIP
> accordingly.
> >
> > On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang 
> wrote:
> > >
> > > Hi Mickael,
> > >
> > > Thanks for the KIP!
> > >
> > > Just some minor comments.
> > >
> > > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java" should
> be
> > > "AlterOffsetsOptions".
> > >
> > > 2. I'd suggest we change the future structure of "AlterOffsetsResult"
> to
> > >
> > > *KafkaFuture>>*
> > >
> > > This is because we will have a hierarchy of two-layers of errors since
> we
> > > need to find out the group coordinator first and then issue the commit
> > > offset request (see e.g. the ListConsumerGroupOffsetsResult which
> exclude
> > > partitions that have errors, or the DeleteMembersResult as part of
> KIP-345).
> > >
> > > If the discover-coordinator returns non-triable error, we would set it
> on
> > > the first layer of the KafkaFuture, and the per-partition error would
> be
> > > set on the second layer of the KafkaFuture.
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe 
> wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > Considering that KIP-496, which adds a way of deleting consumer
> offsets
> > > > from AdminClient, looks like it is going to get in, this seems like
> > > > functionality we should definitely have.
> > > >
> > > > For alterConsumerGroupOffsets, is the intention to ignore partitions
> that
> > > > are not specified in the map?  If so, we should specify that in the
> JavaDoc.
> > > >
> > > > isolationLevel seems like it should be an enum rather than a
> string.  The
> > > > existing enum is in org.apache.kafka.common.requests, so we should
> probably
> > > > create a new one which is public in org.apache.kafka.clients.admin.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > > > > Bumping this thread once again
> > > > >
> > > > > Ismael, have I answered your questions?
> > > > > While this has received a few non-binding +1s, no committers have
> > > > > voted yet. If you have concerns or questions, please let me know.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > > >  wrote:
> > > > > >
> > > > > > Bumping this thread as it's been a couple of weeks.
> > > > > >
> > > > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > > > mickael.mai...@gmail.com> wrote:
> > > > > > >
> > > > > > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > > > > > - Having the reset functionality in the AdminClient:
> > > > > > > The fact we have a command line tool illustrate that this
> operation
> > > > is
> > > > > > > relatively common. I seems valuable to be able to perform this
> > > > > > > operation directly via a proper API in addition of the CLI
> tool.
> > > > > > >
> > > > > > > - Sending an OffsetCommit directly instead of relying on
> > > > KafkaConsumer:
> > > > > > > The KafkaConsumer requires a lot of stuff to commit offsets.
> Its
> > > > group
> > > > > > > cannot change so you need to start a new Consumer every time,
> that
> > > > > > > creates new connections and overal sends more requests. Also
> there
> > > > are
> > > > > > > already  a bunch of AdminClient APIs that have logic very
> close to
> > > > > > > what needs to be done to send a commit request, keeping the
> code
> > > > small
> > > > > > > and consistent.
> > > > > > >
> > > > > > > I've updated the KIP with these details and moved the 2nd part
> to
> > > > > > > "Proposed changes" as it's more an implementation detail.
> > > > > > >
> > > > > > > I hope this answers your question
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma  >
> > > > wrote:
> > > > > > > >
> > > > > > > > The KIP doesn't discuss the option of using KafkaConsumer
> directly
> > > > as far
> > > > > > > > as I can tell. We have tried to avoid having the same
> > > > functionality in
> > > > > > > > multiple clients so it would be good to explain why this is
> > > > necessary here
> > > > > > > > (not saying it isn't).
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > > > mickael.mai...@gmail.com
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Ryanne for the feedback, all suggestions sounded
> good,
> > > > I've
> > > > > > > > > updated the KIP accordingly.
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > > > ryannedo...

Re: [DISCUSS] KIP-481: SerDe Improvements for Connect Decimal type in JSON

[Konstantine Karantasis]

Thanks Almog for preparing this KIP!
I think it will improve usability and troubleshooting with JSON data a lot.

The finalized plan seems quite concrete now. I also liked that some
implementation specific implications (such as setting the ObjectMapper to
deserialize floating point as BigDecimal) are highlighted in the KIP.

Still, as I was reading the KIP, the main obstacle I encountered was around
terminology. I couldn't get used to reading "producer" and "consumer" and
not thinking in terms of Kafka producers and consumers - which are not
relevant to what this KIP proposes. Thus, I'd suggest replacing
"Producer(s)" with "Source Converter(s)" and "Consumer(s)" with "Sink
Converter(s)" (even if "Converter used by Source Connector" or "Converter
used by Sink Connector" would be even more accurate - maybe this could be
an explanation in a footnote). Terminology around converters has been
tricky in the past and adding producers/consumers in the mix might add to
the confusion.

Another example where I'd apply this different terminology would be to a
phrase such as the following:
"Because of this, users must take care to first ensure that all consumers
have upgraded to the new code before upgrading producers to make use of the
NUMERIC serialization format."
which I'd write
"Because of this, users must take care to first ensure that all sink
connectors have upgraded to the new converter code before upgrading source
connectors to make use of the NUMERIC serialization format in
JsonConverter."

Let me know if you think this suggestion makes the KIP easier to follow.
Otherwise I think it's a solid proposal.

I'm concluding with a couple of nits:

- "Upgraded Producer with BASE64 serialization, Legacy Consumer: this
scenario is okay as the upgraded ~producer~ consumer will be able to read
binary as today" (again according to my suggestion above, it could be as
the upgraded source converter ...)

- "consumers cannot consumer NUMERIC data. " -> "consumers cannot read
NUMERIC data"

Best,
Konstantine

On Fri, Aug 9, 2019 at 6:37 PM Almog Gavra  wrote:

> Good catches! Fixed :)
>
> On Thu, Aug 8, 2019 at 10:36 PM Arjun Satish 
> wrote:
>
> > Cool!
> >
> > Couple of nits:
> >
> > - In public interfaces, typo: *json.decimal.serialization.fromat*
> > - In public interfaces, you use the term "HEX" instead of "BASE64".
> >
> >
> >
> > On Wed, Aug 7, 2019 at 9:51 AM Almog Gavra  wrote:
> >
> > > EDIT: everywhere I've been using "HEX" I meant to be using "BASE64". I
> > will
> > > update the KIP to reflect this.
> > >
> > > On Wed, Aug 7, 2019 at 9:44 AM Almog Gavra  wrote:
> > >
> > > > Thanks for the feedback Arjun! I'm happy changing the default config
> to
> > > > HEX instead of BINARY, no strong feelings there.
> > > >
> > > > I'll also clarify the example in the KIP to be clearer:
> > > >
> > > > - serialize the decimal field "foo" with value "10.2345" with the HEX
> > > > setting: {"foo": "D3J5"}
> > > > - serialize the decimal field "foo" with value "10.2345" with the
> > NUMERIC
> > > > setting: {"foo": 10.2345}
> > > >
> > > > With regards to the precision issue, that was my original concern as
> > well
> > > > (and why I originally suggested a TEXT format). Many JSON
> deserializers
> > > > (e.g. Jackson with
> DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS),
> > > > however, have the ability to deserialize decimals correctly so I will
> > > > configure that as the default for Connect's JsonDeserializer. It's
> > > probably
> > > > a good idea to call out that using other deserializers must be done
> > with
> > > > care - I will add that documentation to the serialization config.
> > > >
> > > > Note that there would not be an issue on the _serialization_ side of
> > > > things as Jackson respects BigDecimal.
> > > >
> > > > Almog
> > > >
> > > > On Tue, Aug 6, 2019 at 11:23 PM Arjun Satish  >
> > > > wrote:
> > > >
> > > >> hey Almog, nice work! couple of thoughts (hope I'm not late since
> you
> > > >> started the voting thread already):
> > > >>
> > > >> can you please add some examples to show the changes that you are
> > > >> proposing. makes me think that for a given decimal number, we will
> > have
> > > >> two
> > > >> encodings: “asHex” and “asNumber”.
> > > >>
> > > >> should we call the default config value “HEX” instead of “BINARY”?
> > > >>
> > > >> Should we call out the fact that JS systems might be susceptible to
> > > double
> > > >> precision round offs with the new numeric format? here are some
> people
> > > >> discussing a similar problem
> > > >> https://github.com/EventStore/EventStore/issues/1541
> > > >>
> > > >> On Tue, Aug 6, 2019 at 1:40 PM Almog Gavra 
> > wrote:
> > > >>
> > > >> > Hello Everyone,
> > > >> >
> > > >> > Summarizing an in-person discussion with Randall (this is copied
> > from
> > > >> the
> > > >> > KIP):
> > > >> >
> > > >> > The original KIP suggested supporting an additional
> representation -
> > > >> base10
> > > >> > encoded text (e.g. `{"asText":"10.2345"}`). 

Re: [DISCUSS] KIP-481: SerDe Improvements for Connect Decimal type in JSON

[Almog Gavra]

Thanks for the review Konstantine!

I think the terminology suggestion definitely makes things clearer - I will
update the documentation based on your suggestion (e.g. s/Consumer/Sink
Converter/g and s/Producer/Source Converter/g).

Cheers,
Almog

On Wed, Aug 14, 2019 at 8:13 AM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Thanks Almog for preparing this KIP!
> I think it will improve usability and troubleshooting with JSON data a lot.
>
> The finalized plan seems quite concrete now. I also liked that some
> implementation specific implications (such as setting the ObjectMapper to
> deserialize floating point as BigDecimal) are highlighted in the KIP.
>
> Still, as I was reading the KIP, the main obstacle I encountered was around
> terminology. I couldn't get used to reading "producer" and "consumer" and
> not thinking in terms of Kafka producers and consumers - which are not
> relevant to what this KIP proposes. Thus, I'd suggest replacing
> "Producer(s)" with "Source Converter(s)" and "Consumer(s)" with "Sink
> Converter(s)" (even if "Converter used by Source Connector" or "Converter
> used by Sink Connector" would be even more accurate - maybe this could be
> an explanation in a footnote). Terminology around converters has been
> tricky in the past and adding producers/consumers in the mix might add to
> the confusion.
>
> Another example where I'd apply this different terminology would be to a
> phrase such as the following:
> "Because of this, users must take care to first ensure that all consumers
> have upgraded to the new code before upgrading producers to make use of the
> NUMERIC serialization format."
> which I'd write
> "Because of this, users must take care to first ensure that all sink
> connectors have upgraded to the new converter code before upgrading source
> connectors to make use of the NUMERIC serialization format in
> JsonConverter."
>
> Let me know if you think this suggestion makes the KIP easier to follow.
> Otherwise I think it's a solid proposal.
>
> I'm concluding with a couple of nits:
>
> - "Upgraded Producer with BASE64 serialization, Legacy Consumer: this
> scenario is okay as the upgraded ~producer~ consumer will be able to read
> binary as today" (again according to my suggestion above, it could be as
> the upgraded source converter ...)
>
> - "consumers cannot consumer NUMERIC data. " -> "consumers cannot read
> NUMERIC data"
>
> Best,
> Konstantine
>
> On Fri, Aug 9, 2019 at 6:37 PM Almog Gavra  wrote:
>
> > Good catches! Fixed :)
> >
> > On Thu, Aug 8, 2019 at 10:36 PM Arjun Satish 
> > wrote:
> >
> > > Cool!
> > >
> > > Couple of nits:
> > >
> > > - In public interfaces, typo: *json.decimal.serialization.fromat*
> > > - In public interfaces, you use the term "HEX" instead of "BASE64".
> > >
> > >
> > >
> > > On Wed, Aug 7, 2019 at 9:51 AM Almog Gavra  wrote:
> > >
> > > > EDIT: everywhere I've been using "HEX" I meant to be using "BASE64".
> I
> > > will
> > > > update the KIP to reflect this.
> > > >
> > > > On Wed, Aug 7, 2019 at 9:44 AM Almog Gavra 
> wrote:
> > > >
> > > > > Thanks for the feedback Arjun! I'm happy changing the default
> config
> > to
> > > > > HEX instead of BINARY, no strong feelings there.
> > > > >
> > > > > I'll also clarify the example in the KIP to be clearer:
> > > > >
> > > > > - serialize the decimal field "foo" with value "10.2345" with the
> HEX
> > > > > setting: {"foo": "D3J5"}
> > > > > - serialize the decimal field "foo" with value "10.2345" with the
> > > NUMERIC
> > > > > setting: {"foo": 10.2345}
> > > > >
> > > > > With regards to the precision issue, that was my original concern
> as
> > > well
> > > > > (and why I originally suggested a TEXT format). Many JSON
> > deserializers
> > > > > (e.g. Jackson with
> > DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS),
> > > > > however, have the ability to deserialize decimals correctly so I
> will
> > > > > configure that as the default for Connect's JsonDeserializer. It's
> > > > probably
> > > > > a good idea to call out that using other deserializers must be done
> > > with
> > > > > care - I will add that documentation to the serialization config.
> > > > >
> > > > > Note that there would not be an issue on the _serialization_ side
> of
> > > > > things as Jackson respects BigDecimal.
> > > > >
> > > > > Almog
> > > > >
> > > > > On Tue, Aug 6, 2019 at 11:23 PM Arjun Satish <
> arjun.sat...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > >> hey Almog, nice work! couple of thoughts (hope I'm not late since
> > you
> > > > >> started the voting thread already):
> > > > >>
> > > > >> can you please add some examples to show the changes that you are
> > > > >> proposing. makes me think that for a given decimal number, we will
> > > have
> > > > >> two
> > > > >> encodings: “asHex” and “asNumber”.
> > > > >>
> > > > >> should we call the default config value “HEX” instead of “BINARY”?
> > > > >>
> > > > >> Should we call out the fact that JS systems might be suscep

Re: [VOTE] KIP-503: deleted topics metric

[Robert Barrett]

+1 (non-binding)

This will be good to have, thanks David!

Bob

On Wed, Aug 14, 2019 at 6:08 AM Mickael Maison 
wrote:

> +1 non binding
> Thank you!
>
> On Tue, Aug 13, 2019 at 9:07 PM Stanislav Kozlovski
>  wrote:
> >
> > +1 (non-binding)
> >
> > Thanks for the simple but very useful KIP!
> > Best,
> > Stanislav
> >
> > On Tue, Aug 13, 2019 at 8:32 PM Harsha Chintalapani 
> wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Aug 13, 2019 at 12:08 PM, David Arthur  >
> > > wrote:
> > >
> > > > Hello all,
> > > >
> > > > I'd like to start the vote on KIP-503
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion
> > > >
> > > > Thanks!
> > > > David
> > > >
> > >
> >
> >
> > --
> > Best,
> > Stanislav
>

Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

[Jason Gustafson]

Yeah, my reasoning is that the group metadata is only relevant to the
subscription API. So it makes sense to only expose it to the rebalance
listener.

One option we could consider is bring back the `initTransactions` overload.
Then usage looks something like this:

consumer.subscribe(topics, new RebalanceListener() {
  void onGroupJoined(GroupMetadata metadata) {
producer.initTransactions(metadata);
  }
}

That seems pretty clean. What do you think?

-Jason

On Tue, Aug 13, 2019 at 6:07 PM Boyang Chen 
wrote:

> Hey Guozhang,
>
> thanks for the suggestion. Could you elaborate more on why defining a
> direct consumer API would be easier? The benefit of reusing consumer
> rebalance listener is to consolidate the entry point of consumer internal
> states. Compared with letting consumer generate a deep-copy of metadata
> every time we call #sendOffsetsToTransactions, using a callback seems
> reducing unnecessary updates towards the metadata. WDYT?
>
> Boyang
>
> On Tue, Aug 13, 2019 at 2:14 PM Guozhang Wang  wrote:
>
> > Hi Boyang, Jason,
> >
> > If we are going to expose the generation id / group.instance id etc
> anyways
> > I think its slightly better to just add a new API on KafkaConsumer
> > returning the ConsumerGroupMetadata (option 3) than passing it in on an
> > additional callback of ConsumerRebalanceListener.
> > It feels easier to leverage, than requiring users to pass in the
> listener.
> >
> > Guozhang
> >
> > On Mon, Aug 12, 2019 at 3:41 PM Boyang Chen 
> > wrote:
> >
> > > Thanks Jason, the intuition behind defining a separate callback
> function
> > is
> > > that, with KIP-429 we no longer guarantee to call
> OnPartitionsAssigned()
> > or
> > > OnPartitionsRevoked() with each rebalance. Our requirement is to be
> > > up-to-date with group metadata such as generation information, so
> > callback
> > > like onGroupJoined() would make more sense as it should be invoked
> after
> > > every successful rebalance.
> > >
> > > Best,
> > > Boyang
> > >
> > > On Mon, Aug 12, 2019 at 2:02 PM Jason Gustafson 
> > > wrote:
> > >
> > > > Hey Boyang,
> > > >
> > > > I favor option 4 as well. It's a little more cumbersome than 3 for
> this
> > > use
> > > > case, but it seems like a cleaner separation of concerns. The
> rebalance
> > > > listener is already concerned with events affecting the assignment
> > > > lifecycle and group membership. I think the only thing I'm wondering
> is
> > > > whether it should be a separate callback as you've suggested, or if
> it
> > > > would make sense to overload `onPartitionsAssigned`. If it's
> separate,
> > > > maybe a name like `onGroupJoined` would be clearer?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > >
> > > > On Thu, Aug 8, 2019 at 10:59 PM Boyang Chen <
> > reluctanthero...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thank you Jason. We had some offline discussion on properly keeping
> > > group
> > > > > metadata up to date, and here are some of our options brainstormed:
> > > > > 1. Let the caller of `sendOffsetsToTransaction(offset, metadata)`
> > > > maintain
> > > > > the ever-changing group metadata. This could be done on stream
> side,
> > > but
> > > > > for non-stream EOS the sample code will become complicated as the
> > user
> > > > > needs to implement the partition assignor interface to get the
> update
> > > > from
> > > > > `onAssignment`
> > > > >
> > > > > 2. Get a new API on producer like `refreshGroupMetadata(metadata)`.
> > > This
> > > > is
> > > > > similar to option 1 except that now in the partition assignor
> > callback
> > > we
> > > > > could straightly pass in the producer instance, which simplifies
> the
> > > > > non-stream EOS, however this new API seems weird to define on
> > producer.
> > > > >
> > > > > 3. Make an accessing interface to group metadata, or just expose
> the
> > > > group
> > > > > metadata through a consumer API like `consumer.GroupMetadata()`.
> This
> > > is
> > > > > the old way which avoids the users’ effort to implement partition
> > > > assignor
> > > > > directly.
> > > > >
> > > > > 4. Expose the group metadata through rebalance listener, which is a
> > > more
> > > > > well-known and adopted callback interface. We could do sth like
> > > > > `onGroupMetadataUpdated(ConsumerGroupMetadata metadata)`
> > > > >
> > > > > To simplify the code logic, we believe option 3 & 4 are better
> > > solutions,
> > > > > and of which I slightly prefer option 4 as it is the most clean
> > > solution
> > > > > with less intrusion to both consumer and producer APIs.
> > > > >
> > > > > WDYT?
> > > > >
> > > > > Boyang
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Aug 7, 2019 at 9:20 AM Jason Gustafson  >
> > > > wrote:
> > > > >
> > > > > > Hi Boyang,
> > > > > >
> > > > > > > We already persist member.id, instance.id and generation.id in
> > the
> > > > > > offset
> > > > > > topic, what extra fields we need to store?
> > > > > >
> > > > > > Yeah, you're right. I was a little confused and t

Re: [VOTE] KIP-497: Add inter-broker API to alter ISR

[Satish Duggana]

+1 (non-binding), Thanks Jason for nice improvements on ISR
propagation protocol!

On Wed, Aug 14, 2019 at 8:29 PM David Arthur  wrote:
>
> +1 binding, this looks great!
>
> -David
>
> On Tue, Aug 13, 2019 at 4:55 PM Guozhang Wang  wrote:
>
> > +1 (binding). This is a great KIP, thanks Jason!
> >
> > Regarding the naming of the zkVersion, I'm actually fine to name it more
> > generally and leave a note that at the moment its value is defined as the
> > zk version.
> >
> >
> > Guozhang
> >
> >
> > On Mon, Aug 12, 2019 at 2:22 PM Jason Gustafson 
> > wrote:
> >
> > > Hi Viktor,
> > >
> > > I originally named the field `CurrentVersion`. I didn't have 'Zk' in the
> > > name in anticipation of KIP-500. I thought about it and decided it makes
> > > sense to keep naming consistent with other APIs. Even if KIP-500 passes,
> > > there will be some time during which it only refers to the zk version.
> > > Eventually we'll have to decide whether it makes sense to change the name
> > > or just introduce a new field.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Fri, Aug 9, 2019 at 9:19 AM Viktor Somogyi-Vass <
> > > viktorsomo...@gmail.com>
> > > wrote:
> > >
> > > > Hey Jason,
> > > >
> > > > +1 from me too.
> > > > One note though: since it's a new protocol we could perhaps rename
> > > > CurrentZkVersion to something like "IsrEpoch" or "IsrVersion". I think
> > > > that'd reflect its purpose better.
> > > >
> > > > Best,
> > > > Viktor
> > > >
> > > > On Wed, Aug 7, 2019 at 8:37 PM Jason Gustafson 
> > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I'd like to start a vote on KIP-497:
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-497%3A+Add+inter-broker+API+to+alter+ISR
> > > > > .
> > > > > +1
> > > > > from me.
> > > > >
> > > > > -Jason
> > > > >
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>
>
> --
> David Arthur

[jira] [Reopened] (KAFKA-7912) In-memory key-value store does not support concurrent access

[Sophie Blee-Goldman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sophie Blee-Goldman reopened KAFKA-7912:


The fix for this caused a performance regression and is being rolled back. We 
should look into alternative ways to support concurrent modifications to the 
underlying data structure

> In-memory key-value store does not support concurrent access 
> -
>
> Key: KAFKA-7912
> URL: https://issues.apache.org/jira/browse/KAFKA-7912
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Sophie Blee-Goldman
>Assignee: Sophie Blee-Goldman
>Priority: Major
> Fix For: 2.3.0
>
>
> Currently, the in-memory key-value store uses a Map to store key-value pairs 
> and fetches them by calling subMap and returning an iterator to this submap. 
> This is unsafe as the submap is just a view of the original map and there is 
> risk of concurrent access.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (KAFKA-8802) ConcurrentSkipListMap shows performance regression in cache and in-memory store

[Sophie Blee-Goldman (JIRA)]

Sophie Blee-Goldman created KAFKA-8802:
--

 Summary: ConcurrentSkipListMap shows performance regression in 
cache and in-memory store
 Key: KAFKA-8802
 URL: https://issues.apache.org/jira/browse/KAFKA-8802
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.3.0
Reporter: Sophie Blee-Goldman
 Fix For: 2.4.0, 2.3.1


A significant performance regression was seen between 2.2 and 2.3. For 
in-memory stores, about half of it was improved by removing the 
ConcurrentSkipListMap. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (KAFKA-8803) Stream will not start due to TimeoutException: Timeout expired after 60000milliseconds while awaiting InitProducerId

[Raman Gupta (JIRA)]

Raman Gupta created KAFKA-8803:
--

 Summary: Stream will not start due to TimeoutException: Timeout 
expired after 6milliseconds while awaiting InitProducerId
 Key: KAFKA-8803
 URL: https://issues.apache.org/jira/browse/KAFKA-8803
 Project: Kafka
  Issue Type: Bug
Reporter: Raman Gupta


One streams app is consistently failing at startup with the following exception:

{code}
2019-08-14 17:02:29,568 ERROR --- [2ce1b-StreamThread-2] 
org.apa.kaf.str.pro.int.StreamTask: task [0_36] Timeout 
exception caught when initializing transactions for task 0_36. This might 
happen if the broker is slow to respond, if the network connection to the 
broker was interrupted, or if similar circumstances arise. You can increase 
producer parameter `max.block.ms` to increase this timeout.
org.apache.kafka.common.errors.TimeoutException: Timeout expired after 
6milliseconds while awaiting InitProducerId
{code}

These same brokers are used by many other streams without any issue, including 
some in the very same processes for the stream which consistently throws this 
exception.

I use the static consumer group protocol with Kafka 2.3.0 clients and 2.3.0 
broker. The broker has a patch for KAFKA-8773.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

[Guozhang Wang]

My main concern is to require the overloaded `initTransactions` to be
called repeatedly while the original `initTransactions` still called once
throughout the life time, which is a bit confusing.

Looking into the current POC PR, we actually only need the latest
generation id when fetching offsets, so we can just make the GroupMetadata
returned from the consumer a wrapper of the underlying values, and the
getters of this object would always return the latest value.
The values would be reset internally within the rebalances; and then the
new `initTransactions` would still only be called once.

Guozhang


On Wed, Aug 14, 2019 at 9:53 AM Jason Gustafson  wrote:

> Yeah, my reasoning is that the group metadata is only relevant to the
> subscription API. So it makes sense to only expose it to the rebalance
> listener.
>
> One option we could consider is bring back the `initTransactions` overload.
> Then usage looks something like this:
>
> consumer.subscribe(topics, new RebalanceListener() {
>   void onGroupJoined(GroupMetadata metadata) {
> producer.initTransactions(metadata);
>   }
> }
>
> That seems pretty clean. What do you think?
>
> -Jason
>
> On Tue, Aug 13, 2019 at 6:07 PM Boyang Chen 
> wrote:
>
> > Hey Guozhang,
> >
> > thanks for the suggestion. Could you elaborate more on why defining a
> > direct consumer API would be easier? The benefit of reusing consumer
> > rebalance listener is to consolidate the entry point of consumer internal
> > states. Compared with letting consumer generate a deep-copy of metadata
> > every time we call #sendOffsetsToTransactions, using a callback seems
> > reducing unnecessary updates towards the metadata. WDYT?
> >
> > Boyang
> >
> > On Tue, Aug 13, 2019 at 2:14 PM Guozhang Wang 
> wrote:
> >
> > > Hi Boyang, Jason,
> > >
> > > If we are going to expose the generation id / group.instance id etc
> > anyways
> > > I think its slightly better to just add a new API on KafkaConsumer
> > > returning the ConsumerGroupMetadata (option 3) than passing it in on an
> > > additional callback of ConsumerRebalanceListener.
> > > It feels easier to leverage, than requiring users to pass in the
> > listener.
> > >
> > > Guozhang
> > >
> > > On Mon, Aug 12, 2019 at 3:41 PM Boyang Chen <
> reluctanthero...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Jason, the intuition behind defining a separate callback
> > function
> > > is
> > > > that, with KIP-429 we no longer guarantee to call
> > OnPartitionsAssigned()
> > > or
> > > > OnPartitionsRevoked() with each rebalance. Our requirement is to be
> > > > up-to-date with group metadata such as generation information, so
> > > callback
> > > > like onGroupJoined() would make more sense as it should be invoked
> > after
> > > > every successful rebalance.
> > > >
> > > > Best,
> > > > Boyang
> > > >
> > > > On Mon, Aug 12, 2019 at 2:02 PM Jason Gustafson 
> > > > wrote:
> > > >
> > > > > Hey Boyang,
> > > > >
> > > > > I favor option 4 as well. It's a little more cumbersome than 3 for
> > this
> > > > use
> > > > > case, but it seems like a cleaner separation of concerns. The
> > rebalance
> > > > > listener is already concerned with events affecting the assignment
> > > > > lifecycle and group membership. I think the only thing I'm
> wondering
> > is
> > > > > whether it should be a separate callback as you've suggested, or if
> > it
> > > > > would make sense to overload `onPartitionsAssigned`. If it's
> > separate,
> > > > > maybe a name like `onGroupJoined` would be clearer?
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Aug 8, 2019 at 10:59 PM Boyang Chen <
> > > reluctanthero...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thank you Jason. We had some offline discussion on properly
> keeping
> > > > group
> > > > > > metadata up to date, and here are some of our options
> brainstormed:
> > > > > > 1. Let the caller of `sendOffsetsToTransaction(offset, metadata)`
> > > > > maintain
> > > > > > the ever-changing group metadata. This could be done on stream
> > side,
> > > > but
> > > > > > for non-stream EOS the sample code will become complicated as the
> > > user
> > > > > > needs to implement the partition assignor interface to get the
> > update
> > > > > from
> > > > > > `onAssignment`
> > > > > >
> > > > > > 2. Get a new API on producer like
> `refreshGroupMetadata(metadata)`.
> > > > This
> > > > > is
> > > > > > similar to option 1 except that now in the partition assignor
> > > callback
> > > > we
> > > > > > could straightly pass in the producer instance, which simplifies
> > the
> > > > > > non-stream EOS, however this new API seems weird to define on
> > > producer.
> > > > > >
> > > > > > 3. Make an accessing interface to group metadata, or just expose
> > the
> > > > > group
> > > > > > metadata through a consumer API like `consumer.GroupMetadata()`.
> > This
> > > > is
> > > > > > the old way which avoids the users’ effort to implement partition
> > > > > assignor
> > >

Re: [VOTE] KIP-503: deleted topics metric

[David Jacot]

+1 (non-binding)

Thanks for the KIP! Simple yet very useful.

Best,
David

On Wed, Aug 14, 2019 at 9:24 AM Robert Barrett 
wrote:

> +1 (non-binding)
>
> This will be good to have, thanks David!
>
> Bob
>
> On Wed, Aug 14, 2019 at 6:08 AM Mickael Maison 
> wrote:
>
> > +1 non binding
> > Thank you!
> >
> > On Tue, Aug 13, 2019 at 9:07 PM Stanislav Kozlovski
> >  wrote:
> > >
> > > +1 (non-binding)
> > >
> > > Thanks for the simple but very useful KIP!
> > > Best,
> > > Stanislav
> > >
> > > On Tue, Aug 13, 2019 at 8:32 PM Harsha Chintalapani 
> > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > > On Tue, Aug 13, 2019 at 12:08 PM, David Arthur <
> davidart...@apache.org
> > >
> > > > wrote:
> > > >
> > > > > Hello all,
> > > > >
> > > > > I'd like to start the vote on KIP-503
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion
> > > > >
> > > > > Thanks!
> > > > > David
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Stanislav
> >
>

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

[Gabor Somogyi]

+1 (non-binding)
I've read it through in depth and as Jungtaek said Spark can make good use
of it.

On Wed, 14 Aug 2019, 17:06 Jungtaek Lim,  wrote:

> +1 (non-binding)
>
> I found it very useful for Spark's case. (Discussion on KIP-505 described
> it.)
>
> Thanks for driving the effort!
>
> 2019년 8월 14일 (수) 오후 8:49, Mickael Maison 님이 작성:
>
> > Hi Guozhang,
> >
> > Thanks for taking a look.
> >
> > 1. Right, I updated the titles of the code blocks
> >
> > 2. Yes that's a good idea. I've updated the KIP
> >
> > Thank you
> >
> > On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
> >  wrote:
> > >
> > > Hi Colin,
> > >
> > > Thanks for raising these 2 valid points. I've updated the KIP
> > accordingly.
> > >
> > > On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang 
> > wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > Just some minor comments.
> > > >
> > > > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java"
> should
> > be
> > > > "AlterOffsetsOptions".
> > > >
> > > > 2. I'd suggest we change the future structure of "AlterOffsetsResult"
> > to
> > > >
> > > > *KafkaFuture>>*
> > > >
> > > > This is because we will have a hierarchy of two-layers of errors
> since
> > we
> > > > need to find out the group coordinator first and then issue the
> commit
> > > > offset request (see e.g. the ListConsumerGroupOffsetsResult which
> > exclude
> > > > partitions that have errors, or the DeleteMembersResult as part of
> > KIP-345).
> > > >
> > > > If the discover-coordinator returns non-triable error, we would set
> it
> > on
> > > > the first layer of the KafkaFuture, and the per-partition error would
> > be
> > > > set on the second layer of the KafkaFuture.
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe 
> > wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Considering that KIP-496, which adds a way of deleting consumer
> > offsets
> > > > > from AdminClient, looks like it is going to get in, this seems like
> > > > > functionality we should definitely have.
> > > > >
> > > > > For alterConsumerGroupOffsets, is the intention to ignore
> partitions
> > that
> > > > > are not specified in the map?  If so, we should specify that in the
> > JavaDoc.
> > > > >
> > > > > isolationLevel seems like it should be an enum rather than a
> > string.  The
> > > > > existing enum is in org.apache.kafka.common.requests, so we should
> > probably
> > > > > create a new one which is public in org.apache.kafka.clients.admin.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > > > > > Bumping this thread once again
> > > > > >
> > > > > > Ismael, have I answered your questions?
> > > > > > While this has received a few non-binding +1s, no committers have
> > > > > > voted yet. If you have concerns or questions, please let me know.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > > > >  wrote:
> > > > > > >
> > > > > > > Bumping this thread as it's been a couple of weeks.
> > > > > > >
> > > > > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > > > > mickael.mai...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Thanks Ismael for the feedback. I think your point has 2
> parts:
> > > > > > > > - Having the reset functionality in the AdminClient:
> > > > > > > > The fact we have a command line tool illustrate that this
> > operation
> > > > > is
> > > > > > > > relatively common. I seems valuable to be able to perform
> this
> > > > > > > > operation directly via a proper API in addition of the CLI
> > tool.
> > > > > > > >
> > > > > > > > - Sending an OffsetCommit directly instead of relying on
> > > > > KafkaConsumer:
> > > > > > > > The KafkaConsumer requires a lot of stuff to commit offsets.
> > Its
> > > > > group
> > > > > > > > cannot change so you need to start a new Consumer every time,
> > that
> > > > > > > > creates new connections and overal sends more requests. Also
> > there
> > > > > are
> > > > > > > > already  a bunch of AdminClient APIs that have logic very
> > close to
> > > > > > > > what needs to be done to send a commit request, keeping the
> > code
> > > > > small
> > > > > > > > and consistent.
> > > > > > > >
> > > > > > > > I've updated the KIP with these details and moved the 2nd
> part
> > to
> > > > > > > > "Proposed changes" as it's more an implementation detail.
> > > > > > > >
> > > > > > > > I hope this answers your question
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <
> isma...@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > The KIP doesn't discuss the option of using KafkaConsumer
> > directly
> > > > > as far
> > > > > > > > > as I can tell. We have tried to avoid having the same
> > > > > functionality in
> > > > > > > > > multiple clients so it would be good to explain why this is
> > > > > necessar

Build failed in Jenkins: kafka-trunk-jdk11 #753

[Apache Jenkins Server]

See 


Changes:

[bbejeck] MINOR: Correct typo in test name `TimetampedSegmentsTest` (#7210)

--
[...truncated 2.59 MB...]

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutTaskConfigs STARTED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutTaskConfigs PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateConnectorFailedBasicValidation STARTED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateConnectorFailedBasicValidation PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateConnectorAlreadyExists STARTED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateConnectorAlreadyExists PASSED

org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicyTest
 > testPrincipalPlusOtherConfigs STARTED

org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicyTest
 > testPrincipalPlusOtherConfigs PASSED

org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicyTest
 > testPrincipalOnly STARTED

org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicyTest
 > testPrincipalOnly PASSED

org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicyTest
 > testNoOverrides STARTED

org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicyTest
 > testNoOverrides PASSED

org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicyTest
 > testWithOverrides STARTED

org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicyTest
 > testWithOverrides PASSED

> Task :streams:examples:test

org.apache.kafka.streams.examples.wordcount.WordCountProcessorTest > test 
STARTED

org.apache.kafka.streams.examples.wordcount.WordCountProcessorTest > test PASSED

> Task :streams:streams-scala:test

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed should 
create a Consumed with Serdes STARTED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed should 
create a Consumed with Serdes PASSED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
timestampExtractor and resetPolicy should create a Consumed with Serdes, 
timestampExtractor and resetPolicy STARTED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
timestampExtractor and resetPolicy should create a Consumed with Serdes, 
timestampExtractor and resetPolicy PASSED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
timestampExtractor should create a Consumed with Serdes and timestampExtractor 
STARTED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
timestampExtractor should create a Consumed with Serdes and timestampExtractor 
PASSED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
resetPolicy should create a Consumed with Serdes and resetPolicy STARTED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
resetPolicy should create a Consumed with Serdes and resetPolicy PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilWindowCloses STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilWindowCloses PASSED

org.apache.kafka.str

[jira] [Created] (KAFKA-8804) Internal Connect REST endpoints are insecure

[Chris Egerton (JIRA)]

Chris Egerton created KAFKA-8804:


 Summary: Internal Connect REST endpoints are insecure
 Key: KAFKA-8804
 URL: https://issues.apache.org/jira/browse/KAFKA-8804
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Chris Egerton
Assignee: Chris Egerton


This covers 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints]



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

[Chris Egerton]

Hi all,

I'd like to start discussion on a KIP to secure the internal "POST
/connectors//tasks" endpoint for the Connect framework. The proposed
changes address a vulnerability in the framework in its current state that
allows malicious users to write arbitrary task configurations for
connectors; it is vital that this issue be addressed in order for any
Connect cluster to be secure.

Looking forward to your thoughts,

Chris

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

[Chris Egerton]

The KIP page can be found at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints,
by the way. Apologies for neglecting to include it in my initial email!

On Wed, Aug 14, 2019 at 12:29 PM Chris Egerton  wrote:

> Hi all,
>
> I'd like to start discussion on a KIP to secure the internal "POST
> /connectors//tasks" endpoint for the Connect framework. The proposed
> changes address a vulnerability in the framework in its current state that
> allows malicious users to write arbitrary task configurations for
> connectors; it is vital that this issue be addressed in order for any
> Connect cluster to be secure.
>
> Looking forward to your thoughts,
>
> Chris
>

[jira] [Created] (KAFKA-8805) Bump producer epoch following recoverable errors

[Bob Barrett (JIRA)]

Bob Barrett created KAFKA-8805:
--

 Summary: Bump producer epoch following recoverable errors
 Key: KAFKA-8805
 URL: https://issues.apache.org/jira/browse/KAFKA-8805
 Project: Kafka
  Issue Type: Improvement
  Components: producer 
Affects Versions: 2.3.0
Reporter: Bob Barrett
Assignee: Bob Barrett


As part of KIP-360, the producer needs to call the new InitProducerId API after 
receiving UNKNOWN_PRODUCER_ID and INVALID_PRODUCER_MAPPING errors, which will 
allow the producers to bump their epoch and continue processing unless a new 
producer has already initialized a new producer ID.

The broker change that this depends on is 
https://issues.apache.org/jira/browse/KAFKA-8710.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Colin McCabe]

Hi Rajini,

I think it would be good to rename KafkaRequestContext to something like 
AuthorizableRequestContext, and put it in the 
org.apache.kafka.server.authorizer namespace.  If we put it in the 
org.apache.kafka.common namespace, then it's not really clear that it's part of 
the Authorizer API.  Since this class is purely an interface, with no concrete 
implementation of anything, there's nothing common to really reuse in any case. 
 We definitely don't want someone to accidentally add or remove methods because 
they think this is just another internal class used for requests.

The BrokerInfo class is a nice improvement.  It looks like it will be useful 
for passing in information about the context we're running in.  It would be 
nice to call this class ServerInfo rather than BrokerInfo, since we will want 
to run the authorizer on controllers as well as on brokers, and the controller 
may run as a separate process post KIP-500.  I also think that this class 
should be in the org.apache.kafka.server.authorizer namespace.  Again, it is an 
interface, not a concrete implementation, and it's an interface that is very 
specifically for the authorizer.

I agree that we probably don't have enough information preserved for requests 
currently to always know what entity made them.  So we can leave that out for 
now (except in the special case of Fetch).  Perhaps we can add this later if 
it's needed.

I understand the intention behind AuthorizationMode (which is now called 
AuditFlag in the latest revision).  But it still feels complex.  What if we 
just had two booleans in Action: logSuccesses and logFailures?  That seems to 
cover all the cases here.  MANDATORY_AUTHORIZE = true, true.  
OPTIONAL_AUTHORIZE = true, false.  FILTER = true, false.  LIST_AUTHORIZED = 
false, false.  Would there be anything lost versus having the enum?

best,
Colin


On Wed, Aug 14, 2019, at 06:29, Mickael Maison wrote:
> Hi Rajini,
> 
> Thanks for the KIP!
> I really like that authorize() will be able to take a batch of
> requests, this will speed up many implementations!
> 
> On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram  
> wrote:
> >
> > Thanks David! I have fixed the typo.
> >
> > Also made a couple of changes to make the context interfaces more generic.
> > KafkaRequestContext now returns the 16-bit API key as Colin suggested as
> > well as the friendly name used in metrics which are useful in audit logs.
> > `Authorizer#start` is now provided a generic `BrokerInfo` interface that
> > gives cluster id, broker id and endpoint information. The generic interface
> > can potentially be used in other broker plugins in future and provides
> > dynamically generated configs like broker id and ports which are currently
> > not available to plugins unless these configs are statically configured.
> > Please let me know if there are any concerns.
> >
> > Regards,
> >
> > Rajini
> >
> > On Tue, Aug 13, 2019 at 4:30 PM David Jacot  wrote:
> >
> > > Hi Rajini,
> > >
> > > Thank you for the update! It looks good to me. There is a typo in the
> > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` -> `MANDATORY_AUTHORIZE`.
> > >
> > > Regards,
> > > David
> > >
> > > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram 
> > > wrote:
> > >
> > > > Hi David,
> > > >
> > > > Thanks for reviewing the KIP! Since questions about `authorization mode`
> > > > and `count` have come up multiple times, I have renamed both.
> > > >
> > > > 1) Renamed `count` to `resourceReferenceCount`. It is the number of 
> > > > times
> > > > the resource being authorized is referenced within the request.
> > > >
> > > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided to improve
> > > > audit logging in the authorizer. The enum values have javadoc which
> > > > indicate how the authorization result is used in each of the modes to
> > > > enable authorizers to log audit messages at the right severity level.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot  wrote:
> > > >
> > > > > Hi Rajini,
> > > > >
> > > > > Thank you for the KIP. Overall, it looks good to me. I have few
> > > > > questions/suggestions:
> > > > >
> > > > > 1. It is hard to grasp what `Action#count` is for. I guess I 
> > > > > understand
> > > > > where you want to go with it but it took me a while to figure it out.
> > > > > Perhaps, we could come up with a better name than `count`?
> > > > >
> > > > > 2. I had a hard time trying to understand the `AuthorizationMode`
> > > > concept,
> > > > > especially wrt. the OPTIONAL one. My understanding is that an ACL is
> > > > either
> > > > > defined or not. Could you elaborate a bit more on that?
> > > > >
> > > > > Thanks,
> > > > > David
> > > > >
> > > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai 
> > > > wrote:
> > > > >
> > > > > > Hi Rajini
> > > > > >
> > > > > > 3.2 - This makes sense. Thanks for clarifying.
> > > > > >
> > > > > > Rest looks fine. Once the implementations are don

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

[Colin McCabe]

Thanks,  Mickael.  +1 (binding)

best,
Colin

On Wed, Aug 14, 2019, at 12:07, Gabor Somogyi wrote:
> +1 (non-binding)
> I've read it through in depth and as Jungtaek said Spark can make good use
> of it.
> 
> On Wed, 14 Aug 2019, 17:06 Jungtaek Lim,  wrote:
> 
> > +1 (non-binding)
> >
> > I found it very useful for Spark's case. (Discussion on KIP-505 described
> > it.)
> >
> > Thanks for driving the effort!
> >
> > 2019년 8월 14일 (수) 오후 8:49, Mickael Maison 님이 작성:
> >
> > > Hi Guozhang,
> > >
> > > Thanks for taking a look.
> > >
> > > 1. Right, I updated the titles of the code blocks
> > >
> > > 2. Yes that's a good idea. I've updated the KIP
> > >
> > > Thank you
> > >
> > > On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
> > >  wrote:
> > > >
> > > > Hi Colin,
> > > >
> > > > Thanks for raising these 2 valid points. I've updated the KIP
> > > accordingly.
> > > >
> > > > On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang 
> > > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for the KIP!
> > > > >
> > > > > Just some minor comments.
> > > > >
> > > > > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java"
> > should
> > > be
> > > > > "AlterOffsetsOptions".
> > > > >
> > > > > 2. I'd suggest we change the future structure of "AlterOffsetsResult"
> > > to
> > > > >
> > > > > *KafkaFuture>>*
> > > > >
> > > > > This is because we will have a hierarchy of two-layers of errors
> > since
> > > we
> > > > > need to find out the group coordinator first and then issue the
> > commit
> > > > > offset request (see e.g. the ListConsumerGroupOffsetsResult which
> > > exclude
> > > > > partitions that have errors, or the DeleteMembersResult as part of
> > > KIP-345).
> > > > >
> > > > > If the discover-coordinator returns non-triable error, we would set
> > it
> > > on
> > > > > the first layer of the KafkaFuture, and the per-partition error would
> > > be
> > > > > set on the second layer of the KafkaFuture.
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe 
> > > wrote:
> > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > Considering that KIP-496, which adds a way of deleting consumer
> > > offsets
> > > > > > from AdminClient, looks like it is going to get in, this seems like
> > > > > > functionality we should definitely have.
> > > > > >
> > > > > > For alterConsumerGroupOffsets, is the intention to ignore
> > partitions
> > > that
> > > > > > are not specified in the map?  If so, we should specify that in the
> > > JavaDoc.
> > > > > >
> > > > > > isolationLevel seems like it should be an enum rather than a
> > > string.  The
> > > > > > existing enum is in org.apache.kafka.common.requests, so we should
> > > probably
> > > > > > create a new one which is public in org.apache.kafka.clients.admin.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > > > > > > Bumping this thread once again
> > > > > > >
> > > > > > > Ismael, have I answered your questions?
> > > > > > > While this has received a few non-binding +1s, no committers have
> > > > > > > voted yet. If you have concerns or questions, please let me know.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > > > > >  wrote:
> > > > > > > >
> > > > > > > > Bumping this thread as it's been a couple of weeks.
> > > > > > > >
> > > > > > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > > > > > mickael.mai...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Thanks Ismael for the feedback. I think your point has 2
> > parts:
> > > > > > > > > - Having the reset functionality in the AdminClient:
> > > > > > > > > The fact we have a command line tool illustrate that this
> > > operation
> > > > > > is
> > > > > > > > > relatively common. I seems valuable to be able to perform
> > this
> > > > > > > > > operation directly via a proper API in addition of the CLI
> > > tool.
> > > > > > > > >
> > > > > > > > > - Sending an OffsetCommit directly instead of relying on
> > > > > > KafkaConsumer:
> > > > > > > > > The KafkaConsumer requires a lot of stuff to commit offsets.
> > > Its
> > > > > > group
> > > > > > > > > cannot change so you need to start a new Consumer every time,
> > > that
> > > > > > > > > creates new connections and overal sends more requests. Also
> > > there
> > > > > > are
> > > > > > > > > already  a bunch of AdminClient APIs that have logic very
> > > close to
> > > > > > > > > what needs to be done to send a commit request, keeping the
> > > code
> > > > > > small
> > > > > > > > > and consistent.
> > > > > > > > >
> > > > > > > > > I've updated the KIP with these details and moved the 2nd
> > part
> > > to
> > > > > > > > > "Proposed changes" as it's more an implementation detail.
> > > > > > > > >
> > > > > > > > > I hope this answers your question
> > > > > > > > >
> > > > > > > > > On Mo

Build failed in Jenkins: kafka-trunk-jdk8 #3852

[Apache Jenkins Server]

See 


Changes:

[bbejeck] MINOR: Correct typo in test name `TimetampedSegmentsTest` (#7210)

--
[...truncated 6.51 MB...]

org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicyTest
 > testPrincipalOnly PASSED

org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicyTest
 > testNoOverrides STARTED

org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicyTest
 > testNoOverrides PASSED

org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicyTest
 > testWithOverrides STARTED

org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicyTest
 > testWithOverrides PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > testBytesNullToNumber 
STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > testBytesNullToNumber 
PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testSerializingIncorrectType STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testSerializingIncorrectType PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testDeserializingHeaderWithTooManyBytes STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testDeserializingHeaderWithTooManyBytes PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > testNullToBytes 
STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > testNullToBytes PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testSerializingIncorrectHeader STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testSerializingIncorrectHeader PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testDeserializingDataWithTooManyBytes STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testDeserializingDataWithTooManyBytes PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testConvertingSamplesToAndFromBytes STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testConvertingSamplesToAndFromBytes PASSED

org.apache.kafka.connect.converters.FloatConverterTest > testBytesNullToNumber 
STARTED

org.apache.kafka.connect.converters.FloatConverterTest > testBytesNullToNumber 
PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectType STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectType PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingHeaderWithTooManyBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingHeaderWithTooManyBytes PASSED

org.apache.kafka.connect.converters.FloatConverterTest > testNullToBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > testNullToBytes PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectHeader STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectHeader PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingDataWithTooManyBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingDataWithTooManyBytes PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testConvertingSamplesToAndFromBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testConvertingSamplesToAndFromBytes PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testBytesNullToNumber STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testBytesNullToNumber PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectType STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectType PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingHeaderWithTooManyBytes STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingHeaderWithTooManyBytes PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > testNullToBytes 
STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > testNullToBytes 
PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectHeader STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectHeader PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingDataWithTooManyBytes STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingDataWithTooManyBytes PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testConvertingSamplesToAndFromBytes STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testConvertingSamplesToAndFromBytes PASSED

org.apache.kafka.connect.converters.ByteArrayConverterTest > 
testFrom

Build failed in Jenkins: kafka-2.3-jdk8 #87

[Apache Jenkins Server]

See 


Changes:

[bill] MINOR: Correct typo in test name `TimetampedSegmentsTest` (#7210)

--
[...truncated 2.92 MB...]
kafka.zk.KafkaZkClientTest > testCreateTokenChangeNotification STARTED

kafka.zk.KafkaZkClientTest > testCreateTokenChangeNotification PASSED

kafka.zk.KafkaZkClientTest > testGetTopicsAndPartitions STARTED

kafka.zk.KafkaZkClientTest > testGetTopicsAndPartitions PASSED

kafka.zk.KafkaZkClientTest > testRegisterBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testRegisterBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testRetryRegisterBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testRetryRegisterBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testConsumerOffsetPath STARTED

kafka.zk.KafkaZkClientTest > testConsumerOffsetPath PASSED

kafka.zk.KafkaZkClientTest > testDeleteRecursiveWithControllerEpochVersionCheck 
STARTED

kafka.zk.KafkaZkClientTest > testDeleteRecursiveWithControllerEpochVersionCheck 
PASSED

kafka.zk.KafkaZkClientTest > testControllerManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testControllerManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testTopicAssignmentMethods STARTED

kafka.zk.KafkaZkClientTest > testTopicAssignmentMethods PASSED

kafka.zk.KafkaZkClientTest > testPropagateIsrChanges STARTED

kafka.zk.KafkaZkClientTest > testPropagateIsrChanges PASSED

kafka.zk.KafkaZkClientTest > testControllerEpochMethods STARTED

kafka.zk.KafkaZkClientTest > testControllerEpochMethods PASSED

kafka.zk.KafkaZkClientTest > testDeleteRecursive STARTED

kafka.zk.KafkaZkClientTest > testDeleteRecursive PASSED

kafka.zk.KafkaZkClientTest > testGetTopicPartitionStates STARTED

kafka.zk.KafkaZkClientTest > testGetTopicPartitionStates PASSED

kafka.zk.KafkaZkClientTest > testCreateConfigChangeNotification STARTED

kafka.zk.KafkaZkClientTest > testCreateConfigChangeNotification PASSED

kafka.zk.KafkaZkClientTest > testDelegationTokenMethods STARTED

kafka.zk.KafkaZkClientTest > testDelegationTokenMethods PASSED

kafka.zk.ReassignPartitionsZNodeTest > testDecodeInvalidJson STARTED

kafka.zk.ReassignPartitionsZNodeTest > testDecodeInvalidJson PASSED

kafka.zk.ReassignPartitionsZNodeTest > testEncode STARTED

kafka.zk.ReassignPartitionsZNodeTest > testEncode PASSED

kafka.zk.ReassignPartitionsZNodeTest > testDecodeValidJson STARTED

kafka.zk.ReassignPartitionsZNodeTest > testDecodeValidJson PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDataChange 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDataChange PASSED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperSessionStateMetric STARTED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperSessionStateMetric PASSED

kafka.zookeeper.ZooKeeperClientTest > testExceptionInBeforeInitializingSession 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testExceptionInBeforeInitializingSession 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testConnection STARTED

kafka.zookeeper.ZooKeeperClientTest > testConnection PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForCreation STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForCreation PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetAclExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetAclExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiryDuringClose STARTED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiryDuringClose PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetAclNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetAclNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testConnectionLossRequestTermination 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testConnectionLossRequestTermination 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testExistsNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testExistsNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetDataNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetDataNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testConnectionTimeout STARTED

kafka.zookeeper.ZooKeeperClientTest > testConnectionTimeout PASSED

kafka.zookeeper.ZooKeeperClientTest > 
testBlockOnRequestCompletionFromStateChangeHandler STARTED

kafka.zookeeper.ZooKeeperClientTest > 
testBlockOnRequestCompletionFromStateChangeHandler PASSED

kafka.zookeeper.ZooKeeperClientTest > testUnresolvableConnectString STARTED

kafka.zookeeper.ZooKeeperClientTest > testUnresolvableConnectString PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenNonExistentZNode PASSED

kafka.zookeeper.ZooKee

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

[Harsha Chintalapani]

Thanks for the KIP Mickael. LGTM +1 (binding).
-Harsha


On Wed, Aug 14, 2019 at 1:10 PM, Colin McCabe  wrote:

> Thanks, Mickael. +1 (binding)
>
> best,
> Colin
>
> On Wed, Aug 14, 2019, at 12:07, Gabor Somogyi wrote:
>
> +1 (non-binding)
> I've read it through in depth and as Jungtaek said Spark can make good use
> of it.
>
> On Wed, 14 Aug 2019, 17:06 Jungtaek Lim,  wrote:
>
> +1 (non-binding)
>
> I found it very useful for Spark's case. (Discussion on KIP-505 described
> it.)
>
> Thanks for driving the effort!
>
> 2019년 8월 14일 (수) 오후 8:49, Mickael Maison 님이 작성:
>
> Hi Guozhang,
>
> Thanks for taking a look.
>
> 1. Right, I updated the titles of the code blocks
>
> 2. Yes that's a good idea. I've updated the KIP
>
> Thank you
>
> On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
>  wrote:
>
> Hi Colin,
>
> Thanks for raising these 2 valid points. I've updated the KIP
>
> accordingly.
>
> On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang 
>
> wrote:
>
> Hi Mickael,
>
> Thanks for the KIP!
>
> Just some minor comments.
>
> 1. Java class names are stale, e.g. "CommitOffsetsOptions.java
> "
>
> should
>
> be
>
> "AlterOffsetsOptions".
>
> 2. I'd suggest we change the future structure of "AlterOffsetsResult"
>
> to
>
> *KafkaFuture>>*
>
> This is because we will have a hierarchy of two-layers of errors
>
> since
>
> we
>
> need to find out the group coordinator first and then issue the
>
> commit
>
> offset request (see e.g. the ListConsumerGroupOffsetsResult which
>
> exclude
>
> partitions that have errors, or the DeleteMembersResult as part of
>
> KIP-345).
>
> If the discover-coordinator returns non-triable error, we would set
>
> it
>
> on
>
> the first layer of the KafkaFuture, and the per-partition error would
>
> be
>
> set on the second layer of the KafkaFuture.
>
> Guozhang
>
> On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe 
>
> wrote:
>
> Hi Mickael,
>
> Considering that KIP-496, which adds a way of deleting consumer
>
> offsets
>
> from AdminClient, looks like it is going to get in, this seems like
> functionality we should definitely have.
>
> For alterConsumerGroupOffsets, is the intention to ignore
>
> partitions
>
> that
>
> are not specified in the map? If so, we should specify that in the
>
> JavaDoc.
>
> isolationLevel seems like it should be an enum rather than a
>
> string. The
>
> existing enum is in org.apache.kafka.common.requests, so we should
>
> probably
>
> create a new one which is public in org.apache.kafka.clients.admin.
>
> best,
> Colin
>
> On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
>
> Bumping this thread once again
>
> Ismael, have I answered your questions?
> While this has received a few non-binding +1s, no committers have voted
> yet. If you have concerns or questions, please let me know.
>
> Thanks
>
> On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
>  wrote:
>
> Bumping this thread as it's been a couple of weeks.
>
> On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
>
> mickael.mai...@gmail.com> wrote:
>
> Thanks Ismael for the feedback. I think your point has 2
>
> parts:
>
> - Having the reset functionality in the AdminClient: The fact we have a
> command line tool illustrate that this
>
> operation
>
> is
>
> relatively common. I seems valuable to be able to perform
>
> this
>
> operation directly via a proper API in addition of the CLI
>
> tool.
>
> - Sending an OffsetCommit directly instead of relying on
>
> KafkaConsumer:
>
> The KafkaConsumer requires a lot of stuff to commit offsets.
>
> Its
>
> group
>
> cannot change so you need to start a new Consumer every time,
>
> that
>
> creates new connections and overal sends more requests. Also
>
> there
>
> are
>
> already a bunch of AdminClient APIs that have logic very
>
> close to
>
> what needs to be done to send a commit request, keeping the
>
> code
>
> small
>
> and consistent.
>
> I've updated the KIP with these details and moved the 2nd
>
> part
>
> to
>
> "Proposed changes" as it's more an implementation detail.
>
> I hope this answers your question
>
> On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <
>
> isma...@gmail.com
>
> wrote:
>
> The KIP doesn't discuss the option of using KafkaConsumer
>
> directly
>
> as far
>
> as I can tell. We have tried to avoid having the same
>
> functionality in
>
> multiple clients so it would be good to explain why this is
>
> necessary here
>
> (not saying it isn't).
>
> Ismael
>
> On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
>
> mickael.mai...@gmail.com
>
> wrote:
>
> Thanks Ryanne for the feedback, all suggestions sounded
>
> good,
>
> I've
>
> updated the KIP accordingly.
>
> On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
>
> ryannedo...@gmail.com>
>
> wrote:
>
> +1 (non-binding)
>
> But I suggest:
>
> - drop "get" from getOffset, getTimestamp.
>
> - add to the motivation section why this is better than
>
> constructing a
>
> KafkaConsumer and using seek(), commit() etc.
>
> - add some rejected alternatives.
>
> Ryanne
>
> On Mon,

[jira] [Created] (KAFKA-8806) Kafka.poll spends significant amount of time in KafkaConsumer.updateAssignmentMetadataIfNeeded

[JIRA]

Xavier Léauté created KAFKA-8806:


 Summary: Kafka.poll spends significant amount of time in 
KafkaConsumer.updateAssignmentMetadataIfNeeded
 Key: KAFKA-8806
 URL: https://issues.apache.org/jira/browse/KAFKA-8806
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Xavier Léauté


Comparing the performance profile of 2.2.0 and 2.3.0, we are seeing significant 
performance differences in the 
{{KafkaConumer.updateAssignmentMetadataIfNeeded()}} method.

The call to {{KafkaConsumer.updateAssignmentMetadataIfNeeded()}} now represents 
roughly 40% of CPU time spent in {{KafkaConsumer.poll()}}, when before it only 
represented less than 2%.

Most of the extra time appears to be spent in 
{{KafkaConsumer.validateOffsetsIfNeeded()}}, which did not show up in previous 
profiles.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

[Bill Bejeck]

Thanks for the KIP Mickael, looks very useful.
+1 (binding)

-Bill

On Wed, Aug 14, 2019 at 6:14 PM Harsha Chintalapani  wrote:

> Thanks for the KIP Mickael. LGTM +1 (binding).
> -Harsha
>
>
> On Wed, Aug 14, 2019 at 1:10 PM, Colin McCabe  wrote:
>
> > Thanks, Mickael. +1 (binding)
> >
> > best,
> > Colin
> >
> > On Wed, Aug 14, 2019, at 12:07, Gabor Somogyi wrote:
> >
> > +1 (non-binding)
> > I've read it through in depth and as Jungtaek said Spark can make good
> use
> > of it.
> >
> > On Wed, 14 Aug 2019, 17:06 Jungtaek Lim,  wrote:
> >
> > +1 (non-binding)
> >
> > I found it very useful for Spark's case. (Discussion on KIP-505 described
> > it.)
> >
> > Thanks for driving the effort!
> >
> > 2019년 8월 14일 (수) 오후 8:49, Mickael Maison 님이
> 작성:
> >
> > Hi Guozhang,
> >
> > Thanks for taking a look.
> >
> > 1. Right, I updated the titles of the code blocks
> >
> > 2. Yes that's a good idea. I've updated the KIP
> >
> > Thank you
> >
> > On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
> >  wrote:
> >
> > Hi Colin,
> >
> > Thanks for raising these 2 valid points. I've updated the KIP
> >
> > accordingly.
> >
> > On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang 
> >
> > wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the KIP!
> >
> > Just some minor comments.
> >
> > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java
> > "
> >
> > should
> >
> > be
> >
> > "AlterOffsetsOptions".
> >
> > 2. I'd suggest we change the future structure of "AlterOffsetsResult"
> >
> > to
> >
> > *KafkaFuture>>*
> >
> > This is because we will have a hierarchy of two-layers of errors
> >
> > since
> >
> > we
> >
> > need to find out the group coordinator first and then issue the
> >
> > commit
> >
> > offset request (see e.g. the ListConsumerGroupOffsetsResult which
> >
> > exclude
> >
> > partitions that have errors, or the DeleteMembersResult as part of
> >
> > KIP-345).
> >
> > If the discover-coordinator returns non-triable error, we would set
> >
> > it
> >
> > on
> >
> > the first layer of the KafkaFuture, and the per-partition error would
> >
> > be
> >
> > set on the second layer of the KafkaFuture.
> >
> > Guozhang
> >
> > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe 
> >
> > wrote:
> >
> > Hi Mickael,
> >
> > Considering that KIP-496, which adds a way of deleting consumer
> >
> > offsets
> >
> > from AdminClient, looks like it is going to get in, this seems like
> > functionality we should definitely have.
> >
> > For alterConsumerGroupOffsets, is the intention to ignore
> >
> > partitions
> >
> > that
> >
> > are not specified in the map? If so, we should specify that in the
> >
> > JavaDoc.
> >
> > isolationLevel seems like it should be an enum rather than a
> >
> > string. The
> >
> > existing enum is in org.apache.kafka.common.requests, so we should
> >
> > probably
> >
> > create a new one which is public in org.apache.kafka.clients.admin.
> >
> > best,
> > Colin
> >
> > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> >
> > Bumping this thread once again
> >
> > Ismael, have I answered your questions?
> > While this has received a few non-binding +1s, no committers have voted
> > yet. If you have concerns or questions, please let me know.
> >
> > Thanks
> >
> > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> >  wrote:
> >
> > Bumping this thread as it's been a couple of weeks.
> >
> > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> >
> > mickael.mai...@gmail.com> wrote:
> >
> > Thanks Ismael for the feedback. I think your point has 2
> >
> > parts:
> >
> > - Having the reset functionality in the AdminClient: The fact we have a
> > command line tool illustrate that this
> >
> > operation
> >
> > is
> >
> > relatively common. I seems valuable to be able to perform
> >
> > this
> >
> > operation directly via a proper API in addition of the CLI
> >
> > tool.
> >
> > - Sending an OffsetCommit directly instead of relying on
> >
> > KafkaConsumer:
> >
> > The KafkaConsumer requires a lot of stuff to commit offsets.
> >
> > Its
> >
> > group
> >
> > cannot change so you need to start a new Consumer every time,
> >
> > that
> >
> > creates new connections and overal sends more requests. Also
> >
> > there
> >
> > are
> >
> > already a bunch of AdminClient APIs that have logic very
> >
> > close to
> >
> > what needs to be done to send a commit request, keeping the
> >
> > code
> >
> > small
> >
> > and consistent.
> >
> > I've updated the KIP with these details and moved the 2nd
> >
> > part
> >
> > to
> >
> > "Proposed changes" as it's more an implementation detail.
> >
> > I hope this answers your question
> >
> > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <
> >
> > isma...@gmail.com
> >
> > wrote:
> >
> > The KIP doesn't discuss the option of using KafkaConsumer
> >
> > directly
> >
> > as far
> >
> > as I can tell. We have tried to avoid having the same
> >
> > functionality in
> >
> > multiple clients so it would be good to explain why this is
> >
> > necessary here

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

[Vahid Hashemian]

+1 (binding)

Thanks Michael for the suggestion of simplifying offset
retrieval/alteration operations.

--Vahid

On Wed, Aug 14, 2019 at 4:42 PM Bill Bejeck  wrote:

> Thanks for the KIP Mickael, looks very useful.
> +1 (binding)
>
> -Bill
>
> On Wed, Aug 14, 2019 at 6:14 PM Harsha Chintalapani 
> wrote:
>
> > Thanks for the KIP Mickael. LGTM +1 (binding).
> > -Harsha
> >
> >
> > On Wed, Aug 14, 2019 at 1:10 PM, Colin McCabe 
> wrote:
> >
> > > Thanks, Mickael. +1 (binding)
> > >
> > > best,
> > > Colin
> > >
> > > On Wed, Aug 14, 2019, at 12:07, Gabor Somogyi wrote:
> > >
> > > +1 (non-binding)
> > > I've read it through in depth and as Jungtaek said Spark can make good
> > use
> > > of it.
> > >
> > > On Wed, 14 Aug 2019, 17:06 Jungtaek Lim,  wrote:
> > >
> > > +1 (non-binding)
> > >
> > > I found it very useful for Spark's case. (Discussion on KIP-505
> described
> > > it.)
> > >
> > > Thanks for driving the effort!
> > >
> > > 2019년 8월 14일 (수) 오후 8:49, Mickael Maison 님이
> > 작성:
> > >
> > > Hi Guozhang,
> > >
> > > Thanks for taking a look.
> > >
> > > 1. Right, I updated the titles of the code blocks
> > >
> > > 2. Yes that's a good idea. I've updated the KIP
> > >
> > > Thank you
> > >
> > > On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
> > >  wrote:
> > >
> > > Hi Colin,
> > >
> > > Thanks for raising these 2 valid points. I've updated the KIP
> > >
> > > accordingly.
> > >
> > > On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang 
> > >
> > > wrote:
> > >
> > > Hi Mickael,
> > >
> > > Thanks for the KIP!
> > >
> > > Just some minor comments.
> > >
> > > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java
> > > "
> > >
> > > should
> > >
> > > be
> > >
> > > "AlterOffsetsOptions".
> > >
> > > 2. I'd suggest we change the future structure of "AlterOffsetsResult"
> > >
> > > to
> > >
> > > *KafkaFuture>>*
> > >
> > > This is because we will have a hierarchy of two-layers of errors
> > >
> > > since
> > >
> > > we
> > >
> > > need to find out the group coordinator first and then issue the
> > >
> > > commit
> > >
> > > offset request (see e.g. the ListConsumerGroupOffsetsResult which
> > >
> > > exclude
> > >
> > > partitions that have errors, or the DeleteMembersResult as part of
> > >
> > > KIP-345).
> > >
> > > If the discover-coordinator returns non-triable error, we would set
> > >
> > > it
> > >
> > > on
> > >
> > > the first layer of the KafkaFuture, and the per-partition error would
> > >
> > > be
> > >
> > > set on the second layer of the KafkaFuture.
> > >
> > > Guozhang
> > >
> > > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe 
> > >
> > > wrote:
> > >
> > > Hi Mickael,
> > >
> > > Considering that KIP-496, which adds a way of deleting consumer
> > >
> > > offsets
> > >
> > > from AdminClient, looks like it is going to get in, this seems like
> > > functionality we should definitely have.
> > >
> > > For alterConsumerGroupOffsets, is the intention to ignore
> > >
> > > partitions
> > >
> > > that
> > >
> > > are not specified in the map? If so, we should specify that in the
> > >
> > > JavaDoc.
> > >
> > > isolationLevel seems like it should be an enum rather than a
> > >
> > > string. The
> > >
> > > existing enum is in org.apache.kafka.common.requests, so we should
> > >
> > > probably
> > >
> > > create a new one which is public in org.apache.kafka.clients.admin.
> > >
> > > best,
> > > Colin
> > >
> > > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > >
> > > Bumping this thread once again
> > >
> > > Ismael, have I answered your questions?
> > > While this has received a few non-binding +1s, no committers have voted
> > > yet. If you have concerns or questions, please let me know.
> > >
> > > Thanks
> > >
> > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > >  wrote:
> > >
> > > Bumping this thread as it's been a couple of weeks.
> > >
> > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > >
> > > mickael.mai...@gmail.com> wrote:
> > >
> > > Thanks Ismael for the feedback. I think your point has 2
> > >
> > > parts:
> > >
> > > - Having the reset functionality in the AdminClient: The fact we have a
> > > command line tool illustrate that this
> > >
> > > operation
> > >
> > > is
> > >
> > > relatively common. I seems valuable to be able to perform
> > >
> > > this
> > >
> > > operation directly via a proper API in addition of the CLI
> > >
> > > tool.
> > >
> > > - Sending an OffsetCommit directly instead of relying on
> > >
> > > KafkaConsumer:
> > >
> > > The KafkaConsumer requires a lot of stuff to commit offsets.
> > >
> > > Its
> > >
> > > group
> > >
> > > cannot change so you need to start a new Consumer every time,
> > >
> > > that
> > >
> > > creates new connections and overal sends more requests. Also
> > >
> > > there
> > >
> > > are
> > >
> > > already a bunch of AdminClient APIs that have logic very
> > >
> > > close to
> > >
> > > what needs to be done to send a commit request, keeping the
> > >
> > > code
> > >
> >

Build failed in Jenkins: kafka-trunk-jdk8 #3853

[Apache Jenkins Server]

See 


Changes:

[cmccabe] KAFKA-8345: KIP-455: Admin API changes (Part 2) (#7120)

[jason] KAFKA-7335; Store clusterId locally to ensure broker joins the right

--
[...truncated 2.60 MB...]
org.apache.kafka.connect.json.JsonConverterTest > mapToJsonNonStringKeys STARTED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonNonStringKeys PASSED

org.apache.kafka.connect.json.JsonConverterTest > longToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > longToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > mismatchSchemaJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > mismatchSchemaJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToConnectNonStringKeys 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > mapToConnectNonStringKeys 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
testJsonSchemaMetadataTranslation STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
testJsonSchemaMetadataTranslation PASSED

org.apache.kafka.connect.json.JsonConverterTest > bytesToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > bytesToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > shortToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > shortToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > intToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > intToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndNullValueToJson 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndNullValueToJson 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > timestampToConnectOptional 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > timestampToConnectOptional 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > structToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > structToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > stringToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > stringToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndArrayToJson 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndArrayToJson 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > byteToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > byteToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaPrimitiveToConnect 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaPrimitiveToConnect 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > byteToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > byteToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > intToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > intToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > dateToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndPrimitiveToJson 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndPrimitiveToJson 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
decimalToConnectOptionalWithDefaultValue STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
decimalToConnectOptionalWithDefaultValue PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonStringKeys STARTED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonStringKeys PASSED

org.apache.kafka.connect.json.JsonConverterTest > arrayToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > arrayToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > timeToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > timeToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > structToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > structToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
testConnectSchemaMetadataTranslation STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
testConnectSchemaMetadataTranslation PASSED

org.apache.kafka.connect.json.JsonConverterTest > shortToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > shortToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnect PASSED

org.apach

Request permission

[Mario Molina]

Hi,

I'd like to create a new page for a KIP in the Apache Kafka Confluence site.
The implementation of this KIP is already done but changes some things in
the API that should be approved. You can check it out here
.

My ID in Confluence is "mmolimar".

Regards,
Mario

Jenkins build is back to normal : kafka-trunk-jdk11 #754

[Apache Jenkins Server]

See 

Apply to be contributor of kafka

[xiemian]

Hello,

I am a graduate student studying in United States, major in computer science. I 
am much interested on being a contributor of kafka. Before going to school 
again, I actually had a couple years working experience on Hadoop ecosystem. I 
have built clusters to satisfy the real world requirement and also modified 
some source code of YARN and MapReduce in order to make an improvement. I have 
already read some source code of kafka and would like to be a contributor since 
contributing myself to the open source community is my goal. So could you 
please tell me what I need to do?

My ASF JIRA username is: mianxieEmail: xiemian...@gmail.com

Thanks for your time and patience. I am really looking forward to your reply.

Best.
Mian Xie

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

[Ryanne Dolan]

Chris, I don't understand how the rebalance protocol can be used to give
out session tokens in a secure way. It seems that any attacker could just
join the group and sign requests with the provided token. Am I missing
something?

Ryanne

On Wed, Aug 14, 2019, 2:31 PM Chris Egerton  wrote:

> The KIP page can be found at
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> ,
> by the way. Apologies for neglecting to include it in my initial email!
>
> On Wed, Aug 14, 2019 at 12:29 PM Chris Egerton 
> wrote:
>
> > Hi all,
> >
> > I'd like to start discussion on a KIP to secure the internal "POST
> > /connectors//tasks" endpoint for the Connect framework. The
> proposed
> > changes address a vulnerability in the framework in its current state
> that
> > allows malicious users to write arbitrary task configurations for
> > connectors; it is vital that this issue be addressed in order for any
> > Connect cluster to be secure.
> >
> > Looking forward to your thoughts,
> >
> > Chris
> >
>

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

[Maulin Vasavada]

Just to update - still working on it. Get to work only on and off on it :(

On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada 
wrote:

> Hi Harsha
>
> Let me try to write samples and will let you know.
>
> Thanks
> Maulin
>
> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch  wrote:
>
>> Hi Maulin,
>>  With java security providers can be as custom you would like it
>> to
>> be. If you only want to to implement a custom way of loading the
>> keystore and truststore and not implement any protocol/encryption handling
>> you can leave them empty and no need to implement.
>> Have you looked into the links I pasted before?
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>>
>> Can you please tell me which methods are too complex in above to implement
>> or unnecessary? You are changing anything in SSL/TLS implementations
>> provided by
>>
>> All of the implementations delegating the checks to the default
>> implementation anyway.
>> Spire agent is an example, its nothing but a GRPC server listening on a
>> unix domain socket . Above code is making a RPC call to the local daemon
>> to
>> get the certificate and keys. The mechanics are pretty much same as what
>> you are asking for.
>>
>> Thanks,
>> Harsha
>>
>> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada > >
>> wrote:
>>
>> > Imagine a scenario like - We know we have a custom KMS and as a Kafka
>> owner
>> > we want to comply to using that KMS source to load keys/certs. As a
>> Kafka
>> > owner we know how to integrate with KMS but doesn't necessarily have to
>> > know anything about cipher suites, algorithms, and SSL/TLS
>> implementation.
>> > Going the Provider way requires to know lot more than we should, isn't
>> it?
>> > Not that we would have concern/shy-away knowing those details - but if
>> we
>> > don't have to - why should we?
>> >
>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>> maulin.vasav...@gmail.com>
>> > wrote:
>> >
>> > > Hi Harsha
>> > >
>> > > We don't have spire (or similar) agents and we do not have keys/certs
>> > > locally on any brokers. To elaborate more on my previous email,
>> > >
>> > > I agree that Java security Providers are used in much broader sense -
>> to
>> > > have a particular implementation of an algorithm, use specific cipher
>> > > suites for SSL , OR  in our current team's case have a particular way
>> to
>> > > leverage pre-generated SSL sessions. However, the scope of our KIP
>> (486)
>> > is
>> > > much restricted than that. We merely intend to provide a custom
>> > > keystore/truststore for our SSL connections and not really worry about
>> > > underlying specific SSL/TLS implementation.  This simplifies it a lot
>> for
>> > > us to keep the concerns separate and clear.
>> > >
>> > > I feel our approach is more complimentary such that it allows for
>> using
>> > > keystores of choice while retaining the flexibility to use any
>> > > underlying/available Provider for actually making the SSL call.
>> > >
>> > > We agree with KIP-492's approach based on Providers (and Java's
>> > > recommendation), but also strongly believe that our approach can
>> > compliment
>> > > it very effectively for reasons explained above.
>> > >
>> > > Thanks
>> > > Maulin
>> > >
>> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani 
>> > > wrote:
>> > >
>> > >> Hi Maulin,
>> > >>
>> > >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>> > >> maulin.vasav...@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > Hi Harsha
>> > >> >
>> > >> > The reason we rejected the SslProvider route is that - we only
>> needed
>> > a
>> > >> > custom way to load keys/certs. Not touch any policy that existing
>> > >> Providers
>> > >> > govern like SunJSSE Provider.
>> > >> >
>> > >>
>> > >> We have exactly the same requirements to load certs and keys through
>> > spire
>> > >> agent. We used security.provider to do that exactly. I am not sure
>> why
>> > you
>> > >> would be modifying any policies provided by default SunJSSE provider.
>> > Can
>> > >> you give me an example of having custom provider that will override
>> an
>> > >> existing policy in  SunJSSE provider.
>> > >>
>> > >> As pointed out earlier, this kip
>> > >>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>> > >> allows
>> > >> you to  load security.provider through config.
>> > >> Take a look at the examples I gave before
>> > >>
>> > >>
>> >
>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>> > >> It registers