[jira] [Created] (KAFKA-8343) streams application crashed due to rocksdb

[gaoshu (JIRA)]

gaoshu created KAFKA-8343:
-

 Summary: streams application crashed due to rocksdb
 Key: KAFKA-8343
 URL: https://issues.apache.org/jira/browse/KAFKA-8343
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.0
 Environment: centos 7 jdk8 kafka-streams1.0
Reporter: gaoshu
 Attachments: fullsizeoutput_1.jpeg

my streams application always crashed in few days.  The crash log looks like 
[https://github.com/facebook/rocksdb/issues/5234|[https://github.com/facebook/rocksdb/issues/5234].]
  so I think it may because of RocksDBStore.java closed incorrectly in 
multithread.  I look through the below code,  it means the db.close()  should 
after openiterators.close(). However, db.close() may be executed before 
iterators.close() due to instructions reorder. I hope my guess is correct.
{code:java}
// RocksDBStore.java
@Override
public synchronized void close() {
if (!open) {
return;
}

open = false;
closeOpenIterators();
options.close();
wOptions.close();
fOptions.close();
db.close();

options = null;
wOptions = null;
fOptions = null;
db = null;
}
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-461 Improve Replica Fetcher behavior at handling partition failure

[Dhruvil Shah]

Thanks for the KIP! +1 (non-binding)

On Wed, May 8, 2019 at 10:23 PM Colin McCabe  wrote:

> +1.  Thanks, Aishwarya.
>
> Colin
>
> On Wed, May 8, 2019, at 17:50, Jason Gustafson wrote:
> > +1. Thanks!
> >
> > On Wed, May 8, 2019 at 4:30 PM Aishwarya Gune 
> > wrote:
> >
> > > Hi All!
> > >
> > > I would like to call for a vote on KIP-461 that would improve the
> behavior
> > > of replica fetcher in case of partition failure. The fetcher thread
> would
> > > just stop monitoring the crashed partition instead of terminating.
> > >
> > > Here's a link to the KIP -
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-461+-+Improve+Replica+Fetcher+behavior+at+handling+partition+failure
> > >
> > > Discussion thread -
> > > https://www.mail-archive.com/dev@kafka.apache.org/msg97559.html
> > >
> > > --
> > > Thank you,
> > > Aishwarya
> > >
> >
>

Re: [VOTE] KIP-461 Improve Replica Fetcher behavior at handling partition failure

[Colin McCabe]

+1.  Thanks, Aishwarya.

Colin

On Wed, May 8, 2019, at 17:50, Jason Gustafson wrote:
> +1. Thanks!
> 
> On Wed, May 8, 2019 at 4:30 PM Aishwarya Gune 
> wrote:
> 
> > Hi All!
> >
> > I would like to call for a vote on KIP-461 that would improve the behavior
> > of replica fetcher in case of partition failure. The fetcher thread would
> > just stop monitoring the crashed partition instead of terminating.
> >
> > Here's a link to the KIP -
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-461+-+Improve+Replica+Fetcher+behavior+at+handling+partition+failure
> >
> > Discussion thread -
> > https://www.mail-archive.com/dev@kafka.apache.org/msg97559.html
> >
> > --
> > Thank you,
> > Aishwarya
> >
>

[VOTE] KIP-455: Create an Administrative API for Replica Reassignment

[Colin McCabe]

Hi all,

I'd like to start the vote for KIP-455: Create an Administrative API for 
Replica Reassignment.  I think this KIP is important since it will unlock many 
follow-on improvements to Kafka reassignment (see the "Future work" section, 
plus a lot of the other discussions we've had recently about reassignment).  It 
also furthers the important KIP-4 goal of removing direct access to ZK.

I made a few changes based on the discussion in the [DISCUSS] thread.  As 
Robert suggested, I removed the need to explicitly cancel a reassignment for a 
partition before setting up a different reassignment for that specific 
partition.  I also simplified the API a bit by adding a PartitionReassignment 
class which is used by both the alter and list APIs.

I modified the proposal so that we now deprecate the old znode-based API rather 
than removing it completely.  That should give external rebalancing tools some 
time to transition to the new API.

To clarify a question Viktor asked, I added a note that the 
kafka-reassign-partitions.sh will now use a --bootstrap-server argument to 
contact the admin APIs. 

thanks,
Colin

[jira] [Resolved] (KAFKA-8284) Enable static membership on KStream

[Boyang Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen resolved KAFKA-8284.

Resolution: Fixed

> Enable static membership on KStream
> ---
>
> Key: KAFKA-8284
> URL: https://issues.apache.org/jira/browse/KAFKA-8284
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk8 #3611

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk11 #500

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: cleanup policy doc update (#6692)

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H29 (ubuntu xenial) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision cc4a7f01e872e6d0664360d5148af09c876ca72b 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f cc4a7f01e872e6d0664360d5148af09c876ca72b
Commit message: "MINOR: cleanup policy doc update (#6692)"
 > git rev-list --no-walk e6cff21fd8c5add0eb7e55417a91f0530a7d3a32 # timeout=10
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
[kafka-trunk-jdk11] $ /bin/bash -xe /tmp/jenkins469090719776719255.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.10.2/bin/gradle
/tmp/jenkins469090719776719255.sh: line 4: 
/home/jenkins/tools/gradle/4.10.2/bin/gradle: No such file or directory
Build step 'Execute shell' marked build as failure
Recording test results
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Not sending mail to unregistered user nore...@github.com
Not sending mail to unregistered user csh...@gmail.com
Not sending mail to unregistered user wangg...@gmail.com

[jira] [Created] (KAFKA-8342) Admin tool to setup Kafka Stream topology (internal) topics

[Boyang Chen (JIRA)]

Boyang Chen created KAFKA-8342:
--

 Summary: Admin tool to setup Kafka Stream topology (internal) 
topics
 Key: KAFKA-8342
 URL: https://issues.apache.org/jira/browse/KAFKA-8342
 Project: Kafka
  Issue Type: Bug
Reporter: Boyang Chen


We have seen customers who need to deploy their application to production 
environment but don't have access to create changelog and repartition topics. 
They need to ask admin team to manually create those topics before proceeding 
to start the actual stream job. We could add an admin tool to help them go 
through the process quicker by providing a command that could
 # Read through current stream topology
 # Create corresponding topics as needed, even including output topics.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams

[Guozhang Wang]

Hello all,

Thanks for everyone who've shared their feedbacks for this KIP! If there's
no further comments I'll start the voting thread by end of tomorrow.


Guozhang.

On Wed, May 8, 2019 at 6:36 PM Guozhang Wang  wrote:

> Hello Boyang,
>
> On Wed, May 1, 2019 at 4:51 PM Boyang Chen  wrote:
>
>> Hey Guozhang,
>>
>> thank you for the great write up. Overall the motivation and changes
>> LGTM, just some minor comments:
>>
>>
>>   1.  In "Consumer Coordinator Algorithm", we could reorder alphabet
>> points for 3d~3f from ["ready-to-migrate-partitions",
>> "unknown-but-owned-partitions",  "maybe-revoking-partitions"] to
>> ["maybe-revoking-partitions", "ready-to-migrate-partitions",
>> "unknown-but-owned-partitions"] in order to be consistent with 3c1~3.
>>
>
> Ack. Updated.
>
>
>>   2.  In "Consumer Coordinator Algorithm", 1c suggests to revoke all
>> partition upon heartbeat/commit fail. What's the gain here? Do we want to
>> keep all partitions running at this moment, to be optimistic for the case
>> when no partitions get reassigned?
>>
>
> That's a good catch. When REBALANCE_IN_PROGRESS is received, we can just
> re-join the group with all the currently owned partitions encoded. Updated.
>
>
>>   3.  In "Recommended Upgrade Procedure", remove extra 'those': " The
>> 'sticky' assignor works even those there are "
>>
>
> Ack, should be `even when`.
>
>
>>   4.  Put two "looking into the future" into a separate category from
>> migration session. It seems inconsistent for readers to see this before we
>> finished discussion for everything.
>>
>
> Ack.
>
>
>>   5.  Have we discussed the concern on the serialization? Could the new
>> metadata we are adding grow larger than the message size cap?
>>
>
> We're completing https://issues.apache.org/jira/browse/KAFKA-7149 which
> should largely reduce the message size (will update the wiki accordingly as
> well).
>
>
>>
>> Boyang
>>
>> 
>> From: Guozhang Wang 
>> Sent: Monday, April 15, 2019 9:20 AM
>> To: dev
>> Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams
>>
>> Hello Jason,
>>
>> I agree with you that for range / round-robin it makes less sense to be
>> compatible with cooperative rebalance protocol.
>>
>> As for StickyAssignor, however, I think it would still be possible to make
>> the current implementation to be compatible with cooperative rebalance. So
>> after pondering on different options at hand I'm now proposing this
>> approach as listed in the upgrade section:
>>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol#KIP-429:KafkaConsumerIncrementalRebalanceProtocol-CompatibilityandUpgradePath
>>
>> The idea is to let assignors specify which protocols it would work with,
>> associating with a different name; then the upgrade path would involve a
>> "compatible" protocol which actually still use eager behavior while
>> encoding two assignors if possible. In "Rejected Section" (just to clarify
>> I'm not finalizing it as rejected, just putting it there for now, and if
>> we
>> like this one instead we can always switch them) I listed the other
>> approach we once discussed about, and arguing its cons of duplicated class
>> seems overwhelm the pros of saving the  "rebalance.protocol" config.
>>
>> Let me know WDYT.
>>
>> Guozhang
>>
>> On Fri, Apr 12, 2019 at 6:08 PM Jason Gustafson 
>> wrote:
>>
>> > Hi Guozhang,
>> >
>> > Responses below:
>> >
>> > 2. The interface's default implementation will just be
>> > > `onPartitionRevoked`, so for user's instantiation if they do not make
>> any
>> > > code changes they should be able to recompile the code and continue.
>> >
>> >
>> > Ack, makes sense.
>> >
>> > 4. Hmm.. not sure if it will work. The main issue is that the
>> > > consumer-coordinator behavior (whether to revoke all or none at
>> > > onRebalancePrepare) is independent of the selected protocol's assignor
>> > > (eager or cooperative), so even if the assignor is selected to be the
>> > > old-versioned one, we will still not revoke at the
>> consumer-coordinator
>> > > layer and hence has the same risk of migrating still-owned partitions,
>> > > right?
>> >
>> >
>> > Yeah, basically we would have to push the eager/cooperative logic into
>> the
>> > PartitionAssignor itself and make the consumer aware of the rebalance
>> > protocol it is compatible with. As long as an eager protocol _could_ be
>> > selected, the consumer would have to be pessimistic and do eager
>> > revocation. But if all the assignors configured in the consumer support
>> > cooperative reassignment, then either 1) a cooperative protocol will be
>> > selected and cooperative revocation can be safely used, or 2) if the
>> rest
>> > of the group does not support it, then the consumer will simply fail.
>> >
>> > Another point which you raised offline and I will repeat here is that
>> this
>> > proposal's benefit is mostly limited to sticky assignment logic.
>> Arguably
>> 

Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams

[Guozhang Wang]

Hello Boyang,

On Wed, May 1, 2019 at 4:51 PM Boyang Chen  wrote:

> Hey Guozhang,
>
> thank you for the great write up. Overall the motivation and changes LGTM,
> just some minor comments:
>
>
>   1.  In "Consumer Coordinator Algorithm", we could reorder alphabet
> points for 3d~3f from ["ready-to-migrate-partitions",
> "unknown-but-owned-partitions",  "maybe-revoking-partitions"] to
> ["maybe-revoking-partitions", "ready-to-migrate-partitions",
> "unknown-but-owned-partitions"] in order to be consistent with 3c1~3.
>

Ack. Updated.


>   2.  In "Consumer Coordinator Algorithm", 1c suggests to revoke all
> partition upon heartbeat/commit fail. What's the gain here? Do we want to
> keep all partitions running at this moment, to be optimistic for the case
> when no partitions get reassigned?
>

That's a good catch. When REBALANCE_IN_PROGRESS is received, we can just
re-join the group with all the currently owned partitions encoded. Updated.


>   3.  In "Recommended Upgrade Procedure", remove extra 'those': " The
> 'sticky' assignor works even those there are "
>

Ack, should be `even when`.


>   4.  Put two "looking into the future" into a separate category from
> migration session. It seems inconsistent for readers to see this before we
> finished discussion for everything.
>

Ack.


>   5.  Have we discussed the concern on the serialization? Could the new
> metadata we are adding grow larger than the message size cap?
>

We're completing https://issues.apache.org/jira/browse/KAFKA-7149 which
should largely reduce the message size (will update the wiki accordingly as
well).


>
> Boyang
>
> 
> From: Guozhang Wang 
> Sent: Monday, April 15, 2019 9:20 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams
>
> Hello Jason,
>
> I agree with you that for range / round-robin it makes less sense to be
> compatible with cooperative rebalance protocol.
>
> As for StickyAssignor, however, I think it would still be possible to make
> the current implementation to be compatible with cooperative rebalance. So
> after pondering on different options at hand I'm now proposing this
> approach as listed in the upgrade section:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol#KIP-429:KafkaConsumerIncrementalRebalanceProtocol-CompatibilityandUpgradePath
>
> The idea is to let assignors specify which protocols it would work with,
> associating with a different name; then the upgrade path would involve a
> "compatible" protocol which actually still use eager behavior while
> encoding two assignors if possible. In "Rejected Section" (just to clarify
> I'm not finalizing it as rejected, just putting it there for now, and if we
> like this one instead we can always switch them) I listed the other
> approach we once discussed about, and arguing its cons of duplicated class
> seems overwhelm the pros of saving the  "rebalance.protocol" config.
>
> Let me know WDYT.
>
> Guozhang
>
> On Fri, Apr 12, 2019 at 6:08 PM Jason Gustafson 
> wrote:
>
> > Hi Guozhang,
> >
> > Responses below:
> >
> > 2. The interface's default implementation will just be
> > > `onPartitionRevoked`, so for user's instantiation if they do not make
> any
> > > code changes they should be able to recompile the code and continue.
> >
> >
> > Ack, makes sense.
> >
> > 4. Hmm.. not sure if it will work. The main issue is that the
> > > consumer-coordinator behavior (whether to revoke all or none at
> > > onRebalancePrepare) is independent of the selected protocol's assignor
> > > (eager or cooperative), so even if the assignor is selected to be the
> > > old-versioned one, we will still not revoke at the consumer-coordinator
> > > layer and hence has the same risk of migrating still-owned partitions,
> > > right?
> >
> >
> > Yeah, basically we would have to push the eager/cooperative logic into
> the
> > PartitionAssignor itself and make the consumer aware of the rebalance
> > protocol it is compatible with. As long as an eager protocol _could_ be
> > selected, the consumer would have to be pessimistic and do eager
> > revocation. But if all the assignors configured in the consumer support
> > cooperative reassignment, then either 1) a cooperative protocol will be
> > selected and cooperative revocation can be safely used, or 2) if the rest
> > of the group does not support it, then the consumer will simply fail.
> >
> > Another point which you raised offline and I will repeat here is that
> this
> > proposal's benefit is mostly limited to sticky assignment logic. Arguably
> > the range assignor may have some incidental stickiness, particularly if
> the
> > group is rebalancing for a newly created or deleted topic. For other
> cases,
> > the proposal is mostly additional overhead since it takes an additional
> > rebalance and many of the partitions will move. Perhaps it doesn't make
> as
> > much sense to use the cooperative protocol for 

Re: [VOTE] KIP-461 Improve Replica Fetcher behavior at handling partition failure

[Jason Gustafson]

+1. Thanks!

On Wed, May 8, 2019 at 4:30 PM Aishwarya Gune 
wrote:

> Hi All!
>
> I would like to call for a vote on KIP-461 that would improve the behavior
> of replica fetcher in case of partition failure. The fetcher thread would
> just stop monitoring the crashed partition instead of terminating.
>
> Here's a link to the KIP -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-461+-+Improve+Replica+Fetcher+behavior+at+handling+partition+failure
>
> Discussion thread -
> https://www.mail-archive.com/dev@kafka.apache.org/msg97559.html
>
> --
> Thank you,
> Aishwarya
>

Re: [VOTE] KIP-458: Connector Client Config Override Policy

[Magesh Nandakumar]

Thanks a lot Chris. So far, the KIP has one non-binding vote and I'm still
looking forward to the KIP to be voted by Friday's deadline.

On Tue, May 7, 2019 at 10:00 AM Chris Egerton  wrote:

> Hi Magesh,
>
> This looks great! Very excited to see these changes finally coming to
> Connect.
> +1 (non-binding)
>
> Cheers,
>
> Chris
>
> On Tue, May 7, 2019 at 9:51 AM Magesh Nandakumar 
> wrote:
>
> > Hi All,
> >
> > I would like to start a vote on
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy
> >
> > The discussion thread can be found here
> > .
> >
> > Thanks,
> > Magesh
> >
>

[VOTE] KIP-461 Improve Replica Fetcher behavior at handling partition failure

[Aishwarya Gune]

Hi All!

I would like to call for a vote on KIP-461 that would improve the behavior
of replica fetcher in case of partition failure. The fetcher thread would
just stop monitoring the crashed partition instead of terminating.

Here's a link to the KIP -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-461+-+Improve+Replica+Fetcher+behavior+at+handling+partition+failure

Discussion thread -
https://www.mail-archive.com/dev@kafka.apache.org/msg97559.html

-- 
Thank you,
Aishwarya

[jira] [Reopened] (KAFKA-7830) Convert Kafka RPCs to use automatically generated code

[Boyang Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen reopened KAFKA-7830:


> Convert Kafka RPCs to use automatically generated code
> --
>
> Key: KAFKA-7830
> URL: https://issues.apache.org/jira/browse/KAFKA-7830
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, core
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>Priority: Major
>
> KAFKA-7609 added a way of automatically generating code for reading and 
> writing Kafka RPC message types from JSON schemas.
> Automatically generated code is preferrable to manually written serialization 
> code. 
> * * It is less tedious and error-prone to use than hand-written code.
> * For developers writing Kafka clients in other languages, the JSON schemas 
> are useful in a way that the java serialization code is not.
> * It will eventually be possible to automatically validate aspects of 
> cross-version compatibility, when using JSON message schemas.
> * Once all of the RPCs are converted, we can drop using Structs in favor of 
> serializing directly to ByteBuffer, to reduce GC load.
> This JIRA tracks converting the current hand-written message serialization 
> code to automatically generated serialization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-8196) Replace InitProducerId request/response with automated protocol

[Boyang Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen resolved KAFKA-8196.

Resolution: Fixed

> Replace InitProducerId request/response with automated protocol
> ---
>
> Key: KAFKA-8196
> URL: https://issues.apache.org/jira/browse/KAFKA-8196
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Mickael Maison
>Assignee: Jason Gustafson
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7830) Convert Kafka RPCs to use automatically generated code

[Boyang Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen resolved KAFKA-7830.

Resolution: Fixed

> Convert Kafka RPCs to use automatically generated code
> --
>
> Key: KAFKA-7830
> URL: https://issues.apache.org/jira/browse/KAFKA-7830
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, core
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>Priority: Major
>
> KAFKA-7609 added a way of automatically generating code for reading and 
> writing Kafka RPC message types from JSON schemas.
> Automatically generated code is preferrable to manually written serialization 
> code. 
> * * It is less tedious and error-prone to use than hand-written code.
> * For developers writing Kafka clients in other languages, the JSON schemas 
> are useful in a way that the java serialization code is not.
> * It will eventually be possible to automatically validate aspects of 
> cross-version compatibility, when using JSON message schemas.
> * Once all of the RPCs are converted, we can drop using Structs in favor of 
> serializing directly to ByteBuffer, to reduce GC load.
> This JIRA tracks converting the current hand-written message serialization 
> code to automatically generated serialization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-461 - Improving replica fetcher behavior in case of partition failures

[Aishwarya Gune]

Hi Jun!

Yes, we should exclude. When a replica is deleted with StopReplicaRequest,
the partition is removed from the set of failed partitions. Will update the
KIP to mention it.

On Wed, May 8, 2019 at 1:59 PM Jun Rao  wrote:

> Hi, Aishwarya,
>
> Thanks for the KIP. Looks good to me. Just one minor comment. If a replica
> is deleted on a broker (through a StopReplicaRequest) while it's in the
> failed partition set, should we exclude that partition from the set and
> the FailedPartitionsCount?
>
> Jun
>
> On Mon, May 6, 2019 at 1:21 PM Aishwarya Gune 
> wrote:
>
> > Hey All!
> >
> > I have created a KIP to improve the behavior of replica fetcher when
> > partition failure occurs. Please do have a look at it and let me know
> what
> > you think.
> > KIP 461 -
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-461+-+Improve+Replica+Fetcher+behavior+at+handling+partition+failure
> >
> > --
> > Thank you,
> > Aishwarya
> >
>


-- 
Thank you,
Aishwarya

Build failed in Jenkins: kafka-trunk-jdk8 #3610

[Apache Jenkins Server]

See 


Changes:

[bbejeck] BUGFIX: Add missing recording of close of stand-by task (#6663)

--
[...truncated 4.64 MB...]

kafka.server.MultipleListenersWithDefaultJaasContextTest > testProduceConsume 
STARTED

kafka.server.MultipleListenersWithDefaultJaasContextTest > testProduceConsume 
PASSED

kafka.server.ServerMetricsTest > testMetricsConfig STARTED

kafka.server.ServerMetricsTest > testMetricsConfig PASSED

kafka.server.MetadataRequestTest > testIsrAfterBrokerShutDownAndJoinsBack 
STARTED

kafka.server.MetadataRequestTest > testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.server.MetadataRequestTest > testReplicaDownResponse STARTED

kafka.server.MetadataRequestTest > testReplicaDownResponse PASSED

kafka.server.MetadataRequestTest > testAutoCreateOfCollidingTopics STARTED

kafka.server.MetadataRequestTest > testAutoCreateOfCollidingTopics PASSED

kafka.server.MetadataRequestTest > testRack STARTED

kafka.server.MetadataRequestTest > testRack PASSED

kafka.server.MetadataRequestTest > testIsInternal STARTED

kafka.server.MetadataRequestTest > testIsInternal PASSED

kafka.server.MetadataRequestTest > testControllerId STARTED

kafka.server.MetadataRequestTest > testControllerId PASSED

kafka.server.MetadataRequestTest > testAliveBrokersWithNoTopics STARTED

kafka.server.MetadataRequestTest > testAliveBrokersWithNoTopics PASSED

kafka.server.MetadataRequestTest > testAllTopicsRequest STARTED

kafka.server.MetadataRequestTest > testAllTopicsRequest PASSED

kafka.server.MetadataRequestTest > testClusterIdIsValid STARTED

kafka.server.MetadataRequestTest > testClusterIdIsValid PASSED

kafka.server.MetadataRequestTest > testNoTopicsRequest STARTED

kafka.server.MetadataRequestTest > testNoTopicsRequest PASSED

kafka.server.MetadataRequestTest > 
testAutoCreateTopicWithInvalidReplicationFactor STARTED

kafka.server.MetadataRequestTest > 
testAutoCreateTopicWithInvalidReplicationFactor PASSED

kafka.server.MetadataRequestTest > testPreferredReplica STARTED

kafka.server.MetadataRequestTest > testPreferredReplica PASSED

kafka.server.MetadataRequestTest > testClusterIdWithRequestVersion1 STARTED

kafka.server.MetadataRequestTest > testClusterIdWithRequestVersion1 PASSED

kafka.server.MetadataRequestTest > testAutoTopicCreation STARTED

kafka.server.MetadataRequestTest > testAutoTopicCreation PASSED

kafka.server.DelayedOperationTest > testRequestPurge STARTED

kafka.server.DelayedOperationTest > testRequestPurge PASSED

kafka.server.DelayedOperationTest > testRequestExpiry STARTED

kafka.server.DelayedOperationTest > testRequestExpiry PASSED

kafka.server.DelayedOperationTest > 
shouldReturnNilOperationsOnCancelForKeyWhenKeyDoesntExist STARTED

kafka.server.DelayedOperationTest > 
shouldReturnNilOperationsOnCancelForKeyWhenKeyDoesntExist PASSED

kafka.server.DelayedOperationTest > testDelayedOperationLockOverride STARTED

kafka.server.DelayedOperationTest > testDelayedOperationLockOverride PASSED

kafka.server.DelayedOperationTest > testTryCompleteLockContention STARTED

kafka.server.DelayedOperationTest > testTryCompleteLockContention PASSED

kafka.server.DelayedOperationTest > testTryCompleteWithMultipleThreads STARTED

kafka.server.DelayedOperationTest > testTryCompleteWithMultipleThreads PASSED

kafka.server.DelayedOperationTest > 
shouldCancelForKeyReturningCancelledOperations STARTED

kafka.server.DelayedOperationTest > 
shouldCancelForKeyReturningCancelledOperations PASSED

kafka.server.DelayedOperationTest > testRequestSatisfaction STARTED

kafka.server.DelayedOperationTest > testRequestSatisfaction PASSED

kafka.server.DelayedOperationTest > testDelayedOperationLock STARTED

kafka.server.DelayedOperationTest > testDelayedOperationLock PASSED

kafka.server.ReplicaFetchTest > testReplicaFetcherThread STARTED

kafka.server.ReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.DeleteTopicsRequestTest > testValidDeleteTopicRequests STARTED

kafka.server.DeleteTopicsRequestTest > testValidDeleteTopicRequests PASSED

kafka.server.DeleteTopicsRequestTest > testErrorDeleteTopicRequests STARTED

kafka.server.DeleteTopicsRequestTest > testErrorDeleteTopicRequests PASSED

kafka.server.DeleteTopicsRequestTest > testNotController STARTED

kafka.server.DeleteTopicsRequestTest > testNotController PASSED

kafka.server.AdvertiseBrokerTest > testBrokerAdvertiseHostNameAndPortToZK 
STARTED

kafka.server.AdvertiseBrokerTest > testBrokerAdvertiseHostNameAndPortToZK PASSED

kafka.server.ListOffsetsRequestTest > testListOffsetsErrorCodes STARTED

kafka.server.ListOffsetsRequestTest > testListOffsetsErrorCodes PASSED

kafka.server.ListOffsetsRequestTest > testCurrentEpochValidation STARTED

kafka.server.ListOffsetsRequestTest > testCurrentEpochValidation PASSED

kafka.server.ListOffsetsRequestTest > testResponseIncludesLeaderEpoch STARTED

kafka.server.ListOffsetsRequestTest > 

Re: [VOTE] KIP-454: Expansion of the ConnectClusterState interface

[Chris Egerton]

Hi all,

I am closing this vote as it has been open for six days now and has
achieved lazy majority. Thanks to everyone who voted!

Summary:
Binding (+3): Randall Hauch, Manikumar Reddy, Rajini Sivaram
Non-binding (+2): Magesh Nandakumar, Konstantine Karantasis

If you'd like to check out the implementation PR I've opened, you can view
(and review!) it at https://github.com/apache/kafka/pull/6584.

Cheers,

Chris

On Wed, May 8, 2019 at 12:16 PM Rajini Sivaram 
wrote:

> Hi Chris,
>
> Thanks for the KIP, +1 (binding)
>
> Regards,
>
> Rajini
>
> On Wed, May 8, 2019 at 7:28 PM Manikumar 
> wrote:
>
> > Hi Chris,
> >
> > +1 (binding). Thanks for the KIP.
> >
> > Thanks,
> > Manikumar
> >
> > On Tue, May 7, 2019 at 2:55 AM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > > Nice and useful KIP. Thanks Chris!
> > >
> > > +1 (non-binding)
> > >
> > > Konstantine
> > >
> > >
> > > On Fri, May 3, 2019 at 5:14 PM Randall Hauch  wrote:
> > >
> > > > Nice job, Chris!
> > > >
> > > > +1 (binding)
> > > >
> > > > On Thu, May 2, 2019 at 8:16 PM Magesh Nandakumar <
> mage...@confluent.io
> > >
> > > > wrote:
> > > >
> > > > > Thanks a lot for the work on this KIP Chris.
> > > > >
> > > > > +1(non-binding)
> > > > >
> > > > > On Thu, May 2, 2019, 4:56 PM Chris Egerton 
> > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to start a vote for KIP-454:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-454%3A+Expansion+of+the+ConnectClusterState+interface
> > > > > >
> > > > > > The discussion thread can be found at
> > > > > > https://www.mail-archive.com/dev@kafka.apache.org/msg96911.html
> > > > > >
> > > > > > Thanks to Konstantine Karantasis and Magesh Nandakumar for their
> > > > > thoughtful
> > > > > > feedback!
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (KAFKA-8341) AdminClient should retry coordinator lookup after NOT_COORDINATOR error

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-8341:
--

 Summary: AdminClient should retry coordinator lookup after 
NOT_COORDINATOR error
 Key: KAFKA-8341
 URL: https://issues.apache.org/jira/browse/KAFKA-8341
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson
Assignee: Vikas Singh


If a group operation (e.g. DescribeGroup) fails because the coordinator has 
moved, the AdminClient should lookup the coordinator before retrying the 
operation. Currently we will either fail or just retry anyway. This is similar 
in some ways to controller rediscovery after getting NOT_CONTROLLER errors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8340) ServiceLoader fails when used from isolated plugin path directory

[Chris Egerton (JIRA)]

Chris Egerton created KAFKA-8340:


 Summary: ServiceLoader fails when used from isolated plugin path 
directory
 Key: KAFKA-8340
 URL: https://issues.apache.org/jira/browse/KAFKA-8340
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Chris Egerton


Under some circumstances, the {{ServiceLoader.load}} mechanism will fail when 
used from an isolated plugin path directory and return an incomplete (often 
empty) {{ServiceLoader}} instance.

 

To replicate:
 * Include a {{META-INF/services/...}} file in one of the JARS located in a 
plugin's directory with one or more implementations of that service listed 
inside. For the sake of example, let's say the name of this service is 
{{com.example.MyService}}
 * Program that plugin to invoke 
{{ServiceLoader.load(com.example.MyService.class)}}
 * Start the Connect framework, making sure this plugin is included on the 
plugin path and that it somehow invokes the {{ServiceLoader.load(...)}} method
 * Observe that the services loaded by that invocation do not include the ones 
described in the {{META-INF/services/...}} file contained in the JAR in the 
plugin's directory

 

This is because the 
[ServiceLoader.load(Class)|https://docs.oracle.com/javase/7/docs/api/java/util/ServiceLoader.html#load(java.lang.Class)]
 method uses the current thread's context classloader to locate resources and 
load services. The current thread's context classloader is, in most cases, an 
instance of {{DelegatingClassLoader}}, which will (unless asked to locate 
resources corresponding to a provider-configuration file for a REST extension 
or config provider) simply delegate resource location to the parent and, unless 
asked to locate a class for a recognized plugin, also delegate class loading to 
the parent. Thus, none of the plugin's JARs are scanned for either 
provider-configuration files or for actual service classes.

A viable workaround for some cases is to instead use the 
[ServiceLoader.load(Class, 
ClassLoader)|https://docs.oracle.com/javase/7/docs/api/java/util/ServiceLoader.html#load(java.lang.Class,%20java.lang.ClassLoader)]
 method, specifying the current class's classloader as the second argument. 
This causes the plugin's {{PluginClassLoader}}, which will scan all JARs in the 
plugin's directory to be used to locate resources and classes.

However, this may not be feasible in all cases, especially when working with 
external libraries that may be difficult or impossible to apply this workaround 
on.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8339) At-least-once delivery guarantee seemingly not met due to async commit / produce failure race condition

[tdp (JIRA)]

tdp created KAFKA-8339:
--

 Summary: At-least-once delivery guarantee seemingly not met due to 
async commit / produce failure race condition
 Key: KAFKA-8339
 URL: https://issues.apache.org/jira/browse/KAFKA-8339
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.0.1
Reporter: tdp


We have hit a race condition several times now between the StreamThread 
committing its offsets for a task before the task has fully processed the 
record through the topology.
 
Consider part of a topology that looks like this:
 
TOPIC T1 -> KSTREAM-SOURCE-NODE1 > KSTREAM-TRANSFORMVALUES-NODE1 > 
KSTREAM-FILTER-NODE1 > KSTREAM-MAPVALUES-NODE1 -> KSTREAM-SINK-NODE1 -> TOPIC T2
 
Records are committed to topic T1. KSTREAM-SOURCE-NODE1 consumes these records 
from topic T1. KSTREAM-TRANSFORMVALUES-NODE1 aggregates these records using a 
local state store. KSTREAM-TRANSFORMVALUES-NODE1 returns null if not all 
necessary records from topic T1 have been consumed yet or an object 
representing an aggregation of records if all necessary records from topic T1 
have been consumed. KSTREAM-FILTER-NODE1 then filters out anything that is 
null. Only an aggregation of records is passed to the KSTREAM-MAPVALUES-NODE1 
node. KSTREAM-MAPVALUES-NODE1 then maps the aggregation of records into another 
object type. KSTREAM-SINK-NODE1 then attempts to produce this other object to 
topic T2.
 
The race condition occurs when the stream thread commits its offsets for topic 
T1 after it consumes some or all of the necessary records from topic T1 for an 
aggregation but before it gets the failure response back from the async produce 
kicked off by KSTREAM-SINK-NODE1.
 
We are running with a LogAndFailExceptionHandler, so when the stream thread 
tries to commit the next time it fails and the stream thread shuts itself down. 
The stream task is then reassigned to another stream thread, which reads the 
offsets previously committed by the original stream thread. That means the new 
stream thread's KSTREAM-SOURCE-NODE1 will never be able to consume the messages 
required for the aggregation and the KSTREAM-SINK-NODE1 will never end up 
producing the required records to topic T2. This is why it seems the 
at-least-once delivery guarantee is not met - KSTREAM-SINK-NODE1 never 
successfully processed records and the stream application continued on past it.

Note: we are running with StreamsConfig.RETRIES_CONFIG set to 10, which 
increases the likelihood of occurrence of the issue when all retries fail since 
it widens the window at which the async offset commit can occur before the 
produce record request is marked as failed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-461 - Improving replica fetcher behavior in case of partition failures

[Jun Rao]

Hi, Aishwarya,

Thanks for the KIP. Looks good to me. Just one minor comment. If a replica
is deleted on a broker (through a StopReplicaRequest) while it's in the
failed partition set, should we exclude that partition from the set and
the FailedPartitionsCount?

Jun

On Mon, May 6, 2019 at 1:21 PM Aishwarya Gune 
wrote:

> Hey All!
>
> I have created a KIP to improve the behavior of replica fetcher when
> partition failure occurs. Please do have a look at it and let me know what
> you think.
> KIP 461 -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-461+-+Improve+Replica+Fetcher+behavior+at+handling+partition+failure
>
> --
> Thank you,
> Aishwarya
>

[VOTE] 2.2.1 RC0

[Vahid Hashemian]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 2.2.1, which
includes many bug fixes for Apache Kafka 2.2.

Release notes for the 2.2.1 release:
https://home.apache.org/~vahid/kafka-2.2.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Monday, May 13, 6:00 pm PT.

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~vahid/kafka-2.2.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~vahid/kafka-2.2.1-rc0/javadoc/

* Tag to be voted upon (off 2.2 branch) is the 2.2.1 tag:
https://github.com/apache/kafka/releases/tag/2.2.1-rc0

* Documentation:
https://kafka.apache.org/22/documentation.html

* Protocol:
https://kafka.apache.org/22/protocol.html

* Successful Jenkins builds for the 2.2 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-2.2-jdk8/106/

Thanks,
--Vahid

Re: [VOTE] KIP-454: Expansion of the ConnectClusterState interface

[Rajini Sivaram]

Hi Chris,

Thanks for the KIP, +1 (binding)

Regards,

Rajini

On Wed, May 8, 2019 at 7:28 PM Manikumar  wrote:

> Hi Chris,
>
> +1 (binding). Thanks for the KIP.
>
> Thanks,
> Manikumar
>
> On Tue, May 7, 2019 at 2:55 AM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > Nice and useful KIP. Thanks Chris!
> >
> > +1 (non-binding)
> >
> > Konstantine
> >
> >
> > On Fri, May 3, 2019 at 5:14 PM Randall Hauch  wrote:
> >
> > > Nice job, Chris!
> > >
> > > +1 (binding)
> > >
> > > On Thu, May 2, 2019 at 8:16 PM Magesh Nandakumar  >
> > > wrote:
> > >
> > > > Thanks a lot for the work on this KIP Chris.
> > > >
> > > > +1(non-binding)
> > > >
> > > > On Thu, May 2, 2019, 4:56 PM Chris Egerton 
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to start a vote for KIP-454:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-454%3A+Expansion+of+the+ConnectClusterState+interface
> > > > >
> > > > > The discussion thread can be found at
> > > > > https://www.mail-archive.com/dev@kafka.apache.org/msg96911.html
> > > > >
> > > > > Thanks to Konstantine Karantasis and Magesh Nandakumar for their
> > > > thoughtful
> > > > > feedback!
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

[Rajini Sivaram]

Hi Chris,

Thanks for the explanation. Since there are workarounds and we are not
making it any worse, this should be fine.

Regards,

Rajini


On Wed, May 8, 2019 at 8:06 PM Chris Egerton  wrote:

> Hi Rajini,
>
> That was an initial concern of mine as well but I think we should be fine.
> Connect REST extensions are already capable of intercepting requests that
> contain new connector configurations, through POST calls to the /connectors
> endpoint and PUT calls to /connectors//config. The additional method
> you pointed out would extend that capability to include not just new
> connector configurations but existing connector configurations (by querying
> the Connect herder) as well.
>
> Neither should be a problem because, as of the merging of
> https://github.com/apache/kafka/pull/6129 (which addressed
> https://issues.apache.org/jira/browse/KAFKA-5117), both of those
> configurations can make use of the ConfigProvider mechanism in Connect to
> hide sensitive configs.
>
> If that mechanism is not used, connector configurations are available via
> the Connect REST API through GET calls to /connectors/ and
> /connectors//config, so it seems reasonable to enable REST extensions
> to view them as well.
>
> I hope this addresses your concerns; I'm happy to continue the discussion
> if any follow-up is necessary.
>
> Thanks for your thoughts!
>
> Cheers,
>
> Chris
>
> On Wed, May 8, 2019 at 11:19 AM Rajini Sivaram 
> wrote:
>
> > Hi Chris,
> >
> > Thanks for the KIP, looks good. I have just one question. Can `
> > ConnectClusterState#connectorConfig()` return any sensitive configs like
> > passwords?
> >
> > Thanks,
> >
> > Rajini
> >
> > On Wed, May 8, 2019 at 1:30 AM Chris Egerton 
> wrote:
> >
> > > Hi all,
> > >
> > > Now that  KAFKA-8304 (https://issues.apache.org/jira/browse/KAFKA-8304
> ),
> > > which was a blocker, has been addressed, I've published a PR for these
> > > changes: https://github.com/apache/kafka/pull/6584
> > >
> > > Thanks to everyone who's voted so far! If anyone else is interested,
> the
> > > voting thread can be found here:
> > > https://www.mail-archive.com/dev@kafka.apache.org/msg97458.html.
> Current
> > > status: +1 binding, +2 non-binding.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Tue, Apr 30, 2019 at 12:40 PM Chris Egerton 
> > > wrote:
> > >
> > > > Hi Konstantine,
> > > >
> > > > I've updated the KIP to add default method implementations to the
> list
> > of
> > > > rejected alternatives. Technically this makes the changes in the KIP
> > > > backwards incompatible, but I think I agree that for the majority of
> > > cases
> > > > where it would even be an issue a compile-time error is likely to be
> > more
> > > > beneficial than one at run time.
> > > >
> > > > Thanks for your thoughts and thanks for the LGTM!
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Mon, Apr 29, 2019 at 12:29 PM Konstantine Karantasis <
> > > > konstant...@confluent.io> wrote:
> > > >
> > > >> Hi Chris,
> > > >>
> > > >> Thanks for considering my suggestion regarding default
> implementations
> > > for
> > > >> the new methods.
> > > >> However, given that these implementations don't seem to have sane
> > > defaults
> > > >> and throw UnsupportedOperationException, I think we'll be better
> > without
> > > >> defaults.
> > > >> Seems that a compile time error is preferable here, for those who
> want
> > > to
> > > >> upgrade their implementations.
> > > >>
> > > >> Otherwise, the KIP LGTM.
> > > >>
> > > >> Konstantine
> > > >>
> > > >> On Thu, Apr 25, 2019 at 10:29 PM Magesh Nandakumar <
> > > mage...@confluent.io>
> > > >> wrote:
> > > >>
> > > >> > Thanks a lot, Chris. The KIP looks good to me.
> > > >> >
> > > >> > On Thu, Apr 25, 2019 at 7:35 PM Chris Egerton <
> chr...@confluent.io>
> > > >> wrote:
> > > >> >
> > > >> > > Hi Magesh,
> > > >> > >
> > > >> > > Sounds good; I've updated the KIP to make ConnectClusterDetails
> an
> > > >> > > interface. If we want to leave the door open to expand it in the
> > > >> future
> > > >> > it
> > > >> > > definitely makes sense to treat it similarly to how we're
> treating
> > > the
> > > >> > > ConnectClusterState interface now.
> > > >> > >
> > > >> > > Thanks,
> > > >> > >
> > > >> > > Chris
> > > >> > >
> > > >> > > On Thu, Apr 25, 2019 at 4:18 PM Magesh Nandakumar <
> > > >> mage...@confluent.io>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > HI Chrise,
> > > >> > > >
> > > >> > > > Overall it looks good to me. Just one last comment - I was
> > > >> wondering if
> > > >> > > > ConnectClusterDetail should be an interface just like
> > > >> > > ConnectClusterState.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Magesh
> > > >> > > >
> > > >> > > > On Thu, Apr 25, 2019 at 3:54 PM Chris Egerton <
> > > chr...@confluent.io>
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > Hi Magesh,
> > > >> > > > >
> > > >> > > > > Expanding the type we use to convey cluster metadata from
> > just a
> > > >> > Kafka
> > > >> > > > 

[jira] [Resolved] (KAFKA-8049) remove KafkaMbean when network close

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8049.

Resolution: Duplicate

> remove KafkaMbean when network close
> 
>
> Key: KAFKA-8049
> URL: https://issues.apache.org/jira/browse/KAFKA-8049
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, core
>Affects Versions: 0.10.2.0, 0.10.2.1, 0.10.2.2
>Reporter: limeng
>Priority: Critical
> Fix For: 2.2.2
>
>
> the  broker server will be oom when 
>  * a large number of clients frequently close and reconnect
>  * the clientId changes every time when reconnect，that gives rise to too much 
> kafkaMbean in broker
> the reason is that broker forget to remove kafkaMbean when detect connection 
> closes.
> h2.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

[Chris Egerton]

Hi Rajini,

That was an initial concern of mine as well but I think we should be fine.
Connect REST extensions are already capable of intercepting requests that
contain new connector configurations, through POST calls to the /connectors
endpoint and PUT calls to /connectors//config. The additional method
you pointed out would extend that capability to include not just new
connector configurations but existing connector configurations (by querying
the Connect herder) as well.

Neither should be a problem because, as of the merging of
https://github.com/apache/kafka/pull/6129 (which addressed
https://issues.apache.org/jira/browse/KAFKA-5117), both of those
configurations can make use of the ConfigProvider mechanism in Connect to
hide sensitive configs.

If that mechanism is not used, connector configurations are available via
the Connect REST API through GET calls to /connectors/ and
/connectors//config, so it seems reasonable to enable REST extensions
to view them as well.

I hope this addresses your concerns; I'm happy to continue the discussion
if any follow-up is necessary.

Thanks for your thoughts!

Cheers,

Chris

On Wed, May 8, 2019 at 11:19 AM Rajini Sivaram 
wrote:

> Hi Chris,
>
> Thanks for the KIP, looks good. I have just one question. Can `
> ConnectClusterState#connectorConfig()` return any sensitive configs like
> passwords?
>
> Thanks,
>
> Rajini
>
> On Wed, May 8, 2019 at 1:30 AM Chris Egerton  wrote:
>
> > Hi all,
> >
> > Now that  KAFKA-8304 (https://issues.apache.org/jira/browse/KAFKA-8304),
> > which was a blocker, has been addressed, I've published a PR for these
> > changes: https://github.com/apache/kafka/pull/6584
> >
> > Thanks to everyone who's voted so far! If anyone else is interested, the
> > voting thread can be found here:
> > https://www.mail-archive.com/dev@kafka.apache.org/msg97458.html. Current
> > status: +1 binding, +2 non-binding.
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Apr 30, 2019 at 12:40 PM Chris Egerton 
> > wrote:
> >
> > > Hi Konstantine,
> > >
> > > I've updated the KIP to add default method implementations to the list
> of
> > > rejected alternatives. Technically this makes the changes in the KIP
> > > backwards incompatible, but I think I agree that for the majority of
> > cases
> > > where it would even be an issue a compile-time error is likely to be
> more
> > > beneficial than one at run time.
> > >
> > > Thanks for your thoughts and thanks for the LGTM!
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Mon, Apr 29, 2019 at 12:29 PM Konstantine Karantasis <
> > > konstant...@confluent.io> wrote:
> > >
> > >> Hi Chris,
> > >>
> > >> Thanks for considering my suggestion regarding default implementations
> > for
> > >> the new methods.
> > >> However, given that these implementations don't seem to have sane
> > defaults
> > >> and throw UnsupportedOperationException, I think we'll be better
> without
> > >> defaults.
> > >> Seems that a compile time error is preferable here, for those who want
> > to
> > >> upgrade their implementations.
> > >>
> > >> Otherwise, the KIP LGTM.
> > >>
> > >> Konstantine
> > >>
> > >> On Thu, Apr 25, 2019 at 10:29 PM Magesh Nandakumar <
> > mage...@confluent.io>
> > >> wrote:
> > >>
> > >> > Thanks a lot, Chris. The KIP looks good to me.
> > >> >
> > >> > On Thu, Apr 25, 2019 at 7:35 PM Chris Egerton 
> > >> wrote:
> > >> >
> > >> > > Hi Magesh,
> > >> > >
> > >> > > Sounds good; I've updated the KIP to make ConnectClusterDetails an
> > >> > > interface. If we want to leave the door open to expand it in the
> > >> future
> > >> > it
> > >> > > definitely makes sense to treat it similarly to how we're treating
> > the
> > >> > > ConnectClusterState interface now.
> > >> > >
> > >> > > Thanks,
> > >> > >
> > >> > > Chris
> > >> > >
> > >> > > On Thu, Apr 25, 2019 at 4:18 PM Magesh Nandakumar <
> > >> mage...@confluent.io>
> > >> > > wrote:
> > >> > >
> > >> > > > HI Chrise,
> > >> > > >
> > >> > > > Overall it looks good to me. Just one last comment - I was
> > >> wondering if
> > >> > > > ConnectClusterDetail should be an interface just like
> > >> > > ConnectClusterState.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Magesh
> > >> > > >
> > >> > > > On Thu, Apr 25, 2019 at 3:54 PM Chris Egerton <
> > chr...@confluent.io>
> > >> > > wrote:
> > >> > > >
> > >> > > > > Hi Magesh,
> > >> > > > >
> > >> > > > > Expanding the type we use to convey cluster metadata from
> just a
> > >> > Kafka
> > >> > > > > cluster ID string to its own class seems like a good idea for
> > the
> > >> > sake
> > >> > > of
> > >> > > > > forwards compatibility, but I'm still not sure what the gains
> of
> > >> > > > including
> > >> > > > > the cluster group ID would be--it's a simple map lookup away
> in
> > >> the
> > >> > > REST
> > >> > > > > extension's configure(...) method. Including information on
> > >> whether
> > >> > the
> > >> > > > > cluster is distributed or standalone definitely seems
> > convenient;
> > >> as
> > >> > > far
> 

[jira] [Resolved] (KAFKA-8050) remove KafkaMbean when network close

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8050.

Resolution: Duplicate

> remove KafkaMbean when network close
> 
>
> Key: KAFKA-8050
> URL: https://issues.apache.org/jira/browse/KAFKA-8050
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, core
>Affects Versions: 0.10.2.0, 0.10.2.1, 0.10.2.2
>Reporter: limeng
>Priority: Critical
> Fix For: 2.2.2
>
>
> the  broker server will be oom when 
>  * a large number of clients frequently close and reconnect
>  * the clientId changes every time when reconnect，that gives rise to too much 
> kafkaMbean in broker
> the reason is that broker forget to remove kafkaMbean when detect connection 
> closes.
> h2.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-8048) remove KafkaMbean when network close

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8048.

Resolution: Duplicate

> remove KafkaMbean when network close
> 
>
> Key: KAFKA-8048
> URL: https://issues.apache.org/jira/browse/KAFKA-8048
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, core
>Affects Versions: 0.10.2.0, 0.10.2.1, 0.10.2.2
>Reporter: limeng
>Priority: Critical
> Fix For: 2.2.2
>
>
> the  broker server will be oom when 
>  * a large number of clients frequently close and reconnect
>  * the clientId changes every time when reconnect，that gives rise to too much 
> kafkaMbean in broker
> the reason is that broker forget to remove kafkaMbean when detect connection 
> closes.
> h2.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-8051) remove KafkaMbean when network close

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8051.

Resolution: Duplicate

> remove KafkaMbean when network close
> 
>
> Key: KAFKA-8051
> URL: https://issues.apache.org/jira/browse/KAFKA-8051
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, core
>Affects Versions: 0.10.2.0, 0.10.2.1, 0.10.2.2
>Reporter: limeng
>Priority: Critical
> Fix For: 2.2.2
>
>
> the  broker server will be oom when 
>  * a large number of clients frequently close and reconnect
>  * the clientId changes every time when reconnect，that gives rise to too much 
> kafkaMbean in broker
> the reason is that broker forget to remove kafkaMbean when detect connection 
> closes.
> h2.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8338) Improve consumer offset expiration logic to take subscription into account

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-8338:
---

 Summary: Improve consumer offset expiration logic to take 
subscription into account
 Key: KAFKA-8338
 URL: https://issues.apache.org/jira/browse/KAFKA-8338
 Project: Kafka
  Issue Type: Improvement
Reporter: Gwen Shapira


Currently, we expire consumer offsets for a group after the group is considered 
gone.

There is a case where the consumer group still exists, but is now subscribed to 
different topics. In that case, the offsets of the old topics will never expire 
and if lag is monitored, the monitors will show ever-growing lag on those 
topics. 

We need to improve the logic to expire the consumer offsets if the consumer 
group didn't subscribe to specific topics/partitions for enough time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

[John Roesler]

Hi Daniyar,

No worries about the procedural stuff. Prior experience with KIPs is
not required :)

I was just trying to help you propose this stuff in a way that the
others will find easy to review.

Thanks for updating the KIP. Thanks to the others for helping out with
the syntax.

Given these updates, I'm curious if anyone else has feedback about
this proposal. Personally, I think it sounds fine!

-John

On Wed, May 8, 2019 at 1:01 PM Development  wrote:
>
> Hey,
>
> That worked! I certainly lack Java generics knowledge. Thanks for the 
> snippet. I’ll update KIP again.
>
> Best,
> Daniyar Yeralin
>
> > On May 8, 2019, at 1:39 PM, Chris Egerton  wrote:
> >
> > Hi Daniyar,
> >
> > I think you may want to tweak your syntax a little:
> >
> > public static  Serde> List(Serde innerSerde) {
> >   return new ListSerde(innerSerde);
> > }
> >
> > Does that work?
> >
> > Cheers,
> >
> > Chris
> >
> > On Wed, May 8, 2019 at 10:29 AM Development  > > wrote:
> > Hi John,
> >
> > I updated JIRA and KIP.
> >
> > I didn’t know about the process, and created PR before I knew about KIPs :)
> >
> > As per static declaration, I don’t think Java allows that:
> >
> >
> > Best,
> > Daniyar Yeralin
> >
> >> On May 7, 2019, at 2:22 PM, John Roesler  >> > wrote:
> >>
> >> Thanks for that update. Do you mind making changes primarily on the
> >> KIP document ? 
> >> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List%3CT%3E+serialization+and+deserialization
> >>  
> >> )
> >>
> >> This is the design document that we have to agree on and vote for, the
> >> PR comes later. It can be nice to have an implementation to look at,
> >> but the KIP is the main artifact for this discussion.
> >>
> >> With this in mind, it will help get more reviewers to look at it if
> >> you can tidy up the KIP document so that it stands on its own. People
> >> shouldn't have to look at any other document to understand the
> >> motivation of the proposal, and they shouldn't have to look at a PR to
> >> see what the public API will look like. If it helps, you can take a
> >> look at some other recent KIPs.
> >>
> >> Given that the list serde needs an inner serde, I agree you can't have
> >> a zero-argument static factory method for it, but it seems you could
> >> still have a static method:
> >> `public static Serde> List(Serde innerSerde)`.
> >>
> >> Thoughts?
> >>
> >> On Tue, May 7, 2019 at 12:18 PM Development  >> > wrote:
> >>>
> >>> Absolutely agree. Already pushed another commit to remove comparator 
> >>> argument: https://github.com/apache/kafka/pull/6592 
> >>>  
> >>>  >>> >
> >>>
> >>> Thank you for your input John! I really appreciate it.
> >>>
> >>> What about this point I made:
> >>>
> >>> 1. Since type for List serde needs to be declared before hand, I could 
> >>> not create a static method for List Serde under 
> >>> org.apache.kafka.common.serialization.Serdes. I addressed it in the KIP:
> >>> P.S. Static method corresponding to ListSerde under 
> >>> org.apache.kafka.common.serialization.Serdes (something like static 
> >>> public Serde> List() {...} 
> >>> inorg.apache.kafka.common.serialization.Serdes) class cannot be added 
> >>> because type needs to be defined beforehand. That's why one needs to 
> >>> create List Serde in the following fashion:
> >>> new Serdes.ListSerde(Serdes.String(), 
> >>> Comparator.comparing(String::length));
> >>> (can possibly be simplified by declaring import static 
> >>> org.apache.kafka.common.serialization.Serdes.ListSerde)
> >>>
>  On May 7, 2019, at 11:50 AM, John Roesler   > wrote:
> 
>  Thanks for the reply Daniyar,
> 
>  That makes much more sense! I thought I must be missing something, but I
>  couldn't for the life of me figure it out.
> 
>  What do you think about just taking an argument, instead of for a
>  Comparator, for the Serde of the inner type? That way, the user can 
>  control
>  how exactly the inner data gets serialized, while also bounding the 
>  generic
>  parameter properly. As for the order, since the list is already in a
>  specific order, which the user themselves controls, it doesn't seem
>  strictly necessary to offer an option to sort the data during 
>  serialization.
> 
>  Thanks,
>  -John
> 
>  On Mon, May 6, 2019 at 8:47 PM Development   > wrote:
> 
> > Hi John,
> >
> > I’m really sorry for the confusion. I cloned that JIRA ticket from an 
> > old
> > one about introducing UUID Serde, and I guess was too hasty while 
> > editing
> > the copy 

Re: [VOTE] KIP-454: Expansion of the ConnectClusterState interface

[Manikumar]

Hi Chris,

+1 (binding). Thanks for the KIP.

Thanks,
Manikumar

On Tue, May 7, 2019 at 2:55 AM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Nice and useful KIP. Thanks Chris!
>
> +1 (non-binding)
>
> Konstantine
>
>
> On Fri, May 3, 2019 at 5:14 PM Randall Hauch  wrote:
>
> > Nice job, Chris!
> >
> > +1 (binding)
> >
> > On Thu, May 2, 2019 at 8:16 PM Magesh Nandakumar 
> > wrote:
> >
> > > Thanks a lot for the work on this KIP Chris.
> > >
> > > +1(non-binding)
> > >
> > > On Thu, May 2, 2019, 4:56 PM Chris Egerton 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start a vote for KIP-454:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-454%3A+Expansion+of+the+ConnectClusterState+interface
> > > >
> > > > The discussion thread can be found at
> > > > https://www.mail-archive.com/dev@kafka.apache.org/msg96911.html
> > > >
> > > > Thanks to Konstantine Karantasis and Magesh Nandakumar for their
> > > thoughtful
> > > > feedback!
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

[Rajini Sivaram]

Hi Chris,

Thanks for the KIP, looks good. I have just one question. Can `
ConnectClusterState#connectorConfig()` return any sensitive configs like
passwords?

Thanks,

Rajini

On Wed, May 8, 2019 at 1:30 AM Chris Egerton  wrote:

> Hi all,
>
> Now that  KAFKA-8304 (https://issues.apache.org/jira/browse/KAFKA-8304),
> which was a blocker, has been addressed, I've published a PR for these
> changes: https://github.com/apache/kafka/pull/6584
>
> Thanks to everyone who's voted so far! If anyone else is interested, the
> voting thread can be found here:
> https://www.mail-archive.com/dev@kafka.apache.org/msg97458.html. Current
> status: +1 binding, +2 non-binding.
>
> Cheers,
>
> Chris
>
> On Tue, Apr 30, 2019 at 12:40 PM Chris Egerton 
> wrote:
>
> > Hi Konstantine,
> >
> > I've updated the KIP to add default method implementations to the list of
> > rejected alternatives. Technically this makes the changes in the KIP
> > backwards incompatible, but I think I agree that for the majority of
> cases
> > where it would even be an issue a compile-time error is likely to be more
> > beneficial than one at run time.
> >
> > Thanks for your thoughts and thanks for the LGTM!
> >
> > Cheers,
> >
> > Chris
> >
> > On Mon, Apr 29, 2019 at 12:29 PM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> >> Hi Chris,
> >>
> >> Thanks for considering my suggestion regarding default implementations
> for
> >> the new methods.
> >> However, given that these implementations don't seem to have sane
> defaults
> >> and throw UnsupportedOperationException, I think we'll be better without
> >> defaults.
> >> Seems that a compile time error is preferable here, for those who want
> to
> >> upgrade their implementations.
> >>
> >> Otherwise, the KIP LGTM.
> >>
> >> Konstantine
> >>
> >> On Thu, Apr 25, 2019 at 10:29 PM Magesh Nandakumar <
> mage...@confluent.io>
> >> wrote:
> >>
> >> > Thanks a lot, Chris. The KIP looks good to me.
> >> >
> >> > On Thu, Apr 25, 2019 at 7:35 PM Chris Egerton 
> >> wrote:
> >> >
> >> > > Hi Magesh,
> >> > >
> >> > > Sounds good; I've updated the KIP to make ConnectClusterDetails an
> >> > > interface. If we want to leave the door open to expand it in the
> >> future
> >> > it
> >> > > definitely makes sense to treat it similarly to how we're treating
> the
> >> > > ConnectClusterState interface now.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Chris
> >> > >
> >> > > On Thu, Apr 25, 2019 at 4:18 PM Magesh Nandakumar <
> >> mage...@confluent.io>
> >> > > wrote:
> >> > >
> >> > > > HI Chrise,
> >> > > >
> >> > > > Overall it looks good to me. Just one last comment - I was
> >> wondering if
> >> > > > ConnectClusterDetail should be an interface just like
> >> > > ConnectClusterState.
> >> > > >
> >> > > > Thanks,
> >> > > > Magesh
> >> > > >
> >> > > > On Thu, Apr 25, 2019 at 3:54 PM Chris Egerton <
> chr...@confluent.io>
> >> > > wrote:
> >> > > >
> >> > > > > Hi Magesh,
> >> > > > >
> >> > > > > Expanding the type we use to convey cluster metadata from just a
> >> > Kafka
> >> > > > > cluster ID string to its own class seems like a good idea for
> the
> >> > sake
> >> > > of
> >> > > > > forwards compatibility, but I'm still not sure what the gains of
> >> > > > including
> >> > > > > the cluster group ID would be--it's a simple map lookup away in
> >> the
> >> > > REST
> >> > > > > extension's configure(...) method. Including information on
> >> whether
> >> > the
> >> > > > > cluster is distributed or standalone definitely seems
> convenient;
> >> as
> >> > > far
> >> > > > as
> >> > > > > I can tell there's no easy way to do that from within a REST
> >> > extension
> >> > > at
> >> > > > > the moment, and relying on something like the presence of a
> >> group.id
> >> > > > > property to identify a distributed cluster could result in false
> >> > > > positives.
> >> > > > > However, is there a use case for it? If not, we can note that
> as a
> >> > > > possible
> >> > > > > addition to the ConnectClusterDetails class for later but leave
> it
> >> > out
> >> > > > for
> >> > > > > now.
> >> > > > >
> >> > > > > I've updated the KIP to include the new ConnectClusterDetails
> >> class
> >> > but
> >> > > > > left out the cluster type information for now; let me know what
> >> you
> >> > > > think.
> >> > > > >
> >> > > > > Thanks again for your thoughts!
> >> > > > >
> >> > > > > Cheers,
> >> > > > >
> >> > > > > Chris
> >> > > > >
> >> > > > > On Wed, Apr 24, 2019 at 4:49 PM Magesh Nandakumar <
> >> > > mage...@confluent.io>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi Chris,
> >> > > > > >
> >> > > > > > Instead of calling it ConnectClusterId, perhaps call it
> >> > > > > > ConnectClusterDetails which can include things like groupid,
> >> > > underlying
> >> > > > > > kafkaclusterId, standalone or distributed, etc. This will help
> >> > expose
> >> > > > any
> >> > > > > > cluster related information in the future.
> >> > > > > > Let me know if that would work?
> >> > > > > >
> >> > 

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

[Development]

Hey,

That worked! I certainly lack Java generics knowledge. Thanks for the snippet. 
I’ll update KIP again.

Best,
Daniyar Yeralin

> On May 8, 2019, at 1:39 PM, Chris Egerton  wrote:
> 
> Hi Daniyar,
> 
> I think you may want to tweak your syntax a little:
> 
> public static  Serde> List(Serde innerSerde) {
>   return new ListSerde(innerSerde);
> }
> 
> Does that work?
> 
> Cheers,
> 
> Chris
> 
> On Wed, May 8, 2019 at 10:29 AM Development  > wrote:
> Hi John,
> 
> I updated JIRA and KIP.
> 
> I didn’t know about the process, and created PR before I knew about KIPs :) 
> 
> As per static declaration, I don’t think Java allows that:
> 
> 
> Best,
> Daniyar Yeralin
> 
>> On May 7, 2019, at 2:22 PM, John Roesler > > wrote:
>> 
>> Thanks for that update. Do you mind making changes primarily on the
>> KIP document ? 
>> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List%3CT%3E+serialization+and+deserialization
>>  
>> )
>> 
>> This is the design document that we have to agree on and vote for, the
>> PR comes later. It can be nice to have an implementation to look at,
>> but the KIP is the main artifact for this discussion.
>> 
>> With this in mind, it will help get more reviewers to look at it if
>> you can tidy up the KIP document so that it stands on its own. People
>> shouldn't have to look at any other document to understand the
>> motivation of the proposal, and they shouldn't have to look at a PR to
>> see what the public API will look like. If it helps, you can take a
>> look at some other recent KIPs.
>> 
>> Given that the list serde needs an inner serde, I agree you can't have
>> a zero-argument static factory method for it, but it seems you could
>> still have a static method:
>> `public static Serde> List(Serde innerSerde)`.
>> 
>> Thoughts?
>> 
>> On Tue, May 7, 2019 at 12:18 PM Development > > wrote:
>>> 
>>> Absolutely agree. Already pushed another commit to remove comparator 
>>> argument: https://github.com/apache/kafka/pull/6592 
>>>  
>>> >> >
>>> 
>>> Thank you for your input John! I really appreciate it.
>>> 
>>> What about this point I made:
>>> 
>>> 1. Since type for List serde needs to be declared before hand, I could not 
>>> create a static method for List Serde under 
>>> org.apache.kafka.common.serialization.Serdes. I addressed it in the KIP:
>>> P.S. Static method corresponding to ListSerde under 
>>> org.apache.kafka.common.serialization.Serdes (something like static public 
>>> Serde> List() {...} inorg.apache.kafka.common.serialization.Serdes) 
>>> class cannot be added because type needs to be defined beforehand. That's 
>>> why one needs to create List Serde in the following fashion:
>>> new Serdes.ListSerde(Serdes.String(), 
>>> Comparator.comparing(String::length));
>>> (can possibly be simplified by declaring import static 
>>> org.apache.kafka.common.serialization.Serdes.ListSerde)
>>> 
 On May 7, 2019, at 11:50 AM, John Roesler >>> > wrote:
 
 Thanks for the reply Daniyar,
 
 That makes much more sense! I thought I must be missing something, but I
 couldn't for the life of me figure it out.
 
 What do you think about just taking an argument, instead of for a
 Comparator, for the Serde of the inner type? That way, the user can control
 how exactly the inner data gets serialized, while also bounding the generic
 parameter properly. As for the order, since the list is already in a
 specific order, which the user themselves controls, it doesn't seem
 strictly necessary to offer an option to sort the data during 
 serialization.
 
 Thanks,
 -John
 
 On Mon, May 6, 2019 at 8:47 PM Development >>> > wrote:
 
> Hi John,
> 
> I’m really sorry for the confusion. I cloned that JIRA ticket from an old
> one about introducing UUID Serde, and I guess was too hasty while editing
> the copy to notice the mistake. Just edited the ticket. Sorry for any
> inconvenience .
> 
> As per comparator, I agree. Let’s make user be responsible for
> implementing comparable interface. I was just thinking to make the serde a
> little more flexible (i.e. let user decide in which order records is going
> to be inserted into a change log topic).
> 
> Thank you!
> 
> Best,
> Daniyar Yeralin
> 
> 
>> On May 6, 2019, at 5:37 PM, John Roesler > > wrote:
>> 
>> Hi Daniyar,
>> 
>> Thanks for the proposal!
>> 
>> If I understand the point about the comparator, is it just to capture the
>> 

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

[Chris Egerton]

Hi Daniyar,

I think you may want to tweak your syntax a little:

public static  Serde> List(Serde innerSerde) {
  return new ListSerde(innerSerde);
}

Does that work?

Cheers,

Chris

On Wed, May 8, 2019 at 10:29 AM Development  wrote:

> Hi John,
>
> I updated JIRA and KIP.
>
> I didn’t know about the process, and created PR before I knew about KIPs
> :)
>
> As per static declaration, I don’t think Java allows that:
>
> Best,
> Daniyar Yeralin
>
> On May 7, 2019, at 2:22 PM, John Roesler  wrote:
>
> Thanks for that update. Do you mind making changes primarily on the
> KIP document ? (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List%3CT%3E+serialization+and+deserialization
> )
>
> This is the design document that we have to agree on and vote for, the
> PR comes later. It can be nice to have an implementation to look at,
> but the KIP is the main artifact for this discussion.
>
> With this in mind, it will help get more reviewers to look at it if
> you can tidy up the KIP document so that it stands on its own. People
> shouldn't have to look at any other document to understand the
> motivation of the proposal, and they shouldn't have to look at a PR to
> see what the public API will look like. If it helps, you can take a
> look at some other recent KIPs.
>
> Given that the list serde needs an inner serde, I agree you can't have
> a zero-argument static factory method for it, but it seems you could
> still have a static method:
> `public static Serde> List(Serde innerSerde)`.
>
> Thoughts?
>
> On Tue, May 7, 2019 at 12:18 PM Development  wrote:
>
>
> Absolutely agree. Already pushed another commit to remove comparator
> argument: https://github.com/apache/kafka/pull/6592 <
> https://github.com/apache/kafka/pull/6592>
>
> Thank you for your input John! I really appreciate it.
>
> What about this point I made:
>
> 1. Since type for List serde needs to be declared before hand, I could not
> create a static method for List Serde under
> org.apache.kafka.common.serialization.Serdes. I addressed it in the KIP:
> P.S. Static method corresponding to ListSerde under
> org.apache.kafka.common.serialization.Serdes (something like static public
> Serde> List() {...} inorg.apache.kafka.common.serialization.Serdes)
> class cannot be added because type needs to be defined beforehand. That's
> why one needs to create List Serde in the following fashion:
> new Serdes.ListSerde(Serdes.String(),
> Comparator.comparing(String::length));
> (can possibly be simplified by declaring import static
> org.apache.kafka.common.serialization.Serdes.ListSerde)
>
> On May 7, 2019, at 11:50 AM, John Roesler  wrote:
>
> Thanks for the reply Daniyar,
>
> That makes much more sense! I thought I must be missing something, but I
> couldn't for the life of me figure it out.
>
> What do you think about just taking an argument, instead of for a
> Comparator, for the Serde of the inner type? That way, the user can control
> how exactly the inner data gets serialized, while also bounding the generic
> parameter properly. As for the order, since the list is already in a
> specific order, which the user themselves controls, it doesn't seem
> strictly necessary to offer an option to sort the data during
> serialization.
>
> Thanks,
> -John
>
> On Mon, May 6, 2019 at 8:47 PM Development  wrote:
>
> Hi John,
>
> I’m really sorry for the confusion. I cloned that JIRA ticket from an old
> one about introducing UUID Serde, and I guess was too hasty while editing
> the copy to notice the mistake. Just edited the ticket. Sorry for any
> inconvenience .
>
> As per comparator, I agree. Let’s make user be responsible for
> implementing comparable interface. I was just thinking to make the serde a
> little more flexible (i.e. let user decide in which order records is going
> to be inserted into a change log topic).
>
> Thank you!
>
> Best,
> Daniyar Yeralin
>
>
> On May 6, 2019, at 5:37 PM, John Roesler  wrote:
>
> Hi Daniyar,
>
> Thanks for the proposal!
>
> If I understand the point about the comparator, is it just to capture the
> generic type parameter? If so, then anything that implements a known
> interface would work just as well, right? I've been considering adding
> something like the Jackson TypeReference (or similar classes in many
>
> other
>
> projects). Would this be a good time to do it?
>
> Note that it's not necessary to actually require that the captured type
>
> is
>
> Comparable (as this proposal currently does), it's just a way to make
>
> sure
>
> there is some method that makes use of the generic type parameter, to
>
> force
>
> the compiler to capture the type.
>
> Just to make sure I understand the motivation... You expressed a desire
>
> to
>
> be able to serialize UUIDs, which I didn't follow, since there is a
> built-in UUID serde: org.apache.kafka.common.serialization.Serdes#UUID,
>
> and
>
> also, a UUID isn't a List. Did you mean that you need to use *lists of*
> UUIDs?
>
> Thanks,
> -John
>
> On 

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

[Sophie Blee-Goldman]

Hi Daniyar,

Thanks for the KIP! I had to write my own List serde for testing a while
back and this definitely would have saved me some time. Regarding the
static declaration, I believe you're missing a  between "public static"
and the return type "Serde>"  -- Java should allow this

Cheers,
Sophie

On Wed, May 8, 2019 at 10:29 AM Development  wrote:

> Hi John,
>
> I updated JIRA and KIP.
>
> I didn’t know about the process, and created PR before I knew about KIPs
> :)
>
> As per static declaration, I don’t think Java allows that:
>
> Best,
> Daniyar Yeralin
>
> On May 7, 2019, at 2:22 PM, John Roesler  wrote:
>
> Thanks for that update. Do you mind making changes primarily on the
> KIP document ? (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List%3CT%3E+serialization+and+deserialization
> )
>
> This is the design document that we have to agree on and vote for, the
> PR comes later. It can be nice to have an implementation to look at,
> but the KIP is the main artifact for this discussion.
>
> With this in mind, it will help get more reviewers to look at it if
> you can tidy up the KIP document so that it stands on its own. People
> shouldn't have to look at any other document to understand the
> motivation of the proposal, and they shouldn't have to look at a PR to
> see what the public API will look like. If it helps, you can take a
> look at some other recent KIPs.
>
> Given that the list serde needs an inner serde, I agree you can't have
> a zero-argument static factory method for it, but it seems you could
> still have a static method:
> `public static Serde> List(Serde innerSerde)`.
>
> Thoughts?
>
> On Tue, May 7, 2019 at 12:18 PM Development  wrote:
>
>
> Absolutely agree. Already pushed another commit to remove comparator
> argument: https://github.com/apache/kafka/pull/6592 <
> https://github.com/apache/kafka/pull/6592>
>
> Thank you for your input John! I really appreciate it.
>
> What about this point I made:
>
> 1. Since type for List serde needs to be declared before hand, I could not
> create a static method for List Serde under
> org.apache.kafka.common.serialization.Serdes. I addressed it in the KIP:
> P.S. Static method corresponding to ListSerde under
> org.apache.kafka.common.serialization.Serdes (something like static public
> Serde> List() {...} inorg.apache.kafka.common.serialization.Serdes)
> class cannot be added because type needs to be defined beforehand. That's
> why one needs to create List Serde in the following fashion:
> new Serdes.ListSerde(Serdes.String(),
> Comparator.comparing(String::length));
> (can possibly be simplified by declaring import static
> org.apache.kafka.common.serialization.Serdes.ListSerde)
>
> On May 7, 2019, at 11:50 AM, John Roesler  wrote:
>
> Thanks for the reply Daniyar,
>
> That makes much more sense! I thought I must be missing something, but I
> couldn't for the life of me figure it out.
>
> What do you think about just taking an argument, instead of for a
> Comparator, for the Serde of the inner type? That way, the user can control
> how exactly the inner data gets serialized, while also bounding the generic
> parameter properly. As for the order, since the list is already in a
> specific order, which the user themselves controls, it doesn't seem
> strictly necessary to offer an option to sort the data during
> serialization.
>
> Thanks,
> -John
>
> On Mon, May 6, 2019 at 8:47 PM Development  wrote:
>
> Hi John,
>
> I’m really sorry for the confusion. I cloned that JIRA ticket from an old
> one about introducing UUID Serde, and I guess was too hasty while editing
> the copy to notice the mistake. Just edited the ticket. Sorry for any
> inconvenience .
>
> As per comparator, I agree. Let’s make user be responsible for
> implementing comparable interface. I was just thinking to make the serde a
> little more flexible (i.e. let user decide in which order records is going
> to be inserted into a change log topic).
>
> Thank you!
>
> Best,
> Daniyar Yeralin
>
>
> On May 6, 2019, at 5:37 PM, John Roesler  wrote:
>
> Hi Daniyar,
>
> Thanks for the proposal!
>
> If I understand the point about the comparator, is it just to capture the
> generic type parameter? If so, then anything that implements a known
> interface would work just as well, right? I've been considering adding
> something like the Jackson TypeReference (or similar classes in many
>
> other
>
> projects). Would this be a good time to do it?
>
> Note that it's not necessary to actually require that the captured type
>
> is
>
> Comparable (as this proposal currently does), it's just a way to make
>
> sure
>
> there is some method that makes use of the generic type parameter, to
>
> force
>
> the compiler to capture the type.
>
> Just to make sure I understand the motivation... You expressed a desire
>
> to
>
> be able to serialize UUIDs, which I didn't follow, since there is a
> built-in UUID serde: org.apache.kafka.common.serialization.Serdes#UUID,
>
> 

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

[Development]

Hi John,

I updated JIRA and KIP.

I didn’t know about the process, and created PR before I knew about KIPs :) 

As per static declaration, I don’t think Java allows that:


Best,
Daniyar Yeralin

> On May 7, 2019, at 2:22 PM, John Roesler  wrote:
> 
> Thanks for that update. Do you mind making changes primarily on the
> KIP document ? 
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List%3CT%3E+serialization+and+deserialization)
> 
> This is the design document that we have to agree on and vote for, the
> PR comes later. It can be nice to have an implementation to look at,
> but the KIP is the main artifact for this discussion.
> 
> With this in mind, it will help get more reviewers to look at it if
> you can tidy up the KIP document so that it stands on its own. People
> shouldn't have to look at any other document to understand the
> motivation of the proposal, and they shouldn't have to look at a PR to
> see what the public API will look like. If it helps, you can take a
> look at some other recent KIPs.
> 
> Given that the list serde needs an inner serde, I agree you can't have
> a zero-argument static factory method for it, but it seems you could
> still have a static method:
> `public static Serde> List(Serde innerSerde)`.
> 
> Thoughts?
> 
> On Tue, May 7, 2019 at 12:18 PM Development  wrote:
>> 
>> Absolutely agree. Already pushed another commit to remove comparator 
>> argument: https://github.com/apache/kafka/pull/6592 
>> 
>> 
>> Thank you for your input John! I really appreciate it.
>> 
>> What about this point I made:
>> 
>> 1. Since type for List serde needs to be declared before hand, I could not 
>> create a static method for List Serde under 
>> org.apache.kafka.common.serialization.Serdes. I addressed it in the KIP:
>> P.S. Static method corresponding to ListSerde under 
>> org.apache.kafka.common.serialization.Serdes (something like static public 
>> Serde> List() {...} inorg.apache.kafka.common.serialization.Serdes) 
>> class cannot be added because type needs to be defined beforehand. That's 
>> why one needs to create List Serde in the following fashion:
>> new Serdes.ListSerde(Serdes.String(), 
>> Comparator.comparing(String::length));
>> (can possibly be simplified by declaring import static 
>> org.apache.kafka.common.serialization.Serdes.ListSerde)
>> 
>>> On May 7, 2019, at 11:50 AM, John Roesler  wrote:
>>> 
>>> Thanks for the reply Daniyar,
>>> 
>>> That makes much more sense! I thought I must be missing something, but I
>>> couldn't for the life of me figure it out.
>>> 
>>> What do you think about just taking an argument, instead of for a
>>> Comparator, for the Serde of the inner type? That way, the user can control
>>> how exactly the inner data gets serialized, while also bounding the generic
>>> parameter properly. As for the order, since the list is already in a
>>> specific order, which the user themselves controls, it doesn't seem
>>> strictly necessary to offer an option to sort the data during serialization.
>>> 
>>> Thanks,
>>> -John
>>> 
>>> On Mon, May 6, 2019 at 8:47 PM Development  wrote:
>>> 
 Hi John,
 
 I’m really sorry for the confusion. I cloned that JIRA ticket from an old
 one about introducing UUID Serde, and I guess was too hasty while editing
 the copy to notice the mistake. Just edited the ticket. Sorry for any
 inconvenience .
 
 As per comparator, I agree. Let’s make user be responsible for
 implementing comparable interface. I was just thinking to make the serde a
 little more flexible (i.e. let user decide in which order records is going
 to be inserted into a change log topic).
 
 Thank you!
 
 Best,
 Daniyar Yeralin
 
 
> On May 6, 2019, at 5:37 PM, John Roesler  wrote:
> 
> Hi Daniyar,
> 
> Thanks for the proposal!
> 
> If I understand the point about the comparator, is it just to capture the
> generic type parameter? If so, then anything that implements a known
> interface would work just as well, right? I've been considering adding
> something like the Jackson TypeReference (or similar classes in many
 other
> projects). Would this be a good time to do it?
> 
> Note that it's not necessary to actually require that the captured type
 is
> Comparable (as this proposal currently does), it's just a way to make
 sure
> there is some method that makes use of the generic type parameter, to
 force
> the compiler to capture the type.
> 
> Just to make sure I understand the motivation... You expressed a desire
 to
> be able to serialize UUIDs, which I didn't follow, since there is a
> built-in UUID serde: org.apache.kafka.common.serialization.Serdes#UUID,
 and
> also, a UUID isn't a List. Did you mean that you need to use *lists of*
> UUIDs?
> 
> Thanks,
> -John
> 
> On Mon, 

Build failed in Jenkins: kafka-trunk-jdk11 #499

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-7320; Add consumer configuration to disable auto topic creation

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H33 (ubuntu xenial) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision e6cff21fd8c5add0eb7e55417a91f0530a7d3a32 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e6cff21fd8c5add0eb7e55417a91f0530a7d3a32
Commit message: "KAFKA-7320; Add consumer configuration to disable auto topic 
creation [KIP-361] (#5542)"
 > git rev-list --no-walk 83823aedf2e2d176004402152b69bd67f97d8e12 # timeout=10
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
[kafka-trunk-jdk11] $ /bin/bash -xe /tmp/jenkins5142204601136240683.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.10.2/bin/gradle
/tmp/jenkins5142204601136240683.sh: line 4: 
/home/jenkins/tools/gradle/4.10.2/bin/gradle: No such file or directory
Build step 'Execute shell' marked build as failure
Recording test results
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Not sending mail to unregistered user nore...@github.com
Not sending mail to unregistered user csh...@gmail.com
Not sending mail to unregistered user wangg...@gmail.com

Re: [DISCUSS] KIP-461 - Improving replica fetcher behavior in case of partition failures

[Jason Gustafson]

Hey Aishwarya,

Thanks for the KIP. I'd suggest we move to a vote since this is a
straightforward improvement with a large impact.

-Jason

On Tue, May 7, 2019 at 3:02 PM Aishwarya Gune 
wrote:

> Hi Colin!
>
> Whenever the thread has all of its partitions marked as failed (i.e. thread
> is idle), the thread would be shut down.
> The errors that are not per-partition would probably retry or behave just
> as before.
>
>
> On Tue, May 7, 2019 at 9:57 AM Colin McCabe  wrote:
>
> > Hi Aishwarya,
> >
> > This looks like a great improvement!
> >
> > Will a fetcher thread exit if all of its partitions have been marked
> > failed?  Or will it continue to run?
> >
> > After this KIP is adopted, are there any remaining situations where we
> > would exit a fetcher thread?  I guess some errors are not per-partition,
> > like authentication exceptions.  How will those behave?
> >
> > best,
> > Colin
> >
> > On Mon, May 6, 2019, at 13:21, Aishwarya Gune wrote:
> > > Hey All!
> > >
> > > I have created a KIP to improve the behavior of replica fetcher when
> > > partition failure occurs. Please do have a look at it and let me know
> > > what
> > > you think.
> > > KIP 461 -
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-461+-+Improve+Replica+Fetcher+behavior+at+handling+partition+failure
> > >
> > > --
> > > Thank you,
> > > Aishwarya
> > >
> >
>
>
> --
> Thank you,
> Aishwarya
>

Re: [DISCUSS] KIP-461 - Improving replica fetcher behavior in case of partition failures

[Colin McCabe]

Thanks-- that makes sense.

cheers,
Colin

On Tue, May 7, 2019, at 15:02, Aishwarya Gune wrote:
> Hi Colin!
> 
> Whenever the thread has all of its partitions marked as failed (i.e. thread
> is idle), the thread would be shut down.
> The errors that are not per-partition would probably retry or behave just
> as before.
> 
> 
> On Tue, May 7, 2019 at 9:57 AM Colin McCabe  wrote:
> 
> > Hi Aishwarya,
> >
> > This looks like a great improvement!
> >
> > Will a fetcher thread exit if all of its partitions have been marked
> > failed?  Or will it continue to run?
> >
> > After this KIP is adopted, are there any remaining situations where we
> > would exit a fetcher thread?  I guess some errors are not per-partition,
> > like authentication exceptions.  How will those behave?
> >
> > best,
> > Colin
> >
> > On Mon, May 6, 2019, at 13:21, Aishwarya Gune wrote:
> > > Hey All!
> > >
> > > I have created a KIP to improve the behavior of replica fetcher when
> > > partition failure occurs. Please do have a look at it and let me know
> > > what
> > > you think.
> > > KIP 461 -
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-461+-+Improve+Replica+Fetcher+behavior+at+handling+partition+failure
> > >
> > > --
> > > Thank you,
> > > Aishwarya
> > >
> >
> 
> 
> -- 
> Thank you,
> Aishwarya
>

[jira] [Resolved] (KAFKA-7320) Provide ability to disable auto topic creation in KafkaConsumer

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-7320.

   Resolution: Fixed
Fix Version/s: 2.3.0

> Provide ability to disable auto topic creation in KafkaConsumer
> ---
>
> Key: KAFKA-7320
> URL: https://issues.apache.org/jira/browse/KAFKA-7320
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Dhruvil Shah
>Assignee: Dhruvil Shah
>Priority: Major
> Fix For: 2.3.0
>
>
> Consumers should have a configuration to control whether subscribing to 
> non-existent topics should automatically create the topic or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk11 #498

[Apache Jenkins Server]

See 


Changes:

[bbejeck] BUGFIX: Add missing recording of close of stand-by task (#6663)

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H33 (ubuntu xenial) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 83823aedf2e2d176004402152b69bd67f97d8e12 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 83823aedf2e2d176004402152b69bd67f97d8e12
Commit message: "BUGFIX: Add missing recording of close of stand-by task 
(#6663)"
 > git rev-list --no-walk 5144660040839cee6e213b5146c1ae44340eddb0 # timeout=10
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
[kafka-trunk-jdk11] $ /bin/bash -xe /tmp/jenkins4649178286356794986.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.10.2/bin/gradle
/tmp/jenkins4649178286356794986.sh: line 4: 
/home/jenkins/tools/gradle/4.10.2/bin/gradle: No such file or directory
Build step 'Execute shell' marked build as failure
Recording test results
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Not sending mail to unregistered user nore...@github.com
Not sending mail to unregistered user csh...@gmail.com
Not sending mail to unregistered user wangg...@gmail.com

Re: apply to be a contributor

[Guozhang Wang]

Hello Wei,

I've added you to the contributor list. Cheers.


Guozhang

On Wed, May 8, 2019 at 8:57 AM 魏巍  wrote:

> |
> Hi,
>
> I want to contribute to Apache Kafka.
> Would you please give methe contributor permission?
> My JIRA ID is doubleWei.
> |



-- 
-- Guozhang

I want to be apache kafka contributer

[魏巍]

I want to be apache kafka contributer

apply to be a contributor

[魏巍]

|
Hi,

I want to contribute to Apache Kafka.
Would you please give methe contributor permission?
My JIRA ID is doubleWei.
|

Re: Apply for JIRA contributor permission

[Bill Bejeck]

Done.  Thanks for contributing!

-Bill

On Wed, May 8, 2019 at 11:15 AM Kengo Seki  wrote:

> Hi,
>
> I just filed KAFKA-8337 and would like to assign it to myself.
> Would you add me to the contributors list?
> My username is "sekikn".
>
> Thanks in advance!
>
> Kengo Seki 
>

Apply for JIRA contributor permission

[Kengo Seki]

Hi,

I just filed KAFKA-8337 and would like to assign it to myself.
Would you add me to the contributors list?
My username is "sekikn".

Thanks in advance!

Kengo Seki 

[jira] [Created] (KAFKA-8337) Fix tests/setup.cfg to work with pytest 4.x

[Kengo Seki (JIRA)]

Kengo Seki created KAFKA-8337:
-

 Summary: Fix tests/setup.cfg to work with pytest 4.x
 Key: KAFKA-8337
 URL: https://issues.apache.org/jira/browse/KAFKA-8337
 Project: Kafka
  Issue Type: Bug
  Components: system tests, unit tests
Reporter: Kengo Seki


In accordance with tests/README.md, I ran {{`python setup.py test`}} in the 
{{tests}} directory to execute unit tests for the system tests, but got the 
following error:

{code}
$ python setup.py test
running test

(snip)

Using /home/sekikn/repo/kafka/tests/.eggs/docutils-0.14-py2.7.egg
Searching for pytest
Best match: pytest 4.4.1
Processing pytest-4.4.1-py2.7.egg

(snip)

builtins.Failed: [pytest] section in setup.cfg files is no longer supported, 
change to [tool:pytest] instead.

{code}

This is because [\[pytest\] section in setup.cfg has been removed in pytest 
4.0|https://docs.pytest.org/en/4.1.0/deprecations.html#pytest-section-in-setup-cfg-files].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-456: Helper classes to make it simpler to write test logic with TopologyTestDriver

[John Roesler]

Hi Jukka, thanks for the reply!

I think this is a good summary (the discussion was getting a little
unwieldy. I'll reply inline.

Also, thanks for clarify about your library vs. this KIP. That makes
perfect sense to me.
>
> 1. Add JavaDoc for KIP
>
> Is there a good example of KIP where Javadoc is included, so I can follow?
> I create this KIP based on this as an example::
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams
>
>
> Now added some comments to KIP page to clarify timestamp handling, but I
> did not want to add full JavaDoc of each methods.
> Is that enough?

That's fine. I was just trying to make the review process more
efficient for other reviewers (which makes getting your KIP accepted
more efficient). I reviewed a few recent KIPs, and, indeed, I see that
javadocs are not actually as common as I thought.

> 2. TTD usability changes and swapping ConsumerRecord and ProducerRecord in
> APIs
>
> To my point of view only:
> - changing readRecord to return ConsumerRecord would cause we cannot use
> OutputVerifier

Yes, we'd likely have to provide new methods in OutputVerifier to work
with ConsumerRecord. If you buy into the plan of deprecating most of
the current-style interactions, this wouldn't be that confusing, since
all the ProducerRecord verifications would be deprecated, and only the
ConsumerRecord verifications would remain "live".

> - changing pipeInput to take in ProducerRecord, but not providing easy way
> to contruct those like ConsumerRecordFactory

I didn't follow this as well. The ConsumerRecordFactory is there
because it's a pain to construct ConsumerRecords. Conversely,
ProducerRecord has many convenience constructors, so we wouldn't need
a factory at all. This is a net win for users, since there's less
surface area for them to deal with. Under my proposal, we'd deprecate
the whole ConsumerRecordFactory.

Note that there's an "idea parity check" here: ConsumerRecords are
hard to construct because developers aren't meant to ever construct
them. They are meant to construct ProducerRecords, which is why it's
made easy. TTD has inverted the relationships of these classes, which
is why the ConsumerRecordFactory is necessary, but if we correct it,
and return to a "normal" interaction with the Client library, then we
don't need special support classes.

> - if initializing ConsumerRecord to/from  ProducerRecord  in these classes
> field by field contructor, there are risk new fields are not added to this
> classes if there are changes in ProducerRecord or ConsumerRecord

This risk seems pretty low, to be honest. We will have tests that
exercise this testing framework, so if anyone changes ProducerRecord
or ConsumerRecord, our tests will break. Since both libraries are
build together, the problem would be fixed before the change is ever
merged to trunk.

> I would propose a separate KIP for these and probably other enhanchements:
> -superclass or common interface for ConsumerRecord and ProducerRecord
> -contructors to ConsumerRecord and ProducerRecord to initialize with this
> superclass
> -modify OutputVerifier to work with both ConsumerRecord and ProducerRecord
> -create new RecordFactory to replace ConsumerRecordFactory

I understand your motivation to control the scope of this change, but
I actually think that it's better for user-facing design changes to
occur in fewer, bigger, chunks, rather than many small changes. People
will get fatigued if multiple releases in a row change the
test-support library from under their feet. Better to do it in one
shot.

Plus, this is a design discussion. We need to include the whole scope
of the system in the design, or we may realize in Phase 3 that there
was some design error in Phase 1, since we were only designing
locally. This doesn't mean that we only need one Jira ticket, there
can be many in support of this KIP, or that we only need one PR, it's
certainly better to send multiple small PRs to decrease risk and ease
reviews. But the design discussion doesn't need to be fragmented
similarly.

> 3. return null vs NoSuchElementException when empty queue
>
> Should this be also included to the above TTD usability changes?
> If single row read methods is changed to throw expectiong, it would require
> addition of hasRecords to able to verified the empty queue scenarios.
> I do not know how to implement it currently without modifying TTD to
> provide some kind way to get the queue size or peak items.

Yes, it's absolutely within bounds to propose changes to TTD to
support the ergonomic API you're proposing.

> 4. IllegalArgumentException("topic doesn't exist")
> Is this worth separate ticket?

This is your call. That was just an idea in response to your experience.

> 5. org.apache.kafka.streams.test vs org.apache.kafka.streams
>
> I was thinking org.apache.kafka.streams.test where also OutputVerifier and
> ConsumerRecordFactory exist would be more logical place, but
> I do not know is there some 

Build failed in Jenkins: kafka-trunk-jdk8 #3609

[Apache Jenkins Server]

See 


Changes:

[cshapi] MINOR: correctly parse version OffsetCommitResponse version < 3

[github] KAFKA-8158: Add EntityType for Kafka RPC fields (#6503)

--
[...truncated 4.81 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 

Re: Cleaning up command line tools argument parsing a little

[Viktor Somogyi-Vass]

Hi Sönke,

In KAFKA-1774 I have created a Kafka Shell java tool that unfortunately
didn't get much attention so far from the creators of the jira. It works
similarly to what Colin mentioned, like "kafka.sh topics create -n my-topic
-p 3 -r 3" or has long names like "kafka.sh topics create --name my-topic
--partitions 3 --replicas 3". The bootstrap server everywhere defaults to
:9092 or reads up the configs from a path so in many scenarios you can omit
it, also it uses the admin client of course and all in all provides a much
better experience than the current tools. It has interactive mode and help
too. Wanted to implement "code completion" too but for that I'd have to
exercise the code a little bit more :).
It is currently based on a quite old trunk but if you think it's
interesting I can rebase it and we can continue with raising a KIP.
https://github.com/viktorsomogyi/kafka/tree/kafka_shell

Viktor

On Tue, May 7, 2019 at 11:10 AM Sönke Liebau
 wrote:

> Hi Colin,
>
> thanks!
> While I personally don't like the current command line tools I
> realistically think we'll be stuck with them for a while yet, so a cleanup
> might make sense.
> So I'll start looking into that.
>
> Regarding a central entrypoint, that would indeed be brilliant and I'd love
> to work on that, but I currently have enough other open things to look at,
> so I won't draw that one to myself as well for now.
>
> But I'll keep it in mind for when some time frees up :)
>
> Best regards,
> Sönke
>
>
>
> Colin McCabe  schrieb am Di., 7. Mai 2019, 00:56:
>
> > On Mon, May 6, 2019, at 10:21, Sönke Liebau wrote:
> > > Hi Colin,
> > >
> > > it was my intention to keep the structure of the commands mostly intact
> > > while doing the refactoring - if that is possible, have not really
> > checked
> > > yet to be honest.
> > >
> > > But what I wanted to try and do is recreate the current parsing with
> > > argparse as much as possible. And in the process simply adding
> synonyms,
> > > for example make the kafka-console-producer understand a
> > > bootstrap-parameter in addition to broker-list.
> > > There is a bit of custom logic about which parameters go together etc.
> in
> > > the current classes, so output may look different here and there, but
> in
> > > principle I do believe that it should be possible to recreate the
> current
> > > structure.
> >
> > Sounds like a good idea.  Thanks for the clarification.
> >
> > >
> > > If there is an appetite for a new, hadoop-like entrypoint anyway, then
> > all
> > > of this might be "wasted" effort, or rather effort better spent though,
> > you
> > > are right.
> >
> > I don't think anyone is working on a new entry point right now -- or if
> > they are, they haven't said anything yet :)
> >
> > I just wanted to mention it as a possible approach in case you wanted to
> > do a bigger project.
> >
> > best,
> > Colin
> >
> > >
> > > Best regards,
> > > Sönke
> > >
> > >
> > >
> > > On Mon, May 6, 2019 at 7:13 PM Colin McCabe 
> wrote:
> > >
> > > > Hi Sönke,
> > > >
> > > > #2 is a bit tough because people have come to rely on the way the
> > commands
> > > > are structured right now.
> > > >
> > > > If we want to make big changes, it might be easier just to create a
> > > > separate tool and deprecate the old one(s).  One thing we've talked
> > about
> > > > doing in the past is creating a single entry point for all the tool
> > > > functionality, kind of like hadoop did with the "hadoop" command  Or
> > git
> > > > with the "git" command, etc.  Then we could deprecate the standalone
> > > > commands and remove them after enough time had passed-- kind of like
> > the
> > > > old consumer.
> > > >
> > > > On the other hand, a more incremental change would be standardizing
> > flags
> > > > a bit.  So for example, at least setting it up so that there is a
> > standard
> > > > way of supplying bootstrap brokers, etc.  We could keep the old flags
> > > > around for a while as variants to ease the transition.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote:
> > > > > Hi Colin,
> > > > >
> > > > > I totally agree! Especially the differently named bootstrap server
> > > > options
> > > > > have been annoying me a long time.
> > > > >
> > > > > I'd propose a two-step approach:
> > > > > 1. Add new default options objects similar to CommandLineUtils and
> > > > > CommandDefaultOptions (based on argparse4j) but in the clients
> > project,
> > > > as
> > > > > this is referenced by all command line tools as far as I can tell
> > > > > 2. Refactor tools one by one to use these new helper classes (and
> > thus
> > > > > argparse) and add standardized synonyms for parameters as necessary
> > > > >
> > > > > I think for step 1 we can get away with no KIP, as this doesn't
> > change
> > > > any
> > > > > public interfaces or behavior.
> > > > > Step 2 probably needs a KIP as we are adding new parameters? We can
> > pick
> > > > up
> > > > > KIP-14 again 

Re: Cleaning up command line tools argument parsing a little

[Viktor Somogyi-Vass]

Just to add: I wasn't too pushy by raising a KIP for this as so far I had
the experience that the community is fine with them or at least people
learned to live with the current tools but if there's a need I'd be happy
to jump back working on it or helping you if you have time :)

On Wed, May 8, 2019 at 11:35 AM Viktor Somogyi-Vass 
wrote:

> Hi Sönke,
>
> In KAFKA-1774 I have created a Kafka Shell java tool that unfortunately
> didn't get much attention so far from the creators of the jira. It works
> similarly to what Colin mentioned, like "kafka.sh topics create -n my-topic
> -p 3 -r 3" or has long names like "kafka.sh topics create --name my-topic
> --partitions 3 --replicas 3". The bootstrap server everywhere defaults to
> :9092 or reads up the configs from a path so in many scenarios you can omit
> it, also it uses the admin client of course and all in all provides a much
> better experience than the current tools. It has interactive mode and help
> too. Wanted to implement "code completion" too but for that I'd have to
> exercise the code a little bit more :).
> It is currently based on a quite old trunk but if you think it's
> interesting I can rebase it and we can continue with raising a KIP.
> https://github.com/viktorsomogyi/kafka/tree/kafka_shell
>
> Viktor
>
> On Tue, May 7, 2019 at 11:10 AM Sönke Liebau
>  wrote:
>
>> Hi Colin,
>>
>> thanks!
>> While I personally don't like the current command line tools I
>> realistically think we'll be stuck with them for a while yet, so a cleanup
>> might make sense.
>> So I'll start looking into that.
>>
>> Regarding a central entrypoint, that would indeed be brilliant and I'd
>> love
>> to work on that, but I currently have enough other open things to look at,
>> so I won't draw that one to myself as well for now.
>>
>> But I'll keep it in mind for when some time frees up :)
>>
>> Best regards,
>> Sönke
>>
>>
>>
>> Colin McCabe  schrieb am Di., 7. Mai 2019, 00:56:
>>
>> > On Mon, May 6, 2019, at 10:21, Sönke Liebau wrote:
>> > > Hi Colin,
>> > >
>> > > it was my intention to keep the structure of the commands mostly
>> intact
>> > > while doing the refactoring - if that is possible, have not really
>> > checked
>> > > yet to be honest.
>> > >
>> > > But what I wanted to try and do is recreate the current parsing with
>> > > argparse as much as possible. And in the process simply adding
>> synonyms,
>> > > for example make the kafka-console-producer understand a
>> > > bootstrap-parameter in addition to broker-list.
>> > > There is a bit of custom logic about which parameters go together
>> etc. in
>> > > the current classes, so output may look different here and there, but
>> in
>> > > principle I do believe that it should be possible to recreate the
>> current
>> > > structure.
>> >
>> > Sounds like a good idea.  Thanks for the clarification.
>> >
>> > >
>> > > If there is an appetite for a new, hadoop-like entrypoint anyway, then
>> > all
>> > > of this might be "wasted" effort, or rather effort better spent
>> though,
>> > you
>> > > are right.
>> >
>> > I don't think anyone is working on a new entry point right now -- or if
>> > they are, they haven't said anything yet :)
>> >
>> > I just wanted to mention it as a possible approach in case you wanted to
>> > do a bigger project.
>> >
>> > best,
>> > Colin
>> >
>> > >
>> > > Best regards,
>> > > Sönke
>> > >
>> > >
>> > >
>> > > On Mon, May 6, 2019 at 7:13 PM Colin McCabe 
>> wrote:
>> > >
>> > > > Hi Sönke,
>> > > >
>> > > > #2 is a bit tough because people have come to rely on the way the
>> > commands
>> > > > are structured right now.
>> > > >
>> > > > If we want to make big changes, it might be easier just to create a
>> > > > separate tool and deprecate the old one(s).  One thing we've talked
>> > about
>> > > > doing in the past is creating a single entry point for all the tool
>> > > > functionality, kind of like hadoop did with the "hadoop" command  Or
>> > git
>> > > > with the "git" command, etc.  Then we could deprecate the standalone
>> > > > commands and remove them after enough time had passed-- kind of like
>> > the
>> > > > old consumer.
>> > > >
>> > > > On the other hand, a more incremental change would be standardizing
>> > flags
>> > > > a bit.  So for example, at least setting it up so that there is a
>> > standard
>> > > > way of supplying bootstrap brokers, etc.  We could keep the old
>> flags
>> > > > around for a while as variants to ease the transition.
>> > > >
>> > > > best,
>> > > > Colin
>> > > >
>> > > >
>> > > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote:
>> > > > > Hi Colin,
>> > > > >
>> > > > > I totally agree! Especially the differently named bootstrap server
>> > > > options
>> > > > > have been annoying me a long time.
>> > > > >
>> > > > > I'd propose a two-step approach:
>> > > > > 1. Add new default options objects similar to CommandLineUtils and
>> > > > > CommandDefaultOptions (based on argparse4j) but in the clients
>> > project,
>> > > > as
>> > > 

[jira] [Created] (KAFKA-8336) Enable dynamic update of client-side SSL factory in brokers

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-8336:
-

 Summary: Enable dynamic update of client-side SSL factory in 
brokers
 Key: KAFKA-8336
 URL: https://issues.apache.org/jira/browse/KAFKA-8336
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 2.2.0
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 2.3.0


We currently support dynamic update of server-side keystores. This allows 
expired certs to be updated on brokers without a rolling restart. When mutual 
authentication is enabled for inter-broker-communication 
(ssl.client.auth=required), we dont currently dynamically update client-side 
keystores for controller or transaction coordinator. So a broker restart (or 
controller change) is required for cert update for this case. Since short-lived 
SSL cert is a common usecase, we should enable client-side cert updates for all 
client connections initiated by the broker to ensure that SSL certificate 
expiry can be handled with dynamic config updates on brokers for all 
configurations.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-456: Helper classes to make it simpler to write test logic with TopologyTestDriver

[Jukka Karvanen]

Hi,

This KIP is relating to add classes to kafka-stream-test-utils.
My initial plan was to create new project for these helper classes before I
decided to contribute directly to Apache Kafka with this KIP.
I have one streams project where I have used these classes, that is why I
planned to release this as separate "early access" package
before the Kafka release including this is publicly available. This would
work at the same time "a historically-compatible version of the library".

I try to summarize the open topics after your last email:

1. Add JavaDoc for KIP

Is there a good example of KIP where Javadoc is included, so I can follow?
I create this KIP based on this as an example::
https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams


Now added some comments to KIP page to clarify timestamp handling, but I
did not want to add full JavaDoc of each methods.
Is that enough?

2. TTD usability changes and swapping ConsumerRecord and ProducerRecord in
APIs

To my point of view only:
- changing readRecord to return ConsumerRecord would cause we cannot use
OutputVerifier
- changing pipeInput to take in ProducerRecord, but not providing easy way
to contruct those like ConsumerRecordFactory
- if initializing ConsumerRecord to/from  ProducerRecord  in these classes
field by field contructor, there are risk new fields are not added to this
classes if there are changes in ProducerRecord or ConsumerRecord

I would propose a separate KIP for these and probably other enhanchements:
-superclass or common interface for ConsumerRecord and ProducerRecord
-contructors to ConsumerRecord and ProducerRecord to initialize with this
superclass
-modify OutputVerifier to work with both ConsumerRecord and ProducerRecord
-create new RecordFactory to replace ConsumerRecordFactory


3. return null vs NoSuchElementException when empty queue

Should this be also included to the above TTD usability changes?
If single row read methods is changed to throw expectiong, it would require
addition of hasRecords to able to verified the empty queue scenarios.
I do not know how to implement it currently without modifying TTD to
provide some kind way to get the queue size or peak items.

4. IllegalArgumentException("topic doesn't exist")
Is this worth separate ticket?

5. org.apache.kafka.streams.test vs org.apache.kafka.streams

I was thinking org.apache.kafka.streams.test where also OutputVerifier and
ConsumerRecordFactory exist would be more logical place, but
I do not know is there some technical reasons why TTD are in
org.apache.kafka.streams, not in org.apache.kafka.streams.test where other
classes are.

Did I skip something?

Jukka


ti 7. toukok. 2019 klo 22.02 John Roesler (j...@confluent.io) kirjoitti:

> Thanks for the responses, Jukka!
>
> Thanks for the reference to the javadocs in your library, but I
> actually meant they should be part of the KIP document.
>
> As a general comment, I did get that your intent is to smooth over
> some rough spots with the current testing library, and that many of
> your API/behavior decisions are just reflecting the underlying TTD.
> What I'm wondering is whether we can take the opportunity to make a
> bigger change for an even smoother usability experience.
>
> Regarding the publishing artifact, did you mean that you're proposing
> to add a separate build artifact to the Apache Kafka project, or just
> that you plan to host a historically-compatible version of the library
> on your own? My initial reaction is that we should just include this
> stuff in the test-utils package starting at whatever version. This
> gives us greater latitude to modify the underlying layers in service
> of this KIP. Plus, it's nicer as a user to just have one test-support
> artifact to manage.
>
> Specific responses below:
> > >InputTopic:
> > >
> > >1. Have you considered adding a ProducerRecord input method to the input
> > >topic? This might be unnecessary, given the overloads you provide. I'm
> > >wondering if it decreases the domain mismatch between TopologyTestDriver
> > >and KafkaStreams, though, since in production code, you send input to
> the
> > >app as a ProducerRecord, via a topic. Also, this might let you drop
> some of
> > >the less mainstream overloads, like the ones for headers.
> >
> > Ok, so not same methods as in TopologyTestDriver:
> > pipeInput(ConsumerRecord consumerRecord);
> > pipeInput(List> records);
> > but instead:
> > pipeInput(ProducerRecord  record);
> > pipeInput(List< ProducerRecord  > records);
> > Which would convert ProductRecords to ConsumerRecords before piping to
> TTD
> >
> > and Drop these methods
> > void pipeInput(K key, V value, Headers headers);
> > void pipeInput(K key, V value, Headers headers, long timestampMs);
>
> Yes, this is essentially what I was saying.
>
> >
> > In this case you would not able to create those with
> ConsumerRecordFactory,
> > but needing to create  ProducerRecord   objects directly.one by one 

Build failed in Jenkins: kafka-trunk-jdk8 #3608

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-8304: Fix registration of Connect REST extensions (#6651)

[wangguoz] KAFKA-8284: enable static membership on KStream (#6673)

--
[...truncated 4.81 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 

[jira] [Created] (KAFKA-8335) Log cleaner skips Transactional mark and batch record, causing unlimited growth of __consumer_offsets

[Boquan Tang (JIRA)]

Boquan Tang created KAFKA-8335:
--

 Summary: Log cleaner skips Transactional mark and batch record, 
causing unlimited growth of __consumer_offsets
 Key: KAFKA-8335
 URL: https://issues.apache.org/jira/browse/KAFKA-8335
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Boquan Tang


My Colleague Weichu already sent out a mail to kafka user mailing list 
regarding this issue, but we think it's worth having a ticket tracking it.

We are using Kafka Streams with exactly-once enabled on a Kafka cluster for
a while.

Recently we found that the size of __consumer_offsets partitions grew huge.
Some partition went over 30G. This caused Kafka to take quite long to load
"__consumer_offsets" topic on startup (it loads the topic in order to
become group coordinator).

We dumped the __consumer_offsets segments and found that while normal
offset commits are nicely compacted, transaction records (COMMIT, etc) are
all preserved. Looks like that since these messages don't have a key, the
LogCleaner is keeping them all:

--
$ bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files
/003484332061.log --key-decoder-class
kafka.serializer.StringDecoder 2>/dev/null | cat -v | head
Dumping 003484332061.log
Starting offset: 3484332061
offset: 3484332089 position: 549 CreateTime: 1556003706952 isvalid: true
keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId: 1006
producerEpoch: 2530 sequence: -1 isTransactional: true headerKeys: []
endTxnMarker: COMMIT coordinatorEpoch: 81
offset: 3484332090 position: 627 CreateTime: 1556003706952 isvalid: true
keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId: 4005
producerEpoch: 2520 sequence: -1 isTransactional: true headerKeys: []
endTxnMarker: COMMIT coordinatorEpoch: 84
...
--

Streams is doing transaction commits per 100ms (commit.interval.ms=100 when
exactly-once) so the __consumer_offsets is growing really fast.

Is this (to keep all transactions) by design, or is that a bug for
LogCleaner?  What would be the way to clean up the topic?





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)