[jira] [Created] (KAFKA-8334) Occasional OffsetCommit Timeout

[windkithk (JIRA)]

windkithk created KAFKA-8334:


 Summary: Occasional OffsetCommit Timeout
 Key: KAFKA-8334
 URL: https://issues.apache.org/jira/browse/KAFKA-8334
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 1.1.1
Reporter: windkithk
 Attachments: offsetcommit_p99.9_cut.jpg

h2. 1) Issue Summary
Since we have upgraded to 1.1, we have observed occasional OffsetCommit 
timeouts from clients
{code:java}
Offset commit failed on partition sometopic-somepartition at offset someoffset: 
The request timed out{code}

Normally OffsetCommit completes within 10ms but when we check the 99.9 
percentile, we see the request duration time jumps up to 5000 ms 
(offsets.commit.timeout.ms)

Here is a screenshot of prometheus recording 
kafka_network_request_duration_milliseconds
 !offsetcommit_p99.9_cut.jpg!
and after checking the duration breakdown, most of the time was spent on 
"Remote" Scope

(Below is a request log line produced by inhouse slow request logger
{code:java}
[2019-04-16 13:06:20,339] WARN Slow 
response:RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=2, 
clientId=kafka-python-1.4.6, correlationId=953) -- 
{group_id=wilson-tester,generation_id=28,member_id=kafka-python-1.4.6-69ed979d-a069-4c6d-9862-e4fc34883269,retention_time=-1,topics=[{topic=test,partitions=[{partition=2,offset=63456,metadata=null}]}]}
 from connection 
10.127.114.124:20992-10.127.144.181:35078-31125;totalTime:5001.942000,requestQueueTime:0.03,localTime:0.574000,remoteTime:5001.173000,responseQueueTime:0.058000,sendTime:0.053000,securityProtocol:PLAINTEXT,principal:User:ANONYMOUS
 (kafka.request.logger)
{code}

h2. 2) What got changed in 1.1 from 0.10.2.1?

# Log Level Changed
In 1.1 Kafka Consumer, logging about timed out OffsetCommit is changed from 
DEBUG to WARN
# Group Lock is acquired when trying to complete DelayedProduce of OffsetCommit
This was added after 0.11.0.2
(Ticket: https://issues.apache.org/jira/browse/KAFKA-6042)
(PR: https://github.com/apache/kafka/pull/4103)
(in 1.1 
https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L292)
# Followers do incremental fetch
https://cwiki.apache.org/confluence/display/KAFKA/KIP-227%3A+Introduce+Incremental+FetchRequests+to+Increase+Partition+Scalability

h2. 3) Interaction between OffsetCommit, DelayedProduce and FetchFollower
{quote}
OffsetCommit append a message of committed offset to partition of topic 
`__consumer_offsets`

During the append, it would create a DelayedProduce with lock to 
GroupMetadata.lock (ReentrantLock) and add to delayedProducePurgatory

When follower fetches the partition of topic `__consumer_offsets` and causes an 
increase in HighWaterMark, delayedProducePurgatory would be transversed and all 
operations related to the partition may be completed
{quote}

*DelayedProduce from OffsetCommit may not be completed, if the group metadata 
lock was held by others*

h2. 4) Reproduce
h4. Methodology
{code}
1. DelayedProduce on __consumer_offsets could not be completed if the 
group.lock is acquired by others
2. We spam requests like Heartbeat to keep acquiring group.lock
3. We keep sending OffsetCommit and check the processing time
{code}

h4. Reproduce Script
https://gist.github.com/windkit/3384bb86dc146111d1e0857e66b85861
# jammer.py - join the group "wilson-tester" and keep spamming Heartbeat
# tester.py - fetch one message and do a long processing (or sleep) and then 
commit the offset

h4. Result
||Seq||Operation||Lock||
|1|OffsetCommit Request 
|2|Append to local __consumer_offsets   
|3|DelayedProduce tryComplete   
|4|Added into delayedProducePurgatory   
|5|FetchFollower1 Fetch
|6|FetchFollower2 Fetch 
|7|Heartbeat Request|Acquired group.lock
|8|FetchFollower2 maybeTryComplete DelayedProduce|Failed to acquire group.lock
|9|Heartbeat Response|Released group.lock
| |(NO FetchFollower Requests on the partitions __consumer_offsets)
|10|OffsetCommit Response (Timeout)

h4. Trace Log
{code}
// The OffsetCommit Request
[2019-04-15 23:59:53,736] TRACE [KafkaApi-1] Handling 
request:RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=2, 
clientId=kafka-python-1.4.6, correlationId=2114) -- 
{group_id=wilson-tester,generation_id=20,member_id=kafka-python-1.4.6-60008b58-4d6a-4cfd-948f-dd9e19e7f981,retention_time=-1,topics=[{topic=test,partitions=[{partition=2,offset=22654,metadata=null}]}]}
 from connection 
10.127.114.124:20992-10.127.144.181:57032-29505;securityProtocol:PLAINTEXT,principal:User:ANONYMOUS
 (kafka.server.KafkaApis)
 
// Initial Check of DelayedProduce:tryCompleteElseWatch
// 
https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/server/DelayedOperation.scala#L217
[2019-04-15 23:59:53,736] TRACE Initial partition status for 
__consumer_offsets-48 is [acksPending: true, error: 7, startOffset: 23134, 
requiredOffset: 23135] (kafka.server.DelayedProduce)

Build failed in Jenkins: kafka-trunk-jdk11 #497

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-8158: Add EntityType for Kafka RPC fields (#6503)

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H30 (ubuntu xenial) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 5144660040839cee6e213b5146c1ae44340eddb0 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5144660040839cee6e213b5146c1ae44340eddb0
Commit message: "KAFKA-8158: Add EntityType for Kafka RPC fields (#6503)"
 > git rev-list --no-walk 8308829e46fcc4903b9faa3d2dcdf10f56f262fe # timeout=10
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
[kafka-trunk-jdk11] $ /bin/bash -xe /tmp/jenkins5023201203146428935.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.10.2/bin/gradle
/tmp/jenkins5023201203146428935.sh: line 4: 
/home/jenkins/tools/gradle/4.10.2/bin/gradle: No such file or directory
Build step 'Execute shell' marked build as failure
Recording test results
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Not sending mail to unregistered user nore...@github.com
Not sending mail to unregistered user csh...@gmail.com
Not sending mail to unregistered user wangg...@gmail.com

Re: I want to contribute to Apache Kafka

[Colin McCabe]

Hi,

Sorry, JIRA doesn't seem to have any user named 15332318...@189.cn

best,
Colin

On Sat, May 4, 2019, at 21:21, 15332318...@189.cn wrote:
> Hi,
> 
> I want to contribute to Apache Kafka.
> Would you please give me the contributor permission?
> My JIRA ID is 15332318...@189.cn.
> 
> 
> 
> 
>

Build failed in Jenkins: kafka-trunk-jdk11 #496

[Apache Jenkins Server]

See 


Changes:

[cshapi] MINOR: correctly parse version OffsetCommitResponse version < 3

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H28 (ubuntu xenial) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 8308829e46fcc4903b9faa3d2dcdf10f56f262fe 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8308829e46fcc4903b9faa3d2dcdf10f56f262fe
Commit message: "MINOR: correctly parse version OffsetCommitResponse version < 
3"
 > git rev-list --no-walk b0e82a68b34512d23a5a60a9b5da0db86eb880b2 # timeout=10
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
[kafka-trunk-jdk11] $ /bin/bash -xe /tmp/jenkins2236673357884216048.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.10.2/bin/gradle
/tmp/jenkins2236673357884216048.sh: line 4: 
/home/jenkins/tools/gradle/4.10.2/bin/gradle: No such file or directory
Build step 'Execute shell' marked build as failure
Recording test results
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Not sending mail to unregistered user nore...@github.com
Not sending mail to unregistered user csh...@gmail.com
Not sending mail to unregistered user wangg...@gmail.com

Build failed in Jenkins: kafka-trunk-jdk11 #495

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-8284: enable static membership on KStream (#6673)

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H24 (ubuntu xenial) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision b0e82a68b34512d23a5a60a9b5da0db86eb880b2 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b0e82a68b34512d23a5a60a9b5da0db86eb880b2
Commit message: "KAFKA-8284: enable static membership on KStream (#6673)"
 > git rev-list --no-walk cc097e909c7e5b1617565e4456f5328a826eafbc # timeout=10
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
[kafka-trunk-jdk11] $ /bin/bash -xe /tmp/jenkins3108833113718836585.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.10.2/bin/gradle
/tmp/jenkins3108833113718836585.sh: line 4: 
/home/jenkins/tools/gradle/4.10.2/bin/gradle: No such file or directory
Build step 'Execute shell' marked build as failure
Recording test results
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Not sending mail to unregistered user nore...@github.com
Not sending mail to unregistered user wangg...@gmail.com

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

[Chris Egerton]

Hi all,

Now that  KAFKA-8304 (https://issues.apache.org/jira/browse/KAFKA-8304),
which was a blocker, has been addressed, I've published a PR for these
changes: https://github.com/apache/kafka/pull/6584

Thanks to everyone who's voted so far! If anyone else is interested, the
voting thread can be found here:
https://www.mail-archive.com/dev@kafka.apache.org/msg97458.html. Current
status: +1 binding, +2 non-binding.

Cheers,

Chris

On Tue, Apr 30, 2019 at 12:40 PM Chris Egerton  wrote:

> Hi Konstantine,
>
> I've updated the KIP to add default method implementations to the list of
> rejected alternatives. Technically this makes the changes in the KIP
> backwards incompatible, but I think I agree that for the majority of cases
> where it would even be an issue a compile-time error is likely to be more
> beneficial than one at run time.
>
> Thanks for your thoughts and thanks for the LGTM!
>
> Cheers,
>
> Chris
>
> On Mon, Apr 29, 2019 at 12:29 PM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
>> Hi Chris,
>>
>> Thanks for considering my suggestion regarding default implementations for
>> the new methods.
>> However, given that these implementations don't seem to have sane defaults
>> and throw UnsupportedOperationException, I think we'll be better without
>> defaults.
>> Seems that a compile time error is preferable here, for those who want to
>> upgrade their implementations.
>>
>> Otherwise, the KIP LGTM.
>>
>> Konstantine
>>
>> On Thu, Apr 25, 2019 at 10:29 PM Magesh Nandakumar 
>> wrote:
>>
>> > Thanks a lot, Chris. The KIP looks good to me.
>> >
>> > On Thu, Apr 25, 2019 at 7:35 PM Chris Egerton 
>> wrote:
>> >
>> > > Hi Magesh,
>> > >
>> > > Sounds good; I've updated the KIP to make ConnectClusterDetails an
>> > > interface. If we want to leave the door open to expand it in the
>> future
>> > it
>> > > definitely makes sense to treat it similarly to how we're treating the
>> > > ConnectClusterState interface now.
>> > >
>> > > Thanks,
>> > >
>> > > Chris
>> > >
>> > > On Thu, Apr 25, 2019 at 4:18 PM Magesh Nandakumar <
>> mage...@confluent.io>
>> > > wrote:
>> > >
>> > > > HI Chrise,
>> > > >
>> > > > Overall it looks good to me. Just one last comment - I was
>> wondering if
>> > > > ConnectClusterDetail should be an interface just like
>> > > ConnectClusterState.
>> > > >
>> > > > Thanks,
>> > > > Magesh
>> > > >
>> > > > On Thu, Apr 25, 2019 at 3:54 PM Chris Egerton 
>> > > wrote:
>> > > >
>> > > > > Hi Magesh,
>> > > > >
>> > > > > Expanding the type we use to convey cluster metadata from just a
>> > Kafka
>> > > > > cluster ID string to its own class seems like a good idea for the
>> > sake
>> > > of
>> > > > > forwards compatibility, but I'm still not sure what the gains of
>> > > > including
>> > > > > the cluster group ID would be--it's a simple map lookup away in
>> the
>> > > REST
>> > > > > extension's configure(...) method. Including information on
>> whether
>> > the
>> > > > > cluster is distributed or standalone definitely seems convenient;
>> as
>> > > far
>> > > > as
>> > > > > I can tell there's no easy way to do that from within a REST
>> > extension
>> > > at
>> > > > > the moment, and relying on something like the presence of a
>> group.id
>> > > > > property to identify a distributed cluster could result in false
>> > > > positives.
>> > > > > However, is there a use case for it? If not, we can note that as a
>> > > > possible
>> > > > > addition to the ConnectClusterDetails class for later but leave it
>> > out
>> > > > for
>> > > > > now.
>> > > > >
>> > > > > I've updated the KIP to include the new ConnectClusterDetails
>> class
>> > but
>> > > > > left out the cluster type information for now; let me know what
>> you
>> > > > think.
>> > > > >
>> > > > > Thanks again for your thoughts!
>> > > > >
>> > > > > Cheers,
>> > > > >
>> > > > > Chris
>> > > > >
>> > > > > On Wed, Apr 24, 2019 at 4:49 PM Magesh Nandakumar <
>> > > mage...@confluent.io>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Chris,
>> > > > > >
>> > > > > > Instead of calling it ConnectClusterId, perhaps call it
>> > > > > > ConnectClusterDetails which can include things like groupid,
>> > > underlying
>> > > > > > kafkaclusterId, standalone or distributed, etc. This will help
>> > expose
>> > > > any
>> > > > > > cluster related information in the future.
>> > > > > > Let me know if that would work?
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Magesh
>> > > > > >
>> > > > > > On Wed, Apr 24, 2019 at 4:26 PM Chris Egerton <
>> chr...@confluent.io
>> > >
>> > > > > wrote:
>> > > > > >
>> > > > > > > Hi Magesh,
>> > > > > > >
>> > > > > > > 1. After ruminating for a little while on the inclusion of a
>> > method
>> > > > to
>> > > > > > > retrieve task configurations I've tentatively decided to
>> remove
>> > it
>> > > > from
>> > > > > > the
>> > > > > > > proposal and place it in the rejected alternatives section. If
>> > > anyone
>> > > > > > > presents a reasonable use case for it I'll be 

[jira] [Created] (KAFKA-8333) Load high watermark checkpoint only once when handling LeaderAndIsr requests

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-8333:
--

 Summary: Load high watermark checkpoint only once when handling 
LeaderAndIsr requests
 Key: KAFKA-8333
 URL: https://issues.apache.org/jira/browse/KAFKA-8333
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson
Assignee: Jason Gustafson


Currently we reload the checkpoint file separately for every partition that is 
first initialized on the broker. It would be more efficient to do this one time 
only when we receive the LeaderAndIsr request and to reuse the state.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8332) Regression in handling of JoinGroupRequest disallows deterministic protocol selection based on order of preference

[Konstantine Karantasis (JIRA)]

Konstantine Karantasis created KAFKA-8332:
-

 Summary: Regression in handling of JoinGroupRequest disallows 
deterministic protocol selection based on order of preference
 Key: KAFKA-8332
 URL: https://issues.apache.org/jira/browse/KAFKA-8332
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: Konstantine Karantasis
Assignee: Konstantine Karantasis


When a group of Kafka clients includes more than one embedded protocol in its 
{{JoinGroupRequest}} along with its metadata, the group membership protocol 
defines that the protocol which is supported by all the members of a group is 
selected, and if more than one protocols are supported by all the members the 
protocol is selected based on the order of preference as defined in the 
{{JoinGroupRequest}}. 

A recent change from type {{List}} to type {{Set}} for storing the set of 
supported embedded protocols in the {{JoinGroupRequest}} combined with the old 
type of handling with implicit types in the scala code, has introduced 
non-determinism in the selection of the embedded protocol by the 
{{GroupCoordinator}}, even though the underlying type of the Set in use is a 
variant of LinkedHashSet (it respects order). 

The relevant code is: 
{code:java}
// KafkaApis.scala
val protocols = joinGroupRequest.data().protocols().asScala.map(protocol =>
(protocol.name, protocol.metadata)).toList
{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-439: Deprecate Interface WindowStoreIterator

[Guozhang Wang]

Thanks John.

I think I'm convinced about not collapsing the metrics measuring for
various function calls. Regarding the possible solution I'm a bit inclined
to just distinguish on the metric names directly over on tags: since our
current metrics naming are a tad messed up anyways, fixing them in one shot
as a breaking change sounds reasonable to me.


Guozhang


On Mon, May 6, 2019 at 9:14 AM John Roesler  wrote:

> Thanks all (or range? ;) ) for the discussion. Good points all around.
>
> Although I see the allure of naming the metrics the same as the things
> they're measuring, it seems not to be perfect. Seconding Matthias's latter
> thought, I think it's likely you'd want to measure the method calls
> independently, since the different range variants would have wildly
> different characteristics, which could then lead you to want to orient the
> storage differently to support particular use cases.
>
> Pointing out some structural characteristics (I know you all know this
> stuff, I'm just constructing a table for analysis):
> * Java supports method-name overloading. *Different* methods can have the
> same names, distinguished by arg lists; it doesn't change the fact that
> they are different methods.
> * Metrics does not support metric-name overloading, but metric names do
> have some structure we could exploit, if you consider the tags.
>
> It seems to me that there's actually more domain mismatch if we just have
> one metric named "range", since (e.g., in the SessionStore proposal above)
> the Java API has *four* methods named "range".
>
> Two potential solutions I see:
> * hierarchical metric names: "range-single-key-all-time",
> "range-key-range-all-time", "range-single-key-time-range",
> "range-key-range-time-range", maybe with better names... I'm not the best
> at this stuff. Hopefully, you see the point, though... they all start with
> "range", which provides the association to the method, and all have a
> suffix which identifies the overload being measured.
> * tagged metric names: "range" {"variant": "single-key-all-time"}, "range"
> {"variant": "key-range-all-time"}, "range" {"variant":
> "single-key-time-range"}, "range" {"variant": "key-range-time-range"} . Or
> you could even split the tags up semantically, but my instinct says that
> that would just make it harder to digest the metrics later on.
>
> Just some ideas.
> -John
>
> On Fri, Apr 26, 2019 at 3:51 AM Matthias J. Sax 
> wrote:
>
> > Thanks for the input Guozhang. I was not aware of those dependencies.
> >
> > It might be good to align this KIP with the metrics cleanup. Not sure
> > atm, if we should use different metric names for different overloads,
> > even if those have the same method name?
> >
> > If we rename all method to `range()` and use the same metric name for
> > all, one could argue that this is still fine, because the metric
> > collects how often a range query is executed (regardless of the range
> > itself).
> >
> > On the other hand, this would basically be a "roll up". It could still
> > be valuable to distinguish between single-key-time-range,
> > key-range-time-range, and all-range queries. Users could still aggregate
> > those later if they are not interested in the details, while it's not
> > possible for user to split a pre-aggregated metric into it's component.
> >
> >
> > Input from others might be helpful here, too.
> >
> >
> > -Matthias
> >
> > On 4/11/19 6:00 PM, Guozhang Wang wrote:
> > > While working at KIP-444 (https://github.com/apache/kafka/pull/6498) I
> > > realized there are a bunch of issues on metric names v.s. function
> names,
> > > e.g. some function named `fetchAll` are actually measure with `fetch`,
> > etc.
> > > So in that KIP I proposed to make the function name aligned with
> metrics
> > > name. So suppose we rename the functions from `fetch` to `range` I'd
> > > suggest we make this change as part of KIP-444 as well. Note that it
> > means
> > > different functions with the same name `range` will be measured under a
> > > single metric then.
> > >
> > > But still for function named `all` it will be measured under a separate
> > > metric named `all`, so I'm just clarifying with you if that's the
> > intention.
> > >
> > >
> > > Guozhang
> > >
> > > On Thu, Apr 11, 2019 at 2:04 PM Matthias J. Sax  >
> > > wrote:
> > >
> > >> I did not see a reason to rename `all()` to `range()`. `all()` does
> not
> > >> take any parameters to limit a range and is a good name IMHO. But I am
> > >> not married to keep `all()` and if we think we should rename it, too,
> I
> > >> am fine with it.
> > >>
> > >> Not sure what connection you make to metrics though. Can you
> elaborate?
> > >>
> > >>
> > >> Would be interested to hear others opinions on this, too.
> > >>
> > >>
> > >> -Matthias
> > >>
> > >> On 4/11/19 8:38 AM, Guozhang Wang wrote:
> > >>> I like the renaming, since it also aligns with our metrics cleanup
> > >>> (KIP-444) which touches upon the store level metrics as well.
> > >>>
> > >>> 

[jira] [Created] (KAFKA-8331) Add system test for enabling static membership on KStream

[Boyang Chen (JIRA)]

Boyang Chen created KAFKA-8331:
--

 Summary: Add system test for enabling static membership on KStream
 Key: KAFKA-8331
 URL: https://issues.apache.org/jira/browse/KAFKA-8331
 Project: Kafka
  Issue Type: Sub-task
Reporter: Boyang Chen
Assignee: Boyang Chen






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-8304) Connect susceptible to deadlock while registering REST extensions

[Randall Hauch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch resolved KAFKA-8304.
--
Resolution: Fixed

Merged onto `trunk`, and backported to `2.2`, `2.1`, and `2.0`.

> Connect susceptible to deadlock while registering REST extensions
> -
>
> Key: KAFKA-8304
> URL: https://issues.apache.org/jira/browse/KAFKA-8304
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 2.2.0, 2.1.1, 2.0.2, 2.3.0, 2.1.2, 2.2.1
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Blocker
> Fix For: 2.0.2, 2.3.0, 2.1.2, 2.2.1
>
>
> As part of KAFKA-7503, the {{ConnectClusterStateImpl}} class was altered to 
> use a {{HerderProvider}} instance instead of a {{Herder}}. However, the 
> Connect {{RestServer}} registers REST extensions before a herder is given to 
> that {{HerderProvider}}, so any extensions that invoke, e.g., 
> {{ConnectClusterState.connector()}} in their {{register(...)}} method end up 
> in a deadlock that eventually causes Connect startup to fail with the error 
> message "Timed out waiting for herder to be initialized."
> If possible, the {{HerderProvider}} used for {{ConnectClusterStateImpl}} 
> instances given to REST extensions should be supplied with a {{Herder}} 
> before those extensions are registered. If that isn't feasible, another 
> option could be to install Connect REST extensions on a separate thread so 
> that they don't block the Connect startup process and eventual call of 
> {{HerderProvider.setHerder(...)}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOT] KIP-449: Add connector contexts to Connect worker logs (vote thread)

[Randall Hauch]

+1 (binding)

On Mon, May 6, 2019 at 3:53 PM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Great improvement for multi-tenancy.
> Thanks Randall!
>
> +1 (non-binding)
>
> Konstantine
>
> On Tue, Apr 30, 2019 at 9:18 PM Chris Egerton  wrote:
>
> > +1 (non-binding)
> >
> > Really looking forward to this. Thanks, Randall!
> >
> > On Tue, Apr 30, 2019, 20:47 Magesh Nandakumar 
> > wrote:
> >
> > > This will make connect debugging so much easier. Thanks a lot for
> driving
> > > this Randall.
> > >
> > > +1 (non-binding)
> > >
> > > Thanks,
> > > Magesh
> > >
> > > On Tue, Apr 30, 2019 at 7:19 PM Jeremy Custenborder <
> > > jcustenbor...@gmail.com>
> > > wrote:
> > >
> > > > +1 non binding
> > > >
> > > > On Mon, Apr 29, 2019 at 5:34 PM Randall Hauch 
> > wrote:
> > > > >
> > > > > I would like to start the vote for KIP-258:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-449%3A+Add+connector+contexts+to+Connect+worker+logs
> > > > >
> > > > > The KIP uses the Mapped Diagnostic Contexts (MDC) feature of SLF4J
> > API
> > > to
> > > > > add more context to log messages from within Connect workers and
> > > > connector
> > > > > implementations. This would not be enabled by default, though it
> > would
> > > be
> > > > > easy to enable within the Connect Log4J configuration.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Randall
> > > >
> > >
> >
>

Build failed in Jenkins: kafka-trunk-jdk11 #494

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-8304: Fix registration of Connect REST extensions (#6651)

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H39 (ubuntu xenial) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision cc097e909c7e5b1617565e4456f5328a826eafbc 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f cc097e909c7e5b1617565e4456f5328a826eafbc
Commit message: "KAFKA-8304: Fix registration of Connect REST extensions 
(#6651)"
 > git rev-list --no-walk 889512202c2a2cfba23e358da08747357b60d8ac # timeout=10
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
[kafka-trunk-jdk11] $ /bin/bash -xe /tmp/jenkins4429531215048775590.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.10.2/bin/gradle
/tmp/jenkins4429531215048775590.sh: line 4: 
/home/jenkins/tools/gradle/4.10.2/bin/gradle: No such file or directory
Build step 'Execute shell' marked build as failure
Recording test results
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Not sending mail to unregistered user nore...@github.com
Not sending mail to unregistered user wangg...@gmail.com

Re: [DISCUSS] KIP-461 - Improving replica fetcher behavior in case of partition failures

[Aishwarya Gune]

Hi Colin!

Whenever the thread has all of its partitions marked as failed (i.e. thread
is idle), the thread would be shut down.
The errors that are not per-partition would probably retry or behave just
as before.


On Tue, May 7, 2019 at 9:57 AM Colin McCabe  wrote:

> Hi Aishwarya,
>
> This looks like a great improvement!
>
> Will a fetcher thread exit if all of its partitions have been marked
> failed?  Or will it continue to run?
>
> After this KIP is adopted, are there any remaining situations where we
> would exit a fetcher thread?  I guess some errors are not per-partition,
> like authentication exceptions.  How will those behave?
>
> best,
> Colin
>
> On Mon, May 6, 2019, at 13:21, Aishwarya Gune wrote:
> > Hey All!
> >
> > I have created a KIP to improve the behavior of replica fetcher when
> > partition failure occurs. Please do have a look at it and let me know
> > what
> > you think.
> > KIP 461 -
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-461+-+Improve+Replica+Fetcher+behavior+at+handling+partition+failure
> >
> > --
> > Thank you,
> > Aishwarya
> >
>


-- 
Thank you,
Aishwarya

Build failed in Jenkins: kafka-trunk-jdk11 #493

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-8275; Take throttling into account when choosing least loaded 
node

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H42 (ubuntu xenial) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 889512202c2a2cfba23e358da08747357b60d8ac 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 889512202c2a2cfba23e358da08747357b60d8ac
Commit message: "KAFKA-8275; Take throttling into account when choosing least 
loaded node (#6619)"
 > git rev-list --no-walk a6d5efaf0d06f8a66350a8f1b959baf176fd482a # timeout=10
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
[kafka-trunk-jdk11] $ /bin/bash -xe /tmp/jenkins6925036939371232911.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.10.2/bin/gradle
/tmp/jenkins6925036939371232911.sh: line 4: 
/home/jenkins/tools/gradle/4.10.2/bin/gradle: No such file or directory
Build step 'Execute shell' marked build as failure
Recording test results
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Not sending mail to unregistered user nore...@github.com
Not sending mail to unregistered user wangg...@gmail.com

[jira] [Resolved] (KAFKA-8275) NetworkClient leastLoadedNode selection should consider throttled nodes

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8275.

   Resolution: Fixed
Fix Version/s: 2.3.0

> NetworkClient leastLoadedNode selection should consider throttled nodes
> ---
>
> Key: KAFKA-8275
> URL: https://issues.apache.org/jira/browse/KAFKA-8275
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.3.0
>
>
> The leastLoadedNode() function is intended to find any available node. It is 
> smart in the sense that it considers the number of inflight requests and 
> reconnect backoff, but it has not been updated to take into account client 
> throttling. If we have an available node which is not throttled, we should 
> use it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-456: Helper classes to make it simpler to write test logic with TopologyTestDriver

[John Roesler]

Thanks for the responses, Jukka!

Thanks for the reference to the javadocs in your library, but I
actually meant they should be part of the KIP document.

As a general comment, I did get that your intent is to smooth over
some rough spots with the current testing library, and that many of
your API/behavior decisions are just reflecting the underlying TTD.
What I'm wondering is whether we can take the opportunity to make a
bigger change for an even smoother usability experience.

Regarding the publishing artifact, did you mean that you're proposing
to add a separate build artifact to the Apache Kafka project, or just
that you plan to host a historically-compatible version of the library
on your own? My initial reaction is that we should just include this
stuff in the test-utils package starting at whatever version. This
gives us greater latitude to modify the underlying layers in service
of this KIP. Plus, it's nicer as a user to just have one test-support
artifact to manage.

Specific responses below:
> >InputTopic:
> >
> >1. Have you considered adding a ProducerRecord input method to the input
> >topic? This might be unnecessary, given the overloads you provide. I'm
> >wondering if it decreases the domain mismatch between TopologyTestDriver
> >and KafkaStreams, though, since in production code, you send input to the
> >app as a ProducerRecord, via a topic. Also, this might let you drop some of
> >the less mainstream overloads, like the ones for headers.
>
> Ok, so not same methods as in TopologyTestDriver:
> pipeInput(ConsumerRecord consumerRecord);
> pipeInput(List> records);
> but instead:
> pipeInput(ProducerRecord  record);
> pipeInput(List< ProducerRecord  > records);
> Which would convert ProductRecords to ConsumerRecords before piping to TTD
>
> and Drop these methods
> void pipeInput(K key, V value, Headers headers);
> void pipeInput(K key, V value, Headers headers, long timestampMs);

Yes, this is essentially what I was saying.

>
> In this case you would not able to create those with ConsumerRecordFactory,
> but needing to create  ProducerRecord   objects directly.one by one or with
> some helper methods.
>
> Generally I was hoping that ProducerRecord and ConsumerRecord would have
> inherited to same base class or implemented some kind KafkaRecord
> interface, but
> it is not the case now. So in the most of  case it would have enough to
> pass KafkaRecord instead of ProducerRecord like in OutputVerifier now.
>
> What do you think?
>
I agree it would be nice in some cases to have a symmetric superclass
to capture the common fields in Producer and Consumer records, but I
don't know if I see how that's an impediment here.

Part of what I was thinking below is that the interface you're
proposing may actually render the ConsumerRecordFactory unnecessary as
a public interface. Since we're taking a typed ProducerRecord

> >2. On the "pipeList" methods, it seems like the "start,advance" timestamp
> >approach forces me to do a little mental math, if I actually want to
> >achieve some specific timestamp per record, or to be able to verify the
> >result, given specific timestamps as input. Did you consider a
> >KeyValueTimestamp value type instead? Alternatively, if you like the
> >ProducerRecord approach, above, you could lean on that instead.
>
> This start + advance concept is coming directly from ConsumerRecordFactory.
> I don't see value of adding KeyValueTimestamp class.
>
> My own approach has been piping in the seeds with list and pipe special
> times with:
> void pipeInput(K key, V value, long timestampMs);
>
> This start + n*interval approach will fit the normal happy case scenarios
> and
> for timing relating test special cases I have used pipe single record at a
> time.
>
> And that ProducerRecord approach is one alternative, but you end up
> generating that list some special methods anyway,
> which could also pipe those in one by one.
>

Ok, my intuition may not be the best here, because I do a *lot* of
timestamp-sensitive testing. I think your experience is more typical.
Plus, it sounds like you're no opposed to adding ProducerRecord
input, which renders KeyValueTimestamp totally unnecessary.

I do understand that this "start,advance" API comes from
ConsumerRecordFactory, I'm sorry if it seemed I was being critical of
your proposal. I think that if I actually want to verify the right
thing happens with timestamps, then the ProducerRecord input would let
me do that, and for other cases, where I just need "some" timestamp,
then start+advance works fine.

> >
> >3. I wasn't clear on the semantics of the constructors that take a start
> >timestamp, but no advance time. I also wasn't clear on the semantics when
> >the constructor specifies start/advance, but then we also call the input
> >methods that specify timestamps, or start/advance timestamps. Also related,
> >what's the "default" timestamp, if no start is specified, "zero" or "now"
> >both seem reasonable. Similar with the advance, 

Re: [DISCUSSION] KIP-422: Add support for user/client configuration in the Kafka Admin Client

[Yaodong Yang]

Hello Colin, Rajini and Jun,

I think it makes sense to have new APIs defined in the AdminClient for
quota management only, as Colin described above. For the request and
response protocol, It seems like we can leverage the existing requests:
*IncrementalAlterConfigsRequest* and *DescribeConfigsRequest*. In this way,
we convert the quota management requests (set quota, describe quota and
delete quotas) to configuration changes for User resource and Client
resource (e.g. update a configuration to user 1 ). And then we send the
configuration change request to the broker side. Therefore, we will not
have any protocol changes for this KIP.

Please let me know what you think.

Thanks!
Yaodong


On Mon, Apr 15, 2019 at 12:16 PM Colin McCabe  wrote:

> Hi all,
>
> In KIP-133: Describe and Alter Configs Admin APIs, there is "future work"
> section that explains:
>
> > Future Work
> > ...
> >
>  > 2. Support for reading and updating client, user and replication
> quotas. We
>  > initially included that in the KIP, but it subsequently became apparent
>  > that a separate protocol and AdminClient API would be more appropriate.
>  > The reason is that client/user quotas can be applied on a client id,
> user
>  > or (client id, user) tuple. In the future, the hierarchy may get even
> more
>  > complicated. So, it makes sense to keeping the API simple for the
> simple
>  > cases while introducing a more sophisticated API for the more complex
> case.
>
> In other words, we deliberately didn't implement quotas through
> AlterConfigs because we felt like it the AlterConfigs API wasn't really
> complex enough to handle quotas well.
>
> I think that the original discussion was correct here -- we should have a
> special API for quotas, rather than trying to shoehorn them into the
> AlterConfigs API.
>
> For example, we could have an API like this:
>
> >
> > SetQuotasResults setQuotas(Map quotas,
> SetQuotasOptions options)
> >
> > interface QuotaTarget {
> > }
> >
> > class ClientQuotaTarget implements QuotaTarget {
> >   String clientId;
> > }
> >
> > class PrincipalQuotaTarget implements QuotaTarget {
> >   String principal;
> > }
> >
> > class ClientAndPrincipalQuotaTarget implements QuotaTarget {
> >   String clientId;
> >   String principal;
> > }
> >
> > class QuotaLimit {
> >long bytesWrittenPerSec;
> >long bytesReadPerSec;
> > }
> >
> > DescribeQuotasResults describeQuotas(QuotaTarget target,
> DescribeQuotasOptions options);
> >
> > ListQuotasResults listQuotas(ListQuotasOptions options);
> >
>
> This would avoid the need to parse text strings.  It would also make it
> really clear how to use and extend the API.
>
> best,
> Colin
>
> On Mon, Apr 8, 2019, at 05:29, Rajini Sivaram wrote:
> > Hi Jun, Yaodong,
> >
> > 21. The proposed approach sounds very hacky. User principals can contain
> > arbitrary characters. So we can't simply split `user1/clients/clientA`
> into
> > tokens using '/' as delimiter.  Internally, we sanitize names before
> > storing in ZK, but the names provided by the user are actual principals
> and
> > client-ids. I think we want to have entity names that explicitly specify
> > (type, name) as in the CLI kafka-configs.sh and allow multiple entities
> to
> > be specified together for (user, client-id). That will also enable us to
> > configure defaults in a consistent way.
> >
> > 22. Yes, that sounds reasonable.
> >
> > On Fri, Apr 5, 2019 at 11:13 PM Jun Rao  wrote:
> >
> > > Hi, Yaodong,
> > >
> > > Yes, what you proposed makes sense. A couple of more comments.
> > >
> > > 21.  Could you add those examples to the KIP wiki? It would also be
> useful
> > > to know how to set the ConfigEntry value for quotas at
> > > DefaultClientInUser, DefaultClientDefaultUser and DefaultUser level.
> > >
> > > 22. To support KIP-257, I guess we can just pass in some special string
> > > value in ConfigEntry value through alterConfig and let the customized
> > > implementation of ClientQuotaCallback parse it. It would be useful to
> > > document this. Does that sound reasonable, Rajini?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Fri, Apr 5, 2019 at 2:03 PM Yaodong Yang 
> > > wrote:
> > >
> > >> Hi Jun,
> > >>
> > >> The proposal we have right now is to directly set the quota through
> > >> existing admin client APIs. See following examples:
> > >>
> > >> Example 1: set a user quota
> > >>
> > >> AdminClient adminClient = mock(AdminClient.class);
> > >> Map configs = new HashMap();
> > >> Config config = new Config(Arrays.asList(new ConfigEntry("user1",
> > >> "producer_byte_rate=1024")));
> > >> configs.put(singletonMap(ConfigResource.USER, config));
> > >> adminClient.alterConfigs(configs);
> > >>
> > >>
> > >> Example 2: set a client id quota
> > >>
> > >> AdminClient adminClient = mock(AdminClient.class);
> > >> Map configs = new HashMap();
> > >> Config config = new Config(Arrays.asList(new ConfigEntry("client1",
> > >> "producer_byte_rate=1024")));
> > >> 

Re: [DISCUSS] 2.2.1 Bug Fix Release

[Vahid Hashemian]

Hi John,

Thanks for checking, and sorry for the delay. I'm working on it.
I'm working through some issues with the release script / my environment.
Hopefully not much longer :)

Thanks,
--Vahid

On Tue, May 7, 2019 at 10:53 AM John Roesler  wrote:

> Hi Vahid,
>
> Can you let us know the status of the release? I don't mean to
> pressure you, but I actually just had someone ask me for a status
> update on some of my bugfixes.
>
> Thanks again for driving this!
> -John
>
> On Fri, May 3, 2019 at 5:51 PM Vahid Hashemian
>  wrote:
> >
> > Hi Sophie,
> >
> > Thanks for the heads-up. Once the fix is confirmed, could you please
> create
> > a ticket for it and assign it to 2.2.1 release?
> >
> > Thanks,
> > --Vahid
> >
> > On Fri, May 3, 2019 at 3:24 PM Sophie Blee-Goldman 
> > wrote:
> >
> > > Hey Vahid,
> > >
> > > We also have another minor bug fix we just uncovered and are hoping to
> get
> > > in today although I don't think there's a ticket for it atm...just
> waiting
> > > for the build to pass.
> > >
> > > Thanks for volunteering!
> > >
> > > Cheers,
> > > Sophie
> > >
> > > On Fri, May 3, 2019 at 3:16 PM Vahid Hashemian <
> vahid.hashem...@gmail.com>
> > > wrote:
> > >
> > > > Hi John,
> > > >
> > > > Thanks for confirming.
> > > > I'll wait for final bug fix PR for this issue to get merged so we can
> > > > safely resolve the ticket. That makes it easier with the release
> script.
> > > > Hopefully, the current build passes.
> > > >
> > > > --Vahid
> > > >
> > > > On Fri, May 3, 2019 at 3:07 PM John Roesler 
> wrote:
> > > >
> > > > > Hi Vahid,
> > > > >
> > > > > The fix is merged to 2.2. The ticket isn't resolved yet, because
> the
> > > > tests
> > > > > failed on the 2.1 merge, but I think the 2.2.1 release is unblocked
> > > now.
> > > > >
> > > > > Thanks,
> > > > > -John
> > > > >
> > > > > On Fri, May 3, 2019 at 10:41 AM Vahid Hashemian <
> > > > vahid.hashem...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks for the filter fix and the heads up John.
> > > > > > I'll wait for that to go through then.
> > > > > >
> > > > > > --Vahid
> > > > > >
> > > > > > On Fri, May 3, 2019 at 8:33 AM John Roesler 
> > > wrote:
> > > > > >
> > > > > > > Thanks for volunteering, Vahid!
> > > > > > >
> > > > > > > I noticed that the "unresolved issues" filter on the plan page
> was
> > > > > still
> > > > > > > set to 2.1.1 (I fixed it).
> > > > > > >
> > > > > > > There's one blocker left:
> > > > > > https://issues.apache.org/jira/browse/KAFKA-8289
> > > > > > > ,
> > > > > > > but it's merged to trunk and we're cherry-picking to 2.2 today.
> > > > > > >
> > > > > > > Thanks again!
> > > > > > > -John
> > > > > > >
> > > > > > > On Thu, May 2, 2019 at 10:38 PM Vahid Hashemian <
> > > > > > vahid.hashem...@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > If there are no objections on the proposed plan, I'll start
> > > > preparing
> > > > > > the
> > > > > > > > first release candidate.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > > On Thu, Apr 25, 2019 at 6:27 AM Ismael Juma <
> ism...@juma.me.uk>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Vahid!
> > > > > > > > >
> > > > > > > > > On Wed, Apr 24, 2019 at 10:44 PM Vahid Hashemian <
> > > > > > > > > vahid.hashem...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I'd like to volunteer for the release manager of the
> 2.2.1
> > > bug
> > > > > fix
> > > > > > > > > release.
> > > > > > > > > > Kafka 2.2.0 was released on March 22, 2019.
> > > > > > > > > >
> > > > > > > > > > At this point, there are 29 resolved JIRA issues
> scheduled
> > > for
> > > > > > > > inclusion
> > > > > > > > > in
> > > > > > > > > > 2.2.1:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%202.2.1
> > > > > > > > > >
> > > > > > > > > > The release plan is documented here:
> > > > > > > > > >
> > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.2.1
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > > --Vahid
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Thanks!
> > > > > > --Vahid
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Thanks!
> > > > --Vahid
> > > >
> > >
> >
> >
> > --
> >
> > Thanks!
> > --Vahid
>


-- 

Thanks!
--Vahid

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

[John Roesler]

Thanks for that update. Do you mind making changes primarily on the
KIP document ? 
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List%3CT%3E+serialization+and+deserialization)

This is the design document that we have to agree on and vote for, the
PR comes later. It can be nice to have an implementation to look at,
but the KIP is the main artifact for this discussion.

With this in mind, it will help get more reviewers to look at it if
you can tidy up the KIP document so that it stands on its own. People
shouldn't have to look at any other document to understand the
motivation of the proposal, and they shouldn't have to look at a PR to
see what the public API will look like. If it helps, you can take a
look at some other recent KIPs.

Given that the list serde needs an inner serde, I agree you can't have
a zero-argument static factory method for it, but it seems you could
still have a static method:
`public static Serde> List(Serde innerSerde)`.

Thoughts?

On Tue, May 7, 2019 at 12:18 PM Development  wrote:
>
> Absolutely agree. Already pushed another commit to remove comparator 
> argument: https://github.com/apache/kafka/pull/6592 
> 
>
> Thank you for your input John! I really appreciate it.
>
> What about this point I made:
>
> 1. Since type for List serde needs to be declared before hand, I could not 
> create a static method for List Serde under 
> org.apache.kafka.common.serialization.Serdes. I addressed it in the KIP:
> P.S. Static method corresponding to ListSerde under 
> org.apache.kafka.common.serialization.Serdes (something like static public 
> Serde> List() {...} inorg.apache.kafka.common.serialization.Serdes) 
> class cannot be added because type needs to be defined beforehand. That's why 
> one needs to create List Serde in the following fashion:
> new Serdes.ListSerde(Serdes.String(), 
> Comparator.comparing(String::length));
> (can possibly be simplified by declaring import static 
> org.apache.kafka.common.serialization.Serdes.ListSerde)
>
> > On May 7, 2019, at 11:50 AM, John Roesler  wrote:
> >
> > Thanks for the reply Daniyar,
> >
> > That makes much more sense! I thought I must be missing something, but I
> > couldn't for the life of me figure it out.
> >
> > What do you think about just taking an argument, instead of for a
> > Comparator, for the Serde of the inner type? That way, the user can control
> > how exactly the inner data gets serialized, while also bounding the generic
> > parameter properly. As for the order, since the list is already in a
> > specific order, which the user themselves controls, it doesn't seem
> > strictly necessary to offer an option to sort the data during serialization.
> >
> > Thanks,
> > -John
> >
> > On Mon, May 6, 2019 at 8:47 PM Development  wrote:
> >
> >> Hi John,
> >>
> >> I’m really sorry for the confusion. I cloned that JIRA ticket from an old
> >> one about introducing UUID Serde, and I guess was too hasty while editing
> >> the copy to notice the mistake. Just edited the ticket. Sorry for any
> >> inconvenience .
> >>
> >> As per comparator, I agree. Let’s make user be responsible for
> >> implementing comparable interface. I was just thinking to make the serde a
> >> little more flexible (i.e. let user decide in which order records is going
> >> to be inserted into a change log topic).
> >>
> >> Thank you!
> >>
> >> Best,
> >> Daniyar Yeralin
> >>
> >>
> >>> On May 6, 2019, at 5:37 PM, John Roesler  wrote:
> >>>
> >>> Hi Daniyar,
> >>>
> >>> Thanks for the proposal!
> >>>
> >>> If I understand the point about the comparator, is it just to capture the
> >>> generic type parameter? If so, then anything that implements a known
> >>> interface would work just as well, right? I've been considering adding
> >>> something like the Jackson TypeReference (or similar classes in many
> >> other
> >>> projects). Would this be a good time to do it?
> >>>
> >>> Note that it's not necessary to actually require that the captured type
> >> is
> >>> Comparable (as this proposal currently does), it's just a way to make
> >> sure
> >>> there is some method that makes use of the generic type parameter, to
> >> force
> >>> the compiler to capture the type.
> >>>
> >>> Just to make sure I understand the motivation... You expressed a desire
> >> to
> >>> be able to serialize UUIDs, which I didn't follow, since there is a
> >>> built-in UUID serde: org.apache.kafka.common.serialization.Serdes#UUID,
> >> and
> >>> also, a UUID isn't a List. Did you mean that you need to use *lists of*
> >>> UUIDs?
> >>>
> >>> Thanks,
> >>> -John
> >>>
> >>> On Mon, May 6, 2019 at 11:49 AM Development  wrote:
> >>>
>  Hello,
> 
>  Starting a discussion for KIP-466 adding support for List Serde. PR is
>  created under https://github.com/apache/kafka/pull/6592 <
>  https://github.com/apache/kafka/pull/6592>
> 
>  There are two topics I would like to discuss:
>  1. Since type 

Build failed in Jenkins: kafka-trunk-jdk11 #492

[Apache Jenkins Server]

See 


Changes:

[bbejeck] KAFKA-3522: Interactive Queries must return timestamped stores (#6661)

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H33 (ubuntu xenial) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision a6d5efaf0d06f8a66350a8f1b959baf176fd482a 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a6d5efaf0d06f8a66350a8f1b959baf176fd482a
Commit message: "KAFKA-3522: Interactive Queries must return timestamped stores 
(#6661)"
 > git rev-list --no-walk 8e7b57fbcd13edcab280725bb808f232f2e60a1b # timeout=10
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
[kafka-trunk-jdk11] $ /bin/bash -xe /tmp/jenkins5083490049005478511.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.10.2/bin/gradle
/tmp/jenkins5083490049005478511.sh: line 4: 
/home/jenkins/tools/gradle/4.10.2/bin/gradle: No such file or directory
Build step 'Execute shell' marked build as failure
Recording test results
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Not sending mail to unregistered user wangg...@gmail.com

Re: [DISCUSS] 2.2.1 Bug Fix Release

[John Roesler]

Hi Vahid,

Can you let us know the status of the release? I don't mean to
pressure you, but I actually just had someone ask me for a status
update on some of my bugfixes.

Thanks again for driving this!
-John

On Fri, May 3, 2019 at 5:51 PM Vahid Hashemian
 wrote:
>
> Hi Sophie,
>
> Thanks for the heads-up. Once the fix is confirmed, could you please create
> a ticket for it and assign it to 2.2.1 release?
>
> Thanks,
> --Vahid
>
> On Fri, May 3, 2019 at 3:24 PM Sophie Blee-Goldman 
> wrote:
>
> > Hey Vahid,
> >
> > We also have another minor bug fix we just uncovered and are hoping to get
> > in today although I don't think there's a ticket for it atm...just waiting
> > for the build to pass.
> >
> > Thanks for volunteering!
> >
> > Cheers,
> > Sophie
> >
> > On Fri, May 3, 2019 at 3:16 PM Vahid Hashemian 
> > wrote:
> >
> > > Hi John,
> > >
> > > Thanks for confirming.
> > > I'll wait for final bug fix PR for this issue to get merged so we can
> > > safely resolve the ticket. That makes it easier with the release script.
> > > Hopefully, the current build passes.
> > >
> > > --Vahid
> > >
> > > On Fri, May 3, 2019 at 3:07 PM John Roesler  wrote:
> > >
> > > > Hi Vahid,
> > > >
> > > > The fix is merged to 2.2. The ticket isn't resolved yet, because the
> > > tests
> > > > failed on the 2.1 merge, but I think the 2.2.1 release is unblocked
> > now.
> > > >
> > > > Thanks,
> > > > -John
> > > >
> > > > On Fri, May 3, 2019 at 10:41 AM Vahid Hashemian <
> > > vahid.hashem...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Thanks for the filter fix and the heads up John.
> > > > > I'll wait for that to go through then.
> > > > >
> > > > > --Vahid
> > > > >
> > > > > On Fri, May 3, 2019 at 8:33 AM John Roesler 
> > wrote:
> > > > >
> > > > > > Thanks for volunteering, Vahid!
> > > > > >
> > > > > > I noticed that the "unresolved issues" filter on the plan page was
> > > > still
> > > > > > set to 2.1.1 (I fixed it).
> > > > > >
> > > > > > There's one blocker left:
> > > > > https://issues.apache.org/jira/browse/KAFKA-8289
> > > > > > ,
> > > > > > but it's merged to trunk and we're cherry-picking to 2.2 today.
> > > > > >
> > > > > > Thanks again!
> > > > > > -John
> > > > > >
> > > > > > On Thu, May 2, 2019 at 10:38 PM Vahid Hashemian <
> > > > > vahid.hashem...@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > If there are no objections on the proposed plan, I'll start
> > > preparing
> > > > > the
> > > > > > > first release candidate.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > --Vahid
> > > > > > >
> > > > > > > On Thu, Apr 25, 2019 at 6:27 AM Ismael Juma 
> > > > wrote:
> > > > > > >
> > > > > > > > Thanks Vahid!
> > > > > > > >
> > > > > > > > On Wed, Apr 24, 2019 at 10:44 PM Vahid Hashemian <
> > > > > > > > vahid.hashem...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I'd like to volunteer for the release manager of the 2.2.1
> > bug
> > > > fix
> > > > > > > > release.
> > > > > > > > > Kafka 2.2.0 was released on March 22, 2019.
> > > > > > > > >
> > > > > > > > > At this point, there are 29 resolved JIRA issues scheduled
> > for
> > > > > > > inclusion
> > > > > > > > in
> > > > > > > > > 2.2.1:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%202.2.1
> > > > > > > > >
> > > > > > > > > The release plan is documented here:
> > > > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.2.1
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > --Vahid
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Thanks!
> > > > > > > --Vahid
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Thanks!
> > > > > --Vahid
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Thanks!
> > > --Vahid
> > >
> >
>
>
> --
>
> Thanks!
> --Vahid

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

[Development]

Absolutely agree. Already pushed another commit to remove comparator argument: 
https://github.com/apache/kafka/pull/6592 


Thank you for your input John! I really appreciate it.

What about this point I made:

1. Since type for List serde needs to be declared before hand, I could not 
create a static method for List Serde under 
org.apache.kafka.common.serialization.Serdes. I addressed it in the KIP: 
P.S. Static method corresponding to ListSerde under 
org.apache.kafka.common.serialization.Serdes (something like static public 
Serde> List() {...} inorg.apache.kafka.common.serialization.Serdes) 
class cannot be added because type needs to be defined beforehand. That's why 
one needs to create List Serde in the following fashion:
new Serdes.ListSerde(Serdes.String(), 
Comparator.comparing(String::length));
(can possibly be simplified by declaring import static 
org.apache.kafka.common.serialization.Serdes.ListSerde)

> On May 7, 2019, at 11:50 AM, John Roesler  wrote:
> 
> Thanks for the reply Daniyar,
> 
> That makes much more sense! I thought I must be missing something, but I
> couldn't for the life of me figure it out.
> 
> What do you think about just taking an argument, instead of for a
> Comparator, for the Serde of the inner type? That way, the user can control
> how exactly the inner data gets serialized, while also bounding the generic
> parameter properly. As for the order, since the list is already in a
> specific order, which the user themselves controls, it doesn't seem
> strictly necessary to offer an option to sort the data during serialization.
> 
> Thanks,
> -John
> 
> On Mon, May 6, 2019 at 8:47 PM Development  wrote:
> 
>> Hi John,
>> 
>> I’m really sorry for the confusion. I cloned that JIRA ticket from an old
>> one about introducing UUID Serde, and I guess was too hasty while editing
>> the copy to notice the mistake. Just edited the ticket. Sorry for any
>> inconvenience .
>> 
>> As per comparator, I agree. Let’s make user be responsible for
>> implementing comparable interface. I was just thinking to make the serde a
>> little more flexible (i.e. let user decide in which order records is going
>> to be inserted into a change log topic).
>> 
>> Thank you!
>> 
>> Best,
>> Daniyar Yeralin
>> 
>> 
>>> On May 6, 2019, at 5:37 PM, John Roesler  wrote:
>>> 
>>> Hi Daniyar,
>>> 
>>> Thanks for the proposal!
>>> 
>>> If I understand the point about the comparator, is it just to capture the
>>> generic type parameter? If so, then anything that implements a known
>>> interface would work just as well, right? I've been considering adding
>>> something like the Jackson TypeReference (or similar classes in many
>> other
>>> projects). Would this be a good time to do it?
>>> 
>>> Note that it's not necessary to actually require that the captured type
>> is
>>> Comparable (as this proposal currently does), it's just a way to make
>> sure
>>> there is some method that makes use of the generic type parameter, to
>> force
>>> the compiler to capture the type.
>>> 
>>> Just to make sure I understand the motivation... You expressed a desire
>> to
>>> be able to serialize UUIDs, which I didn't follow, since there is a
>>> built-in UUID serde: org.apache.kafka.common.serialization.Serdes#UUID,
>> and
>>> also, a UUID isn't a List. Did you mean that you need to use *lists of*
>>> UUIDs?
>>> 
>>> Thanks,
>>> -John
>>> 
>>> On Mon, May 6, 2019 at 11:49 AM Development  wrote:
>>> 
 Hello,
 
 Starting a discussion for KIP-466 adding support for List Serde. PR is
 created under https://github.com/apache/kafka/pull/6592 <
 https://github.com/apache/kafka/pull/6592>
 
 There are two topics I would like to discuss:
 1. Since type for List serve needs to be declared before hand, I could
>> not
 create a static method for List Serde under
 org.apache.kafka.common.serialization.Serdes. I addressed it in the KIP:
 P.S. Static method corresponding to ListSerde under
 org.apache.kafka.common.serialization.Serdes (something like static
>> public
 Serde> List() {...}
>> inorg.apache.kafka.common.serialization.Serdes)
 class cannot be added because type needs to be defined beforehand.
>> That's
 why one needs to create List Serde in the following fashion:
 new Serdes.ListSerde(Serdes.String(),
 Comparator.comparing(String::length));
 (can possibly be simplified by declaring import static
 org.apache.kafka.common.serialization.Serdes.ListSerde)
 
 2. @miguno Michael G. Noll  is questioning
 whether I need to pass a comparator to ListDeserializer. This certainly
>> is
 not required. Feel free to add your input:
 https://github.com/apache/kafka/pull/6592#discussion_r281152067
 
 Thank you!
 
 Best,
 Daniyar Yeralin
 
> On May 6, 2019, at 11:59 AM, Daniyar Yeralin (JIRA) 
 wrote:
> 
> Daniyar Yeralin created KAFKA-8326:

Re: [VOTE] KIP-460: Admin Leader Election RPC

[Jose Armando Garcia Sancio]

Thanks Gwen for the clarification!

On Mon, May 6, 2019 at 5:03 PM Gwen Shapira  wrote:

> All committer votes are binding, so you have binding +1 from Colin, Jason
> and myself - which is just the 3 you need for the KIP to be accepted.
> Mickael added non-binding community support, which is great signal as well.
>
> On Mon, May 6, 2019 at 4:39 PM Jose Armando Garcia Sancio <
> jsan...@confluent.io> wrote:
>
> > I am closing the voting. KIP is accepted with:
> >
> > +1 (binding): Colin McCabe
> > +1 (non-binding): Jason Gustafson, Gwen Shapira, Mickael Maison
> >
> > Thanks!
> >
> > On Fri, May 3, 2019 at 9:59 AM Mickael Maison 
> > wrote:
> >
> > > +1 (non binding)
> > > Thanks for the KIP
> > >
> > > On Thu, May 2, 2019 at 11:02 PM Colin McCabe 
> wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > thanks, Jose.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Wed, May 1, 2019, at 14:44, Jose Armando Garcia Sancio wrote:
> > > > > Hi all,
> > > > >
> > > > > I would like to start the voting for KIP-460:
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-460%3A+Admin+Leader+Election+RPC
> > > > >
> > > > > The thread discussion is here:
> > > > > https://www.mail-archive.com/dev@kafka.apache.org/msg97226.html
> > > > >
> > > > > Thanks!
> > > > > -Jose
> > > > >
> > >
> >
> >
> > --
> > -Jose
> >
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter  | blog
> 
>


-- 
-Jose

Manual Partition Trigger

[Harsh Bansal]

Hi,

I would like to know if there is any provision by which we can initiate a 
re-partition once all the partition are assigned a consumer. This will help in 
scaling consumers as with re-partition of partition, there would be more 
partitions available and hence more consumer can be attached. Using this we can 
achieve scaling.

Sent from Mail for Windows 10

[VOTE] KIP-458: Connector Client Config Override Policy

[Magesh Nandakumar]

Hi All,

I would like to start a vote on
https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy

The discussion thread can be found here
.

Thanks,
Magesh

Re: [VOTE] KIP-458: Connector Client Config Override Policy

[Chris Egerton]

Hi Magesh,

This looks great! Very excited to see these changes finally coming to
Connect.
+1 (non-binding)

Cheers,

Chris

On Tue, May 7, 2019 at 9:51 AM Magesh Nandakumar 
wrote:

> Hi All,
>
> I would like to start a vote on
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy
>
> The discussion thread can be found here
> .
>
> Thanks,
> Magesh
>

Re: [DISCUSS] KIP-461 - Improving replica fetcher behavior in case of partition failures

[Colin McCabe]

Hi Aishwarya,

This looks like a great improvement!

Will a fetcher thread exit if all of its partitions have been marked failed?  
Or will it continue to run?

After this KIP is adopted, are there any remaining situations where we would 
exit a fetcher thread?  I guess some errors are not per-partition, like 
authentication exceptions.  How will those behave?

best,
Colin

On Mon, May 6, 2019, at 13:21, Aishwarya Gune wrote:
> Hey All!
> 
> I have created a KIP to improve the behavior of replica fetcher when
> partition failure occurs. Please do have a look at it and let me know 
> what
> you think.
> KIP 461 -
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-461+-+Improve+Replica+Fetcher+behavior+at+handling+partition+failure
> 
> -- 
> Thank you,
> Aishwarya
>

[VOTE] KIP-458: Connector Client Config Override Policy

[Magesh Nandakumar]

Hi All,

I would like to start a vote on
https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy

The discussion thread can be found here
.

Thanks,
Magesh

On Tue, May 7, 2019 at 9:35 AM Magesh Nandakumar 
wrote:

> I have addressed all the outstanding discussion points and I believe we
> have consensus on the design. Thanks, everyone for the feedback. I will
> start a VOTE thread on this KIP.
>
> On Mon, May 6, 2019 at 10:13 PM Magesh Nandakumar 
> wrote:
>
>> Randall,
>>
>> Thanks a lot for the suggestions. I have incorporated the comments in the
>> KIP.
>>
>> Thanks,
>> Magesh
>>
>> On Mon, May 6, 2019 at 6:52 PM Randall Hauch  wrote:
>>
>>> Thanks, Magesh. I do have a few pretty minor suggestions.
>>>
>>> 1) Define a bit more clearly in the "Proposed Changes" whether the
>>> configs
>>> passed to the validate method via the ConnectorClientConfigRequest object
>>> have or do not have the prefixes.
>>> 2) Specify more clearly in (or around) the table which is the default
>>> policy. Currently the Ignore policy "Behavior" just mentions that it's
>>> the
>>> current behavior, but I think it would help that it is described as the
>>> default for the property.
>>>
>>> Otherwise, this looks good to me.
>>>
>>> Best regards,
>>>
>>> Randall
>>>
>>> On Mon, May 6, 2019 at 8:12 PM Magesh Nandakumar 
>>> wrote:
>>>
>>> > Konstantine,
>>> >
>>> > Thanks a lot for your feedback on the KIP. I have incorporated the
>>> feedback
>>> > using generics for Class. I have also updated the KIP to handle the
>>> default
>>> > value per Randall's suggestion. Let me know if you have any questions.
>>> >
>>> > Thanks,
>>> > Magesh
>>> >
>>> >
>>> > On Mon, May 6, 2019 at 1:58 PM Konstantine Karantasis <
>>> > konstant...@confluent.io> wrote:
>>> >
>>> > > Thanks for the KIP Magesh, it's quite useful towards the goals for
>>> more
>>> > > general multi-tenancy in Connect.
>>> > >
>>> > > Couple of comments from me too:
>>> > >
>>> > > I think the fact that the default policy is 'null' (no
>>> implementation)
>>> > > should be mentioned on the table next to the available
>>> implementations.
>>> > > Currently the KIP says: 'In addition to the default implementation,
>>> ..."
>>> > > but this is not very accurate because there is no concrete default
>>> > > implementation. Just special handling of 'null' in
>>> > > 'connector.client.config.policy'
>>> > >
>>> > > Regarding passing the overrides to the connector 'configure' method,
>>> I
>>> > feel
>>> > > it wouldn't hurt to pass them, but I also agree that leaving this
>>> out at
>>> > > the moment is the safest option.
>>> > >
>>> > > Since the interfaces and classes are listed in the KIP, I'd like to
>>> note
>>> > > that Class is used as a raw type in field and return value
>>> declarations.
>>> > We
>>> > > should use the generic type instead.
>>> > >
>>> > > Thanks for this improvement proposal!
>>> > > Konstantine
>>> > >
>>> > > On Mon, May 6, 2019 at 11:11 AM Magesh Nandakumar <
>>> mage...@confluent.io>
>>> > > wrote:
>>> > >
>>> > > > Randall,
>>> > > >
>>> > > > I was wondering if you had any thoughts on the above alternatives
>>> to
>>> > deal
>>> > > > with a default policy.  If it's possible, I would like to finalize
>>> the
>>> > > > discussions and start a vote.
>>> > > > Let me know your thoughts.
>>> > > >
>>> > > > Thanks,
>>> > > > Magesh
>>> > > >
>>> > > > On Tue, Apr 30, 2019 at 8:46 PM Magesh Nandakumar <
>>> > mage...@confluent.io>
>>> > > > wrote:
>>> > > >
>>> > > > > Randall,
>>> > > > >
>>> > > > > The approach to return the to override configs could possibly
>>> make it
>>> > > > > cumbersome to implement a custom policy. This is a new
>>> configuration
>>> > > and
>>> > > > if
>>> > > > > you don't explicitly set it the existing behavior remains as-is.
>>> Like
>>> > > > > Chris, I also preferred this approach for the sake of
>>> simplicity.  If
>>> > > not
>>> > > > > for the default `null` I would prefer to fall back to using
>>> `Ignore`
>>> > > > which
>>> > > > > is a misnomer to the interface spec but still gets the job done
>>> via
>>> > > > > instanceOf checks. The other options I could think of are as
>>> below:-
>>> > > > >
>>> > > > >- have an enforcePolicy() method in the interface which by
>>> default
>>> > > > >returns true and the Ignore implementation could return false
>>> > > > >- introduce another worker config
>>> allow.connector.config.overrides
>>> > > > >with a default value of false and the default policy can be
>>> None
>>> > > > >
>>> > > > > Let me know what you think.
>>> > > > >
>>> > > > > Thanks
>>> > > > > Magesh
>>> > > > >
>>> > > > > On Tue, Apr 30, 2019 at 6:52 PM Randall Hauch 
>>> > > wrote:
>>> > > > >
>>> > > > >> Thanks, Chris. I still think it's strange to have a non-policy,
>>> > since
>>> > > > >> there's now special behavior for when the policy is not
>>> specified.
>>> > > > >>

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

[Magesh Nandakumar]

I have addressed all the outstanding discussion points and I believe we
have consensus on the design. Thanks, everyone for the feedback. I will
start a VOTE thread on this KIP.

On Mon, May 6, 2019 at 10:13 PM Magesh Nandakumar 
wrote:

> Randall,
>
> Thanks a lot for the suggestions. I have incorporated the comments in the
> KIP.
>
> Thanks,
> Magesh
>
> On Mon, May 6, 2019 at 6:52 PM Randall Hauch  wrote:
>
>> Thanks, Magesh. I do have a few pretty minor suggestions.
>>
>> 1) Define a bit more clearly in the "Proposed Changes" whether the configs
>> passed to the validate method via the ConnectorClientConfigRequest object
>> have or do not have the prefixes.
>> 2) Specify more clearly in (or around) the table which is the default
>> policy. Currently the Ignore policy "Behavior" just mentions that it's the
>> current behavior, but I think it would help that it is described as the
>> default for the property.
>>
>> Otherwise, this looks good to me.
>>
>> Best regards,
>>
>> Randall
>>
>> On Mon, May 6, 2019 at 8:12 PM Magesh Nandakumar 
>> wrote:
>>
>> > Konstantine,
>> >
>> > Thanks a lot for your feedback on the KIP. I have incorporated the
>> feedback
>> > using generics for Class. I have also updated the KIP to handle the
>> default
>> > value per Randall's suggestion. Let me know if you have any questions.
>> >
>> > Thanks,
>> > Magesh
>> >
>> >
>> > On Mon, May 6, 2019 at 1:58 PM Konstantine Karantasis <
>> > konstant...@confluent.io> wrote:
>> >
>> > > Thanks for the KIP Magesh, it's quite useful towards the goals for
>> more
>> > > general multi-tenancy in Connect.
>> > >
>> > > Couple of comments from me too:
>> > >
>> > > I think the fact that the default policy is 'null' (no implementation)
>> > > should be mentioned on the table next to the available
>> implementations.
>> > > Currently the KIP says: 'In addition to the default implementation,
>> ..."
>> > > but this is not very accurate because there is no concrete default
>> > > implementation. Just special handling of 'null' in
>> > > 'connector.client.config.policy'
>> > >
>> > > Regarding passing the overrides to the connector 'configure' method, I
>> > feel
>> > > it wouldn't hurt to pass them, but I also agree that leaving this out
>> at
>> > > the moment is the safest option.
>> > >
>> > > Since the interfaces and classes are listed in the KIP, I'd like to
>> note
>> > > that Class is used as a raw type in field and return value
>> declarations.
>> > We
>> > > should use the generic type instead.
>> > >
>> > > Thanks for this improvement proposal!
>> > > Konstantine
>> > >
>> > > On Mon, May 6, 2019 at 11:11 AM Magesh Nandakumar <
>> mage...@confluent.io>
>> > > wrote:
>> > >
>> > > > Randall,
>> > > >
>> > > > I was wondering if you had any thoughts on the above alternatives to
>> > deal
>> > > > with a default policy.  If it's possible, I would like to finalize
>> the
>> > > > discussions and start a vote.
>> > > > Let me know your thoughts.
>> > > >
>> > > > Thanks,
>> > > > Magesh
>> > > >
>> > > > On Tue, Apr 30, 2019 at 8:46 PM Magesh Nandakumar <
>> > mage...@confluent.io>
>> > > > wrote:
>> > > >
>> > > > > Randall,
>> > > > >
>> > > > > The approach to return the to override configs could possibly
>> make it
>> > > > > cumbersome to implement a custom policy. This is a new
>> configuration
>> > > and
>> > > > if
>> > > > > you don't explicitly set it the existing behavior remains as-is.
>> Like
>> > > > > Chris, I also preferred this approach for the sake of
>> simplicity.  If
>> > > not
>> > > > > for the default `null` I would prefer to fall back to using
>> `Ignore`
>> > > > which
>> > > > > is a misnomer to the interface spec but still gets the job done
>> via
>> > > > > instanceOf checks. The other options I could think of are as
>> below:-
>> > > > >
>> > > > >- have an enforcePolicy() method in the interface which by
>> default
>> > > > >returns true and the Ignore implementation could return false
>> > > > >- introduce another worker config
>> allow.connector.config.overrides
>> > > > >with a default value of false and the default policy can be
>> None
>> > > > >
>> > > > > Let me know what you think.
>> > > > >
>> > > > > Thanks
>> > > > > Magesh
>> > > > >
>> > > > > On Tue, Apr 30, 2019 at 6:52 PM Randall Hauch 
>> > > wrote:
>> > > > >
>> > > > >> Thanks, Chris. I still think it's strange to have a non-policy,
>> > since
>> > > > >> there's now special behavior for when the policy is not
>> specified.
>> > > > >>
>> > > > >> Perhaps the inability for a policy implementation to represent
>> the
>> > > > >> existing
>> > > > >> behavior suggests that the policy interface isn't quite right.
>> Could
>> > > the
>> > > > >> policy's "validate" method take the overrides that were supplied
>> and
>> > > > >> return
>> > > > >> the overrides that should be passed to the connector, yet still
>> > > throwing
>> > > > >> an
>> > > > >> exception if any supplied overrides are not allowed. Then the
>> > > 

Build failed in Jenkins: kafka-trunk-jdk11 #491

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-8131; Move --version implementation into CommandLineUtils (#6481)

[jason] MINOR: Remove unused field in `ListenerConnectionQuota`

[wangguoz] MINOR: MetricsIntegrationTest should set 
StreamsConfig.STATE_DIR_CONFIG

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H30 (ubuntu xenial) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 8e7b57fbcd13edcab280725bb808f232f2e60a1b 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8e7b57fbcd13edcab280725bb808f232f2e60a1b
Commit message: "MINOR: MetricsIntegrationTest should set 
StreamsConfig.STATE_DIR_CONFIG (#6687)"
 > git rev-list --no-walk 407bcdf78e06f83f2b358d2cbd96aed348a5c28f # timeout=10
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
[kafka-trunk-jdk11] $ /bin/bash -xe /tmp/jenkins4722060420697827122.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.10.2/bin/gradle
/tmp/jenkins4722060420697827122.sh: line 4: 
/home/jenkins/tools/gradle/4.10.2/bin/gradle: No such file or directory
Build step 'Execute shell' marked build as failure
Recording test results
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_10_2_HOME=/home/jenkins/tools/gradle/4.10.2
Not sending mail to unregistered user wangg...@gmail.com

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

[John Roesler]

Thanks for the reply Daniyar,

That makes much more sense! I thought I must be missing something, but I
couldn't for the life of me figure it out.

What do you think about just taking an argument, instead of for a
Comparator, for the Serde of the inner type? That way, the user can control
how exactly the inner data gets serialized, while also bounding the generic
parameter properly. As for the order, since the list is already in a
specific order, which the user themselves controls, it doesn't seem
strictly necessary to offer an option to sort the data during serialization.

Thanks,
-John

On Mon, May 6, 2019 at 8:47 PM Development  wrote:

> Hi John,
>
> I’m really sorry for the confusion. I cloned that JIRA ticket from an old
> one about introducing UUID Serde, and I guess was too hasty while editing
> the copy to notice the mistake. Just edited the ticket. Sorry for any
> inconvenience .
>
> As per comparator, I agree. Let’s make user be responsible for
> implementing comparable interface. I was just thinking to make the serde a
> little more flexible (i.e. let user decide in which order records is going
> to be inserted into a change log topic).
>
> Thank you!
>
> Best,
> Daniyar Yeralin
>
>
> > On May 6, 2019, at 5:37 PM, John Roesler  wrote:
> >
> > Hi Daniyar,
> >
> > Thanks for the proposal!
> >
> > If I understand the point about the comparator, is it just to capture the
> > generic type parameter? If so, then anything that implements a known
> > interface would work just as well, right? I've been considering adding
> > something like the Jackson TypeReference (or similar classes in many
> other
> > projects). Would this be a good time to do it?
> >
> > Note that it's not necessary to actually require that the captured type
> is
> > Comparable (as this proposal currently does), it's just a way to make
> sure
> > there is some method that makes use of the generic type parameter, to
> force
> > the compiler to capture the type.
> >
> > Just to make sure I understand the motivation... You expressed a desire
> to
> > be able to serialize UUIDs, which I didn't follow, since there is a
> > built-in UUID serde: org.apache.kafka.common.serialization.Serdes#UUID,
> and
> > also, a UUID isn't a List. Did you mean that you need to use *lists of*
> > UUIDs?
> >
> > Thanks,
> > -John
> >
> > On Mon, May 6, 2019 at 11:49 AM Development  wrote:
> >
> >> Hello,
> >>
> >> Starting a discussion for KIP-466 adding support for List Serde. PR is
> >> created under https://github.com/apache/kafka/pull/6592 <
> >> https://github.com/apache/kafka/pull/6592>
> >>
> >> There are two topics I would like to discuss:
> >> 1. Since type for List serve needs to be declared before hand, I could
> not
> >> create a static method for List Serde under
> >> org.apache.kafka.common.serialization.Serdes. I addressed it in the KIP:
> >> P.S. Static method corresponding to ListSerde under
> >> org.apache.kafka.common.serialization.Serdes (something like static
> public
> >> Serde> List() {...}
> inorg.apache.kafka.common.serialization.Serdes)
> >> class cannot be added because type needs to be defined beforehand.
> That's
> >> why one needs to create List Serde in the following fashion:
> >> new Serdes.ListSerde(Serdes.String(),
> >> Comparator.comparing(String::length));
> >> (can possibly be simplified by declaring import static
> >> org.apache.kafka.common.serialization.Serdes.ListSerde)
> >>
> >> 2. @miguno Michael G. Noll  is questioning
> >> whether I need to pass a comparator to ListDeserializer. This certainly
> is
> >> not required. Feel free to add your input:
> >> https://github.com/apache/kafka/pull/6592#discussion_r281152067
> >>
> >> Thank you!
> >>
> >> Best,
> >> Daniyar Yeralin
> >>
> >>> On May 6, 2019, at 11:59 AM, Daniyar Yeralin (JIRA) 
> >> wrote:
> >>>
> >>> Daniyar Yeralin created KAFKA-8326:
> >>> --
> >>>
> >>>Summary: Add List Serde
> >>>Key: KAFKA-8326
> >>>URL: https://issues.apache.org/jira/browse/KAFKA-8326
> >>>Project: Kafka
> >>> Issue Type: Improvement
> >>> Components: clients, streams
> >>>   Reporter: Daniyar Yeralin
> >>>
> >>>
> >>> I propose adding serializers and deserializers for the java.util.List
> >> class.
> >>>
> >>> I have many use cases where I want to set the key of a Kafka message to
> >> be a UUID. Currently, I need to turn UUIDs into strings or byte arrays
> and
> >> use their associated Serdes, but it would be more convenient to
> serialize
> >> and deserialize UUIDs directly.
> >>>
> >>> I believe there are many use cases where one would want to have a List
> >> serde. Ex. [
> >>
> https://stackoverflow.com/questions/41427174/aggregate-java-objects-in-a-list-with-kafka-streams-dsl-windows
> ],
> >> [
> >>
> https://stackoverflow.com/questions/46365884/issue-with-arraylist-serde-in-kafka-streams-api
> >> ]
> >>>
> >>>
> >>>
> >>> KIP Link: [

Re: [VOTE] KIP-434: Dead replica fetcher and log cleaner metrics

[Harsha]

Thanks for the kip. LGTM +1.

-Harsha

On Mon, Apr 29, 2019, at 8:14 AM, Viktor Somogyi-Vass wrote:
> Hi Jason,
> 
> I too agree this is more of a problem in older versions and therefore we
> could backport it. Were you thinking of any specific versions? I guess the
> 2.x and 1.x versions are definitely targets here but I was thinking that we
> might not want to further.
> 
> Viktor
> 
> On Mon, Apr 29, 2019 at 12:55 AM Stanislav Kozlovski 
> wrote:
> 
> > Thanks for the work done, Viktor! +1 (non-binding)
> >
> > I strongly agree with Jason that this monitoring-focused KIP is worth
> > porting back to older versions. I am sure users will find it very useful
> >
> > Best,
> > Stanislav
> >
> > On Fri, Apr 26, 2019 at 9:38 PM Jason Gustafson 
> > wrote:
> >
> > > Thanks, that works for me. +1
> > >
> > > By the way, we don't normally port KIPs to older releases, but I wonder
> > if
> > > it's worth making an exception here. From recent experience, it tends to
> > be
> > > the older versions that are more prone to fetcher failures. Thoughts?
> > >
> > > -Jason
> > >
> > > On Fri, Apr 26, 2019 at 5:18 AM Viktor Somogyi-Vass <
> > > viktorsomo...@gmail.com>
> > > wrote:
> > >
> > > > Let me have a second thought, I'll just add the clientId instead to
> > > follow
> > > > the convention, so it'll change DeadFetcherThreadCount but with the
> > > > clientId tag.
> > > >
> > > > On Fri, Apr 26, 2019 at 11:29 AM Viktor Somogyi-Vass <
> > > > viktorsomo...@gmail.com> wrote:
> > > >
> > > > > Hi Jason,
> > > > >
> > > > > Yea I think it could make sense. In this case I would rename the
> > > > > DeadFetcherThreadCount to DeadReplicaFetcherThreadCount and introduce
> > > the
> > > > > metric you're referring to as DeadLogDirFetcherThreadCount.
> > > > > I'll update the KIP to reflect this.
> > > > >
> > > > > Viktor
> > > > >
> > > > > On Thu, Apr 25, 2019 at 8:07 PM Jason Gustafson 
> > > > > wrote:
> > > > >
> > > > >> Hi Viktor,
> > > > >>
> > > > >> This looks good. Just one question I had is whether we may as well
> > > cover
> > > > >> the log dir fetchers as well.
> > > > >>
> > > > >> Thanks,
> > > > >> Jason
> > > > >>
> > > > >>
> > > > >> On Thu, Apr 25, 2019 at 7:46 AM Viktor Somogyi-Vass <
> > > > >> viktorsomo...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Folks,
> > > > >> >
> > > > >> > This thread sunk a bit but I'd like to bump it hoping to get some
> > > > >> feedback
> > > > >> > and/or votes.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Viktor
> > > > >> >
> > > > >> > On Thu, Mar 28, 2019 at 8:47 PM Viktor Somogyi-Vass <
> > > > >> > viktorsomo...@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Sorry, the end of the message cut off.
> > > > >> > >
> > > > >> > > So I tried to be consistent with the convention in LogManager,
> > > hence
> > > > >> the
> > > > >> > > hyphens and in AbstractFetcherManager, hence the camel case. It
> > > > would
> > > > >> be
> > > > >> > > nice though to decide with one convention across the whole
> > > project,
> > > > >> > however
> > > > >> > > it requires a major refactor (especially for the components that
> > > > >> leverage
> > > > >> > > metrics for monitoring).
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Viktor
> > > > >> > >
> > > > >> > > On Thu, Mar 28, 2019 at 8:44 PM Viktor Somogyi-Vass <
> > > > >> > > viktorsomo...@gmail.com> wrote:
> > > > >> > >
> > > > >> > >> Hi Dhruvil,
> > > > >> > >>
> > > > >> > >> Thanks for the feedback and the vote. I fixed the typo in the
> > > KIP.
> > > > >> > >> The naming is interesting though. Unfortunately kafka overall
> > is
> > > > not
> > > > >> > >> consistent in metric naming but at least I tried to be
> > consistent
> > > > >> among
> > > > >> > the
> > > > >> > >> other metrics used in LogManager
> > > > >> > >>
> > > > >> > >> On Thu, Mar 28, 2019 at 7:32 PM Dhruvil Shah <
> > > dhru...@confluent.io
> > > > >
> > > > >> > >> wrote:
> > > > >> > >>
> > > > >> > >>> Thanks for the KIP, Viktor! This is a useful addition. +1
> > > overall.
> > > > >> > >>>
> > > > >> > >>> Minor nits:
> > > > >> > >>> > I propose to add three gauge: DeadFetcherThreadCount for the
> > > > >> fetcher
> > > > >> > >>> threads, log-cleaner-dead-thread-count for the log cleaner.
> > > > >> > >>> I think you meant two instead of three.
> > > > >> > >>>
> > > > >> > >>> Also, would it make sense to name these metrics consistency,
> > > > >> something
> > > > >> > >>> like
> > > > >> > >>> `log-cleaner-dead-thread-count` and
> > > > >> > `replica-fetcher-dead-thread-count`?
> > > > >> > >>>
> > > > >> > >>> Thanks,
> > > > >> > >>> Dhruvil
> > > > >> > >>>
> > > > >> > >>> On Thu, Mar 28, 2019 at 11:27 AM Viktor Somogyi-Vass <
> > > > >> > >>> viktorsomo...@gmail.com> wrote:
> > > > >> > >>>
> > > > >> > >>> > Hi All,
> > > > >> > >>> >
> > > > >> > >>> > I'd like to start a vote on KIP-434.
> > > > >> > >>> > This basically would add a metrics to count dead threads in
> > > > >> > >>> > 

Re: [VOTE] KIP-453: Add close() method to RocksDBConfigSetter

[Guozhang Wang]

+1 （binding）

On Mon, May 6, 2019 at 6:44 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> +1 (non-binding). Thanks for the KIP.
>
> On Tue, May 7, 2019 at 3:50 AM Bill Bejeck  wrote:
>
> > Thanks for the KIP Sophie.
> >
> > +1(binding)
> >
> > On Mon, May 6, 2019 at 4:51 PM John Roesler  wrote:
> >
> > > Thanks, Sophie, I reviewed the KIP, and I agree this is the best /
> > > only-practical approach.
> > >
> > > +1 (non-binding)
> > >
> > > On Mon, May 6, 2019 at 2:23 PM Matthias J. Sax 
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > >
> > > >
> > > > On 5/6/19 6:28 PM, Sophie Blee-Goldman wrote:
> > > > > Hi all,
> > > > >
> > > > > I'd like to call for a vote on a minor KIP that adds a close()
> method
> > > to
> > > > > the RocksDBConfigSetter interface of Streams.
> > > > >
> > > > > Link:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-453%3A+Add+close%28%29+method+to+RocksDBConfigSetter
> > > > >
> > > > > This is important for users who have created RocksOBjects in
> > > > > RocksDBConfigSetter#setConfig to avoid leaking off-heap memory
> > > > >
> > > > > Thanks!
> > > > > Sophie
> > > > >
> > > >
> > > >
> > >
> >
>


-- 
-- Guozhang

Re: [VOTE] KIP-419: Safely notify Kafka Connect SourceTask is stopped

[Andrew Schofield]

It should be possible to get this small KIP into Kafka 2.3 but it's proving 
hard to get some binding votes. Please could the committers with background in 
Kafka Connect take a look and vote this week.

Thanks,
Andrew Schofield

On 25/04/2019, 16:11, "Andrew Schofield"  wrote:

I'd like to encourage some more votes on KIP-419. It's a pretty small KIP 
to make it easier to handle resource clean up in Kafka Connect SourceTasks.

Currently only +2 non-binding.

KIP: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-419%3A+Safely+notify+Kafka+Connect+SourceTask+is+stopped
PR: https://github.com/apache/kafka/pull/6551

Thanks,
Andrew Schofield
IBM Event Streams

On 15/04/2019, 15:59, "Edoardo Comar"  wrote:

Thanks Andrew.

+1 (non-binding)

--

Edoardo Comar

IBM Event Streams
IBM UK Ltd, Hursley Park, SO21 2JN




From:   Mickael Maison 
To: dev 
Date:   10/04/2019 10:14
Subject:Re: [VOTE] KIP-419: Safely notify Kafka Connect 
SourceTask 
is stopped



+1 (non-binding)
Thanks for the KIP!

On Mon, Apr 8, 2019 at 8:07 PM Andrew Schofield
 wrote:
>
> Hi,
> I’d like to begin the voting thread for KIP-419. This is a minor KIP 
to 
add a new stopped() method to the SourceTask interface in Kafka 
Connect. 
Its purpose is to give the task a safe opportunity to clean up its 
resources, in the knowledge that this is the final call to the task.
>
> KIP: 

https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D419-253A-2BSafely-2Bnotify-2BKafka-2BConnect-2BSourceTask-2Bis-2Bstopped%26d%3DDwIFaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DEzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ%26m%3DvBvXztcRTgKwMpQ54ziN_GoOo0_fHSvTEMoXwQABvfs%26s%3DParyN6mWVuOGJR7kA84NOshRJA2LAK6htiD2gqf-h_M%26edata=02%7C01%7C%7Cffb96440b207419a7fcd08d6c1b2ed17%7C84df9e7fe9f640afb435%7C1%7C0%7C636909371551269648sdata=WJ4ZgMEIUTl83QXBIm%2Fn3ekWWabpZTIWsPbQOQGR6J8%3Dreserved=0=

> PR: 

https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__github.com_apache_kafka_pull_6551%26d%3DDwIFaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DEzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ%26m%3DvBvXztcRTgKwMpQ54ziN_GoOo0_fHSvTEMoXwQABvfs%26s%3DR_udYap1tpd83ISv1Rh0TY6ttH6RuEIwQ0KwOFMB3zU%26edata=02%7C01%7C%7Cffb96440b207419a7fcd08d6c1b2ed17%7C84df9e7fe9f640afb435%7C1%7C0%7C636909371551279653sdata=4LrcZVcLG9acQm7rjZz8%2F9MO2UeKK08242TW1SSJdlE%3Dreserved=0=

> JIRA: 

https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__issues.apache.org_jira_browse_KAFKA-2D7841%26d%3DDwIFaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DEzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ%26m%3DvBvXztcRTgKwMpQ54ziN_GoOo0_fHSvTEMoXwQABvfs%26s%3D5WqDQPU2J8yAxRXsjOgydtzJSE8yQCoB7qX0TtQyHA0%26edata=02%7C01%7C%7Cffb96440b207419a7fcd08d6c1b2ed17%7C84df9e7fe9f640afb435%7C1%7C0%7C636909371551279653sdata=JxRlNBP9FmuCmSVHIj6T30eT3uMPijbHi%2B%2F1QsUfA5U%3Dreserved=0=

>
> Thanks,
> Andrew Schofield
> IBM




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with 
number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 
3AU

[jira] [Created] (KAFKA-8330) Separate Replica and Reassignment Throttling

[Viktor Somogyi-Vass (JIRA)]

Viktor Somogyi-Vass created KAFKA-8330:
--

 Summary: Separate Replica and Reassignment Throttling
 Key: KAFKA-8330
 URL: https://issues.apache.org/jira/browse/KAFKA-8330
 Project: Kafka
  Issue Type: Improvement
  Components: core, replication
Affects Versions: 2.3.0
Reporter: Viktor Somogyi-Vass
Assignee: Viktor Somogyi-Vass


Currently Kafka doesn't separate reassignment related replication from ISR 
replication. That is dangerous because if replication is throttled during 
reassignment and some replicas fall out of ISR they could get further behind 
while reassignment is also slower. We need a method that throttles reassignment 
related replication only.

A KIP is in progress for this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8329) Flaky Test LogOffsetTest#testEmptyLogsGetOffsets

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-8329:
--

 Summary: Flaky Test LogOffsetTest#testEmptyLogsGetOffsets
 Key: KAFKA-8329
 URL: https://issues.apache.org/jira/browse/KAFKA-8329
 Project: Kafka
  Issue Type: Bug
  Components: core, unit tests
Affects Versions: 2.3.0
Reporter: Matthias J. Sax
 Fix For: 2.3.0


[https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/4325/testReport/junit/kafka.server/LogOffsetTest/testEmptyLogsGetOffsets/]
{quote}org.scalatest.exceptions.TestFailedException: Partition [kafka-,0] 
metadata not propagated after 15000 ms at 
org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530) at 
org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529) at 
org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389) at 
org.scalatest.Assertions.fail(Assertions.scala:1091) at 
org.scalatest.Assertions.fail$(Assertions.scala:1087) at 
org.scalatest.Assertions$.fail(Assertions.scala:1389) at 
kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:788) at 
kafka.utils.TestUtils$.waitUntilMetadataIsPropagated(TestUtils.scala:877) at 
kafka.utils.TestUtils$.$anonfun$createTopic$3(TestUtils.scala:320) at 
kafka.utils.TestUtils$.$anonfun$createTopic$3$adapted(TestUtils.scala:319) at 
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) at 
scala.collection.immutable.Range.foreach(Range.scala:158) at 
scala.collection.TraversableLike.map(TraversableLike.scala:237) at 
scala.collection.TraversableLike.map$(TraversableLike.scala:230) at 
scala.collection.AbstractTraversable.map(Traversable.scala:108) at 
kafka.utils.TestUtils$.createTopic(TestUtils.scala:319) at 
kafka.integration.KafkaServerTestHarness.createTopic(KafkaServerTestHarness.scala:125)
 at 
kafka.server.LogOffsetTest.testEmptyLogsGetOffsets(LogOffsetTest.scala:141){quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Cleaning up command line tools argument parsing a little

[Sönke Liebau]

Hi Colin,

thanks!
While I personally don't like the current command line tools I
realistically think we'll be stuck with them for a while yet, so a cleanup
might make sense.
So I'll start looking into that.

Regarding a central entrypoint, that would indeed be brilliant and I'd love
to work on that, but I currently have enough other open things to look at,
so I won't draw that one to myself as well for now.

But I'll keep it in mind for when some time frees up :)

Best regards,
Sönke



Colin McCabe  schrieb am Di., 7. Mai 2019, 00:56:

> On Mon, May 6, 2019, at 10:21, Sönke Liebau wrote:
> > Hi Colin,
> >
> > it was my intention to keep the structure of the commands mostly intact
> > while doing the refactoring - if that is possible, have not really
> checked
> > yet to be honest.
> >
> > But what I wanted to try and do is recreate the current parsing with
> > argparse as much as possible. And in the process simply adding synonyms,
> > for example make the kafka-console-producer understand a
> > bootstrap-parameter in addition to broker-list.
> > There is a bit of custom logic about which parameters go together etc. in
> > the current classes, so output may look different here and there, but in
> > principle I do believe that it should be possible to recreate the current
> > structure.
>
> Sounds like a good idea.  Thanks for the clarification.
>
> >
> > If there is an appetite for a new, hadoop-like entrypoint anyway, then
> all
> > of this might be "wasted" effort, or rather effort better spent though,
> you
> > are right.
>
> I don't think anyone is working on a new entry point right now -- or if
> they are, they haven't said anything yet :)
>
> I just wanted to mention it as a possible approach in case you wanted to
> do a bigger project.
>
> best,
> Colin
>
> >
> > Best regards,
> > Sönke
> >
> >
> >
> > On Mon, May 6, 2019 at 7:13 PM Colin McCabe  wrote:
> >
> > > Hi Sönke,
> > >
> > > #2 is a bit tough because people have come to rely on the way the
> commands
> > > are structured right now.
> > >
> > > If we want to make big changes, it might be easier just to create a
> > > separate tool and deprecate the old one(s).  One thing we've talked
> about
> > > doing in the past is creating a single entry point for all the tool
> > > functionality, kind of like hadoop did with the "hadoop" command  Or
> git
> > > with the "git" command, etc.  Then we could deprecate the standalone
> > > commands and remove them after enough time had passed-- kind of like
> the
> > > old consumer.
> > >
> > > On the other hand, a more incremental change would be standardizing
> flags
> > > a bit.  So for example, at least setting it up so that there is a
> standard
> > > way of supplying bootstrap brokers, etc.  We could keep the old flags
> > > around for a while as variants to ease the transition.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote:
> > > > Hi Colin,
> > > >
> > > > I totally agree! Especially the differently named bootstrap server
> > > options
> > > > have been annoying me a long time.
> > > >
> > > > I'd propose a two-step approach:
> > > > 1. Add new default options objects similar to CommandLineUtils and
> > > > CommandDefaultOptions (based on argparse4j) but in the clients
> project,
> > > as
> > > > this is referenced by all command line tools as far as I can tell
> > > > 2. Refactor tools one by one to use these new helper classes (and
> thus
> > > > argparse) and add standardized synonyms for parameters as necessary
> > > >
> > > > I think for step 1 we can get away with no KIP, as this doesn't
> change
> > > any
> > > > public interfaces or behavior.
> > > > Step 2 probably needs a KIP as we are adding new parameters? We can
> pick
> > > up
> > > > KIP-14 again for that I think. A lot of work has been done on that
> > > already.
> > > >
> > > > Does that sound useful to everybody?
> > > >
> > > > Best regards,
> > > > Sönke
> > > >
> > > >
> > > > On Thu, Apr 18, 2019 at 1:44 AM Colin McCabe 
> wrote:
> > > >
> > > > > If we are going to standardize on one argument parsing library, it
> > > should
> > > > > certainly be argparse4j, I think.
> > > > >  argparse4j is simply a better argument parsing library with
> support
> > > for
> > > > > more features.  One example is mutually exclusive options.
> argparse4j
> > > > > supports this with MutuallyExclusiveGroup.  jopt doesn't support
> this,
> > > so
> > > > > when it is needed, we have to add extra code to manually check that
> > > > > mutually exclusive options are not set.
> > > > >
> > > > > argparse4j also has subcommands.  If you want something like "git
> add"
> > > > > with some set of flags, and "git remove" with another, you can do
> this
> > > with
> > > > > argparse4j, but not with jopt.  This would be very helpful for
> > > clearing up
> > > > > confusion in a lot of our shell scripts which have accumulated
> dozens
> > > of
> > > > > arguments, most of which are only relevant to a 

Re: [DISCUSS] KIP-456: Helper classes to make it simpler to write test logic with TopologyTestDriver

[Jukka Karvanen]

Thanks John for feedback.

General comment first, which affect many of the individual answers.
My current approach has been abstracting the current functionality of
TopologyTestDriver and ConsumerRecordFactory
and that way the Contructors and pipe-methods signatures of TestInputTopic are
replicated from ConsumerRecordFactory.
Main addition are:
-constructor versions with serde instead of serializer / deserializer
-TestInputTopic possibility to feed in Value list instead of only
KeyValueList
-TestOutputTopic methods to unwrap  ProducerRecord and return collections
of unwrapped key values and values

The JavaDoc of current implementation can be found:
https://jukkakarvanen.github.io/kafka-streams-test-topics/
The classes and methods are identical even the package name is different.
Also these JavaDoc comments are replicating a lot of descriptions from
existing class methods JavaDocs..

I was planning to publish these classes as separate artifact so these can
be used also with older Kafka versions.
In my test with Kafka version 1.1, only the method including Headers
failed, all other functionality was working without modifications.


>InputTopic:
>
>1. Have you considered adding a ProducerRecord input method to the input
>topic? This might be unnecessary, given the overloads you provide. I'm
>wondering if it decreases the domain mismatch between TopologyTestDriver
>and KafkaStreams, though, since in production code, you send input to the
>app as a ProducerRecord, via a topic. Also, this might let you drop some of
>the less mainstream overloads, like the ones for headers.

Ok, so not same methods as in TopologyTestDriver:
pipeInput(ConsumerRecord consumerRecord);
pipeInput(List> records);
but instead:
pipeInput(ProducerRecord  record);
pipeInput(List< ProducerRecord  > records);
Which would convert ProductRecords to ConsumerRecords before piping to TTD

and Drop these methods
void pipeInput(K key, V value, Headers headers);
void pipeInput(K key, V value, Headers headers, long timestampMs);

In this case you would not able to create those with ConsumerRecordFactory,
but needing to create  ProducerRecord   objects directly.one by one or with
some helper methods.

Generally I was hoping that ProducerRecord and ConsumerRecord would have
inherited to same base class or implemented some kind KafkaRecord
interface, but
it is not the case now. So in the most of  case it would have enough to
pass KafkaRecord instead of ProducerRecord like in OutputVerifier now.

What do you think?

>2. On the "pipeList" methods, it seems like the "start,advance" timestamp
>approach forces me to do a little mental math, if I actually want to
>achieve some specific timestamp per record, or to be able to verify the
>result, given specific timestamps as input. Did you consider a
>KeyValueTimestamp value type instead? Alternatively, if you like the
>ProducerRecord approach, above, you could lean on that instead.

This start + advance concept is coming directly from ConsumerRecordFactory.
I don't see value of adding KeyValueTimestamp class.

My own approach has been piping in the seeds with list and pipe special
times with:
void pipeInput(K key, V value, long timestampMs);

This start + n*interval approach will fit the normal happy case scenarios
and
for timing relating test special cases I have used pipe single record at a
time.

And that ProducerRecord approach is one alternative, but you end up
generating that list some special methods anyway,
which could also pipe those in one by one.

>
>3. I wasn't clear on the semantics of the constructors that take a start
>timestamp, but no advance time. I also wasn't clear on the semantics when
>the constructor specifies start/advance, but then we also call the input
>methods that specify timestamps, or start/advance timestamps. Also related,
>what's the "default" timestamp, if no start is specified, "zero" or "now"
>both seem reasonable. Similar with the advance, "1ms" and "0ms" both seem
>reasonable defaults.

This is also based on the implementation of ConsumerRecordFactory, which
seems to
set the base time with System.currentTimeMillis() if not set,advanceMs is 0
and
you can overwrite these timestamp providing it with the method call.

>
>OutputTopic:
>
>4. Tentatively, ProducerRecord seems like a strange output type, since as a
>user, I'm "consuming" the results. How about using ConsumerRecord instead?

This is also directly from TTD. Also for this that KafkaRecord concept
would be better, but would
require alignment with existing classes, too.

>
>5. We have methods above for producing with specific timestamps, but none
>for observing the timestamps. How can we strengthen the symmetry?

My own experience with testing with TTD.
Those timestamp are important when feeding in the data, so can test for
example
timewindows and joins with left side or right side arriving first.

For the result verification, I have not so far checked the Kafka timestamps.
Timewindow related information is 

Re: [DISCUSS] KIP-440: Extend Connect Converter to support headers

[sapiensy]

Hi Randall,

Sorry, I was away. Just started a vote a few minutes ago.

On 2019/05/06 17:46:23, Randall Hauch  wrote: 
> Have we started a vote on this? I don't see the separate "[VOTE]" thread.
> 
> On Mon, Apr 29, 2019 at 6:19 PM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
> 
> > Thanks Yaroslav, this KIP LGTM now too!
> >
> > To give some context regarding my previous comment: headers in Connect
> > would probably have followed a similar approach if default methods in
> > interfaces could be used at the time. But we could not have assumed java 8
> > or later yet in the AK version that Connect headers were added, so, I
> > believe, that led to two different converter interfaces.
> >
> > Thanks for the nicely written KIP!
> > Konstantine
> >
> >
> >
> > On Mon, Apr 29, 2019 at 3:39 PM Randall Hauch  wrote:
> >
> > > Thanks for the update. Yes, IMO this KIP is ready for a vote.
> > >
> > > On Fri, Apr 26, 2019 at 12:15 AM sapie...@gmail.com 
> > > wrote:
> > >
> > > > Hi Randall, Konstantine,
> > > >
> > > > I've updated the KIP to reflect the details we discussed here. Let me
> > > know
> > > > if it looks good and we can go to the voting phase.
> > > >
> > > > Thanks!
> > > >
> > > > On 2019/04/22 21:07:31, Randall Hauch  wrote:
> > > > > I think it would be helpful to clarify this in the KIP, just so that
> > > > > readers are aware that the headers will be the raw header bytes that
> > > are
> > > > > the same as what is in the Kafka record.
> > > > >
> > > > > The alternative I was referring to is exposing the Connect `Headers`
> > > > > interface, which is different.
> > > > >
> > > > > On Mon, Apr 22, 2019 at 1:45 PM sapie...@gmail.com <
> > sapie...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Konstantine, Randall,
> > > > > >
> > > > > > As you can see in the updated Converter interface, it always
> > operates
> > > > on
> > > > > > `org.apache.kafka.common.header.Headers`.
> > > > > >
> > > > > > WorkerSinkTask simply uses Kafka message headers and passes them to
> > > the
> > > > > > `toConnectData` method.
> > > > > >
> > > > > > WorkerSourceTask leverages header converter to extract
> > RecordHeaders,
> > > > > > which implements Headers interface. Then RecordHeaders are passed
> > to
> > > > the
> > > > > > `fromConnectData` method.
> > > > > >
> > > > > > So header converter is used as a way to get headers when converting
> > > > data
> > > > > > from internal Connect format to Kafka messages (cause there is no
> > > > other way
> > > > > > to get the headers in this case).
> > > > > >
> > > > > > I can add this to the KIP if it's helpful.
> > > > > >
> > > > > > Randall, what is the alternative approach you're referring to?
> > > > > >
> > > > > > On 2019/04/22 18:09:24, Randall Hauch  wrote:
> > > > > > > Konstantine raises a good point. Which `Headers` is being
> > > referenced
> > > > in
> > > > > > the
> > > > > > > API? The Connect `org.apache.kafka.connect.header.Headers` would
> > > > > > correspond
> > > > > > > to what was already deserialized by the `HeaderConverter` or what
> > > > will
> > > > > > yet
> > > > > > > be serialized by the `HeaderConverter`. Alternatively, the
> > common `
> > > > > > > org.apache.kafka.common.header.Headers` would correspond to the
> > raw
> > > > > > header
> > > > > > > pairs from the underlying Kafka record.
> > > > > > >
> > > > > > > So, we probably want to be a bit more specific, and also mention
> > > > why. And
> > > > > > > we probably want to mention the other approach in the rejected
> > > > > > alternatives.
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Randall
> > > > > > >
> > > > > > > On Mon, Apr 22, 2019 at 11:59 AM Konstantine Karantasis <
> > > > > > > konstant...@confluent.io> wrote:
> > > > > > >
> > > > > > > > Thanks for the KIP Yaroslav!
> > > > > > > >
> > > > > > > > Apologies for the late comment. However, after reading the KIP
> > > it's
> > > > > > still
> > > > > > > > not very clear to me what happens to the existing
> > > > > > > > HeaderConverter interface and what's the expectation for
> > existing
> > > > code
> > > > > > > > implementing this interface out there.
> > > > > > > >
> > > > > > > > Looking at the PR I see that the existing code is leaving the
> > > > existing
> > > > > > > > connect headers conversion unaffected. I'd expect by reading
> > the
> > > > KIP to
> > > > > > > > understand what's the interplay of the current proposal with
> > the
> > > > > > existing
> > > > > > > > implementation of KIP-145 that introduced headers in Connect.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Konstantine
> > > > > > > >
> > > > > > > > On Mon, Apr 22, 2019 at 9:07 AM Randall Hauch <
> > rha...@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for updating the KIP. It looks good to me, and since
> > > there
> > > > > > haven't
> > > > > > > > > been any other issue mentioned in this month-long thread,
> > it's
> > > > > > 

[jira] [Created] (KAFKA-8328) Kafka smooth expansion

[ChenLin (JIRA)]

ChenLin created KAFKA-8328:
--

 Summary: Kafka smooth expansion
 Key: KAFKA-8328
 URL: https://issues.apache.org/jira/browse/KAFKA-8328
 Project: Kafka
  Issue Type: Improvement
  Components: clients, core
Affects Versions: 0.10.2.0
Reporter: ChenLin
 Fix For: 0.10.2.0
 Attachments: DiskUtil.png, producerP999.png

When expanding the kafka cluster, the new follower will read the data from the 
earliest offset. This can result in a large amount of historical data being 
read from the disk, putting a lot of pressure on the disk and affecting the 
performance of the kafka service, for example, the producer write latency will 
increase. In general, kafka's original expansion mechanism has the following 
problems:
  1. The new follower will put a lot of pressure on the disk;
  2. Causes the producer write latency to increase;
  3. Causes the consumer read latency to increase;
   In order to solve these problems, we have proposed a solution for smooth 
expansion. The main idea of the scheme is that the newly added follower reads 
data from the HW position, and when the newly added follower reads the data to 
a certain time threshold or data size threshold, the follower enters the ISR 
queue. . Since the new follower reads data from the HW location, most of the 
data read is in the operating system's cache, so it does not put pressure on 
the disk and does not affect the performance of the kafka service, thus solving 
the above problems. problem.
   In order to illustrate the problems of the original expansion scheme, we 
have done some tests, and there are corresponding test charts in the attachment.

!image-2019-05-07-14-12-41-314.png!         !image-2019-05-07-14-12-03-005.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[VOTE] KIP-440: Extend Connect Converter to support headers

[Yaroslav Tkachenko]

Hi everyone,

I'd like to start a vote for KIP-440: Extend Connect Converter to support
headers (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-440%3A+Extend+Connect+Converter+to+support+headers
)

Discussion:
https://lists.apache.org/thread.html/1fc1e3d2cddd8311d3db7c98f0d09a1a137ca4b20d1f3c8ab203a855@%3Cdev.kafka.apache.org%3E

Thanks!