Re: [DISCUSS] Apache Kafka 2.6.1 release

[Ismael Juma]

Hi Mickael,

Any updates on this release?

Ismael

On Thu, Oct 1, 2020 at 7:40 AM Mickael Maison 
wrote:

> Hi,
>
> I'd like to volunteer to be the release manager for the next bugfix
> release, 2.6.1.
> I created the release plan on the wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.6.1
>
> Thanks
>

Jenkins build is back to normal : Kafka » kafka-trunk-jdk11 #194

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #193

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10664: Delete existing checkpoint when writing empty offsets 
(#9534)


--
[...truncated 3.45 MB...]
org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@11a98cc4, 
timestamped = false, caching = false, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@a68e52e, 
timestamped = false, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@a68e52e, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@62625ddb, 
timestamped = false, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@62625ddb, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@3c905d41, 
timestamped = false, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@3c905d41, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@618229bb, 
timestamped = false, caching = true, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@618229bb, 
timestamped = false, caching = true, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@2c8ef130, 
timestamped = false, caching = true, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@2c8ef130, 
timestamped = false, caching = true, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@5cf2fdff, 
timestamped = false, caching = true, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@5cf2fdff, 
timestamped = false, caching = true, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@62b6c975, 
timestamped = false, caching = true, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@62b6c975, 
timestamped = false, caching = true, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@1209eb69, 
timestamped = false, caching = false, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@1209eb69, 
timestamped = false, caching = false, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@49558f69, 
timestamped = false, caching = false, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@49558f69, 
timestamped = false, caching = false, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@2038eb9, 
timestamped = false, caching = false, logging = fal

[jira] [Created] (KAFKA-10668) Avoid deserialization on second hop for request forwarding

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-10668:
---

 Summary: Avoid deserialization on second hop for request forwarding
 Key: KAFKA-10668
 URL: https://issues.apache.org/jira/browse/KAFKA-10668
 Project: Kafka
  Issue Type: Sub-task
Reporter: Boyang Chen


Right now on forwarding broker we would deserialize the response and serialize 
it again to respond to the client. It should be able to keep the response data 
sealed and send back to the client to save some CPU cost.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10667) Add timeout for forwarding requests

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-10667:
---

 Summary: Add timeout for forwarding requests
 Key: KAFKA-10667
 URL: https://issues.apache.org/jira/browse/KAFKA-10667
 Project: Kafka
  Issue Type: Sub-task
Reporter: Boyang Chen


It makes sense to handle timeout for forwarding request coming from the client, 
instead of retry indefinitely. We could either use the api timeout, or a 
customized timeout hook which could be defined by different request types.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10666) Kafka doesn't used keystore / key / truststore passwords for named SSL connections

[Jason (Jira)]

Jason created KAFKA-10666:
-

 Summary: Kafka doesn't used keystore / key / truststore passwords 
for named SSL connections
 Key: KAFKA-10666
 URL: https://issues.apache.org/jira/browse/KAFKA-10666
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 2.6.0, 2.5.0
 Environment: kafka in an openjdk-11 docker container, the client java 
application is in an alpine container. zookeeper in a separate container. 
Reporter: Jason


When configuring named listener SSL connections with ssl key and keystore with 
passwords including listener.name.ourname.ssl.key.password, 
listener.name.ourname.ssl.keystore.password, and 
listener.name.ourname.ssl.truststore.password via via the AdminClient the 
settings are not used and the setting is not accepted if the default 
ssl.key.password or ssl.keystore.password are not set.  We configure all 
keystore and truststore values for the named listener in a single batch using 
incrementalAlterConfigs. Additionally, when ssl.keystore.password is set to the 
value of our keystore password the keystore is loaded for SSL communication 
without issue, however if ssl.keystore.password is incorrect and 
listener.name.ourname.keystore.password is correct, we are unable to load the 
keystore with bad password errors.  It appears that only the default 
ssl.xxx.password settings are used. This setting is immutable as when we 
attempt to set it we get an error indicating that the listener.name. setting 
can be set. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10665) Flaky Test StreamTableJoinTopologyOptimizationIntegrationTest.shouldDoStreamTableJoinWithDifferentNumberOfPartitions[Optimization = all]

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-10665:
--

 Summary: Flaky Test 
StreamTableJoinTopologyOptimizationIntegrationTest.shouldDoStreamTableJoinWithDifferentNumberOfPartitions[Optimization
 = all]
 Key: KAFKA-10665
 URL: https://issues.apache.org/jira/browse/KAFKA-10665
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: A. Sophie Blee-Goldman


{code:java}
java.nio.file.DirectoryNotEmptyException: 
/tmp/kafka-13241964730537515637/app-StreamTableJoinTopologyOptimizationIntegrationTestshouldDoStreamTableJoinWithDifferentNumberOfPartitions_Optimization___all_
at 
java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:246)
at 
java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
at java.base/java.nio.file.Files.delete(Files.java:1146)
at 
org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:869)
at 
org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:839)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2822)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2876)
at org.apache.kafka.common.utils.Utils.delete(Utils.java:839)
at org.apache.kafka.common.utils.Utils.delete(Utils.java:825)
at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.purgeLocalStreamsState(IntegrationTestUtils.java:151)
at 
org.apache.kafka.streams.integration.StreamTableJoinTopologyOptimizationIntegrationTest.whenShuttingDown(StreamTableJoinTopologyOptimizationIntegrationTest.java:122)
{code}
https://github.com/apache/kafka/pull/9515/checks?check_run_id=1333753280



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-584: Versioning scheme for features

[Kowshik Prakasam]

Hi all,

I wanted to let you know that I have made the following small change to the
`kafka-features` CLI tool description in the KIP-584 write up. The purpose
is to ensure the design is compatible with post KIP-500 world. I have
eliminated the facility in Admin#describeFeatures API to be able to
optionally send a describeFeatures request to the controller. This facility
was originally seen useful for (1) debugability and (2) slightly better
consistency guarantees in the CLI tool that reads features before updating
them. But in hindsight it now poses a hindrance to post KIP-500 world where
no client would be able to access the controller directly. So, when looking
at cost vs benefit, this facility does not feel useful enough and therefore
I've removed it. We can discuss if it becomes necessary in the future, and
implement a suitable solution.

The corresponding PR containing this change is
https://github.com/apache/kafka/pull/9536 .

Please let me know if you have any questions or concerns.


Cheers,
Kowshik


On Thu, Oct 15, 2020 at 10:17 AM Jun Rao  wrote:

> Hi, Kowshik,
>
> Thanks for the update. Those changes look good to me.
>
> Jun
>
> On Tue, Oct 13, 2020 at 4:50 PM Kowshik Prakasam 
> wrote:
>
> > Hi all,
> >
> > I wanted to let you know that I have made the following minor changes to
> > the `kafka-features` CLI tool description in the KIP-584 write up. The
> > purpose is to ensure the design is correct for a few things which came up
> > during implementation:
> >
> > 1. The CLI tool now produces a tab-formatted output instead of JSON. This
> > aligns with the type of format produced by other admin CLI tools of
> Kafka,
> > ex: `kafka-topics`.
> > 2. Whenever feature updates are performed, the output of the CLI tool
> shows
> > the result of each feature update that was applied.
> > 3. The CLI tool accepts an optional argument `--dry-run` which lets the
> > user preview the feature updates before applying them.
> >
> > The following section of the KIP has been updated with the above changes:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP584:Versioningschemeforfeatures-Toolingsupport
> >
> > Please let me know if you have any questions.
> >
> >
> > Cheers,
> > Kowshik
> >
> >
> > On Thu, Oct 8, 2020 at 1:12 AM Kowshik Prakasam 
> > wrote:
> >
> > > Hi Jun,
> > >
> > > This is a very good point. I have updated the feature version
> deprecation
> > > section mentioning the same:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP584:Versioningschemeforfeatures-Featureversiondeprecation
> > > .
> > >
> > > Thank you for the suggestion.
> > >
> > >
> > > Cheers,
> > > Kowshik
> > >
> > >
> > > On Tue, Oct 6, 2020 at 5:30 PM Jun Rao  wrote:
> > >
> > >> Hi, Kowshik,
> > >>
> > >> Thanks for the follow up. Both look good to me.
> > >>
> > >> For 2, it would be useful to also add that an admin should make sure
> > that
> > >> no clients are using a deprecated feature version (e.g. using the
> client
> > >> version metric) before deploying a release that deprecates it.
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >> On Tue, Oct 6, 2020 at 3:46 PM Kowshik Prakasam <
> kpraka...@confluent.io
> > >
> > >> wrote:
> > >>
> > >> > Hi Jun,
> > >> >
> > >> > I have added the following details in the KIP-584 write up:
> > >> >
> > >> > 1. Deployment, IBP deprecation and avoidance of double rolls. This
> > >> section
> > >> > talks about the various phases of work that would be required to use
> > >> this
> > >> > KIP to eventually avoid Broker double rolls in the cluster (whenever
> > IBP
> > >> > values are advanced). Link to section:
> > >> >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP584:Versioningschemeforfeatures-Deployment,IBPdeprecationandavoidanceofdoublerolls
> > >> > .
> > >> >
> > >> > 2. Feature version deprecation. This section explains the idea for
> > >> feature
> > >> > version deprecation (using highest supported feature min version)
> > which
> > >> you
> > >> > had proposed during code review:
> > >> >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP584:Versioningschemeforfeatures-Featureversiondeprecation
> > >> > .
> > >> >
> > >> > Please let me know if you have any questions.
> > >> >
> > >> >
> > >> > Cheers,
> > >> > Kowshik
> > >> >
> > >> >
> > >> > On Tue, Sep 29, 2020 at 11:07 AM Jun Rao  wrote:
> > >> >
> > >> > > Hi, Kowshik,
> > >> > >
> > >> > > Thanks for the update. Regarding enabling a single rolling restart
> > in
> > >> the
> > >> > > future, could we sketch out a bit how this will work by treating
> IBP
> > >> as a
> > >> > > feature? For example, IBP currently uses the release version and
> > this
> > >> KIP
> > >> > > uses an integer for versions. How do we bridge the gap between the
> > >> two?
> > >> > > Does min

Re: [DISCUSS] Apache Kafka 2.7.0 release

[Bill Bejeck]

Hi Sophie,

Thanks for the update.

-Bill

Jenkins build is back to normal : Kafka » kafka-trunk-jdk15 #217

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #192

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Add missing DESCRIBE_QUORUM ApiKey in 
AbstractRequest.parseRequest (#9537)


--
[...truncated 3.45 MB...]
org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@79000e69, 
timestamped = false, caching = true, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@46eb2c, timestamped 
= false, caching = true, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@46eb2c, timestamped 
= false, caching = true, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@6f39e96b, 
timestamped = false, caching = true, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@6f39e96b, 
timestamped = false, caching = true, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@4cdfed4a, 
timestamped = false, caching = true, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@4cdfed4a, 
timestamped = false, caching = true, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@35d5d1fe, 
timestamped = false, caching = true, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@35d5d1fe, 
timestamped = false, caching = true, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@2fd43e3, 
timestamped = false, caching = false, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@2fd43e3, 
timestamped = false, caching = false, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@22af8066, 
timestamped = false, caching = false, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@22af8066, 
timestamped = false, caching = false, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@11a98cc4, 
timestamped = false, caching = false, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@11a98cc4, 
timestamped = false, caching = false, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@a68e52e, 
timestamped = false, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@a68e52e, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@62625ddb, 
timestamped = false, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@62625ddb, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@3c905d41, 
timestamped = false, caching = false, logging = false] STARTED

Jenkins build is back to normal : Kafka » kafka-2.7-jdk8 #45

[Apache Jenkins Server]

See