[jira] [Resolved] (KAFKA-9684) Add support for SNI names in SSL request

[Badai Aqrandista (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Badai Aqrandista resolved KAFKA-9684.
-
Resolution: Not A Problem

Java 7 should include SNI names by default.

> Add support for SNI names in SSL request
> 
>
> Key: KAFKA-9684
> URL: https://issues.apache.org/jira/browse/KAFKA-9684
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Badai Aqrandista
>Priority: Minor
>
> When running Kafka cluster with SSL security behind HA Proxy, we need the 
> client to send SSL packets with SNI name extension [1]. This will allow HA 
> Proxy to forward the request to the relevant broker behind it (passthrough).
> Java 7 and higher supports this by adding SNIHostName [2] to SSLParameters 
> [3]. 
> [1] https://www.ietf.org/rfc/rfc6066.txt
> [2] https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SNIHostName.html
> [3] 
> https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SSLParameters.html#setServerNames-java.util.List-



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9685) Solve Set concatenation perf issue in AclAuthorizer

[Jiao Zhang (Jira)]

Jiao Zhang created KAFKA-9685:
-

 Summary: Solve Set concatenation perf issue in AclAuthorizer
 Key: KAFKA-9685
 URL: https://issues.apache.org/jira/browse/KAFKA-9685
 Project: Kafka
  Issue Type: Improvement
  Components: security
Affects Versions: 1.1.0
Reporter: Jiao Zhang


In version 1.1, 
https://github.com/apache/kafka/blob/71b1e19fc60b5e1f9bba33025737ec2b7fb1c2aa/core/src/main/scala/kafka/security/auth/SimpleAclAuthorizer.scala#L110
the logic for checking acls is preparing a merged acl Set with 
'acls = getAcls(new Resource(resource.resourceType, Resource.WildCardResource)) 
++ getAcls(resource);' and then pass it as aclMatch's parameter.
We found scala's Set '++' operation is very slow for example in the case that 
the Set on right hand of '++' has more than 100 entries.
And the bad performance of '++' is due to iterating every entry of the Set on 
right hand of '++' in which the calculation of HashCode seems heavy.
The performance of 'authorize' is important as each request delivered to broker 
goes through the logic, that's the reason we can't leave it as-is although the 
change for this proposal seems trivial.

Here is the approach. We propose to solve this issue by introducing a new class 
'AclSets' which takes multiple Sets as parameters and do 'find' against them 
one by one.
``` 
class AclSets(sets: Set[Acl]*) {
  def find(p: Acl => Boolean): Option[Acl] = sets.flatMap(_.find(p)).headOption
  def isEmpty: Boolean = !sets.exists(_.nonEmpty)
}
 ``` 
This approach avoids the Set '++' operation, and thus outperforms a lot 
compared to old '++' logic.

The benchmark result(we did the test with kafka version 1.1) shows notable 
difference under the condition:
1. set on left consists of 60 entries
2. set of right consists of 30 entries
3. search for absent entry (so that all entries are iterated)

Benchmark Results is as following.

Mode                                             Cnt    Score   Error   Units
ScalaSetConcatination.Set thrpt 3 281.974 ± 140.029 ops/ms
ScalaSetConcatination.AclSets thrpt 3 887.426 ± 40.261 ops/ms

As the upstream also use the similar ++ operation, 
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/security/authorizer/AclAuthorizer.scala#L360
we think it's necessary to fix this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Subject: [VOTE] 2.4.1 RC0

[Gwen Shapira]

+1 (binding)

Verified signatures, built jars from source, quickstart passed and local unit 
tests all passed.

Thank you for the release Bill!

Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

On Sat, Mar 07, 2020 at 8:15 PM, Vahid Hashemian < vahid.hashem...@gmail.com > 
wrote:

> 
> 
> 
> +1 (binding)
> 
> 
> 
> Verified signature, built from source, and ran quickstart successfully
> (using openjdk version "11.0.6"). I also ran unit tests locally which
> resulted in a few flaky tests for which there are already open Jiras:
> 
> 
> 
> ReassignPartitionsClusterTest.shouldMoveSinglePartitionWithinBroker
> ConsumerBounceTest.testCloseDuringRebalance
> 
> 
> 
> ConsumerBounceTest.testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize
> PlaintextEndToEndAuthorizationTest.testNoConsumeWithDescribeAclViaAssign
> 
> 
> 
> SaslClientsWithInvalidCredentialsTest.testManualAssignmentConsumerWithAuthenticationFailure
> SaslMultiMechanismConsumerTest.testCoordinatorFailover
> 
> 
> 
> Thanks for running the release Bill.
> 
> 
> 
> Regards,
> --Vahid
> 
> 
> 
> On Fri, Mar 6, 2020 at 9:20 AM Colin McCabe < cmccabe@ apache. org (
> cmcc...@apache.org ) > wrote:
> 
> 
>> 
>> 
>> +1 (binding)
>> 
>> 
>> 
>> Checked the git hash and branch, looked at the docs a bit. Ran quickstart
>> (although not the connect or streams parts). Looks good.
>> 
>> 
>> 
>> best,
>> Colin
>> 
>> 
>> 
>> On Fri, Mar 6, 2020, at 07:31, David Arthur wrote:
>> 
>> 
>>> 
>>> 
>>> +1 (binding)
>>> 
>>> 
>>> 
>>> Download kafka_2.13-2.4.1 and verified signature, ran quickstart,
>>> everything looks good.
>>> 
>>> 
>>> 
>>> Thanks for running this release, Bill!
>>> 
>>> 
>>> 
>>> -David
>>> 
>>> 
>>> 
>>> On Wed, Mar 4, 2020 at 6:06 AM Eno Thereska < eno. thereska@ gmail. com (
>>> eno.there...@gmail.com ) >
>>> 
>>> 
>> 
>> 
>> 
>> wrote:
>> 
>> 
>>> 
 
 
 Hi Bill,
 
 
 
 I built from source and ran unit and integration tests. They passed. There
 was a large number of skipped tests, but I'm assuming that is intentional.
 
 
 
 
 Cheers
 Eno
 
 
 
 On Tue, Mar 3, 2020 at 8:42 PM Eric Lalonde < eric@ autonomic. ai (
 e...@autonomic.ai ) > wrote:
 
 
> 
> 
> Hi,
> 
> 
> 
> I ran:
> $
> 
> 
 
 
>>> 
>>> 
>> 
>> 
>> 
>> https:/ / github. com/ elalonde/ kafka/ blob/ master/ bin/ verify-kafka-rc.
>> sh ( https://github.com/elalonde/kafka/blob/master/bin/verify-kafka-rc.sh )
>> 
>> 
>> 
>>> 
 
 
 < https:/ / github. com/ elalonde/ kafka/ blob/ master/ bin/ 
 verify-kafka-rc.
 sh ( https://github.com/elalonde/kafka/blob/master/bin/verify-kafka-rc.sh )
 >
 2.4.1 https:/ / home. apache. org/ ~bbejeck/ kafka-2. 4. 1-rc0 (
 https://home.apache.org/~bbejeck/kafka-2.4.1-rc0 ) < https:/ / home. 
 apache.
 org/ ~bbejeck/ kafka-2. 4. 1-rc0 (
 https://home.apache.org/~bbejeck/kafka-2.4.1-rc0 ) >
 
 
> 
> 
> All checksums and signatures are good and all unit and integration
> 
> 
 
 
>>> 
>>> 
>> 
>> 
>> 
>> tests
>> 
>> 
>>> 
 
 
 that were executed passed successfully.
 
 
> 
> 
> - Eric
> 
> 
>> 
>> 
>> On Mar 2, 2020, at 6:39 PM, Bill Bejeck < bbejeck@ gmail. com (
>> bbej...@gmail.com ) > wrote:
>> 
>> 
>> 
>> Hello Kafka users, developers and client-developers,
>> 
>> 
>> 
>> This is the first candidate for release of Apache Kafka 2.4.1.
>> 
>> 
>> 
>> This is a bug fix release and it includes fixes and improvements
>> 
>> 
> 
> 
 
 
>>> 
>>> 
>> 
>> 
>> 
>> from
>> 
>> 
>>> 
 
 
 38
 
 
> 
>> 
>> 
>> JIRAs, including a few critical bugs.
>> 
>> 
>> 
>> Release notes for the 2.4.1 release:
>> 
>> 
> 
> 
 
 
>>> 
>>> 
>> 
>> 
>> 
>> https:/ / home. apache. org/ ~bbejeck/ kafka-2. 4. 1-rc0/ RELEASE_NOTES. html
>> ( https://home.apache.org/~bbejeck/kafka-2.4.1-rc0/RELEASE_NOTES.html )
>> 
>> 
>>> 
 
> 
>> 
>> 
>> *Please download, test and vote by Thursday, March 5, 9 am PT*
>> 
>> 
>> 
>> Kafka's KEYS file containing PGP keys we use to sign the release: 
>> https:/ /
>> kafka. apache. org/ KEYS ( https://kafka.apache.org/KEYS )
>> 
>> 
>> 
>> * Release artifacts to be voted upon (source and binary): https:/ / 
>> home. apache.
>> org/ ~bbejeck/ kafka-2. 4. 1-rc0/ (
>> https://home.apache.org/~bbejeck/kafka-2.4.1-rc0/ )
>> 
>> 
>> 
>> * Maven artifacts to be voted upon:
>> 
>> 
> 
> 
 
 
>>> 
>>> 
>> 
>> 
>> 
>> https:/ / repository. apache. org/ content/ groups/ staging/ org/ apache/ 
>> kafka/
>> ( https://repository.apache.org/content/groups/staging/org/apache/kafka/ )
>> 
>> 
>> 
>>> 
 
>>

[DISCUSS] KIP-576

[Cheng Tan]

Hi developers,

KIP-226 added support for dynamic update of  broker configuration.  In this 
KIP, we propose to extend the support to dynamic update of a group of new 
dynamic broker configs which will benefit the replication process. In this KIP, 
we propose to extend the support to dynamic update of a group of new dynamic 
broker configs which will benefit the replication process.

fetch.max.bytes Maximum bytes expected for the entire fetch response.
failed.authentication.delay Connection close delay on failed authentication.
replica.fetch.response.max.bytesMaximum bytes expected for the entire 
fetch response.
replica.fetch.wait.max.ms   Max wait time for each fetcher request issued 
by follower replicas.
follower.replication.throttled.replicas A list of replicas for which log 
replication should be throttled on the follower side.
leader.replication.throttled.replicas   A list of replicas for which log 
replication should be throttled on the leader side.


The wiki link for KIP-576:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-576%3A+Support+dynamic+update+of+more+configs+related+to+replication
 


Please feel free to discuss and suggest any other broker configs worth making 
dynamic. Thanks.

Regards, - Cheng Tan

[jira] [Created] (KAFKA-9684) Add support for SNI names in SSL request

[Badai Aqrandista (Jira)]

Badai Aqrandista created KAFKA-9684:
---

 Summary: Add support for SNI names in SSL request
 Key: KAFKA-9684
 URL: https://issues.apache.org/jira/browse/KAFKA-9684
 Project: Kafka
  Issue Type: Improvement
Reporter: Badai Aqrandista


When running Kafka cluster with SSL security behind HA Proxy, we need the 
client to send SSL packets with SNI name extension [1]. This will allow HA 
Proxy to forward the request to the relevant broker behind it (passthrough).

Java 7 and higher supports this by adding SNIHostName [2] to SSLParameters [3]. 


[1] https://www.ietf.org/rfc/rfc6066.txt
[2] https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SNIHostName.html
[3] 
https://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SSLParameters.html#setServerNames-java.util.List-



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9683) Support dynamic update of fetch.max.bytes/failed.authentication.delay/replica.fetch.response.max.bytes/replica.fetch.wait.max.ms/follower.replication.throttled.replicas/l

[Cheng Tan (Jira)]

Cheng Tan created KAFKA-9683:


 Summary: Support dynamic update of 
fetch.max.bytes/failed.authentication.delay/replica.fetch.response.max.bytes/replica.fetch.wait.max.ms/follower.replication.throttled.replicas/leader.replication.throttled.replicas
 Key: KAFKA-9683
 URL: https://issues.apache.org/jira/browse/KAFKA-9683
 Project: Kafka
  Issue Type: Bug
Reporter: Cheng Tan


[KIP-226|https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration]
 added support for dynamic update of  broker configuration.  In this KIP, we 
propose to extend the support to dynamic update of a group of new dynamic 
broker configs which will benefit the replication process.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-8019) Better Scaling Experience for KStream

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-8019.

Resolution: Duplicate

Closing this a duplicate of KAFKA-6145. Should be address via KIP-441.

> Better Scaling Experience for KStream
> -
>
> Key: KAFKA-8019
> URL: https://issues.apache.org/jira/browse/KAFKA-8019
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
>  Labels: kip
>
> In our day-to-day work, we found it really hard to scale up a stateful stream 
> application when its state store is very heavy. The caveat is that when the 
> newly spinned hosts take ownership of some active tasks, so that they need to 
> use non-trivial amount of time to restore the state store from changelog 
> topic. The reassigned tasks would be available for unpredicted long time, 
> which is not favorable. Secondly the current global rebalance stops the 
> entire application process, which in a rolling host swap scenario would 
> suggest an infinite resource shuffling without actual progress.
> Following the community's [cooperative 
> rebalancing|https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies]
>  proposal, we need to build something similar for KStream to better handle 
> the auto scaling experience.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Add a customized logo for Kafka Streams

[Sophie Blee-Goldman]

Seems the mailing list may have filtered the inlined prototype logo,
attaching it here instead

On Sat, Mar 7, 2020 at 11:54 PM Sophie Blee-Goldman 
wrote:

> Matthias makes a good point about being careful not to position Streams as
> outside of Apache Kafka. One obvious thing we could do it just include the
> Kafka logo as-is in the Streams logo, somehow.
>
> I have some unqualified opinions on what that might look like:
> A good logo is simple and clean, so incorporating the Kafka logo as a minor
> detail within a more complicated image is probably not the best way to get
> the quick and easy comprehension/recognition that we're going for.
>
> That said I'd throw out the idea of just attaching something to the Kafka
> logo,
> perhaps a stream-dwelling animal, perhaps a (river) otter? It could be
> "swimming" left of the Kafka logo, with its head touching the upper circle
> and
> its tail touching the bottom one. Like Streams, it starts with Kafka and
> ends
> with Kafka (ie reading input topics and writing to output topics).
>
> Without further ado, here's my very rough prototype for the Kafka Streams
> logo:
>
> [image: image.png]
> Obviously the real thing would be colored and presumably done by someone
> with actual artist talent/experience (or at least photoshop ability).
>
> Thoughts?
>
> On Sat, Mar 7, 2020, 1:08 PM Matthias J. Sax  wrote:
>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA512
>>
>> Boyang,
>>
>> thanks for starting this discussion. I like the idea in general
>> however we need to be a little careful IMHO -- as you mentioned Kafka
>> is one project and thus we should avoid the impression that Kafka
>> Streams is not part of Apache Kafka.
>>
>> Besides this, many projects use animals that are often very adorable.
>> Maybe we could find a cute Streams related mascot? :)
>>
>> I would love to hear opinions especially from the PMC if having a logo
>> for Kafka Streams is a viable thing to do.
>>
>>
>> - -Matthias
>>
>> On 3/3/20 1:01 AM, Patrik Kleindl wrote:
>> > Hi Boyang Great idea, that would help in some discussions. To throw
>> > in a first idea: https://imgur.com/a/UowXaMk best regards Patrik
>> >
>> > On Mon, 2 Mar 2020 at 18:23, Boyang Chen
>> >  wrote:
>> >
>> >> Hey Apache Kafka committers and community folks,
>> >>
>> >> over the years Kafka Streams has been widely adopted and tons of
>> >> blog posts and tech talks have been trying to introduce it to
>> >> people with need of stream processing. As it is part of Apache
>> >> Kafka project, there is always an awkward situation where Kafka
>> >> Streams could not be campaigned as a standalone streaming engine,
>> >> and makes people confused about its relation to Kafka.
>> >>
>> >> So, do we want to introduce a customized logo just for Streams?
>> >> The immediate benefit is when people are making technical
>> >> decisions, we could list Streams as a logo just like Flink and
>> >> Spark Streaming, instead of putting Kafka logo there as it is not
>> >> literally a legitimate comparison between processing framework
>> >> and messaging system. Should we do a KIP for this?
>> >>
>> >> Boyang
>> >>
>> >
>> -BEGIN PGP SIGNATURE-
>>
>> iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5kDVQACgkQO4miYXKq
>> /Og2Hg/+IOjjz3yrvHOz/p/Qbi9hXDK+GZT6Ixzk2HrrpMiLAAZs6WrNGQMwwI6a
>> qrPdMRLKA45F53wwMXBqhLYe0O0vOIRSur3pef8UuTVEkbFstY6dNzduPjTXK4vV
>> Ahzb+pu1lZsF+n2DbMuXWvkmvMAsSrKQXbH58rA7I4qx7Zr6g1a/KL2b2oOoo3kI
>> 4x3PJfG6oLSnQHwkJxmM79ZjM7MLZh0d8cRqb7Oudy5MJiMzHY+Rm5aTu4nhPgRr
>> cLxA8kz1PbGPboxjD9/ZGuZJMWfVnvY1wJcOp5UnOUs4kX5uYDyWw1sKIn3DcnW8
>> YVzoto0syCHTAdbl89H2fxhJbtVp8JSxbBx9AW8mdgLOLsYRZGHZ1cbdJ3h4NkeE
>> xTPTuTptQbhdcjbSVX6F0q+h1hiPCU5PKqcR12zGVTBI4rOGkhPIhDdnIti5Qp2e
>> MQ1Urh/tWCfwiJZbOyjZ9Tz/2vbUBB4kzUI9DxwWpD15jgMuN3JfqFwcwSIm7NrY
>> jHcM3UB2QlUt+uymU78xjy6er2AFiGrfL7UXeFHzzVIWBU3fdw4wqpuIuWjFhL3t
>> NWnAvamXOc7kfE4VXQ8igoRUVDZ8xCCqNvVoaJZ1cWboY/Cfk+4sgd9QnDMZN9jU
>> bzRpghQp3R93Vp5FH4L9z478uuh8DFM2/+6pqqapd9efydGtKrU=
>> =OFBs
>> -END PGP SIGNATURE-
>>
>