[jira] [Created] (KAFKA-12710) Consider enabling (at least some) optimizations by default

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-12710:
--

 Summary: Consider enabling (at least some) optimizations by default
 Key: KAFKA-12710
 URL: https://issues.apache.org/jira/browse/KAFKA-12710
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: A. Sophie Blee-Goldman


Topology optimizations such as the repartition consolidation and source topic 
changelog are extremely useful at reducing the footprint of a Kafka Streams 
application on the broker. The additional storage and resource utilization due 
to changelogs and repartitions is a very real pain point, and has even been 
cited as the reason for turning to other stream processing frameworks in the 
past (though of course I question that judgement)

The repartition topic optimization, at the very least, should be enabled by 
default. The problem is that we can't just flip the switch without breaking 
existing applications during upgrade, since the location and name of such 
topics in the topology may change. One possibility is to just detect this 
situation and disable the optimization if we find that it would produce an 
incompatible topology for an existing application. We can determine that this 
is the case simply by looking for pre-existing repartition topics. If any such 
topics are present, and match the set of repartition topics in the un-optimized 
topology, then we know we need to switch the optimization off. If we don't find 
any repartition topics, or they match the optimized topology, then we're safe 
to enable it by default.

Alternatively, we could just do a KIP to indicate that we intend to change the 
default in the next breaking release and that existing applications should 
override this config if necessary. We should be able to implement a fail-safe 
and shut down if a user misses or forgets to do so, using the method mentioned 
above.

The source topic optimization is perhaps more controversial, as there have been 
a few issues raised with regards to things like [restoring bad data and 
asymmetric serdes|https://issues.apache.org/jira/browse/KAFKA-8037], or more 
recently the bug discovered in the [emit-on-change semantics for 
KTables|https://issues.apache.org/jira/browse/KAFKA-12508?focusedCommentId=17306323=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17306323].
 However for this case at least there are no compatibility concerns. It's safe 
to upgrade from using a separate changelog for a source KTable to just using 
that source topic directly, although the reverse is not true. We could even 
automatically delete the no-longer-necessary changelog for upgrading 
applications



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Colin McCabe]

Hi Sarwar,

It seems like we didn't reach a consensus last time about this KIP.  So I would 
suggest revising the KIP, and then creating a new DISCUSS thread when it's 
ready for review.

regards,
Colin


On Wed, Apr 14, 2021, at 01:26, Sarwar Bhuiyan wrote:
> Forgive me while I find my feet, but since this reached a VOTE stage, is it
> really required to go through a DISCUSS thread again or can we continue to
> address the issues on this thread since the issues were raised on this
> thread? If I do have to create a new DISCUSS thread, do I just copy the
> messages from the old thread to it to address them?
> 
> Thank you.
> 
> Sarwar
> 
> On Wed, Apr 14, 2021 at 9:18 AM Stanislav Kozlovski 
> wrote:
> 
> > Hey all,
> >
> > To revive this old KIP, Sarwar Bhuiyan has volunteered to take over
> > ownership.
> > He will continue to drive this KIP to approval and completion - I
> > understand that he will re-start the discussion with a new [DISCUSS] or
> > [VOTE] thread.
> >
> > Thank you Sarwar!
> >
> > Best,
> > Stanislav
> >
> > On Fri, Jan 10, 2020 at 5:55 PM Gwen Shapira  wrote:
> >
> >> Sorry for the super late reply, but since the vote thread showed up,
> >> I've read the KIP and have a concern.
> >> The concern was raised by Matthias Sax earlier, but I didn't see it
> >> addressed.
> >>
> >> Basically the current iteration of the KIP unifies deserialization
> >> errors with corruption. As was pointed out, these are not the same
> >> thing. Corruption means that the message itself (including envelope,
> >> not just the payload) is broken. De-serialization errors mean that
> >> either key or value serializers have a problem. It can even be a
> >> temporary problem of connecting to schema registry, I believe. Corrupt
> >> messages can only be skipped. De-serialization errors can (and
> >> arguably should) be retried with a different serializer. Something
> >> like Connect will need to skip corrupt messages, but messages with
> >> SerDe issues should probably go into a dead-letter queue.
> >>
> >> Anyway, IMO we need exceptions that will let us tell the difference.
> >>
> >> Gwen
> >>
> >> On Fri, Oct 11, 2019 at 10:05 AM Stanislav Kozlovski
> >>  wrote:
> >> >
> >> > Thanks Jason. I've edited the KIP with the latest proposal.
> >> >
> >> > On Thu, Oct 10, 2019 at 2:00 AM Jason Gustafson 
> >> wrote:
> >> >
> >> > > Hi Stanislav,
> >> > >
> >> > > Sorry for the late comment. I'm happy with the current proposal. Just
> >> one
> >> > > small request is to include an example which shows how a user could
> >> use
> >> > > this to skip over a record.
> >> > >
> >> > > I'd suggest pushing this to a vote to see if anyone else has feedback.
> >> > >
> >> > > Thanks,
> >> > > Jason
> >> > >
> >> > > On Sat, Jul 13, 2019 at 2:27 PM Stanislav Kozlovski <
> >> > > stanis...@confluent.io>
> >> > > wrote:
> >> > >
> >> > > > Hello,
> >> > > >
> >> > > > Could we restart the discussion here again?
> >> > > >
> >> > > > My last message was sent on the 3rd of June but I haven't received
> >> > > replies
> >> > > > since then.
> >> > > >
> >> > > > I'd like to get this KIP to a finished state, regardless of whether
> >> that
> >> > > is
> >> > > > merged-in or discarded. It has been almost one year since the
> >> publication
> >> > > > of the KIP.
> >> > > >
> >> > > > Thanks,
> >> > > > Stanislav
> >> > > >
> >> > > > On Mon, Jun 3, 2019 at 11:19 AM Stanislav Kozlovski <
> >> > > > stanis...@confluent.io>
> >> > > > wrote:
> >> > > >
> >> > > > > Do people agree with the approach I outlined in my last reply?
> >> > > > >
> >> > > > > On Mon, May 6, 2019 at 2:12 PM Stanislav Kozlovski <
> >> > > > stanis...@confluent.io>
> >> > > > > wrote:
> >> > > > >
> >> > > > >> Hey there Kamal,
> >> > > > >>
> >> > > > >> I'm sincerely sorry for missing your earlier message. As I open
> >> this
> >> > > > >> thread up, I see I have an unsent draft message about resuming
> >> > > > discussion
> >> > > > >> from some time ago.
> >> > > > >>
> >> > > > >> In retrospect, I think I may have been too pedantic with the
> >> exception
> >> > > > >> naming and hierarchy.
> >> > > > >> I now believe a single exception type of
> >> > > > `RecordDeserializationException`
> >> > > > >> is enough. Let's go with that.
> >> > > > >>
> >> > > > >> On Mon, May 6, 2019 at 6:40 AM Kamal Chandraprakash <
> >> > > > >> kamal.chandraprak...@gmail.com> wrote:
> >> > > > >>
> >> > > > >>> Matthias,
> >> > > > >>>
> >> > > > >>> We already have CorruptRecordException which doesn't extend the
> >> > > > >>> SerializationException. So, we need an alternate
> >> > > > >>> name suggestion for the corrupted record error if we decide to
> >> extend
> >> > > > the
> >> > > > >>> FaultyRecordException class.
> >> > > > >>>
> >> > > > >>> Stanislav,
> >> > > > >>>
> >> > > > >>> Our users are also facing this error. Could we bump up this
> >> > > discussion?
> >> > > > >>>
> >> > > > >>> I think we can have a single exception type
> >> > > > >>> 

Re: Requesting review of PR 10377

[Colin McCabe]

Hi feyman2009,

Thanks for the PR.  I pinged a few people about reviewing it.  I don't know if 
anyone will be able to look at it this week, but probably next week we can.  
The good news is that this field is optional (tagged) so the protocol will 
still work with older clients.

best,
Colin


On Sat, Apr 17, 2021, at 06:50, feyman2009 wrote:
> Hi team,
> Could anyone help to review PR 10377, thanks!
>  KAFKA-12515 ApiVersionManager should create a response based on the request
> https://github.com/apache/kafka/pull/10377/files
> 
> Haoran Xuan

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #64

[Apache Jenkins Server]

See 

Re: [ANNOUNCE] New Kafka PMC Member: Bill Bejeck

[lobo xu]

 Congratulations Bill !

 Lobo
> 在 2021年4月22日，13:05，Navinder Brar  写道：
> 
> Congratulations Bill!

[jira] [Created] (KAFKA-12709) Admin API for ListTransactions

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-12709:
---

 Summary: Admin API for ListTransactions
 Key: KAFKA-12709
 URL: https://issues.apache.org/jira/browse/KAFKA-12709
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jason Gustafson
Assignee: Jason Gustafson


Add the `listTransactions` API described in KIP-664: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+transactions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-12586) Admin API for DescribeTransactions

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-12586.
-
Resolution: Fixed

> Admin API for DescribeTransactions
> --
>
> Key: KAFKA-12586
> URL: https://issues.apache.org/jira/browse/KAFKA-12586
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
>
> Add the Admin API for DescribeTransactions documented on KIP-664: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+transactions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #63

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-12371) MirrorMaker 2.0 documentation is incorrect

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-12371.

Resolution: Duplicate

> MirrorMaker 2.0 documentation is incorrect
> --
>
> Key: KAFKA-12371
> URL: https://issues.apache.org/jira/browse/KAFKA-12371
> Project: Kafka
>  Issue Type: Improvement
>  Components: docs, documentation
>Affects Versions: 2.7.0
>Reporter: Scott Kirkpatrick
>Priority: Minor
>
> There are a few places in the official MirrorMaker 2.0 docs that are either 
> confusing or incorrect. Here are a few examples I've found:
> The documentation for the 'sync.group.offsets.enabled' config states that 
> it's enabled by default 
> [here|https://github.com/apache/kafka-site/blob/61f4707381c369a98a7a77e1a7c3a11d5983909c/27/ops.html#L802],
>  but the actual source code indicates that it's disabled by default 
> [here|https://github.com/apache/kafka/blob/f75efb96fae99a22eb54b5d0ef4e23b28fe8cd2d/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java#L185].
>  I'm unsure if the intent is to have it enabled or disabled by default.
> There are also some numerical typos, 
> [here|https://github.com/apache/kafka-site/blob/61f4707381c369a98a7a77e1a7c3a11d5983909c/27/ops.html#L791]
>  and 
> [here|https://github.com/apache/kafka-site/blob/61f4707381c369a98a7a77e1a7c3a11d5983909c/27/ops.html#L793].
>  These lines state that the default is 6000 seconds (and incorrectly that 
> it's equal to 10 minutes), but the actual default is 600 seconds, shown 
> [here|https://github.com/apache/kafka/blob/f75efb96fae99a22eb54b5d0ef4e23b28fe8cd2d/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java#L145]
>  and 
> [here|https://github.com/apache/kafka/blob/f75efb96fae99a22eb54b5d0ef4e23b28fe8cd2d/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java#L152]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-12529) kafka-configs.sh does not work while changing the sasl jaas configurations.

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-12529.

Resolution: Duplicate

see KAFKA-12530

> kafka-configs.sh does not work while changing the sasl jaas configurations.
> ---
>
> Key: KAFKA-12529
> URL: https://issues.apache.org/jira/browse/KAFKA-12529
> Project: Kafka
>  Issue Type: Bug
>Reporter: kaushik srinivas
>Priority: Major
>
> We are trying to use kafka-configs script to modify the sasl jaas 
> configurations, but unable to do so.
> Command used:
> ./kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers 
> --entity-name 59 --alter --add-config 'sasl.jaas.config=KafkaServer \{\n 
> org.apache.kafka.common.security.plain.PlainLoginModule required \n 
> username=\"test\" \n password=\"test\"; \n };'
> error:
> requirement failed: Invalid entity config: all configs to be added must be in 
> the format "key=val".
> command 2:
> kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers 
> --entity-name 59 --alter --add-config 
> 'sasl.jaas.config=[username=test,password=test]'
> output:
> command does not return , but kafka broker logs below error:
> DEBUG", "neid":"kafka-cfd5ccf2af7f47868e83471a5b603408", "system":"kafka", 
> "time":"2021-03-23T08:29:00.946", "timezone":"UTC", 
> "log":\{"message":"data-plane-kafka-network-thread-1001-ListenerName(SASL_PLAINTEXT)-SASL_PLAINTEXT-2
>  - org.apache.kafka.common.security.authenticator.SaslServerAuthenticator - 
> Set SASL server state to FAILED during authentication"}}
> {"type":"log", "host":"kf-kaudynamic-0", "level":"INFO", 
> "neid":"kafka-cfd5ccf2af7f47868e83471a5b603408", "system":"kafka", 
> "time":"2021-03-23T08:29:00.946", "timezone":"UTC", 
> "log":\{"message":"data-plane-kafka-network-thread-1001-ListenerName(SASL_PLAINTEXT)-SASL_PLAINTEXT-2
>  - org.apache.kafka.common.network.Selector - [SocketServer brokerId=1001] 
> Failed authentication with /127.0.0.1 (Unexpected Kafka request of type 
> METADATA during SASL handshake.)"}}
> We have below issues:
> 1. If one installs kafka broker with SASL mechanism and wants to change the 
> SASL jaas config via kafka-configs scripts, how is it supposed to be done ?
>  does kafka-configs needs client credentials to do the same ? 
> 2. Can anyone point us to example commands of kafka-configs to alter the 
> sasl.jaas.config property of kafka broker. We do not see any documentation or 
> examples for the same.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-12528) kafka-configs.sh does not work while changing the sasl jaas configurations.

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-12528.

Resolution: Duplicate

see KAFKA-12530

> kafka-configs.sh does not work while changing the sasl jaas configurations.
> ---
>
> Key: KAFKA-12528
> URL: https://issues.apache.org/jira/browse/KAFKA-12528
> Project: Kafka
>  Issue Type: Bug
>  Components: admin, core
>Reporter: kaushik srinivas
>Priority: Major
>
> We are trying to modify the sasl jaas configurations for the kafka broker 
> runtime using the dynamic config update functionality using the 
> kafka-configs.sh script. But we are unable to get it working.
> Below is our command:
> ./kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers 
> --entity-name 59 --alter --add-config 'sasl.jaas.config=KafkaServer \{\n 
> org.apache.kafka.common.security.plain.PlainLoginModule required \n 
> username=\"test\" \n password=\"test\"; \n };'
>  
> command is exiting with error:
> requirement failed: Invalid entity config: all configs to be added must be in 
> the format "key=val".
>  
> we also tried below format as well:
> kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers 
> --entity-name 59 --alter --add-config 
> 'sasl.jaas.config=[username=test,password=test]'
> command does not return but the kafka broker logs prints the below error 
> messages.
> org.apache.kafka.common.security.authenticator.SaslServerAuthenticator - Set 
> SASL server state to FAILED during authentication"}}
> {"type":"log", "host":"kf-kaudynamic-0", "level":"INFO", 
> "neid":"kafka-cfd5ccf2af7f47868e83471a5b603408", "system":"kafka", 
> "time":"2021-03-23T08:29:00.946", "timezone":"UTC", 
> "log":\{"message":"data-plane-kafka-network-thread-1001-ListenerName(SASL_PLAINTEXT)-SASL_PLAINTEXT-2
>  - org.apache.kafka.common.network.Selector - [SocketServer brokerId=1001] 
> Failed authentication with /127.0.0.1 (Unexpected Kafka request of type 
> METADATA during SASL handshake.)"}}
>  
> 1. If one has SASL enabled and with a single listener, how are we supposed to 
> change the sasl credentials using this command ?
> 2. can anyone point us out to some example commands for modifying the sasl 
> jaas configurations ?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12708) Rewrite org.apache.kafka.test.Microbenchmarks by JMH

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-12708:
--

 Summary: Rewrite org.apache.kafka.test.Microbenchmarks by JMH
 Key: KAFKA-12708
 URL: https://issues.apache.org/jira/browse/KAFKA-12708
 Project: Kafka
  Issue Type: Task
Reporter: Chia-Ping Tsai


The benchmark code is a bit obsolete and it would be better to rewrite it by JMH



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #62

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-12707) KafkaProducer should have a clearer error message on sasl.mechanism misconfiguration

[Rohan Desai (Jira)]

Rohan Desai created KAFKA-12707:
---

 Summary: KafkaProducer should have a clearer error message on 
sasl.mechanism misconfiguration
 Key: KAFKA-12707
 URL: https://issues.apache.org/jira/browse/KAFKA-12707
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Rohan Desai


Not sure if this is producer-specific, but I accidentally configured my 
producer by setting:

```

sasl.mechanism=plain

```

instead of 

```

sasl.mechanism=PLAIN

```

 

When I did this, the producer just hangs and logs in a loop like this, which 
isn't very informative:
[2021-04-21 21:33:20,519] WARN [Producer clientId=producer-1] Bootstrap broker 
pkc-g91q1.us-west-2.aws.stag.cpdev.cloud:9092 (id: -1 rack: null) disconnected 
(org.apache.kafka.clients.NetworkClient:1050)
[2021-04-21 21:33:21,584] INFO [Producer clientId=producer-1] Failed to create 
channel due to  (org.apache.kafka.common.network.SaslChannelBuilder:239)
org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure 
SaslClientAuthenticator
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed 
to create SaslClient with mechanism plain
[2021-04-21 21:33:21,584] WARN [Producer clientId=producer-1] Error connecting 
to node pkc-g91q1.us-west-2.aws.stag.cpdev.cloud:9092 (id: -1 rack: null) 
(org.apache.kafka.clients.NetworkClient:982)
java.io.IOException: Channel could not be created for socket 
java.nio.channels.SocketChannel[closed]
at 
org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:348)
at 
org.apache.kafka.common.network.Selector.registerChannel(Selector.java:329)
at org.apache.kafka.common.network.Selector.connect(Selector.java:256)
at 
org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:977)
at 
org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:73)
at 
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1148)
at 
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1036)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)
at 
org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.KafkaException: 
org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure 
SaslClientAuthenticator
at 
org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:240)
at 
org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:338)
... 10 more
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed 
to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed 
to create SaslClient with mechanism plain
[2021-04-21 21:33:21,584] WARN [Producer clientId=producer-1] Bootstrap broker 
pkc-g91q1.us-west-2.aws.stag.cpdev.cloud:9092 (id: -1 rack: null) disconnected 
(org.apache.kafka.clients.NetworkClient:1050)
 

It would be better to early-exit with a clear error message



--
This message was sent by Atlassian Jira
(v8.3.4#803005)