[jira] [Created] (KAFKA-14977) testDescribeStateOfExistingGroupWithRoundRobinAssignor is flaky

[Philip Nee (Jira)]

Philip Nee created KAFKA-14977:
--

 Summary: testDescribeStateOfExistingGroupWithRoundRobinAssignor is 
flaky
 Key: KAFKA-14977
 URL: https://issues.apache.org/jira/browse/KAFKA-14977
 Project: Kafka
  Issue Type: Bug
  Components: unit tests
Reporter: Philip Nee
 Attachments: failed_test.log

Relevent ticket: 
- KAFKA-8110 Flaky Test 
DescribeConsumerGroupTest#testDescribeMembersWithConsumersWithoutAssignedPartitions
- KAFKA-7969 Flaky Test 
DescribeConsumerGroupTest#testDescribeOffsetsOfExistingGroupWithNoMembers
- KAFKA-8068 Flaky Test 
DescribeConsumerGroupTest#testDescribeMembersOfExistingGroup
- KAFKA-8706 Kafka 2.3.0 Transient Unit Test Failures on Oracle Linux - See 
attached for details
 
See the attachment for failure



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14918) KRaft controller sending ZK controller RPCs to KRaft brokers

[Colin McCabe (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-14918.
--
Resolution: Fixed

> KRaft controller sending ZK controller RPCs to KRaft brokers
> 
>
> Key: KAFKA-14918
> URL: https://issues.apache.org/jira/browse/KAFKA-14918
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Critical
> Fix For: 3.5.0
>
>
> During the migration, when upgrading a ZK broker to KRaft, the controller is 
> incorrectly sending UpdateMetadata requests to the KRaft controller. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14698) Received request api key LEADER_AND_ISR which is not enabled

[Colin McCabe (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-14698.
--
Fix Version/s: (was: 3.4.1)
   Resolution: Duplicate

> Received request api key LEADER_AND_ISR which is not enabled
> 
>
> Key: KAFKA-14698
> URL: https://issues.apache.org/jira/browse/KAFKA-14698
> Project: Kafka
>  Issue Type: Bug
>  Components: kraft
>Affects Versions: 3.4.0
>Reporter: Mickael Maison
>Assignee: Akhilesh Chaganti
>Priority: Major
> Fix For: 3.5.0
>
> Attachments: broker0.log, controller.log, test_online_migration.tar.gz
>
>
> I started from a Kafka cluster (with ZooKeeper) with 2 brokers. There's a 
> single topic "test" with 2 partitions and 2 replicas and the internal 
> __consumer_offsets topics.
> While following the ZooKeeper to KRaft migration steps from 
> [https://kafka.apache.org/documentation/#kraft_zk_migration], I'm hitting 
> issues at the Migrating brokers to KRaft step.
> When I restart a broker as KRaft, it repetitively prints the following error:
> {code:java}
> org.apache.kafka.common.errors.InvalidRequestException: Received request api 
> key LEADER_AND_ISR which is not enabled
> [2023-02-09 16:14:30,334] ERROR Closing socket for 
> 192.168.1.11:9092-192.168.1.11:63737-371 because of error 
> (kafka.network.Processor)
> {code}
> The controller repetitively prints the following error:
> {code:java}
> [2023-02-09 16:12:27,456] WARN [Controller id=1000, targetBrokerId=0] 
> Connection to node 0 (mmaison-mac.home/192.168.1.11:9092) could not be 
> established. Broker may not be available. 
> (org.apache.kafka.clients.NetworkClient)
> [2023-02-09 16:12:27,456] INFO [Controller id=1000, targetBrokerId=0] Client 
> requested connection close from node 0 
> (org.apache.kafka.clients.NetworkClient)
> [2023-02-09 16:12:27,560] INFO [Controller id=1000, targetBrokerId=0] Node 0 
> disconnected. (org.apache.kafka.clients.NetworkClient)
> {code}
> Attached the controller logs and logs from broker-0
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14976) Left/outer stream-stream joins create KV stores that aren't customizable

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-14976:
--

 Summary: Left/outer stream-stream joins create KV stores that 
aren't customizable
 Key: KAFKA-14976
 URL: https://issues.apache.org/jira/browse/KAFKA-14976
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: A. Sophie Blee-Goldman


It appears that we only give the illusion of full customizability when it comes 
to the state stores of a windowed join. This arose due to an 
[optimization|https://github.com/apache/kafka/pull/11252] for the performance 
of the spurious results fix, and means that these joins now come with one 
additional, and possibly unexpected, state store:

 
{code:java}
final StoreBuilder, 
LeftOrRightValue>> builder =
            new ListValueStoreBuilder<>(
         |--[   persistent ? 
this-->  |         Stores.persistentKeyValueStore(storeName) : 
         |--[      Stores.inMemoryKeyValueStore(storeName),
                timestampedKeyAndJoinSideSerde,
                leftOrRightValueSerde,
                Time.SYSTEM
            ); {code}
 

where persistent is defined above that as
{code:java}
final boolean persistent = streamJoinedInternal.thisStoreSupplier() == null || 
streamJoinedInternal.thisStoreSupplier().get().persistent(); {code}
 
This means regardless of whether a custom state store implementation was passed 
in to the join, we will still insert one of our RocksDB or InMemory state 
stores. Which might be very surprising since the API makes it seem like the 
underlying stores are fully configurable.

I'm adding a warning line for this in PR 
[#13682|https://github.com/apache/kafka/pull/13682/files#diff-9ce43046fdef1233ab762e728abd1d3d44d7c270b28dcf6b63aa31a93a30af07R334-R336]
 but we should really make this hidden state store fully configurable like the 
window stores currently are (which will require a KIP)

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-923: Add A Grace Period to Stream Table Join

[Matthias J. Sax]

Thanks for the KIP! Also some question/comments from my side:

10) Notation: you use the term "late data" but I think you mean 
out-of-order. We reserve the term "late" to records that arrive after 
grace period passed, and thus, "late == out-of-order data that is dropped".



20) "There is only one option from the stream side and only recently is 
there a second option on the table side."


What are those options? Victoria already asked about the table side, but 
I am also not sure what option you mean for the stream side?



30) "If the table side uses a materialized version store the value is 
the latest by stream time rather than by offset within its defined grace 
period."


The phrase "the value is the latest by stream time" is confusing -- in 
the end, a versioned stores multiple versions, not just one.



40) I am also wondering about ordering. In general, KS tries to preserve 
offset-order during processing (with some exception, when offset order 
preservation is not clearly defined). Given that the stream-side buffer 
is really just a "linear buffer", we could easily preserve offset-order. 
But I also see a benefit of re-ordering and emitting out-of-order data 
right away when read (instead of blocking them behind in-order records 
that are not ready yet). -- It might even be a possibility, to let users 
pick a emit strategy eg "EmitStrategy.preserveOffsets" (name just a 
placeholder).


The KIP should explain this in more detail and also discuss different 
options and mention them in "Rejected alternatives" in case we don't 
want to include them.



50) What happens when users change the grace period? Especially, when 
they turn it on/off (but also increasing/decreasing is an interesting 
point)? I think we should try to support this if possible; the 
"Compatibility" section needs to cover switching on/off in more detail.



-Matthias




On 5/2/23 2:06 PM, Victoria Xia wrote:

Cool KIP, Walker! Thanks for sharing this proposal.

A few clarifications:

1. Is the order that records exit the buffer in necessarily the same as the
order that records enter the buffer in, or no? Based on the description in
the KIP, it sounds like the answer is no, i.e., records will exit the
buffer in increasing timestamp order, which means that they may be ordered
(even for the same key) compared to the input order.

2. What happens if the join grace period is nonzero, and a stream-side
record arrives with a timestamp that is older than the current stream time
minus the grace period? Will this record trigger a join result, or will it
be dropped? Based on the description for what happens when the join grace
period is set to zero, it sounds like the late record will be dropped, even
if the join grace period is nonzero. Is that true?

3. What could cause stream time to advance, for purposes of removing
records from the join buffer? For example, will new records arriving on the
table side of the join cause stream time to advance? From the KIP it sounds
like only stream-side records will advance stream time -- does that mean
that the join processor itself will have to track this stream time?

Also +1 to Lucas's question about what options will be available for
configuring the join buffer. Will users have the option to choose whether
they want the buffer to be in-memory vs persistent?

- Victoria

On Fri, Apr 28, 2023 at 11:54 AM Lucas Brutschy
 wrote:


HI Walker,

thanks for the KIP! We definitely need this. I have two questions:

  - Have you considered allowing the customization of the underlying
buffer implementation? As I can see, `StreamJoined` lets you customize
the underlying store via a `WindowStoreSupplier`. Would it make sense
for `Joined` to have this as well? I can imagine one may want to limit
the number of records in the buffer, for example. If we hit the
maximum, the only option would be to drop semantic guarantees, but
users may still want to do this.
  - With "second option on the table side" you are referring to
versioned tables, right? Will the buffer on the stream side behave any
different whether the table side is versioned or not?

Finally, I think a simple example in the motivation section could help
non-experts understand the KIP.

Best,
Lucas

On Tue, Apr 25, 2023 at 9:13 PM Walker Carlson
 wrote:


Hello everybody,

I have a stream proposal to improve the stream table join by adding a

grace

period and buffer to the stream side of the join to allow processing in
timestamp order matching the recent improvements of the versioned tables.

Please take a look here 

and

share your thoughts.

best,
Walker

[jira] [Created] (KAFKA-14975) Make TopicBasedRemoteLogMetadataManager methods wait for initialize to complete

[Matthew de Detrich (Jira)]

Matthew de Detrich created KAFKA-14975:
--

 Summary: Make TopicBasedRemoteLogMetadataManager methods wait for 
initialize to complete
 Key: KAFKA-14975
 URL: https://issues.apache.org/jira/browse/KAFKA-14975
 Project: Kafka
  Issue Type: Task
Reporter: Matthew de Detrich






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14974) Restore backward compatibility in KafkaBasedLog

[Yash Mayya (Jira)]

Yash Mayya created KAFKA-14974:
--

 Summary: Restore backward compatibility in KafkaBasedLog
 Key: KAFKA-14974
 URL: https://issues.apache.org/jira/browse/KAFKA-14974
 Project: Kafka
  Issue Type: Task
Reporter: Yash Mayya
Assignee: Yash Mayya


{{KafkaBasedLog}} is a widely used utility class that provides a generic 
implementation of a shared, compacted log of records in a Kafka topic. It isn't 
in Connect's public API, but has been used outside of Connect and we try to 
preserve backward compatibility whenever possible. 
https://issues.apache.org/jira/browse/KAFKA-14455 modified the two overloaded 
void {{KafkaBasedLog::send}} methods to return a {{{}Future{}}}. While this 
change is source compatible, it isn't binary compatible. We can restore 
backward compatibility simply by re-instating the older send methods, and 
renaming the new Future returning send methods.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14973) Inconsistent treatment of invalid config when creating or altering a topic

[James Shaw (Jira)]

James Shaw created KAFKA-14973:
--

 Summary: Inconsistent treatment of invalid config when creating or 
altering a topic
 Key: KAFKA-14973
 URL: https://issues.apache.org/jira/browse/KAFKA-14973
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 3.4.0
Reporter: James Shaw


{{createTopics}} throws {{InvalidConfigurationException}} on receiving an 
invalid config entry name or value.
{{incrementalAlterConfigs}} throws {{InvalidConfigurationException}} on 
receiving an invalid config entry name, but throws {{InvalidRequestException}} 
on receiving an invalid entry value.

The {{incrementalAlterConfigs}} javadoc mentions that 
{{InvalidRequestException}} is anticipated; the {{createTopics}} javadoc says 
nothing about exception types.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14662) ACL listings in documentation are out of date

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14662.
---
Fix Version/s: 3.6.0
   Resolution: Fixed

> ACL listings in documentation are out of date
> -
>
> Key: KAFKA-14662
> URL: https://issues.apache.org/jira/browse/KAFKA-14662
> Project: Kafka
>  Issue Type: Bug
>  Components: core, docs
>Reporter: Mickael Maison
>Assignee: Gantigmaa Selenge
>Priority: Major
> Fix For: 3.6.0
>
>
> ACLs listed in 
> https://kafka.apache.org/documentation/#operations_resources_and_protocols 
> are out of date. They only cover API keys up to 47 (OffsetDelete) and don't 
> include DescribeClientQuotas, AlterClientQuotas, 
> DescribeUserScramCredentials, AlterUserScramCredentials, DescribeQuorum, 
> AlterPartition, UpdateFeatures, DescribeCluster, DescribeProducers, 
> UnregisterBroker, DescribeTransactions, ListTransactions, AllocateProducerIds.
> This is hard to keep up to date so we should consider whether this could be 
> generated automatically.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)