Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2456

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2455

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15984) Client disconnections can cause hanging transactions on __consumer_offsets

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-15984:
--

 Summary: Client disconnections can cause hanging transactions on 
__consumer_offsets
 Key: KAFKA-15984
 URL: https://issues.apache.org/jira/browse/KAFKA-15984
 Project: Kafka
  Issue Type: Task
Reporter: Justine Olshan


When investigating frequent hanging transactions on __consumer_offsets 
partitions, we realized that many of them were cause by the same offset being 
committed with duplicates and one with `"isDisconnectedClient":true`. 

TxnOffsetCommits do not have sequence numbers and thus are not protected 
against duplicates in the same way idempotent produce requests are. Thus, when 
a client is disconnected (and flushes its requests), we may see the duplicate 
get appended to the log. 

KIP-890 part 1 should protect against this as the duplicate will not succeed 
verification. KIP-890 part 2 strengthens this further as duplicates (from 
previous transactions) can not be added to new transactions if the partitions 
is re-added since the epoch will be bumped. 

Another possible solution is to do duplicate checking on the group coordinator 
side when the request comes in. This solution could be used instead of KIP-890 
part 1 to prevent hanging transactions but given that part 1 only has one open 
PR remaining, we may not need to do this. However, this can also prevent 
duplicates from being added to a new transaction – something only part 2 will 
protect against.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15983) Kafka-acls should return authorization already done if repeat work is issued

[Chen He (Jira)]

Chen He created KAFKA-15983:
---

 Summary: Kafka-acls should return authorization already done if 
repeat work is issued
 Key: KAFKA-15983
 URL: https://issues.apache.org/jira/browse/KAFKA-15983
 Project: Kafka
  Issue Type: Improvement
  Components: security
Affects Versions: 3.6.0
Reporter: Chen He


kafka-acls.sh cmd will always issue normal operation for a cmd if customer 
already authorized a user. It should reports something like "user {} already 
authorized with {} resources" instead of do it again and again. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-996: Pre-Vote

[Alyssa Huang]

>From Jose -

> 1. In the schema for VoteRequest and VoteResponse, you are using
> "boolean" as the type keyword. The correct keyword should be "bool"
> instead.
>
Thanks!


> 2. In the states and state transaction table you have the following entry:
> >  * Candidate transitions to:
> > *...
> > *Prospective: After expiration of the election timeout
>
> Can you explain the reason a candidate would transition back to
> prospective? If a voter transitions to the candidate state it is
> because the voters don't support KIP-996 or the replica was able to
> win the majority of the votes at some point in the past. Are we
> concerned that the network partition might have occurred after the
> replica has become a candidate? If so, I think we should state this
> explicitly in the KIP.
>
Added this under Proposed Changes

Also, if a Candidate is unable to be elected (transition to Leader) before
its election timeout expires, it will transition back to Prospective. This
will handle the case if a network partition occurs while the server is in
Candidate state and prevent unnecessary loss of leadership.


3. In the proposed section and state transition section, I think it
> would be helpful to explicitly state that we have an invariant that
> only the prospective state can transition to the candidate state. This
> transition to the candidate state from the prospective state can only
> happen because the replica won the majority of the votes or there is
> at least one remote voter that doesn't support pre-vote.
>
 Added this under Proposed Changes

A follower will now transition to Prospective instead of Candidate when its
fetch timeout expires. Servers will only be able to transition to Candidate
state from the Prospective state.

4. I am a bit confused by this paragraph
> > A candidate will now send a VoteRequest with the PreVote field set to
> true and CandidateEpoch set to its [epoch + 1] when its election timeout
> expires. If [majority - 1] of VoteResponse grant the vote, the candidate
> will then bump its epoch up and send a VoteRequest with PreVote set to
> false which is our standard vote that will cause state changes for servers
> receiving the request.
>
> I am assuming that "candidate" refers to the states enumerated on the
> table above this quote. If so, I think you mean "prospective" for the
> first candidate.
>
> CandidateEpoch should be ReplicaEpoch.
>
> [epoch + 1] should just be epoch. I thought we agreed that replicas
> will always send their current epoch to the remote replicas.

Thanks, Luke also pointed out that I missed modifying this section. It
should read correctly now.


> 5. I am a bit confused by this bullet section
> > true if the server receives less than [majority] VoteResponse with
> VoteGranted set to false within [election.timeout.ms + a little
> randomness] and the first bullet point does not apply
>  Explanation for why we don't send a standard vote at this point
> is explained in rejected alternatives.
>
> Can we explain this case in plain english? I assume that this case is
> trying to cover the scenario where the election timer expired but the
> prospective candidate hasn't received enough votes (granted or
> rejected) to make a decision if it could win an election.
>
 Yes, thanks for the better wording! Modified to the following


   - true if the server does not receive enough votes (granted or rejected)
   within [election.timeout.ms + a little randomness]


6.
> > Yes. If a leader is unable to receive fetch responses from a majority of
> servers, it can impede followers that are able to communicate with it from
> voting in an eligible leader that can communicate with a majority of the
> cluster.
>
> In general, leaders don't receive fetch responses. They receive FETCH
> requests. Did you mean "if a leader is able to send FETCH responses to
> the majority - 1 of the voters, it can impede fetching voters
> (followers) from granting their vote to prospective candidates. This
> should stop prospective candidates from getting enough votes to
> transition to the candidate state and increase their epoch".

7.
> > Check Quorum ensures a leader steps down if it is unable to receive
> fetch responses from a majority of servers.
>
> I think you mean "... if it is unable to receive FETCH requests from
> the majority - 1 of the voters".
>
Yes, thanks for this catch! The section now reads as

Yes. If a leader is unable to send FETCH responses to [majority - 1] of
servers, it can impede its connected followers from granting their vote to
prospectives which *can* communicate with a majority of the cluster. This
is the reason why an additional "Check Quorum" safeguard is needed which is
what KAFKA-15489  implements.
Check Quorum ensures a leader steps down if it is unable to receive FETCH
requests from a majority of servers. This will allow all servers to grant
their votes to eligible prospectives.



> 8. At the end of the Proposed chan

[jira] [Created] (KAFKA-15982) Move GenericGroup state metrics to `GroupCoordinatorMetricsShard`

[Jeff Kim (Jira)]

Jeff Kim created KAFKA-15982:


 Summary: Move GenericGroup state metrics to 
`GroupCoordinatorMetricsShard`
 Key: KAFKA-15982
 URL: https://issues.apache.org/jira/browse/KAFKA-15982
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jeff Kim
Assignee: Jeff Kim


Currently, the generic group state metrics exist inside 
`GroupCoordinatorMetrics` as global metrics. This causes issues as during 
unload, we need to traverse through all groups and decrement the group size 
counters. 

Move the generic group state metrics to the shard level so that when a 
partition is unloaded we automatically remove the counter.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15981) only record GenericGroup state metrics once per group during load

[Jeff Kim (Jira)]

Jeff Kim created KAFKA-15981:


 Summary: only record GenericGroup state metrics once per group 
during load
 Key: KAFKA-15981
 URL: https://issues.apache.org/jira/browse/KAFKA-15981
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jeff Kim
Assignee: Jeff Kim


Currently, we increment generic group metrics whenever we create a new 
GenericGroup object while we load the partition. This is incorrect as the 
partition may contain several records for the same group if in the active 
segment or if the segment has not yet been compacted.

Instead, increment the metrics for each group we successfully loaded.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2454

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-15972) Add support to exclude labels in client telemetry

[Apoorv Mittal (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apoorv Mittal resolved KAFKA-15972.
---
Resolution: Done

> Add support to exclude labels in client telemetry
> -
>
> Key: KAFKA-15972
> URL: https://issues.apache.org/jira/browse/KAFKA-15972
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apoorv Mittal
>Assignee: Apoorv Mittal
>Priority: Major
>
> Some of the labels/tags which are present in metric should be skipped while 
> collecting telemetry as data might already be known to broker hence, we 
> should minimize the data transfer. One of such labels is client_id which is 
> already present in RequestContext hence broker can append that label prior 
> emitting metrics to telemetry backend. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15684) Add support to describe all subscriptions through utility

[Jun Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-15684.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

merged the PR to trunk.

> Add support to describe all subscriptions through utility
> -
>
> Key: KAFKA-15684
> URL: https://issues.apache.org/jira/browse/KAFKA-15684
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apoorv Mittal
>Assignee: Apoorv Mittal
>Priority: Major
> Fix For: 3.7.0
>
>
> Open PR to support client-metrics through kafka-configs.sh doesn't list all 
> subscriptions. The functionality is missing because of missing support to 
> list client subscription in config repository and admin client. This task 
> should provide a workaround to fetch all subscriptions from config repository 
> by adding a method in KRaftMetadataCache. Later a KIP might be needed to add 
> support in AdminClient.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15877) Support change of temporality in Java client

[Apoorv Mittal (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apoorv Mittal resolved KAFKA-15877.
---
Resolution: Done

> Support change of temporality in Java client 
> -
>
> Key: KAFKA-15877
> URL: https://issues.apache.org/jira/browse/KAFKA-15877
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apoorv Mittal
>Assignee: Apoorv Mittal
>Priority: Major
>
> Details: https://github.com/apache/kafka/pull/14620#discussion_r1401554867



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15979) Add KIP-1001 CurrentControllerId metric

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-15979:


 Summary: Add KIP-1001 CurrentControllerId metric
 Key: KAFKA-15979
 URL: https://issues.apache.org/jira/browse/KAFKA-15979
 Project: Kafka
  Issue Type: Improvement
Reporter: Colin McCabe
Assignee: Colin McCabe






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15980) Add KIP-1001 CurrentControllerId metric

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-15980:


 Summary: Add KIP-1001 CurrentControllerId metric
 Key: KAFKA-15980
 URL: https://issues.apache.org/jira/browse/KAFKA-15980
 Project: Kafka
  Issue Type: Improvement
Reporter: Colin McCabe
Assignee: Colin McCabe






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-995: Allow users to specify initial offsets while creating connectors

[Chris Egerton]

Hi Ashwin,

Regarding point 4--I think this is still a bit unwise. When workers pick up
a new connector config from the config topic, they participate in a
rebalance. It may be safe to write offsets during that rebalance, but in
the name of simplicity, do you think we can write the offsets for the
connector before its config? The sequence of steps would be something like
this:

1. Validate offsets and connector config (can be done in any order)
2. Write offsets
3. Write connector config (with whatever initial state is specified in the
request, or the default if none is specified)

Cheers,

Chris

On Wed, Dec 6, 2023 at 9:13 AM Ashwin  wrote:

> Hello Chris,
>
> Thanks for the quick and detailed review. Please see my responses below
>
> High-level thoughts:
>
> 1. I had not thought of this till now, thanks for pointing it out. I
> would lean towards the second option of cleaning previous offsets as
> it will result in the fewer surprises for the user.
>
> 2 and 3. I agree and have updated the wiki to that effect. I just
> wanted to use the connector name as a mutex - we can handle the race
> in other ways.
>
> 4. Yes, I meant submitting config to the config topic. Have updated
> the wiki to make it clearer.
>
>
> Nits:
>
> Thanks for pointing these - I have made the necessary changes.
>
>
> Thanks again,
>
> Ashwin
>
>
> On Mon, Dec 4, 2023 at 9:05 PM Chris Egerton 
> wrote:
>
> > Hi Ashwin,
> >
> > Thanks for the KIP! This would be a nice simplification to the process
> for
> > migrating connectors enabled by KIP-980, and would also add global
> support
> > for a feature I've seen implemented by hand in at least a couple
> connectors
> > over the years.
> >
> >
> > High-level thoughts:
> >
> > 1. If there are leftover offsets from a previous connector, what will the
> > impact on those offsets (if any) be if a new connector is created with
> the
> > same name with initial offsets specified? I can think of at least two
> > options: we leave those offsets as-are but allow any of the initial
> offsets
> > in the new connector request to overwrite them, or we automatically wipe
> > all existing offsets for the connector first before writing its initial
> > offsets and then creating it. I have a slight preference for the first
> > because it's simpler to implement and aligns with existing precedent for
> > offset handling where we never wipe them unless explicitly requested by
> the
> > user or connector, but it could be argued that the second is less likely
> to
> > generate footguns for users. Interested in your thoughts!
> >
> > 2. IMO preflight validation (i.e., the "Validate initial_offset before
> > creating the connector in STOPPED state rejected alternative) is a
> > must-have for this kind of feature. I acknowledge the possible race
> > condition (and I don't think it's worth it to track in-flight connector
> > creation requests in the herder in order to prevent this race, since
> > ever-blocking Connector operations would be cumbersome to deal with), and
> > the extra implementation effort. But I don't think either of those tip
> the
> > scales far enough to override the benefit of ensuring the submitted
> offsets
> > are valid before creating the connector.
> >
> > 3. On the topic of preflight validation--I also think it's important that
> > we validate both the connector config and the initial offsets before
> either
> > creating the connector or storing the initial offsets. I don't think this
> > point requires any changes to the KIP that aren't already proposed with
> > point 2. above, but wanted to see if we could adopt this as a primary
> goal
> > for the design of the feature and keep it in mind with future changes.
> >
> > 4. In the proposed changes section, the first step is "Create a connector
> > in STOPPED state", and the second step is "Validate the offset...". What
> > exactly is entailed by "Create a connector"? Submit the config to the
> > config topic (presumably after a preflight validation of the config)?
> > Participate in the ensuing rebalance? I'm a little hesitant to start
> > performing rebalances inside REST requests, wondering if we can find a
> > lighter-weight way to implement this.
> >
> >
> > Nits:
> >
> > 5. You can remove the italicized "This page is meant as a template for
> > writing a KIP" section after the table of contents.
> >
> > 6. Can you file a JIRA ticket for this change and add a link to it in the
> > KIP under the status section?
> >
> > 7. What do you think about changing the name for the new field from
> > "initial_offset" (singular) to "initial_offsets" (plural)? This is
> > especially relevant for connectors that read from multiple source
> > partitions, like MM2 and the various JDBC source connectors out there.
> >
> > 8. IMO it's not necessary to include this section: "Please note that sink
> > and source connectors have different schemas for offset." While it's
> > technically true that the fields of the objects inside the "partition"
> and
> > "offs

[jira] [Resolved] (KAFKA-15871) Implement kafka-client-metrics.sh tool

[Jun Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-15871.
-
Resolution: Fixed

merged the PR to trunk

> Implement kafka-client-metrics.sh tool
> --
>
> Key: KAFKA-15871
> URL: https://issues.apache.org/jira/browse/KAFKA-15871
> Project: Kafka
>  Issue Type: Sub-task
>  Components: admin
>Affects Versions: 3.7.0
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Major
> Fix For: 3.7.0
>
>
> Implement the `kafka-client-metrics.sh` tool which is introduced in KIP-714 
> and enhanced in KIP-1000.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] MINOR: Add 3.6.1 to downloads page [kafka-site]

[via GitHub]

mimaison commented on code in PR #570:
URL: https://github.com/apache/kafka-site/pull/570#discussion_r1417763416


##
downloads.html:
##
@@ -6,12 +6,40 @@

 Download
 
-3.6.0 is the latest release. The current stable version is 3.6.0
+3.6.1 is the latest release. The current stable version is 3.6.1
 
 
 You can verify your download by following these https://www.apache.org/info/verification.html";>procedures and using 
these https://downloads.apache.org/kafka/KEYS";>KEYS.
 
 
+
+3.6.1
+
+
+Released Dec 8, 2023
+
+
+https://downloads.apache.org/kafka/3.6.1/RELEASE_NOTES.html";>Release 
Notes
+
+
+Source download: https://downloads.apache.org/kafka/3.6.1/kafka-3.6.1-src.tgz";>kafka-3.6.1-src.tgz
 (https://downloads.apache.org/kafka/3.6.1/kafka-3.6.1-src.tgz.asc";>asc,
 https://downloads.apache.org/kafka/3.6.1/kafka-3.6.1-src.tgz.sha512";>sha512)
+
+
+Binary downloads:
+
+Scala 2.12  - https://downloads.apache.org/kafka/3.6.1/kafka_2.12-3.6.1.tgz";>kafka_2.12-3.6.1.tgz
 (https://downloads.apache.org/kafka/3.6.1/kafka_2.12-3.6.1.tgz.asc";>asc,
 https://downloads.apache.org/kafka/3.6.1/kafka_2.12-3.6.1.tgz.sha512";>sha512)
+Scala 2.13  - https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz";>kafka_2.13-3.6.1.tgz
 (https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz.asc";>asc,
 https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz.sha512";>sha512)
+
+We build for multiple versions of Scala. This only matters if you 
are using Scala and you want a version
+built for the same Scala version you use. Otherwise any version 
should work (2.13 is recommended).
+
+
+
+
+Kafka 3.6.1 fixes 30 issues since the 3.6.0 release.
+For more information, please read the detailed https://downloads.apache.org/kafka/3.6.1/RELEASE_NOTES.html";>Release 
Notes

Review Comment:
   We used to only publish announcements to the Apache blog for major and minor 
releases. As we now have our own blog, it's easy to add an entry. I've pushed 
an update adding a short blog post.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-15932) Flaky test - PlaintextConsumerTest.testSeek("kraft+kip-848","consumer")

[Andrew Schofield (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Schofield resolved KAFKA-15932.
--
Resolution: Fixed

> Flaky test - PlaintextConsumerTest.testSeek("kraft+kip-848","consumer")
> ---
>
> Key: KAFKA-15932
> URL: https://issues.apache.org/jira/browse/KAFKA-15932
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 3.7.0
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Major
>  Labels: flaky-test
>
> Intermittently failing test for the new consumer.
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14859/1/tests/
> ```Error
> org.opentest4j.AssertionFailedError: Timed out before consuming expected 1 
> records. The number consumed was 0.
> Stacktrace
> org.opentest4j.AssertionFailedError: Timed out before consuming expected 1 
> records. The number consumed was 0.
>   at 
> app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
>   at app//org.junit.jupiter.api.Assertions.fail(Assertions.java:134)
>   at 
> app//kafka.api.AbstractConsumerTest.consumeRecords(AbstractConsumerTest.scala:161)
>   at 
> app//kafka.api.AbstractConsumerTest.consumeAndVerifyRecords(AbstractConsumerTest.scala:128)
>   at 
> app//kafka.api.PlaintextConsumerTest.testSeek(PlaintextConsumerTest.scala:616)
>   at 
> java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>  Method)
>   at 
> java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base@11.0.16.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base@11.0.16.1/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
>   at 
> app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>   at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>   at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
>   at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
>   at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94)
>   at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
>   at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
>   at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
>   at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
>   at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
>   at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
>   at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
>   at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
>   at 
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
>   at 
> app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>   at 
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
>   at 
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
>   at 
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
>   at 
> app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
>   at 
> app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>   at 
> app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
>   at 
> app//org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
>   at 
> app//org.

[jira] [Resolved] (KAFKA-15910) New group coordinator needs to generate snapshots while loading

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-15910.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

> New group coordinator needs to generate snapshots while loading
> ---
>
> Key: KAFKA-15910
> URL: https://issues.apache.org/jira/browse/KAFKA-15910
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jeff Kim
>Assignee: Jeff Kim
>Priority: Major
> Fix For: 3.7.0
>
>
> After the new coordinator loads a __consumer_offsets partition, it logs the 
> following exception when making a read operation (fetch/list groups, etc):
>  
> {{{}java.lang.RuntimeException: No in-memory snapshot for epoch 740745. 
> Snapshot epochs are:{}}}{{{}at 
> org.apache.kafka.timeline.SnapshotRegistry.getSnapshot(SnapshotRegistry.java:178){}}}{{{}at
>  
> org.apache.kafka.timeline.SnapshottableHashTable.snapshottableIterator(SnapshottableHashTable.java:407){}}}{{{}at
>  
> org.apache.kafka.timeline.TimelineHashMap$ValueIterator.(TimelineHashMap.java:283){}}}{{{}at
>  
> org.apache.kafka.timeline.TimelineHashMap$Values.iterator(TimelineHashMap.java:271){}}}
> {{...}}
>  
> This happens because we don't have a snapshot at the last updated high 
> watermark after loading. We cannot generate a snapshot at the high watermark 
> after loading all batches because it may contain records that have not yet 
> been committed. We also don't know where the high watermark will advance up 
> to so we need to generate a snapshot for each offset the loader observes to 
> be greater than the current high watermark. Then once we add the high 
> watermark listener and update the high watermark we can delete all of the 
> snapshots prior. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15026) Implement min-cost flow balancing tasks for same subtopology

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-15026.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

> Implement min-cost flow balancing tasks for same subtopology
> 
>
> Key: KAFKA-15026
> URL: https://issues.apache.org/jira/browse/KAFKA-15026
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Hao Li
>Assignee: Hao Li
>Priority: Major
> Fix For: 3.7.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15022) Support rack aware task assignment in Kafka streams

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-15022.
-
Fix Version/s: 3.7.0
   3.6.0
   Resolution: Fixed

> Support rack aware task assignment in Kafka streams 
> 
>
> Key: KAFKA-15022
> URL: https://issues.apache.org/jira/browse/KAFKA-15022
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Hao Li
>Assignee: Hao Li
>Priority: Major
>  Labels: kip, kip-925
> Fix For: 3.7.0, 3.6.0
>
>
> For KIP-925: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15347) Single-Key_multi-timestamp IQs with versioned state stores

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-15347.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

> Single-Key_multi-timestamp IQs with versioned state stores
> --
>
> Key: KAFKA-15347
> URL: https://issues.apache.org/jira/browse/KAFKA-15347
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Alieh Saeedi
>Assignee: Alieh Saeedi
>Priority: Major
> Fix For: 3.7.0
>
>
> [KIP-968|https://cwiki.apache.org/confluence/display/KAFKA/KIP-968%3A+Support+single-key_multi-timestamp+interactive+queries+%28IQv2%29+for+versioned+state+stores]
> This ticket covers just four query types:
> *Key Queries with multiple timestamps:*
>  # single-key query with upper bound timestamp
>  # single-key query with lower bound timestamp
>  # single-key query with timestamp range
>  # single-key all versions query



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15978) New consumer sends OffsetCommit with empty member ID

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-15978:


 Summary: New consumer sends OffsetCommit with empty member ID
 Key: KAFKA-15978
 URL: https://issues.apache.org/jira/browse/KAFKA-15978
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 3.7.0
Reporter: Andrew Schofield
Assignee: Andrew Schofield
 Fix For: 3.7.0


Running the trogdor tests with the new consumer, it seemed that offsets were 
not being committed correctly, although the records were being fetched 
successfully. Upon investigation, it seems that the commit request manager uses 
a cached member ID which means that its OffsetCommit requests are rejected by 
the group coordinator.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2453

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15977) DelegationTokenEndToEndAuthorizationWithOwnerTest leaks threads

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-15977:
--

 Summary: DelegationTokenEndToEndAuthorizationWithOwnerTest leaks 
threads
 Key: KAFKA-15977
 URL: https://issues.apache.org/jira/browse/KAFKA-15977
 Project: Kafka
  Issue Type: Bug
Reporter: Lucas Brutschy


[https://ci-builds.apache.org/blue/rest/organizations/jenkins/pipelines/Kafka/pipelines/kafka-pr/branches/PR-14878/runs/8/nodes/11/steps/90/log/?start=0]

 

I had an unrelated PR fail with the following thread leak:

 

```

Gradle Test Run :core:test > Gradle Test Executor 95 > 
DelegationTokenEndToEndAuthorizationWithOwnerTest > executionError STARTED 
kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest.executionError 
failed, log available in 
/home/jenkins/workspace/Kafka_kafka-pr_PR-14878/core/build/reports/testOutput/kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest.executionError.test.stdout
 Gradle Test Run :core:test > Gradle Test Executor 95 > 
DelegationTokenEndToEndAuthorizationWithOwnerTest > executionError FAILED 
org.opentest4j.AssertionFailedError: Found 1 unexpected threads during 
@AfterAll: `kafka-admin-client-thread | adminclient-483` ==> expected:  
but was: 

```

 

All the following tests on that error fail with initialization errors, because 
the admin client thread is never closed.

 

This is preceded by the following test failure:

 

```

Gradle Test Run :core:test > Gradle Test Executor 95 > 
DelegationTokenEndToEndAuthorizationWithOwnerTest > 
testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl(String, boolean) > [1] 
quorum=kraft, isIdempotenceEnabled=true FAILED 
org.opentest4j.AssertionFailedError: expected acls: 
(principal=User:scram-user2, host=*, operation=CREATE_TOKENS, 
permissionType=ALLOW) (principal=User:scram-user2, host=*, 
operation=DESCRIBE_TOKENS, permissionType=ALLOW) but got: at 
app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38) at 
app//org.junit.jupiter.api.Assertions.fail(Assertions.java:134) at 
app//kafka.utils.TestUtils$.waitAndVerifyAcls(TestUtils.scala:1142) at 
app//kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest.$anonfun$configureSecurityAfterServersStart$1(DelegationTokenEndToEndAuthorizationWithOwnerTest.scala:71)
 at 
app//kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest.$anonfun$configureSecurityAfterServersStart$1$adapted(DelegationTokenEndToEndAuthorizationWithOwnerTest.scala:70)
 at app//scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576) at 
app//scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574) at 
app//scala.collection.AbstractIterable.foreach(Iterable.scala:933) at 
app//kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest.configureSecurityAfterServersStart(DelegationTokenEndToEndAuthorizationWithOwnerTest.scala:70)

```

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-995: Allow users to specify initial offsets while creating connectors

[Ashwin]

Hello Chris,

Thanks for the quick and detailed review. Please see my responses below

High-level thoughts:

1. I had not thought of this till now, thanks for pointing it out. I
would lean towards the second option of cleaning previous offsets as
it will result in the fewer surprises for the user.

2 and 3. I agree and have updated the wiki to that effect. I just
wanted to use the connector name as a mutex - we can handle the race
in other ways.

4. Yes, I meant submitting config to the config topic. Have updated
the wiki to make it clearer.


Nits:

Thanks for pointing these - I have made the necessary changes.


Thanks again,

Ashwin


On Mon, Dec 4, 2023 at 9:05 PM Chris Egerton 
wrote:

> Hi Ashwin,
>
> Thanks for the KIP! This would be a nice simplification to the process for
> migrating connectors enabled by KIP-980, and would also add global support
> for a feature I've seen implemented by hand in at least a couple connectors
> over the years.
>
>
> High-level thoughts:
>
> 1. If there are leftover offsets from a previous connector, what will the
> impact on those offsets (if any) be if a new connector is created with the
> same name with initial offsets specified? I can think of at least two
> options: we leave those offsets as-are but allow any of the initial offsets
> in the new connector request to overwrite them, or we automatically wipe
> all existing offsets for the connector first before writing its initial
> offsets and then creating it. I have a slight preference for the first
> because it's simpler to implement and aligns with existing precedent for
> offset handling where we never wipe them unless explicitly requested by the
> user or connector, but it could be argued that the second is less likely to
> generate footguns for users. Interested in your thoughts!
>
> 2. IMO preflight validation (i.e., the "Validate initial_offset before
> creating the connector in STOPPED state rejected alternative) is a
> must-have for this kind of feature. I acknowledge the possible race
> condition (and I don't think it's worth it to track in-flight connector
> creation requests in the herder in order to prevent this race, since
> ever-blocking Connector operations would be cumbersome to deal with), and
> the extra implementation effort. But I don't think either of those tip the
> scales far enough to override the benefit of ensuring the submitted offsets
> are valid before creating the connector.
>
> 3. On the topic of preflight validation--I also think it's important that
> we validate both the connector config and the initial offsets before either
> creating the connector or storing the initial offsets. I don't think this
> point requires any changes to the KIP that aren't already proposed with
> point 2. above, but wanted to see if we could adopt this as a primary goal
> for the design of the feature and keep it in mind with future changes.
>
> 4. In the proposed changes section, the first step is "Create a connector
> in STOPPED state", and the second step is "Validate the offset...". What
> exactly is entailed by "Create a connector"? Submit the config to the
> config topic (presumably after a preflight validation of the config)?
> Participate in the ensuing rebalance? I'm a little hesitant to start
> performing rebalances inside REST requests, wondering if we can find a
> lighter-weight way to implement this.
>
>
> Nits:
>
> 5. You can remove the italicized "This page is meant as a template for
> writing a KIP" section after the table of contents.
>
> 6. Can you file a JIRA ticket for this change and add a link to it in the
> KIP under the status section?
>
> 7. What do you think about changing the name for the new field from
> "initial_offset" (singular) to "initial_offsets" (plural)? This is
> especially relevant for connectors that read from multiple source
> partitions, like MM2 and the various JDBC source connectors out there.
>
> 8. IMO it's not necessary to include this section: "Please note that sink
> and source connectors have different schemas for offset." While it's
> technically true that the fields of the objects inside the "partition" and
> "offset" fields will likely differ between sink and source connectors,
> they'll also likely differ in the exact same way across different sink
> connectors. I think it's enough to link to the relevant section in KIP-875
> on the details for the format.
>
> 9. Instead of "Connector-defined source partition" and "Connector-defined
> source offset" in the comments for the sample connector creation body
> (which aren't strictly accurate for sink connectors), can we say something
> like "Source partition" and "Desired initial offset"?
>
> 10. In the compatibility section the KIP states that "This new feature will
> use the current OffsetStorageWriter." IMO we should refrain from referring
> to internal API class names in KIPs when possible, since those class names
> may change and users may also mistakenly assume that they're part of the
> public API

Re: [DISCUSS] KIP-969: Support range interactive queries for versioned state stores

[Lucas Brutschy]

Hi Alieh,

I think we do not have to restrict ourselves too much for the future
and complicate the implementation. The user can always store away and
sort, so we should only provide the ordering guarantee we can provide
efficiently, and we shouldn't restrict our future evolution too much
by this. I think a global ordering by timestamp is sufficient for this
KIP, so I vote for option 2.

Cheers,
Lucas

On Fri, Dec 1, 2023 at 8:45 PM Alieh Saeedi
 wrote:
>
> Hi all,
> I updated the KIP based on the changes made in the former KIP (KIP-968). So
> with the `ResultOrder` enum, the class `MultiVersionedRangeQuery` had some
> changes both in the defined fields and defined methods.
>
> Based on the PoC PR, what we currently promise in the KIP about ordering
> seems like a dream. I intended to enable the user to have a global ordering
> based on the key or timestamp (using `orderByKey()` or
> `orderByTimestamp()`) and then even have a partial ordering based on the
> other parameter.  For example, if the user specifies
> `orderByKey().withDescendingKey().withAscendingTimestamps()`, then the
> global ordering is based on keys in a descending order, and then all the
> records with the same key are ordered ascendingly based on ts. The result
> will be something like (k3,v1,t1), (k3,v2,t2), (k2,v2,t3), (k1.v1.t1)
> (assuming that k1
> But in reality, we have limitations for having a global ordering based on
> keys since we are iterating over the segments in a lazy manner. Therefore,
> when we are processing the current segment, we have no knowledge of the
> keys in the next segment.
>
> Now I have two suggestions:
> 1. Changing the `MultiVersionedRangeQuery` class as follows:
>
> private final ResultOrder *segmentOrder*;
> private final contentOrder *segmentContentOrder*; // can be KEY_WISE or
> TIMESTAMP_WISE
> private final ResultOrder  *keyOrder*;
> private final ResultOrder *timestampOrder*;
>
> This way, the global ordering is specified by the `segmentOrder`. It means
> we either show the results from the oldest to the latest segment
> (ASCENDING) or from the latest to the oldest segment (DESCENDING).
> Then, inside each segment, we guarantee a `segmentContentOrder` which can
> be `KEY_WISE` or `TIMESTAMP_WISE`. The key order and timestamp order are
> specified by the `keyOrder` and `timestampOrder` properties, respectively.
> If the content of a segment must be ordered key-wise and then we have two
> records with the same key (it happens in older segments), then the
> `timestampOrder` determines the order between them.
>
> 2. We define that global ordering can only be based on timestamps (the
> `timestampOrder` property), and if two records have the same timestamp, the
> `keyOrder` determines the order between them.
>
> I think the first suggestion gives more flexibility to the user, but it is
> more complicated. I mean, it needs good Javadocs.
>
> I look forward to your ideas.
>
> Cheers,
> Alieh
>
>
> On Mon, Nov 6, 2023 at 3:08 PM Alieh Saeedi  wrote:
>
> > Thank you, Bruno and Matthias.
> > I updated the KIP as follows:
> > 1. The one remaining `asOf` word in the KIP is removed.
> > 2. Example 2 is updated. Thanks, Bruno for the correction.
> >
> > Discussions and open questions
> > 1. Yes, Bruno. We need `orderByKey()` and `orderByTimestamp()` as well.
> > Because the results must have a global ordering. Either based on key or
> > based on ts. For example, we can have
> > `orderByKey().withDescendingKey().withAscendingTimestamps()`. Then the
> > global ordering is based on keys in a descending order, and then all the
> > records with the same key are ordered ascendingly based on ts. The result
> > will be something like (k3,v1,t1), (k3,v2,t2), (k3,v1,t1), (k2,v2,t2),
> > (k1.v1.t1) (assuming that k1 > 2. About having the `latest()` method: it seems like we are undecided yet.
> > Adding a new class or ignoring `latest()` for VersionedRangeQuery and
> > instead using the `TimestampedRangeQuery` as Matthias suggested.
> >
> > Cheers,
> > Alieh
> >
> > On Sat, Nov 4, 2023 at 1:38 AM Matthias J. Sax  wrote:
> >
> >> Great discussion. Seems we are making good progress.
> >>
> >> I see advantages and disadvantages in splitting out a "single-ts
> >> key-range" query type. I guess, the main question might be, what
> >> use-cases do we anticipate and how common we believe they are?
> >>
> >> We should also take KIP-992 (adding `TimestampedRangeQuery`) into account.
> >>
> >> (1) The most common use case seems to be, a key-range over latest. For
> >> this one, we could use `TimestampedRangeQuery` -- it would return a
> >> `ValueAndTimestamp` instead of a `VersionedRecord` but the won't
> >> be any information loss, because "toTimestamp" would be `null` anyway.
> >>
> >>
> >> (2) The open question is, how common is a key-range in a point in the
> >> past? For this case, using
> >> `MultiVersionedRangeQuery.withKeyRange().from(myTs).to(myTs)` seems
> >> actually not to be a bad UX, and also does not really need to be
> >> 

Re: [DISCUSS] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

[Taras Ledkov]

Hi Greg,

> Taras, are you interested in dynamic SSL reconfiguration in Connect?
> Would you be willing to investigate the details of that for the KIP?
I would prefer to separate this functionality into the next KIP. But
if you / (the community) consider it necessary to combine this in one
KIP/patch, then I’m ready to do it.

The entry point for dynamic reconfiguration for Connect is not clear for me.
For a broker it is the DynamicConfig functionality. Should we add a
REST endpoint to reconfigure Connect or use another way?

--
With best regards,
Taras Ledkov

On Tue, Dec 5, 2023 at 2:42 AM Greg Harris  wrote:
>
> Hi Chris,
>
> Thank you for your comments above. I disagree with your recommendation
> for a new SslEngineFactory variant/hierarchy.
>
> 1. A superinterface could be more confusing to users. Since this is an
> interface in `clients`, the connect-specific interface would also need
> to be in clients, despite being superfluous for clients users and
> broker developers.
> 2. A user could implement the reduced interface, and then have issues
> loading their implementation in a broker, and need to find
> documentation/javadocs to explain to them why.
> 3. This interface has been in use since 2020, so I don't believe that
> the burden of implementing these methods has been excessive. I assume
> there's at least one "static" implementation out there that would have
> benefitted from the proposed super-interface, but they managed to
> adapt to the standardized interface.
> 4. Implementations that don't want to provide basic implementations
> can throw UnsupportedOperationException from the extra methods as a
> last resort.
>
> On the other hand, how much would it take to support the full suite of
> SslEngineFactory operations in Connect? Could part of this KIP be
> improving Connect to make use of these methods? This would help unify
> the experience for users that implement the interface specifically for
> the dynamic reconfiguration support, and rely on it on the broker
> side.
>
> Taras, are you interested in dynamic SSL reconfiguration in Connect?
> Would you be willing to investigate the details of that for the KIP?
>
> Thanks,
> Greg
>
> On Mon, Dec 4, 2023 at 1:17 PM Chris Egerton  wrote:
> >
> > Hi Taras,
> >
> > Regarding slimming down the interface: IMO, we should do this right the
> > first time, and that includes not requiring unnecessary methods from users.
> > I think BaseSslEngineFactory is good enough as a superinterface.
> >
> >
> > Regarding the parsing logic: I think the KIP needs to be more explicit. We
> > should add something like this to the proposed changes section:
> >
> > "If any properties are present in the worker config with a prefix of
> > "listeners.https.", then only properties with that prefix will be passed to
> > the SSL engine factory. Otherwise, all top-level worker properties will be
> > passed to the SSL engine factory. Note that this differs slightly from
> > existing logic in that the set of properties (prefixed or otherwise) will
> > not be filtered based on a predefined set of keys; this will enable custom
> > SSL engine factories to define and accept custom properties."
> >
> > I also took a quick look at the prototype (I usually try not to do this
> > since we vote on KIP documents, not PRs). I don't think we should populate
> > default values for SSL-related properties before sending properties to the
> > SSL engine factory, since it may confuse users who have written custom SSL
> > engine factories to see that properties not specified in their Connect
> > worker config are being passed to their factory. Instead, the default SSL
> > engine factory used by Connect can perform this logic, and we can let other
> > custom factories be responsible for their own default values.
> >
> >
> > Cheers,
> >
> > Chris
> >
> > On Wed, Nov 29, 2023 at 8:36 AM Taras Ledkov  wrote:
> >
> > > Hi team,
> > >
> > > Ping for review / vote for KIP-967 [1].
> > > Voting thread is here [2]
> > >
> > > [1].
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-967%3A+Support+custom+SSL+configuration+for+Kafka+Connect+RestServer
> > > [2]. https://github.com/apache/kafka/pull/14203
> > > [2]. https://lists.apache.org/thread/wc4v5v3pynl15g1q547m2croqsqgmzpw
> > >
> > > --
> > > With best regards,
> > > Taras Ledkov
> > >

Re: [DISCUSS] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

[Taras Ledkov]

Hi Chris,

>  I don't think we should populate default values for SSL-related properties
> before sending properties to the SSL engine factory,
> since it may confuse users who have written custom SSL engine factories
> to see that properties not specified in their Connect worker config are being 
> passed to their factory.
I slightly disagree with this.. The behavior implemented in PR is
consistent with SSL configuration for clients and brokers.
This behavior does not prevent the implementation of custom SSL
factories for the broker and client.

But you are right about reviewing PR and KIP. Let's focus on the KIP.

--
With best regards,
Taras Ledkov

On Tue, Dec 5, 2023 at 12:18 AM Chris Egerton  wrote:
>
> Hi Taras,
>
> Regarding slimming down the interface: IMO, we should do this right the
> first time, and that includes not requiring unnecessary methods from users.
> I think BaseSslEngineFactory is good enough as a superinterface.
>
>
> Regarding the parsing logic: I think the KIP needs to be more explicit. We
> should add something like this to the proposed changes section:
>
> "If any properties are present in the worker config with a prefix of
> "listeners.https.", then only properties with that prefix will be passed to
> the SSL engine factory. Otherwise, all top-level worker properties will be
> passed to the SSL engine factory. Note that this differs slightly from
> existing logic in that the set of properties (prefixed or otherwise) will
> not be filtered based on a predefined set of keys; this will enable custom
> SSL engine factories to define and accept custom properties."
>
> I also took a quick look at the prototype (I usually try not to do this
> since we vote on KIP documents, not PRs). I don't think we should populate
> default values for SSL-related properties before sending properties to the
> SSL engine factory, since it may confuse users who have written custom SSL
> engine factories to see that properties not specified in their Connect
> worker config are being passed to their factory. Instead, the default SSL
> engine factory used by Connect can perform this logic, and we can let other
> custom factories be responsible for their own default values.
>
>
> Cheers,
>
> Chris
>
> On Wed, Nov 29, 2023 at 8:36 AM Taras Ledkov  wrote:
>
> > Hi team,
> >
> > Ping for review / vote for KIP-967 [1].
> > Voting thread is here [2]
> >
> > [1].
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-967%3A+Support+custom+SSL+configuration+for+Kafka+Connect+RestServer
> > [2]. https://github.com/apache/kafka/pull/14203
> > [2]. https://lists.apache.org/thread/wc4v5v3pynl15g1q547m2croqsqgmzpw
> >
> > --
> > With best regards,
> > Taras Ledkov
> >

Re: [PR] MINOR: Add 3.6.1 to downloads page [kafka-site]

[via GitHub]

jlprat commented on code in PR #570:
URL: https://github.com/apache/kafka-site/pull/570#discussion_r1417102361


##
downloads.html:
##
@@ -6,12 +6,40 @@

 Download
 
-3.6.0 is the latest release. The current stable version is 3.6.0
+3.6.1 is the latest release. The current stable version is 3.6.1
 
 
 You can verify your download by following these https://www.apache.org/info/verification.html";>procedures and using 
these https://downloads.apache.org/kafka/KEYS";>KEYS.
 
 
+
+3.6.1
+
+
+Released Dec 8, 2023
+
+
+https://downloads.apache.org/kafka/3.6.1/RELEASE_NOTES.html";>Release 
Notes
+
+
+Source download: https://downloads.apache.org/kafka/3.6.1/kafka-3.6.1-src.tgz";>kafka-3.6.1-src.tgz
 (https://downloads.apache.org/kafka/3.6.1/kafka-3.6.1-src.tgz.asc";>asc,
 https://downloads.apache.org/kafka/3.6.1/kafka-3.6.1-src.tgz.sha512";>sha512)
+
+
+Binary downloads:
+
+Scala 2.12  - https://downloads.apache.org/kafka/3.6.1/kafka_2.12-3.6.1.tgz";>kafka_2.12-3.6.1.tgz
 (https://downloads.apache.org/kafka/3.6.1/kafka_2.12-3.6.1.tgz.asc";>asc,
 https://downloads.apache.org/kafka/3.6.1/kafka_2.12-3.6.1.tgz.sha512";>sha512)
+Scala 2.13  - https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz";>kafka_2.13-3.6.1.tgz
 (https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz.asc";>asc,
 https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz.sha512";>sha512)
+
+We build for multiple versions of Scala. This only matters if you 
are using Scala and you want a version
+built for the same Scala version you use. Otherwise any version 
should work (2.13 is recommended).
+
+
+
+
+Kafka 3.6.1 fixes 30 issues since the 3.6.0 release.
+For more information, please read the detailed https://downloads.apache.org/kafka/3.6.1/RELEASE_NOTES.html";>Release 
Notes

Review Comment:
   There won't be a blog post for this release? Checking previous releases, we 
had one also for patch versions 
(https://kafka.apache.org/blog#apache_kafka_351_release_announcement)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] MINOR: Add 3.6.1 to downloads page [kafka-site]

[via GitHub]

mimaison opened a new pull request, #570:
URL: https://github.com/apache/kafka-site/pull/570

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-15976) KIP-995: Allow users to specify initial offsets while creating connectors

[Ashwin Pankaj (Jira)]

Ashwin Pankaj created KAFKA-15976:
-

 Summary: KIP-995: Allow users to specify initial offsets while 
creating connectors
 Key: KAFKA-15976
 URL: https://issues.apache.org/jira/browse/KAFKA-15976
 Project: Kafka
  Issue Type: Improvement
  Components: connect
Reporter: Ashwin Pankaj
Assignee: Ashwin Pankaj


Allow setting the initial offset for a connector in the connector creation API.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.7.0 Release

[Stanislav Kozlovski]

Hello again,

Time is flying by! It is feature freeze day!

By today, we expect to have major features merged and to begin working on
their stabilisation. Minor features should have PRs.

I am planning to cut the release branch soon - on Monday EU daytime. When I
do that, I will create a new e-mail thread titled "New release branch
3.7.0" to notify you, so be on the lookout for that. I will also notify
this thread.

Thank you for your contributions. Let's get this release shipped!

Best,
Stanislav


On Fri, Nov 24, 2023 at 6:11 PM Stanislav Kozlovski 
wrote:

> Hey all,
>
> The KIP Freeze has passed. I count 31 KIPs that will be going into the 3.7
> Release. Thank you all for your hard work!
>
> They are the following (some of these were accepted in previous releases
> and have minor parts going out, some targeting a Preview release and the
> rest being fully released as regular.):
>  - KIP-1000: List Client Metrics Configuration Resources
>  - KIP-1001: Add CurrentControllerId Metric
>  - KIP-405: Kafka Tiered Storage
>  - KIP-580: Exponential Backoff for Kafka Clients
>  - KIP-714: Client metrics and observability
>  - KIP-770: Replace "buffered.records.per.partition" &
> "cache.max.bytes.buffering" with
> "{statestore.cache}/{input.buffer}.max.bytes"
>  - KIP-848: The Next Generation of the Consumer Rebalance Protocol
>  - KIP-858: Handle JBOD broker disk failure in KRaft
>  - KIP-890: Transactions Server-Side Defense
>  - KIP-892: Transactional StateStores
>  - KIP-896: Remove old client protocol API versions in Kafka 4.0 -
> metrics/request log changes to identify deprecated apis
>  - KIP-925: Rack aware task assignment in Kafka Streams
>  - KIP-938: Add more metrics for measuring KRaft performance
>  - KIP-951 - Leader discovery optimizations for the client
>  - KIP-954: expand default DSL store configuration to custom types
>  - KIP-959: Add BooleanConverter to Kafka Connect
>  - KIP-960: Single-key single-timestamp IQv2 for state stores
>  - KIP-963: Additional metrics in Tiered Storage
>  - KIP-968: Support single-key_multi-timestamp Interactive Queries (IQv2)
> for Versioned State Stores
>  - KIP-970: Deprecate and remove Connect's redundant task configurations
> endpoint
>  - KIP-975: Docker Image for Apache Kafka
>  - KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect
>  - KIP-978: Allow dynamic reloading of certificates with different DN /
> SANs
>  - KIP-979: Allow independently stop KRaft processes
>  - KIP-980: Allow creating connectors in a stopped state
>  - KIP-985: Add reverseRange and reverseAll query over kv-store in IQv2
>  - KIP-988: Streams Standby Update Listener
>  - KIP-992: Proposal to introduce IQv2 Query Types: TimestampedKeyQuery
> and TimestampedRangeQuery
>  - KIP-998: Give ProducerConfig(props, doLog) constructor protected access
>
> Notable KIPs that didn't make the Freeze were KIP-977 - it only got 2/3
> votes.
>
> For the full list and latest source of truth, refer to the Release Plan
> 3.7.0 Document
> .
>
> Thanks for your contributions once again!
> Best,
> Stan
>
>
> On Thu, Nov 23, 2023 at 2:27 PM Nick Telford 
> wrote:
>
>> Hi Stan,
>>
>> I'd like to propose including KIP-892 in the 3.7 release. The KIP has been
>> accepted and I'm just working on rebasing the implementation against trunk
>> before I open a PR.
>>
>> Regards,
>> Nick
>>
>> On Tue, 21 Nov 2023 at 11:27, Mayank Shekhar Narula <
>> mayanks.nar...@gmail.com> wrote:
>>
>> > Hi Stan
>> >
>> > Can you include KIP-951 to the 3.7 release plan? All PRs are merged in
>> the
>> > trunk.
>> >
>> > On Wed, Nov 15, 2023 at 4:05 PM Stanislav Kozlovski
>> >  wrote:
>> >
>> > > Friendly reminder to everybody that the KIP Freeze is *exactly 7 days
>> > away*
>> > > - November 22.
>> > >
>> > > A KIP must be accepted by this date in order to be considered for this
>> > > release. Note, any KIP that may not be implemented in time, or
>> otherwise
>> > > risks heavily destabilizing the release, should be deferred.
>> > >
>> > > Best,
>> > > Stan
>> > >
>> > > On Fri, Nov 3, 2023 at 6:03 AM Sophie Blee-Goldman <
>> > sop...@responsive.dev>
>> > > wrote:
>> > >
>> > > > Looks great, thank you! +1
>> > > >
>> > > > On Thu, Nov 2, 2023 at 10:21 AM David Jacot
>> > > > > >
>> > > > wrote:
>> > > >
>> > > > > +1 from me as well. Thanks, Stan!
>> > > > >
>> > > > > David
>> > > > >
>> > > > > On Thu, Nov 2, 2023 at 6:04 PM Ismael Juma 
>> > wrote:
>> > > > >
>> > > > > > Thanks Stanislav, +1
>> > > > > >
>> > > > > > Ismael
>> > > > > >
>> > > > > > On Thu, Nov 2, 2023 at 7:01 AM Stanislav Kozlovski
>> > > > > >  wrote:
>> > > > > >
>> > > > > > > Hi all,
>> > > > > > >
>> > > > > > > Given the discussion here and the lack of any pushback, I have
>> > > > changed
>> > > > > > the
>> > > > > > > dates of the release:
>> > > > > > > - KIP Freeze - *November 22 *(moved 4 days later)
>> > > > > > > - Feature Freeze - *December 6 *(moved 2 days earl

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2452

[Apache Jenkins Server]

See 

Re: KIP-993: Allow restricting files accessed by File and Directory ConfigProviders

[Mickael Maison]

Hi,

I'm not aware of any other mechanisms to explore the filesystem. If
you have ideas, please reach out to the security list.

Thanks,
Mickael

On Tue, Dec 5, 2023 at 1:05 PM Gantigmaa Selenge  wrote:
>
> Hi everyone,
>
>
> Apologies for the very delayed response. Thank you both for the feedback.
>
>
> > For clarity it might make sense to mention this feature will be useful
>
> when using a ConfigProvider with Kafka Connect as providers are set in
>
> the runtime and can then be used by connectors. This feature has no
>
> use when using a ConfigProvider in server.properties or in clients.
>
>
> I have updated the KIP to address this suggestion. Please let me know if
> it's not clear enough.
>
>
> > When trying to use a path not allowed, you propose returning an error.
>
> With Connect does that mean the connector will be failed? The
>
> EnvVarConfigProvider returns empty string in case a user tries to
>
> access an environment variable not allowed. I wonder if we should
>
> follow the same pattern so the behavior is "consistent" across all
>
> built-in providers.
>
>
> I agree with this, it makes sense to have consistent behaviour across all
> the providers. I made this update.
>
>
> > 1. In the past Connect removed the FileStream connectors in order to
>
> prevent a REST API attacker from accessing the filesystem. Is this the
>
> only remaining attack vector for reading the file system? Meaning, if
>
> this feature is configured and all custom plugins are audited for
>
> filesystem accesses, would someone with access to the REST API be
>
> unable to access arbitrary files on disk?
>
>
> Once this feature is configured, it will stop someone from accessing the
> file system via config providers.
>
> However, I’m not sure whether there are other ways users can access file
> systems via REST API.
>
>
> Mickael, perhaps you have some thoughts on this?
>
>
> > 2. Could you explain how this feature would prevent a path traversal
>
> attack, and how we will verify that such attacks are not feasible?
>
>
> The intention is to generate File objects based on the String value
> provided for allowed.paths and the String path passed to the get() function.
>
> This would allow validation of path inclusion within the specified allowed
> paths using their corresponding Path objects, rather than doing String
> comparisons.
>
> This hopefully will mitigate the risk of path traversal. The implementation
> should include unit tests to verify this.
>
>
> > 3. This applies a single "allowed paths" to a whole worker, but I've
>
> seen situations where preventing one connector from accessing
>
> another's secrets may also be desirable. Is there any way to extend
>
> this feature now or in the future to make that possible?
>
>
> One approach could be creating multiple providers, each assigned a unique
> name and specific allowed.paths configuration. Users would then be assigned
> a provider name, granting them appropriate access on the file system to
> load variables for their connectors. However, during provider
> configuration, administrators would have to anticipate and specify the
> files and directories users may require access to.
>
>
> Regards,
>
> Tina
>
> On Wed, Nov 8, 2023 at 7:49 PM Greg Harris 
> wrote:
>
> > Hey Tina,
> >
> > Thanks for the KIP! Unrestricted file system access over a REST API is
> > an unfortunate anti-pattern, so I'm glad that you're trying to change
> > it. I had a few questions, mostly from the Connect perspective.
> >
> > 1. In the past Connect removed the FileStream connectors in order to
> > prevent a REST API attacker from accessing the filesystem. Is this the
> > only remaining attack vector for reading the file system? Meaning, if
> > this feature is configured and all custom plugins are audited for
> > filesystem accesses, would someone with access to the REST API be
> > unable to access arbitrary files on disk?
> > 2. Could you explain how this feature would prevent a path traversal
> > attack, and how we will verify that such attacks are not feasible?
> > 3. This applies a single "allowed paths" to a whole worker, but I've
> > seen situations where preventing one connector from accessing
> > another's secrets may also be desirable. Is there any way to extend
> > this feature now or in the future to make that possible?
> >
> > Thanks!
> > Greg
> >
> > On Tue, Nov 7, 2023 at 7:06 AM Mickael Maison 
> > wrote:
> > >
> > > Hi Tina,
> > >
> > > Thanks for the KIP.
> > > For clarity it might make sense to mention this feature will be useful
> > > when using a ConfigProvider with Kafka Connect as providers are set in
> > > the runtime and can then be used by connectors. This feature has no
> > > use when using a ConfigProvider in server.properties or in clients.
> > >
> > > When trying to use a path not allowed, you propose returning an error.
> > > With Connect does that mean the connector will be failed? The
> > > EnvVarConfigProvider returns empty string in case a user tries to
> > > access