[jira] [Resolved] (KAFKA-14367) Introduce `GroupCoordinator` interface

2023-01-19 Thread David Jacot (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-14367.
-
Fix Version/s: 3.5.0
   Resolution: Fixed

> Introduce `GroupCoordinator` interface
> --
>
> Key: KAFKA-14367
> URL: https://issues.apache.org/jira/browse/KAFKA-14367
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Jacot
>Assignee: David Jacot
>Priority: Major
> Fix For: 3.5.0
>
>
> The goal is to introduce a new GroupCoordinator interface and to convert the 
> existing coordinator to use it. The new coordinator will use it later on.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1524

2023-01-19 Thread Apache Jenkins Server 
See

[jira] [Created] (KAFKA-14643) TopicBased RemoteLogMetadataManager can't support reassign user-partitions

2023-01-19 Thread hzh0425 (Jira) 
hzh0425 created KAFKA-14643:
---

 Summary: TopicBased RemoteLogMetadataManager can't support 
reassign user-partitions
 Key: KAFKA-14643
 URL: https://issues.apache.org/jira/browse/KAFKA-14643
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: hzh0425
Assignee: hzh0425


{*}Background{*}:
In [KIP-405: Kafka Tiered Storage - Apache Kafka - Apache Software 
Foundation|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage],
  kafka introduced the feature of hierarchical storage.
Also, KAFKA-9555 Topic-based implementation for the RemoteLogMetadataManager - 
ASF JIRA (apache.org) implements the default RLMM - 'TopicBased-RLMM'.

{*}Problem{*}:
TopicBased-RLMM will only subscribe to the Partitions where the current Broker 
is Leader or Follower. If the current Broker is not the Leader or Follower, 
then RLMM will directly skip the metadata records related to these Partitions.

When reassign user-partitions occurs, rlmm will subscribe to new 
user-partitions, assuming that the metadata-partition to which the new 
user-partition belongs is 'metadata-partition0', and RLMM has consumed 
'metadata-partition0' *to offset = 100* before the reassign partition occurs, 
then {*}after reassign{*}, RMLM will *not* consume 'metadata-partition0' 
{*}from the beginning{*}, and finally cause the metadata records related to the 
new user-partition to *be lost with offset < 100.*

*Solution*

Let RLMM subscribe to all user-patitions, instead of only subscribing to 
partitions where the current broker is leader or follower.
In this way, when reassign partition occurs, RLMM will have new partition's 
metadata records.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14642) TopicBased RemoteLogMetadataManager can't support reassign user-partitions

2023-01-19 Thread hzh0425 (Jira) 
hzh0425 created KAFKA-14642:
---

 Summary: TopicBased RemoteLogMetadataManager can't support 
reassign user-partitions
 Key: KAFKA-14642
 URL: https://issues.apache.org/jira/browse/KAFKA-14642
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 3.3.1
Reporter: hzh0425
Assignee: hzh0425


{*}Background{*}:
In [KIP-405: Kafka Tiered Storage - Apache Kafka - Apache Software 
Foundation|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage],
  kafka introduced the feature of hierarchical storage.
Also, [KAFKA-9555] Topic-based implementation for the RemoteLogMetadataManager 
- ASF JIRA (apache.org) implements the default RLMM - 'TopicBased-RLMM'.

{*}Problem{*}:
TopicBased-RLMM will only subscribe to the Partitions where the current Broker 
is Leader or Follower. If the current Broker is not the Leader or Follower, 
then RLMM will directly skip the metadata records related to these Partitions.

When reassign user-partitions occurs, rlmm will subscribe to new 
user-partitions, assuming that the metadata-partition to which the new 
user-partition belongs is 'metadata-partition0', and RLMM has consumed 
'metadata-partition0' *to offset = 100* before the reassign partition occurs, 
then {*}after reassign{*}, RMLM will *not* consume 'metadata-partition0' 
\{*}from the beginning{*}, and finally cause the metadata records related to 
the new user-partition to *be lost with offset < 100.*

*Solution*

Let RLMM subscribe to all user-patitions, instead of only subscribing to 
partitions where the current broker is leader or follower.
In this way, when reassign partition occurs, RLMM will have new partition's 
metadata records.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1523

2023-01-19 Thread Apache Jenkins Server 
See

[jira] [Resolved] (KAFKA-14638) Documentation for transaction.timeout.ms should be more precise

2023-01-19 Thread Matthias J. Sax (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-14638.
-
Fix Version/s: 3.5.0
   Resolution: Fixed

> Documentation for transaction.timeout.ms should be more precise
> ---
>
> Key: KAFKA-14638
> URL: https://issues.apache.org/jira/browse/KAFKA-14638
> Project: Kafka
>  Issue Type: Improvement
>  Components: docs
>Reporter: Alex Sorokoumov
>Assignee: Alex Sorokoumov
>Priority: Minor
> Fix For: 3.5.0
>
>
> The documentation for {{transaction.timeout.ms}} is
> {quote}
> The maximum amount of time in ms that the transaction coordinator will wait 
> for a transaction status update from the producer before proactively aborting 
> the ongoing transaction. If this value is larger than the 
> transaction.max.timeout.ms setting in the broker, the request will fail with 
> a InvalidTxnTimeoutException error.
> {quote}
> It would be easier to reason about this timeout if the documentation would 
> elaborate on when the timer starts ticking and under what circumstances it 
> might reset.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-900: KRaft kafka-storage.sh API additions to support SCRAM

2023-01-19 Thread Colin McCabe 
Hi Proven,

Thanks for putting this together.

We always intended to have a way to bootstrap into using an all-SCRAM cluster, 
from scratch.

I have two big comments here. First, I think we need a better interface than 
raw records. And second, I'm not sure that kafka-storage.sh is the right place 
to put this.

I think raw records are going to be tough for people to use, because there are 
a lot of fields, and the values to set them to are not intuitive. For example, 
to set SHA512, the user needs to set "mechanism" equal to 2. That is going to 
be impossible to remember or figure out without looking at the source code. The 
other thing of course is that we may add more fields over time, including 
mandatory ones. So any documentation could quickly get out of date.

I think people are going to want to specify SCRAM users here the same way they 
do when using the kafka-configs.sh tool. As a reminder, using kafka-configs.sh, 
they specify users like this:

./bin/kafka-configs --bootstrap-server localhost:9092 --alter \
  --add-config 'SCRAM-SHA-256=[iterations=8192,password=pass]' \
  --entity-type users \
  --entity-name alice

Of course, in this example, we're not specifying a salt. So we'd have to 
evaluate whether that's what we want for our use-case as well. On the plus 
side, specifying a salt could ensure that the bootstrap files end up identical 
on every node. On the minus side, it is another random number that users would 
need to generate and explicitly pass in.

I would lean towards auto-generating the salt. I don't think the salt needs to 
be the same on all nodes. Only one controller will become active and write the 
bootstrap records to the log; no other controllers will do that. Brokers don't 
need to read the SCRAM records out of the bootstrap file.

If we put all the functionality into kafka-storage.sh, it might look something 
like this:

./bin/kafka-storage.sh format \
  --config [my-config-path] \
  --cluster-id mb0Zz1YPTUeVzpedHHPT-Q \
  --release-version 3.5-IV0 \
  --scram-user alice \
  --scram-config 'SCRAM-SHA-256=[iterations=8192,password=alicepass]' \
  --scram-user bob \
  --scram-config 'SCRAM-SHA-256=[password=bobpass]'

(Here I am assuming that each --scram-user must be followed by exactly on 
--scram-config line)

Perhaps it's worth considering whether it woudl be better to add a mode to 
kafka-configs.sh where it appends to a bootstrap file.

If we do put everything into kafka-storage.sh, we should consider the plight of 
people with low limits on the maximum length of their command lines. One fix 
for these people could be allowing them to read their arguments from a file 
like this:

$ ./bin/kafka-storage.sh @myfile
$ cat myfile:
  ./bin/kafka-storage.sh format \
--config [my-config-path] \
  ...
[etc, etc.]

Argparse4j supports this natively with fromFilePrefix. See 
https://argparse4j.github.io/usage.html#fromfileprefix

best,
Colin


On Thu, Jan 19, 2023, at 11:08, Proven Provenzano wrote:
> I have written a KIP describing the API additions needed to 
> kafka-storage
> to store SCRAM
> credentials at bootstrap time. Please take a look at
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-900%3A+KRaft+kafka-storage.sh+API+additions+to+support+SCRAM+for+Kafka+Brokers
>
> -- 
> --Proven

Re: [ANNOUNCE] New committer: Justine Olshan

2023-01-19 Thread Colin McCabe 
Congratuations, Justine! Happy to hear it.

I guess my message is a bit late too :)

best,
Colin

On Tue, Jan 17, 2023, at 17:23, Kowshik Prakasam wrote:
> Congrats, Justine!
>
>
> Cheers,
> Kowshik
>
> On Tue, Jan 17, 2023, 4:53 PM Guozhang Wang 
> wrote:
>
>> Congratulations, Justine (I'm also late)!
>>
>> On Wed, Jan 11, 2023 at 12:17 AM Bruno Cadonna  wrote:
>>
>> > Hi Justine,
>> >
>> > Re-reading my message I realized that my message might be
>> > misinterpreted. I meant that I am late with congratulating you due to
>> > the holidays, NOT that it took you long becoming a committer!
>> >
>> > Sorry for the potential confusion!
>> >
>> > Best,
>> > Bruno
>> >
>> > On 11.01.23 08:57, Bruno Cadonna wrote:
>> > > Better late than never!
>> > >
>> > > Congrats!
>> > >
>> > > Best,
>> > > Bruno
>> > >
>> > > On 04.01.23 20:25, Kirk True wrote:
>> > >> Congratulations!
>> > >>
>> > >> On Tue, Jan 3, 2023, at 7:34 PM, John Roesler wrote:
>> > >>> Congrats, Justine!
>> > >>> -John
>> > >>>
>> > >>> On Tue, Jan 3, 2023, at 13:03, Matthias J. Sax wrote:
>> >  Congrats!
>> > 
>> >  On 12/29/22 6:47 PM, ziming deng wrote:
>> > > Congratulations Justine!
>> > > —
>> > > Best,
>> > > Ziming
>> > >
>> > >> On Dec 30, 2022, at 10:06, Luke Chen  wrote:
>> > >>
>> > >> Congratulations, Justine!
>> > >> Well deserved!
>> > >>
>> > >> Luke
>> > >>
>> > >> On Fri, Dec 30, 2022 at 9:15 AM Ron Dagostino 
>> > >> wrote:
>> > >>
>> > >>> Congratulations, Justine!Well-deserved., and I’m very happy
>> > >>> for you.
>> > >>>
>> > >>> Ron
>> > >>>
>> >  On Dec 29, 2022, at 6:13 PM, Israel Ekpo 
>> >  wrote:
>> > 
>> >  Congratulations Justine!
>> > 
>> > 
>> > > On Thu, Dec 29, 2022 at 5:05 PM Greg Harris
>> > >>> 
>> > > wrote:
>> > >
>> > > Congratulations Justine!
>> > >
>> > >> On Thu, Dec 29, 2022 at 1:37 PM Bill Bejeck
>> > >>  wrote:
>> > >>
>> > >> Congratulations Justine!
>> > >>
>> > >>
>> > >> -Bill
>> > >>
>> > >>> On Thu, Dec 29, 2022 at 4:36 PM Philip Nee <
>> > philip...@gmail.com>
>> > >>> wrote:
>> > >>
>> > >>> wow congrats!
>> > >>>
>> > >>> On Thu, Dec 29, 2022 at 1:05 PM Chris Egerton <
>> > >>> fearthecel...@gmail.com
>> > >>
>> > >>> wrote:
>> > >>>
>> >  Congrats, Justine!
>> > 
>> >  On Thu, Dec 29, 2022, 15:58 David Jacot 
>> >  wrote:
>> > 
>> > > Hi all,
>> > >
>> > > The PMC of Apache Kafka is pleased to announce a new Kafka
>> > > committer
>> > > Justine
>> > > Olshan.
>> > >
>> > > Justine has been contributing to Kafka since June 2019. She
>> > >> contributed
>> >  53
>> > > PRs including the following KIPs.
>> > >
>> > > KIP-480: Sticky Partitioner
>> > > KIP-516: Topic Identifiers & Topic Deletion State
>> > Improvements
>> > > KIP-854: Separate configuration for producer ID expiry
>> > > KIP-890: Transactions Server-Side Defense (in progress)
>> > >
>> > > Congratulations, Justine!
>> > >
>> > > Thanks,
>> > >
>> > > David (on behalf of the Apache Kafka PMC)
>> > >
>> > 
>> > >>>
>> > >>
>> > >
>> > >>>
>> > >
>> > >
>> > >>>
>> > >>
>> >
>>

Re: [ANNOUNCE] New committer: Stanislav Kozlovski

2023-01-19 Thread Colin McCabe 
Congratulations, Stan! Well deserved.

best,
Colin

On Tue, Jan 17, 2023, at 07:50, Jun Rao wrote:
> Hi, Everyone,
>
> The PMC of Apache Kafka is pleased to announce a new Kafka committer
> Stanislav Kozlovski.
>
> Stan has been contributing to Apache Kafka since June 2018. He made various
> contributions including the following KIPs.
>
> KIP-455: Create an Administrative API for Replica Reassignment
> KIP-412: Extend Admin API to support dynamic application log levels
>
> Congratulations, Stan!
>
> Thanks,
>
> Jun (on behalf of the Apache Kafka PMC)

[jira] [Created] (KAFKA-14641) Cleanup CommitNeeded after EOS-V1 is removed

2023-01-19 Thread Guozhang Wang (Jira) 
Guozhang Wang created KAFKA-14641:
-

 Summary: Cleanup CommitNeeded after EOS-V1 is removed
 Key: KAFKA-14641
 URL: https://issues.apache.org/jira/browse/KAFKA-14641
 Project: Kafka
  Issue Type: Improvement
Reporter: Guozhang Wang


This is a follow-up of KAFKA-14294.

Today we have several flags to determine if KS need to execute a commit: 1) 
task-level "commitNeeded" which is set whenever process() or punctuator() is 
called, 2) if there are input topic offsets to commit, retrieved from the 
"task.prepareCommit()", 3) the "transactionInFlight" flag from producer as a 
fix of KAFKA-14294 (this subsumes the first "commitNeeded" functionality).

Given that we are still having EOS-v1, cleanup this would be a bit complex. But 
after the deprecated EOS-V1 is removed, we can cleanup those controls since for 
any commit cases, we would need to commit all tasks anyways whereas in EOS-v1, 
we would commit probably a subset of tasks since they are done by different 
producers and hence different txns.

A quick thought is the following:

1) We would not need the per-task "commitNeeded" anymore.
2) We would maintain a single "commitNeeded" flag on the task-executor, hence 
on the thread level. It is set whenever `process()` or `punctuator` is called.
3) Whenever we need to commit, either a) periodically, b) upon revocation, c) 
upon user request, we simply check that flag, and if necessary commit all tasks 
and reset the flag.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-869: Improve Streams State Restoration Visibility

2023-01-19 Thread Guozhang Wang 
Hello Matthias,

Thanks for the feedback. I was on vacation for a while. Pardon for the
late replies. Please see them inline below

On Thu, Dec 1, 2022 at 11:23 PM Matthias J. Sax  wrote:
>
> Seems I am late to the party... Great KIP. Couple of questions from my side:
>
> (1) What is the purpose of `standby-updating-tasks`? It seems to be the
> same as the number of assigned standby task? Not sure how useful it
> would be?
>
In general, yes, it is the number of assigned standby tasks --- there
will be transit times when the assigned standby tasks are not yet
being updated but it would not last long --- but we do not yet have a
direct gauge to expose this before, and users have to infer this from
other indirect metrics.

>
>
> (2) `active-paused-tasks` / `standby-paused-tasks` -- what does "paused"
> exactly mean? There was a discussion about renaming the callback method
> from pause to suspended. So should this be called `suspended`, too? And
> if yes, how is it useful for users?
>
Pausing here refers to "KIP-834: Pause / Resume KafkaStreams
Topologies" 
(https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832).
When a topology is paused, all its tasks including standbys will be
paused too.

I'm not aware of a discussion to rename the call name to "suspend" for
KIP-834. Could you point me to the reference?

>
>
> (3) `restore-ratio`: the description says
>
> > The fraction of time the thread spent on restoring active or standby tasks
>
> I find the term "restoring" does only apply to active tasks, but not to
> standbys. Can we reword this?
>
Yeah I have been discussing this with others in the community a bit as
well, but so far I have not been convinced of a better name than it.
Some other alternatives being discussed but not win everyone's love is
"restore-or-update-ratio", "process-ratio" (for the restore thread
that means restoring or updating), and "io-ratio".

The only one so far that I feel is probably better, is
"state-update-ratio". If folks feel this one is better than
"restore-ratio" I'm happy to update.

>
> (4) `restore-call-rate`: not sure what you exactly mean by "restore calls"?
>
This is similar to the "io-calls-rate" in the selector classes, i.e.
the number of "restore" function calls made. It's argurably a very
low-level metrics but I included it since it could be useful in some
debugging scenarios.

>
> (5) `restore-remaining-records-total` -- why is this a task metric?
> Seems we could roll it up into a thread metric that we report at INFO
> level (we could still have per-task DEBUG level metric for it in addition).
>
The rationale behind it is the general principle in metrics design
that "Kafka would provide the lowest necessary metrics levels, and
users can do the roll-ups however they want".

>
> (6) What about "warmup tasks"? Internally, we treat them as standbys,
> but it seems it's hard for users to reason about it in the scale-out
> warm-up case. Would it be helpful (and possible) to report "warmup
> progress" explicitly?
>
At the restore thread level, we cannot differentiate standby tasks
from warmup tasks since the latter is created exactly just like the
former. But I do agree this is an issue for visibility that worth
addressing, I think another KIP would be needed to first consider
distinguishing these two at the class level.

>
> -Matthias
>
>
> On 11/1/22 2:44 AM, Lucas Brutschy wrote:
> > We need this!
> >
> > + 1 non binding
> >
> > Cheers,
> > Lucas
> >
> > On Tue, Nov 1, 2022 at 10:01 AM Bruno Cadonna  wrote:
> >>
> >> Guozhang,
> >>
> >> Thanks for the KIP!
> >>
> >> +1 (binding)
> >>
> >> Best,
> >> Bruno
> >>
> >> On 25.10.22 22:07, Walker Carlson wrote:
> >>> +1 non binding
> >>>
> >>> Thanks for the kip!
> >>>
> >>> On Thu, Oct 20, 2022 at 10:25 PM John Roesler  wrote:
> >>>
>  Thanks for the KIP, Guozhang!
> 
>  I'm +1 (binding)
> 
>  -John
> 
>  On Wed, Oct 12, 2022, at 16:36, Nick Telford wrote:
> > Can't wait!
> > +1 (non-binding)
> >
> > On Wed, 12 Oct 2022, 18:02 Guozhang Wang, 
> > wrote:
> >
> >> Hello all,
> >>
> >> I'd like to start a vote for the following KIP, aiming to improve Kafka
> >> Stream's restoration visibility via new metrics and callback methods:
> >>
> >>
> >>
>  https://cwiki.apache.org/confluence/display/KAFKA/KIP-869%3A+Improve+Streams+State+Restoration+Visibility
> >>
> >>
> >> Thanks!
> >> -- Guozhang
> >>
> 
> >>>

[jira] [Created] (KAFKA-14640) Update AddPartitionsToTxn protocol to batch and handle verifyOnly requests

2023-01-19 Thread Justine Olshan (Jira) 
Justine Olshan created KAFKA-14640:
--

 Summary: Update AddPartitionsToTxn protocol to batch and handle 
verifyOnly requests
 Key: KAFKA-14640
 URL: https://issues.apache.org/jira/browse/KAFKA-14640
 Project: Kafka
  Issue Type: Sub-task
Reporter: Justine Olshan


As part of KIP-890 we are making some changes to this protocol.

1. We can send a request to verify a partition is added to a transaction

2. We can batch multiple transactional IDs



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.4 #44

2023-01-19 Thread Apache Jenkins Server 
See

Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect

2023-01-19 Thread Chris Egerton 
Hi Greg,

Thanks for your thoughts. Responses inline:

> Does this mean that a connector may be assigned to a non-leader worker
in the cluster, an alter request comes in, and a connector instance is
temporarily started on the leader to service the request?

> While the alterOffsets method is being called, will the leader need to
ignore potential requestTaskReconfiguration calls?
On the surface, this seems to conflict with the semantics of the rebalance
subsystem, as connectors are being started where they are not assigned.
Additionally, it seems to conflict with the semantics of the STOPPED state,
which when read literally, might imply that the connector is not started
_anywhere_ in the cluster.

Good catch! This was an oversight on my part when adding the Connector API
for altering offsets. I'd like to keep delegating offset alter requests to
the leader (for reasons discussed below), but it does seem like relying on
Connector::start to deliver a configuration to connectors before invoking
that method is likely to cause more problems than it solves.

One alternative is to add the connector config as an argument to the
alterOffsets method; this would behave similarly to the validate method,
which accepts a raw config and doesn't provide any guarantees about where
the connector is hosted when that method is called or whether start has or
has not yet been invoked on it. Thoughts?

> How will the check for the STOPPED state on alter requests be
implemented, is it from reading the config topic or the status topic?

Config topic only. If it's critical that alter requests not be overwritten
by zombie tasks, fairly strong guarantees can be provided already:
- For sink connectors, the request will naturally fail if there are any
active members of the consumer group for the connector
- For source connectors, exactly-once support can be enabled, at which
point the preemptive zombie fencing round we perform before proceeding with
the reset request should disable any zombie source tasks' producers from
writing any more records/offsets to Kafka

We can also check the config topic to ensure that the connector's set of
task configs is empty (discussed further below).

> Is there synchronization to ensure that a connector on a non-leader is
STOPPED before an instance is started on the leader? If not, there might be
a risk of the non-leader connector overwriting the effects of the
alterOffsets on the leader connector.

A few things to keep in mind here:

1. Connector instances cannot (currently) produce offsets on their own
2. By delegating alter requests to the leader, we can ensure that the
connector's set of task configs is empty before proceeding with the
request. Because task configs are only written to the config topic by the
leader, there's no risk of new tasks spinning up in between the leader
doing that check and servicing the offset alteration request (well,
technically there is if you have a zombie leader in your cluster, but that
extremely rare scenario is addressed naturally if exactly-once source
support is enabled on the cluster since we use a transactional producer for
writes to the config topic then). This, coupled with the zombie-handling
logic described above, should be sufficient to address your concerns, but
let me know if I've missed anything.

Cheers,

Chris

On Wed, Jan 18, 2023 at 2:57 PM Greg Harris 
wrote:

> Hi Chris,
>
> I had some clarifying questions about the alterOffsets hooks. The KIP
> includes these elements of the design:
>
> * The Javadoc for the methods mentions that the alterOffsets methods are
> only called on started and initialized connector objects.
> * The 'Altering' and 'Resetting' offsets descriptions indicate that the
> requests are forwarded to the leader.
> * And finally, the description of "A new STOPPED state" includes a note
> that the Connector will not be started, and it will not be able to generate
> new task configurations
>
> 1. Does this mean that a connector may be assigned to a non-leader worker
> in the cluster, an alter request comes in, and a connector instance is
> temporarily started on the leader to service the request?
> 2. While the alterOffsets method is being called, will the leader need to
> ignore potential requestTaskReconfiguration calls?
> On the surface, this seems to conflict with the semantics of the rebalance
> subsystem, as connectors are being started where they are not assigned.
> Additionally, it seems to conflict with the semantics of the STOPPED state,
> which when read literally, might imply that the connector is not started
> _anywhere_ in the cluster.
>
> I think that if we wish to provide these alterOffsets methods, they must be
> called on started and initialized connector objects.
> And if that's the case, then we will need to ignore
> requestTaskReconfigurationCalls.
> But we may need to relax the wording on the Stopped state to add an
> exception for temporary starts, while still preventing it from using
> resources in the backgrou

[GitHub] [kafka-site] stanislavkozlovski merged pull request #480: MINOR: Add Stanislav Kozlovski to committer list

2023-01-19 Thread GitBox 


stanislavkozlovski merged PR #480:
URL: https://github.com/apache/kafka-site/pull/480


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[DISCUSS] KIP-900: KRaft kafka-storage.sh API additions to support SCRAM

2023-01-19 Thread Proven Provenzano 
I have written a KIP describing the API additions needed to kafka-storage
to store SCRAM
credentials at bootstrap time. Please take a look at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-900%3A+KRaft+kafka-storage.sh+API+additions+to+support+SCRAM+for+Kafka+Brokers

-- 
--Proven

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #43

2023-01-19 Thread Apache Jenkins Server 
See 


Changes:


--
[...truncated 436031 lines...]
[2023-01-19T19:00:13.553Z] > Task :connect:api:testSrcJar
[2023-01-19T19:00:13.553Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-01-19T19:00:13.553Z] > Task :connect:api:publishToMavenLocal
[2023-01-19T19:00:14.566Z] 
[2023-01-19T19:00:14.566Z] > Task :clients:javadoc
[2023-01-19T19:00:14.566Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerLoginCallbackHandler.java:151:
 warning - Tag @link: reference not found: 
[2023-01-19T19:00:15.573Z] 
[2023-01-19T19:00:15.573Z] > Task :streams:javadoc
[2023-01-19T19:00:15.573Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-01-19T19:00:15.573Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-01-19T19:00:15.573Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-01-19T19:00:15.573Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:84:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:136:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:147:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:62:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:62:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/TopologyDescription.java:38:
 warning - Tag @link: reference not found: ProcessorContext#forward(Object, 
Object) forwards
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2023-01-19T19:00:15.574Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:109:
 warning - Tag @link: reference not found: this#getResult()
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureReason()
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureMessage()
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:154:
 warning - Tag @link: reference not found: this#isSuccess()
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:154:
 warning - Tag @link: reference not found: this#isFailure()
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2023-01-19T19:00:15.574Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.4/streams/src/main/

[jira] [Resolved] (KAFKA-14637) Upgrade to 3.4 from old versions (< 0.10) are failing due to incompatible meta.properties check

2023-01-19 Thread Akhilesh Chaganti (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-14637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akhilesh Chaganti resolved KAFKA-14637.
---
Fix Version/s: 3.4.0
   Resolution: Fixed

> Upgrade to 3.4 from old versions (< 0.10) are failing due to incompatible 
> meta.properties check
> ---
>
> Key: KAFKA-14637
> URL: https://issues.apache.org/jira/browse/KAFKA-14637
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Akhilesh Chaganti
>Assignee: Akhilesh Chaganti
>Priority: Blocker
> Fix For: 3.4.0
>
>
> 3.4 has a check in broker startup to ensure cluster.id is provided in 
> `metadata.properties`. This is not always the case if the previous version of 
> Kafka is < 0.10.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1521

2023-01-19 Thread Apache Jenkins Server 
See

[GitHub] [kafka-site] stanislavkozlovski commented on pull request #480: MINOR: Add Stanislav Kozlovski to committer list

2023-01-19 Thread GitBox 


stanislavkozlovski commented on PR #480:
URL: https://github.com/apache/kafka-site/pull/480#issuecomment-1396858053

   tested locally, looks like this: 
   ![Uploading image.png…]()
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] stanislavkozlovski commented on a diff in pull request #480: MINOR: Add Stanislav Kozlovski to committer list

2023-01-19 Thread GitBox 


stanislavkozlovski commented on code in PR #480:
URL: https://github.com/apache/kafka-site/pull/480#discussion_r1081162645


##
committers.html:
##
@@ -469,6 +469,18 @@ The committers
 
 
   
+  

Review Comment:
   afaict we put a  each two committers



##
committers.html:
##
@@ -469,6 +469,18 @@ The committers
 
 
   
+  

Review Comment:
   afaict we put a `` each two committers



##
committers.html:
##
@@ -469,6 +469,18 @@ The committers
 
 
   
+  

Review Comment:
   afaict we put a `` every two committers



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] stanislavkozlovski opened a new pull request, #480: MINOR: Add Stanislav Kozlovski to committer list

2023-01-19 Thread GitBox 


stanislavkozlovski opened a new pull request, #480:
URL: https://github.com/apache/kafka-site/pull/480

   This patch adds Stanislav Kozlovski to the committer page, including an 
image and a few social media links through which one can reach him (me)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-14639) Kafka CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle

2023-01-19 Thread Bojan Blagojevic (Jira) 
Bojan Blagojevic created KAFKA-14639:


 Summary: Kafka CooperativeStickyAssignor revokes/assigns partition 
in one rebalance cycle
 Key: KAFKA-14639
 URL: https://issues.apache.org/jira/browse/KAFKA-14639
 Project: Kafka
  Issue Type: Bug
  Components: clients, consumer
Affects Versions: 3.2.1
Reporter: Bojan Blagojevic


I have an application that runs 6 consumers in parallel. I am getting some 
unexpected results when I use {{{}CooperativeStickyAssignor{}}}. If I 
understand the mechanism correctly, if the consumer looses partition in one 
rebalance cycle, the partition should be assigned in the next rebalance cycle.

This assumption is based on the 
[RebalanceProtocol|https://kafka.apache.org/31/javadoc/org/apache/kafka/clients/consumer/ConsumerPartitionAssignor.RebalanceProtocol.html]
 documentation and few blog posts that describe the protocol, like [this 
one|https://www.confluent.io/blog/cooperative-rebalancing-in-kafka-streams-consumer-ksqldb/]
 on Confluent blog.
{quote}The assignor should not reassign any owned partitions immediately, but 
instead may indicate consumers the need for partition revocation so that the 
revoked partitions can be reassigned to other consumers in the next rebalance 
event. This is designed for sticky assignment logic which attempts to minimize 
partition reassignment with cooperative adjustments.
{quote}
{quote}Any member that revoked partitions then rejoins the group, triggering a 
second rebalance so that its revoked partitions can be assigned. Until then, 
these partitions are unowned and unassigned.
{quote}
These are the logs from the application that uses 
{{{}protocol='cooperative-sticky'{}}}. In the same rebalance cycle 
({{{}generationId=640{}}}) {{partition 74}} moves from {{consumer-3}} to 
{{{}consumer-4{}}}. I omitted the lines that are logged by the other 4 
consumers.

Mind that the log is in reverse(bottom to top)
2022-12-14 11:18:24   1 --- [consumer-3] x.y.z.MyRebalanceHandler1   : New 
partition assignment: partition-59, seek to min common offset: 85120524
2022-12-14 11:18:24   1 --- [consumer-3] x.y.z.MyRebalanceHandler2   : 
Partitions [partition-59] assigned successfully
2022-12-14 11:18:24   1 --- [consumer-3] x.y.z.MyRebalanceHandler1   : 
Partitions assigned: [partition-59]
2022-12-14 11:18:24   1 --- [consumer-3] 
o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer 
clientId=partition-3-my-client-id-my-group-id, groupId=my-group-id] Adding 
newly assigned partitions: partition-59
2022-12-14 11:18:24   1 --- [consumer-3] 
o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer 
clientId=partition-3-my-client-id-my-group-id, groupId=my-group-id] Notifying 
assignor about the new Assignment(partitions=[partition-59])
2022-12-14 11:18:24   1 --- [consumer-3] 
o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer 
clientId=partition-3-my-client-id-my-group-id, groupId=my-group-id] Request 
joining group due to: need to revoke partitions [partition-26, partition-74] as 
indicated by the current assignment and re-join
2022-12-14 11:18:24   1 --- [consumer-3] x.y.z.MyRebalanceHandler2   : 
Partitions [partition-26, partition-74] revoked successfully
2022-12-14 11:18:24   1 --- [consumer-3] x.y.z.MyRebalanceHandler1   : Finished 
removing partition data
2022-12-14 11:18:24   1 --- [consumer-4] 
o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer 
clientId=partition-4-my-client-id-my-group-id, groupId=my-group-id] 
(Re-)joining group
2022-12-14 11:18:24   1 --- [consumer-4] x.y.z.MyRebalanceHandler1   : New 
partition assignment: partition-74, seek to min common offset: 107317730
2022-12-14 11:18:24   1 --- [consumer-4] x.y.z.MyRebalanceHandler2   : 
Partitions [partition-74] assigned successfully
2022-12-14 11:18:24   1 --- [consumer-4] x.y.z.MyRebalanceHandler1   : 
Partitions assigned: [partition-74]
2022-12-14 11:18:24   1 --- [consumer-4] 
o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer 
clientId=partition-4-my-client-id-my-group-id, groupId=my-group-id] Adding 
newly assigned partitions: partition-74
2022-12-14 11:18:24   1 --- [consumer-4] 
o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer 
clientId=partition-4-my-client-id-my-group-id, groupId=my-group-id] Notifying 
assignor about the new Assignment(partitions=[partition-74])
2022-12-14 11:18:24   1 --- [consumer-4] 
o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer 
clientId=partition-4-my-client-id-my-group-id, groupId=my-group-id] Request 
joining group due to: need to revoke partitions [partition-57] as indicated by 
the current assignment and re-join
2022-12-14 11:18:24   1 --- [consumer-4] x.y.z.MyRebalanceHandler2   : 
Partitions [partition-57] revoked successfully
2022-12-14 11:18:24   1 --- [consumer-4] x.y.z.MyRebalanceHandler1   : Finished 
removing partition data
2022-12-14 11:18:22   1 --- [consumer-3] x.y.z.MyRebalanceHandler1   : 
Partitions rev