Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #10

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2152

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-15397) Deserializing produce requests may cause memory leaks when exceptions occur

[hudeqi (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hudeqi resolved KAFKA-15397.

Resolution: Resolved

> Deserializing produce requests may cause memory leaks when exceptions occur
> ---
>
> Key: KAFKA-15397
> URL: https://issues.apache.org/jira/browse/KAFKA-15397
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, core
>Affects Versions: 2.8.1
>Reporter: hudeqi
>Assignee: hudeqi
>Priority: Blocker
> Attachments: SeaTalk_IMG_1692796505.png, SeaTalk_IMG_1692796533.png, 
> SeaTalk_IMG_1692796604.png, SeaTalk_IMG_1692796891.png
>
>
> When the client sends a produce request in an abnormal way and the server 
> accepts it for deserialization, a "java.lang.IllegalArgumentException" may 
> occur, which will cause a large number of "TopicProduceData" objects to be 
> instantiated without being cleaned up. In the end, the entire service of 
> kafka is OOM.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2151

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

[Justine Olshan]

Hey Calvin,

Thanks for the KIP. This will close some of the gaps in leader election! I
has a few questions:

*>* *High Watermark can only advance if the ISR size is larger or equal
to min.insync.replicas*.

For my understanding, will we block replication? Or just the high watermark
advancement?
Also in the acks=1 case, if folks want to continue the previous behavior,
they also need to set min.insync.replicas to 1, correct? It seems like this
change takes some control away from clients when it comes to durability vs
availability.

*> *
*ELR + ISR size will not be dropped below the min ISR unless the controller
discovers an ELR member has an unclean shutdown. *
The KIP seems to suggest that we remove from ELR when we start up again and
notice we do not have the clean shutdown file. Is there a chance we have an
offline broker in ELR that had an unclean shutdown that we elect as a
leader before we get the change to realize the shutdown was unclean?
This seems like it could cause some problems. I may have missed how we
avoid this scenario though.

*> When updating the config **min.insync.replicas, *
*if the new min ISR <= current ISR, the ELR will be removed.*Would this be
the case for strictly a smaller min ISR? I suppose if we increase the ISR,
we can't reason about ELR. Can we reason about high water mark in this
case--seems like we will have the broker out of ISR not in ISR or ELR?
(Forgive me if we can't increase min ISR if the increase will put us under
it)

*> Unclean recovery. *

   - *The unclean leader election will be replaced by the unclean recovery.*
   - *unclean.leader.election.enable will only be replaced by
   the unclean.recovery.strategy after ELR is delivered.*
   - *As there is no change to the ISR, the "last known leader" behavior is
   maintained.*

What does "last known leader behavior maintained" mean here? I thought we
said *"*The above “*Last Leader” behavior can’t be maintained with an empty
ISR and it should be removed." *My understanding is once metadata version
is updated we will always take the more thoughtful unclean election process
(ie, inspect the logs)

Overall though, the general KIP is pretty solid. Looking at the rejected
alternatives, it looks like a lot was considered, so it's nice to see the
final proposal.

Justine

On Mon, Aug 14, 2023 at 8:50 AM Calvin Liu 
wrote:

>1. Yes, the new protocol requires 2 things to advance the HWM. a) The
>messages have been replicated to the controller-committed ISR members.
> b)
>The number of ISR members should be at least the min ISR.
>2. With the current protocol, we are not able to select broker 1 as the
>leader. If we first imply we have the new HWM requirement in place, then
>broker 1 is a good candidate to choose. The following part of the KIP
> (ELR)
>part will explain a new mechanism to enable us to choose broker 1.
> Note, if
>both HWM and ELR are in place, broker 1 will be actually elected in T3.
>
>
> On Fri, Aug 11, 2023 at 10:05 AM Jeff Kim 
> wrote:
>
> > Hi Calvin,
> >
> > Thanks for the KIP! I'm still digesting it but I have two questions:
> >
> > > In the scenario raised in the motivation section, the server may
> receive
> > ack=1 messages during T1 and advance High Watermark when the leader
> > is the only one in ISR.
> >
> > To confirm, the current protocol allows advancing the HWM if all brokers
> in
> > the ISR append to their logs (in this case only the leader). And we're
> > proposing
> > to advance the HWM only when  brokers
> > replicate. Is this correct?
> >
> > > Then, if we elect broker 1 as the leader at T4, though we can guarantee
> > the safety of ack=all messages, the High Watermark may move backward
> > which causes further impacts on the consumers.
> >
> > How can broker 1 become the leader if it was ineligible in T3? Or are
> > you referring to broker 2?
> >
> > Thanks,
> > Jeff
> >
> > On Thu, Aug 10, 2023 at 6:48 PM Calvin Liu 
> > wrote:
> >
> > > Hi everyone,
> > > I'd like to discuss a series of enhancement to the replication
> protocol.
> > >
> > > A partition replica can experience local data loss in unclean shutdown
> > > scenarios where unflushed data in the OS page cache is lost - such as
> an
> > > availability zone power outage or a server error. The Kafka replication
> > > protocol is designed to handle these situations by removing such
> replicas
> > > from the ISR and only re-adding them once they have caught up and
> > therefore
> > > recovered any lost data. This prevents replicas that lost an arbitrary
> > log
> > > suffix, which included committed data, from being elected leader.
> > > However, there is a "last replica standing" state which when combined
> > with
> > > a data loss unclean shutdown event can turn a local data loss scenario
> > into
> > > a global data loss scenario, i.e., committed data can be removed from
> all
> > > replicas. When the last replica in the ISR experiences an unclean
> > shutdown
> > > and loses committed data, it 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2150

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-965: Support disaster recovery between clusters by MirrorMaker

[Greg Harris]

Hi hudeqi,

Thanks for the KIP! I think the original behavior (removing WRITE
permissions during the sync) is a good default, but is not acceptable
in every situation. I think providing a configuration for this
behavior is the right idea.

I had a few questions:

1. Is this configuration only relevant to the MirrorSourceConnector?
Since we split the different connector configurations, we can omit
this configuration from the Checkpoint and Heartbeat connectors when
deployed in a connect cluster.
2. Is this configuration only able to be configured globally for an
entire Dedicated MirrorMaker2? Can it be configured for one flow in a
dedicated deployment and not another by specifying
`source->target.sync.full.acl.enabled`?
3. Is the documentation going to include the "disaster recovery"
language, or is that a left-over from an earlier revision in the KIP?
I don't think that "disaster recovery" is a very clear term in this
situation, and we should probably be very specific in the
documentation about what this configuration is changing.
4. Did you consider any use-cases where a more restrictive ACL sync
would be desirable? Right now we are downgrading ALL/removing WRITE,
but leaving CREATE/DELETE/ALTER/etc ACLs as-is. Perhaps users would
like to choose between an ACL sync which is more locked-down, the
current behavior, or more permissive.
5. Currently MM2 only syncs topic ACLs, and not group ACLs or SCRAM
credentials, so those would be new capabilities. Can you here (or in
the KIP) go into more detail about how these would work?
6. Is there a reason to have one configuration control these three
different syncs? Could users want to change the topic ACL sync
semantics, while not using the group sync or the SCRAM sync?

Thanks,
Greg

On Mon, Aug 28, 2023 at 2:10 AM hudeqi <16120...@bjtu.edu.cn> wrote:
>
> Hi, all, this is a vote about kip-965, thanks.
>
> best,
> hudeqi
>
>
>  -原始邮件-
>  发件人: hudeqi 16120...@bjtu.edu.cn
>  发送时间: 2023-08-17 18:03:49 (星期四)
>  收件人: dev@kafka.apache.org
>  抄送:
>  主题: Re: [DISCUSSION] KIP-965: Support disaster recovery between clusters 
> by MirrorMaker
> 

[jira] [Resolved] (KAFKA-13327) Preflight validations of connectors leads to 500 responses

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-13327.
---
Fix Version/s: 3.7.0
   Resolution: Fixed

> Preflight validations of connectors leads to 500 responses
> --
>
> Key: KAFKA-13327
> URL: https://issues.apache.org/jira/browse/KAFKA-13327
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
> Fix For: 3.7.0
>
>
> The Connect framework performs some preflight validations for all connectors 
> that are created in addition to allowing connectors to define their own 
> custom validation logic by providing a {{ConfigDef}} object in 
> [Connector::config|https://kafka.apache.org/30/javadoc/org/apache/kafka/connect/connector/Connector.html#config()]
>  and performing multi-property validation in 
> [Connector::validate|https://kafka.apache.org/30/javadoc/org/apache/kafka/connect/connector/Connector.html#validate(java.util.Map)].
> When performed correctly, this validation information is surfaced to the user 
> in the form of a 
> [ConfigInfos|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/entities/ConfigInfos.java]
>  object containing a list of [config 
> objects|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/entities/ConfigInfo.java#L42-L45]
>  whose 
> [values|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/entities/ConfigInfo.java#L42-L45]
>  contain one or more [error 
> messages|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/entities/ConfigValueInfo.java#L61-L64].
>  This can be used as the response for a REST request to PUT 
> /connector-plugins/\{connectorType}/config/validate and allows programmatic 
> UIs to render error messages for every invalid property to the user.
> However, some validations performed by the Connect framework do not follow 
> this pattern and instead result in a 500 response being returned to the user. 
> For example, logic specific to sink connectors (see 
> [AbstractHerder|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L436]
>  and 
> [SinkConnectorConfig|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/SinkConnectorConfig.java#L88-L125])
>  simply throws an exception instead of documenting the error with the 
> offending property and returning it in a standard response.
>  
> We should correct this logic wherever possible so that configurations that 
> are not fatally invalid (i.e., may have invalid properties but can still be 
> translated into a meaningful {{ConfigInfos}} response object) do not cause a 
> 500 response to be returned to the user.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14875) Implement Wakeup()

[Jun Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-14875.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

merged the PR to trunk

> Implement Wakeup()
> --
>
> Key: KAFKA-14875
> URL: https://issues.apache.org/jira/browse/KAFKA-14875
> Project: Kafka
>  Issue Type: Task
>  Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Philip Nee
>Priority: Major
>  Labels: consumer-threading-refactor
> Fix For: 3.7.0
>
>
> Implement wakeup() and WakeupException.  This would be different to the 
> current implementation because I think we just need to interrupt the blocking 
> futures.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2149

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15418) Update statement on decompression location

[Lucia Cerchie (Jira)]

Lucia Cerchie created KAFKA-15418:
-

 Summary: Update statement on decompression location 
 Key: KAFKA-15418
 URL: https://issues.apache.org/jira/browse/KAFKA-15418
 Project: Kafka
  Issue Type: Improvement
  Components: docs
Reporter: Lucia Cerchie
 Fix For: 3.5.1


The docs 
[state|[https://github.com/apache/kafka/blob/0912ca27e2a229d2ebe02f4d1dabc40ed5fab0bb/docs/design.html#L139-L140]]:
 
 Kafka supports this with an efficient batching format. A batch of messages can 
be clumped together compressed and sent to the server in this form. This batch 
of messages will be written in compressed form and will
    remain compressed in the log and will only be decompressed by the consumer.
However, brokers always perform some amount of batch decompression in order to 
validate data. Internally, the LogValidator class implements the validation 
logic, and message decompression ultimately happens in the relevant 
CompressionType's wrapForInput implementation. These lines need updating to 
reflect that, including the scenarios that require full payload decompression 
on the broker (compacted topic, or topic level compression codec different from 
producer's codec). 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15417) JoinWindow does not seem to work properly with a KStream - KStream - LeftJoin()

[Victor van den Hoven (Jira)]

Victor van den Hoven created KAFKA-15417:


 Summary: JoinWindow does not  seem to work properly with a KStream 
- KStream - LeftJoin()
 Key: KAFKA-15417
 URL: https://issues.apache.org/jira/browse/KAFKA-15417
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 3.4.0
Reporter: Victor van den Hoven
 Attachments: SimpleStreamTopology.java, SimpleStreamTopologyTest.java

In Kafka-streams 3.4.0 :

According to the javadoc of the Joinwindow:

_There are three different window configuration supported:_
 * _before = after = time-difference_
 * _before = 0 and after = time-difference_
 * _*before = time-difference and after = 0*_

 

However if I use a joinWindow with *before = time-difference and after = 0* 
on a kstream-kstream-leftjoin the *after=0* part does not seem to work.

When using _stream1.leftjoin(stream2, joinWindow)_ with 
{_}joinWindow{_}.{_}after=0 and joinWindow.before=30s{_}, any new message on 
stream 1 that can not be joined with any messages on stream2 should be joined 
with a null-record after the _joinWindow.after_ has ended and a new message has 
arrived on stream1.

It does not.

Only if the new message arrives after the value of _joinWindow.before_ the 
previous message will be joined with a null-record.

 

Attached you can find two files with a TopologyTestDriver Unit test to 
reproduce.

topology:   stream1.leftjoin( stream2, joiner, joinwindow)

joinWindow has before=5000ms and after=0

message1(key1) ->  stream1

after 4000ms message2(key2) -> stream1  ->  NO null-record join was made, but 
the after period was expired.
after 4900ms message2(key2) -> stream1  ->  NO null-record join was made, but 
the after period was expired.
after 5000ms message2(key2) -> stream1  ->  A null-record join was made,  
before period was expired.
after 6000ms message2(key2) -> stream1  ->  A null-record join was made,  
before period was expired.



 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15416) Flaky test TopicAdminTest::retryEndOffsetsShouldRetryWhenTopicNotFound

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-15416:
-

 Summary: Flaky test 
TopicAdminTest::retryEndOffsetsShouldRetryWhenTopicNotFound
 Key: KAFKA-15416
 URL: https://issues.apache.org/jira/browse/KAFKA-15416
 Project: Kafka
  Issue Type: Test
  Components: KafkaConnect
Reporter: Chris Egerton


This test fails frequently when I run unit tests locally, but I've never seen 
it fail during a CI build.

Failure message:
{quote}    org.apache.kafka.connect.errors.ConnectException: Failed to list 
offsets for topic partitions.

        at 
app//org.apache.kafka.connect.util.TopicAdmin.retryEndOffsets(TopicAdmin.java:777)

        at 
app//org.apache.kafka.connect.util.TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound(TopicAdminTest.java:570)

 

        Caused by:

        org.apache.kafka.connect.errors.ConnectException: Fail to list offsets 
for topic partitions after 1 attempts.  Reason: Timed out while waiting to get 
end offsets for topic 'myTopic' on brokers at \{retry.backoff.ms=0}

            at 
app//org.apache.kafka.connect.util.RetryUtil.retryUntilTimeout(RetryUtil.java:106)

            at 
app//org.apache.kafka.connect.util.RetryUtil.retryUntilTimeout(RetryUtil.java:56)

            at 
app//org.apache.kafka.connect.util.TopicAdmin.retryEndOffsets(TopicAdmin.java:768)

            ... 1 more

 

            Caused by:

            org.apache.kafka.common.errors.TimeoutException: Timed out while 
waiting to get end offsets for topic 'myTopic' on brokers at 
\{retry.backoff.ms=0}

 

                Caused by:

                java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the 
call. Call: listOffsets(api=METADATA)

                    at 
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)

                    at 
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)

                    at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)

                    at 
org.apache.kafka.connect.util.TopicAdmin.endOffsets(TopicAdmin.java:716)

                    at 
org.apache.kafka.connect.util.TopicAdmin.lambda$retryEndOffsets$7(TopicAdmin.java:769)

                    at 
org.apache.kafka.connect.util.RetryUtil.retryUntilTimeout(RetryUtil.java:87)

                    at 
org.apache.kafka.connect.util.RetryUtil.retryUntilTimeout(RetryUtil.java:56)

                    at 
org.apache.kafka.connect.util.TopicAdmin.retryEndOffsets(TopicAdmin.java:768)

                    at 
org.apache.kafka.connect.util.TopicAdminTest.retryEndOffsetsShouldRetryWhenTopicNotFound(TopicAdminTest.java:570)

 

                    Caused by:

                    org.apache.kafka.common.errors.TimeoutException: Timed out 
waiting to send the call. Call: listOffsets(api=METADATA)
{quote}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Request: Permissions to contribute to Apache Kafka

[Josep Prat]

Hi Krishna,

Thank you for your interest in contributing to Apache Kafka.
Your accounts are all set.
Let me know if you have any questions.

Best,

On Tue, Aug 29, 2023 at 4:06 PM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi,
> I would like to propose a KIP and contribute to Apache Kafka.
> Can I please be provided with the necessary permissions.
>
> Wiki ID: krishna06
> Jira ID: krishna06
>
> Regards,
> Krishna
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Request: Permissions to contribute to Apache Kafka

[Krishna Agarwal]

Hi,
I would like to propose a KIP and contribute to Apache Kafka.
Can I please be provided with the necessary permissions.

Wiki ID: krishna06
Jira ID: krishna06

Regards,
Krishna

[jira] [Created] (KAFKA-15415) In Java-client, backoff should be skipped for retried producer-batch for a new leader

[Mayank Shekhar Narula (Jira)]

Mayank Shekhar Narula created KAFKA-15415:
-

 Summary: In Java-client, backoff should be skipped for retried 
producer-batch for a new leader
 Key: KAFKA-15415
 URL: https://issues.apache.org/jira/browse/KAFKA-15415
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Reporter: Mayank Shekhar Narula
Assignee: Mayank Shekhar Narula
 Fix For: 3.7.0


Right now in java-client, producer-batches backoff upto 
`retry.backoff.ms`(100ms by default). This Jira proposes that backoff should be 
skipped if in a sub-sequent retry, client knows of a newer-leader for the 
partition. This would help improve the latency of the produce-request around 
leadership changes of a partition.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

[Yash Mayya]

Hi Sagar,

> The size of offsets topic can be controlled by
> setting appropriate topic retention values and
> that is a standard practice in Kafka

Kafka Connect enforces the `cleanup.policy` configuration for the offsets
topic to be "compact" only (references - [1], [2]), so the topic retention
related configurations won't be relevant right?

> Deleting offsets is not something which should
> be done very frequently and should be handled
> with care

> Agreed this involves some toil but it's not something
> that should be done on a very regular basis.

I'm not sure I follow how we came to this conclusion, could you please
expand on the pitfalls of allowing connector plugins to wipe the offsets
for source partitions that they no longer care about?

> The usecases you highlighted are edge cases at
> best. As I have been saying, if it is needed we can
> always add it in the future but that doesn't look like
> a problem we need to solve upfront.

I agree that these cases might not be too common, but I'm just trying to
understand the reasoning behind preventing this use case since null offsets
don't require any separate handling from the Connect runtime's point of
view (and wouldn't need any additional implementation work in this KIP).

Thanks,
Yash

[1] - https://kafka.apache.org/documentation/#connect_running
[2] -
https://github.com/apache/kafka/blob/0912ca27e2a229d2ebe02f4d1dabc40ed5fab0bb/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaTopicBasedBackingStore.java#L47

On Mon, Aug 28, 2023 at 1:38 PM Sagar  wrote:

> Hey Yash,
>
> Thanks for your further comments. Here are my responses:
>
> 1) Deleting offsets via updateOffsets.
>
> Hmm, I am not sure this is really necessary to be part of the KIP at this
> point, and we can always add it later on if needed. I say this for the
> following reasons:
>
>
>- The size of offsets topic can be controlled by setting appropriate
>topic retention values and that is a standard practice in Kafka. Sure
> it's
>not always possible to get the right values but as I said it is a
> standard
>practice. For Connect specifically, there is also a KIP (KIP-943
><
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255073470
> >)
>which is trying to solve the problem of a large connect-offsets topic.
> So,
>if that is really the motivation, then these are being addressed
> separately
>anyways.
>- Deleting offsets is not something which should be done very frequently
>and should be handled with care. That is why KIP-875's mechanism to have
>users/ cluster admin do this externally is the right thing to do. Agreed
>this involves some toil but it's not something that should be done on a
>very regular basis.
>- There is no stopping connector implementations to send tombstone
>records as offsets but in practice how many connectors actually do it?
>Maybe 1 or 2 from what we discussed.
>- The usecases you highlighted are edge cases at best. As I have been
>saying, if it is needed we can always add it in the future but that
> doesn't
>look like a problem we need to solve upfront.
>
> Due to these reasons, I don't think this is a point that we need to stress
> so much upon. I say this because offsets topic's purging/clean up can be
> handled either via standard Kafka techniques (point #1 above) or via
> Connect runtime techniques (Pt #2  above). IMO the problem we are trying to
> solve via this KIP has been solved by connectors using techniques which
> have been termed as having higher maintenance cost or a high cognitive load
> (i.e separate topic) and that needs to be addressed upfront. And since you
> yourself termed it as a nice to have feature, we can leave it to that and
> take it up as Future Work. Hope that's ok with you and other community
> members.
>
> 2) Purpose of offsets parameter in updateOffsets
>
> The main purpose is to provide the task with the visibility into what
> partitions are getting their offsets committed. It is not necessary that a
> task might choose to update offsets everytime it sees that a given source
> partition is missing from the about to be committed offsets. Maybe it
> chooses to wait for some X iterations or X amount of time and send out an
> updated offset for a partition only when such thresholds are breached. Even
> here we could argue that since it's sending the partition/offsets it can do
> the tracking on it's own, but IMO that is too much work given that the
> information is already available via offsets to be committed.
>
> Thanks!
> Sagar.
>

Kafka jenkins is unavailable now

[Luke Chen]

Hi all,

Just found Kafka jenkins is unavailable currently.
Filed an INFRA issue: https://issues.apache.org/jira/browse/INFRA-24940
FYI.

Thanks.
Luke

Re: [REVIEW REQUEST] Move ReassignPartitionsCommandArgsTest to java

[Николай Ижиков]

Hello, Mickael.

Thanks for the reply and reviewing previous patches for this feature :).

Just to let all know - I’ve got one approval from non commiter.
So, if any committer has spare cycle, please, join the simple review that will 
help to rewrite tools in java

https://github.com/apache/kafka/pull/14217


> 25 авг. 2023 г., в 13:25, Mickael Maison  
> написал(а):
> 
> Hi Nikolay,
> 
> Thanks for working on this feature. Take into account that many people
> are on PTO at the moment and there's a lot of PRs to review.
> Not sure why fixVersion was set to 3.6.0. Apart for key features or
> fixes, it's usually best to only set the fixVersion once the code has
> been merged. I've cleared that field.
> 
> Thanks,
> Mickael
> 
> On Fri, Aug 25, 2023 at 9:26 AM Николай Ижиков  wrote:
>> 
>> Hello, Luke
>> 
>> Thanks for reply.
>> 
>> Actually, KAFKA-14595 [1] has fixVersion = 3.6.0
>> And the PR is part of the ticket.
>> 
>> Several review required If we want to include java version of 
>> ReassignPartitionCommand in 3.6.0
>> 
>> [1] https://issues.apache.org/jira/browse/KAFKA-14595
>> 
>> 
>>> 23 авг. 2023 г., в 10:07, Luke Chen  написал(а):
>>> 
>>> Hi,
>>> 
>>> Sorry that we're mostly working on features for v3.6.0, which is expected
>>> to be released in the following weeks.
>>> I'll review your PR after releasing. (Please ping me then if I forget it!)
>>> 
>>> Also, it'd be good if the devs in the community can help on PR review when
>>> available.
>>> That'll help a lot.
>>> Besides, PR review is also one kind of contribution, not just code
>>> commitment.
>>> 
>>> Thanks.
>>> Luke
>>> 
>>> 
>>> 
>>> On Tue, Aug 22, 2023 at 7:15 PM Николай Ижиков  wrote:
>>> 
 Hello.
 
 Please, join the simple review)
 We have few steps left to completely rewrite ReassignPartitionsCommand in
 java.
 
> 17 авг. 2023 г., в 17:16, Николай Ижиков 
 написал(а):
> 
> Hello.
> 
> I’m working on [1].
> The goal of ticket is to rewire `ReassignPartitionCommand` in java.
> 
> The PR that moves whole command is pretty big so it makes sense to split
 it.
> I prepared the PR [2] that moves single test
 (ReassignPartitionsCommandArgsTest) to java.
> 
> It relatively small and simple(touches only 3 files):
> 
> To review - https://github.com/apache/kafka/pull/14217
> Big PR  - https://github.com/apache/kafka/pull/13247
> 
> Please, review.
> 
> [1] https://issues.apache.org/jira/browse/KAFKA-14595
> [2] https://github.com/apache/kafka/pull/14217
 
 
>> 

Re: Need Access to create KIP & Jira Tickets

[Josep Prat]

Hi Raghu,
Thanks for your interest in Apache Kafka.
As Justine rightly points out, for the change of the Jira ID you should
file a Jira issue under the INFRA project [1]. If you end up creating
another account, I would also recommend creating a Jira issue under INFRA
to merge the 2 accounts.

Best,
[1]: https://issues.apache.org/jira/projects/INFRA/issues

On Mon, Aug 28, 2023 at 10:29 PM Justine Olshan
 wrote:

> Hey Raghu,
>
> I've added your ID to give you permissions to the wiki.
>
> I'm not sure if committers can change your jira ID. You may want to try to
> create a new account or file a ticket with apache for that.
>
> Let me know if there are any issues.
>
> Justine
>
> On Mon, Aug 28, 2023 at 11:54 AM Raghu Baddam  wrote:
>
> > Hi Team,
> >
> > Please find wiki ID and Jira ID and help me by providing me access to
> > create KIP's and Jira Tickets on Apache Kafka space.
> >
> > wiki ID: rbaddam
> > Jira ID: raghu98...@gmail.com
> >
> > Also If possible I also need help with changing my Jira ID same as wiki
> ID
> > i.e. *rbaddam*
> >
> > Thanks,
> > Raghu
> >
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B