Re: Apache Kafka 3.6.0 release

[Yash Mayya]

Hi Satish,

KIP-793 [1] just passed voting and we should be able to wrap up the
implementation in time for the 3.6.0 feature freeze. Could we add it to the
release plan?

[1] -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-793%3A+Allow+sink+connectors+to+be+used+with+topic-mutating+SMTs

Thanks,
Yash

On Mon, Jun 12, 2023 at 3:52 PM Satish Duggana 
wrote:

> Hi,
> I have created a release plan for Apache Kafka version 3.6.0 on the
> wiki. You can access the release plan and all related information by
> following this link:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0
>
> The release plan outlines the key milestones and important dates for
> version 3.6.0. Currently, the following dates have been set for the
> release:
>
> KIP Freeze: 26th July 23
> Feature Freeze : 16th Aug 23
> Code Freeze : 30th Aug 23
>
> Please review the release plan and provide any additional information
> or updates regarding KIPs targeting version 3.6.0. If you have
> authored any KIPs that are missing a status or if there are incorrect
> status details, please make the necessary updates and inform me so
> that I can keep the plan accurate and up to date.
>
> Thanks,
> Satish.
>
> On Mon, 17 Apr 2023 at 21:17, Luke Chen  wrote:
> >
> > Thanks for volunteering!
> >
> > +1
> >
> > Luke
> >
> > On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma  wrote:
> >
> > > Thanks for volunteering Satish. +1.
> > >
> > > Ismael
> > >
> > > On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana <
> satish.dugg...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > I would like to volunteer as release manager for the next release,
> > > > which will be Apache Kafka 3.6.0.
> > > >
> > > > If there are no objections, I will start a release plan a week after
> > > > 3.5.0 release(around early May).
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > >
>

Re: [VOTE] KIP-793: Allow sink connectors to be used with topic-mutating SMTs

[Yash Mayya]

Hi all,

Thanks to everyone who participated in the voting and the discussion. I'll
close the voting since it has been open for over 72 hours and we have a
sufficient number of votes. KIP-793 has been accepted with the following +1
votes (and no +0 or -1 votes):


   - Chris Egerton (binding)
   - Greg Harris
   - Mickael Maison (binding)
   - Randall Hauch (binding)

I'll start working on the implementation next week and try to get it merged
in time for the 3.6.0 release.

Thanks,
Yash

On Thu, Jul 6, 2023 at 11:32 PM Randall Hauch  wrote:

> Hi, Yash.
>
> Thanks for the KIP and for fixing this long standing issue.
> +1 (binding)
>
> Randall
>
> On Tue, Jul 4, 2023 at 8:32 AM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > +1 (binding)
> > Thanks for the KIP
> >
> > Mickael
> >
> >
> > On Mon, Jul 3, 2023 at 8:18 PM Greg Harris  >
> > wrote:
> > >
> > > Hey Yash,
> > >
> > > Thanks so much for your effort in the design and discussion phase!
> > >
> > > +1 (non-binding)
> > >
> > > Greg
> > >
> > > On Mon, Jul 3, 2023 at 7:19 AM Chris Egerton 
> > wrote:
> > > >
> > > > Hi Yash,
> > > >
> > > > Thanks for the KIP! +1 (binding)
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Mon, Jul 3, 2023 at 7:02 AM Yash Mayya 
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to start a vote on KIP-793 which enables sink connector
> > > > > implementations to be used with SMTs that mutate the topic /
> > partition /
> > > > > offset information of a record.
> > > > >
> > > > > KIP -
> > > > >
> > > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-793%3A+Allow+sink+connectors+to+be+used+with+topic-mutating+SMTs
> > > > >
> > > > > Discussion thread -
> > > > > https://lists.apache.org/thread/dfo3spv0xtd7vby075qoxvcwsgx5nkj8
> > > > >
> > > > > Thanks,
> > > > > Yash
> > > > >
> >
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1975

[Apache Jenkins Server]

See 

Re: Permission to contribute to Apache Kafka

[Luke Chen]

Hi Shay,

Your accounts are all set.

Thanks.
Luke

On Fri, Jul 7, 2023 at 7:08 AM Shay Lin  wrote:

> Hi all,
>
> Could you give me the appropriate access (edit Wiki/KIP, Jira etc) to
> contribute to AK?
>
> My confluence and Jira IDs are the same: lqxshay
>
> Thanks in advance,
> Shay
>

Permission to contribute to Apache Kafka

[Shay Lin]

Hi all,

Could you give me the appropriate access (edit Wiki/KIP, Jira etc) to
contribute to AK?

My confluence and Jira IDs are the same: lqxshay

Thanks in advance,
Shay

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

[Erik van Oosten]

Dear PMCs,

So far there have been 0 responses to KIP-944. I understand this may not 
be something that keeps you busy, but this KIP is important to people 
that use async runtimes like Zio, Cats and Kotlin.


Is there anything you need to come to a decision?

Kind regards,
    Erik.


Op 05-07-2023 om 11:38 schreef Erik van Oosten:

Hello all,

I'd like to call a vote on KIP-944 Support async runtimes in consumer. 
It has has been 'under discussion' for 7 days now. 'Under discussion' 
between quotes, because there were 0 comments so far. I hope the KIP 
is clear!


KIP description: https://cwiki.apache.org/confluence/x/chw0Dw

Kind regards,
    Erik.


--
Erik van Oosten
e.vanoos...@grons.nl
https://day-to-day-stuff.blogspot.com

[GitHub] [kafka-site] stevenbooke commented on pull request #521: KAFKA-14995: Automate asf.yaml collaborators refresh

[via GitHub]

stevenbooke commented on PR #521:
URL: https://github.com/apache/kafka-site/pull/521#issuecomment-1624159774

   @mimaison Could you review this and merge if there is no issues please?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-793: Allow sink connectors to be used with topic-mutating SMTs

[Randall Hauch]

Hi, Yash.

Thanks for the KIP and for fixing this long standing issue.
+1 (binding)

Randall

On Tue, Jul 4, 2023 at 8:32 AM Mickael Maison 
wrote:

> Hi,
>
> +1 (binding)
> Thanks for the KIP
>
> Mickael
>
>
> On Mon, Jul 3, 2023 at 8:18 PM Greg Harris 
> wrote:
> >
> > Hey Yash,
> >
> > Thanks so much for your effort in the design and discussion phase!
> >
> > +1 (non-binding)
> >
> > Greg
> >
> > On Mon, Jul 3, 2023 at 7:19 AM Chris Egerton 
> wrote:
> > >
> > > Hi Yash,
> > >
> > > Thanks for the KIP! +1 (binding)
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Mon, Jul 3, 2023 at 7:02 AM Yash Mayya 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start a vote on KIP-793 which enables sink connector
> > > > implementations to be used with SMTs that mutate the topic /
> partition /
> > > > offset information of a record.
> > > >
> > > > KIP -
> > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-793%3A+Allow+sink+connectors+to+be+used+with+topic-mutating+SMTs
> > > >
> > > > Discussion thread -
> > > > https://lists.apache.org/thread/dfo3spv0xtd7vby075qoxvcwsgx5nkj8
> > > >
> > > > Thanks,
> > > > Yash
> > > >
>

[jira] [Resolved] (KAFKA-15069) Refactor scanning hierarchy out of DelegatingClassLoader

[Greg Harris (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Harris resolved KAFKA-15069.
-
  Reviewer: Chris Egerton
Resolution: Fixed

> Refactor scanning hierarchy out of DelegatingClassLoader
> 
>
> Key: KAFKA-15069
> URL: https://issues.apache.org/jira/browse/KAFKA-15069
> Project: Kafka
>  Issue Type: Task
>  Components: KafkaConnect
>Reporter: Greg Harris
>Assignee: Greg Harris
>Priority: Major
> Fix For: 3.6.0
>
>
> The DelegatingClassLoader is involved in both scanning and using the results 
> of scanning to process classloading.
> Instead, the scanning should take place outside of the DelegatingClassLoader, 
> and results of scanning be passed back into the DelegatingClassLoader for 
> classloading functionality.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [kafka-site] stevenbooke commented on pull request #521: KAFKA-14995: Automate asf.yaml collaborators refresh

[via GitHub]

stevenbooke commented on PR #521:
URL: https://github.com/apache/kafka-site/pull/521#issuecomment-1624069492

   Ok, I will make the change now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-932: Queues for Kafka

[Matthias J. Sax]

Thanks for the KIP.

It seems we are in very early stage, and some very important sections in 
the KIP are still marked as TODO. In particular, I am curious about the 
protocol changes, how the "queuing state" will be represented and made 
durable, and all the error edge case / fail-over / fencing 
(broker/clients) that we need to put in place.



A few other comments/question from my side:

(1) Fetch from follower: this was already touched on, but the point is 
really that the consumer does not decide about it, but the broker does. 
When a consumer sends it's first fetch request it will always go to the 
leader, and the broker would reply to the consumer "go and fetch from 
this other broker". -- I think it's ok to exclude fetch from follower in 
the first version of the KIP, but it would need a broker change such 
that the broker knows it's a "queue fetch" request. -- It would also be 
worth to explore how fetch from follow could work in the future and 
ensure that our initial design allows for it and is future proof.



(2) Why do we not allow pattern subscription and what happens if 
different consumers subscribe to different topics? It's not fully 
explained in the KIP.



(3) auto.offset.reset and SPSO/SPSE -- I don't understand why we would 
not allow auto.offset.reset? In the discussion, you mentioned that 
"first consumer would win, if two consumers have a different config" -- 
while this is correct, it's the same for a consumer group right now. 
Maybe we should not try to solve a "non problem"? -- In general, my 
impression is that we are going to do Kafkaeque Queuing, what is fine, 
but it might be to our advantage to carry over as many established 
concepts as we can? And if not, have a very good reason not to.


In the end, it find if very clumsy to only have an admin API to change 
the starting point of a consumer.


(3B) What happens if lag grows and data is purged broker side?

(3C) What happens if the broker released records (based on "timeout / 
exceeding deliver count), and the "ack/reject" comes afterwards?


(3D) How to find out what records got archived but where not acked (ie, 
lost) for re-processing/debugging purpose? The question was already 
asked and the answer was "not supported", but I think it would be 
must-have before the feature is usable in production? We can of course 
also only do it in a future release and not the first "MVP" 
implementation, but the KIP should address it. In the end, the overall 
group monitoring story is missing.



(4) I am also wondering about the overall design with regard to "per 
record" vs "per batch" granularity. In the end, queuing usually aims for 
"per records" semantics, but "per record" implies to keep track of a lot 
of metadata. Kafka is designed on a "per batch" granularity, and it's 
unclear to me how both will go together?


(4A) Do we keep "ack/reject/..." state per-record, or per batch? It 
seems per record, but it would require to hold a lot of meta-data. Also, 
how does it work for the current protocol, is a batch is partially acked 
and we need to re-deliver? Would we add metadata and the let client 
filter acked messages (similar to how "read-committed" mode works)?


(4B) What does "the share-partition leader prefers to return complete
 record batches." exactly mean? "Prefers" is a fuzzy word. What happens 
if we cannot return a complete record batch?


(4C) What happens if different consumer of the same group configure 
different batch sizes for fetching records? How do we track the 
corresponding meta-data?


(4D)


In the situation where some records in a batch have been released or rejected 
separately, subsequent fetches of those records are more likely to have gaps.


What does this mean?

(4E)


For efficiency, the consumer preferentially returns complete record sets with 
no gaps


Can you elaborate on the details?


API contract:

(5A)

acks must be issued in the order in which the records appear


Why is this the case? Sounds like an arbitrary restriction to me? Can 
you share your reasoning?



(5B) How to "reject" (or just "release") all records of a batch at once? 
It seem the API only allows to "ack" all record of a batch at once.


(5C) Currently, `ConsumerRecords` object may contain records from 
different partitions? Would this still be the case?



(6) Group management / re-balancing:

(6A) The KIP should explain better how heart-beating works (was already 
partially discussed). How does `max.poll.interval.ms` interact? Would it 
trigger a "release" of records if violated?


(6B) You mentioned that a consumer that does not heartbeat would just be 
removed from the group with a rebalance: Given the current design to 
assign all partitions to every consumer in the group, that would be ok. 
But as you mentioned on the KIP, we might want to be more clever with 
regard to assigning partitions in the future, and I think we would 
actually need to trigger a rebalance to avoid a later protocol change: 
otherwise, 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1973

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 397495 lines...]
Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterWithVersionedStores[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterWithVersionedStores[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterWithRightVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterWithRightVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithVersionedStores[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithVersionedStores[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithRightVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithRightVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerInner[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerInner[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerOuter[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerOuter[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerLeft[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerLeft[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterInner[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterInner[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > 

[GitHub] [kafka-site] mimaison commented on pull request #521: KAFKA-14995: Automate asf.yaml collaborators refresh

[via GitHub]

mimaison commented on PR #521:
URL: https://github.com/apache/kafka-site/pull/521#issuecomment-1623947447

   The latest committer is Divij Vaidya, see at the bottom of 
https://kafka.apache.org/committers.
   His Github username is `divijvaidya`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] stevenbooke commented on pull request #521: KAFKA-14995: Automate asf.yaml collaborators refresh

[via GitHub]

stevenbooke commented on PR #521:
URL: https://github.com/apache/kafka-site/pull/521#issuecomment-1623938985

   Who is the new Committer? What is their Github username?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-15159) Update minor dependencies in preparation for 3.5.1

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15159:


 Summary: Update minor dependencies in preparation for 3.5.1
 Key: KAFKA-15159
 URL: https://issues.apache.org/jira/browse/KAFKA-15159
 Project: Kafka
  Issue Type: Improvement
Reporter: Divij Vaidya


Go through the list of dependencies in dependencies.gradle. Check if newer 
minor versions (backward compatible) are available, is yes, then create a PR 
for them. Especially important are the ones which fix a CVE.

We will merge them into trunk and backport it to 3.5 branch.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15153) Use Python `is` instead of `==` to compare for None

[Divij Vaidya (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Divij Vaidya resolved KAFKA-15153.
--
  Reviewer: Divij Vaidya
Resolution: Fixed

> Use Python `is` instead of  `==` to compare for None
> 
>
> Key: KAFKA-15153
> URL: https://issues.apache.org/jira/browse/KAFKA-15153
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Divij Vaidya
>Assignee: Lan Ding
>Priority: Minor
>  Labels: newbie
> Fix For: 3.6.0
>
>
> *This is a good item to be picked by first time contributors to Kafka code 
> base*
> At release_notes.py Line: 47
> The {{==}} and {{!=}} operators use the compared objects' {{__eq__}} method 
> to test if they are equal. To check if an object is a singleton, such as 
> {{{}None{}}}, we recommend that you use the {{is}} identity comparison 
> operator.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15158) Add metrics for RemoteRequestsPerSec

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15158:


 Summary: Add metrics for RemoteRequestsPerSec
 Key: KAFKA-15158
 URL: https://issues.apache.org/jira/browse/KAFKA-15158
 Project: Kafka
  Issue Type: Sub-task
Reporter: Divij Vaidya
 Fix For: 3.6.0


Add the following metrics for better observability into the RemoteLog related 
activities inside the broker.

1. RemoteWriteRequestsPerSec


2. RemoteDeleteRequestsPerSec


3. BuildRemoteLogAuxStateRequestsPerSec

 

These metrics will be calculated at topic level (we can add them at 
brokerTopicStats)


*RemoteWriteRequestsPerSec* will be marked on every call to RemoteLogManager#
copyLogSegmentsToRemote()
 
*RemoteDeleteRequestsPerSec* will be marked on every call to 
RemoteLogManager#cleanupExpiredRemoteLogSegments(). This method is introduced 
in [https://github.com/apache/kafka/pull/13561] 

*BuildRemoteLogAuxStateRequestsPerSec* will be marked on every call to 
ReplicaFetcherTierStateMachine#buildRemoteLogAuxState()
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Permission to contribute to Apache Kafka

[Chris Egerton]

Hi,

Thanks for your interest in contributing! You should be all set to go now.

Cheers,

Chris

On Thu, Jul 6, 2023 at 9:18 AM Jacob  wrote:

> Hi master,
>
> Could you help grant me permissions to contribute to Kafka?
>
> Wiki ID:  zixuanbai
> Jira ID: zixuan
>
> Thanks a lot!
>
> Best regards,
>

[jira] [Created] (KAFKA-15157) Print startup time for RemoteIndexCache

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15157:


 Summary: Print startup time for RemoteIndexCache
 Key: KAFKA-15157
 URL: https://issues.apache.org/jira/browse/KAFKA-15157
 Project: Kafka
  Issue Type: Sub-task
Reporter: Divij Vaidya


When RemoteIndexCache starts up, it will try to re-build the in-memory cache 
using the files already present on the disk in the remote-index-cache 
directory. The process involves:
1. deleting existing files which are pending delete i.e. have a .delete suffix
2. read the cached index files, if present.
3. creating the indexes (this step will create a MMapped'buffer)
4. perform sanity check on the indexes
5. add to internal cache

The steps 2-5 are bound by the maximum number of entries in the cache. But step 
1 could be arbitrary large.

To debug a slow cache startup, we should add a info statement that prints the 
time it took to initialize the cache.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Permission to contribute to Apache Kafka

[Jacob]

Hi master,

Could you help grant me permissions to contribute to Kafka?

Wiki ID:  zixuanbai
Jira ID: zixuan

Thanks a lot!

Best regards,

Re: [DISCUSS] KIP-943: Add independent "offset.storage.segment.bytes" for connect-distributed.properties

[Yash Mayya]

Also, just adding to the above point - we don't necessarily need to
explicitly add a new worker configuration right? Instead, we could
potentially just use the new proposed default value internally which can be
overridden by users through setting a value for
"offset.storage.segment.bytes" (via the existing KIP-605 based mechanism).

On Thu, Jul 6, 2023 at 6:04 PM Yash Mayya  wrote:

> Hi hudeqi,
>
> Thanks for the KIP! Just to clarify - since KIP-605 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-605%3A+Expand+Connect+Worker+Internal+Topic+Settings)
> already allows configuring "segment.bytes" for the Connect cluster's
> offsets topic via a worker configuration ("offset.storage.segment.bytes",
> same as what is being proposed in this KIP), the primary motivation of this
> KIP is essentially to override the default value for that topic
> configuration to a smaller value and decouple it from the backing Kafka
> cluster's "log.segment.bytes" configuration? Also, I'm curious about how
> the new default value of 50 MB was chosen (if there were any experiments
> that were run etc.)?
>
> Thanks,
> Yash
>
> On Mon, Jul 3, 2023 at 6:08 PM hudeqi <16120...@bjtu.edu.cn> wrote:
>
>> Is anyone following this KIP? Bump this thread.
>
>

Re: [DISCUSS] KIP-943: Add independent "offset.storage.segment.bytes" for connect-distributed.properties

[Yash Mayya]

Hi hudeqi,

Thanks for the KIP! Just to clarify - since KIP-605 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-605%3A+Expand+Connect+Worker+Internal+Topic+Settings)
already allows configuring "segment.bytes" for the Connect cluster's
offsets topic via a worker configuration ("offset.storage.segment.bytes",
same as what is being proposed in this KIP), the primary motivation of this
KIP is essentially to override the default value for that topic
configuration to a smaller value and decouple it from the backing Kafka
cluster's "log.segment.bytes" configuration? Also, I'm curious about how
the new default value of 50 MB was chosen (if there were any experiments
that were run etc.)?

Thanks,
Yash

On Mon, Jul 3, 2023 at 6:08 PM hudeqi <16120...@bjtu.edu.cn> wrote:

> Is anyone following this KIP? Bump this thread.

[jira] [Resolved] (KAFKA-15151) Missing connector-stopped-task-count metric

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-15151.

Resolution: Invalid

> Missing connector-stopped-task-count metric
> ---
>
> Key: KAFKA-15151
> URL: https://issues.apache.org/jira/browse/KAFKA-15151
> Project: Kafka
>  Issue Type: Task
>  Components: KafkaConnect
>Reporter: Mickael Maison
>Assignee: Yash Mayya
>Priority: Major
>
> We have task-count metrics for all other states but when adding the STOPPED 
> state we did not add the respective metric.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-948: Allow custom prefix for internal topic names in Kafka Streams

[Igor Buzatovic]

Hi everyone,
I would like to start a discussion on the following KIP:

KIP-948: Allow custom prefix for internal topic names in Kafka Streams
https://cwiki.apache.org/confluence/display/KAFKA/KIP-948%3A+Allow+custom+prefix+for+internal+topic+names+in+Kafka+Streams

The goal is to provide greater flexibility in naming internal topics. The
main issue is that it affects Kafka Streams application reset tool (as
described in the KIP), so I guess the pros and cons should be weighed to
decide if this proposal makes sense.

Your feedback is appreciated. Thanks!

Igor Buzatović
Porsche Digital Croatia d.o.o.

[jira] [Created] (KAFKA-15156) Update cipherInformation correctly in DefaultChannelMetadataRegistry

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15156:


 Summary: Update cipherInformation correctly in 
DefaultChannelMetadataRegistry
 Key: KAFKA-15156
 URL: https://issues.apache.org/jira/browse/KAFKA-15156
 Project: Kafka
  Issue Type: Bug
Reporter: Divij Vaidya


At 
[https://github.com/apache/kafka/blob/4a61b48d3dca81e28b57f10af6052f36c50a05e3/clients/src/main/java/org/apache/kafka/common/network/DefaultChannelMetadataRegistry.java#L25]
 

we do not end up assigning the new value of cipherInformation to the member 
variable. 

The code over here, should be the following so that we can update the cipher 
information.
{noformat}
if (cipherInformation == null) {     
throw Illegal exception.
}
this.cipherInformation = cipherInformation{noformat}
 

 
While this class is only used in tests, we should still fix this. It's a minor 
bug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15155) Follow PEP 8 best practice in Python to check if a container is empty

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15155:


 Summary: Follow PEP 8 best practice in Python to check if a 
container is empty
 Key: KAFKA-15155
 URL: https://issues.apache.org/jira/browse/KAFKA-15155
 Project: Kafka
  Issue Type: Improvement
Reporter: Divij Vaidya


*This is a good task for first time contributors to Kafka*

At release.py Line:94, we don't follow PEP 8 [1] best practices.

To check if a container or sequence (string, list, tuple) is empty, use if not 
val. Do not compare its length using if len(val) == 0

[1] 
https://peps.python.org/pep-0008/#programming-recommendations#:~:text=if%20not%20seq



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15154) Potential bug: We don't acquire lock when reading checkQuotas

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15154:


 Summary: Potential bug: We don't acquire lock when reading 
checkQuotas
 Key: KAFKA-15154
 URL: https://issues.apache.org/jira/browse/KAFKA-15154
 Project: Kafka
  Issue Type: Bug
Reporter: Divij Vaidya


At sensor.java line:254, we call `this.metrics.values()`. metrics is not a 
thread safe map and that is why we acquire a lock whenever we want to 
add/remove entries from it. For example, see add(), hasMetrics() method.

However, we don't acquire a lock when calling Sensor#checkQuotas(timeMs).

This could lead to a situation where this metrics map may be left in an 
inconsistent state (since it is not thread safe for concurrent read/write 
access).

The objective of this task is to validate what I said above is correct and if 
yes, then fix the situation by enclosing this read in a lock. As a stretch 
task, we should consider if we can replace the metrics data structure which 
allows concurrent reads but exclusive writes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15153) Use Python `is` instead of `==` to compare for None

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15153:


 Summary: Use Python `is` instead of  `==` to compare for None
 Key: KAFKA-15153
 URL: https://issues.apache.org/jira/browse/KAFKA-15153
 Project: Kafka
  Issue Type: Improvement
Reporter: Divij Vaidya


*This is a good item to be picked by first time contributors to Kafka code base*

At release_notes.py Line: 47

The {{==}} and {{!=}} operators use the compared objects' {{__eq__}} method to 
test if they are equal. To check if an object is a singleton, such as 
{{{}None{}}}, we recommend that you use the {{is}} identity comparison operator.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15152) Fix incorrect format specifiers when formatting string

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15152:


 Summary: Fix incorrect format specifiers when formatting string
 Key: KAFKA-15152
 URL: https://issues.apache.org/jira/browse/KAFKA-15152
 Project: Kafka
  Issue Type: Improvement
Reporter: Divij Vaidya


*This is a good Jira to be picked up by first time contributors to Kafka code 
base.*

The objective of this Jira is to fix incorrect formatting of string at multiple 
places in the code which can cause incorrect print when used in a different 
locale.

*1. FollowerState.java Line: 121*

This code uses '%s' to format long: updatedHighWatermark (declared at line 
117). This is a potential locale-sensitive handling issue. It might cause 
errors in the handling and processing of the statement at line: 121. Consider 
formatting this data with '%d' instead.

*2. MemoryRecordsBuilder.java Line: 441*

This code uses '%s' to format long: offset (declared at line 434), lastOffset. 
This is a potential locale-sensitive handling issue. It might cause errors in 
the handling and processing of the statement at line: 441. Consider formatting 
this data with '%d' instead.

*3. HttpAccessTokenRetriever.java Line: 340*

This code uses '%s' to format int: MAX_RESPONSE_BODY_LENGTH, actualLength 
(declared at line 338). This is a potential locale-sensitive handling issue. It 
might cause errors in the handling and processing of the statement at line: 
340. Consider formatting this data with '%d' instead.

4. *KafkaAdminClient.java Line: 1256*

This code uses '%s' to format int: correlationId (declared at line 1226). This 
is a potential locale-sensitive handling issue. It might cause errors in the 
handling and processing of the statement at line: 1256. Consider formatting 
this data with '%d' instead.

*5 . Batch.java Line: 170*
This code uses '%s' to format int: epoch (declared at line 163). This is a 
potential locale-sensitive handling issue. It might cause errors in the 
handling and processing of the statement at line: 170. Consider formatting this 
data with '%d' instead.

*6. Batch.java Line: 207*
This code uses '%s' to format int: epoch (declared at line 200). This is a 
potential locale-sensitive handling issue. It might cause errors in the 
handling and processing of the statement at line: 207. Consider formatting this 
data with '%d' instead.

*7. BatchBuilder.java Line: 330*
This code uses '%s' to format int: 'size' expression. This is a potential 
locale-sensitive handling issue. It might cause errors in the handling and 
processing of the statement at line: 330. Consider formatting this data with 
'%d' instead.

*8. CopartitionedTopicsEnforcer.java Line: 104*
This code uses '%s' to format int: numberOfPartitionsOfInternalTopic (declared 
at line 99), numPartitionsToUseForRepartitionTopics (defined at line 81). This 
is a potential locale-sensitive handling issue. It might cause errors in the 
handling and processing of the statement at line: 104. Consider formatting this 
data with '%d' instead.

 

*9. RefreshingHttpsJwks.java Line: 337*
This code uses '%s' to format int: MISSING_KEY_ID_MAX_KEY_LENGTH, actualLength 
(declared at line 335). This is a potential locale-sensitive handling issue. It 
might cause errors in the handling and processing of the statement at line: 
337. Consider formatting this data with '%d' instead.

*10. SerializedJwt.java Line: 47*
This code uses '%s' to format int: splits.length. This is a potential 
locale-sensitive handling issue. It might cause errors in the handling and 
processing of the statement at line: 47. Consider formatting this data with 
'%d' instead.

*11. ControllerResultAndOffset.java Line: 57*
This code uses '%s' to format long: offset. This is a potential 
locale-sensitive handling issue. It might cause errors in the handling and 
processing of the statement at line: 57. Consider formatting this data with 
'%d' instead.

*12. ReplicatedLog.java Line: 101*
This code uses '%s' to format long: 'startOffset' expression. This is a 
potential locale-sensitive handling issue. It might cause errors in the 
handling and processing of the statement at line: 101. Consider formatting this 
data with '%d' instead.

*13. HttpAccessTokenRetriever.java Line: 275*
This code uses '%s' to format int: responseCode (declared at line 240). This is 
a potential locale-sensitive handling issue. It might cause errors in the 
handling and processing of the statement at line: 275. Consider formatting this 
data with '%d' instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15151) Missing connector-stopped-task-count metric

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-15151:
--

 Summary: Missing connector-stopped-task-count metric
 Key: KAFKA-15151
 URL: https://issues.apache.org/jira/browse/KAFKA-15151
 Project: Kafka
  Issue Type: Task
  Components: KafkaConnect
Reporter: Mickael Maison


We have task-count metrics for all other states but when adding the STOPPED 
state we did not add the respective metric.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)