Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #869

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-613: Add end-to-end latency metrics to Streams

[John Roesler]

Hi Chaimaa,

Interesting... That basis for those metrics is the difference between a 
record's timestamp and the current system time while processing it. I wonder if 
it's negative because of a difference in the way that the producer sets the 
records' timestamps versus the current time on the Streams instances? For the 
times that you see NaN, I believe that's expected when there have not yet been 
any data points measured in the metric yet.

I hope this helps,
-John

On Thu, Apr 14, 2022, at 04:46, Chaimaa LOTFI wrote:
> CONFIDENTIAL & RESTRICTED
>
>
> Hello, I hope that you are doing well.
> I have a quick question please, I created a Kafka streams dashboard 
> which works well, expect that we got a negative latency for record 
> end2end latency with this metric 
> (kafka_streams_stream_processor_node_metrics_record_e2e_latency_min), I 
> was looking why, while I saw in Thanos that it gives either NaN or a 
> negative value! so I am still looking for that. Do you have any idea 
> why?
> Like we can't get a negative latency anyways, so what does this metric 
> calculate exactly?
> Thank you in advance!
> Best regards
>
> On 2020/05/13 02:27:54 Sophie Blee-Goldman wrote:
>> Hey all,
>>
>> I'd like to kick off discussion on KIP-613 which aims to add end-to-end
>> latency metrics to Streams. Please take a look:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams
>>
>> Cheers,
>> Sophie
>>

Re: [DISCUSS] KIP-829: (console-consumer) add print.topic property

[John Roesler]

Hi Evans,

Thanks for the KIP!

I agree with Mickael; it would be good to see an example in the KIP.

Otherwise, I'm in favor.

Thanks,
-John 

On Fri, Apr 15, 2022, at 04:33, Mickael Maison wrote:
> Hi,
>
> I can't believe this was missing! Thanks for addressing this.
> Can you add an example of the expected output with the topic in the KIP?
>
> Thanks,
> Mickael
>
> On Sat, Apr 9, 2022 at 12:11 PM Evans Jahja  wrote:
>>
>> Good evening.
>>
>> My name is Evans, I am proposing KIP-829.
>>
>> Brief: Using `kafka-console-consumer` with `--include` argument listens
>> from multiple topics, but there is no way to know which topic the message
>> came from. This proposal allows printing the topic name.
>>
>> Requesting for comments.
>>
>> Confluence
>> 
>> Github 
>>
>> Best Regards,
>>
>> Evans Jahja

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #868

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.2 #33

[Apache Jenkins Server]

See 

[VOTE] 3.2.0 RC0

[Bruno Cadonna]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.2.0.

* log4j 1.x is replaced with reload4j (KAFKA-9366)
* StandardAuthorizer for KRaft (KIP-801)
* Send a hint to the partition leader to recover the partition (KIP-704)
* Top-level error code field in DescribeLogDirsResponse (KIP-784)
* kafka-console-producer writes headers and null values (KIP-798 and 
KIP-810)

* JoinGroupRequest and LeaveGroupRequest have a reason attached (KIP-800)
* Static membership protocol lets the leader skip assignment (KIP-814)
* Rack-aware standby task assignment in Kafka Streams (KIP-708)
* Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
* Connect APIs list all connector plugins and retrieve their 
configuration (KIP-769)

* TimestampConverter SMT supports different unix time precisions (KIP-808)
* Connect source tasks handle producer exceptions (KIP-779)

Release notes for the 3.2.0 release:
https://home.apache.org/~cadonna/kafka-3.2.0-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Monday, April 25, 9am CEST

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~cadonna/kafka-3.2.0-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~cadonna/kafka-3.2.0-rc0/javadoc/

* Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
https://github.com/apache/kafka/releases/tag/3.2.0-rc0

* Documentation (not yet ported to kafka-site):
https://kafka.apache.org/32/documentation.html

* Protocol:
https://kafka.apache.org/32/protocol.html

* Successful Jenkins builds for the 3.2 branch:
I'll share a link once the builds complete


/**

Thanks,
Bruno

Re: [VOTE] KIP-825: introduce a new API to control when aggregated results are produced

[Hao Li]

Hi Guozhang,

Thanks for the suggestion.

1. I have no problem with

window-aggregate-final-emit-(rate | total) and
window-aggregate-final-emit-latency-(avg | max)

2. Good question, it should be the time to emit final when it COULD be
emitted. Will update KIP and PR.

Hao


On Thu, Apr 14, 2022 at 5:38 PM Guozhang Wang  wrote:

> Hello Hao,
>
> Just a bit further suggestion to align with KIP-444 (
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams
> ):
> in the suppression processor node we have an existing metric as follows:
>
> suppression-emit (rate | total)
>
>
> So I'm feeling maybe we can name it more explicitly as
> window-aggregate-final-emit-(rate | total) and
> window-aggregate-final-emit-latency-(avg | max). WDYT?
>
> Also I'm wondering how the latency is measured in the metrics? Is that
> measured as the time difference between "when a final result record COULD
> be emitted" and "when that final result record is actually emitted"?
> Anyways, maybe it's better to elaborate clearly on how the latency is
> measured.
>
>
> Guozhang
>
> On Tue, Apr 12, 2022 at 9:50 AM Hao Li  wrote:
>
> > Thanks for the feedback Bruno and John.
> >
> > 1. Proposed name by John sounds good to me!
> > 2. I will use processor-node metrics with debug level since this doesn't
> > seem like top level metrics and make more sense to not mix them in task.
> > 3. Will update KIP as Bruno suggested.
> >
> > Thanks,
> > Hao
> >
> > On Tue, Apr 12, 2022 at 1:30 AM Bruno Cadonna 
> wrote:
> >
> > > Hi Hao,
> > >
> > > Thanks for the addition!
> > >
> > > I second what John said about the naming.
> > >
> > > Could you please describe the metrics as has been previously done in
> > > KIP-471, KIP-613, or KIP-761?
> > > That would make the metrics more concise and clear. In addition,
> > > TaskMetrics is an internal class that is an implementation detail and
> > > hence not intended to be shown in a KIP.
> > >
> > > Best,
> > > Bruno
> > >
> > > On 12.04.22 06:03, John Roesler wrote:
> > > > Thanks, Hao!
> > > >
> > > > I have no concern about amending the KIP to add metrics.
> > > > Thanks for thinking of it.
> > > >
> > > > Can you comment on the choice to add them as a task-level metric
> > > > instead of a processor-level metric? This will cause the metrics for
> > > > all windowed aggregations in a task that use final emission to be
> > > > mixed together. It might be fine, but we should at least document
> > > > that it was anticipated and the reasons for the choice. By the way,
> > > > if we do add them as processor-node metrics but want them to
> > > > be measured at info level, we should also state it, since processor-
> > > > node metrics are usually debug.
> > > >
> > > > Also, I'm concerned that the name `emitted-records` will be
> > > > ambiguous in the larger context of all Kafka Streams metrics. If I'm
> > > > right in thinking that these metrics are only for measuring the
> > > > behavior of emit-final windowed aggregations, then we should
> > > > make sure that the metric name says as much. Maybe:
> > > >
> > > > emit-final-records-[rate|total]
> > > > emit-final-latency-[avg|max]
> > > >
> > > > Thanks!
> > > > -John
> > > >
> > > > On Mon, Apr 11, 2022, at 14:25, Hao Li wrote:
> > > >> Hi all,
> > > >>
> > > >> I would like to introduce two metrics in this KIP as well to measure
> > the
> > > >> latency and number of records emitted for emit final. They are
> named:
> > > >>
> > > >> `emit-final-latency`
> > > >> `emitted-records`
> > > >>
> > > >> I've updated the KIP with details in
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-825%3A+introduce+a+new+API+to+control+when+aggregated+results+are+produced
> > > >>
> > > >> Can you take a look and see if you have any concerns for the
> metrics?
> > > >>
> > > >> Thanks,
> > > >> Hao
> > > >>
> > > >> On Fri, Mar 25, 2022 at 8:27 AM Hao Li  wrote:
> > > >>
> > > >>> Got it. Forgot that. Yeah, it’s still open and ppl can still vote.
> > > Thanks
> > > >>> for reminding!
> > > >>>
> > > >>> Hao Li
> > > >>>
> > >  On Mar 25, 2022, at 8:22 AM, Guozhang Wang 
> > > wrote:
> > > 
> > >  Hello Hao,
> > > 
> > >  According to bylaws the voting has to last for at least 72
> business
> > > >>> hours.
> > >  So let's wait a bit longer to see if there are different opinions
> > > before
> > >  calling it close.
> > > 
> > > > On Thu, Mar 24, 2022 at 4:20 PM Hao Li  >
> > > >>> wrote:
> > > >
> > > > The vote happened in the discussion thread since I started the
> vote
> > > >>> there
> > > > by mistake. But it passed there. To avoid having everyone vote
> > > again. I
> > > > copied the content from that thread here:
> > > >
> > > >  end of discussion thread vote
> > > > ==
> > > > The vote passed with 5 binding votes from John, Guozhang, Bruno,
> > > >>> Matthias
> > > > and 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #867

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 527016 lines...]
[2022-04-15T18:27:02.792Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = true] PASSED
[2022-04-15T18:27:02.792Z] 
[2022-04-15T18:27:02.792Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] STARTED
[2022-04-15T18:27:10.312Z] 
[2022-04-15T18:27:10.312Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] PASSED
[2022-04-15T18:27:10.312Z] 
[2022-04-15T18:27:10.312Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = true] STARTED
[2022-04-15T18:27:11.938Z] 
[2022-04-15T18:27:11.938Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftInner[caching enabled = false] PASSED
[2022-04-15T18:27:11.938Z] 
[2022-04-15T18:27:11.938Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = false] STARTED
[2022-04-15T18:27:19.823Z] 
[2022-04-15T18:27:19.823Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = true] PASSED
[2022-04-15T18:27:19.823Z] 
[2022-04-15T18:27:19.823Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = true] STARTED
[2022-04-15T18:27:25.729Z] 
[2022-04-15T18:27:25.729Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = false] PASSED
[2022-04-15T18:27:25.729Z] 
[2022-04-15T18:27:25.729Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftLeft[caching enabled = false] STARTED
[2022-04-15T18:27:30.881Z] 
[2022-04-15T18:27:30.881Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = true] PASSED
[2022-04-15T18:27:30.881Z] 
[2022-04-15T18:27:30.881Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testMultiInner[caching enabled = true] STARTED
[2022-04-15T18:27:39.312Z] 
[2022-04-15T18:27:39.312Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftLeft[caching enabled = false] PASSED
[2022-04-15T18:27:39.312Z] 
[2022-04-15T18:27:39.312Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterLeft[caching enabled = false] STARTED
[2022-04-15T18:27:46.362Z] 
[2022-04-15T18:27:46.362Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testMultiInner[caching enabled = true] PASSED
[2022-04-15T18:27:46.362Z] 
[2022-04-15T18:27:46.362Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeftRepartitioned[caching enabled = true] STARTED
[2022-04-15T18:27:54.861Z] 
[2022-04-15T18:27:54.861Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterLeft[caching enabled = false] PASSED
[2022-04-15T18:27:54.861Z] 
[2022-04-15T18:27:54.861Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInner[caching enabled = false] STARTED
[2022-04-15T18:28:03.536Z] 
[2022-04-15T18:28:03.536Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeftRepartitioned[caching enabled = true] PASSED
[2022-04-15T18:28:03.536Z] 
[2022-04-15T18:28:03.536Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = false] STARTED
[2022-04-15T18:28:08.097Z] 
[2022-04-15T18:28:08.097Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInner[caching enabled = false] PASSED
[2022-04-15T18:28:08.097Z] 
[2022-04-15T18:28:08.097Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuter[caching enabled = false] STARTED
[2022-04-15T18:28:15.178Z] 
[2022-04-15T18:28:15.178Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = false] PASSED
[2022-04-15T18:28:15.178Z] 
[2022-04-15T18:28:15.178Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = false] STARTED
[2022-04-15T18:28:19.364Z] 
[2022-04-15T18:28:19.364Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuter[caching enabled = false] PASSED
[2022-04-15T18:28:19.364Z] 
[2022-04-15T18:28:19.364Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeft[caching enabled = false] STARTED
[2022-04-15T18:28:29.777Z] 
[2022-04-15T18:28:29.777Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = false] PASSED
[2022-04-15T18:28:29.777Z] 
[2022-04-15T18:28:29.777Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = false] 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.2 #32

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 523771 lines...]
[2022-04-15T17:02:35.639Z] 
org.apache.kafka.streams.integration.RocksDBMetricsIntegrationTest > 
shouldExposeRocksDBMetricsBeforeAndAfterFailureWithEmptyStateDir[exactly_once] 
PASSED
[2022-04-15T17:02:35.639Z] 
[2022-04-15T17:02:35.639Z] 
org.apache.kafka.streams.integration.RocksDBMetricsIntegrationTest > 
shouldExposeRocksDBMetricsBeforeAndAfterFailureWithEmptyStateDir[exactly_once_v2]
 STARTED
[2022-04-15T17:02:36.692Z] 
[2022-04-15T17:02:36.693Z] 
org.apache.kafka.streams.processor.internals.HandlingSourceTopicDeletionIntegrationTest
 > shouldThrowErrorAfterSourceTopicDeleted PASSED
[2022-04-15T17:02:40.861Z] streams-2: SMOKE-TEST-CLIENT-CLOSED
[2022-04-15T17:02:40.861Z] streams-0: SMOKE-TEST-CLIENT-CLOSED
[2022-04-15T17:02:40.861Z] streams-4: SMOKE-TEST-CLIENT-CLOSED
[2022-04-15T17:02:40.861Z] streams-6: SMOKE-TEST-CLIENT-CLOSED
[2022-04-15T17:02:40.861Z] streams-3: SMOKE-TEST-CLIENT-CLOSED
[2022-04-15T17:02:40.861Z] streams-1: SMOKE-TEST-CLIENT-CLOSED
[2022-04-15T17:02:40.861Z] streams-5: SMOKE-TEST-CLIENT-CLOSED
[2022-04-15T17:03:25.227Z] 
[2022-04-15T17:03:25.227Z] 
org.apache.kafka.streams.integration.RocksDBMetricsIntegrationTest > 
shouldExposeRocksDBMetricsBeforeAndAfterFailureWithEmptyStateDir[exactly_once_v2]
 PASSED
[2022-04-15T17:03:28.224Z] 
[2022-04-15T17:03:28.224Z] 
org.apache.kafka.streams.integration.StandbyTaskCreationIntegrationTest > 
shouldNotCreateAnyStandByTasksForStateStoreWithLoggingDisabled STARTED
[2022-04-15T17:03:30.121Z] 
[2022-04-15T17:03:30.121Z] 
org.apache.kafka.streams.integration.StandbyTaskCreationIntegrationTest > 
shouldNotCreateAnyStandByTasksForStateStoreWithLoggingDisabled PASSED
[2022-04-15T17:03:30.121Z] 
[2022-04-15T17:03:30.121Z] 
org.apache.kafka.streams.integration.StandbyTaskCreationIntegrationTest > 
shouldCreateStandByTasksForMaterializedAndOptimizedSourceTables STARTED
[2022-04-15T17:03:31.853Z] 
[2022-04-15T17:03:31.853Z] 
org.apache.kafka.streams.integration.StandbyTaskCreationIntegrationTest > 
shouldCreateStandByTasksForMaterializedAndOptimizedSourceTables PASSED
[2022-04-15T17:03:32.865Z] 
[2022-04-15T17:03:32.865Z] 
org.apache.kafka.streams.integration.StateDirectoryIntegrationTest > 
testNotCleanUpStateDirIfNotEmpty STARTED
[2022-04-15T17:03:36.892Z] 
[2022-04-15T17:03:36.892Z] 
org.apache.kafka.streams.integration.StateDirectoryIntegrationTest > 
testNotCleanUpStateDirIfNotEmpty PASSED
[2022-04-15T17:03:36.892Z] 
[2022-04-15T17:03:36.892Z] 
org.apache.kafka.streams.integration.StateDirectoryIntegrationTest > 
testCleanUpStateDirIfEmpty STARTED
[2022-04-15T17:03:38.650Z] 
[2022-04-15T17:03:38.650Z] 
org.apache.kafka.streams.integration.StateDirectoryIntegrationTest > 
testCleanUpStateDirIfEmpty PASSED
[2022-04-15T17:03:39.851Z] 
[2022-04-15T17:03:39.851Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificActivePartitionStores STARTED
[2022-04-15T17:03:44.078Z] 
[2022-04-15T17:03:44.078Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificActivePartitionStores PASSED
[2022-04-15T17:03:44.078Z] 
[2022-04-15T17:03:44.078Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryAllStalePartitionStores STARTED
[2022-04-15T17:03:49.910Z] 
[2022-04-15T17:03:49.910Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryAllStalePartitionStores PASSED
[2022-04-15T17:03:49.910Z] 
[2022-04-15T17:03:49.910Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads STARTED
[2022-04-15T17:03:54.048Z] 
[2022-04-15T17:03:54.048Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads PASSED
[2022-04-15T17:03:54.048Z] 
[2022-04-15T17:03:54.048Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStores STARTED
[2022-04-15T17:03:57.845Z] 
[2022-04-15T17:03:57.845Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStores PASSED
[2022-04-15T17:03:57.845Z] 
[2022-04-15T17:03:57.845Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryOnlyActivePartitionStoresByDefault STARTED
[2022-04-15T17:04:02.652Z] 
[2022-04-15T17:04:02.652Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryOnlyActivePartitionStoresByDefault PASSED
[2022-04-15T17:04:02.652Z] 
[2022-04-15T17:04:02.652Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread STARTED
[2022-04-15T17:04:10.120Z] 
[2022-04-15T17:04:10.120Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread PASSED

[jira] [Resolved] (KAFKA-13624) Add Metric for Store Cache Size

[Sagar Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sagar Rao resolved KAFKA-13624.
---
Resolution: Won't Fix

Taken care as part of KIP-770

> Add Metric for Store Cache Size
> ---
>
> Key: KAFKA-13624
> URL: https://issues.apache.org/jira/browse/KAFKA-13624
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: needs-kip
>
> KIP-770 introduced a new metric called `{*}input-buffer-bytes-total`{*} to 
> track the total amount of bytes accumulated by a task. While working through 
> it's PR, it was suggested to add a similar metric for 
> *cache-size-bytes-total* to track the cache size in bytes for a task. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #866

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Mickael Maison]

Hi Luke,

7. I've updated the KIP to clarify these sizes are in bytes.

Thanks,
Mickael

On Fri, Apr 15, 2022 at 12:16 PM Luke Chen  wrote:
>
> Hi Mickael,
>
> Thanks for the KIP!
> This is a good improvement.
>
> (3) +1 for not adding the number of files in the directory. Counting the
> file numbers should be slow.
> (7) Could you make the fields clear in `DescribeLogDirsResponse`, to
> mention the returned number is size in Byte (or not?)
>
> Thank you.
> Luke
>
> On Fri, Apr 15, 2022 at 5:27 PM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > Thanks for the feedback.
> >
> > 3. Yes that's right. Also the number of file descriptors is really not
> > a property of log directories. Administrators typically tracked that
> > count per process and for the whole operating system.
> >
> > 5. That's a good point, I've updated the KIP to mention sizes will be
> > capped to Long.MAX_VALUE even if the actual storage is larger.
> >
> > 6. Brokers would never return UNKNOWN_SPACE. When new clients query
> > older brokers via the admin API, the admin client will use
> > UNKNOWN_SPACE to indicate these values weren't provided by brokers.
> >
> > Thanks,
> > Mickael
> >
> > On Fri, Apr 8, 2022 at 5:00 PM Divij Vaidya 
> > wrote:
> > >
> > > Thanks for replying. I still have a few lingering questions/comments.
> > >
> > > *Reg#1* Understood. I checked and the underlying system call is statvfs
> > for
> > > unix systems which should be ok to call here.
> > > *Reg#2* Fair point. I checked again and yes, log.dir always means local
> > > storage even when tiered storage is enabled.
> > > *Reg#3* The rationale for adding these new (size) fields to the
> > > `DescribeLogDirs` is to allow the administrator to monitor or perhaps
> > take
> > > automated action based on results. Doesn't monitoring the number of file
> > > descriptors fall in the same category of use cases? I am assuming that we
> > > want to add the size information in the API response because JVM makes it
> > > possible to get this information in a platform agnostic manner which is
> > not
> > > true for open file descriptors, correct?
> > > *Reg#4* Agree.
> > > *New#5*: As an FYI, Java FileStore API breaks on large storage sizes.
> > See:
> > > https://bugs.openjdk.java.net/browse/JDK-8162520. ElasticSearch has been
> > > hit by these limitations in the past. For JDK 11, you will probably have
> > to
> > > add defensive checks such as
> > >
> > https://github.com/opensearch-project/OpenSearch/blob/b74d71fb747cc2873d4c2ffae825944da4d06e1b/server/src/main/java/org/opensearch/monitor/fs/FsProbe.java#L148
> > .
> > > The documentation of the API mentioned in KIP will also be modified to
> > > account for this edge case.
> > > *New#6*: Can you please provide an example where the return for these
> > APIs
> > > would be UNKNOWN_SPACE? Doesn't JVM guarantee that this API will
> > definitely
> > > return results (else it throws an IOException)? I would propose that we
> > get
> > > rid of default since JVM guarantees that this would work on all
> > platforms.
> > > If it doesn't then it's a bug and should be uncovered via an exception.
> > >
> > > Also, I would like to volunteer to code review (of course, it would be
> > > non-binding) your implementation once this KIP is approved.
> > >
> > > Regards,
> > > Divij Vaidya
> > >
> > > On Fri, Apr 8, 2022 at 11:35 AM Mickael Maison  > >
> > > wrote:
> > >
> > > > Hi Divij,
> > > >
> > > > Thanks for taking a look!
> > > >
> > > > 1. In order to retrieve the sizes, the plan is to use getTotalSpace()
> > > > and getUsableSpace() from java.nio.file.FileStore. The implementations
> > > > may vary depending on the filesystem but these calls typically don't
> > > > depend on the size of storage but instead just return metadata the
> > > > filesystem maintains.
> > > > 2. I'm not an expert on KIP-405, so correct me if I'm wrong. As far as
> > > > I understand brokers will still have local log dirs and remote volumes
> > > > are not counted as log dirs. KIP-405 does not mention updating the
> > > > DescribeLogDirs API. So I don't think this KIP needs to do anything
> > > > special to be compatible with KIP-405. On the other hand, I wonder if
> > > > KIP-405 should update DescribeLogDirs to provide details about the
> > > > location of replicas.
> > > > 3. Counting files can be a slow operation as it requires exploring all
> > > > paths recursively to find all files. Administrators should
> > > > definitively monitor file descriptors via host metrics but I'm not
> > > > sure it's something we want to expose via the Kafka API. As mentioned
> > > > it could be slow to compute and files are not really a Kafka concept.
> > > > 4. DescribeLogDirs is usually a low volume API. This change should not
> > > > significantly affect the latency of this API.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Apr 7, 2022 at 1:41 PM Divij Vaidya 
> > > > wrote:
> > > > >
> > > > > Hi Mickael
> > > > >
> > > > > 

Re: maxParallelForks while running tests

[Unmesh Joshi]

I am not running into any specific issue with kafka code. Was just curious
to know if there were any specific fixes done to handle this issue.

On Fri, Apr 15, 2022 at 8:24 AM Luke Chen  wrote:

> Hi Unmesh,
>
> Are you running into any issue with that?
>
> So far, the `maxParallelForks` can be set via gradle argument:
> https://github.com/apache/kafka/blob/trunk/build.gradle#L78
>
> And in Jenkins, it looks like we default to 2.
> https://github.com/apache/kafka/blob/trunk/Jenkinsfile#L40
>
> Thank you.
> Luke
>
> On Fri, Apr 15, 2022 at 1:24 AM Unmesh Joshi 
> wrote:
>
> > Hi,
> > I came across this issue which has discussion about capping
> > maxParallelForks while running tests.
> > https://issues.apache.org/jira/browse/KAFKA-2613
> > Is this still the case?
> >
> > Thanks,
> > Unmesh
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #865

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.2 #31

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #864

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Luke Chen]

Hi Mickael,

Thanks for the KIP!
This is a good improvement.

(3) +1 for not adding the number of files in the directory. Counting the
file numbers should be slow.
(7) Could you make the fields clear in `DescribeLogDirsResponse`, to
mention the returned number is size in Byte (or not?)

Thank you.
Luke

On Fri, Apr 15, 2022 at 5:27 PM Mickael Maison 
wrote:

> Hi,
>
> Thanks for the feedback.
>
> 3. Yes that's right. Also the number of file descriptors is really not
> a property of log directories. Administrators typically tracked that
> count per process and for the whole operating system.
>
> 5. That's a good point, I've updated the KIP to mention sizes will be
> capped to Long.MAX_VALUE even if the actual storage is larger.
>
> 6. Brokers would never return UNKNOWN_SPACE. When new clients query
> older brokers via the admin API, the admin client will use
> UNKNOWN_SPACE to indicate these values weren't provided by brokers.
>
> Thanks,
> Mickael
>
> On Fri, Apr 8, 2022 at 5:00 PM Divij Vaidya 
> wrote:
> >
> > Thanks for replying. I still have a few lingering questions/comments.
> >
> > *Reg#1* Understood. I checked and the underlying system call is statvfs
> for
> > unix systems which should be ok to call here.
> > *Reg#2* Fair point. I checked again and yes, log.dir always means local
> > storage even when tiered storage is enabled.
> > *Reg#3* The rationale for adding these new (size) fields to the
> > `DescribeLogDirs` is to allow the administrator to monitor or perhaps
> take
> > automated action based on results. Doesn't monitoring the number of file
> > descriptors fall in the same category of use cases? I am assuming that we
> > want to add the size information in the API response because JVM makes it
> > possible to get this information in a platform agnostic manner which is
> not
> > true for open file descriptors, correct?
> > *Reg#4* Agree.
> > *New#5*: As an FYI, Java FileStore API breaks on large storage sizes.
> See:
> > https://bugs.openjdk.java.net/browse/JDK-8162520. ElasticSearch has been
> > hit by these limitations in the past. For JDK 11, you will probably have
> to
> > add defensive checks such as
> >
> https://github.com/opensearch-project/OpenSearch/blob/b74d71fb747cc2873d4c2ffae825944da4d06e1b/server/src/main/java/org/opensearch/monitor/fs/FsProbe.java#L148
> .
> > The documentation of the API mentioned in KIP will also be modified to
> > account for this edge case.
> > *New#6*: Can you please provide an example where the return for these
> APIs
> > would be UNKNOWN_SPACE? Doesn't JVM guarantee that this API will
> definitely
> > return results (else it throws an IOException)? I would propose that we
> get
> > rid of default since JVM guarantees that this would work on all
> platforms.
> > If it doesn't then it's a bug and should be uncovered via an exception.
> >
> > Also, I would like to volunteer to code review (of course, it would be
> > non-binding) your implementation once this KIP is approved.
> >
> > Regards,
> > Divij Vaidya
> >
> > On Fri, Apr 8, 2022 at 11:35 AM Mickael Maison  >
> > wrote:
> >
> > > Hi Divij,
> > >
> > > Thanks for taking a look!
> > >
> > > 1. In order to retrieve the sizes, the plan is to use getTotalSpace()
> > > and getUsableSpace() from java.nio.file.FileStore. The implementations
> > > may vary depending on the filesystem but these calls typically don't
> > > depend on the size of storage but instead just return metadata the
> > > filesystem maintains.
> > > 2. I'm not an expert on KIP-405, so correct me if I'm wrong. As far as
> > > I understand brokers will still have local log dirs and remote volumes
> > > are not counted as log dirs. KIP-405 does not mention updating the
> > > DescribeLogDirs API. So I don't think this KIP needs to do anything
> > > special to be compatible with KIP-405. On the other hand, I wonder if
> > > KIP-405 should update DescribeLogDirs to provide details about the
> > > location of replicas.
> > > 3. Counting files can be a slow operation as it requires exploring all
> > > paths recursively to find all files. Administrators should
> > > definitively monitor file descriptors via host metrics but I'm not
> > > sure it's something we want to expose via the Kafka API. As mentioned
> > > it could be slow to compute and files are not really a Kafka concept.
> > > 4. DescribeLogDirs is usually a low volume API. This change should not
> > > significantly affect the latency of this API.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > >
> > >
> > > On Thu, Apr 7, 2022 at 1:41 PM Divij Vaidya 
> > > wrote:
> > > >
> > > > Hi Mickael
> > > >
> > > > Thanks for starting this. It is a very useful feature.
> > > >
> > > > Some initial thoughts (I am new to Kafka so please excuse if these
> are
> > > > naive suggestions):
> > > > 1. What is the impact on latency of the DescribeLogDirs API due to
> this
> > > > change? Would calculating the totalSpace from each logdir be a
> bottleneck
> > > > for the API? What if we are talking 

Re: [DISCUSS] KIP-829: (console-consumer) add print.topic property

[Mickael Maison]

Hi,

I can't believe this was missing! Thanks for addressing this.
Can you add an example of the expected output with the topic in the KIP?

Thanks,
Mickael

On Sat, Apr 9, 2022 at 12:11 PM Evans Jahja  wrote:
>
> Good evening.
>
> My name is Evans, I am proposing KIP-829.
>
> Brief: Using `kafka-console-consumer` with `--include` argument listens
> from multiple topics, but there is no way to know which topic the message
> came from. This proposal allows printing the topic name.
>
> Requesting for comments.
>
> Confluence
> 
> Github 
>
> Best Regards,
>
> Evans Jahja

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Mickael Maison]

Hi,

Thanks for the feedback.

3. Yes that's right. Also the number of file descriptors is really not
a property of log directories. Administrators typically tracked that
count per process and for the whole operating system.

5. That's a good point, I've updated the KIP to mention sizes will be
capped to Long.MAX_VALUE even if the actual storage is larger.

6. Brokers would never return UNKNOWN_SPACE. When new clients query
older brokers via the admin API, the admin client will use
UNKNOWN_SPACE to indicate these values weren't provided by brokers.

Thanks,
Mickael

On Fri, Apr 8, 2022 at 5:00 PM Divij Vaidya  wrote:
>
> Thanks for replying. I still have a few lingering questions/comments.
>
> *Reg#1* Understood. I checked and the underlying system call is statvfs for
> unix systems which should be ok to call here.
> *Reg#2* Fair point. I checked again and yes, log.dir always means local
> storage even when tiered storage is enabled.
> *Reg#3* The rationale for adding these new (size) fields to the
> `DescribeLogDirs` is to allow the administrator to monitor or perhaps take
> automated action based on results. Doesn't monitoring the number of file
> descriptors fall in the same category of use cases? I am assuming that we
> want to add the size information in the API response because JVM makes it
> possible to get this information in a platform agnostic manner which is not
> true for open file descriptors, correct?
> *Reg#4* Agree.
> *New#5*: As an FYI, Java FileStore API breaks on large storage sizes. See:
> https://bugs.openjdk.java.net/browse/JDK-8162520. ElasticSearch has been
> hit by these limitations in the past. For JDK 11, you will probably have to
> add defensive checks such as
> https://github.com/opensearch-project/OpenSearch/blob/b74d71fb747cc2873d4c2ffae825944da4d06e1b/server/src/main/java/org/opensearch/monitor/fs/FsProbe.java#L148.
> The documentation of the API mentioned in KIP will also be modified to
> account for this edge case.
> *New#6*: Can you please provide an example where the return for these APIs
> would be UNKNOWN_SPACE? Doesn't JVM guarantee that this API will definitely
> return results (else it throws an IOException)? I would propose that we get
> rid of default since JVM guarantees that this would work on all platforms.
> If it doesn't then it's a bug and should be uncovered via an exception.
>
> Also, I would like to volunteer to code review (of course, it would be
> non-binding) your implementation once this KIP is approved.
>
> Regards,
> Divij Vaidya
>
> On Fri, Apr 8, 2022 at 11:35 AM Mickael Maison 
> wrote:
>
> > Hi Divij,
> >
> > Thanks for taking a look!
> >
> > 1. In order to retrieve the sizes, the plan is to use getTotalSpace()
> > and getUsableSpace() from java.nio.file.FileStore. The implementations
> > may vary depending on the filesystem but these calls typically don't
> > depend on the size of storage but instead just return metadata the
> > filesystem maintains.
> > 2. I'm not an expert on KIP-405, so correct me if I'm wrong. As far as
> > I understand brokers will still have local log dirs and remote volumes
> > are not counted as log dirs. KIP-405 does not mention updating the
> > DescribeLogDirs API. So I don't think this KIP needs to do anything
> > special to be compatible with KIP-405. On the other hand, I wonder if
> > KIP-405 should update DescribeLogDirs to provide details about the
> > location of replicas.
> > 3. Counting files can be a slow operation as it requires exploring all
> > paths recursively to find all files. Administrators should
> > definitively monitor file descriptors via host metrics but I'm not
> > sure it's something we want to expose via the Kafka API. As mentioned
> > it could be slow to compute and files are not really a Kafka concept.
> > 4. DescribeLogDirs is usually a low volume API. This change should not
> > significantly affect the latency of this API.
> >
> > Thanks,
> > Mickael
> >
> >
> >
> >
> > On Thu, Apr 7, 2022 at 1:41 PM Divij Vaidya 
> > wrote:
> > >
> > > Hi Mickael
> > >
> > > Thanks for starting this. It is a very useful feature.
> > >
> > > Some initial thoughts (I am new to Kafka so please excuse if these are
> > > naive suggestions):
> > > 1. What is the impact on latency of the DescribeLogDirs API due to this
> > > change? Would calculating the totalSpace from each logdir be a bottleneck
> > > for the API? What if we are talking about a large storage size in the
> > order
> > > of hundred (or tens) of GBs?
> > > 2. How does this fit in with RemoteStorage (KIP-405)? I think integration
> > > with KIP-405 is worth discussing in the scope of this KIP. My
> > > recommendation will be to add a new API in the RLMM
> > > (RemoteLogMetadataManager) called GetLogSize() and leave it upto the
> > remote
> > > storage to perform a concrete implementation for this
> > > interface. DescribeLogDirs could call this interface internally to
> > provide
> > > the relevant information.
> > > 3. Do you think adding the 

[jira] [Created] (KAFKA-13831) Kafka retention can use old value of retention.ms

[Jira]

Maciej Bryński created KAFKA-13831:
--

 Summary: Kafka retention can use old value of retention.ms
 Key: KAFKA-13831
 URL: https://issues.apache.org/jira/browse/KAFKA-13831
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.8.0
Reporter: Maciej Bryński


Hi,
I think I have found a bug in Kafka retention.
I'm using Confluent Platform 6.2.2 (Kafka 2.8.0).
I changed retention.ms for topic twice:
1. From 43200ms to 18ms (to clean the topic)
2. Back to 43200ms.

After second change retention thread is still using 18ms value.
Only broker restart fixes this issue.

Logs:
{code}
server.log.2022-04-15-03:[2022-04-15 03:29:08,445] INFO [Log 
partition=pm.hwe.lte.lcell.inc.intrarat.ho.x2.raw-0, dir=/data/kafka] Deleting 
segment LogSegment(baseOffset=1029819055, size=22996644, 
lastModifiedTime=1650007299179, largestRecordTimestamp=Some(1650007299178)) due 
to retention time 18ms breach based on the largest record timestamp in the 
segment (kafka.log.Log)
{code}

Topic description:
{code}
kafka-topics --bootstrap-server localhost:9092 --describe --topic 
pm.hwe.lte.lcell.inc.intrarat.ho.x2.raw
Topic: pm.hwe.lte.lcell.inc.intrarat.ho.x2.raw TopicId: svLdGbOaRXmdkHGsdlaPUQ 
PartitionCount: 1 ReplicationFactor: 3 Configs: 
min.insync.replicas=2,segment.bytes=1073741824,retention.ms=43200,segment.ms=8640
{code}




--
This message was sent by Atlassian Jira
(v8.20.1#820001)