[jira] [Created] (KAFKA-13807) Ensure that we can set log.flush.interval.ms with IncrementalAlterConfigs

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-13807:


 Summary: Ensure that we can set log.flush.interval.ms with 
IncrementalAlterConfigs
 Key: KAFKA-13807
 URL: https://issues.apache.org/jira/browse/KAFKA-13807
 Project: Kafka
  Issue Type: Bug
Reporter: Colin McCabe






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13806) Check CRC when reading snapshots

[Jose Armando Garcia Sancio (Jira)]

Jose Armando Garcia Sancio created KAFKA-13806:
--

 Summary: Check CRC when reading snapshots
 Key: KAFKA-13806
 URL: https://issues.apache.org/jira/browse/KAFKA-13806
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jose Armando Garcia Sancio
Assignee: Jose Armando Garcia Sancio






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Performance Engineering Track CFP for ApacheCon NA New Orleans

[sharanf]

Hi All

I hope that you have already heard that ApacheCon NA is back as a live 
event in New Orleans later this year. You can find out more details 
here: https://apachecon.com/acna2022/


For the first time ever - we will be running a Performance Engineering 
track. So what is Performance Engineering? You can find a definition and 
a track description here: https://s.apache.org/3ykqk


So why are you getting this message? Well - we took a look at all the 
ASF projects that may have an interest in Performance Engineering and 
this project was on the list :-)


If you are interested in making a submission to this new track then you 
can find a link to the CFP here: https://apachecon.com/acna2022/cfp.html


We are looking forward to receiving your submissions and hopefully 
seeing those of you who can make it to New Orleans in October.


Thanks
Sharan

[jira] [Created] (KAFKA-13805) Upgrade vulnerable dependencies march 2022

[Shivakumar (Jira)]

Shivakumar created KAFKA-13805:
--

 Summary: Upgrade vulnerable dependencies march 2022
 Key: KAFKA-13805
 URL: https://issues.apache.org/jira/browse/KAFKA-13805
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.8.1
Reporter: Shivakumar
 Fix For: 3.0.1, 3.2.0, 3.1.1


|Packages|Package Version|CVSS|Fix Status|
|com.fasterxml.jackson.core_jackson-databind| 2.10.5.1| 7.5| fixed in 2.14, 
2.13.1, 2.12.6|
| | | | |

Our security scan detected the above vulnerabilities

upgrade to correct versions for fixing vulnerabilities



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #846

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 605582 lines...]
[2022-04-07T14:09:13.636Z] 
[2022-04-07T14:09:13.636Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = false] PASSED
[2022-04-07T14:09:13.636Z] 
[2022-04-07T14:09:13.636Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = false] STARTED
[2022-04-07T14:09:14.152Z] 
[2022-04-07T14:09:14.152Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys PASSED
[2022-04-07T14:09:14.152Z] 
[2022-04-07T14:09:14.152Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys STARTED
[2022-04-07T14:09:18.888Z] 
[2022-04-07T14:09:18.888Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = false] PASSED
[2022-04-07T14:09:18.888Z] 
[2022-04-07T14:09:18.888Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testMultiInner[caching enabled = false] STARTED
[2022-04-07T14:09:25.854Z] 
[2022-04-07T14:09:25.854Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testMultiInner[caching enabled = false] PASSED
[2022-04-07T14:09:25.854Z] 
[2022-04-07T14:09:25.854Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeftRepartitioned[caching enabled = false] STARTED
[2022-04-07T14:09:33.464Z] 
[2022-04-07T14:09:33.464Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargeNumConsumers PASSED
[2022-04-07T14:09:33.464Z] 
[2022-04-07T14:09:33.464Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount STARTED
[2022-04-07T14:09:33.464Z] 
[2022-04-07T14:09:33.464Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount PASSED
[2022-04-07T14:09:33.464Z] 
[2022-04-07T14:09:33.464Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient STARTED
[2022-04-07T14:09:34.403Z] 
[2022-04-07T14:09:34.403Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient PASSED
[2022-04-07T14:09:34.403Z] 
[2022-04-07T14:09:34.403Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys STARTED
[2022-04-07T14:09:35.468Z] 
[2022-04-07T14:09:35.468Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys PASSED
[2022-04-07T14:09:35.468Z] 
[2022-04-07T14:09:35.468Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers STARTED
[2022-04-07T14:09:35.468Z] 
[2022-04-07T14:09:35.468Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers PASSED
[2022-04-07T14:09:35.468Z] 
[2022-04-07T14:09:35.468Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers STARTED
[2022-04-07T14:09:35.640Z] 
[2022-04-07T14:09:35.640Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeftRepartitioned[caching enabled = false] PASSED
[2022-04-07T14:09:35.640Z] 
[2022-04-07T14:09:35.640Z] 
org.apache.kafka.streams.integration.StreamTableJoinTopologyOptimizationIntegrationTest
 > shouldDoStreamTableJoinWithDifferentNumberOfPartitions[Optimization = all] 
STARTED
[2022-04-07T14:09:37.736Z] 
[2022-04-07T14:09:37.736Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers PASSED
[2022-04-07T14:09:37.736Z] streams-4: SMOKE-TEST-CLIENT-CLOSED
[2022-04-07T14:09:37.736Z] streams-5: SMOKE-TEST-CLIENT-CLOSED
[2022-04-07T14:09:37.736Z] streams-0: SMOKE-TEST-CLIENT-CLOSED
[2022-04-07T14:09:37.736Z] streams-3: SMOKE-TEST-CLIENT-CLOSED
[2022-04-07T14:09:37.736Z] streams-2: SMOKE-TEST-CLIENT-CLOSED
[2022-04-07T14:09:37.736Z] streams-1: SMOKE-TEST-CLIENT-CLOSED
[2022-04-07T14:09:39.189Z] 
[2022-04-07T14:09:39.189Z] 
org.apache.kafka.streams.integration.StreamTableJoinTopologyOptimizationIntegrationTest
 > shouldDoStreamTableJoinWithDifferentNumberOfPartitions[Optimization = all] 
PASSED
[2022-04-07T14:09:39.189Z] 
[2022-04-07T14:09:39.189Z] 
org.apache.kafka.streams.integration.StreamTableJoinTopologyOptimizationIntegrationTest
 > shouldDoStreamTableJoinWithDifferentNumberOfPartitions[Optimization = none] 
STARTED
[2022-04-07T14:09:42.372Z] 
[2022-04-07T14:09:42.372Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest 

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Divij Vaidya]

Hi Mickael

Thanks for starting this. It is a very useful feature.

Some initial thoughts (I am new to Kafka so please excuse if these are
naive suggestions):
1. What is the impact on latency of the DescribeLogDirs API due to this
change? Would calculating the totalSpace from each logdir be a bottleneck
for the API? What if we are talking about a large storage size in the order
of hundred (or tens) of GBs?
2. How does this fit in with RemoteStorage (KIP-405)? I think integration
with KIP-405 is worth discussing in the scope of this KIP. My
recommendation will be to add a new API in the RLMM
(RemoteLogMetadataManager) called GetLogSize() and leave it upto the remote
storage to perform a concrete implementation for this
interface. DescribeLogDirs could call this interface internally to provide
the relevant information.
3. Do you think adding the number of files in the directory as part of the
API response will be useful as well? e.g. a use case where this information
will be useful is to monitor/alarm the situations when the number of files
are dangerously reaching the max value of file descriptors configured at
the OS.
4. Please add an API latency perf test as part of the release criteria for
this change. We want to avoid regression.

Regards,
Divij Vaidya



On Thu, Apr 7, 2022 at 11:17 AM Mickael Maison 
wrote:

> Hi,
>
> I wrote a small KIP to expose the total and usable space of logdirs
> via the DescribeLogDirs API:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
>
> Please take a look and let me know if you have any feedback.
>
> Thanks,
> Mickael
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #845

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 444252 lines...]
[2022-04-07T11:33:14.178Z] 
[2022-04-07T11:33:14.178Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuter[caching enabled = false] STARTED
[2022-04-07T11:33:23.435Z] 
[2022-04-07T11:33:23.435Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread PASSED
[2022-04-07T11:33:23.435Z] 
[2022-04-07T11:33:23.435Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology STARTED
[2022-04-07T11:33:25.462Z] 
[2022-04-07T11:33:25.462Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuter[caching enabled = false] PASSED
[2022-04-07T11:33:25.462Z] 
[2022-04-07T11:33:25.462Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeft[caching enabled = false] STARTED
[2022-04-07T11:33:29.400Z] 
[2022-04-07T11:33:29.400Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology PASSED
[2022-04-07T11:33:29.400Z] 
[2022-04-07T11:33:29.400Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = true] STARTED
[2022-04-07T11:33:38.780Z] 
[2022-04-07T11:33:38.780Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeft[caching enabled = false] PASSED
[2022-04-07T11:33:38.780Z] 
[2022-04-07T11:33:38.780Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerInner[caching enabled = false] STARTED
[2022-04-07T11:33:43.137Z] 
[2022-04-07T11:33:43.137Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = true] PASSED
[2022-04-07T11:33:43.137Z] 
[2022-04-07T11:33:43.137Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = true] STARTED
[2022-04-07T11:33:51.633Z] 
[2022-04-07T11:33:51.633Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerInner[caching enabled = false] PASSED
[2022-04-07T11:33:51.633Z] 
[2022-04-07T11:33:51.633Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerOuter[caching enabled = false] STARTED
[2022-04-07T11:34:01.764Z] 
[2022-04-07T11:34:01.764Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = true] PASSED
[2022-04-07T11:34:01.764Z] 
[2022-04-07T11:34:01.764Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] STARTED
[2022-04-07T11:34:07.518Z] 
[2022-04-07T11:34:07.518Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerOuter[caching enabled = false] PASSED
[2022-04-07T11:34:07.518Z] 
[2022-04-07T11:34:07.518Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerLeft[caching enabled = false] STARTED
[2022-04-07T11:34:10.149Z] 
[2022-04-07T11:34:10.149Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] PASSED
[2022-04-07T11:34:10.149Z] 
[2022-04-07T11:34:10.149Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = true] STARTED
[2022-04-07T11:34:20.669Z] 
[2022-04-07T11:34:20.669Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerLeft[caching enabled = false] PASSED
[2022-04-07T11:34:20.669Z] 
[2022-04-07T11:34:20.669Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterInner[caching enabled = false] STARTED
[2022-04-07T11:34:21.606Z] 
[2022-04-07T11:34:21.606Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = true] PASSED
[2022-04-07T11:34:21.606Z] 
[2022-04-07T11:34:21.606Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = true] STARTED
[2022-04-07T11:34:34.018Z] 
[2022-04-07T11:34:34.018Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = true] PASSED
[2022-04-07T11:34:34.018Z] 
[2022-04-07T11:34:34.018Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testMultiInner[caching enabled = true] STARTED
[2022-04-07T11:34:35.946Z] 
[2022-04-07T11:34:35.946Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterInner[caching enabled = false] PASSED
[2022-04-07T11:34:35.946Z] 
[2022-04-07T11:34:35.946Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterOuter[caching enabled = false] STARTED
[2022-04-07T11:34:50.034Z] 
[2022-04-07T11:34:50.034Z] 

[jira] [Created] (KAFKA-13804) Log broker shutdown reason at the end of log output

[Luke Chen (Jira)]

Luke Chen created KAFKA-13804:
-

 Summary: Log broker shutdown reason at the end of log output
 Key: KAFKA-13804
 URL: https://issues.apache.org/jira/browse/KAFKA-13804
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Luke Chen
Assignee: Luke Chen


Currently, when the broker hit exception, or SIGINT signal received, we will 
first log the exception (or the reason we started to shutdown) ex:

_INFO Terminating process due to signal SIGINT_ 

 

And then go through the shutdown process, to close all the resources (i.e. 
threads, sockets, metrics...), and in the end, output:

_[KafkaServer id=0] shut down completed (kafka.server.KafkaServer)_

 

Sometimes, during closing the resources, we will also hit some exceptions, or 
waiting for some other dependencies close, and output many more logs. In the 
end, when we tried to find out why the broker is shutdown, we have to go 
through many lines of logs from the tail, and identify the specific reason why 
the broker shutting down.

We could improve it by adding the reason why the broker is shutting down at the 
end of log, something like this:

 
{code:java}
[2022-04-07 18:19:33,005] INFO Terminating process due to signal SIGINT 
(org.apache.kafka.common.utils.LoggingSignalHandler)
[2022-04-07 18:19:33,007] INFO [KafkaServer id=0] shutting down 
(kafka.server.KafkaServer)
[2022-04-07 18:19:33,008] INFO [KafkaServer id=0] Starting controlled shutdown 
(kafka.server.KafkaServer)
[2022-04-07 18:19:33,016] INFO [KafkaServer id=0] Controlled shutdown request 
returned successfully after 6ms (kafka.server.KafkaServer)

[2022-04-07 18:19:33,227] INFO Broker and topic stats closed 
(kafka.server.BrokerTopicStats)
[2022-04-07 18:19:33,227] INFO App info kafka.server for 0 unregistered 
(org.apache.kafka.common.utils.AppInfoParser)
[2022-04-07 18:19:33,227] INFO [KafkaServer id=0] shut down completed 
(kafka.server.KafkaServer)



// new added log
[2022-04-07 18:19:33,227] INFO [KafkaServer id=0] the broker is shut down due 
to: signal SIGINT{code}
 

 

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #844

[Apache Jenkins Server]

See 

[DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Mickael Maison]

Hi,

I wrote a small KIP to expose the total and usable space of logdirs
via the DescribeLogDirs API:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API

Please take a look and let me know if you have any feedback.

Thanks,
Mickael

[jira] [Resolved] (KAFKA-13801) Kafka server does not respect MetricsReporter interface contract for dynamically configured reporters

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-13801.
-
Fix Version/s: 3.3.0
 Reviewer: David Jacot
 Assignee: Xavier Léauté
   Resolution: Fixed

> Kafka server does not respect MetricsReporter interface contract for 
> dynamically configured reporters
> -
>
> Key: KAFKA-13801
> URL: https://issues.apache.org/jira/browse/KAFKA-13801
> Project: Kafka
>  Issue Type: Bug
>  Components: metrics
>Reporter: Xavier Léauté
>Assignee: Xavier Léauté
>Priority: Minor
> Fix For: 3.3.0
>
>
> MetricsReporter.contextChange contract states the method should always
> be called first before MetricsReporter.init is called. This is done
> correctly for reporters enabled by default (e.g. JmxReporter) but not
> for metrics reporters configured dynamically



--
This message was sent by Atlassian Jira
(v8.20.1#820001)