[GitHub] [kafka] cshannon closed pull request #10845: KAFKA-12916: Add new AUTO_CREATE ACL for auto topic creation
cshannon closed pull request #10845: KAFKA-12916: Add new AUTO_CREATE ACL for auto topic creation URL: https://github.com/apache/kafka/pull/10845 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13535: KAFKA-9579 Fetch implementation for records in the remote storage through a specific purgatory.
satishd commented on code in PR #13535:
URL: https://github.com/apache/kafka/pull/13535#discussion_r1184873142
##
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##
@@ -670,6 +875,14 @@ public void close() {
} catch (InterruptedException e) {
// ignore
}
+remoteStorageReaderThreadPool.shutdownNow();
+//waits for 2 mins to terminate the current tasks
+try {
+remoteStorageReaderThreadPool.awaitTermination(2,
TimeUnit.MINUTES);
Review Comment:
It does not require that to be completed in 5 mins. That code
`lifecycleManager.controlledShutdownFuture` is more about processing the
controlled shutdown event to the controller for that broker. It will wait for 5
mins before proceeding with other sequence of actions. But that will not get
affected because of the code introduced here.
Logging subsystem handles unclean shutdown for log segments and it would
have been already finished before RemoteLogManager is closed. So, they will not
get affected because of this timeout. But we can have a short duration here
like 10 secs, we can revisit introducing a config if it is really needed for
closing the remote log subsystem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] omkreddy commented on pull request #13737: MINOR: Add System Properties to config documentation section
omkreddy commented on PR #13737: URL: https://github.com/apache/kafka/pull/13737#issuecomment-1555946866 Merged to 3.5 and 3.4 branches -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] omkreddy merged pull request #13737: MINOR: Add System Properties to config documentation section
omkreddy merged PR #13737: URL: https://github.com/apache/kafka/pull/13737 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] omkreddy opened a new pull request, #13737: MINOR: Add System Properties to config documentation section
omkreddy opened a new pull request, #13737: URL: https://github.com/apache/kafka/pull/13737 Kafka repo update for https://github.com/apache/kafka-site/pull/512/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (KAFKA-15005) Status of KafkaConnect task not correct
[
https://issues.apache.org/jira/browse/KAFKA-15005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724518#comment-17724518
]
Yu Wang edited comment on KAFKA-15005 at 5/20/23 10:11 AM:
---
[~sagarrao] Sorry for the confusion, I will change the wroker_id.
Yes, the RUNNIG status comes from worker-1 and UNASSIGNED status comes from
worker-2.
was (Author: lucentwong):
[~sagarrao] Sorry for the confusion.
Yes, the RUNNIG status comes from worker-1 and UNASSIGNED status comes from
worker-2.
> Status of KafkaConnect task not correct
> ---
>
> Key: KAFKA-15005
> URL: https://issues.apache.org/jira/browse/KAFKA-15005
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.5.1, 3.0.0, 3.3.2
>Reporter: Yu Wang
>Priority: Major
>
> Our MM2 is running version 2.5.1.
> After a rebalance of our MM2 source tasks, we found there were several tasks
> always in *UNASSIGNED* status, even the real tasks already started.
> So we dump the payload of the status topic of Kafka Connect, and found the
> last two status change is status *RUNNING* followed by status
> {*}UNASSIGNED{*}.
> {code:java}
> LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1
> headerKeys: [] key: task-7 payload:
> {"state":"RUNNING","trace":null,"worker_id":"worker-1","generation":437643}
> LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1
> headerKeys: [] key: task-7 payload:
> {"state":"UNASSIGNED","trace":null,"worker_id":"worker-2","generation":437643}
> {code}
> But usually, the RUNNING status should be appended after the UNASSIGNED,
> because the worker coordinator will revoked the tasks before start new tasks.
> Then we checked the log of our MM2 worker. And found that, during that time,
> there was a task that revoked on worker-2 and started on worker-1.
>
> Worker-1
> {code:java}
> [2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1,
> groupId=__group] Starting task task-7
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
> [2023-05-15 09:24:45,951] INFO Creating task task-7
> (org.apache.kafka.connect.runtime.Worker) {code}
> Worker-2
> {code:java}
> [2023-05-15 09:24:40,922] INFO Stopping task task-7
> (org.apache.kafka.connect.runtime.Worker) {code}
>
> So I think the incorrect status was caused by the revoked task finished later
> than the new started task, which made the UNASSIGNED status append to that
> status topic after the RUNNING status.
>
> After reading the code of DistributeHerder, I found that the task revoking is
> running in a thread pool, the revoke operation just return after submit all
> the callables. So I think even in the same worker, there is not a guarantee
> that the revoke operation will always finish before the new tasks start.
> {code:java}
> for (final ConnectorTaskId taskId : tasks) {
> callables.add(getTaskStoppingCallable(taskId));
> }
> // The actual timeout for graceful task/connector stop is applied in worker's
> // stopAndAwaitTask/stopAndAwaitConnector methods.
> startAndStop(callables);
> log.info("Finished stopping tasks in preparation for rebalance"); {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15005) Status of KafkaConnect task not correct
[
https://issues.apache.org/jira/browse/KAFKA-15005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Wang updated KAFKA-15005:
Description:
Our MM2 is running version 2.5.1.
After a rebalance of our MM2 source tasks, we found there were several tasks
always in *UNASSIGNED* status, even the real tasks already started.
So we dump the payload of the status topic of Kafka Connect, and found the last
two status change is status *RUNNING* followed by status {*}UNASSIGNED{*}.
{code:java}
LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"RUNNING","trace":null,"worker_id":"worker-1","generation":437643}
LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"UNASSIGNED","trace":null,"worker_id":"worker-2","generation":437643}
{code}
But usually, the RUNNING status should be appended after the UNASSIGNED,
because the worker coordinator will revoked the tasks before start new tasks.
Then we checked the log of our MM2 worker. And found that, during that time,
there was a task that revoked on worker-2 and started on worker-1.
Worker-1
{code:java}
[2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1, groupId=__group]
Starting task task-7
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[2023-05-15 09:24:45,951] INFO Creating task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
Worker-2
{code:java}
[2023-05-15 09:24:40,922] INFO Stopping task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
So I think the incorrect status was caused by the revoked task finished later
than the new started task, which made the UNASSIGNED status append to that
status topic after the RUNNING status.
After reading the code of DistributeHerder, I found that the task revoking is
running in a thread pool, the revoke operation just return after submit all the
callables. So I think even in the same worker, there is not a guarantee that
the revoke operation will always finish before the new tasks start.
{code:java}
for (final ConnectorTaskId taskId : tasks) {
callables.add(getTaskStoppingCallable(taskId));
}
// The actual timeout for graceful task/connector stop is applied in worker's
// stopAndAwaitTask/stopAndAwaitConnector methods.
startAndStop(callables);
log.info("Finished stopping tasks in preparation for rebalance"); {code}
was:
Our MM2 is running version 2.5.1.
After a rebalance of our MM2 source tasks, we found there were several tasks
always in *UNASSIGNED* status, even the real tasks already started.
So we dump the payload of the status topic of Kafka Connect, and found the last
two status change is status *RUNNING* followed by status {*}UNASSIGNED{*}.
{code:java}
LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"RUNNING","trace":null,"worker_id":"x","generation":437643}
LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1 headerKeys:
[] key: task-7 payload:
{"state":"UNASSIGNED","trace":null,"worker_id":"x","generation":437643}
{code}
But usually, the RUNNING status should be appended after the UNASSIGNED,
because the worker coordinator will revoked the tasks before start new tasks.
Then we checked the log of our MM2 worker. And found that, during that time,
there was a task that revoked on worker-2 and started on worker-1.
Worker-1
{code:java}
[2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1, groupId=__group]
Starting task task-7
(org.apache.kafka.connect.runtime.distributed.DistributedHerder)
[2023-05-15 09:24:45,951] INFO Creating task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
Worker-2
{code:java}
[2023-05-15 09:24:40,922] INFO Stopping task task-7
(org.apache.kafka.connect.runtime.Worker) {code}
So I think the incorrect status was caused by the revoked task finished later
than the new started task, which made the UNASSIGNED status append to that
status topic after the RUNNING status.
After reading the code of DistributeHerder, I found that the task revoking is
running in a thread pool, the revoke operation just return after submit all the
callables. So I think even in the same worker, there is not a guarantee that
the revoke operation will always finish before the new tasks start.
{code:java}
for (final ConnectorTaskId taskId : tasks) {
callables.add(getTaskStoppingCallable(taskId));
}
// The actual timeout for graceful task/connector stop is applied in worker's
// stopAndAwaitTask/stopAndAwaitConnector methods.
startAndStop(callables);
log.info("Finished stopping tasks in preparation for rebalance"); {code}
> Status of KafkaConnect task not correct
> ---
>
> Key: KAFKA-15005
> URL: https://issues.apache.org/jira/browse/KAFKA-15005
>
[jira] [Commented] (KAFKA-15005) Status of KafkaConnect task not correct
[
https://issues.apache.org/jira/browse/KAFKA-15005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724518#comment-17724518
]
Yu Wang commented on KAFKA-15005:
-
[~sagarrao] Sorry for the confusion.
Yes, the RUNNIG status comes from worker-1 and UNASSIGNED status comes from
worker-2.
> Status of KafkaConnect task not correct
> ---
>
> Key: KAFKA-15005
> URL: https://issues.apache.org/jira/browse/KAFKA-15005
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.5.1, 3.0.0, 3.3.2
>Reporter: Yu Wang
>Priority: Major
>
> Our MM2 is running version 2.5.1.
> After a rebalance of our MM2 source tasks, we found there were several tasks
> always in *UNASSIGNED* status, even the real tasks already started.
> So we dump the payload of the status topic of Kafka Connect, and found the
> last two status change is status *RUNNING* followed by status
> {*}UNASSIGNED{*}.
> {code:java}
> LogAppendTime: 1684167885961 keysize: 64 valuesize: 95 sequence: -1
> headerKeys: [] key: task-7 payload:
> {"state":"RUNNING","trace":null,"worker_id":"x","generation":437643}
> LogAppendTime: 1684167903456 keysize: 64 valuesize: 98 sequence: -1
> headerKeys: [] key: task-7 payload:
> {"state":"UNASSIGNED","trace":null,"worker_id":"x","generation":437643}
> {code}
> But usually, the RUNNING status should be appended after the UNASSIGNED,
> because the worker coordinator will revoked the tasks before start new tasks.
> Then we checked the log of our MM2 worker. And found that, during that time,
> there was a task that revoked on worker-2 and started on worker-1.
>
> Worker-1
> {code:java}
> [2023-05-15 09:24:45,950] INFO [Worker clientId=connect-1,
> groupId=__group] Starting task task-7
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
> [2023-05-15 09:24:45,951] INFO Creating task task-7
> (org.apache.kafka.connect.runtime.Worker) {code}
> Worker-2
> {code:java}
> [2023-05-15 09:24:40,922] INFO Stopping task task-7
> (org.apache.kafka.connect.runtime.Worker) {code}
>
> So I think the incorrect status was caused by the revoked task finished later
> than the new started task, which made the UNASSIGNED status append to that
> status topic after the RUNNING status.
>
> After reading the code of DistributeHerder, I found that the task revoking is
> running in a thread pool, the revoke operation just return after submit all
> the callables. So I think even in the same worker, there is not a guarantee
> that the revoke operation will always finish before the new tasks start.
> {code:java}
> for (final ConnectorTaskId taskId : tasks) {
> callables.add(getTaskStoppingCallable(taskId));
> }
> // The actual timeout for graceful task/connector stop is applied in worker's
> // stopAndAwaitTask/stopAndAwaitConnector methods.
> startAndStop(callables);
> log.info("Finished stopping tasks in preparation for rebalance"); {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14559) Handle object name with wildcards in the Jmx tool
[
https://issues.apache.org/jira/browse/KAFKA-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamal Chandraprakash updated KAFKA-14559:
-
Fix Version/s: 3.6.0
> Handle object name with wildcards in the Jmx tool
> -
>
> Key: KAFKA-14559
> URL: https://issues.apache.org/jira/browse/KAFKA-14559
> Project: Kafka
> Issue Type: Task
>Reporter: Kamal Chandraprakash
>Assignee: Kamal Chandraprakash
>Priority: Minor
> Fix For: 3.6.0
>
>
> {code}
> ❯ sh kafka-run-class.sh kafka.tools.JmxTool --jmx-url
> service:jmx:rmi:///jndi/rmi://127.0.0.1:/jmxrmi --object-name
> kafka.server:type=BrokerTopicMetrics,*
> Trying to connect to JMX url:
> service:jmx:rmi:///jndi/rmi://127.0.0.1:/jmxrmi.
> Exception in thread "main" java.lang.NullPointerException
> at kafka.tools.JmxTool$.main(JmxTool.scala:194)
> at kafka.tools.JmxTool.main(JmxTool.scala)
> ❯ sh kafka-run-class.sh kafka.tools.JmxTool --jmx-url
> service:jmx:rmi:///jndi/rmi://localhost:/jmxrmi --object-name
> kafka.server:type=BrokerTopicMetrics,* --attributes Count,FifteenMinuteRate
> Trying to connect to JMX url:
> service:jmx:rmi:///jndi/rmi://127.0.0.1:/jmxrmi.
> Exception in thread "main" java.lang.NullPointerException
> at kafka.tools.JmxTool$.queryAttributes(JmxTool.scala:254)
> at kafka.tools.JmxTool$.main(JmxTool.scala:214)
> at kafka.tools.JmxTool.main(JmxTool.scala)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
