[jira] [Commented] (KAFKA-4629) Per topic MBeans leak
[ https://issues.apache.org/jira/browse/KAFKA-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825545#comment-15825545 ] huxi commented on KAFKA-4629: - Thanks for [~alberto.fo...@natwestmarkets.com]'s response. After some debugging, I found that BrokerTopicMetrics.close could always be invoked even the corresponding MBeans did not get removed. Seems that it's not caused by Kafka code. Might need to investigate whether it's a known issue for metrics-core library. > Per topic MBeans leak > - > > Key: KAFKA-4629 > URL: https://issues.apache.org/jira/browse/KAFKA-4629 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.0.1 >Reporter: Alberto Forti >Priority: Minor > > Hi, > In our application we create and delete topics dynamically. Most of the times > when a topic is deleted the related MBeans are not deleted. Example of MBean: > kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=dw_06b5f828-e452-4e22-89c9-67849a65603d > Also, deleting a topic often produces (what I think is) noise in the logs at > WARN level. One example is: > WARN PartitionStateMachine$DeleteTopicsListener:83 - [DeleteTopicsListener > on 1]: Ignoring request to delete non-existing topics > dw_fe8ff14b-aa9b-4f24-9bc1-6fbce15d20d2 > Easy reproducible with a basic Kafka cluster with two brokers, just create > and delete topics few times. Sometimes the MBeans for the topic are deleted > and sometimes are not. > I'm creating and deleting topics using the AdminUtils class in the Java API: > AdminUtils.deleteTopic(zkUtils, topicName); > AdminUtils.createTopic(zkUtils, topicName, partitions, replicationFactor, > topicConfiguration, kafka.admin.RackAwareMode.Enforced$.MODULE$); > Kafka version: 0.10.0.1 (haven't tried other versions) > Thanks, > Alberto -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-4629) Per topic MBeans leak
[ https://issues.apache.org/jira/browse/KAFKA-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824221#comment-15824221 ] Alberto Forti commented on KAFKA-4629: -- Just an update regarding [~huxi_2b] request. Waiting doesn't make any difference, I've just tried to create 100 topics and deleted them after one hour or so... some mbeans are still there. > Per topic MBeans leak > - > > Key: KAFKA-4629 > URL: https://issues.apache.org/jira/browse/KAFKA-4629 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.0.1 >Reporter: Alberto Forti >Priority: Minor > > Hi, > In our application we create and delete topics dynamically. Most of the times > when a topic is deleted the related MBeans are not deleted. Example of MBean: > kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=dw_06b5f828-e452-4e22-89c9-67849a65603d > Also, deleting a topic often produces (what I think is) noise in the logs at > WARN level. One example is: > WARN PartitionStateMachine$DeleteTopicsListener:83 - [DeleteTopicsListener > on 1]: Ignoring request to delete non-existing topics > dw_fe8ff14b-aa9b-4f24-9bc1-6fbce15d20d2 > Easy reproducible with a basic Kafka cluster with two brokers, just create > and delete topics few times. Sometimes the MBeans for the topic are deleted > and sometimes are not. > I'm creating and deleting topics using the AdminUtils class in the Java API: > AdminUtils.deleteTopic(zkUtils, topicName); > AdminUtils.createTopic(zkUtils, topicName, partitions, replicationFactor, > topicConfiguration, kafka.admin.RackAwareMode.Enforced$.MODULE$); > Kafka version: 0.10.0.1 (haven't tried other versions) > Thanks, > Alberto -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-4629) Per topic MBeans leak
[
https://issues.apache.org/jira/browse/KAFKA-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823725#comment-15823725
]
Ismael Juma commented on KAFKA-4629:
There are 2 points here:
1. What's the best way to do this kind of thing in tests? Kafka's own tests do
something like the following in TestUtils:
{code}
def createTopic(zkUtils: ZkUtils,
topic: String,
numPartitions: Int = 1,
replicationFactor: Int = 1,
servers: Seq[KafkaServer],
topicConfig: Properties = new Properties) :
scala.collection.immutable.Map[Int, Option[Int]] = {
// create topic
AdminUtils.createTopic(zkUtils, topic, numPartitions, replicationFactor,
topicConfig)
// wait until the update metadata request for new topic reaches all servers
(0 until numPartitions).map { case i =>
TestUtils.waitUntilMetadataIsPropagated(servers, topic, i)
i -> TestUtils.waitUntilLeaderIsElectedOrChanged(zkUtils, topic, i)
}.toMap
}
{code}
2. The topic leak can happen outside of tests too. It would be good to fix
that, if possible. I haven't looked at the code, but maybe we should check if
the topic hasn't been queued for deletion when registering the metrics.
> Per topic MBeans leak
> -
>
> Key: KAFKA-4629
> URL: https://issues.apache.org/jira/browse/KAFKA-4629
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Alberto Forti
>Priority: Minor
>
> Hi,
> In our application we create and delete topics dynamically. Most of the times
> when a topic is deleted the related MBeans are not deleted. Example of MBean:
> kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=dw_06b5f828-e452-4e22-89c9-67849a65603d
> Also, deleting a topic often produces (what I think is) noise in the logs at
> WARN level. One example is:
> WARN PartitionStateMachine$DeleteTopicsListener:83 - [DeleteTopicsListener
> on 1]: Ignoring request to delete non-existing topics
> dw_fe8ff14b-aa9b-4f24-9bc1-6fbce15d20d2
> Easy reproducible with a basic Kafka cluster with two brokers, just create
> and delete topics few times. Sometimes the MBeans for the topic are deleted
> and sometimes are not.
> I'm creating and deleting topics using the AdminUtils class in the Java API:
> AdminUtils.deleteTopic(zkUtils, topicName);
> AdminUtils.createTopic(zkUtils, topicName, partitions, replicationFactor,
> topicConfiguration, kafka.admin.RackAwareMode.Enforced$.MODULE$);
> Kafka version: 0.10.0.1 (haven't tried other versions)
> Thanks,
> Alberto
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4629) Per topic MBeans leak
[ https://issues.apache.org/jira/browse/KAFKA-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823679#comment-15823679 ] Alberto Forti commented on KAFKA-4629: -- Hi Huxi, I've noticed this behaviour in our dev environment where we have integration tests continuously running. The test that relies on a new Kafka topic does the following: creates a topic, sends 6 messages, checks the messages are consumed and then deletes the topic. As you can see the topic is not deleted immediately after creation. Anyway I'll add a pause after topic creation to see if it helps. Thanks, Alberto > Per topic MBeans leak > - > > Key: KAFKA-4629 > URL: https://issues.apache.org/jira/browse/KAFKA-4629 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.0.1 >Reporter: Alberto Forti >Priority: Minor > > Hi, > In our application we create and delete topics dynamically. Most of the times > when a topic is deleted the related MBeans are not deleted. Example of MBean: > kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=dw_06b5f828-e452-4e22-89c9-67849a65603d > Also, deleting a topic often produces (what I think is) noise in the logs at > WARN level. One example is: > WARN PartitionStateMachine$DeleteTopicsListener:83 - [DeleteTopicsListener > on 1]: Ignoring request to delete non-existing topics > dw_fe8ff14b-aa9b-4f24-9bc1-6fbce15d20d2 > Easy reproducible with a basic Kafka cluster with two brokers, just create > and delete topics few times. Sometimes the MBeans for the topic are deleted > and sometimes are not. > I'm creating and deleting topics using the AdminUtils class in the Java API: > AdminUtils.deleteTopic(zkUtils, topicName); > AdminUtils.createTopic(zkUtils, topicName, partitions, replicationFactor, > topicConfiguration, kafka.admin.RackAwareMode.Enforced$.MODULE$); > Kafka version: 0.10.0.1 (haven't tried other versions) > Thanks, > Alberto -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-4629) Per topic MBeans leak
[ https://issues.apache.org/jira/browse/KAFKA-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823329#comment-15823329 ] huxi commented on KAFKA-4629: - [~alberto.fo...@natwestmarkets.com] Yes, multiple brokers DOES help reproducing the issue. And some interesting things were found. If deleting topic command was issued right after creating the topic, some MBeans at the topic level indeed failed to get removed. But if some delay time was put on between these two commands, then all the topic-level MBeans could be removed as expected. In the former case, the controller is still doing many background tasks to complete the topic creation although the CREATE command returns successfully. So could you wait some time before issuing the delete command to see whether you would run into this issue? > Per topic MBeans leak > - > > Key: KAFKA-4629 > URL: https://issues.apache.org/jira/browse/KAFKA-4629 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.0.1 >Reporter: Alberto Forti >Priority: Minor > > Hi, > In our application we create and delete topics dynamically. Most of the times > when a topic is deleted the related MBeans are not deleted. Example of MBean: > kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=dw_06b5f828-e452-4e22-89c9-67849a65603d > Also, deleting a topic often produces (what I think is) noise in the logs at > WARN level. One example is: > WARN PartitionStateMachine$DeleteTopicsListener:83 - [DeleteTopicsListener > on 1]: Ignoring request to delete non-existing topics > dw_fe8ff14b-aa9b-4f24-9bc1-6fbce15d20d2 > Easy reproducible with a basic Kafka cluster with two brokers, just create > and delete topics few times. Sometimes the MBeans for the topic are deleted > and sometimes are not. > I'm creating and deleting topics using the AdminUtils class in the Java API: > AdminUtils.deleteTopic(zkUtils, topicName); > AdminUtils.createTopic(zkUtils, topicName, partitions, replicationFactor, > topicConfiguration, kafka.admin.RackAwareMode.Enforced$.MODULE$); > Kafka version: 0.10.0.1 (haven't tried other versions) > Thanks, > Alberto -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-4629) Per topic MBeans leak
[ https://issues.apache.org/jira/browse/KAFKA-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823180#comment-15823180 ] Alberto Forti commented on KAFKA-4629: -- Hi, You need at least two brokers to reproduce the issue. As for the WARN logs, yes I confirm the topic exists before invoking AdminUtils.deleteTopic. Thanks, Alberto > Per topic MBeans leak > - > > Key: KAFKA-4629 > URL: https://issues.apache.org/jira/browse/KAFKA-4629 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.0.1 >Reporter: Alberto Forti >Priority: Minor > > Hi, > In our application we create and delete topics dynamically. Most of the times > when a topic is deleted the related MBeans are not deleted. Example of MBean: > kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=dw_06b5f828-e452-4e22-89c9-67849a65603d > Also, deleting a topic often produces (what I think is) noise in the logs at > WARN level. One example is: > WARN PartitionStateMachine$DeleteTopicsListener:83 - [DeleteTopicsListener > on 1]: Ignoring request to delete non-existing topics > dw_fe8ff14b-aa9b-4f24-9bc1-6fbce15d20d2 > Easy reproducible with a basic Kafka cluster with two brokers, just create > and delete topics few times. Sometimes the MBeans for the topic are deleted > and sometimes are not. > I'm creating and deleting topics using the AdminUtils class in the Java API: > AdminUtils.deleteTopic(zkUtils, topicName); > AdminUtils.createTopic(zkUtils, topicName, partitions, replicationFactor, > topicConfiguration, kafka.admin.RackAwareMode.Enforced$.MODULE$); > Kafka version: 0.10.0.1 (haven't tried other versions) > Thanks, > Alberto -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-4629) Per topic MBeans leak
[ https://issues.apache.org/jira/browse/KAFKA-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823029#comment-15823029 ] huxi commented on KAFKA-4629: - I tried in my kafka0.10.0.1 single-broker test environment with no luck to reproduce the issue. The partition number and replication factor were both set to 1 which I am sure if the setting matters. As for the WARN logs, seems you are requesting a deletion for a non-exist topic. Could you confirm that the topic exists before invoking AdminUtils.deleteTopic? > Per topic MBeans leak > - > > Key: KAFKA-4629 > URL: https://issues.apache.org/jira/browse/KAFKA-4629 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.10.0.1 >Reporter: Alberto Forti >Priority: Minor > > Hi, > In our application we create and delete topics dynamically. Most of the times > when a topic is deleted the related MBeans are not deleted. Example of MBean: > kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=dw_06b5f828-e452-4e22-89c9-67849a65603d > Also, deleting a topic often produces (what I think is) noise in the logs at > WARN level. One example is: > WARN PartitionStateMachine$DeleteTopicsListener:83 - [DeleteTopicsListener > on 1]: Ignoring request to delete non-existing topics > dw_fe8ff14b-aa9b-4f24-9bc1-6fbce15d20d2 > Easy reproducible with a basic Kafka cluster with two brokers, just create > and delete topics few times. Sometimes the MBeans for the topic are deleted > and sometimes are not. > I'm creating and deleting topics using the AdminUtils class in the Java API: > AdminUtils.deleteTopic(zkUtils, topicName); > AdminUtils.createTopic(zkUtils, topicName, partitions, replicationFactor, > topicConfiguration, kafka.admin.RackAwareMode.Enforced$.MODULE$); > Kafka version: 0.10.0.1 (haven't tried other versions) > Thanks, > Alberto -- This message was sent by Atlassian JIRA (v6.3.4#6332)
