[
https://issues.apache.org/jira/browse/KAFKA-13672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509374#comment-17509374
]
Liam Clarke-Hutchinson edited comment on KAFKA-13672 at 3/20/22, 4:56 AM:
--------------------------------------------------------------------------
I've renamed this issue (was {_}Flaky test
{{kafka.server.DynamicBrokerReconfigurationTest.testThreadPoolResize()}}{_})
because the test was actually exposing a race condition in
{{DynamicBrokerConfig}} that occurred when the broker under test was running
sufficiently slowly. Many thanks are due to [~showuon] for all of his guidance
and insight, and for identifying the root cause.
{{DynamicBrokerConfig}} stores a mutable buffer of {{{}Reconfigurable{}}}s
which are protected by a read lock and a write lock. However, these locks can't
prevent the issue this test was exposing in the CI environment, which is - if
a {{Reconfigurable}} is added by one thread while another thread is updating
dynamic configs (which involves iterating over the buffer of
{{{}Reconfigurable{}}}s) then this causes a {{ConcurrentModificationException}}
for the thread that is iterating, thus preventing some dynamic configs from
being updated.
I'll be submitting a PR shortly to guard against this race condition.
was (Author: JIRAUSER279886):
I've renamed this issue (was Flaky test
{{{}kafka.server.DynamicBrokerReconfigurationTest.testThreadPoolResize(){}}})
because the test was actually exposing a race condition in
{{DynamicBrokerConfig}} that occurred when the broker under test was running
sufficiently slowly. Many thanks are due to [~showuon] for all of his guidance
and insight, and for identifying the root cause.
{{DynamicBrokerConfig}} stores a mutable buffer of {{{}Reconfigurable{}}}s
which are protected by a read lock and a write lock. However, these locks can't
prevent the issue this test was exposing in the CI environment, which is - if
a {{Reconfigurable}} is added by one thread while another thread is updating
dynamic configs (which involves iterating over the buffer of
{{{}Reconfigurable{}}}s) then this causes a {{ConcurrentModificationException}}
for the thread that is iterating, thus preventing some dynamic configs from
being updated.
I'll be submitting a PR shortly to guard against this race condition.
> Race condition in DynamicBrokerConfig
> -------------------------------------
>
> Key: KAFKA-13672
> URL: https://issues.apache.org/jira/browse/KAFKA-13672
> Project: Kafka
> Issue Type: Bug
> Reporter: Bruno Cadonna
> Assignee: Liam Clarke-Hutchinson
> Priority: Blocker
> Fix For: 3.2.0
>
>
> Stacktrace:
> {code:java}
> org.opentest4j.AssertionFailedError: expected: <2> but was: <1>
> at
> app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
> at
> app//org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
> at
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
> at
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
> at
> app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.$anonfun$waitForConfigOnServer$1(DynamicBrokerReconfigurationTest.scala:1500)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.waitForConfigOnServer(DynamicBrokerReconfigurationTest.scala:1500)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.$anonfun$waitForConfig$1(DynamicBrokerReconfigurationTest.scala:1495)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.$anonfun$waitForConfig$1$adapted(DynamicBrokerReconfigurationTest.scala:1495)
> at app//scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
> at
> app//scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
> at app//scala.collection.AbstractIterable.foreach(Iterable.scala:926)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.waitForConfig(DynamicBrokerReconfigurationTest.scala:1495)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.reconfigureServers(DynamicBrokerReconfigurationTest.scala:1440)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.resizeThreadPool$1(DynamicBrokerReconfigurationTest.scala:775)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.$anonfun$testThreadPoolResize$3(DynamicBrokerReconfigurationTest.scala:768)
> at app//scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:190)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.testThreadPoolResize(DynamicBrokerReconfigurationTest.scala:784)
> {code}
> Job:
> https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-11751/5/testReport/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
