[jira] [Created] (KAFKA-6370) MirrorMakerIntegrationTest#testCommaSeparatedRegex may fail due to NullPointerException
Ted Yu created KAFKA-6370: - Summary: MirrorMakerIntegrationTest#testCommaSeparatedRegex may fail due to NullPointerException Key: KAFKA-6370 URL: https://issues.apache.org/jira/browse/KAFKA-6370 Project: Kafka Issue Type: Bug Reporter: Ted Yu Priority: Minor >From >https://builds.apache.org/job/kafka-trunk-jdk8/2277/testReport/junit/kafka.tools/MirrorMakerIntegrationTest/testCommaSeparatedRegex/ > : {code} java.lang.NullPointerException at scala.collection.immutable.StringLike.$anonfun$format$1(StringLike.scala:351) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38) at scala.collection.TraversableLike.map(TraversableLike.scala:234) at scala.collection.TraversableLike.map$(TraversableLike.scala:227) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at scala.collection.immutable.StringLike.format(StringLike.scala:351) at scala.collection.immutable.StringLike.format$(StringLike.scala:350) at scala.collection.immutable.StringOps.format(StringOps.scala:29) at kafka.metrics.KafkaMetricsGroup$.$anonfun$toScope$3(KafkaMetricsGroup.scala:170) at scala.collection.immutable.List.map(List.scala:283) at kafka.metrics.KafkaMetricsGroup$.kafka$metrics$KafkaMetricsGroup$$toScope(KafkaMetricsGroup.scala:170) at kafka.metrics.KafkaMetricsGroup.explicitMetricName(KafkaMetricsGroup.scala:67) at kafka.metrics.KafkaMetricsGroup.explicitMetricName$(KafkaMetricsGroup.scala:51) at kafka.network.RequestMetrics.explicitMetricName(RequestChannel.scala:352) at kafka.metrics.KafkaMetricsGroup.metricName(KafkaMetricsGroup.scala:47) at kafka.metrics.KafkaMetricsGroup.metricName$(KafkaMetricsGroup.scala:42) at kafka.network.RequestMetrics.metricName(RequestChannel.scala:352) at kafka.metrics.KafkaMetricsGroup.newHistogram(KafkaMetricsGroup.scala:81) at kafka.metrics.KafkaMetricsGroup.newHistogram$(KafkaMetricsGroup.scala:80) at kafka.network.RequestMetrics.newHistogram(RequestChannel.scala:352) at kafka.network.RequestMetrics.(RequestChannel.scala:364) at kafka.network.RequestChannel$Metrics.$anonfun$new$2(RequestChannel.scala:57) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:52) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at kafka.network.RequestChannel$Metrics.(RequestChannel.scala:56) at kafka.network.RequestChannel.(RequestChannel.scala:243) at kafka.network.SocketServer.(SocketServer.scala:71) at kafka.server.KafkaServer.startup(KafkaServer.scala:238) at kafka.utils.TestUtils$.createServer(TestUtils.scala:135) at kafka.integration.KafkaServerTestHarness.$anonfun$setUp$1(KafkaServerTestHarness.scala:93) {code} Here is the code from KafkaMetricsGroup.scala : {code} .map { case (key, value) => "%s.%s".format(key, value.replaceAll("\\.", "_"))} {code} It seems (some) value was null. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6384) TransactionsTest#testFencingOnSendOffsets sometimes fails with ProducerFencedException
Ted Yu created KAFKA-6384: - Summary: TransactionsTest#testFencingOnSendOffsets sometimes fails with ProducerFencedException Key: KAFKA-6384 URL: https://issues.apache.org/jira/browse/KAFKA-6384 Project: Kafka Issue Type: Bug Reporter: Ted Yu >From >https://builds.apache.org/job/kafka-trunk-jdk8/2283/testReport/junit/kafka.api/TransactionsTest/testFencingOnSendOffsets/ > : {code} org.scalatest.junit.JUnitTestFailedError: Got an unexpected exception from a fenced producer. at org.scalatest.junit.AssertionsForJUnit$class.newAssertionFailedException(AssertionsForJUnit.scala:100) at org.scalatest.junit.JUnitSuite.newAssertionFailedException(JUnitSuite.scala:71) at org.scalatest.Assertions$class.fail(Assertions.scala:1105) at org.scalatest.junit.JUnitSuite.fail(JUnitSuite.scala:71) at kafka.api.TransactionsTest.testFencingOnSendOffsets(TransactionsTest.scala:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) at com.sun.proxy.$Proxy1.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:108) at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:146) at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:128) at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404) at org.gradle.internal.concurrent.Exec
[jira] [Created] (KAFKA-6412) Improve synchronization in CachingKeyValueStore methods
Ted Yu created KAFKA-6412: - Summary: Improve synchronization in CachingKeyValueStore methods Key: KAFKA-6412 URL: https://issues.apache.org/jira/browse/KAFKA-6412 Project: Kafka Issue Type: Improvement Reporter: Ted Yu Currently CachingKeyValueStore methods are synchronized at method level. It seems we can use read lock for getter and write lock for put / delete methods. For getInternal(), if the underlying thread is streamThread, the getInternal() may trigger eviction. This can be handled by obtaining write lock at the beginning of the method for streamThread. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6413) ReassignPartitionsCommand#parsePartitionReassignmentData() should give better error message when JSON is malformed
Ted Yu created KAFKA-6413: - Summary: ReassignPartitionsCommand#parsePartitionReassignmentData() should give better error message when JSON is malformed Key: KAFKA-6413 URL: https://issues.apache.org/jira/browse/KAFKA-6413 Project: Kafka Issue Type: Improvement Reporter: Ted Yu Priority: Minor In this thread: http://search-hadoop.com/m/Kafka/uyzND1J9Hizcxo0X?subj=Partition+reassignment+data+file+is+empty , Allen gave an example JSON string with extra comma where partitionsToBeReassigned returned by ReassignPartitionsCommand#parsePartitionReassignmentData() was empty. I tried the following example where a right bracket is removed: {code} val (partitionsToBeReassigned, replicaAssignment) = ReassignPartitionsCommand.parsePartitionReassignmentData( "{\"version\":1,\"partitions\":[{\"topic\":\"metrics\",\"partition\":0,\"replicas\":[1,2]},{\"topic\":\"metrics\",\"partition\":1,\"replicas\":[2,3]},}"); {code} The returned partitionsToBeReassigned is empty. The parser should give better error message for malformed JSON string. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6424) QueryableStateIntegrationTest#queryOnRebalance should be accept raw text
Ted Yu created KAFKA-6424: - Summary: QueryableStateIntegrationTest#queryOnRebalance should be accept raw text Key: KAFKA-6424 URL: https://issues.apache.org/jira/browse/KAFKA-6424 Project: Kafka Issue Type: Improvement Reporter: Ted Yu Priority: Minor I was using QueryableStateIntegrationTest#queryOnRebalance for some performance test by adding more sentences to inputValues. I found that when the sentence contains upper case letter, the test would timeout. I get around this limitation by calling {{sentence.toLowerCase(Locale.ROOT)}} before the split. Ideally we can specify the path to text file which contains the text. The test can read the text file and generate the input array. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-6384) TransactionsTest#testFencingOnSendOffsets sometimes fails with ProducerFencedException
[ https://issues.apache.org/jira/browse/KAFKA-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-6384. --- Resolution: Cannot Reproduce > TransactionsTest#testFencingOnSendOffsets sometimes fails with > ProducerFencedException > -- > > Key: KAFKA-6384 > URL: https://issues.apache.org/jira/browse/KAFKA-6384 > Project: Kafka > Issue Type: Bug >Reporter: Ted Yu >Priority: Major > > From > https://builds.apache.org/job/kafka-trunk-jdk8/2283/testReport/junit/kafka.api/TransactionsTest/testFencingOnSendOffsets/ > : > {code} > org.scalatest.junit.JUnitTestFailedError: Got an unexpected exception from a > fenced producer. > at > org.scalatest.junit.AssertionsForJUnit$class.newAssertionFailedException(AssertionsForJUnit.scala:100) > at > org.scalatest.junit.JUnitSuite.newAssertionFailedException(JUnitSuite.scala:71) > at org.scalatest.Assertions$class.fail(Assertions.scala:1105) > at org.scalatest.junit.JUnitSuite.fail(JUnitSuite.scala:71) > at > kafka.api.TransactionsTest.testFencingOnSendOffsets(TransactionsTest.scala:357) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) > at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) > at > org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) > at com.sun.proxy.$Proxy1.processTestClass(Unknown Source) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:108) > at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(Messa
[jira] [Resolved] (KAFKA-5889) MetricsTest is flaky
[ https://issues.apache.org/jira/browse/KAFKA-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-5889. --- Resolution: Cannot Reproduce > MetricsTest is flaky > > > Key: KAFKA-5889 > URL: https://issues.apache.org/jira/browse/KAFKA-5889 > Project: Kafka > Issue Type: Test >Reporter: Ted Yu >Priority: Major > > The following appeared in several recent builds (e.g. > https://builds.apache.org/job/kafka-trunk-jdk7/2758) : > {code} > kafka.metrics.MetricsTest > testMetricsLeak FAILED > java.lang.AssertionError: expected:<1216> but was:<1219> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at org.junit.Assert.assertEquals(Assert.java:631) > at > kafka.metrics.MetricsTest$$anonfun$testMetricsLeak$1.apply$mcVI$sp(MetricsTest.scala:68) > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) > at kafka.metrics.MetricsTest.testMetricsLeak(MetricsTest.scala:66) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-6232) SaslSslAdminClientIntegrationTest sometimes fails
[ https://issues.apache.org/jira/browse/KAFKA-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-6232. --- Resolution: Cannot Reproduce > SaslSslAdminClientIntegrationTest sometimes fails > - > > Key: KAFKA-6232 > URL: https://issues.apache.org/jira/browse/KAFKA-6232 > Project: Kafka > Issue Type: Test >Reporter: Ted Yu >Priority: Major > Labels: security > Attachments: saslSslAdminClientIntegrationTest-203.out > > > Here was one recent occurrence: > https://builds.apache.org/job/kafka-trunk-jdk9/203/testReport/kafka.api/SaslSslAdminClientIntegrationTest/testLogStartOffsetCheckpoint/ > {code} > java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.LeaderNotAvailableException: There is no > leader for this topic-partition as we are in the middle of a leadership > election. > at > org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) > at > org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) > at > org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104) > at > org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:225) > at > kafka.api.AdminClientIntegrationTest.$anonfun$testLogStartOffsetCheckpoint$3(AdminClientIntegrationTest.scala:762) > {code} > In the test output, I saw: > {code} > [2017-11-17 23:15:45,593] ERROR [KafkaApi-1] Error when handling request > {filters=[{resource_type=2,resource_name=foobar,principal=User:ANONYMOUS,host=*,operation=3,permission_type=3},{resource_type=5,resource_name=transactional_id,principal=User:ANONYMOUS,host=*,operation=4,permission_type=3}]} > (kafka.server.KafkaApis:107) > org.apache.kafka.common.errors.ClusterAuthorizationException: Request > Request(processor=2, connectionId=127.0.0.1:36295-127.0.0.1:58183-0, > session=Session(User:client2,localhost/127.0.0.1), > listenerName=ListenerName(SASL_SSL), securityProtocol=SASL_SSL, buffer=null) > is not authorized. > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6531) SocketServerTest#closingChannelException fails sometimes
Ted Yu created KAFKA-6531: - Summary: SocketServerTest#closingChannelException fails sometimes Key: KAFKA-6531 URL: https://issues.apache.org/jira/browse/KAFKA-6531 Project: Kafka Issue Type: Test Reporter: Ted Yu >From >https://builds.apache.org/job/kafka-trunk-jdk9/361/testReport/junit/kafka.network/SocketServerTest/closingChannelException/ > : {code} java.lang.AssertionError: Channels not removed at kafka.utils.TestUtils$.fail(TestUtils.scala:355) at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:865) at kafka.network.SocketServerTest.assertProcessorHealthy(SocketServerTest.scala:914) at kafka.network.SocketServerTest.$anonfun$closingChannelException$1(SocketServerTest.scala:763) at kafka.network.SocketServerTest.$anonfun$closingChannelException$1$adapted(SocketServerTest.scala:747) {code} Among the test output, I saw: {code} [2018-02-04 18:51:15,995] ERROR Processor 0 closed connection from /127.0.0.1:48261 (kafka.network.SocketServerTest$$anon$5$$anon$1:73) java.lang.IllegalStateException: There is already a connection for id 127.0.0.1:1-127.0.0.1:2-0 at org.apache.kafka.common.network.Selector.ensureNotRegistered(Selector.java:260) at org.apache.kafka.common.network.Selector.register(Selector.java:254) at kafka.network.SocketServerTest$TestableSelector.super$register(SocketServerTest.scala:1043) at kafka.network.SocketServerTest$TestableSelector.$anonfun$register$2(SocketServerTest.scala:1043) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) at kafka.network.SocketServerTest$TestableSelector.runOp(SocketServerTest.scala:1037) at kafka.network.SocketServerTest$TestableSelector.register(SocketServerTest.scala:1043) at kafka.network.Processor.configureNewConnections(SocketServer.scala:723) at kafka.network.Processor.run(SocketServer.scala:532) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (KAFKA-5889) MetricsTest is flaky
[ https://issues.apache.org/jira/browse/KAFKA-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened KAFKA-5889: --- As of 332e698ac9c74ce29317021b03a54512c92ac8b3 , I got: {code} kafka.metrics.MetricsTest > testMetricsLeak FAILED java.lang.AssertionError: expected:<1421> but was:<1424> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:834) at org.junit.Assert.assertEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:631) at kafka.metrics.MetricsTest$$anonfun$testMetricsLeak$1.apply$mcVI$sp(MetricsTest.scala:68) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) at kafka.metrics.MetricsTest.testMetricsLeak(MetricsTest.scala:66) {code} > MetricsTest is flaky > > > Key: KAFKA-5889 > URL: https://issues.apache.org/jira/browse/KAFKA-5889 > Project: Kafka > Issue Type: Test >Reporter: Ted Yu >Priority: Major > > The following appeared in several recent builds (e.g. > https://builds.apache.org/job/kafka-trunk-jdk7/2758) : > {code} > kafka.metrics.MetricsTest > testMetricsLeak FAILED > java.lang.AssertionError: expected:<1216> but was:<1219> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at org.junit.Assert.assertEquals(Assert.java:631) > at > kafka.metrics.MetricsTest$$anonfun$testMetricsLeak$1.apply$mcVI$sp(MetricsTest.scala:68) > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) > at kafka.metrics.MetricsTest.testMetricsLeak(MetricsTest.scala:66) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-6228) Intermittent test failure in FetchRequestTest.testDownConversionWithConnectionFailure
[ https://issues.apache.org/jira/browse/KAFKA-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-6228. --- Resolution: Cannot Reproduce > Intermittent test failure in > FetchRequestTest.testDownConversionWithConnectionFailure > - > > Key: KAFKA-6228 > URL: https://issues.apache.org/jira/browse/KAFKA-6228 > Project: Kafka > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From > https://builds.apache.org/job/kafka-trunk-jdk8/2219/testReport/junit/kafka.server/FetchRequestTest/testDownConversionWithConnectionFailure/ > : > {code} > java.lang.AssertionError: Fetch size too small 42, broker may have run out of > memory > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > kafka.server.FetchRequestTest.kafka$server$FetchRequestTest$$fetch$1(FetchRequestTest.scala:214) > at > kafka.server.FetchRequestTest$$anonfun$testDownConversionWithConnectionFailure$2.apply(FetchRequestTest.scala:226) > at > kafka.server.FetchRequestTest$$anonfun$testDownConversionWithConnectionFailure$2.apply(FetchRequestTest.scala:226) > at scala.collection.immutable.Range.foreach(Range.scala:160) > at > kafka.server.FetchRequestTest.testDownConversionWithConnectionFailure(FetchRequestTest.scala:226) > {code} > I ran FetchRequestTest locally which passed. > {code} > assertTrue(s"Fetch size too small $size, broker may have run out of > memory", > size > maxPartitionBytes - batchSize) > {code} > The assertion message should include maxPartitionBytes and batchSize which > would give us more information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6678) Upgrade dependencies with later release versions
Ted Yu created KAFKA-6678: - Summary: Upgrade dependencies with later release versions Key: KAFKA-6678 URL: https://issues.apache.org/jira/browse/KAFKA-6678 Project: Kafka Issue Type: Improvement Reporter: Ted Yu {code} The following dependencies have later release versions: - net.sourceforge.argparse4j:argparse4j [0.7.0 -> 0.8.1] - org.bouncycastle:bcpkix-jdk15on [1.58 -> 1.59] - com.puppycrawl.tools:checkstyle [6.19 -> 8.8] - org.owasp:dependency-check-gradle [3.0.2 -> 3.1.1] - org.ajoberstar:grgit [1.9.3 -> 2.1.1] - org.glassfish.jersey.containers:jersey-container-servlet [2.25.1 -> 2.26] - org.eclipse.jetty:jetty-client [9.2.24.v20180105 -> 9.4.8.v20171121] - org.eclipse.jetty:jetty-server [9.2.24.v20180105 -> 9.4.8.v20171121] - org.eclipse.jetty:jetty-servlet [9.2.24.v20180105 -> 9.4.8.v20171121] - org.eclipse.jetty:jetty-servlets [9.2.24.v20180105 -> 9.4.8.v20171121] - org.openjdk.jmh:jmh-core [1.19 -> 1.20] - org.openjdk.jmh:jmh-core-benchmarks [1.19 -> 1.20] - org.openjdk.jmh:jmh-generator-annprocess [1.19 -> 1.20] - org.lz4:lz4-java [1.4 -> 1.4.1] - org.apache.maven:maven-artifact [3.5.2 -> 3.5.3] - org.jacoco:org.jacoco.agent [0.7.9 -> 0.8.0] - org.jacoco:org.jacoco.ant [0.7.9 -> 0.8.0] - org.rocksdb:rocksdbjni [5.7.3 -> 5.11.3] - org.scala-lang:scala-library [2.11.12 -> 2.12.4] - com.typesafe.scala-logging:scala-logging_2.11 [3.7.2 -> 3.8.0] - org.scala-lang:scala-reflect [2.11.12 -> 2.12.4] - org.scalatest:scalatest_2.11 [3.0.4 -> 3.0.5] {code} Looks like we can consider upgrading scalatest, jmh-core and checkstyle -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6698) ConsumerBounceTest#testClose sometimes fails
Ted Yu created KAFKA-6698: - Summary: ConsumerBounceTest#testClose sometimes fails Key: KAFKA-6698 URL: https://issues.apache.org/jira/browse/KAFKA-6698 Project: Kafka Issue Type: Test Reporter: Ted Yu Saw the following in https://builds.apache.org/job/kafka-1.1-jdk7/94/testReport/junit/kafka.api/ConsumerBounceTest/testClose/ : {code} org.apache.kafka.common.errors.TimeoutException: The consumer group command timed out while waiting for group to initialize: Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException: The coordinator is not available. {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6716) discardChannel should be released in MockSelector#completeSend
Ted Yu created KAFKA-6716: - Summary: discardChannel should be released in MockSelector#completeSend Key: KAFKA-6716 URL: https://issues.apache.org/jira/browse/KAFKA-6716 Project: Kafka Issue Type: Test Reporter: Ted Yu {code} private void completeSend(Send send) throws IOException { // Consume the send so that we will be able to send more requests to the destination ByteBufferChannel discardChannel = new ByteBufferChannel(send.size()); while (!send.completed()) { send.writeTo(discardChannel); } completedSends.add(send); } {code} The {{discardChannel}} should be closed before returning from the method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6734) TopicMetadataTest is flaky
Ted Yu created KAFKA-6734: - Summary: TopicMetadataTest is flaky Key: KAFKA-6734 URL: https://issues.apache.org/jira/browse/KAFKA-6734 Project: Kafka Issue Type: Test Reporter: Ted Yu I got two different test failures in two runs of test suite: {code} kafka.integration.TopicMetadataTest > testAutoCreateTopic FAILED kafka.common.KafkaException: fetching topic metadata for topics [Set(testAutoCreateTopic)] from broker [List(BrokerEndPoint(0,,41557))] failed at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:77) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98) at kafka.integration.TopicMetadataTest.testAutoCreateTopic(TopicMetadataTest.scala:105) Caused by: java.net.SocketTimeoutException at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:122) at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:131) at kafka.network.BlockingChannel.receive(BlockingChannel.scala:122) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79) at kafka.producer.SyncProducer.send(SyncProducer.scala:124) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:63) ... 2 more {code} {code} kafka.integration.TopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack FAILED java.lang.AssertionError: Topic metadata is not correctly updated for broker kafka.server.KafkaServer@4c45dc9f. Expected ISR: List(BrokerEndPoint(0,localhost,40822), BrokerEndPoint(1,localhost,39030)) Actual ISR : Vector(BrokerEndPoint(0,localhost,40822)) at kafka.utils.TestUtils$.fail(TestUtils.scala:355) at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:865) at kafka.integration.TopicMetadataTest$$anonfun$checkIsr$1.apply(TopicMetadataTest.scala:191) at kafka.integration.TopicMetadataTest$$anonfun$checkIsr$1.apply(TopicMetadataTest.scala:189) at scala.collection.immutable.List.foreach(List.scala:392) at kafka.integration.TopicMetadataTest.checkIsr(TopicMetadataTest.scala:189) at kafka.integration.TopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack(TopicMetadataTest.scala:231) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6735) Document how to skip findbugs / checkstyle when running unit test
Ted Yu created KAFKA-6735: - Summary: Document how to skip findbugs / checkstyle when running unit test Key: KAFKA-6735 URL: https://issues.apache.org/jira/browse/KAFKA-6735 Project: Kafka Issue Type: Test Reporter: Ted Yu Even when running single unit test, findbugs dependency would result in some time spent before the test is actually run. We should document how findbugs dependency can be skipped in such scenario: {code} -x findbugsMain -x findbugsTest {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6747) kafka-streams Invalid transition attempted from state READY to state ABORTING_TRANSACTION
Ted Yu created KAFKA-6747: - Summary: kafka-streams Invalid transition attempted from state READY to state ABORTING_TRANSACTION Key: KAFKA-6747 URL: https://issues.apache.org/jira/browse/KAFKA-6747 Project: Kafka Issue Type: Bug Reporter: Ted Yu I running tests against kafka-streams 1.1 and get the following stack trace (everything was working alright using kafka-streams 1.0): {code} ERROR org.apache.kafka.streams.processor.internals.AssignedStreamsTasks - stream-thread [feedBuilder-XXX-StreamThread-4] Failed to close stream task, 0_2 org.apache.kafka.common.KafkaException: TransactionalId feedBuilder-0_2: Invalid transition attempted from state READY to state ABORTING_TRANSACTION at org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:757) at org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:751) at org.apache.kafka.clients.producer.internals.TransactionManager.beginAbort(TransactionManager.java:230) at org.apache.kafka.clients.producer.KafkaProducer.abortTransaction(KafkaProducer.java:660) at org.apache.kafka.streams.processor.internals.StreamTask.closeSuspended(StreamTask.java:486) at org.apache.kafka.streams.processor.internals.StreamTask.close(StreamTask.java:546) at org.apache.kafka.streams.processor.internals.AssignedTasks.closeNonRunningTasks(AssignedTasks.java:166) at org.apache.kafka.streams.processor.internals.AssignedTasks.suspend(AssignedTasks.java:151) at org.apache.kafka.streams.processor.internals.TaskManager.suspendTasksAndState(TaskManager.java:242) at org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsRevoked(StreamThread.java:291) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:414) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:359) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:290) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1149) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115) at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:827) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:784) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:750) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:720) {code} This happens when starting the same stream-processing application on 3 JVMs all running on the same linux box, JVMs are named JVM-[2-4]. All 3 instances use separate stream state.dir. No record is ever processed because the input kafka topics are empty at this stage. JVM-2 starts first, joined shortly after by JVM-4 and JVM-3, find the state transition logs below. The above stacktrace is from JVM-4 {code} [JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to REBALANCING [JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to RUNNING [JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to REBALANCING [JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to REBALANCING [JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to RUNNING [JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to REBALANCING [JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to REBALANCING JVM-4 crashes here with above stacktrace [JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to RUNNING [JVM-4] stream-client [feedBuilder-XXX] State transition from REBALANCING to ERROR [JVM-4] stream-client [feedBuilder-XXX] State transition from ERROR to PENDING_SHUTDOWN [JVM-4] stream-client [feedBuilder-XXX] State transition from PENDING_SHUTDOWN to NOT_RUNNING [JVM-4] stream-client [feedBuilder-XXX] State transition from RUNNING to REBALANCING [JVM-3] stream-client [feedBuilder-XXX] State transition from REBALANCING to RUNNING [JVM-2] stream-client [feedBuilder-XXX] State transition from RUNNING to REBALANCING [JVM-3] stream-client [feedBuilder-XXX] State transition from RUNNING to REBALANCING [JVM-2] stream-client [feedBuilder-XXX] State transition from REBALANCING to RUNNING [JVM-3] stream-client [feedBuilder-XXX] State transition from REBALANCING to RUNNING [JVM-4] stream-client [feedBuilder-XXX] State transition from REBALANCING to
[jira] [Created] (KAFKA-6775) AbstractProcessor created in SimpleBenchmark should call super#init
Ted Yu created KAFKA-6775: - Summary: AbstractProcessor created in SimpleBenchmark should call super#init Key: KAFKA-6775 URL: https://issues.apache.org/jira/browse/KAFKA-6775 Project: Kafka Issue Type: Bug Reporter: Ted Yu Around line 610: {code} return new AbstractProcessor() { @Override public void init(ProcessorContext context) { } {code} super.init should be called above. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-6531) SocketServerTest#closingChannelException fails sometimes
[ https://issues.apache.org/jira/browse/KAFKA-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-6531. --- Resolution: Cannot Reproduce > SocketServerTest#closingChannelException fails sometimes > > > Key: KAFKA-6531 > URL: https://issues.apache.org/jira/browse/KAFKA-6531 > Project: Kafka > Issue Type: Test > Components: core >Reporter: Ted Yu >Priority: Minor > > From > https://builds.apache.org/job/kafka-trunk-jdk9/361/testReport/junit/kafka.network/SocketServerTest/closingChannelException/ > : > {code} > java.lang.AssertionError: Channels not removed > at kafka.utils.TestUtils$.fail(TestUtils.scala:355) > at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:865) > at > kafka.network.SocketServerTest.assertProcessorHealthy(SocketServerTest.scala:914) > at > kafka.network.SocketServerTest.$anonfun$closingChannelException$1(SocketServerTest.scala:763) > at > kafka.network.SocketServerTest.$anonfun$closingChannelException$1$adapted(SocketServerTest.scala:747) > {code} > Among the test output, I saw: > {code} > [2018-02-04 18:51:15,995] ERROR Processor 0 closed connection from > /127.0.0.1:48261 (kafka.network.SocketServerTest$$anon$5$$anon$1:73) > java.lang.IllegalStateException: There is already a connection for id > 127.0.0.1:1-127.0.0.1:2-0 > at > org.apache.kafka.common.network.Selector.ensureNotRegistered(Selector.java:260) > at org.apache.kafka.common.network.Selector.register(Selector.java:254) > at > kafka.network.SocketServerTest$TestableSelector.super$register(SocketServerTest.scala:1043) > at > kafka.network.SocketServerTest$TestableSelector.$anonfun$register$2(SocketServerTest.scala:1043) > at > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) > at > kafka.network.SocketServerTest$TestableSelector.runOp(SocketServerTest.scala:1037) > at > kafka.network.SocketServerTest$TestableSelector.register(SocketServerTest.scala:1043) > at > kafka.network.Processor.configureNewConnections(SocketServer.scala:723) > at kafka.network.Processor.run(SocketServer.scala:532) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-6734) TopicMetadataTest is flaky
[ https://issues.apache.org/jira/browse/KAFKA-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-6734. --- Resolution: Cannot Reproduce > TopicMetadataTest is flaky > -- > > Key: KAFKA-6734 > URL: https://issues.apache.org/jira/browse/KAFKA-6734 > Project: Kafka > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > I got two different test failures in two runs of test suite: > {code} > kafka.integration.TopicMetadataTest > testAutoCreateTopic FAILED > kafka.common.KafkaException: fetching topic metadata for topics > [Set(testAutoCreateTopic)] from broker [List(BrokerEndPoint(0,,41557))] failed > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:77) > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98) > at > kafka.integration.TopicMetadataTest.testAutoCreateTopic(TopicMetadataTest.scala:105) > Caused by: > java.net.SocketTimeoutException > at > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > at > java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) > at > org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:122) > at > kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:131) > at > kafka.network.BlockingChannel.receive(BlockingChannel.scala:122) > at > kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82) > at > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79) > at kafka.producer.SyncProducer.send(SyncProducer.scala:124) > at > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:63) > ... 2 more > {code} > {code} > kafka.integration.TopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack > FAILED > java.lang.AssertionError: Topic metadata is not correctly updated for > broker kafka.server.KafkaServer@4c45dc9f. > Expected ISR: List(BrokerEndPoint(0,localhost,40822), > BrokerEndPoint(1,localhost,39030)) > Actual ISR : Vector(BrokerEndPoint(0,localhost,40822)) > at kafka.utils.TestUtils$.fail(TestUtils.scala:355) > at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:865) > at > kafka.integration.TopicMetadataTest$$anonfun$checkIsr$1.apply(TopicMetadataTest.scala:191) > at > kafka.integration.TopicMetadataTest$$anonfun$checkIsr$1.apply(TopicMetadataTest.scala:189) > at scala.collection.immutable.List.foreach(List.scala:392) > at > kafka.integration.TopicMetadataTest.checkIsr(TopicMetadataTest.scala:189) > at > kafka.integration.TopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack(TopicMetadataTest.scala:231) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-6736) ReassignPartitionsClusterTest#shouldMoveSubsetOfPartitions is flaky
[ https://issues.apache.org/jira/browse/KAFKA-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-6736. --- Resolution: Cannot Reproduce > ReassignPartitionsClusterTest#shouldMoveSubsetOfPartitions is flaky > --- > > Key: KAFKA-6736 > URL: https://issues.apache.org/jira/browse/KAFKA-6736 > Project: Kafka > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > Saw this from > https://builds.apache.org/job/kafka-trunk-jdk8/2518/testReport/junit/kafka.admin/ReassignPartitionsClusterTest/shouldMoveSubsetOfPartitions/ > : > {code} > kafka.common.AdminCommandFailedException: Partition reassignment currently in > progress for Map(topic1-0 -> Buffer(100, 102), topic1-2 -> Buffer(100, 102), > topic2-1 -> Buffer(101, 100), topic2-2 -> Buffer(100, 102)). Aborting > operation > at > kafka.admin.ReassignPartitionsCommand.reassignPartitions(ReassignPartitionsCommand.scala:612) > at > kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:215) > at > kafka.admin.ReassignPartitionsClusterTest.shouldMoveSubsetOfPartitions(ReassignPartitionsClusterTest.scala:242) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6875) EosIntegrationTest#shouldNotViolateEosIfOneTaskFails is flaky
Ted Yu created KAFKA-6875: - Summary: EosIntegrationTest#shouldNotViolateEosIfOneTaskFails is flaky Key: KAFKA-6875 URL: https://issues.apache.org/jira/browse/KAFKA-6875 Project: Kafka Issue Type: Test Reporter: Ted Yu >From >https://builds.apache.org/job/kafka-trunk-jdk10/81/testReport/junit/org.apache.kafka.streams.integration/EosIntegrationTest/shouldNotViolateEosIfOneTaskFails/ > : {code} java.lang.AssertionError: Condition not met within timeout 6. SteamsTasks did not request commit. at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:276) at org.apache.kafka.streams.integration.EosIntegrationTest.shouldNotViolateEosIfOneTaskFails(EosIntegrationTest.java:339) {code} >From test output: {code} [2018-05-07 19:04:18,236] ERROR [Controller id=2 epoch=3] Controller 2 epoch 3 failed to change state for partition __transaction_state-34 from OnlinePartition to OnlinePartition (state.change.logger:76) kafka.common.StateChangeFailedException: Failed to elect leader for partition __transaction_state-34 under strategy ControlledShutdownPartitionLeaderElectionStrategy at kafka.controller.PartitionStateMachine.$anonfun$doElectLeaderForPartitions$9(PartitionStateMachine.scala:328) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:52) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at kafka.controller.PartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:326) at kafka.controller.PartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:254) at kafka.controller.PartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:175) at kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:116) at kafka.controller.KafkaController$ControlledShutdown.doControlledShutdown(KafkaController.scala:1055) at kafka.controller.KafkaController$ControlledShutdown.$anonfun$process$1(KafkaController.scala:1031) at scala.util.Try$.apply(Try.scala:209) at kafka.controller.KafkaController$ControlledShutdown.process(KafkaController.scala:1031) at kafka.controller.ControllerEventManager$ControllerEventThread.$anonfun$doWork$1(ControllerEventManager.scala:69) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6904) DynamicBrokerReconfigurationTest#testAdvertisedListenerUpdate is flaky
Ted Yu created KAFKA-6904: - Summary: DynamicBrokerReconfigurationTest#testAdvertisedListenerUpdate is flaky Key: KAFKA-6904 URL: https://issues.apache.org/jira/browse/KAFKA-6904 Project: Kafka Issue Type: Test Reporter: Ted Yu >From >https://builds.apache.org/job/kafka-pr-jdk10-scala2.12/820/testReport/junit/kafka.server/DynamicBrokerReconfigurationTest/testAdvertisedListenerUpdate/ > : {code} kafka.server.DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate Failing for the past 1 build (Since Failed#820 ) Took 21 sec. Error Message java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. Stacktrace java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:77) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29) at kafka.server.DynamicBrokerReconfigurationTest.$anonfun$verifyProduceConsume$3(DynamicBrokerReconfigurationTest.scala:996) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234) at scala.collection.Iterator.foreach(Iterator.scala:944) at scala.collection.Iterator.foreach$(Iterator.scala:944) at scala.collection.AbstractIterator.foreach(Iterator.scala:1432) at scala.collection.IterableLike.foreach(IterableLike.scala:71) at scala.collection.IterableLike.foreach$(IterableLike.scala:70) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike.map(TraversableLike.scala:234) at scala.collection.TraversableLike.map$(TraversableLike.scala:227) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at kafka.server.DynamicBrokerReconfigurationTest.verifyProduceConsume(DynamicBrokerReconfigurationTest.scala:996) at kafka.server.DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate(DynamicBrokerReconfigurationTest.scala:742) {code} The above happened with jdk 10. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-6875) EosIntegrationTest#shouldNotViolateEosIfOneTaskFails is flaky
[ https://issues.apache.org/jira/browse/KAFKA-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-6875. --- Resolution: Cannot Reproduce > EosIntegrationTest#shouldNotViolateEosIfOneTaskFails is flaky > - > > Key: KAFKA-6875 > URL: https://issues.apache.org/jira/browse/KAFKA-6875 > Project: Kafka > Issue Type: Test > Components: streams, unit tests >Reporter: Ted Yu >Priority: Minor > Labels: newbie++ > Attachments: EosIntegrationTest.out > > > From > https://builds.apache.org/job/kafka-trunk-jdk10/81/testReport/junit/org.apache.kafka.streams.integration/EosIntegrationTest/shouldNotViolateEosIfOneTaskFails/ > : > {code} > java.lang.AssertionError: Condition not met within timeout 6. SteamsTasks > did not request commit. > at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:276) > at > org.apache.kafka.streams.integration.EosIntegrationTest.shouldNotViolateEosIfOneTaskFails(EosIntegrationTest.java:339) > {code} > From test output: > {code} > [2018-05-07 19:04:18,236] ERROR [Controller id=2 epoch=3] Controller 2 epoch > 3 failed to change state for partition __transaction_state-34 from > OnlinePartition to OnlinePartition (state.change.logger:76) > kafka.common.StateChangeFailedException: Failed to elect leader for partition > __transaction_state-34 under strategy > ControlledShutdownPartitionLeaderElectionStrategy > at > kafka.controller.PartitionStateMachine.$anonfun$doElectLeaderForPartitions$9(PartitionStateMachine.scala:328) > at > scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59) > at > scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:52) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at > kafka.controller.PartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:326) > at > kafka.controller.PartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:254) > at > kafka.controller.PartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:175) > at > kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:116) > at > kafka.controller.KafkaController$ControlledShutdown.doControlledShutdown(KafkaController.scala:1055) > at > kafka.controller.KafkaController$ControlledShutdown.$anonfun$process$1(KafkaController.scala:1031) > at scala.util.Try$.apply(Try.scala:209) > at > kafka.controller.KafkaController$ControlledShutdown.process(KafkaController.scala:1031) > at > kafka.controller.ControllerEventManager$ControllerEventThread.$anonfun$doWork$1(ControllerEventManager.scala:69) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-6698) ConsumerBounceTest#testClose sometimes fails
[ https://issues.apache.org/jira/browse/KAFKA-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-6698. --- Resolution: Cannot Reproduce > ConsumerBounceTest#testClose sometimes fails > > > Key: KAFKA-6698 > URL: https://issues.apache.org/jira/browse/KAFKA-6698 > Project: Kafka > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > Saw the following in > https://builds.apache.org/job/kafka-1.1-jdk7/94/testReport/junit/kafka.api/ConsumerBounceTest/testClose/ > : > {code} > org.apache.kafka.common.errors.TimeoutException: The consumer group command > timed out while waiting for group to initialize: > Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException: > The coordinator is not available. > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7049) InternalTopicIntegrationTest sometimes fails
Ted Yu created KAFKA-7049: - Summary: InternalTopicIntegrationTest sometimes fails Key: KAFKA-7049 URL: https://issues.apache.org/jira/browse/KAFKA-7049 Project: Kafka Issue Type: Test Reporter: Ted Yu Saw the following based on commit fa1d0383902260576132e09bdf9efcc2784b55b4 : {code} org.apache.kafka.streams.integration.InternalTopicIntegrationTest > shouldCompactTopicsForKeyValueStoreChangelogs FAILED java.lang.RuntimeException: Timed out waiting for completion. lagMetrics=[0/2] totalLag=[0.0] at org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForCompletion(IntegrationTestUtils.java:227) at org.apache.kafka.streams.integration.InternalTopicIntegrationTest.shouldCompactTopicsForKeyValueStoreChangelogs(InternalTopicIntegrationTest.java:164) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7124) Number of AnyLogDir should match the length of the replicas list
Ted Yu created KAFKA-7124: - Summary: Number of AnyLogDir should match the length of the replicas list Key: KAFKA-7124 URL: https://issues.apache.org/jira/browse/KAFKA-7124 Project: Kafka Issue Type: Bug Reporter: Ted Yu See discussion under 'Partitions reassignment is failing in Kafka 1.1.0' thread. Here is snippet from generated json file: {code} {"topic": "Topic3", "partition": 7, "log_dirs": ["any"], "replicas": [3, 0, 2]} {code} Code snippet from ReassignPartitionsCommand.scala : {code} "log_dirs" -> replicas.map(r => replicaLogDirAssignment.getOrElse(new TopicPartitionReplica(tp.topic, tp.partition, r), AnyLogDir)).asJava {code} We know that the appearance of "any" was due to the OrElse clause. There is a bug in the above code that the number of AnyLogDir should match the length of the replicas list, or "log_dirs" should be omitted in such case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7124) Number of AnyLogDir should match the length of the replicas list
[ https://issues.apache.org/jira/browse/KAFKA-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-7124. --- Resolution: Not A Problem > Number of AnyLogDir should match the length of the replicas list > > > Key: KAFKA-7124 > URL: https://issues.apache.org/jira/browse/KAFKA-7124 > Project: Kafka > Issue Type: Bug >Reporter: Ted Yu >Priority: Major > > See discussion under 'Partitions reassignment is failing in Kafka 1.1.0' > thread reported by Debraj Manna. > Here is snippet from generated json file: > {code} > {"topic": "Topic3", "partition": 7, "log_dirs": ["any"], "replicas": [3, 0, > 2]} > {code} > Code snippet from ReassignPartitionsCommand.scala : > {code} > "log_dirs" -> replicas.map(r => > replicaLogDirAssignment.getOrElse(new TopicPartitionReplica(tp.topic, > tp.partition, r), AnyLogDir)).asJava > {code} > We know that the appearance of "any" was due to the OrElse clause. > There is a bug in the above code that the number of AnyLogDir should match > the length of the replicas list, or "log_dirs" should be omitted in such case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-6335) SimpleAclAuthorizerTest#testHighConcurrencyModificationOfResourceAcls fails intermittently
[ https://issues.apache.org/jira/browse/KAFKA-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-6335. --- Resolution: Cannot Reproduce > SimpleAclAuthorizerTest#testHighConcurrencyModificationOfResourceAcls fails > intermittently > -- > > Key: KAFKA-6335 > URL: https://issues.apache.org/jira/browse/KAFKA-6335 > Project: Kafka > Issue Type: Test >Reporter: Ted Yu >Assignee: Manikumar >Priority: Major > > From > https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/3045/testReport/junit/kafka.security.auth/SimpleAclAuthorizerTest/testHighConcurrencyModificationOfResourceAcls/ > : > {code} > java.lang.AssertionError: expected acls Set(User:36 has Allow permission for > operations: Read from hosts: *, User:7 has Allow permission for operations: > Read from hosts: *, User:21 has Allow permission for operations: Read from > hosts: *, User:39 has Allow permission for operations: Read from hosts: *, > User:43 has Allow permission for operations: Read from hosts: *, User:3 has > Allow permission for operations: Read from hosts: *, User:35 has Allow > permission for operations: Read from hosts: *, User:15 has Allow permission > for operations: Read from hosts: *, User:16 has Allow permission for > operations: Read from hosts: *, User:22 has Allow permission for operations: > Read from hosts: *, User:26 has Allow permission for operations: Read from > hosts: *, User:11 has Allow permission for operations: Read from hosts: *, > User:38 has Allow permission for operations: Read from hosts: *, User:8 has > Allow permission for operations: Read from hosts: *, User:28 has Allow > permission for operations: Read from hosts: *, User:32 has Allow permission > for operations: Read from hosts: *, User:25 has Allow permission for > operations: Read from hosts: *, User:41 has Allow permission for operations: > Read from hosts: *, User:44 has Allow permission for operations: Read from > hosts: *, User:48 has Allow permission for operations: Read from hosts: *, > User:2 has Allow permission for operations: Read from hosts: *, User:9 has > Allow permission for operations: Read from hosts: *, User:14 has Allow > permission for operations: Read from hosts: *, User:46 has Allow permission > for operations: Read from hosts: *, User:13 has Allow permission for > operations: Read from hosts: *, User:5 has Allow permission for operations: > Read from hosts: *, User:29 has Allow permission for operations: Read from > hosts: *, User:45 has Allow permission for operations: Read from hosts: *, > User:6 has Allow permission for operations: Read from hosts: *, User:37 has > Allow permission for operations: Read from hosts: *, User:23 has Allow > permission for operations: Read from hosts: *, User:19 has Allow permission > for operations: Read from hosts: *, User:24 has Allow permission for > operations: Read from hosts: *, User:17 has Allow permission for operations: > Read from hosts: *, User:34 has Allow permission for operations: Read from > hosts: *, User:12 has Allow permission for operations: Read from hosts: *, > User:42 has Allow permission for operations: Read from hosts: *, User:4 has > Allow permission for operations: Read from hosts: *, User:47 has Allow > permission for operations: Read from hosts: *, User:18 has Allow permission > for operations: Read from hosts: *, User:31 has Allow permission for > operations: Read from hosts: *, User:49 has Allow permission for operations: > Read from hosts: *, User:33 has Allow permission for operations: Read from > hosts: *, User:1 has Allow permission for operations: Read from hosts: *, > User:27 has Allow permission for operations: Read from hosts: *) but got > Set(User:36 has Allow permission for operations: Read from hosts: *, User:7 > has Allow permission for operations: Read from hosts: *, User:21 has Allow > permission for operations: Read from hosts: *, User:39 has Allow permission > for operations: Read from hosts: *, User:43 has Allow permission for > operations: Read from hosts: *, User:3 has Allow permission for operations: > Read from hosts: *, User:35 has Allow permission for operations: Read from > hosts: *, User:15 has Allow permission for operations: Read from hosts: *, > User:16 has Allow permission for operations: Read from hosts: *, User:22 has > Allow permission for operations: Read from hosts: *, User:26 has Allow > permission for operations: Read from hosts: *, User:11 has Allow permission > for operations: Read from hosts: *, User:38 has Allow permission for > operations: Read from hosts: *, User:8 has Allow permission for operations: > Read from hosts: *, User:28 has Allow permission for operations: Read from > hosts: *, User:32 has Allow permission for ope
[jira] [Created] (KAFKA-7174) Improve version probing of subscription info
Ted Yu created KAFKA-7174: - Summary: Improve version probing of subscription info Key: KAFKA-7174 URL: https://issues.apache.org/jira/browse/KAFKA-7174 Project: Kafka Issue Type: Improvement Reporter: Ted Yu During code review for KAFKA-5037, [~guozhang] made the following suggestion: Currently the version probing works as the following: when leader receives the subscription info encoded with a higher version that it can understand (e.g. the leader is on version 3, while one of the subscription received is encode with version 4), it will send back an empty assignment with the assignment encoded with version 3, and also latestSupportedVersion set to 3. when the member receives the assignment, it checks if latestSupportedVersion is smaller than the version it used for encoding the sent subscription (i.e. the above logic). If it is smaller, then it means that leader cannot understand, in this case, version 4. It will then set the flag and then re-subscribe but with a down-graded encoding format of version 3. NOW with PR #5322, we can let leader to clearly communicate this error via the error code, and upon receiving the assignment, if the error code is VERSION_PROBING, then the member can immediately know what happens, and hence can simplify the above logic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7175) Make version checking logic more flexible in streams_upgrade_test.py
Ted Yu created KAFKA-7175: - Summary: Make version checking logic more flexible in streams_upgrade_test.py Key: KAFKA-7175 URL: https://issues.apache.org/jira/browse/KAFKA-7175 Project: Kafka Issue Type: Improvement Reporter: Ted Yu During debugging of system test failure for KAFKA-5037, it was re-discovered that the version numbers inside version probing related messages are hard coded in streams_upgrade_test.py This is in-flexible. We should correlate latest version from Java class with the expected version numbers. Matthias made the following suggestion: We should also make this more generic and test upgrades from 3 -> 4, 3 -> 5 and 4 -> 5. The current code does only go from latest version to future version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7195) StreamStreamJoinIntegrationTest fails in 2.0 Jenkins
Ted Yu created KAFKA-7195: - Summary: StreamStreamJoinIntegrationTest fails in 2.0 Jenkins Key: KAFKA-7195 URL: https://issues.apache.org/jira/browse/KAFKA-7195 Project: Kafka Issue Type: Test Reporter: Ted Yu >From >https://builds.apache.org/job/kafka-2.0-jdk8/87/testReport/junit/org.apache.kafka.streams.integration/StreamStreamJoinIntegrationTest/testOuter_caching_enabled___false_/ > : {code} java.lang.AssertionError: Expected: is <[A-null]> but: was <[A-a, A-b, A-c, A-d]> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8) at org.apache.kafka.streams.integration.AbstractJoinIntegrationTest.checkResult(AbstractJoinIntegrationTest.java:171) at org.apache.kafka.streams.integration.AbstractJoinIntegrationTest.runTest(AbstractJoinIntegrationTest.java:212) at org.apache.kafka.streams.integration.AbstractJoinIntegrationTest.runTest(AbstractJoinIntegrationTest.java:184) at org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest.testOuter(StreamStreamJoinIntegrationTest.java:198) {code} However, some test output was missing: {code} [2018-07-23 20:51:36,363] INFO Socket c ...[truncated 1627692 chars]... 671) {code} I ran the test locally which passed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7276) Consider using re2j to speed up regex operations
Ted Yu created KAFKA-7276: - Summary: Consider using re2j to speed up regex operations Key: KAFKA-7276 URL: https://issues.apache.org/jira/browse/KAFKA-7276 Project: Kafka Issue Type: Task Reporter: Ted Yu https://github.com/google/re2j re2j claims to do linear time regular expression matching in Java. Its benefit is most obvious for deeply nested regex (such as a | b | c | d). We should consider using re2j to speed up regex operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7316) Use of filter method in KTable.scala may result in StackOverflowError
Ted Yu created KAFKA-7316: - Summary: Use of filter method in KTable.scala may result in StackOverflowError Key: KAFKA-7316 URL: https://issues.apache.org/jira/browse/KAFKA-7316 Project: Kafka Issue Type: Bug Reporter: Ted Yu In this thread: http://search-hadoop.com/m/Kafka/uyzND1dNbYKXzC4F1?subj=Issue+in+Kafka+2+0+0+ Druhin reported seeing StackOverflowError when using filter method from KTable.scala This can be reproduced with the following change: {code} diff --git a/streams/streams-scala/src/test/scala/org/apache/kafka/streams/scala/StreamToTableJoinScalaIntegrationTestImplicitSerdes.scala b/streams/streams-scala/src/test/scala index 3d1bab5..e0a06f2 100644 --- a/streams/streams-scala/src/test/scala/org/apache/kafka/streams/scala/StreamToTableJoinScalaIntegrationTestImplicitSerdes.scala +++ b/streams/streams-scala/src/test/scala/org/apache/kafka/streams/scala/StreamToTableJoinScalaIntegrationTestImplicitSerdes.scala @@ -58,6 +58,7 @@ class StreamToTableJoinScalaIntegrationTestImplicitSerdes extends StreamToTableJ val userClicksStream: KStream[String, Long] = builder.stream(userClicksTopic) val userRegionsTable: KTable[String, String] = builder.table(userRegionsTopic) +userRegionsTable.filter { case (_, count) => true } // Compute the total per region by summing the individual click counts per region. val clicksPerRegion: KTable[String, Long] = {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7334) Suggest changing config for state.dir in case of FileNotFoundException
Ted Yu created KAFKA-7334: - Summary: Suggest changing config for state.dir in case of FileNotFoundException Key: KAFKA-7334 URL: https://issues.apache.org/jira/browse/KAFKA-7334 Project: Kafka Issue Type: Improvement Reporter: Ted Yu Quoting stack trace from KAFKA-5998 : {code} WARN [2018-08-22 03:17:03,745] org.apache.kafka.streams.processor.internals.ProcessorStateManager: task [0_45] Failed to write offset checkpoint file to /tmp/kafka-streams/ {{ /0_45/.checkpoint: {}}} {{ ! java.nio.file.NoSuchFileException: /tmp/kafka-streams//0_45/.checkpoint.tmp}} {{ ! at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)}} {{ ! at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)}} {code} When state.dir is left at default configuration, there is a chance that certain files under the state directory are cleaned by OS. [~mjsax] and I proposed to suggest user, through exception message, to change the location for state.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7344) Return early when all tasks are assigned in StickyTaskAssignor#assignActive
Ted Yu created KAFKA-7344: - Summary: Return early when all tasks are assigned in StickyTaskAssignor#assignActive Key: KAFKA-7344 URL: https://issues.apache.org/jira/browse/KAFKA-7344 Project: Kafka Issue Type: Improvement Components: streams Reporter: Ted Yu After re-assigning existing active tasks to clients that previously had the same active task, there is chance that {{taskIds.size() == assigned.size()}}, i.e. all tasks are assigned . The method continues with: {code} final Set unassigned = new HashSet<>(taskIds); unassigned.removeAll(assigned); {code} We can check the above condition and return early before allocating HashSet. Similar optimization can be done before the following (around line 112): {code} // assign any remaining unassigned tasks final List sortedTasks = new ArrayList<>(unassigned); {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7345) Potentially unclosed FileChannel in StateDirectory#unlock
Ted Yu created KAFKA-7345: - Summary: Potentially unclosed FileChannel in StateDirectory#unlock Key: KAFKA-7345 URL: https://issues.apache.org/jira/browse/KAFKA-7345 Project: Kafka Issue Type: Bug Components: streams Reporter: Ted Yu {code} lockAndOwner.lock.release(); log.debug("{} Released state dir lock for task {}", logPrefix(), taskId); final FileChannel fileChannel = channels.remove(taskId); if (fileChannel != null) { fileChannel.close(); {code} If {{lockAndOwner.lock.release()}} throws IOE, the closing of the FileChannel would be skipped. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-5802) ScramServerCallbackHandler#handle should check username not being null before calling credentialCache.get()
Ted Yu created KAFKA-5802: - Summary: ScramServerCallbackHandler#handle should check username not being null before calling credentialCache.get() Key: KAFKA-5802 URL: https://issues.apache.org/jira/browse/KAFKA-5802 Project: Kafka Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} String username = null; for (Callback callback : callbacks) { if (callback instanceof NameCallback) username = ((NameCallback) callback).getDefaultName(); else if (callback instanceof ScramCredentialCallback) ((ScramCredentialCallback) callback).scramCredential(credentialCache.get(username)); {code} Since ConcurrentHashMap, used by CredentialCache, doesn't allow null keys, we should check that username is not null before calling credentialCache.get() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5820) Remove unneeded synchronized keyword in StreamThread
Ted Yu created KAFKA-5820: - Summary: Remove unneeded synchronized keyword in StreamThread Key: KAFKA-5820 URL: https://issues.apache.org/jira/browse/KAFKA-5820 Project: Kafka Issue Type: Improvement Reporter: Ted Yu Priority: Minor There are three methods in StreamThread which have unnecessary synchronized keyword since the variable accessed, state, is volatile : isRunningAndNotRebalancing isRunning shutdown synchronized keyword can be dropped for these methods. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5821) Intermittent test failure in SaslPlainSslEndToEndAuthorizationTest.testAcls
Ted Yu created KAFKA-5821: - Summary: Intermittent test failure in SaslPlainSslEndToEndAuthorizationTest.testAcls Key: KAFKA-5821 URL: https://issues.apache.org/jira/browse/KAFKA-5821 Project: Kafka Issue Type: Test Reporter: Ted Yu Priority: Minor >From >https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/7245/testReport/junit/kafka.api/SaslPlainSslEndToEndAuthorizationTest/testAcls/ > : {code} java.lang.SecurityException: zookeeper.set.acl is true, but the verification of the JAAS login file failed. at kafka.server.KafkaServer.initZk(KafkaServer.scala:329) at kafka.server.KafkaServer.startup(KafkaServer.scala:192) at kafka.utils.TestUtils$.createServer(TestUtils.scala:134) at kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:94) at kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:93) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.integration.KafkaServerTestHarness.setUp(KafkaServerTestHarness.scala:93) at kafka.api.IntegrationTestHarness.setUp(IntegrationTestHarness.scala:66) at kafka.api.EndToEndAuthorizationTest.setUp(EndToEndAuthorizationTest.scala:158) at kafka.api.SaslEndToEndAuthorizationTest.setUp(SaslEndToEndAuthorizationTest.scala:48) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) at com.sun.proxy.$Proxy1.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.gradle.internal.dispatch.Reflection
[jira] [Created] (KAFKA-5833) Reset thread interrupt state in case of InterruptedException
Ted Yu created KAFKA-5833: - Summary: Reset thread interrupt state in case of InterruptedException Key: KAFKA-5833 URL: https://issues.apache.org/jira/browse/KAFKA-5833 Project: Kafka Issue Type: Bug Reporter: Ted Yu There are some places where InterruptedException is caught but thread interrupt state is not reset. e.g. from WorkerSourceTask#execute() : {code} } catch (InterruptedException e) { // Ignore and allow to exit. {code} Proper way of handling InterruptedException is to reset thread interrupt state. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5840) TransactionsTest#testBasicTransactions hangs
Ted Yu created KAFKA-5840: - Summary: TransactionsTest#testBasicTransactions hangs Key: KAFKA-5840 URL: https://issues.apache.org/jira/browse/KAFKA-5840 Project: Kafka Issue Type: Test Reporter: Ted Yu Here is part of the stack trace: {code} "Test worker" #20 prio=5 os_prio=0 tid=0x7feb449fc000 nid=0x5f69 waiting on condition [0x7feb05f8c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x81272ec0> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) at org.apache.kafka.clients.producer.internals.ProduceRequestResult.await(ProduceRequestResult.java:76) at org.apache.kafka.clients.producer.internals.RecordAccumulator.awaitFlushCompletion(RecordAccumulator.java:573) at org.apache.kafka.clients.producer.KafkaProducer.flush(KafkaProducer.java:948) at kafka.api.TransactionsTest.testBasicTransactions(TransactionsTest.scala:93) {code} {code} Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T19:39:06Z) Maven home: /apache-maven-3.5.0 Java version: 1.8.0_131, vendor: Oracle Corporation Java home: /jdk1.8.0_131/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-327.28.3.el7.x86_64", arch: "amd64", family: "unix" {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5842) QueryableStateIntegrationTest may fail with JDK 7
Ted Yu created KAFKA-5842: - Summary: QueryableStateIntegrationTest may fail with JDK 7 Key: KAFKA-5842 URL: https://issues.apache.org/jira/browse/KAFKA-5842 Project: Kafka Issue Type: Test Reporter: Ted Yu Priority: Minor Found the following when running test suite for 0.11.0.1 RC0 : {code} org.apache.kafka.streams.integration.QueryableStateIntegrationTest > concurrentAccesses FAILED java.lang.AssertionError: Key not found one at org.junit.Assert.fail(Assert.java:88) at org.apache.kafka.streams.integration.QueryableStateIntegrationTest.verifyGreaterOrEqual(QueryableStateIntegrationTest.java:893) at org.apache.kafka.streams.integration.QueryableStateIntegrationTest.concurrentAccesses(QueryableStateIntegrationTest.java:399) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5846) Use singleton NoOpConsumerRebalanceListener in subscribe() call where listener is not specified
Ted Yu created KAFKA-5846: - Summary: Use singleton NoOpConsumerRebalanceListener in subscribe() call where listener is not specified Key: KAFKA-5846 URL: https://issues.apache.org/jira/browse/KAFKA-5846 Project: Kafka Issue Type: Task Reporter: Ted Yu Priority: Minor Currently KafkaConsumer creates instance of NoOpConsumerRebalanceListener for each subscribe() call where ConsumerRebalanceListener is not specified: {code} public void subscribe(Pattern pattern) { subscribe(pattern, new NoOpConsumerRebalanceListener()); {code} We can create a singleton NoOpConsumerRebalanceListener to be used in such scenarios. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5863) Potential null dereference in DistributedHerder#reconfigureConnector()
Ted Yu created KAFKA-5863: - Summary: Potential null dereference in DistributedHerder#reconfigureConnector() Key: KAFKA-5863 URL: https://issues.apache.org/jira/browse/KAFKA-5863 Project: Kafka Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is the call chain: {code} RestServer.httpRequest(reconfigUrl, "POST", taskProps, null); {code} In httpRequest(): {code} } else if (responseCode >= 200 && responseCode < 300) { InputStream is = connection.getInputStream(); T result = JSON_SERDE.readValue(is, responseFormat); {code} For readValue(): {code} public T readValue(InputStream src, TypeReference valueTypeRef) throws IOException, JsonParseException, JsonMappingException { return (T) _readMapAndClose(_jsonFactory.createParser(src), _typeFactory.constructType(valueTypeRef)); {code} Then there would be NPE in constructType(): {code} public JavaType constructType(TypeReference typeRef) { // 19-Oct-2015, tatu: Simpler variant like so should work return _fromAny(null, typeRef.getType(), EMPTY_BINDINGS); {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5889) MetricsTest is flaky
Ted Yu created KAFKA-5889: - Summary: MetricsTest is flaky Key: KAFKA-5889 URL: https://issues.apache.org/jira/browse/KAFKA-5889 Project: Kafka Issue Type: Test Reporter: Ted Yu The following appeared in several recent builds (e.g. https://builds.apache.org/job/kafka-trunk-jdk7/2758) : {code} kafka.metrics.MetricsTest > testMetricsLeak FAILED java.lang.AssertionError: expected:<1216> but was:<1219> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:834) at org.junit.Assert.assertEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:631) at kafka.metrics.MetricsTest$$anonfun$testMetricsLeak$1.apply$mcVI$sp(MetricsTest.scala:68) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) at kafka.metrics.MetricsTest.testMetricsLeak(MetricsTest.scala:66) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5911) Avoid creation of extra Map for futures in KafkaAdminClient
Ted Yu created KAFKA-5911: - Summary: Avoid creation of extra Map for futures in KafkaAdminClient Key: KAFKA-5911 URL: https://issues.apache.org/jira/browse/KAFKA-5911 Project: Kafka Issue Type: Bug Reporter: Ted Yu In various methods from KafkaAdminClient, there is extra Map created when constructing XXResult instance. e.g. {code} return new DescribeReplicaLogDirResult(new HashMap>(futures)); {code} Prior to returning, futures Map is already filled. Calling get() and values() does not involve the internals of HashMap when we consider thread-safety. The extra Map doesn't need to be created. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5916) Upgrade rocksdb dependency to 5.7.3
Ted Yu created KAFKA-5916: - Summary: Upgrade rocksdb dependency to 5.7.3 Key: KAFKA-5916 URL: https://issues.apache.org/jira/browse/KAFKA-5916 Project: Kafka Issue Type: Improvement Reporter: Ted Yu Priority: Minor Currently we use 5.3.6. The latest release is 5.7.3 : https://github.com/facebook/rocksdb/releases We should upgrade to latest rocksdb release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5943) Reduce dependency on mock in connector tests
Ted Yu created KAFKA-5943: - Summary: Reduce dependency on mock in connector tests Key: KAFKA-5943 URL: https://issues.apache.org/jira/browse/KAFKA-5943 Project: Kafka Issue Type: Improvement Reporter: Ted Yu Currently connector tests make heavy use of mock (easymock, power mock). This often hides the real logic behind operations and makes finding bugs difficult. We should reduce the use of mocks so that developers can debug connector code using unit tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5946) Give connector method parameter better name
Ted Yu created KAFKA-5946: - Summary: Give connector method parameter better name Key: KAFKA-5946 URL: https://issues.apache.org/jira/browse/KAFKA-5946 Project: Kafka Issue Type: Improvement Reporter: Ted Yu During the development of KAFKA-5657, there were several iterations where method call didn't match what the connector parameter actually represents. [~ewencp] had used connType as equivalent to connClass because Type wasn't used to differentiate source vs sink. [~ewencp] proposed the following: It would help to convert all the uses of connType to connClass first, then standardize on class == java class, type == source/sink, name == user-specified name. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-5821) Intermittent test failure in SaslPlainSslEndToEndAuthorizationTest.testAcls
[ https://issues.apache.org/jira/browse/KAFKA-5821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-5821. --- Resolution: Cannot Reproduce > Intermittent test failure in SaslPlainSslEndToEndAuthorizationTest.testAcls > --- > > Key: KAFKA-5821 > URL: https://issues.apache.org/jira/browse/KAFKA-5821 > Project: Kafka > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From > https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/7245/testReport/junit/kafka.api/SaslPlainSslEndToEndAuthorizationTest/testAcls/ > : > {code} > java.lang.SecurityException: zookeeper.set.acl is true, but the verification > of the JAAS login file failed. > at kafka.server.KafkaServer.initZk(KafkaServer.scala:329) > at kafka.server.KafkaServer.startup(KafkaServer.scala:192) > at kafka.utils.TestUtils$.createServer(TestUtils.scala:134) > at > kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:94) > at > kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:93) > at scala.collection.Iterator$class.foreach(Iterator.scala:891) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at > kafka.integration.KafkaServerTestHarness.setUp(KafkaServerTestHarness.scala:93) > at > kafka.api.IntegrationTestHarness.setUp(IntegrationTestHarness.scala:66) > at > kafka.api.EndToEndAuthorizationTest.setUp(EndToEndAuthorizationTest.scala:158) > at > kafka.api.SaslEndToEndAuthorizationTest.setUp(SaslEndToEndAuthorizationTest.scala:48) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) > at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) > at > org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) > at com.sun.proxy.$Proxy1.processTestClass(Unknown Source) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.jav
[jira] [Resolved] (KAFKA-5840) TransactionsTest#testBasicTransactions sometimes hangs
[ https://issues.apache.org/jira/browse/KAFKA-5840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-5840. --- Resolution: Cannot Reproduce > TransactionsTest#testBasicTransactions sometimes hangs > -- > > Key: KAFKA-5840 > URL: https://issues.apache.org/jira/browse/KAFKA-5840 > Project: Kafka > Issue Type: Test >Reporter: Ted Yu > Attachments: 5840.stack > > > While testing 0.11.0.1 RC0 , I found TransactionsTest hanging. > Here is part of the stack trace: > {code} > "Test worker" #20 prio=5 os_prio=0 tid=0x7feb449fc000 nid=0x5f69 waiting > on condition [0x7feb05f8c000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x81272ec0> (a > java.util.concurrent.CountDownLatch$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) > at > org.apache.kafka.clients.producer.internals.ProduceRequestResult.await(ProduceRequestResult.java:76) > at > org.apache.kafka.clients.producer.internals.RecordAccumulator.awaitFlushCompletion(RecordAccumulator.java:573) > at > org.apache.kafka.clients.producer.KafkaProducer.flush(KafkaProducer.java:948) > at > kafka.api.TransactionsTest.testBasicTransactions(TransactionsTest.scala:93) > {code} > {code} > Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; > 2017-04-03T19:39:06Z) > Maven home: /apache-maven-3.5.0 > Java version: 1.8.0_131, vendor: Oracle Corporation > Java home: /jdk1.8.0_131/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "linux", version: "3.10.0-327.28.3.el7.x86_64", arch: "amd64", > family: "unix" > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5967) Ineffective check of negative value in CompositeReadOnlyKeyValueStore#approximateNumEntries()
Ted Yu created KAFKA-5967: - Summary: Ineffective check of negative value in CompositeReadOnlyKeyValueStore#approximateNumEntries() Key: KAFKA-5967 URL: https://issues.apache.org/jira/browse/KAFKA-5967 Project: Kafka Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} long total = 0; for (ReadOnlyKeyValueStore store : stores) { total += store.approximateNumEntries(); } return total < 0 ? Long.MAX_VALUE : total; {code} The check for negative value seems to account for wrapping. However, wrapping can happen within the for loop. So the check should be performed inside the loop. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5988) Consider removing StreamThread#STREAM_THREAD_ID_SEQUENCE
Ted Yu created KAFKA-5988: - Summary: Consider removing StreamThread#STREAM_THREAD_ID_SEQUENCE Key: KAFKA-5988 URL: https://issues.apache.org/jira/browse/KAFKA-5988 Project: Kafka Issue Type: Improvement Reporter: Ted Yu Priority: Minor StreamThread#STREAM_THREAD_ID_SEQUENCE is used for naming (numbering) StreamThread's . It is used in create() which is called from a loop in KafkaStreams ctor. We can remove STREAM_THREAD_ID_SEQUENCE and pass the loop index to create() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-5842) QueryableStateIntegrationTest may fail with JDK 7
[ https://issues.apache.org/jira/browse/KAFKA-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-5842. --- Resolution: Cannot Reproduce > QueryableStateIntegrationTest may fail with JDK 7 > - > > Key: KAFKA-5842 > URL: https://issues.apache.org/jira/browse/KAFKA-5842 > Project: Kafka > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > Found the following when running test suite for 0.11.0.1 RC0 : > {code} > org.apache.kafka.streams.integration.QueryableStateIntegrationTest > > concurrentAccesses FAILED > java.lang.AssertionError: Key not found one > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.kafka.streams.integration.QueryableStateIntegrationTest.verifyGreaterOrEqual(QueryableStateIntegrationTest.java:893) > at > org.apache.kafka.streams.integration.QueryableStateIntegrationTest.concurrentAccesses(QueryableStateIntegrationTest.java:399) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-5916) Upgrade rocksdb dependency to 5.8
[ https://issues.apache.org/jira/browse/KAFKA-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-5916. --- Resolution: Duplicate With KAFKA-5576 > Upgrade rocksdb dependency to 5.8 > - > > Key: KAFKA-5916 > URL: https://issues.apache.org/jira/browse/KAFKA-5916 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 0.11.0.0 >Reporter: Ted Yu >Priority: Minor > > Currently we use 5.3.6. > The latest release is 5.8 : > https://github.com/facebook/rocksdb/releases > We should upgrade to latest rocksdb release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6023) ThreadCache#sizeBytes() should check overflow
Ted Yu created KAFKA-6023: - Summary: ThreadCache#sizeBytes() should check overflow Key: KAFKA-6023 URL: https://issues.apache.org/jira/browse/KAFKA-6023 Project: Kafka Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} long sizeBytes() { long sizeInBytes = 0; for (final NamedCache namedCache : caches.values()) { sizeInBytes += namedCache.sizeInBytes(); } return sizeInBytes; } {code} The summation w.r.t. sizeInBytes may overflow. Check similar to what is done in size() should be performed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6024) Consider moving validation in KafkaConsumer ahead of call to acquireAndEnsureOpen()
Ted Yu created KAFKA-6024: - Summary: Consider moving validation in KafkaConsumer ahead of call to acquireAndEnsureOpen() Key: KAFKA-6024 URL: https://issues.apache.org/jira/browse/KAFKA-6024 Project: Kafka Issue Type: Improvement Reporter: Ted Yu Priority: Minor In several methods, parameter validation is done after calling acquireAndEnsureOpen() : {code} public void seek(TopicPartition partition, long offset) { acquireAndEnsureOpen(); try { if (offset < 0) throw new IllegalArgumentException("seek offset must not be a negative number"); {code} Since the value of parameter would not change per invocation, it seems performing validation ahead of acquireAndEnsureOpen() call would be better. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-5988) Consider removing StreamThread#STREAM_THREAD_ID_SEQUENCE
[ https://issues.apache.org/jira/browse/KAFKA-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-5988. --- Resolution: Won't Fix > Consider removing StreamThread#STREAM_THREAD_ID_SEQUENCE > > > Key: KAFKA-5988 > URL: https://issues.apache.org/jira/browse/KAFKA-5988 > Project: Kafka > Issue Type: Improvement >Reporter: Ted Yu >Assignee: siva santhalingam >Priority: Minor > > StreamThread#STREAM_THREAD_ID_SEQUENCE is used for naming (numbering) > StreamThread's . > It is used in create() which is called from a loop in KafkaStreams ctor. > We can remove STREAM_THREAD_ID_SEQUENCE and pass the loop index to create() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-5911) Avoid creation of extra Map for futures in KafkaAdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-5911. --- Resolution: Later > Avoid creation of extra Map for futures in KafkaAdminClient > --- > > Key: KAFKA-5911 > URL: https://issues.apache.org/jira/browse/KAFKA-5911 > Project: Kafka > Issue Type: Bug >Reporter: Ted Yu > Labels: client > Attachments: 5911.v1.txt > > > In various methods from KafkaAdminClient, there is extra Map created when > constructing XXResult instance. > e.g. > {code} > return new DescribeReplicaLogDirResult(new > HashMap>(futures)); > {code} > Prior to returning, futures Map is already filled. > Calling get() and values() does not involve the internals of HashMap when we > consider thread-safety. > The extra Map doesn't need to be created. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6109) ResetIntegrationTest may fail due to IllegalArgumentException
Ted Yu created KAFKA-6109: - Summary: ResetIntegrationTest may fail due to IllegalArgumentException Key: KAFKA-6109 URL: https://issues.apache.org/jira/browse/KAFKA-6109 Project: Kafka Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/kafka-trunk-jdk7/2918 : {code} org.apache.kafka.streams.integration.ResetIntegrationTest > testReprocessingFromScratchAfterResetWithIntermediateUserTopic FAILED java.lang.IllegalArgumentException: Setting the time to 1508791687000 while current time 1508791687475 is newer; this is not allowed at org.apache.kafka.common.utils.MockTime.setCurrentTimeMs(MockTime.java:81) at org.apache.kafka.streams.integration.AbstractResetIntegrationTest.beforePrepareTest(AbstractResetIntegrationTest.java:114) at org.apache.kafka.streams.integration.ResetIntegrationTest.before(ResetIntegrationTest.java:55) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6135) TransactionsTest#testFencingOnCommit may fail due to unexpected KafkaException
Ted Yu created KAFKA-6135: - Summary: TransactionsTest#testFencingOnCommit may fail due to unexpected KafkaException Key: KAFKA-6135 URL: https://issues.apache.org/jira/browse/KAFKA-6135 Project: Kafka Issue Type: Bug Reporter: Ted Yu >From >https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/2293/testReport/junit/kafka.api/TransactionsTest/testFencingOnCommit/ > : {code} org.scalatest.junit.JUnitTestFailedError: Got an unexpected exception from a fenced producer. at org.scalatest.junit.AssertionsForJUnit.newAssertionFailedException(AssertionsForJUnit.scala:100) at org.scalatest.junit.AssertionsForJUnit.newAssertionFailedException$(AssertionsForJUnit.scala:99) at org.scalatest.junit.JUnitSuite.newAssertionFailedException(JUnitSuite.scala:71) at org.scalatest.Assertions.fail(Assertions.scala:1105) at org.scalatest.Assertions.fail$(Assertions.scala:1101) at org.scalatest.junit.JUnitSuite.fail(JUnitSuite.scala:71) at kafka.api.TransactionsTest.testFencingOnCommit(TransactionsTest.scala:319) ... Caused by: org.apache.kafka.common.KafkaException: Cannot execute transactional method because we are in an error state at org.apache.kafka.clients.producer.internals.TransactionManager.maybeFailWithError(TransactionManager.java:782) at org.apache.kafka.clients.producer.internals.TransactionManager.beginCommit(TransactionManager.java:220) at org.apache.kafka.clients.producer.KafkaProducer.commitTransaction(KafkaProducer.java:617) at kafka.api.TransactionsTest.testFencingOnCommit(TransactionsTest.scala:313) ... 48 more Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker. {code} Confirmed with [~apurva] that the above would not be covered by his fix for KAFKA-6119 Temporarily marking this as bug. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6137) RestoreIntegrationTest sometimes fails with assertion error
Ted Yu created KAFKA-6137: - Summary: RestoreIntegrationTest sometimes fails with assertion error Key: KAFKA-6137 URL: https://issues.apache.org/jira/browse/KAFKA-6137 Project: Kafka Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/kafka-1.0-jdk7/62 : {code} org.apache.kafka.streams.integration.RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled FAILED java.lang.AssertionError at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.kafka.streams.integration.RestoreIntegrationTest.shouldSuccessfullyStartWhenLoggingDisabled(RestoreIntegrationTest.java:195) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-6109) ResetIntegrationTest may fail due to IllegalArgumentException
[ https://issues.apache.org/jira/browse/KAFKA-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-6109. --- Resolution: Cannot Reproduce > ResetIntegrationTest may fail due to IllegalArgumentException > - > > Key: KAFKA-6109 > URL: https://issues.apache.org/jira/browse/KAFKA-6109 > Project: Kafka > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From https://builds.apache.org/job/kafka-trunk-jdk7/2918 : > {code} > org.apache.kafka.streams.integration.ResetIntegrationTest > > testReprocessingFromScratchAfterResetWithIntermediateUserTopic FAILED > java.lang.IllegalArgumentException: Setting the time to 1508791687000 > while current time 1508791687475 is newer; this is not allowed > at > org.apache.kafka.common.utils.MockTime.setCurrentTimeMs(MockTime.java:81) > at > org.apache.kafka.streams.integration.AbstractResetIntegrationTest.beforePrepareTest(AbstractResetIntegrationTest.java:114) > at > org.apache.kafka.streams.integration.ResetIntegrationTest.before(ResetIntegrationTest.java:55) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-6137) RestoreIntegrationTest sometimes fails with assertion error
[ https://issues.apache.org/jira/browse/KAFKA-6137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-6137. --- Resolution: Cannot Reproduce > RestoreIntegrationTest sometimes fails with assertion error > --- > > Key: KAFKA-6137 > URL: https://issues.apache.org/jira/browse/KAFKA-6137 > Project: Kafka > Issue Type: Test > Components: streams, unit tests >Reporter: Ted Yu >Priority: Minor > Labels: flaky-test > > From https://builds.apache.org/job/kafka-1.0-jdk7/62 : > {code} > org.apache.kafka.streams.integration.RestoreIntegrationTest > > shouldSuccessfullyStartWhenLoggingDisabled FAILED > java.lang.AssertionError > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.kafka.streams.integration.RestoreIntegrationTest.shouldSuccessfullyStartWhenLoggingDisabled(RestoreIntegrationTest.java:195) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6193) ReassignPartitionsClusterTest.shouldPerformMultipleReassignmentOperationsOverVariousTopics fails sometimes
Ted Yu created KAFKA-6193: - Summary: ReassignPartitionsClusterTest.shouldPerformMultipleReassignmentOperationsOverVariousTopics fails sometimes Key: KAFKA-6193 URL: https://issues.apache.org/jira/browse/KAFKA-6193 Project: Kafka Issue Type: Test Reporter: Ted Yu >From >https://builds.apache.org/job/kafka-trunk-jdk8/2198/testReport/junit/kafka.admin/ReassignPartitionsClusterTest/shouldPerformMultipleReassignmentOperationsOverVariousTopics/ > : {code} java.lang.AssertionError: expected: but was: at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:834) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at kafka.admin.ReassignPartitionsClusterTest.shouldPerformMultipleReassignmentOperationsOverVariousTopics(ReassignPartitionsClusterTest.scala:524) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-6135) TransactionsTest#testFencingOnCommit may fail due to unexpected KafkaException
[ https://issues.apache.org/jira/browse/KAFKA-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-6135. --- Resolution: Cannot Reproduce > TransactionsTest#testFencingOnCommit may fail due to unexpected KafkaException > -- > > Key: KAFKA-6135 > URL: https://issues.apache.org/jira/browse/KAFKA-6135 > Project: Kafka > Issue Type: Bug >Reporter: Ted Yu > Attachments: 6135.out > > > From > https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/2293/testReport/junit/kafka.api/TransactionsTest/testFencingOnCommit/ > : > {code} > org.scalatest.junit.JUnitTestFailedError: Got an unexpected exception from a > fenced producer. > at > org.scalatest.junit.AssertionsForJUnit.newAssertionFailedException(AssertionsForJUnit.scala:100) > at > org.scalatest.junit.AssertionsForJUnit.newAssertionFailedException$(AssertionsForJUnit.scala:99) > at > org.scalatest.junit.JUnitSuite.newAssertionFailedException(JUnitSuite.scala:71) > at org.scalatest.Assertions.fail(Assertions.scala:1105) > at org.scalatest.Assertions.fail$(Assertions.scala:1101) > at org.scalatest.junit.JUnitSuite.fail(JUnitSuite.scala:71) > at > kafka.api.TransactionsTest.testFencingOnCommit(TransactionsTest.scala:319) > ... > Caused by: org.apache.kafka.common.KafkaException: Cannot execute > transactional method because we are in an error state > at > org.apache.kafka.clients.producer.internals.TransactionManager.maybeFailWithError(TransactionManager.java:782) > at > org.apache.kafka.clients.producer.internals.TransactionManager.beginCommit(TransactionManager.java:220) > at > org.apache.kafka.clients.producer.KafkaProducer.commitTransaction(KafkaProducer.java:617) > at > kafka.api.TransactionsTest.testFencingOnCommit(TransactionsTest.scala:313) > ... 48 more > Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer > attempted an operation with an old epoch. Either there is a newer producer > with the same transactionalId, or the producer's transaction has been expired > by the broker. > {code} > Confirmed with [~apurva] that the above would not be covered by his fix for > KAFKA-6119 > Temporarily marking this as bug. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6215) KafkaStreamsTest fails in trunk
Ted Yu created KAFKA-6215: - Summary: KafkaStreamsTest fails in trunk Key: KAFKA-6215 URL: https://issues.apache.org/jira/browse/KAFKA-6215 Project: Kafka Issue Type: Test Reporter: Ted Yu Two subtests fail. https://builds.apache.org/job/kafka-trunk-jdk9/193/testReport/junit/org.apache.kafka.streams/KafkaStreamsTest/testCannotCleanupWhileRunning/ {code} org.apache.kafka.streams.errors.StreamsException: org.apache.kafka.streams.errors.ProcessorStateException: state directory [/tmp/kafka-streams/testCannotCleanupWhileRunning] doesn't exist and couldn't be created at org.apache.kafka.streams.KafkaStreams.(KafkaStreams.java:618) at org.apache.kafka.streams.KafkaStreams.(KafkaStreams.java:505) at org.apache.kafka.streams.KafkaStreamsTest.testCannotCleanupWhileRunning(KafkaStreamsTest.java:462) {code} testCleanup fails in similar manner. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6228) Intermittent test failure in FetchRequestTest.testDownConversionWithConnectionFailure
Ted Yu created KAFKA-6228: - Summary: Intermittent test failure in FetchRequestTest.testDownConversionWithConnectionFailure Key: KAFKA-6228 URL: https://issues.apache.org/jira/browse/KAFKA-6228 Project: Kafka Issue Type: Test Reporter: Ted Yu Priority: Minor >From >https://builds.apache.org/job/kafka-trunk-jdk8/2219/testReport/junit/kafka.server/FetchRequestTest/testDownConversionWithConnectionFailure/ > : {code} java.lang.AssertionError: Fetch size too small 42, broker may have run out of memory at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at kafka.server.FetchRequestTest.kafka$server$FetchRequestTest$$fetch$1(FetchRequestTest.scala:214) at kafka.server.FetchRequestTest$$anonfun$testDownConversionWithConnectionFailure$2.apply(FetchRequestTest.scala:226) at kafka.server.FetchRequestTest$$anonfun$testDownConversionWithConnectionFailure$2.apply(FetchRequestTest.scala:226) at scala.collection.immutable.Range.foreach(Range.scala:160) at kafka.server.FetchRequestTest.testDownConversionWithConnectionFailure(FetchRequestTest.scala:226) {code} I ran FetchRequestTest locally which passed. {code} assertTrue(s"Fetch size too small $size, broker may have run out of memory", size > maxPartitionBytes - batchSize) {code} The assertion message should include maxPartitionBytes and batchSize which would give us more information. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6232) SaslSslAdminClientIntegrationTest sometimes fails
Ted Yu created KAFKA-6232: - Summary: SaslSslAdminClientIntegrationTest sometimes fails Key: KAFKA-6232 URL: https://issues.apache.org/jira/browse/KAFKA-6232 Project: Kafka Issue Type: Test Reporter: Ted Yu Here was one recent occurrence: https://builds.apache.org/job/kafka-trunk-jdk9/203/testReport/kafka.api/SaslSslAdminClientIntegrationTest/testLogStartOffsetCheckpoint/ {code} java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.LeaderNotAvailableException: There is no leader for this topic-partition as we are in the middle of a leadership election. at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:225) at kafka.api.AdminClientIntegrationTest.$anonfun$testLogStartOffsetCheckpoint$3(AdminClientIntegrationTest.scala:762) {code} In the test output, I saw: {code} [2017-11-17 23:15:45,593] ERROR [KafkaApi-1] Error when handling request {filters=[{resource_type=2,resource_name=foobar,principal=User:ANONYMOUS,host=*,operation=3,permission_type=3},{resource_type=5,resource_name=transactional_id,principal=User:ANONYMOUS,host=*,operation=4,permission_type=3}]} (kafka.server.KafkaApis:107) org.apache.kafka.common.errors.ClusterAuthorizationException: Request Request(processor=2, connectionId=127.0.0.1:36295-127.0.0.1:58183-0, session=Session(User:client2,localhost/127.0.0.1), listenerName=ListenerName(SASL_SSL), securityProtocol=SASL_SSL, buffer=null) is not authorized. {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6300) SelectorTest may fail with ConcurrentModificationException
Ted Yu created KAFKA-6300: - Summary: SelectorTest may fail with ConcurrentModificationException Key: KAFKA-6300 URL: https://issues.apache.org/jira/browse/KAFKA-6300 Project: Kafka Issue Type: Test Reporter: Ted Yu Priority: Minor >From >https://builds.apache.org/job/kafka-trunk-jdk8/2255/testReport/junit/org.apache.kafka.common.network/SelectorTest/testImmediatelyConnectedCleaned/ > : {code} java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) at java.util.ArrayList$Itr.next(ArrayList.java:859) at org.apache.kafka.common.network.EchoServer.closeConnections(EchoServer.java:115) at org.apache.kafka.common.network.EchoServer.close(EchoServer.java:121) at org.apache.kafka.common.network.SelectorTest.tearDown(SelectorTest.java:95) {code} It seems sockets ArrayList was modified during closing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6303) Potential lack of synchronization in NioEchoServer#AcceptorThread
Ted Yu created KAFKA-6303: - Summary: Potential lack of synchronization in NioEchoServer#AcceptorThread Key: KAFKA-6303 URL: https://issues.apache.org/jira/browse/KAFKA-6303 Project: Kafka Issue Type: Bug Reporter: Ted Yu Priority: Minor In the run() method: {code} SocketChannel socketChannel = ((ServerSocketChannel) key.channel()).accept(); socketChannel.configureBlocking(false); newChannels.add(socketChannel); {code} Modification to newChannels should be protected by synchronized block. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6307) mBeanName should be removed before returning from JmxReporter#removeAttribute()
Ted Yu created KAFKA-6307: - Summary: mBeanName should be removed before returning from JmxReporter#removeAttribute() Key: KAFKA-6307 URL: https://issues.apache.org/jira/browse/KAFKA-6307 Project: Kafka Issue Type: Bug Reporter: Ted Yu JmxReporter$KafkaMbean showed up near the top in the first histo output from KAFKA-6199. In JmxReporter#removeAttribute() : {code} KafkaMbean mbean = this.mbeans.get(mBeanName); if (mbean != null) mbean.removeAttribute(metricName.name()); return mbean; {code} mbeans.remove(mBeanName) should be called before returning. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-6335) SimpleAclAuthorizerTest#testHighConcurrencyModificationOfResourceAcls fails intermittently
Ted Yu created KAFKA-6335: - Summary: SimpleAclAuthorizerTest#testHighConcurrencyModificationOfResourceAcls fails intermittently Key: KAFKA-6335 URL: https://issues.apache.org/jira/browse/KAFKA-6335 Project: Kafka Issue Type: Test Reporter: Ted Yu Priority: Minor >From >https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/3045/testReport/junit/kafka.security.auth/SimpleAclAuthorizerTest/testHighConcurrencyModificationOfResourceAcls/ > : {code} java.lang.AssertionError: expected acls Set(User:36 has Allow permission for operations: Read from hosts: *, User:7 has Allow permission for operations: Read from hosts: *, User:21 has Allow permission for operations: Read from hosts: *, User:39 has Allow permission for operations: Read from hosts: *, User:43 has Allow permission for operations: Read from hosts: *, User:3 has Allow permission for operations: Read from hosts: *, User:35 has Allow permission for operations: Read from hosts: *, User:15 has Allow permission for operations: Read from hosts: *, User:16 has Allow permission for operations: Read from hosts: *, User:22 has Allow permission for operations: Read from hosts: *, User:26 has Allow permission for operations: Read from hosts: *, User:11 has Allow permission for operations: Read from hosts: *, User:38 has Allow permission for operations: Read from hosts: *, User:8 has Allow permission for operations: Read from hosts: *, User:28 has Allow permission for operations: Read from hosts: *, User:32 has Allow permission for operations: Read from hosts: *, User:25 has Allow permission for operations: Read from hosts: *, User:41 has Allow permission for operations: Read from hosts: *, User:44 has Allow permission for operations: Read from hosts: *, User:48 has Allow permission for operations: Read from hosts: *, User:2 has Allow permission for operations: Read from hosts: *, User:9 has Allow permission for operations: Read from hosts: *, User:14 has Allow permission for operations: Read from hosts: *, User:46 has Allow permission for operations: Read from hosts: *, User:13 has Allow permission for operations: Read from hosts: *, User:5 has Allow permission for operations: Read from hosts: *, User:29 has Allow permission for operations: Read from hosts: *, User:45 has Allow permission for operations: Read from hosts: *, User:6 has Allow permission for operations: Read from hosts: *, User:37 has Allow permission for operations: Read from hosts: *, User:23 has Allow permission for operations: Read from hosts: *, User:19 has Allow permission for operations: Read from hosts: *, User:24 has Allow permission for operations: Read from hosts: *, User:17 has Allow permission for operations: Read from hosts: *, User:34 has Allow permission for operations: Read from hosts: *, User:12 has Allow permission for operations: Read from hosts: *, User:42 has Allow permission for operations: Read from hosts: *, User:4 has Allow permission for operations: Read from hosts: *, User:47 has Allow permission for operations: Read from hosts: *, User:18 has Allow permission for operations: Read from hosts: *, User:31 has Allow permission for operations: Read from hosts: *, User:49 has Allow permission for operations: Read from hosts: *, User:33 has Allow permission for operations: Read from hosts: *, User:1 has Allow permission for operations: Read from hosts: *, User:27 has Allow permission for operations: Read from hosts: *) but got Set(User:36 has Allow permission for operations: Read from hosts: *, User:7 has Allow permission for operations: Read from hosts: *, User:21 has Allow permission for operations: Read from hosts: *, User:39 has Allow permission for operations: Read from hosts: *, User:43 has Allow permission for operations: Read from hosts: *, User:3 has Allow permission for operations: Read from hosts: *, User:35 has Allow permission for operations: Read from hosts: *, User:15 has Allow permission for operations: Read from hosts: *, User:16 has Allow permission for operations: Read from hosts: *, User:22 has Allow permission for operations: Read from hosts: *, User:26 has Allow permission for operations: Read from hosts: *, User:11 has Allow permission for operations: Read from hosts: *, User:38 has Allow permission for operations: Read from hosts: *, User:8 has Allow permission for operations: Read from hosts: *, User:28 has Allow permission for operations: Read from hosts: *, User:32 has Allow permission for operations: Read from hosts: *, User:25 has Allow permission for operations: Read from hosts: *, User:41 has Allow permission for operations: Read from hosts: *, User:44 has Allow permission for operations: Read from hosts: *, User:48 has Allow permission for operations: Read from hosts: *, User:2 has Allow permission for operations: Read from hosts: *, User:9 has Allow permission for
[jira] [Commented] (KAFKA-3235) Unclosed stream in AppInfoParser static block
[ https://issues.apache.org/jira/browse/KAFKA-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155887#comment-15155887 ] Ted Yu commented on KAFKA-3235: --- clients/src/main/java/org/apache/kafka/common/utils/AppInfoParser.java appears multiple times in the patch. Not sure if the patch was generated correctly. > Unclosed stream in AppInfoParser static block > - > > Key: KAFKA-3235 > URL: https://issues.apache.org/jira/browse/KAFKA-3235 > Project: Kafka > Issue Type: Bug >Reporter: Ted Yu >Priority: Minor > Fix For: 0.9.1.0 > > > {code} > static { > try { > Properties props = new Properties(); > > props.load(AppInfoParser.class.getResourceAsStream("/kafka/kafka-version.properties")); > version = props.getProperty("version", version).trim(); > commitId = props.getProperty("commitId", commitId).trim(); > {code} > The stream returned by getResourceAsStream() should be closed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-3235) Unclosed stream in AppInfoParser static block
Ted Yu created KAFKA-3235: - Summary: Unclosed stream in AppInfoParser static block Key: KAFKA-3235 URL: https://issues.apache.org/jira/browse/KAFKA-3235 Project: Kafka Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} static { try { Properties props = new Properties(); props.load(AppInfoParser.class.getResourceAsStream("/kafka/kafka-version.properties")); version = props.getProperty("version", version).trim(); commitId = props.getProperty("commitId", commitId).trim(); {code} The stream returned by getResourceAsStream() should be closed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-9462) Correct exception message in DistributedHerder
Ted Yu created KAFKA-9462: - Summary: Correct exception message in DistributedHerder Key: KAFKA-9462 URL: https://issues.apache.org/jira/browse/KAFKA-9462 Project: Kafka Issue Type: Bug Reporter: Ted Yu There are a few exception messages in DistributedHerder which were copied from other exception message. This task corrects the messages to reflect actual condition -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9463) Transient failure in KafkaAdminClientTest.testListOffsets
Ted Yu created KAFKA-9463: - Summary: Transient failure in KafkaAdminClientTest.testListOffsets Key: KAFKA-9463 URL: https://issues.apache.org/jira/browse/KAFKA-9463 Project: Kafka Issue Type: Test Reporter: Ted Yu When running tests with Java 11, I got the following test failure: {code} java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) at org.apache.kafka.clients.admin.KafkaAdminClientTest.testListOffsets(KafkaAdminClientTest.java:2336) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. {code} KafkaAdminClientTest.testListOffsets passes when it is run alone. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9464) Close the producer in completeShutdown
Ted Yu created KAFKA-9464: - Summary: Close the producer in completeShutdown Key: KAFKA-9464 URL: https://issues.apache.org/jira/browse/KAFKA-9464 Project: Kafka Issue Type: Bug Reporter: Ted Yu In StreamThread#completeShutdown, the producer (if not null) should be closed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9465) Enclose consumer call with catching InvalidOffsetException
Ted Yu created KAFKA-9465: - Summary: Enclose consumer call with catching InvalidOffsetException Key: KAFKA-9465 URL: https://issues.apache.org/jira/browse/KAFKA-9465 Project: Kafka Issue Type: Improvement Reporter: Ted Yu In maybeUpdateStandbyTasks, the try block encloses restoreConsumer.poll and record handling. Since InvalidOffsetException is thrown by restoreConsumer.poll, we should enclose this call in the try block. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9464) Close the producer in completeShutdown
[ https://issues.apache.org/jira/browse/KAFKA-9464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-9464. --- Resolution: Not A Problem > Close the producer in completeShutdown > -- > > Key: KAFKA-9464 > URL: https://issues.apache.org/jira/browse/KAFKA-9464 > Project: Kafka > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > > In StreamThread#completeShutdown, the producer (if not null) should be closed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9471) Return empty collection for PENDING_SHUTDOWN
Ted Yu created KAFKA-9471: - Summary: Return empty collection for PENDING_SHUTDOWN Key: KAFKA-9471 URL: https://issues.apache.org/jira/browse/KAFKA-9471 Project: Kafka Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu In StreamThreadStateStoreProvider we have: {code} if (streamThread.state() == StreamThread.State.DEAD) { return Collections.emptyList(); {code} PENDING_SHUTDOWN should be treated the same way as DEAD. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9471) Throw exception for DEAD StreamThread.State
[ https://issues.apache.org/jira/browse/KAFKA-9471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved KAFKA-9471. --- Resolution: Duplicate > Throw exception for DEAD StreamThread.State > --- > > Key: KAFKA-9471 > URL: https://issues.apache.org/jira/browse/KAFKA-9471 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > > In StreamThreadStateStoreProvider we have: > {code} > if (streamThread.state() == StreamThread.State.DEAD) { > return Collections.emptyList(); > {code} > If user cannot retry anymore, we should throw exception which is handled in > the else block. -- This message was sent by Atlassian Jira (v8.3.4#803005)