smengcl commented on PR #6413:
URL: https://github.com/apache/ozone/pull/6413#issuecomment-2048481034
fyi with the latest commit as of a few seconds ago there are still two known
issues when running `testKeyLargerThan2GB` with `--key-size` set to 5 MB (at
least 2 chunk buffers allocated). The `testKeyLargerThan2GB` sometimes passes
and sometimes not (flaky).
1. `BlockOutputStream has been closed` thrown from `handleFlushInternal`.
Regardless of whether `testKeyLargerThan2GB` has passed or not:
```log
2024-04-10 14:31:34,838 [pool-234-thread-1] WARN io.KeyOutputStream
(KeyOutputStream.java:handleFlushOrClose(571)) - !!! handleFlushOrClose(op =
FULL)
2024-04-10 14:31:34,840 [pool-234-thread-1] WARN storage.BlockOutputStream
(BlockOutputStream.java:handleFlushInternal(659)) - handleFlushInternal(close =
true)
2024-04-10 14:31:34,852 [client-write-TID-1] ERROR storage.BlockOutputStream
(BlockOutputStream.java:lambda$handleFlushInternal$4(702)) - IOException caught
but ignored in this POC
java.io.IOException: BlockOutputStream has been closed.
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.checkOpen(BlockOutputStream.java:791)
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.watchForCommit(BlockOutputStream.java:498)
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.lambda$handleFlushInternal$4(BlockOutputStream.java:693)
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
2024-04-10 14:31:34,852 [pool-234-thread-1] ERROR storage.BlockOutputStream
(BlockOutputStream.java:close(717)) - InterruptedException or
ExecutionException caught but ignored in this POC
java.util.concurrent.ExecutionException: java.io.IOException:
BlockOutputStream has been closed.
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.close(BlockOutputStream.java:714)
at
org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.close(BlockOutputStreamEntry.java:174)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.handleStreamAction(KeyOutputStream.java:630)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.handleFlushOrClose(KeyOutputStream.java:580)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:284)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:242)
at
org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:94)
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:823)
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1200(RandomKeyGenerator.java:90)
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:717)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: BlockOutputStream has been closed.
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.checkOpen(BlockOutputStream.java:791)
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.watchForCommit(BlockOutputStream.java:498)
at
org.apache.hadoop.hdds.scm.storage.BlockOutputStream.lambda$handleFlushInternal$4(BlockOutputStream.java:693)
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
... 3 more
2024-04-10 14:31:34,854 [pool-234-thread-1] WARN storage.BufferPool
(BufferPool.java:allocateBuffer(94)) - !! allocateBuffer(increment = 0):
capacity = 2, currentBufferIndex = 1, nextBufferIndex = 1, bufferList.size() = 2
2024-04-10 14:31:34,913 [client-write-TID-0] WARN storage.BufferPool
(BufferPool.java:releaseBuffer(110)) - !! releaseBuffer(chunkBuffer =
ChunkBufferImplWithByteBuffer:limit=1048576@1):currentBufferIndex = 1,
bufferList.indexOf(chunkBuffer) = 0
```
2. When `testKeyLargerThan2GB` fails, it still indicates some "buffer not
released" issue:
```log
2024-04-10 14:31:34,959 [pool-234-thread-1] ERROR freon.RandomKeyGenerator
(RandomKeyGenerator.java:createKey(848)) - Exception while adding key:
key-0-10754 in bucket: bucket-0-45857 of volume: vol-0-11123.
java.lang.IllegalStateException: capacity: expected == 0 but computed ==
1048576
at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:76)
at org.apache.ratis.util.Preconditions.assertSame(Preconditions.java:81)
at
org.apache.hadoop.hdds.utils.db.CodecBuffer.release(CodecBuffer.java:299)
at
org.apache.hadoop.hdds.utils.db.CodecBuffer.close(CodecBuffer.java:291)
at
org.apache.hadoop.ozone.common.ChunkBufferImplWithByteBuffer.close(ChunkBufferImplWithByteBuffer.java:50)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at
org.apache.hadoop.hdds.scm.storage.BufferPool.clearBufferPool(BufferPool.java:130)
at
org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.cleanup(BlockOutputStreamEntryPool.java:407)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.markStreamClosed(KeyOutputStream.java:424)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:289)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:242)
at
org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:94)
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:823)
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1200(RandomKeyGenerator.java:90)
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:717)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Suppressed: java.lang.IllegalStateException: capacity: expected == 0
but computed == 1048576
at
org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:76)
at
org.apache.ratis.util.Preconditions.assertSame(Preconditions.java:81)
at
org.apache.hadoop.hdds.utils.db.CodecBuffer.release(CodecBuffer.java:299)
at
org.apache.hadoop.hdds.utils.db.CodecBuffer.close(CodecBuffer.java:291)
at
org.apache.hadoop.ozone.common.ChunkBufferImplWithByteBuffer.close(ChunkBufferImplWithByteBuffer.java:50)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at
org.apache.hadoop.hdds.scm.storage.BufferPool.clearBufferPool(BufferPool.java:130)
at
org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.cleanup(BlockOutputStreamEntryPool.java:407)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:685)
at
org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:105)
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:833)
... 5 more
2024-04-10 14:31:35,400 [Thread-649] INFO freon.ProgressBar
(ProgressBar.java:logProgressBar(168)) - Progress: 0.00 % (0 out of 1)
2024-04-10 14:31:36,412 [Thread-649] INFO freon.ProgressBar
(ProgressBar.java:logProgressBar(168)) - Progress: 0.00 % (0 out of 1)
2024-04-10 14:31:37,418 [Thread-649] INFO freon.ProgressBar
(ProgressBar.java:logProgressBar(168)) - Progress: 0.00 % (0 out of 1)
2024-04-10 14:31:38,423 [Thread-649] INFO freon.ProgressBar
(ProgressBar.java:logProgressBar(168)) - Progress: 0.00 % (0 out of 1)
2024-04-10 14:31:39,456 [Thread-649] INFO freon.ProgressBar
(ProgressBar.java:logProgressBar(168)) - Progress: 0.00 % (0 out of 1)
2024-04-10 14:31:40,461 [Thread-649] INFO freon.ProgressBar
(ProgressBar.java:logProgressBar(168)) - Progress: 0.00 % (0 out of 1)
java.lang.RuntimeException: java.lang.IllegalStateException: capacity:
expected == 0 but computed == 1048576
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator.call(RandomKeyGenerator.java:391)
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator.call(RandomKeyGenerator.java:83)
at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at
picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
at
picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
at picocli.CommandLine.execute(CommandLine.java:2170)
at
org.apache.hadoop.ozone.freon.TestRandomKeyGenerator.testKeyLargerThan2GB(TestRandomKeyGenerator.java:164)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at
org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at
org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:198)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:169)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:93)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:58)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:141)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:57)
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:103)
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:85)
at
org.junit.platform.launcher.core.DelegatingLauncher.execute(DelegatingLauncher.java:47)
at
org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:63)
at
com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at
com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.IllegalStateException: capacity: expected == 0 but
computed == 1048576
at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:76)
at org.apache.ratis.util.Preconditions.assertSame(Preconditions.java:81)
at
org.apache.hadoop.hdds.utils.db.CodecBuffer.release(CodecBuffer.java:299)
at
org.apache.hadoop.hdds.utils.db.CodecBuffer.close(CodecBuffer.java:291)
at
org.apache.hadoop.ozone.common.ChunkBufferImplWithByteBuffer.close(ChunkBufferImplWithByteBuffer.java:50)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at
org.apache.hadoop.hdds.scm.storage.BufferPool.clearBufferPool(BufferPool.java:130)
at
org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.cleanup(BlockOutputStreamEntryPool.java:407)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.markStreamClosed(KeyOutputStream.java:424)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:289)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:242)
at
org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:94)
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:823)
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1200(RandomKeyGenerator.java:90)
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:717)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Suppressed: java.lang.IllegalStateException: capacity: expected == 0
but computed == 1048576
at
org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:76)
at
org.apache.ratis.util.Preconditions.assertSame(Preconditions.java:81)
at
org.apache.hadoop.hdds.utils.db.CodecBuffer.release(CodecBuffer.java:299)
at
org.apache.hadoop.hdds.utils.db.CodecBuffer.close(CodecBuffer.java:291)
at
org.apache.hadoop.ozone.common.ChunkBufferImplWithByteBuffer.close(ChunkBufferImplWithByteBuffer.java:50)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at
org.apache.hadoop.hdds.scm.storage.BufferPool.clearBufferPool(BufferPool.java:130)
at
org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.cleanup(BlockOutputStreamEntryPool.java:407)
at
org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:685)
at
org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:105)
at
org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:833)
... 5 more
org.opentest4j.AssertionFailedError:
Expected :1
Actual :0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
