[jira] [Commented] (HDDS-1373) KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException
[ https://issues.apache.org/jira/browse/HDDS-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819900#comment-16819900 ] Hudson commented on HDDS-1373: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16431 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16431/]) HDDS-1373. KeyOutputStream, close after write request fails after (shashikant: rev d608be660f1c7a3a60787f5ea5d38fe783f0b6b1) * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ContainerTestHelper.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailures.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStream.java * (edit) hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/common/helpers/ExcludeList.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneClientRetriesOnException.java > KeyOutputStream, close after write request fails after retries, runs into > IllegalArgumentException > -- > > Key: HDDS-1373 > URL: https://issues.apache.org/jira/browse/HDDS-1373 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0, 0.5.0 >Reporter: Mukul Kumar Singh >Assignee: Shashikant Banerjee >Priority: Major > Labels: MiniOzoneChaosCluster, pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1373.000.patch, HDDS-1373.001.patch > > Time Spent: 40m > Remaining Estimate: 0h > > In this code, the stream is closed via try with resource. > {code} > try (OzoneOutputStream stream = ozoneBucket.createKey(keyName, > bufferCapacity, ReplicationType.RATIS, ReplicationFactor.THREE, > new HashMap<>())) { > stream.write(buffer.array()); > } catch (Exception e) { > LOG.error("LOADGEN: Create key:{} failed with exception", keyName, e); > break; > } > {code} > Here, the write call fails correctly as expected, However the close doesn't > fail with the same exception. > The exception stack stack is as following > {code} > 2019-04-03 00:52:54,116 ERROR ozone.MiniOzoneLoadGenerator > (MiniOzoneLoadGenerator.java:load(101)) - LOADGEN: Create > key:pool-431-thread-9-8126 failed with exception > java.io.IOException: Retry request failed. retries get failed due to exceeded > maximum allowed retries number: 5 > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:492) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:287) > at > org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49) > at java.i
[jira] [Commented] (HDDS-1373) KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException
[ https://issues.apache.org/jira/browse/HDDS-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819887#comment-16819887 ] Hudson commented on HDDS-1373: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16430 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16430/]) Revert "HDDS-1373. KeyOutputStream, close after write request fails (shashikant: rev 082f1e04376f72487a1c90b3ddf8148b566c6e08) * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ContainerTestHelper.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStream.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/common/helpers/ExcludeList.java * (edit) hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java * (delete) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneClientRetriesOnException.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailures.java > KeyOutputStream, close after write request fails after retries, runs into > IllegalArgumentException > -- > > Key: HDDS-1373 > URL: https://issues.apache.org/jira/browse/HDDS-1373 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0, 0.5.0 >Reporter: Mukul Kumar Singh >Assignee: Shashikant Banerjee >Priority: Major > Labels: MiniOzoneChaosCluster, pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1373.000.patch, HDDS-1373.001.patch > > Time Spent: 40m > Remaining Estimate: 0h > > In this code, the stream is closed via try with resource. > {code} > try (OzoneOutputStream stream = ozoneBucket.createKey(keyName, > bufferCapacity, ReplicationType.RATIS, ReplicationFactor.THREE, > new HashMap<>())) { > stream.write(buffer.array()); > } catch (Exception e) { > LOG.error("LOADGEN: Create key:{} failed with exception", keyName, e); > break; > } > {code} > Here, the write call fails correctly as expected, However the close doesn't > fail with the same exception. > The exception stack stack is as following > {code} > 2019-04-03 00:52:54,116 ERROR ozone.MiniOzoneLoadGenerator > (MiniOzoneLoadGenerator.java:load(101)) - LOADGEN: Create > key:pool-431-thread-9-8126 failed with exception > java.io.IOException: Retry request failed. retries get failed due to exceeded > maximum allowed retries number: 5 > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:492) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:287) >
[jira] [Commented] (HDDS-1373) KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException
[ https://issues.apache.org/jira/browse/HDDS-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819868#comment-16819868 ] Hudson commented on HDDS-1373: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16429 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16429/]) Revert "Merge pull request #729 from bshashikant/HDDS-1373" (shashikant: rev e5a918cda3e22ec5d49be2003e9b273ffc2dfd45) * (edit) hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailures.java * (delete) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/common/helpers/ExcludeList.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStream.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ContainerTestHelper.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneClientRetriesOnException.java HDDS-1373. KeyOutputStream, close after write request fails after (shashikant: rev df2ae27f3e40cb4cedda9c7bed1a464be60fd339) * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailures.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneClientRetriesOnException.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/common/helpers/ExcludeList.java * (add) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ContainerTestHelper.java * (edit) hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStream.java > KeyOutputStream, close after write request fails after retries, runs into > IllegalArgumentException > -- > > Key: HDDS-1373 > URL: https://issues.apache.org/jira/browse/HDDS-1373 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0, 0.5.0 >Reporter: Mukul Kumar Singh >Assignee: Shashikant Banerjee >Priority: Major > Labels: MiniOzoneChaosCluster, pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1373.000.patch, HDDS-1373.001.patch > > Time Spent: 40m > Remaining Estimate: 0h > > In this code, the stream is closed via try with resource. > {code} > try (OzoneOutputStream stream = ozoneBucket.createKey(keyName, > bufferCapacity, ReplicationType.RATIS, ReplicationFactor.THREE, > new HashMap<>())) { > stream.write(buffer.array()); > } catch (Exception e) { > LOG.error("LOADGEN: Create key:{} failed with exception", keyName, e); > break; > } > {code} > Here, the write call fails correctly as expected, However the close doesn't > fail with the same exception. > The exception stack stack is as following > {code} > 2019-04-03 00:52:54,116 ERROR ozone.MiniOzoneLoadGenerator > (MiniOzoneLoadGenerator.java:load(101)) - LOADGEN: Create > key:pool-431-thread-9-8126 failed with exception > java.io.IOException: Retry request failed. retries get failed due to exceeded > maximum allowed retries number: 5 > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:492) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org
[jira] [Commented] (HDDS-1373) KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException
[ https://issues.apache.org/jira/browse/HDDS-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819867#comment-16819867 ] Hudson commented on HDDS-1373: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16428 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16428/]) HDDS-1373. KeyOutputStream, close after write request fails after (shashikant: rev 22d0468b39b138d37de252ac572aa4259fc25224) * (add) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailures.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStream.java * (edit) hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ContainerTestHelper.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/common/helpers/ExcludeList.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneClientRetriesOnException.java > KeyOutputStream, close after write request fails after retries, runs into > IllegalArgumentException > -- > > Key: HDDS-1373 > URL: https://issues.apache.org/jira/browse/HDDS-1373 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0, 0.5.0 >Reporter: Mukul Kumar Singh >Assignee: Shashikant Banerjee >Priority: Major > Labels: MiniOzoneChaosCluster, pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1373.000.patch, HDDS-1373.001.patch > > Time Spent: 40m > Remaining Estimate: 0h > > In this code, the stream is closed via try with resource. > {code} > try (OzoneOutputStream stream = ozoneBucket.createKey(keyName, > bufferCapacity, ReplicationType.RATIS, ReplicationFactor.THREE, > new HashMap<>())) { > stream.write(buffer.array()); > } catch (Exception e) { > LOG.error("LOADGEN: Create key:{} failed with exception", keyName, e); > break; > } > {code} > Here, the write call fails correctly as expected, However the close doesn't > fail with the same exception. > The exception stack stack is as following > {code} > 2019-04-03 00:52:54,116 ERROR ozone.MiniOzoneLoadGenerator > (MiniOzoneLoadGenerator.java:load(101)) - LOADGEN: Create > key:pool-431-thread-9-8126 failed with exception > java.io.IOException: Retry request failed. retries get failed due to exceeded > maximum allowed retries number: 5 > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:492) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:287) >
[jira] [Commented] (HDDS-1373) KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException
[ https://issues.apache.org/jira/browse/HDDS-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16814632#comment-16814632 ] Mukul Kumar Singh commented on HDDS-1373: - Thanks for working on this [~shashikant]. The patch looks really good to me. I feel that this check should go into the caller functions i.e. handleWrite & handleFlushOrClose calls. As the exceptions will eventually roll up to these places. > KeyOutputStream, close after write request fails after retries, runs into > IllegalArgumentException > -- > > Key: HDDS-1373 > URL: https://issues.apache.org/jira/browse/HDDS-1373 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0, 0.5.0 >Reporter: Mukul Kumar Singh >Assignee: Shashikant Banerjee >Priority: Major > Labels: MiniOzoneChaosCluster > Attachments: HDDS-1373.000.patch, HDDS-1373.001.patch > > > In this code, the stream is closed via try with resource. > {code} > try (OzoneOutputStream stream = ozoneBucket.createKey(keyName, > bufferCapacity, ReplicationType.RATIS, ReplicationFactor.THREE, > new HashMap<>())) { > stream.write(buffer.array()); > } catch (Exception e) { > LOG.error("LOADGEN: Create key:{} failed with exception", keyName, e); > break; > } > {code} > Here, the write call fails correctly as expected, However the close doesn't > fail with the same exception. > The exception stack stack is as following > {code} > 2019-04-03 00:52:54,116 ERROR ozone.MiniOzoneLoadGenerator > (MiniOzoneLoadGenerator.java:load(101)) - LOADGEN: Create > key:pool-431-thread-9-8126 failed with exception > java.io.IOException: Retry request failed. retries get failed due to exceeded > maximum allowed retries number: 5 > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:492) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:287) > at > org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49) > at java.io.OutputStream.write(OutputStream.java:75) > at > org.apache.hadoop.ozone.MiniOzoneLoadGenerator.load(MiniOzoneLoadGenerator.java:99) > at > org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$0(MiniOzoneLoadGenerator.java:137) > at > java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Suppressed: java.lang.IllegalArgumentException > at > com.google.common.base.Preconditions.checkArgument(Preconditions.java:72) > at > org.apache.hadoop.oz
[jira] [Commented] (HDDS-1373) KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException
[ https://issues.apache.org/jira/browse/HDDS-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808646#comment-16808646 ] Hadoop QA commented on HDDS-1373: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 10s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m 30s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 22s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 11s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 27s{color} | {color:orange} hadoop-ozone: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 20s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 37s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 12s{color} | {color:red} hadoop-hdds in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 58s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 38s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 80m 49s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.om.TestOzoneManager | | | hadoop.ozone.TestMiniChaosOzoneCluster | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/PreCommit-HDDS-Build/2630/artifact/out/Dockerfile | | JIRA Issue | HDDS-1373 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12964686/HDDS-1373.001.patch | | Optio
[jira] [Commented] (HDDS-1373) KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException
[ https://issues.apache.org/jira/browse/HDDS-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808585#comment-16808585 ] Shashikant Banerjee commented on HDDS-1373: --- Patch v1 fixes the unit tests introduced. > KeyOutputStream, close after write request fails after retries, runs into > IllegalArgumentException > -- > > Key: HDDS-1373 > URL: https://issues.apache.org/jira/browse/HDDS-1373 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0, 0.5.0 >Reporter: Mukul Kumar Singh >Assignee: Shashikant Banerjee >Priority: Major > Labels: MiniOzoneChaosCluster > Attachments: HDDS-1373.000.patch, HDDS-1373.001.patch > > > In this code, the stream is closed via try with resource. > {code} > try (OzoneOutputStream stream = ozoneBucket.createKey(keyName, > bufferCapacity, ReplicationType.RATIS, ReplicationFactor.THREE, > new HashMap<>())) { > stream.write(buffer.array()); > } catch (Exception e) { > LOG.error("LOADGEN: Create key:{} failed with exception", keyName, e); > break; > } > {code} > Here, the write call fails correctly as expected, However the close doesn't > fail with the same exception. > The exception stack stack is as following > {code} > 2019-04-03 00:52:54,116 ERROR ozone.MiniOzoneLoadGenerator > (MiniOzoneLoadGenerator.java:load(101)) - LOADGEN: Create > key:pool-431-thread-9-8126 failed with exception > java.io.IOException: Retry request failed. retries get failed due to exceeded > maximum allowed retries number: 5 > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:492) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:514) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:468) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:344) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:287) > at > org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49) > at java.io.OutputStream.write(OutputStream.java:75) > at > org.apache.hadoop.ozone.MiniOzoneLoadGenerator.load(MiniOzoneLoadGenerator.java:99) > at > org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$0(MiniOzoneLoadGenerator.java:137) > at > java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Suppressed: java.lang.IllegalArgumentException > at > com.google.common.base.Preconditions.checkArgument(Preconditions.java:72) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:643) > at > org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:60) > at > org
[jira] [Commented] (HDDS-1373) KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException
[ https://issues.apache.org/jira/browse/HDDS-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808239#comment-16808239 ] Hadoop QA commented on HDDS-1373: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 45s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 58s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 13s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 4m 22s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 51s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 18s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 26s{color} | {color:orange} hadoop-ozone: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 53s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 45s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 55s{color} | {color:green} hadoop-hdds in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 47s{color} | {color:red} hadoop-ozone in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 33s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 95m 34s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ozone.container.TestContainerReplication | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/PreCommit-HDDS-Build/2626/artifact/out/Dockerfile | | JIRA Issue | HDDS-1373 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12964622/HDDS-1373.000.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient