[jira] [Commented] (HDFS-8409) HDFS client RPC call throws "java.lang.IllegalStateException"
[ https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814873#comment-16814873 ] Hadoop QA commented on HDFS-8409: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 8s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 32s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 36s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 1s{color} | {color:orange} root: The patch generated 2 new + 139 unchanged - 0 fixed = 141 total (was 139) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 59s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 12s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 42s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 47s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}185m 18s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.hdfs.tools.TestDFSZKFailoverController | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-8409 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12735456/HDFS-8409.003.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 48e13cf7b9c5 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 0e770a6 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | |
[jira] [Commented] (HDFS-8409) HDFS client RPC call throws "java.lang.IllegalStateException"
[ https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814726#comment-16814726 ] Xue Liu commented on HDFS-8409: --- We are also seeing a lot of this in our prod cluster. I think that this exception will be thrown if callId is not reset and then retry happens. Can someone take a further look at this? Thanks! > HDFS client RPC call throws "java.lang.IllegalStateException" > - > > Key: HDFS-8409 > URL: https://issues.apache.org/jira/browse/HDFS-8409 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Juan Yu >Assignee: Juan Yu >Priority: Major > Attachments: HDFS-8409.001.patch, HDFS-8409.002.patch, > HDFS-8409.003.patch > > > When the HDFS client RPC calls need to retry, it sometimes throws > "java.lang.IllegalStateException" and retry is aborted and cause the client > call will fail. > {code} > Caused by: java.lang.IllegalStateException > at > com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99) > at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) > {code} > Here is the check that throws exception > {code} > public static void setCallIdAndRetryCount(int cid, int rc) { > ... > Preconditions.checkState(callId.get() == null); > } > {code} > The RetryInvocationHandler tries to call it with not null callId and causes > exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8409) HDFS client RPC call throws "java.lang.IllegalStateException"
[ https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892062#comment-15892062 ] Yongtao Yang commented on HDFS-8409: I found a clue that may be useful to this problem. DFSClient#namenode is a proxy whose InvocationHandler is RetryInvocationHandler. If namenode.toString() or namenode.hashCode() or some other non-rpc methods are called, then the next real rpc method will failed at setCallIdAndRetryCount(). The cause of this is that toString() is also forwarded to RetryInvocationHandler.invoke() and setCallIdAndRetryCount() is executed, then ipc.Client#callId.get() will not be null any more. For real rpc methods, ipc.Client#callId will be set to null when creating a Call instance(org.apache.hadoop.ipc.Client.Call.Call(RpcKind, Writable)), but for toString() or hashCode(), it can't reach the Call constructor, so ipc.Client#callId will never recover to be null and the next method(whether it is a real rpc or not) will fail. > HDFS client RPC call throws "java.lang.IllegalStateException" > - > > Key: HDFS-8409 > URL: https://issues.apache.org/jira/browse/HDFS-8409 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Juan Yu >Assignee: Juan Yu > Attachments: HDFS-8409.001.patch, HDFS-8409.002.patch, > HDFS-8409.003.patch > > > When the HDFS client RPC calls need to retry, it sometimes throws > "java.lang.IllegalStateException" and retry is aborted and cause the client > call will fail. > {code} > Caused by: java.lang.IllegalStateException > at > com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99) > at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) > {code} > Here is the check that throws exception > {code} > public static void setCallIdAndRetryCount(int cid, int rc) { > ... > Preconditions.checkState(callId.get() == null); > } > {code} > The RetryInvocationHandler tries to call it with not null callId and causes > exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8409) HDFS client RPC call throws "java.lang.IllegalStateException"
[ https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830064#comment-15830064 ] Hadoop QA commented on HDFS-8409: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 12s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 40s{color} | {color:orange} root: The patch generated 2 new + 144 unchanged - 0 fixed = 146 total (was 144) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 38s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 12s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 45s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}158m 38s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | HDFS-8409 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12735456/HDFS-8409.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 9f698a75d354 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 85e4961 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/18211/artifact/patchprocess/diff-checkstyle-root.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/18211/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/18211/testReport/ | |
[jira] [Commented] (HDFS-8409) HDFS client RPC call throws "java.lang.IllegalStateException"
[ https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829812#comment-15829812 ] Zhaofei Meng commented on HDFS-8409: 2017-01-16 06:00:50,236 ERROR metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(155)) - MetaException(message:java.lang.IllegalStateException) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5568) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1503) at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) at com.sun.proxy.$Proxy10.create_table_with_environment_context(Unknown Source) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:9817) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:9801) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:129) at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99) at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1701) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1124) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120) at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:475) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1430) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1489) ... 18 more > HDFS client RPC call throws "java.lang.IllegalStateException" > - > > Key: HDFS-8409 > URL: https://issues.apache.org/jira/browse/HDFS-8409 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Juan Yu >Assignee: Juan Yu > Attachments: HDFS-8409.001.patch, HDFS-8409.002.patch, > HDFS-8409.003.patch > > > When the HDFS client RPC calls need to retry, it sometimes throws > "java.lang.IllegalStateException" and retry is aborted and cause the client > call will fail. > {code} > Caused by: java.lang.IllegalStateException > at > com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99) > at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) > {code} > Here is the check that throws exception > {code} > public static void setCallIdAndRetryCount(int cid, int rc) { > ... >
[jira] [Commented] (HDFS-8409) HDFS client RPC call throws java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568011#comment-14568011 ] Andrew Wang commented on HDFS-8409: --- I guess I'm confused where these exceptions are coming from. This error happens if we call {{setCallIdAndRetryCount}} twice without an intervening creation of a {{Call}}. This means an exception somewhere in {{invokeMethod}} before the {{Call}} is created. I looked at {{ProtobufRpcEngine#invoke}} and I don't see any non-fatal exceptions that would be fixed by a retry. Am I off-base here? Do you have more logging from when this happened? HDFS client RPC call throws java.lang.IllegalStateException - Key: HDFS-8409 URL: https://issues.apache.org/jira/browse/HDFS-8409 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Juan Yu Assignee: Juan Yu Attachments: HDFS-8409.001.patch, HDFS-8409.002.patch, HDFS-8409.003.patch When the HDFS client RPC calls need to retry, it sometimes throws java.lang.IllegalStateException and retry is aborted and cause the client call will fail. {code} Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:129) at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99) at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) {code} Here is the check that throws exception {code} public static void setCallIdAndRetryCount(int cid, int rc) { ... Preconditions.checkState(callId.get() == null); } {code} The RetryInvocationHandler tries to call it with not null callId and causes exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8409) HDFS client RPC call throws java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565649#comment-14565649 ] Juan Yu commented on HDFS-8409: --- If it happens at retry, not the initial call. For example, the initial call gets an exception after call object is created and sent. so it needs retry, but during retry, somehow it gets exception again and this time even before call object (should be same callId as the initial call) is created. In my patch, I added a test to simulate it, does it make sense? HDFS client RPC call throws java.lang.IllegalStateException - Key: HDFS-8409 URL: https://issues.apache.org/jira/browse/HDFS-8409 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Juan Yu Assignee: Juan Yu Attachments: HDFS-8409.001.patch, HDFS-8409.002.patch, HDFS-8409.003.patch When the HDFS client RPC calls need to retry, it sometimes throws java.lang.IllegalStateException and retry is aborted and cause the client call will fail. {code} Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:129) at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99) at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) {code} Here is the check that throws exception {code} public static void setCallIdAndRetryCount(int cid, int rc) { ... Preconditions.checkState(callId.get() == null); } {code} The RetryInvocationHandler tries to call it with not null callId and causes exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8409) HDFS client RPC call throws java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565605#comment-14565605 ] Andrew Wang commented on HDFS-8409: --- Hey Juan, when would an exception before creation of a Call object not be a fatal error? HDFS client RPC call throws java.lang.IllegalStateException - Key: HDFS-8409 URL: https://issues.apache.org/jira/browse/HDFS-8409 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Juan Yu Assignee: Juan Yu Attachments: HDFS-8409.001.patch, HDFS-8409.002.patch, HDFS-8409.003.patch When the HDFS client RPC calls need to retry, it sometimes throws java.lang.IllegalStateException and retry is aborted and cause the client call will fail. {code} Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:129) at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99) at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) {code} Here is the check that throws exception {code} public static void setCallIdAndRetryCount(int cid, int rc) { ... Preconditions.checkState(callId.get() == null); } {code} The RetryInvocationHandler tries to call it with not null callId and causes exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8409) HDFS client RPC call throws java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14560345#comment-14560345 ] Hadoop QA commented on HDFS-8409: - \\ \\ | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 14m 49s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 28s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 3m 22s | There were no new checkstyle issues. | | {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that end in whitespace. | | {color:green}+1{color} | install | 1m 35s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 4m 45s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | common tests | 23m 4s | Tests passed in hadoop-common. | | {color:green}+1{color} | hdfs tests | 162m 40s | Tests passed in hadoop-hdfs. | | | | 228m 21s | | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12735456/HDFS-8409.003.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / cdbd66b | | hadoop-common test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11134/artifact/patchprocess/testrun_hadoop-common.txt | | hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11134/artifact/patchprocess/testrun_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/11134/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11134/console | This message was automatically generated. HDFS client RPC call throws java.lang.IllegalStateException - Key: HDFS-8409 URL: https://issues.apache.org/jira/browse/HDFS-8409 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Juan Yu Assignee: Juan Yu Attachments: HDFS-8409.001.patch, HDFS-8409.002.patch, HDFS-8409.003.patch When the HDFS client RPC calls need to retry, it sometimes throws java.lang.IllegalStateException and retry is aborted and cause the client call will fail. {code} Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:129) at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99) at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) {code} Here is the check that throws exception {code} public static void setCallIdAndRetryCount(int cid, int rc) { ... Preconditions.checkState(callId.get() == null); } {code} The RetryInvocationHandler tries to call it with not null callId and causes exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8409) HDFS client RPC call throws java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557501#comment-14557501 ] Hadoop QA commented on HDFS-8409: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 14m 34s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 31s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 1m 54s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 33s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 4m 43s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | common tests | 22m 39s | Tests passed in hadoop-common. | | {color:red}-1{color} | hdfs tests | 161m 36s | Tests failed in hadoop-hdfs. | | | | 225m 5s | | \\ \\ || Reason || Tests || | Failed unit tests | hadoop.hdfs.server.datanode.TestDataNodeMetrics | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12735023/HDFS-8409.002.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / 446d515 | | whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/5/artifact/patchprocess/whitespace.txt | | hadoop-common test log | https://builds.apache.org/job/PreCommit-HDFS-Build/5/artifact/patchprocess/testrun_hadoop-common.txt | | hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/5/artifact/patchprocess/testrun_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/5/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/5/console | This message was automatically generated. HDFS client RPC call throws java.lang.IllegalStateException - Key: HDFS-8409 URL: https://issues.apache.org/jira/browse/HDFS-8409 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Juan Yu Assignee: Juan Yu Attachments: HDFS-8409.001.patch, HDFS-8409.002.patch When the HDFS client RPC calls need to retry, it sometimes throws java.lang.IllegalStateException and retry is aborted and cause the client call will fail. {code} Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:129) at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99) at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) {code} Here is the check that throws exception {code} public static void setCallIdAndRetryCount(int cid, int rc) { ... Preconditions.checkState(callId.get() == null); } {code} The RetryInvocationHandler tries to call it with not null callId and causes exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8409) HDFS client RPC call throws java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557409#comment-14557409 ] Juan Yu commented on HDFS-8409: --- The test failure is because {code}java.lang.NoClassDefFoundError: org/apache/hadoop/tracing/SpanReceiverHost {code} that's not related with my change. I ran those tests locally and both passed. HDFS client RPC call throws java.lang.IllegalStateException - Key: HDFS-8409 URL: https://issues.apache.org/jira/browse/HDFS-8409 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Juan Yu Assignee: Juan Yu Attachments: HDFS-8409.001.patch When the HDFS client RPC calls need to retry, it sometimes throws java.lang.IllegalStateException and retry is aborted and cause the client call will fail. {code} Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:129) at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99) at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) {code} Here is the check that throws exception {code} public static void setCallIdAndRetryCount(int cid, int rc) { ... Preconditions.checkState(callId.get() == null); } {code} The RetryInvocationHandler tries to call it with not null callId and causes exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8409) HDFS client RPC call throws java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14555802#comment-14555802 ] Hadoop QA commented on HDFS-8409: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 14m 32s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 31s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 3m 18s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 35s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 4m 42s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | common tests | 23m 5s | Tests passed in hadoop-common. | | {color:red}-1{color} | hdfs tests | 163m 56s | Tests failed in hadoop-hdfs. | | | | 229m 17s | | \\ \\ || Reason || Tests || | Failed unit tests | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations | | Timed out tests | org.apache.hadoop.hdfs.server.mover.TestMover | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12734734/HDFS-8409.001.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / cf2b569 | | whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/11097/artifact/patchprocess/whitespace.txt | | hadoop-common test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11097/artifact/patchprocess/testrun_hadoop-common.txt | | hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11097/artifact/patchprocess/testrun_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/11097/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11097/console | This message was automatically generated. HDFS client RPC call throws java.lang.IllegalStateException - Key: HDFS-8409 URL: https://issues.apache.org/jira/browse/HDFS-8409 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Juan Yu Assignee: Juan Yu Attachments: HDFS-8409.001.patch When the HDFS client RPC calls need to retry, it sometimes throws java.lang.IllegalStateException and retry is aborted and cause the client call will fail. {code} Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:129) at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99) at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) {code} Here is the check that throws exception {code} public static void setCallIdAndRetryCount(int cid, int rc) { ... Preconditions.checkState(callId.get() == null); } {code} The RetryInvocationHandler tries to call it with not null callId and causes exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8409) HDFS client RPC call throws java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1441#comment-1441 ] Juan Yu commented on HDFS-8409: --- Client.callId is only reset to null when a new RPC call object created. During retry, we keep the same callId. if exception happened before RPC call object creation, Client.callId won't be reset to null and {code}Preconditions.checkState(callId.get() == null);{code} check will fail and cause RPC retry fail. We should accept not null callId as long as it's the same as cid. HDFS client RPC call throws java.lang.IllegalStateException - Key: HDFS-8409 URL: https://issues.apache.org/jira/browse/HDFS-8409 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Juan Yu Assignee: Juan Yu Attachments: HDFS-8409.001.patch When the HDFS client RPC calls need to retry, it sometimes throws java.lang.IllegalStateException and retry is aborted and cause the client call will fail. {code} Caused by: java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:129) at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99) at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) {code} Here is the check that throws exception {code} public static void setCallIdAndRetryCount(int cid, int rc) { ... Preconditions.checkState(callId.get() == null); } {code} The RetryInvocationHandler tries to call it with not null callId and causes exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)