[jira] [Commented] (HBASE-12974) Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no detail
[ https://issues.apache.org/jira/browse/HBASE-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310279#comment-14310279 ] stack commented on HBASE-12974: --- Thanks [~nkeywal] It was an odd case that provoked this. Let me see if I can improve it... (Yeah, I see how after ten times we get more detail... nice) > Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no > detail > --- > > Key: HBASE-12974 > URL: https://issues.apache.org/jira/browse/HBASE-12974 > Project: HBase > Issue Type: Sub-task > Components: integration tests >Affects Versions: 1.0.0 >Reporter: stack >Assignee: stack > > I'm trying to do longer running tests but when I up the numbers for a task I > run into this: > {code} > 2015-02-04 15:35:10,267 FATAL [IPC Server handler 17 on 43975] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: > attempt_1419986015214_0204_m_02_3 - exited : > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 > action: IOException: 1 time, > at > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227) > at > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207) > at > org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1658) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:141) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:98) > at > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.persist(IntegrationTestBigLinkedList.java:449) > at > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:407) > at > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:355) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > {code} > Its telling me an action failed but 1 time only with an empty IOE? > I'm kinda stumped. > Starting up this issue to see if I can get to the bottom of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12974) Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no detail
[ https://issues.apache.org/jira/browse/HBASE-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307976#comment-14307976 ] stack commented on HBASE-12974: --- Making this a subtask of HBASE-12978. Related it seems. The opaque shows up first and then I see the missing row which may be the root cause (speculation). > Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no > detail > --- > > Key: HBASE-12974 > URL: https://issues.apache.org/jira/browse/HBASE-12974 > Project: HBase > Issue Type: Sub-task > Components: integration tests >Affects Versions: 1.0.0 >Reporter: stack >Assignee: stack > > I'm trying to do longer running tests but when I up the numbers for a task I > run into this: > {code} > 2015-02-04 15:35:10,267 FATAL [IPC Server handler 17 on 43975] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: > attempt_1419986015214_0204_m_02_3 - exited : > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 > action: IOException: 1 time, > at > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227) > at > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207) > at > org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1658) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:141) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:98) > at > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.persist(IntegrationTestBigLinkedList.java:449) > at > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:407) > at > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:355) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > {code} > Its telling me an action failed but 1 time only with an empty IOE? > I'm kinda stumped. > Starting up this issue to see if I can get to the bottom of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12974) Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no detail
[ https://issues.apache.org/jira/browse/HBASE-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307249#comment-14307249 ] Nicolas Liochon commented on HBASE-12974: - bq. 1 time only We don't keep the history of the exceptions. the "time" is only about the last exception. So if you have 1 action that failed you will have "1 time". If 10 actions fail for the same reason you will have '10 times'. Yes it's kind of useless. We used to start to log after 10 retries or so, so the log should contain more information (at the info level iirc). > Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no > detail > --- > > Key: HBASE-12974 > URL: https://issues.apache.org/jira/browse/HBASE-12974 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.0.0 >Reporter: stack >Assignee: stack > > I'm trying to do longer running tests but when I up the numbers for a task I > run into this: > {code} > 2015-02-04 15:35:10,267 FATAL [IPC Server handler 17 on 43975] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: > attempt_1419986015214_0204_m_02_3 - exited : > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 > action: IOException: 1 time, > at > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227) > at > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207) > at > org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1658) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:141) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:98) > at > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.persist(IntegrationTestBigLinkedList.java:449) > at > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:407) > at > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:355) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > {code} > Its telling me an action failed but 1 time only with an empty IOE? > I'm kinda stumped. > Starting up this issue to see if I can get to the bottom of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12974) Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no detail
[ https://issues.apache.org/jira/browse/HBASE-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306754#comment-14306754 ] stack commented on HBASE-12974: --- Happened again for me again: ${HOME}/hadoop/bin/hadoop --config ${HOME}/conf_hadoop org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList --monkey serverKilling Generator 5 4 g.$date If rows are > 2 per task, I seem to get it. Wish I could get more detail out of it. Let me try and add some debug. > Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no > detail > --- > > Key: HBASE-12974 > URL: https://issues.apache.org/jira/browse/HBASE-12974 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.0.0 >Reporter: stack >Assignee: stack > > I'm trying to do longer running tests but when I up the numbers for a task I > run into this: > {code} > 2015-02-04 15:35:10,267 FATAL [IPC Server handler 17 on 43975] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: > attempt_1419986015214_0204_m_02_3 - exited : > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 > action: IOException: 1 time, > at > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227) > at > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207) > at > org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1658) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:141) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:98) > at > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.persist(IntegrationTestBigLinkedList.java:449) > at > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:407) > at > org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:355) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > {code} > Its telling me an action failed but 1 time only with an empty IOE? > I'm kinda stumped. > Starting up this issue to see if I can get to the bottom of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)