[jira] [Commented] (HBASE-12974) Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no detail
[
https://issues.apache.org/jira/browse/HBASE-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310279#comment-14310279
]
stack commented on HBASE-12974:
---
Thanks [~nkeywal] It was an odd case that provoked this. Let me see if I can
improve it... (Yeah, I see how after ten times we get more detail... nice)
Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no
detail
---
Key: HBASE-12974
URL: https://issues.apache.org/jira/browse/HBASE-12974
Project: HBase
Issue Type: Sub-task
Components: integration tests
Affects Versions: 1.0.0
Reporter: stack
Assignee: stack
I'm trying to do longer running tests but when I up the numbers for a task I
run into this:
{code}
2015-02-04 15:35:10,267 FATAL [IPC Server handler 17 on 43975]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1419986015214_0204_m_02_3 - exited :
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: IOException: 1 time,
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227)
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207)
at
org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1658)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:141)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:98)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.persist(IntegrationTestBigLinkedList.java:449)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:407)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:355)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
{code}
Its telling me an action failed but 1 time only with an empty IOE?
I'm kinda stumped.
Starting up this issue to see if I can get to the bottom of it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12974) Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no detail
[
https://issues.apache.org/jira/browse/HBASE-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307249#comment-14307249
]
Nicolas Liochon commented on HBASE-12974:
-
bq. 1 time only
We don't keep the history of the exceptions. the time is only about the last
exception. So if you have 1 action that failed you will have 1 time. If 10
actions fail for the same reason you will have '10 times'. Yes it's kind of
useless. We used to start to log after 10 retries or so, so the log should
contain more information (at the info level iirc).
Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no
detail
---
Key: HBASE-12974
URL: https://issues.apache.org/jira/browse/HBASE-12974
Project: HBase
Issue Type: Bug
Components: integration tests
Affects Versions: 1.0.0
Reporter: stack
Assignee: stack
I'm trying to do longer running tests but when I up the numbers for a task I
run into this:
{code}
2015-02-04 15:35:10,267 FATAL [IPC Server handler 17 on 43975]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1419986015214_0204_m_02_3 - exited :
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: IOException: 1 time,
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227)
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207)
at
org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1658)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:141)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:98)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.persist(IntegrationTestBigLinkedList.java:449)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:407)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:355)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
{code}
Its telling me an action failed but 1 time only with an empty IOE?
I'm kinda stumped.
Starting up this issue to see if I can get to the bottom of it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12974) Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no detail
[
https://issues.apache.org/jira/browse/HBASE-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307976#comment-14307976
]
stack commented on HBASE-12974:
---
Making this a subtask of HBASE-12978. Related it seems. The opaque shows up
first and then I see the missing row which may be the root cause (speculation).
Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no
detail
---
Key: HBASE-12974
URL: https://issues.apache.org/jira/browse/HBASE-12974
Project: HBase
Issue Type: Sub-task
Components: integration tests
Affects Versions: 1.0.0
Reporter: stack
Assignee: stack
I'm trying to do longer running tests but when I up the numbers for a task I
run into this:
{code}
2015-02-04 15:35:10,267 FATAL [IPC Server handler 17 on 43975]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1419986015214_0204_m_02_3 - exited :
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: IOException: 1 time,
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227)
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207)
at
org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1658)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:141)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:98)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.persist(IntegrationTestBigLinkedList.java:449)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:407)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:355)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
{code}
Its telling me an action failed but 1 time only with an empty IOE?
I'm kinda stumped.
Starting up this issue to see if I can get to the bottom of it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12974) Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no detail
[
https://issues.apache.org/jira/browse/HBASE-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306754#comment-14306754
]
stack commented on HBASE-12974:
---
Happened again for me again:
${HOME}/hadoop/bin/hadoop --config ${HOME}/conf_hadoop
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList --monkey
serverKilling Generator 5 4 g.$date
If rows are 2 per task, I seem to get it. Wish I could get more
detail out of it. Let me try and add some debug.
Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no
detail
---
Key: HBASE-12974
URL: https://issues.apache.org/jira/browse/HBASE-12974
Project: HBase
Issue Type: Bug
Components: integration tests
Affects Versions: 1.0.0
Reporter: stack
Assignee: stack
I'm trying to do longer running tests but when I up the numbers for a task I
run into this:
{code}
2015-02-04 15:35:10,267 FATAL [IPC Server handler 17 on 43975]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1419986015214_0204_m_02_3 - exited :
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: IOException: 1 time,
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227)
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207)
at
org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1658)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:141)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:98)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.persist(IntegrationTestBigLinkedList.java:449)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:407)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.map(IntegrationTestBigLinkedList.java:355)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
{code}
Its telling me an action failed but 1 time only with an empty IOE?
I'm kinda stumped.
Starting up this issue to see if I can get to the bottom of it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
