[
https://issues.apache.org/jira/browse/HBASE-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652658#action_12652658
]
Andrew Purtell commented on HBASE-1038:
---------------------------------------
I have right now a hung mapred task (running on 0.18.1 Hadoop and 0.18.1 HBase)
that keeps going back to the same regionserver, now sick after OOME, throwing
NSREs but not reporting itself as sick or dead to the master.
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server 10.30.94.51:60020 for region
content,630574f07070d338cfa3c49ad1665cc3,1228236720128, row
'6306dc97a60102228cc63a3ecf134f45', but failed after 10 attempts.
Exceptions:
java.io.IOException: Call failed on local exception
java.io.IOException: Call failed on local exception
java.io.IOException: Call failed on local exception
java.io.IOException: Call failed on local exception
java.io.IOException: Call failed on local exception
java.io.IOException: Call failed on local exception
java.io.IOException: Call failed on local exception
java.io.IOException: Call failed on local exception
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException:
content,630574f07070d338cfa3c49ad1665cc3,1228236720128
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException:
content,630574f07070d338cfa3c49ad1665cc3,1228236720128
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1524)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1110)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:863)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:964)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:950)
at
org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:72)
at
org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:53)
at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:300)
at wtp.back.exec.tasks.DocumentParseReduce.reduce(Unknown Source)
at wtp.back.exec.tasks.DocumentParseReduce.reduce(Unknown Source)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> Master should accept client feedback about inconsistencies in META
> ------------------------------------------------------------------
>
> Key: HBASE-1038
> URL: https://issues.apache.org/jira/browse/HBASE-1038
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: client, master
> Reporter: Andrew Purtell
> Fix For: 0.20.0
>
>
> Jim Firby: Clients should be able to say "I've asked for a regions location
> 10 times now and Mr. Master, you've given me the same response ten times in a
> row and each time, the answer was wrong. Revisit any notion that said region
> is at said location". Mr. Master would then go off and do something drastic
> like close and reassign the region.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
