[
https://issues.apache.org/jira/browse/HADOOP-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HADOOP-2203:
--------------------------
Attachment: TriplesTest.java
Holger Stenzhorn gave his blessing. Here is the class. If you want the data
file, let me know and I'll arrange it so you can pull it (too big to attach
here).
> [hbase] Hung MR job writing hbase
> ---------------------------------
>
> Key: HADOOP-2203
> URL: https://issues.apache.org/jira/browse/HADOOP-2203
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Reporter: stack
> Priority: Minor
> Attachments: TriplesTest.java
>
>
> Running a simple, fairly large job on the cluster, a few of the reduce steps
> failed; all that did are effectively hung. Looking at failures, I see this
> strange output:
> {code}
> 2007-11-13 02:09:09,606 DEBUG org.apache.hadoop.hbase.HTable: reloading table
> servers because: java.io.IOException: Region
> triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 closed
> at org.apache.hadoop.hbase.HRegion.startUpdate(HRegion.java:1142)
> at
> org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1239)
> at
> org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:19,778 DEBUG org.apache.hadoop.hbase.HTable: reloading table
> servers because: org.apache.hadoop.hbase.NotServingRegionException:
> triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> at
> org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> at
> org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> at
> org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> at
> org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:30,039 DEBUG org.apache.hadoop.hbase.HTable: reloading table
> servers because: org.apache.hadoop.hbase.NotServingRegionException:
> triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> at
> org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> at
> org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> at
> org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> at
> org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:40,275 DEBUG org.apache.hadoop.hbase.HTable: reloading table
> servers because: org.apache.hadoop.hbase.NotServingRegionException:
> triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> at
> org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> at
> org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> at
> org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> at
> org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> 2007-11-13 02:09:55,379 WARN org.apache.hadoop.mapred.TaskTracker: Error
> running child
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException:
> triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610
> at
> org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1325)
> at
> org.apache.hadoop.hbase.HRegionServer.getRegion(HRegionServer.java:1297)
> at
> org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1238)
> at
> org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1113)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> at org.apache.hadoop.hbase.HTable.commit(HTable.java:734)
> at org.apache.hadoop.hbase.HTable.commit(HTable.java:706)
> at
> org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:89)
> at
> org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:63)
> at org.apache.hadoop.mapred.ReduceTask$2.collect(ReduceTask.java:308)
> at TriplesTest$TestReducer.reduce(TriplesTest.java:66)
> at TriplesTest$TestReducer.reduce(TriplesTest.java:51)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:326)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1935)
> {code}
> Why is the reducer client being pig-headed trying to go against a region that
> has been closed? Why not recalibrate?
> Here is master log from around same time on the problem region:
> {code}
> ...
> 2007-11-13 02:10:17,867 INFO org.apache.hadoop.hbase.HMaster: region
> triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 split.
> New regions are:
> triples,http://purl.org/dc/terms/bibliographicCitation,1194919707661,
> triples,http://purl.org/dc/terms/references,1194919707661
> ...
> 2007-11-13 02:11:47,313 DEBUG org.apache.hadoop.hbase.HMaster:
> HMaster.metaScanner scanner: -3075797136896920808 regioninfo: {regionname:
> triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610,
> startKey: <http://purl.org/dc/terms/bibliographicCitation>, offline: true,
> split: true, tableDesc: {name: triples, families: {triples:={name: triples,
> max versions: 3, compression: NONE, in memory: false, max length: 2147483647,
> bloom filter: none}}}}, server: 208.76.44.140:60010, startCode:
> -647590197693256616
> ...
> 2007-11-13 02:11:47,317 INFO org.apache.hadoop.hbase.HMaster: Deleting region
> triples,http://purl.org/dc/terms/bibliographicCitation,1194919500610 because
> daughter splits no longer hold references
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
