[
https://issues.apache.org/jira/browse/HADOOP-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559294#action_12559294
]
viper799 edited comment on HADOOP-2619 at 1/15/08 4:05 PM:
----------------------------------------------------------------
I tryed to kill the region server that the region was on and it fail over to a
different server fine but compaction failed on that server too same error as
above.
but when I tryed to stop the whole cluster stop-hbase.sh and stop-all.sh I get
this error on the new regionserver for this region
{code}
2008-01-15 17:47:30,611 WARN org.apache.hadoop.fs.DFSClient: Problem renewing
lease for DFSClient_647476091: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:159)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:575)
at org.apache.hadoop.ipc.Client.call(Client.java:498)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
at org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:586)
at java.lang.Thread.run(Thread.java:595)
{code}
I have to kill the region server my self with kill pid
then on restart of the cluster I get this meta error reprted many times over
and over on the region server serveing .META.
{code}
2008-01-15 17:55:09,315 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0
on 60020, call openScanner(.META.,,1, [Lorg.apache.hadoop.io.Text;@167e3a5, ,
1200441309302, null) from 10.0.0.1:49221: error: java.io.IOException:
HStoreScanner failed construction
java.io.IOException: HStoreScanner failed construction
at
org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(HStore.java:2139)
at org.apache.hadoop.hbase.HStore$HStoreScanner.<init>(HStore.java:2260)
at org.apache.hadoop.hbase.HStore.getScanner(HStore.java:2050)
at org.apache.hadoop.hbase.HRegion$HScanner.<init>(HRegion.java:1593)
at org.apache.hadoop.hbase.HRegion.getScanner(HRegion.java:1167)
at
org.apache.hadoop.hbase.HRegionServer.openScanner(HRegionServer.java:1517)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:908)
Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
does not exist:
/gfs_storage/hadoop-root/hbase/.META./1028785192/info/mapfiles/5207253989210860351/data
at org.apache.hadoop.dfs.FSDirectory.getFileInfo(FSDirectory.java:543)
at
org.apache.hadoop.dfs.FSNamesystem.getFileInfo(FSNamesystem.java:1464)
at org.apache.hadoop.dfs.NameNode.getFileInfo(NameNode.java:439)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:409)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:908)
at org.apache.hadoop.ipc.Client.call(Client.java:509)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:442)
at
org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:331)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:536)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1349)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1344)
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:254)
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:242)
at
org.apache.hadoop.hbase.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:595)
at
org.apache.hadoop.hbase.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:650)
at org.apache.hadoop.hbase.HStoreFile.getReader(HStoreFile.java:424)
at
org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(HStore.java:2114)
... 11 more
{code}
was (Author: viper799):
I tryed to kill the region server that the region was on and it fail over
to a different server fine but compaction failed on that server too same error
as above.
but when I tryed to stop the whole cluster stop-hbase.sh and stop-all.sh I get
this error on the new regionserver for this region
{code}
2008-01-15 17:47:30,611 WARN org.apache.hadoop.fs.DFSClient: Problem renewing
lease for DFSClient_647476091: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:159)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:575)
at org.apache.hadoop.ipc.Client.call(Client.java:498)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy1.renewLease(Unknown Source)
at org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:586)
at java.lang.Thread.run(Thread.java:595)
{code}
then on restart of the cluster I get this meta error reprted many times over
and over on the region server serveing .META.
{code}
2008-01-15 17:55:09,315 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0
on 60020, call openScanner(.META.,,1, [Lorg.apache.hadoop.io.Text;@167e3a5, ,
1200441309302, null) from 10.0.0.1:49221: error: java.io.IOException:
HStoreScanner failed construction
java.io.IOException: HStoreScanner failed construction
at
org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(HStore.java:2139)
at org.apache.hadoop.hbase.HStore$HStoreScanner.<init>(HStore.java:2260)
at org.apache.hadoop.hbase.HStore.getScanner(HStore.java:2050)
at org.apache.hadoop.hbase.HRegion$HScanner.<init>(HRegion.java:1593)
at org.apache.hadoop.hbase.HRegion.getScanner(HRegion.java:1167)
at
org.apache.hadoop.hbase.HRegionServer.openScanner(HRegionServer.java:1517)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:908)
Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
does not exist:
/gfs_storage/hadoop-root/hbase/.META./1028785192/info/mapfiles/5207253989210860351/data
at org.apache.hadoop.dfs.FSDirectory.getFileInfo(FSDirectory.java:543)
at
org.apache.hadoop.dfs.FSNamesystem.getFileInfo(FSNamesystem.java:1464)
at org.apache.hadoop.dfs.NameNode.getFileInfo(NameNode.java:439)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:409)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:908)
at org.apache.hadoop.ipc.Client.call(Client.java:509)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy1.getFileInfo(Unknown Source)
at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:442)
at
org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:331)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:536)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1349)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1344)
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:254)
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:242)
at
org.apache.hadoop.hbase.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:595)
at
org.apache.hadoop.hbase.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:650)
at org.apache.hadoop.hbase.HStoreFile.getReader(HStoreFile.java:424)
at
org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(HStore.java:2114)
... 11 more
{code}
> Compaction errors after a region splits
> ---------------------------------------
>
> Key: HADOOP-2619
> URL: https://issues.apache.org/jira/browse/HADOOP-2619
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Environment: hadoop snv 612165
> Reporter: Billy Pearson
> Fix For: 0.16.0
>
> Attachments: hbase-root-regionserver-PE1750-4.log
>
>
> I am getting compaction errors from regions after they split not all of them
> have this problem but some do
> I attached a log I picked out one region
> webdata,com.technorati/tag/potiron:http,1200430376177
> it is loaded then splits at
> 2008-01-15 14:52:56,116
> the split is finshed at
> 2008-01-15 14:53:01,653
> the first compaction for the new top half region starts at
> 2008-01-15 14:54:07,612 -
> webdata,com.technorati/tag/potiron:http,1200430376177
> and ends successful at
> 2008-01-15 14:54:30,229
> ten the next compaction starts at
> 2008-01-15 14:56:16,315
> This one ends with an error at
> 2008-01-15 14:56:40,246
> {code}
> 2008-01-15 14:57:53,002 ERROR org.apache.hadoop.hbase.HRegionServer:
> Compaction failed for region
> webdata,com.technorati/tag/potiron:http,1200430376177
> org.apache.hadoop.dfs.LeaseExpiredException:
> org.apache.hadoop.dfs.LeaseExpiredException: No lease on
> /gfs_storage/hadoop-root/hbase/webdata/compaction.dir/1438658724/in_rank/mapfiles/8222904438849251562/data
> at org.apache.hadoop.dfs.FSNamesystem.checkLease(FSNamesystem.java:1123)
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1061)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:303)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:409)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:908)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:48)
> at
> org.apache.hadoop.hbase.HRegionServer$Compactor.run(HRegionServer.java:418)
> {code}
> and all other compaction's for this region fail after this one fail with the
> same error I will have to keep testing to see if it ever finishes
> successfully.
> maybe after a restart it will successfully finsh a compaction.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
