Hi,
I am trying to use hbase with s3, using s3n, but I get the below errors, when
starting the master. I am testing this in a pseudo distributed mode on my
laptop.
I've also set hbase.rootdir to
s3n://kdiaa-hbase.s3-us-west-2.amazonaws.com:80/root, where the corresponding
bucket and directory are already created on s3. I've also set
fs.s3n.awsAccessKeyId, and fs.s3n.awsSecretAccessKey to the appropriate values
in hbase-site.xml
So, I must be missing something. Any advice is appreciated.
2014-10-31 16:47:15,312 WARN [master:172.16.209.239:60000]
httpclient.RestS3Service: Response '/root' - Unexpected response code 404,
expected 200
2014-10-31 16:47:15,349 WARN [master:172.16.209.239:60000]
httpclient.RestS3Service: Response '/root_%24folder%24' - Unexpected response
code 404, expected 200
2014-10-31 16:47:15,420 WARN [master:172.16.209.239:60000]
httpclient.RestS3Service: Response '/' - Unexpected response code 404, expected
200
2014-10-31 16:47:15,420 WARN [master:172.16.209.239:60000]
httpclient.RestS3Service: Response '/' - Received error response with XML
message
2014-10-31 16:47:15,601 FATAL [master:172.16.209.239:60000] master.HMaster:
Unhandled exception. Starting shutdown.
org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3
GET failed for '/' XML Error Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The specified bucket
does not
exist</Message><BucketName>kdiaa-hbase.s3-us-west-2.amazonaws.com</BucketName><RequestId>1589CC5DB70ED750</RequestId><HostId>cb2ZGGlNkxtf5fredweXt/wxJlAHLkioUJC86pkh0JxQfBJ1CMYoZuxHU1g+CnTB</HostId></Error>
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:245)
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:181)
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:158)
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:151)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at org.apache.hadoop.fs.s3native.$Proxy9.list(Unknown Source)
at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:432)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:439)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:147)
at
org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:802)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:615)
at java.lang.Thread.run(Thread.java:695)
Caused by: org.jets3t.service.S3ServiceException: S3 GET failed for '/' XML
Error Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The specified bucket
does not
exist</Message><BucketName>kdiaa-hbase.s3-us-west-2.amazonaws.com</BucketName><RequestId>1589CC5DB70ED750</RequestId><HostId>cb2ZGGlNkxtf5fredweXt/wxJlAHLkioUJC86pkh0JxQfBJ1CMYoZuxHU1g+CnTB</HostId></Error>
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:424)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:686)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsInternal(RestS3Service.java:1083)
at
org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsChunkedImpl(RestS3Service.java:1053)
at org.jets3t.service.S3Service.listObjectsChunked(S3Service.java:1333)
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:168)
... 17 more
2014-10-31 16:47:15,604 INFO [master:172.16.209.239:60000] master.HMaster:
Aborting
2014-10-31 16:47:15,612 DEBUG [master:172.16.209.239:60000] master.HMaster:
Stopping service threads
2014-10-31 16:47:15,612 INFO [master:172.16.209.239:60000] ipc.RpcServer:
Stopping server on 60000
2014-10-31 16:47:15,612 INFO [RpcServer.listener,port=60000] ipc.RpcServer:
RpcServer.listener,port=60000: stopping
2014-10-31 16:47:15,619 INFO [master:172.16.209.239:60000] master.HMaster:
Stopping infoServer
2014-10-31 16:47:15,633 INFO [RpcServer.responder] ipc.RpcServer:
RpcServer.responder: stopped
2014-10-31 16:47:15,633 INFO [RpcServer.responder] ipc.RpcServer:
RpcServer.responder: stopping
2014-10-31 16:47:15,660 INFO [master:172.16.209.239:60000] mortbay.log:
Stopped SelectChannelConnector@0.0.0.0:60010
2014-10-31 16:47:15,804 INFO [master:172.16.209.239:60000]
zookeeper.ZooKeeper: Session: 0x149689a7dd80000 closed
2014-10-31 16:47:15,804 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
2014-10-31 16:47:15,804 INFO [master:172.16.209.239:60000] master.HMaster:
HMaster main thread exiting
2014-10-31 16:47:15,804 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2803)
