It's a reliability/stability problem. the S3 implementation of the FS doesn't provide the characteristics we rely on because S3 doesn't have these characteristics. It may be that there are improvements to be made in the s3 or s3n drivers, but I believe there's a fundamental difference in the semantics of S3 compared to HDFS. Agreed that this would be incredibly useful deployment model. I've heard you can run HBase directly against the Azue equivalent of S3, if you're willing to try a different cloud. Haven't tried it myself.
On Sat, Nov 1, 2014 at 12:21 PM, Khaled Elmeleegy <kd...@hotmail.com> wrote: > So, is it a performance problem? or a correctness problem, e.g. reads may > have stale values? or a reliability problem, where not strongly consistent > file system can drive HBase unstable introducing failures? > I can live with a performance hit, given the simplicity of running > directly on S3. Obviously, I can't take a reliability hit though. > > > From: ndimi...@gmail.com > > Date: Fri, 31 Oct 2014 17:40:16 -0700 > > Subject: Re: s3n with hbase > > To: user@hbase.apache.org > > > > Please don't do this. S3 is not a strongly consistent filesystem. HBase > > will not be happy there. Better to run on HDFS and to snapshots/copytable > > backup, restore to S3. > > > > On Fri, Oct 31, 2014 at 4:53 PM, Khaled Elmeleegy <kd...@hotmail.com> > wrote: > > > > > Hi, > > > > > > I am trying to use hbase with s3, using s3n, but I get the below > errors, > > > when starting the master. I am testing this in a pseudo distributed > mode on > > > my laptop. > > > I've also set hbase.rootdir to s3n:// > > > kdiaa-hbase.s3-us-west-2.amazonaws.com:80/root, where the > corresponding > > > bucket and directory are already created on s3. I've also set > > > fs.s3n.awsAccessKeyId, and fs.s3n.awsSecretAccessKey to the appropriate > > > values in hbase-site.xml > > > > > > So, I must be missing something. Any advice is appreciated. > > > > > > > > > > > > 2014-10-31 16:47:15,312 WARN [master:172.16.209.239:60000] > > > httpclient.RestS3Service: Response '/root' - Unexpected response code > 404, > > > expected 200 > > > 2014-10-31 16:47:15,349 WARN [master:172.16.209.239:60000] > > > httpclient.RestS3Service: Response '/root_%24folder%24' - Unexpected > > > response code 404, expected 200 > > > 2014-10-31 16:47:15,420 WARN [master:172.16.209.239:60000] > > > httpclient.RestS3Service: Response '/' - Unexpected response code 404, > > > expected 200 > > > 2014-10-31 16:47:15,420 WARN [master:172.16.209.239:60000] > > > httpclient.RestS3Service: Response '/' - Received error response with > XML > > > message > > > 2014-10-31 16:47:15,601 FATAL [master:172.16.209.239:60000] > > > master.HMaster: Unhandled exception. Starting shutdown. > > > org.apache.hadoop.fs.s3.S3Exception: > > > org.jets3t.service.S3ServiceException: S3 GET failed for '/' XML Error > > > Message: <?xml version="1.0" > > > encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The > specified > > > bucket does not exist</Message><BucketName> > > > kdiaa-hbase.s3-us-west-2.amazonaws.com > > > > </BucketName><RequestId>1589CC5DB70ED750</RequestId><HostId>cb2ZGGlNkxtf5fredweXt/wxJlAHLkioUJC86pkh0JxQfBJ1CMYoZuxHU1g+CnTB</HostId></Error> > > > at > > > > org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:245) > > > at > > > > org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:181) > > > at > > > > org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:158) > > > at > > > > org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:151) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > > at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > > at java.lang.reflect.Method.invoke(Method.java:597) > > > at > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > > > at > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > > > at org.apache.hadoop.fs.s3native.$Proxy9.list(Unknown Source) > > > at > > > > org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:432) > > > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397) > > > at > > > > org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:439) > > > at > > > > org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:147) > > > at > > > > org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128) > > > at > > > > org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:802) > > > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:615) > > > at java.lang.Thread.run(Thread.java:695) > > > Caused by: org.jets3t.service.S3ServiceException: S3 GET failed for '/' > > > XML Error Message: <?xml version="1.0" > > > encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The > specified > > > bucket does not exist</Message><BucketName> > > > kdiaa-hbase.s3-us-west-2.amazonaws.com > > > > </BucketName><RequestId>1589CC5DB70ED750</RequestId><HostId>cb2ZGGlNkxtf5fredweXt/wxJlAHLkioUJC86pkh0JxQfBJ1CMYoZuxHU1g+CnTB</HostId></Error> > > > at > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:424) > > > at > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:686) > > > at > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsInternal(RestS3Service.java:1083) > > > at > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsChunkedImpl(RestS3Service.java:1053) > > > at > > > org.jets3t.service.S3Service.listObjectsChunked(S3Service.java:1333) > > > at > > > > org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:168) > > > ... 17 more > > > 2014-10-31 16:47:15,604 INFO [master:172.16.209.239:60000] > > > master.HMaster: Aborting > > > 2014-10-31 16:47:15,612 DEBUG [master:172.16.209.239:60000] > > > master.HMaster: Stopping service threads > > > 2014-10-31 16:47:15,612 INFO [master:172.16.209.239:60000] > > > ipc.RpcServer: Stopping server on 60000 > > > 2014-10-31 16:47:15,612 INFO [RpcServer.listener,port=60000] > > > ipc.RpcServer: RpcServer.listener,port=60000: stopping > > > 2014-10-31 16:47:15,619 INFO [master:172.16.209.239:60000] > > > master.HMaster: Stopping infoServer > > > 2014-10-31 16:47:15,633 INFO [RpcServer.responder] ipc.RpcServer: > > > RpcServer.responder: stopped > > > 2014-10-31 16:47:15,633 INFO [RpcServer.responder] ipc.RpcServer: > > > RpcServer.responder: stopping > > > 2014-10-31 16:47:15,660 INFO [master:172.16.209.239:60000] > mortbay.log: > > > Stopped SelectChannelConnector@0.0.0.0:60010 > > > 2014-10-31 16:47:15,804 INFO [master:172.16.209.239:60000] > > > zookeeper.ZooKeeper: Session: 0x149689a7dd80000 closed > > > 2014-10-31 16:47:15,804 INFO [main-EventThread] zookeeper.ClientCnxn: > > > EventThread shut down > > > 2014-10-31 16:47:15,804 INFO [master:172.16.209.239:60000] > > > master.HMaster: HMaster main thread exiting > > > 2014-10-31 16:47:15,804 ERROR [main] master.HMasterCommandLine: Master > > > exiting > > > java.lang.RuntimeException: HMaster Aborted > > > at > > > > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194) > > > at > > > > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135) > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > > > at > > > > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) > > > at > org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2803) > > > > > > > >
