bq. 13/10/08 13:32:01 INFO mapred.JobClient: MISSING_FILES=1 Are you able to provide more context from job output ?
Thanks On Tue, Oct 8, 2013 at 6:35 AM, Adrian Sandulescu < [email protected]> wrote: > Hello everyone, > > I'm using this tool to export and "import" snapshots from S3: > > https://github.com/lospro7/snapshot-s3-util/blob/master/src/main/java/com/imgur/backup/SnapshotS3Util.java > > I'm using this tool because it seems like a better option than ExportTable, > considering there isn't another HDFS cluster on hand. > > It uses the following trick to make exportSnapshot "import" from S3 to the > local HDFS. > > // Override dfs configuration to point to S3 > config.set("fs.default.name", s3protocol + accessKey + ":" > + accessSecret + "@" + bucketName); > config.set("fs.defaultFS", s3protocol + accessKey + ":" + > accessSecret + "@" + bucketName); > config.set("fs.s3.awsAccessKeyId", accessKey); > config.set("fs.s3.awsSecretAccessKey", accessSecret); > config.set("hbase.tmp.dir", "/tmp/hbase-${user.name}"); > config.set("hbase.rootdir", s3Url); > > > Imports work great, but only when using the s3n:// protocol (which means > and HFile limit of 5GB). > When using the s3:// protocol, I get the following: > 13/10/08 13:32:01 INFO mapred.JobClient: MISSING_FILES=1 > > The author said he wasn't able to debug it and just uses s3n:// until it > becomes a problem. > > Has anyone encountered this when using exportSnapshot? > Can you please point me in the right direction? > > Adrian >
