Hadoop with S3 instead of local storage

[Alok Kumar]

Hi,

Followed instructions from this link for setup
http://wiki.apache.org/hadoop/AmazonS3.

my core-site.xml  contains only these 3 properties :
property
  namefs.default.name/name
  values3://BUCKET/value
/property

property
  namefs.s3.awsAccessKeyId/name
  valueID/value
/property

property
  namefs.s3.awsSecretAccessKey/name
  valueSECRET/value
/property

hdfs-site.xml is empty!

Namenode log says, its trying to connect to local HDFS not S3.
Am i missing anything?

Regards,
Alok

Re: Can I change hadoo.tmp.dir for each jon run without formatting

[Alok Kumar]

Hi Abhay,

On Wed, Jul 25, 2012 at 10:44 PM, Abhay Ratnaparkhi
abhay.ratnapar...@gmail.com wrote:
 hadoop.tmp.dir points to the directory on local disk to store
 intermediate task related data.

 It's currently mounted to /tmp/hadoop for me. Some of my jobs are running
 and Filesystem on which '/tmp' is mounted is getting full.
 Is it possible to change hadoop.tmp.dir parameter before submitting a new
 job?

You can override hadoop.tmp.dir everytime before submitting your Job.
I tried like this :

Configuration configuration = new Configuration();
config.set(hadoop.tmp.dir, /home/user/some-other-path);
Job job = new Job(config, Job1);

It produced same result (I didn't format anything)

Thanks
-- 
Alok

problem configuring hadoop with s3 bucket

[Alok Kumar]

Hello Group,

I've hadoop setup locally running.

Now I want to use Amazon s3://mybucket as my data store,
so i changed like  dfs.data.dir=s3://mybucket/hadoop/  in my
hdfs-site.xml, Is it the correct way?
I'm getting error :

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in
dfs.data.dir: can not create directory: s3://mybucket/hadoop
2012-07-23 13:15:06,260 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in
dfs.data.dir are invalid.

and
when i changed like  dfs.data.dir=s3://mybucket/ 
I got error :
 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
java.lang.IllegalArgumentException: Wrong FS: s3://mybucket/, expected:
file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at
org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1574)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)

Also,
When I'm changing fs.default.name=s3://mybucket , Namenode is not coming
up with error : ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
java.net.BindException: (Any way I want to run namenode locally, so I
reverted it back to hdfs://localhost:9000 )

Your help is highly appreciated!
Thanks


-- 
Alok Kumar