Another question Can set expired time for /tmp or set yarn/mapreduce to remove the expired tmp files periodically?
Thanks, Jack 2014-04-29 16:56 GMT+08:00 Meng QingPing <mqingp...@gmail.com>: > Thanks for all replies. > > The files in /tmp most are generated by hadoop jobs. Can set the > yarn/mapreduce to specify one repication for tmp files? > > Thanks, > Jack > > > 2014-04-29 16:40 GMT+08:00 sudhakara st <sudhakara...@gmail.com>: > > Hello Nitin, >> >> HDFS replication factor is always associated with file level. When your >> copying or creating file for any directory will set to default replicas. >> but you can specify your replication when creating or copying files >> hadoop fs -D dfs.replication=2 -put foo.txt fsput >> >> and in java >> >> FileSystem fs = FileSystem.get(new Configuration()); >> fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1); >> >> >> >> >> On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <nitinpawar...@gmail.com>wrote: >> >>> Sudhakar, >>> >>> will this set for the new files being written to those directories? >>> >>> >>> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <sudhakara...@gmail.com>wrote: >>> >>>> Changes the replication factor of a file. -R option is for recursively >>>> increasing the replication factor of files within a directory. >>>> >>>> Example: >>>> >>>> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1 >>>> >>>> hadoop dfs -setrep -R -w 1 /dir/ >>>> >>>> >>>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar >>>> <nitinpawar...@gmail.com>wrote: >>>> >>>>> DFS replication is set on the file level (block level) or at cluster >>>>> level (if you do not specify the replication factor while writing the file >>>>> then this one is picked). >>>>> As per my understanding, there is nothing for directories. >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mqingp...@gmail.com>wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as >>>>>> 3 for dfs /user. How to configure? Both /tmp and /user are generated by >>>>>> mapreduce job or hive or sqoop. >>>>>> >>>>>> Thanks, >>>>>> Jack >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Nitin Pawar >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Regards, >>>> ...sudhakara >>>> >>>> >>> >>> >>> >>> -- >>> Nitin Pawar >>> >> >> >> >> -- >> >> Regards, >> ...sudhakara >> >> > > > > -- > Thanks, > Qingping > -- Thanks, Qingping