Can set different dfs.replication for different dirs

Hi, I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3 for dfs /user. How to configure? Both /tmp and /user are generated by mapreduce job or hive or sqoop. Thanks, Jack

Re: Can set different dfs.replication for different dirs

DFS replication is set on the file level (block level) or at cluster level (if you do not specify the replication factor while writing the file then this one is picked). As per my understanding, there is nothing for directories. On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing

copyFromLocal: unexpected URISyntaxException

Hi, My file name contains : and I got error copyFromLocal: unexpected URISyntaxException when I try to copy this file to Hadoop. See below. [patcharee@compute-1-0 ~]$ hadoop fs -copyFromLocal wrfout_d01_2001-01-01_00:00:00 netcdf_data/ copyFromLocal: unexpected URISyntaxException I am

Re: copyFromLocal: unexpected URISyntaxException

try putting escape chars around it On Mon, Apr 28, 2014 at 2:52 PM, Patcharee Thongtra patcharee.thong...@uni.no wrote: Hi, My file name contains : and I got error copyFromLocal: unexpected URISyntaxException when I try to copy this file to Hadoop. See below. [patcharee@compute-1-0 ~]$

Re: copyFromLocal: unexpected URISyntaxException

Hi, I tried to put escape chars around it, but it does not work. Patcharee On 04/28/2014 11:45 AM, Nitin Pawar wrote: try putting escape chars around it On Mon, Apr 28, 2014 at 2:52 PM, Patcharee Thongtra patcharee.thong...@uni.no mailto:patcharee.thong...@uni.no wrote: Hi, My

Re: copyFromLocal: unexpected URISyntaxException

Colons in path names may not currently work well: https://issues.apache.org/jira/browse/HADOOP-7945 On Mon, Apr 28, 2014 at 3:26 PM, Patcharee Thongtra patcharee.thong...@uni.no wrote: Hi, I tried to put escape chars around it, but it does not work. Patcharee On 04/28/2014 11:45 AM, Nitin

答复: hdfs write partially

Hi Harsh, I’m using HDFS client to write GZIP compressed files, I want to write once a file, in order to not uncompressing it. So I should make every write completely, otherwise file will corrupted. I’m raising up the client’s write packet size to avoid partially write. But it doesn’t work,

Install Hadoop 2.4.0 from Source - Compile error

Hi, I am trying to install Hadoop 2.4.0 from source, I got the following error, please help!! Can anyone share the apache-maven-3.1.1/conf/settings.xml” setting? Regards O/S Ubuntu: 12.04 (64-bit) Java: java version 1.6.0_45 protoc —version: libprotoc 2.5.0 Command: mvn package

Re: Install Hadoop 2.4.0 from Source - Compile error

Are you sure that CMake is installed? Best, Silvina On 28 April 2014 13:05, ascot.m...@gmail.com ascot.m...@gmail.com wrote: Hi, I am trying to install Hadoop 2.4.0 from source, I got the following error, please help!! Can anyone share the apache-maven-3.1.1/conf/settings.xml” setting?

Disable Checksum

Hello, I wish to disable checksum calculation verification in Hadoop's framework, I run the TestDfsIO application with the parameter dfs.checksum.type=NULL, To my best understanding, this was supposed to turnoff check summing in the framework, But I received the following error - ... ... ...

Re: Can set different dfs.replication for different dirs

Changes the replication factor of a file. -R option is for recursively increasing the replication factor of files within a directory. Example: - hadoop fs -setrep -w 3 -R /user/hadoop/dir1 hadoop dfs -setrep -R -w 1 /dir/ On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar

Re: Can set different dfs.replication for different dirs

Sudhakar, will this set for the new files being written to those directories? On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st sudhakara...@gmail.comwrote: Changes the replication factor of a file. -R option is for recursively increasing the replication factor of files within a directory.

Re: Install Hadoop 2.4.0 from Source - Compile error

Hi Silvina, Thanks for your reply. cmake is installed, I try the following: apt-get install cmake Reading package lists... Done Building dependency tree Reading state information... Done cmake is already the newest version. 0 upgraded, 0 newly

Re: What codes to chmod 755 to yarn.nodemanager.log-dirs?

Not 755, but yeah. See DefaultContainerExecutor.createAppLogDirs(). You may have to debug more though. +Vinod Hortonworks Inc. http://hortonworks.com/ On Fri, Apr 25, 2014 at 8:42 AM, sam liu samliuhad...@gmail.com wrote: My version is 2.1.0 and the cluster uses DefaultContainerExecutor. Is

CDH4 administration through one account

Hi,   I have a cluster CDH4. How can one perform hadoop admin without root access. Basically an account like 'john1' on the cluster want to have access to hdfs, mapred etc.,   Should 'john1' be included in the 'sudoers' file ?   What instructions should I ask System Admin team to have 'john1'

Running YARN in-process Application Master?

Hi All, I just joined this group, and not sure whether this question was discussed before. Is it possible to run Application Master within the same process as Node Manager? If not, any plan to support it future? I am asking this because we might want to use YARN as a job dispatching system. We

upgrade to CDH5 from CDH4.6 hadoop 2.0

Hello, I'm upgrading to CDH5. I download latest parcel from http://archive.cloudera.com/cdh5/parcels/latest/ to /oprt/cloudera/parcel-repo next to cluster on cludera under parcels -- I hit the distribution button, started to distribute got to 50% but it does not go any further. any ideas how to

Re: upgrade to CDH5 from CDH4.6 hadoop 2.0

Regards, Motty This kind of questions, I think that should be asked in the CDH Users mailing list. There, you will obtain a better and a faster answer. Best wishes On Monday, April 28, 2014 01:00:13 PM motty cruz wrote: Hello, I'm upgrading to CDH5. I download latest parcel from

Re: 答复: hdfs write partially

You do not need to alter the packet size to write files - why do you think you need larger packets than the default one? On Mon, Apr 28, 2014 at 4:04 PM, td...@126.com wrote: Hi Harsh, I’m using HDFS client to write GZIP compressed files, I want to write once a file, in order to not

Practical examples

Hi, I have been reading the definitive guide and taking online courses. Now I would like to understand how Hadoop is used for more real-time scenarios. Are machine learning, language processing and fraud detection examples available ? What are the other practical usecases ? I am familiar

hdfs write partially

Hi Harsh, Hadoop write once a packet, and GZIP compressed file should write completely, so I think if the packet size bigger than the compressed file, I can make sure the compressed file is not written at all or completed written. Is it right ? Thanks a lot. Regards, Ken Huang -邮件原件-

Re: Practical examples

For Machine Learning based applications of Hadoop you can check-out Mahout framework. Regards, Shahab On Mon, Apr 28, 2014 at 10:02 PM, Mohan Radhakrishnan radhakrishnan.mo...@gmail.com wrote: Hi, I have been reading the definitive guide and taking online courses. Now I would like

Re: CDH4 administration through one account

Hi Raj, Should 'john1' be included in the 'sudoers' file ? Hadoop don't use root privileges. But it has some built-in users and groups like hdfs, mapred, etc. I think you should add your admin user at least to groups hdfs and mapred. -- Thanks, Sergey On 28/04/14 23:40, Raj Hadoop wrote: Hi,

hadoop.tmp.dir directory size

Hi Folks, I am running a map only (no reduce) job on Hadoop 2.3.0 and the /tmp/hadoop-df/nm-local-dir (df being my user name) directory is growing exponentially causing me run out of my 80GB disk space. I would like to know if there is any mechanism(automatic or manual) by which I can free up

Re: Practical examples

I am interested in ML but I want a Hadoop base because I am learning hadoop. Mahout seems to be for ML at this time. Not Hadoop. Thanks, Mohan On Tue, Apr 29, 2014 at 7:38 AM, Shahab Yunus shahab.yu...@gmail.comwrote: For Machine Learning based applications of Hadoop you can check-out Mahout