Re: Failed to run distcp against ftp server installed on Windows.

1886 2015-04-27 16:36 GMT+08:00 sam liu : > Hi Experts, > > It is really weird that DistCp could successfully get the file from > FileZilla ftp server on Windows7, but failed from the IIS ftp server on the > same Windows7 OS(but I can get file using wget directly: 'wget > ft

Re: Failed to run distcp against ftp server installed on Windows.

ls.DistCp.run(DistCp.java:121) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:401) Thanks! 2015-02-02 15:41 GMT+08:00 sam liu : > Hi Experts, > > I could run distcp against ftp server installed on Linux

Failed to run distcp against ftp server installed on Windows.

Hi Experts, I could run distcp against ftp server installed on Linux, but could NOT run distcp against ftp server installed on Windows. Below are the steps. Is this a DistCp bug? Any comments? [Scenario 1] I installed a BI cluster using trunk build on HadoopNode1, and then could copy file from a

Re: Questions on rollback/upgrade HDFS with QJM HA enabled

e.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:460) at java.security.AccessController.doPrivileged(AccessController.java:369) 2015-01-26 10:26 GMT+08:00 sam liu : > Could any expert please help answer the questions? > > Thanks in advance! > > 2015-01-24 21:31 GMT+08:00 sam liu : &g

Re: Questions on rollback/upgrade HDFS with QJM HA enabled

Could any expert please help answer the questions? Thanks in advance! 2015-01-24 21:31 GMT+08:00 sam liu : > Hi Experts, > > I have questions on rollback/upgrade HDFS with QJM HA enabled. > > On the website > http://hadoop.apache.org/docs/current/hadoop-projec

Questions on rollback/upgrade HDFS with QJM HA enabled

Hi Experts, I have questions on rollback/upgrade HDFS with QJM HA enabled. On the website http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html#HDFS_UpgradeFinalizationRollback_with_HA_Enabled, it says: 'To perform a rollback of an upgrade, both NN

Does MRv2 support the deprecated properties?

Hi Experts, In Hadoop 2.x code, there are lots of deprecated properties and many of them have new version in 2.x code. For example, org.apache.hadoop.mapred.JobConf#MAPRED_JOB_MAP_MEMORY_MB_PROPERTY is deprecated, and its new version is MAPREDUCE_JOB_MAP_MEMORY_MB_PROPERTY. Does MRv2 support the

Re: Both hadoop fsck and dfsadmin can not detect missing replica in time?

ly remove the block? > > sent from my mobile > Daemeon C.M. Reiydelle > USA 415.501.0198 > London +44.0.20.8144.9872 > On Nov 12, 2014 9:45 PM, "sam liu" wrote: > >> Hi Experts, >> >> In my hdfs, there is a file named /tmp/test.txt belonging to 1 block wi

Re: Both hadoop fsck and dfsadmin can not detect missing replica in time?

Is that a bug of hadoop fsck or dfsadmin? As they really did not defect the missing replica data on a datanode. 2014-11-13 13:43 GMT+08:00 sam liu : > Hi Experts, > > In my hdfs, there is a file named /tmp/test.txt belonging to 1 block with > 2 replica. The block id is blk_1073742

Both hadoop fsck and dfsadmin can not detect missing replica in time?

Hi Experts, In my hdfs, there is a file named /tmp/test.txt belonging to 1 block with 2 replica. The block id is blk_1073742304_1480 and the 2 replica resides on datanode1 and datanode2. Today I manually removed the block file on datanode2: ./current/BP-1640683473-9.181.64.230-1415757100604/curre

Re: Can add a regular check in DataNode on free disk space?

Hadoop daemons will keep writing to log > files, and it will help you diagnose the issue. > > Hope it helps. > > Regards, > Aitor > > On 20 October 2014 11:27, sam liu wrote: > >> Hi Dhiraj, >> >> My cluster only includes 1 datanode and its log does no

Re: Can add a regular check in DataNode on free disk space?

Cluster) > > > > Regards, > > Dhiraj > > > > *From:* sam liu [mailto:samliuhad...@gmail.com] > *Sent:* Monday, October 20, 2014 11:51 AM > *To:* user@hadoop.apache.org > *Subject:* Re: Can add a regular check in DataNode on free disk space? > > > > Hi

Re: Can add a regular check in DataNode on free disk space?

s from > /var/lib/hadoop-hdfs/cache/hdfs/dfs/name > 3. Format namenode > 4. Start all Hadoop daemons. > > On Mon, Oct 20, 2014 at 8:26 AM, sam liu wrote: > >> Hi Experts and Developers, >> >> At present, if a DataNode does not has free disk space, we can n

Can add a regular check in DataNode on free disk space?

Hi Experts and Developers, At present, if a DataNode does not has free disk space, we can not get this bad situation from anywhere, including DataNode log. At the same time, under this situation, the hdfs writing operation will fail and return error msg as below. However, from the error msg, user

Re: Failed to rollback from hadoop-2.4.1 to hadoop 2.2.0

eProgress status > > If it says upgrade is complete then you can do some sanity check by hdfs > fsck. > > Stop the servers by stop-dfs.sh and then rollback by command > start-dfs.sh -rollback > > On 9/18/14, sam liu wrote: > > Thanks for your comment! > > > &

Re: Failed to rollback from hadoop-2.4.1 to hadoop 2.2.0

09-18 1:51 GMT-07:00 Susheel Kumar Gadalay : > You have to upgrade both name node and data node. > > Better issue start-dfs.sh -upgrade. > > Check whether current and previous directories are present in both > dfs.namenode.name.dir and dfs.datanode.data.dir directory. > &g

Failed to rollback from hadoop-2.4.1 to hadoop 2.2.0

Hi Expert, Below are my steps and is it a hadoop bug or did I miss any thing? Thanks! Step: [A] Upgrade 1. Install Hadoop 2.2.0 cluster 2. Stop Hadoop services 3. Replace 2.2.0 binaries with 2.4.1 binaries 4. Start datanodes: $HADOOP_HOME/sbin/hadoop-daemon.sh start datanode 5. Start namenode wit

Start standby namenode using bootstrapStandby hangs

Hi Experts, My hadoop cluster is enabled HA with QJM and I failed to upgrade it from version 2.2.0 to 2.4.1. Why? Is this a existing issue? My steps: 1. Stop hadoop cluster 2. On each node, upgrade hadoop binary with the newer version 3. On each JournalNode: sbin/hadoop-daemon.sh start journalnod

HDFS rollingUpgrade failed due to unexpected storage info

Hi Experts, According to section 'Upgrading Non-Federated Clusters' of http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html, I tried to upgrade hadoop 2.2.0 to hadoop 2.4.1. However, I failed on step 2.2 'Start NN2 as standby with the "-rollingUpgrade starte

Re: How to use docker in Hadoop, with patch of YARN-1964?

lExecutor.java:897) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919) at java.lang.Thread.run(Thread.java:738) Thanks! 2014-08-12 2:18 GMT-07:00 sam liu : > Hi Experts, > > I am very interesting that Hadoop could work with Docker and doing

How to use docker in Hadoop, with patch of YARN-1964?

Hi Experts, I am very interesting that Hadoop could work with Docker and doing some trial on patch of YARN-1964. I applied patch yarn-1964-branch-2.2.0-docker.patch of jira YARN-1964 on branch 2.2 and am going to install a Hadoop cluster using the new generated tarball including the patch. Then,

Re: File Permission Issue using Distributed Cache of Hadoop-2.2.0

execution permission after putting file onto hdfs?* This issue confuses us for a long time, and any comments/suggestions will be appreciated! 2014-05-30 17:03 GMT+08:00 sam liu : > Hi, > > On Hadoop 1.1.1, I did a test on execution permission as below: > 1. Set '*dfs.umaskmode*&

Re: File Permission Issue using Distributed Cache of Hadoop-2.2.0

/jira/browse/HADOOP-3078 > > https://issues.apache.org/jira/browse/HDFS-4659 > > > > Cheers > > Seb. > > > > *From:* sam liu [mailto:samliuhad...@gmail.com] > *Sent:* Wednesday, May 28, 2014 7:40 AM > *To:* user@hadoop.apache.org > *Subject:* Re: File Permission Is

Re: File Permission Issue using Distributed Cache of Hadoop-2.2.0

Is this possible a Hadoop issue? Or any options is wrong in my cluster? 2014-05-27 13:58 GMT+08:00 sam liu : > Hi Experts, > > The original local file has execution permission, and then it was > distributed to multiple nodemanager nodes with Distributed Cache feature of > Hadoop

File Permission Issue using Distributed Cache of Hadoop-2.2.0

Hi Experts, The original local file has execution permission, and then it was distributed to multiple nodemanager nodes with Distributed Cache feature of Hadoop-2.2.0, but the distributed file has lost the execution permission. However I did not encounter such issue in Hadoop-1.1.1. Why this hap

Re: Failed to run 'mvn package' on hadoop-2.2 using Cygwin

emails. You can download the win version > and put that into your classpath: > https://code.google.com/p/protobuf/downloads/detail?name=protoc-2.5.0-win32.zip&can=2&q= > > Jason > > > On Fri, May 23, 2014 at 5:27 AM, sam liu wrote: > >> Btw, I found a strange

Re: Failed to run 'mvn package' on hadoop-2.2 using Cygwin

he system cannot find the file specified.* 2014-05-23 11:34 GMT+08:00 sam liu : > I install JDK in Cygwin. After replacing '\\' with '/', still failed. > > Even after I reinstalled protobuf in Cygwin, I still failed and met same > exception... > > I am c

Re: Failed to run 'mvn package' on hadoop-2.2 using Cygwin

ws, and then in Cygwin, I point to that JDK. If you build and install > JDK under Cygwin, the file path with "\\" might not work since Cygwin > (Linux) uses "/" as separator. > > > On Wed, May 21, 2014 at 1:52 AM, Krishna Chaitanya > wrote: > >> Try insta

Re: Failed to run 'mvn package' on hadoop-2.2 using Cygwin

gwin64\\usr\\local\\bin\\protoc.exe"; > > Then the build worked for me. > > > On Tue, May 20, 2014 at 11:08 PM, sam liu wrote: > >> A more detailed exception info: >> >> [ERROR] Failed to execute goal >> org.apache.hadoop:hadoop-maven-plugins:2.2.0:pro

Re: Failed to run 'mvn package' on hadoop-2.2 using Cygwin

correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-common 2014-05-21 14:04 GMT+08:00 sam liu : > Still failed... > > I added /opt/software/protobuf-2.5.0/src/.libs/libprotoc.so and other > libprotoc.so.* files to LD_LIBRARY _PATH. But still f

Re: Failed to run 'mvn package' on hadoop-2.2 using Cygwin

11:45 GMT+08:00 Krishna Chaitanya : > Try adding the path to libprotoc.so to the variable LD_LIBRARY _PATH and > retry again. > On May 21, 2014 9:00 AM, "sam liu" wrote: > >> Hi Experts, >> >> I can use Cygwin to build hadoop-1.1.1

Failed to run 'mvn package' on hadoop-2.2 using Cygwin

Hi Experts, I can use Cygwin to build hadoop-1.1.1, however failed on hadoop-2.2, as I always encountter issue: *[INFO] --- hadoop-maven-plugins:2.2.0:protoc (compile-protoc) @ hadoop-common ---[WARNING] [/home/sam/protobuf-win/bin/protoc, --version] failed: java.io.IOException: Cannot run

Questions about Hadoop logs and mapred.local.dir

Hi Experts, 1. The size of mapred.local.dir is big(30 GB), how many methods could clean it correctly? 2. For logs of NameNode/DataNode/JobTracker/TaskTracker, are they all rolling type log? What's their max size? I can not find the specific settings for them in log4j.properties. 3. I find the size

Re: For QJM HA solution, after failover, application must update NameNode IP?

> dfs.namenode.http-address.MYCLUSTER.nn2 >dnsOfNameNode2:50070 > > > > dfs.client.failover.proxy.provider.gilbert-prod > > org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider > > > > On Tue, Apr 29, 2014 at 9:07 AM, sam l

For QJM HA solution, after failover, application must update NameNode IP?

Hi Experts, For example, at the beginning, the application will access NameNode using IP of active NameNode(IP: 9.123.22.1). However, after failover, the IP of active NameNode is changed to 9.123.22.2 which is the IP of previous standby NameNode. In this case, application must update NameNode IP?

Re: What codes to chmod 755 to "yarn.nodemanager.log-dirs"?

rtonworks.com/ > > On Thu, Apr 24, 2014 at 9:05 AM, sam liu wrote: > >> Hi Experts, >> >> When the nodemanager log-dirs not exists, I think >> LocalDirsHandlerService#serviceInit will invoke >> DirectoryCollection#createDir to create the log dirs, and chmod 755 t

What codes to chmod 755 to "yarn.nodemanager.log-dirs"?

Hi Experts, When the nodemanager log-dirs not exists, I think LocalDirsHandlerService#serviceInit will invoke DirectoryCollection#createDir to create the log dirs, and chmod 755 to it. However, when nodemanager log-dirs already exists and with a non 755 permission(like 775), I found its permissio

Re: Differences between HistoryServer and Yarn TimeLine server?

; > BTW, if you mean MapReduce JobHistoryServer by HistoryServer, it's a > different daemon, which serves the historic information of MapReduce jobs > only. > > > On Tue, Apr 22, 2014 at 8:44 PM, sam liu wrote: > >> Hi Experts, >> >> I am confusing

Differences between HistoryServer and Yarn TimeLine server?

Hi Experts, I am confusing on these two concepts. Could you help explain the differences? Thanks!

Re: Why block sizes shown by 'fsck' and '-stat' are inconsistent?

04-05 17:12 GMT+08:00 Harsh J : > The fsck is showing you an "average block size", not the block size > metadata attribute of the file like stat shows. In this specific case, > the average is just the length of your file, which is lesser than one > whole block. > > On Sat

Why block sizes shown by 'fsck' and '-stat' are inconsistent?

Hi Experts, First, I believe it's no doubt that HDFS use only what it needs on the local file system. For example, we store a file(12 KB size) to HDFS, and HDFS only use 12 KB on the local file system, and won't use 64 MB(block size) on the local file system for that file. However, I found the bl

Re: How to submit the patch MAPREDUCE-4490.patch which works for branch-1.2, not trunk?

to manually run 'test-patch.sh'? 2014-02-15 5:25 GMT+08:00 Arpit Agarwal : > Hi Sam, > > Hadoop Jenkins does not accept patches for 1.x. > > You can manually run 'test-patch.sh' to verify there are no regressions > introduced by your patch and copy-pas

How to submit the patch MAPREDUCE-4490.patch which works for branch-1.2, not trunk?

Hi Experts, I have been working on the JIRA https://issues.apache.org/jira/browse/MAPREDUCE-4490 and attached MAPREDUCE-4490.patch which could fix this jira. I would like to contribute my patch to community, but encountered some issues. MAPREDUCE-4490 is an issue on Hadoop-1.x versions, and my pa

What are the methods to share dynamic data among mappers/reducers?

Hi, As I know, the Distributed Cache will copy the shared data to the slaves before starting job, and won't change the shared data after that. So are there any solutions to share dynamic data among mappers/reducers? Thanks!

Any method to get input splits by column?

Hi, By default, MR inputformat classes break input file into splits by rows. However, we have a specilal requirement on MR app: get input splits by column. Is there any good method? Thanks!

Re: How to set the handler count related parameters?

is is setting which is set that the child processes of tasktrackers will > start with this option if you do not overwrite them from client side. I may > be wrong in this. > > > > > > On Sun, Dec 22, 2013 at 6:50 PM, sam liu wrote: > >> Hi, >> >> We have 20 nod

How to set the handler count related parameters?

Hi, We have 20 nodes cluster(1 namenode, 1 jobtracker, 18 datanodes). Each node has 20 cpu cores and 64 GB memory. How to set the values for following parameters? - *dfs.namenode.handler.count- dfs.datanode.handler.count- * *mapreduce.jobtracker.handler.count* - *mapred.child.java.opts* Thank

Re: hdfs unable to create new block with 'Too many open fiiles' exception

s' user. > > Cheers > > > On Sat, Dec 21, 2013 at 8:30 AM, sam liu wrote: > >> Hi Experts, >> >> We failed to run an MR job which accesses hive, as hdfs is unable to >> create new block during reduce phase. The exceptions: >> 1) In tasklog: &g

hdfs unable to create new block with 'Too many open fiiles' exception

Hi Experts, We failed to run an MR job which accesses hive, as hdfs is unable to create new block during reduce phase. The exceptions: 1) In tasklog: hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable to create new block 2) In HDFS data node log: DataXceiveServer: IOException

Re: About conception and usage of Uber

Any comments/corrections on my understanding on Uber? Thanks in advance! 2013/11/8 sam liu > Hi Experts, > > In previous discussions, I found following descriptions: > "mapreduce.job.ubertask.enable | (false) | 'Whether to enable the > small-jobs "uber

About conception and usage of Uber

Hi Experts, In previous discussions, I found following descriptions: "mapreduce.job.ubertask.enable | (false) | 'Whether to enable the small-jobs "ubertask" optimization, which runs "sufficiently small" jobs sequentially within a single JVM. "Small" is defined by the following maxmaps, maxreduces,

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

e config can be > bypassed by clients that do not read the configs, place a manual API > override, etc.. > > If you want to really define a hard maximum and catch such clients, > try setting dfs.replication.max to 2 at your NameNode. > > On Thu, Aug 1, 2013 at 8:07 AM, sam liu w

Re: Yarn never use TeraSort#TotalOrderPartitioner when run TeraSort job?

7;p' is the path of '_partition.lst'. But I am not clear two details: - Where is the location of 'p'? It's on hdfs or Linux file system? What's its absolute path? - Which part or phase of Hadoop MapReduce copy the _partition.lst file to the path 'p'? I am ve

Re: Yarn never use TeraSort#TotalOrderPartitioner when run TeraSort job?

uses the new org.apache.hadoop.mapreduce apis (not > org.apache.hadoop.mapred). > > Did you fiddle with the right TotalOrderPartitioner > i.e. org.apache.hadoop.mapreduce.TotalOrderPartitioner? > > Arun > > On Oct 17, 2013, at 8:12 PM, sam liu wrote: > > It's re

Re: Yarn never use TeraSort#TotalOrderPartitioner when run TeraSort job?

It's really weird and confusing me. Anyone can help this question? Thanks! 2013/10/16 sam liu > Hi Experts, > > In Hadoop-2.0.4, the TeraSort leverage TeraSort#TotalOrderPartitioner as > its Partitioner: 'job.setPartitionerClass(TotalOrderPartitioner.class);'. &g

Yarn never use TeraSort#TotalOrderPartitioner when run TeraSort job?

Hi Experts, In Hadoop-2.0.4, the TeraSort leverage TeraSort#TotalOrderPartitioner as its Partitioner: 'job.setPartitionerClass(TotalOrderPartitioner.class);'. However, seems Yarn did not execute the methods of TeraSort#TotalOrderPartitioner at all. I did some tests to verify it as below: Test 1:

Re: ArrayIndexOutOfBoundsException occurs from MapTask

gt; from 2047 to 512, then jobs are executed success > > > > It might help in your case also.. > > > > -- > > *From:* sam liu [samliuhad...@gmail.com] > *Sent:* Thursday, September 12, 2013 9:37 AM > *To:* user@hadoop.apache.org > *Subj

ArrayIndexOutOfBoundsException occurs from MapTask

Hi, We are using Hadoop-1.1.1. When run a big workload mapreduce application, it threw a ArrayIndexOutOfBoundsException as below. Seems the issue was threw in line 'System.arraycopy(b, off, kvbuffer, bufindex, gaplen);' in MapTask. I am not familiar with the code of MapTask.java, and want to know

Re: UnsupportedOperationException occurs with Hadoop-2.1.0-beta jar files

This is an env issue. Hadoop-2.10-beta upgraded protobuf to 2.5 from 2.4.1, but the version of protobuf in my env is still 2.4.1, so the sqoop unit tests failed on my env. After I upgraded my protobuf to 2.5, all sqoop unit tests passed. 2013/9/9 sam liu > Hi, > > With Hadoop-2.1.0

UnsupportedOperationException occurs with Hadoop-2.1.0-beta jar files

Hi, With Hadoop-2.1.0-beta jar files, Sqoop-1.4.3 test TestAllTables failed by exception UnsupportedOperationException, however it works with Hadoop-2.0.4-alpha jar files. Below are the details. Any comments? Thanks! 4202 [main] INFO org.apache.hadoop.mapreduce.JobSubmitter - Cleaning up the stag

How to get username/jobid/taskid/localdir in LinuxTaskController#createLogDir?

Hi Developers, This is an issue when I prepare a patch for MAPREDUCE-4490, and I added following comments in that JIRA, but no response so far. We are eager to resolve this issue in our env, so any one could provide help? Thanks very much! JIR

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

; The step (a) points to your problem and solution both. You have files > being created with repl=3 on a 2 DN cluster which will prevent > decommission. This is not a bug. > > On Wed, Jul 31, 2013 at 12:09 PM, sam liu wrote: > > I opened a jira for tracking this issue: > > https

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

I opened a jira for tracking this issue: https://issues.apache.org/jira/browse/HDFS-5046 2013/7/2 sam liu > Yes, the default replication factor is 3. However, in my case, it's > strange: during decommission hangs, I found some block's expected replicas > is 3, but the 

Re: Parameter 'yarn.nodemanager.resource.cpu-cores' does not work

air > Scheduler will support CPU-based scheduling as well starting in 2.1.0-beta. > > -Sandy > > > On Sat, Jul 20, 2013 at 11:04 PM, sam liu wrote: > >> Thanks, but seems it does not work for me. >> >> My hadoop version is 'Hadoop 2.0.4-alpha', and seem

Re: Parameter 'yarn.nodemanager.resource.cpu-cores' does not work

manager.linux-container-executor.cgroups.mount > > yarn.nodemanager.linux-container-executor.cgroups.mount-path > > ** ** > > -Chuan > > ** ** > > *From:* sam liu [mailto:samliuhad...@gmail.com] > *Sent:* Tuesday, July 02, 2013 8:33 PM > *To:* use

Parameter 'yarn.nodemanager.resource.cpu-cores' does not work

Hi, With Hadoop 2.0.4-alpha, yarn.nodemanager.resource.cpu-cores does not work for me: 1. The performance of running same terasort job do not change, even after increasing or decreasing the value of 'yarn.nodemanager.resource.cpu-cores' in yarn-site.xml and restart the yarn cluster. 2. Even if I

Re: How to write/run MPI program on Yarn?

Any one could help answer above questions? Thanks a lot! 2013/7/1 sam liu > Thanks Pramod and Clark! > > 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project? > 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned > there is alr

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

default replication is 3 > and all files it will create (as part of the app or via a job config) > will be with replication factor 3. > > You can do an -lsr to find all files and filter which ones have been > created with a factor of 3 (versus expected config of 2). > > On F

Re: How to write/run MPI program on Yarn?

ight interest you. >> https://github.com/clarkyzl/mpich2-yarn >> >> >> >> Pramod N <http://atmachinelearner.blogspot.in> >> Bruce Wayne of web >> @machinelearner <https://twitter.com/machinelearner> >> >> -- >> >> >> On Sun, Jun 30, 2013 a

How to write/run MPI program on Yarn?

Hi Experts, Does Hadoop 2.0.5-alpha supports MPI programs? - If yes, is there any example of writing a MPI program on Yarn? How to write the client side code to configure and submit the MPI job? - If no, which Hadoop version will support MPI programming? Thanks!

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

cause it needs 3 nodes to achieve the factor > (replicas are not created on the same node). If you set the replication > factor=2 i think you will not have this issue. So in general you must make > sure that the rep factor is <= to the available datanodes. > > BR, > George >

Hang when add/remove a datanode into/from a 2 datanode cluster

Hi, I encountered an issue which hangs the decommission operatoin. Its steps: 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in hdfs-site.xml, set the 'dfs.replication' to 2 2. Add node dn3 into the cluster as a new datanode, and did not change the ' dfs.replication' value

Re: How Yarn execute MRv1 job?

ryy Yu >> >>> Hi Sam, >>> please look at :http://hbase.apache.org/book.html#d2617e499 >>> >>> generally, we said YARN is Hadoop-2.x, you can download >>> hadoop-2.0.4-alpha. and Hive-0.10 supports hadoop-2.x very well. >>> >>&g

Re: How Yarn execute MRv1 job?

oad > hadoop-2.0.4-alpha. and Hive-0.10 supports hadoop-2.x very well. > > > > On Thu, Jun 20, 2013 at 2:11 PM, sam liu wrote: > >> Thanks Arun! >> >> #1, Yes, I did tests and found that the MRv1 jobs could run against YARN >> directly, without recompiling &g

Re: How Yarn execute MRv1 job?

? 2013/6/20 Arun C Murthy > > On Jun 19, 2013, at 6:45 PM, sam liu wrote: > > Appreciating for the detailed answers! Here are three further questions: > > - Yarn maintains backwards compatibility, and MRv1 job could run on Yarn. > If yarn does not ask existing MRv1 job to do

Re: How Yarn execute MRv1 job?

27;yarn.nodemanager.resource.memory-mb'. But how to set the default size of >>> physical mem of a container? >>> >>> ApplicationMaster is responsible for getting the containers from RM by >>> sending the resource requests. For MR Job, you can use >>> "mapreduce.map.memor

How Yarn execute MRv1 job?

Hi, 1.In Hadoop 1.x, a job will be executed by map task and reduce task together, with a typical process(map > shuffle > reduce). In Yarn, as I know, a MRv1 job will be executed only by ApplicationMaster. - Yarn could run multiple kinds of jobs(MR, MPI, ...), but, MRv1 job has special execution pr

Re: Why my tests shows Yarn is worse than MRv1 for terasort?

x27;mapreduce.task.io.sort.mb' and 'mapreduce.map.sort.spill.percent' > > Yes, all of these properties will still work. Old properties specific > to JobTracker or TaskTracker (usually found as a keyword in the config > name) will not apply anymore

Re: Why my tests shows Yarn is worse than MRv1 for terasort?

e > > with two different values. > > > > Your max JVM memory of 1000 MB is (dangerously?) close to the default > > mapreduce.map/reduce.memory.mb of 1024 MB. Are any of your tasks getting > > killed for running over resource limits? > > > > -Sandy > >

Re: Why my tests shows Yarn is worse than MRv1 for terasort?

s at > this point none of us can tell what it is. > > Obviously, it isn't our goal to make MR2 slower for users and to not > care about such things :) > > On Fri, Jun 7, 2013 at 8:45 AM, sam liu wrote: > > At the begining, I just want to do a fast comparision of MRv1 and Y

Re: Why my tests shows Yarn is worse than MRv1 for terasort?

ld be nice that if you do this, share your results in a blog post or > in a research article, to spread the word about your findings. > > Best wishes. > > > 2013/6/6 sam liu > >> At the begining, I just want to do a fast comparision of MRv1 and Yarn. >> But they h

Re: Why my tests shows Yarn is worse than MRv1 for terasort?

not to tune the configurations? > Both frameworks have many areas to tune: > - Combiners, Shuffle optimization, Block size, etc > > > > 2013/6/6 sam liu > >> Hi Experts, >> >> We are thinking about whether to use Yarn or not in the near future, and >

Why my tests shows Yarn is worse than MRv1 for terasort?

Hi Experts, We are thinking about whether to use Yarn or not in the near future, and I ran teragen/terasort on Yarn and MRv1 for comprison. My env is three nodes cluster, and each node has similar hardware: 2 cpu(4 core), 32 mem. Both Yarn and MRv1 cluster are set on the same env. To be fair, I d

Re: The minimum memory requirements to datanode and namenode?

ry and for what purpose? > > > On Mon, May 13, 2013 at 11:28 AM, sam liu wrote: > >> Nitin, >> >> In my cluster, the tasktracker and datanode already have been launched, >> and are still running now. But the free/available mem of node3 now is just >> 16

Re: The minimum memory requirements to datanode and namenode?

datanode and > tasktracker. > > > On Mon, May 13, 2013 at 11:18 AM, sam liu wrote: > >> For node3, the memory is: >>total used free sharedbuffers >> cached >> Mem: 3834 3666167

Re: The minimum memory requirements to datanode and namenode?

node almost inaccessible as per my experience. > > > > On Sun, May 12, 2013 at 8:25 PM, sam liu wrote: > >> Got some exceptions on node3: >> 1. datanode log: >> 2013-04-17 11:13:44,719 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock

Re: The minimum memory requirements to datanode and namenode?

licated 0 times. > > > > > On Sun, May 12, 2013 at 7:28 PM, sam liu wrote: > >> Hi, >> >> I setup a cluster with 3 nodes, and after that I did not submit any job >> on it. But, after few days, I found the cluster is unhealthy: >> - No result returned afte

The minimum memory requirements to datanode and namenode?

lable I guess the issue of my cluster is caused by lacking of memeory, and my questions are: - Without running jobs, what's the minimum memory requirements to datanode and namenode? - How to define the minimum memeory for datanode and namenode? Thanks! Sam Liu

Re: Why could not find finished jobs in yarn.resourcemanager.webapp.address?

t; On Thu, May 2, 2013 at 6:43 PM, sam liu wrote: > >> I did not restart resourcemanager after running the job, and just >> launched a sample job directly using command 'hadoop jar >> share/hadoop/mapreduce/hadoop- >> mapreduce-examples-2.0.3-alpha.jar pi 2 30'

Re: Why could not find finished jobs in yarn.resourcemanager.webapp.address?

mapreduce-dev and > hdfs-dev in the bcc. Have you made sure you're not using the local job > runner? Did you restart the resourcemanager after running the job? > > -Sandy > > > On Thu, May 2, 2013 at 6:31 PM, sam liu wrote: > >> Can anyone help this issue? Th

Re: Why could not find finished jobs in yarn.resourcemanager.webapp.address?

Can anyone help this issue? Thanks! 2013/5/2 sam liu > Hi, > > I launched yarn and its webapp on port 18088, and then successfully > launched and executed some test MR jobs like 'hadoop jar > share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.3-alpha.jar pi 2 30'. &g

Failed to install openssl-devel 1.0.0-20.el6 on OS RHELS 6.3 x86_64

x86_64/6.3) glibc-common = 2.12-1.80.el6 Available: glibc-common-2.12-1.47.el6.x86_64 (rhel-cd) glibc-common = 2.12-1.47.el6 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest Sam Liu Thanks!