Re: What makes a map to fail?

!--yarn-site.xml-- property descriptionWhether to enable log aggregation/description nameyarn.log-aggregation-enable/name valuetrue/value /property yarn.nodemanager.remote-app-log-dir Specifies the directory where logs are aggregated. Create the

FAILED EMFILE: Too many open files

While i am trying to run a MR Job I am getting FAILED EMFILE: Too many open files at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method) at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172) at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310) at

dataDir in fileChannel gets increasing - flume 1.3

Hi, I am using fileChannel AvroSink at the agent side HBase sink at collector machines. I am able to get all the log events at the collector, then also the dataDir location for fileChannels at agent machine shows more than 300MB size. Could you please give some info how dataDir works which

Content of FSImage

Hi All, I read that block information is stored in memory by hadoop once it receives block report from the datanodes. EditLog logs the changes. What is exactly stored in the FSImage file? Does it store information on the files in HDFS and how many blocks are there etc? Thanks

Re: Newbie: How to set up HDFS file system

Read as much as you can about Hadoop. Installing Hadoop will install HDFS. On Jan 7, 2014 2:03 AM, Ashish Jain ashja...@gmail.com wrote: Hello, Can someone provide some pointers on how to set up a HDFS file system? I tried searching in the document but could not find anything with respect to

Re: Content of FSImage

Yes - The entire file system namespace, including the mapping of blocks to files and file system properties, is stored in a file called the FsImage. The FsImage is stored as a file in the NameNode’s local file system too. When the NameNode starts up, it reads the FsImage and EditLog from disk,

Re: Newbie: How to set up HDFS file system

Installing Hadoop will install HDFS, and you will need to declare storage directories on the host nodes, etc. There is also the question of what setup you want to use, there is what is called pseudo-distributed mode where all the Hadoop daemons are running in one host. Are you a student looking

Network delay between mappers and reducers.

Hi I would like to measure the network delay between mappers and reducers. How can I calcute this time? Log file is having this info? Thanks Navaz

Error: Could not find or load main class hdfs

Hello, I am trying to run the WordCount example using Hadoop 2.2.0 on a single node. I tried to follow the directions from http://nextgenhadoop.blogspot.in/2013/10/steps-to-install-hadoop-220-stable.html. However, when I type in the command bin/hadoop hdfs -copyFromLocal input /input, I get

Re: Error: Could not find or load main class hdfs

does input directory exist in hdfs? you can check by hadoop fs -ls On Tue, Jan 7, 2014 at 11:16 AM, Allen, Ronald L. allen...@ornl.gov wrote: Hello, I am trying to run the WordCount example using Hadoop 2.2.0 on a single node. I tried to follow the directions from

RE: Error: Could not find or load main class hdfs

Thank you for responding. I entered hadoop fs -ls and it returned this: 14/01/07 11:40:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 2 items drwxr-xr-x - r9rhadoop supergroup 0 2014-01-06

Re: Error: Could not find or load main class hdfs

hadoop fs -copyFromLocal path to your file for e.g /user/home/hadoop/input/file1.txt On Tue, Jan 7, 2014 at 11:41 AM, Allen, Ronald L. allen...@ornl.gov wrote: Thank you for responding. I entered hadoop fs -ls and it returned this: 14/01/07 11:40:50 WARN util.NativeCodeLoader: Unable to

Re: Newbie: How to set up HDFS file system

Go to www.hortonworks.com and follow documentation to install HDP. Thank you! Leonid Fedotov Systems Architect - Professional Services lfedo...@hortonworks.com office: +1 855 846 7866 ext 292 mobile: +1 650 430 1673 On Jan 6, 2014, at 23:03, Ashish Jain ashja...@gmail.com wrote: Hello,

Re: JAVA cannot execute binary file

Hadoop env export JAVA_HOME=/usr/lib/jvm/jdk1.7.0/ I followed the above steps. hduser@base:~$ *java -version* *java version 1.7.0_45* *Java(TM) SE Runtime Environment (build 1.7.0_45-b18)* *Java HotSpot(TM) Client VM (build 24.45-b08, mixed mode, sharing)* hduser@base:~$ hduser@base:~$ *which

Re: JAVA cannot execute binary file

Seems like that you have a wrong JAVA_HOME. You can check whether the directory exists, or search around a little bit to find the right configuration with respect to your distribution. ~Haohui On Tue, Jan 7, 2014 at 2:07 PM, navaz navaz@gmail.com wrote: Hadoop env export

Ways to manage user accounts on hadoop cluster when using kerberos security

Hi, From the documentation + code, when kerberos is enabled, all tasks are run as the end user (e..g as user joe and not as hadoop user mapred) using the task-controller (which is setuid root and when it runs, it does a setuid/setgid etc. to Joe and his groups ). For this to work, user joe linux

Re: JAVA cannot execute binary file

hduser@base:~$ */usr/local/hadoop/bin/hadoop namenode -format* */usr/local/hadoop/bin/hadoop: line 320: /usr/lib/jvm/jdk1.7.0//bin/java: cannot execute binary file* */usr/local/hadoop/bin/hadoop: line 390: /usr/lib/jvm/jdk1.7.0//bin/java: cannot execute binary file* */usr/local/hadoop/bin/hadoop:

A question about Hadoop 1 job user id used for group mapping, which could lead to performance degradatioin

Hi, I looked at Hadoop 1.X source code and found some logic that I could not understand. In the org.apache.hadoop.mapred.Child class, there were two UGIs defined as follows. UserGroupInformation current = UserGroupInformation.getCurrentUser(); current.addToken(jt);

Cannot see MR logs in Hadoop 2.x

Hey guys My Environment: HDP 2 (Hadoop 2.2.0). When using Hadoop 1.x, I write log imformation in my MapReduce code, then see the log message in TaskTracker log. But In YARN, I cannot find the log message in any log files(historyserver, nodemanager, resourcemanager), even using

RE: JAVA cannot execute binary file

I have had, at some point on earlier versions of hadoop: Inside hadoop-env.sh where you set /usr/lib/jvm/jdk1.7.0/ Leave off the final slash. instead it should be /usr/lib/jvm/jdk1.7.0 Somewhere at some point it adds the /bin/java on itself. So you are getting a double slash

RE: JAVA cannot execute binary file

Hi, It really does not matter in OS whether any double slashes are there in the path or not. Always OS will normalize it. Problem could be, your java installation might not be matching with the architecture of the machine. Are you sure that /usr/lib/jvm/jdk1.7.0/ is of same architecture

Re: XML to TEXT

Hi, I am using hive. As suggest i am using xpath in select clause, but the error is coming as invalid expression. Please give some sample xml to process xml in hive. Thanks in advance Ranjini On Tue, Jan 7, 2014 at 5:14 PM, Ranjini Rathinam ranjinibe...@gmail.comwrote: Hi Gutierrez , As

Re: Newbie: How to set up HDFS file system

Thanks everyone. I was able to set up a single node hadoop. I was also able to run few HDFS commands like -copyFromLocal. Also ran the word count program. I have few more questions which I will put in another email. ++Ashish On Tue, Jan 7, 2014 at 11:45 PM, Leonid Fedotov

What is hdfs://, data.dir

Hello, This is my second post in the forum and I would like to say that I am amused with the kind of support from other folks. Thanks a lot. Here are few more questions on how stuff works in hadoop: Q1. What is hdfs://. Actually I ran a -copyFromLocal command and gave source directory as

issue about how to assiging map output to reducer?

hi,maillist: i look the containers log from hadoop fs -cat /var/log/hadoop-yarn/apps/root/logs/application_1388730279827_2770/CHBM221_50853 and log say it get 25 map output , and assiging 7 to fetcher 5, assiging 7 to fetcher 4 and assiging 11 to fetcher 3,my question is why not

Re: Cannot see MR logs in Hadoop 2.x

Sorry for my rough asking. Refine my question. When using hadoop 1.x, I can see the MR job logs under userlogs folder in local file system. But I cannot find the MR log in hadoop 2.x environment. There is no userlogs folder. Can anybody tell me where to find the log files? Kyle

what all can be done using MR

Can we do aggregation with in Hadoop MR like find min,max,sum,avg of a column in a csv file. -- *Thanks Regards* Unmesha Sreeveni U.B Junior Developer http://www.unmeshasreeveni.blogspot.in/