RE: Staging directory ENOTDIR error.

Hi Jay, Here client is trying to create a staging directory in local file system, which actually should create in HDFS. Could you check whether do you have configured "fs.defaultFS" configuration in client with the HDFS. Thanks Devaraj k From: Jay Vyas [mailto:jayunit...@gmail.com] Sent:

RE: CompositeInputFormat

Hi Andrew, You could make use of hadoop data join classes to perform the join or you can refer these classes for better idea to perform join. http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-datajoin Thanks Devaraj k From: Botelho, Andrew [mailto:andrew.bote...@emc.com]

Re: Issues Running Hadoop 1.1.2 on multi-node cluster

I figured out the issue! The problem was in the permission to rum Hadoop scripts from root user. I create a dedicated hadoop user to rum hadoop cluster but one of the time i accidentally started hadoop from root. Hence, some of the permissions of hadoop scripts changed. The solution is to again c

Staging directory ENOTDIR error.

Hi , I'm getting an ungoogleable exception, never seen this before. This is on a hadoop 1.1. cluster... It appears that its permissions related... Any thoughts as to how this could crop up? I assume its a bug in my filesystem, but not sure. 13/07/11 18:39:43 ERROR security.UserGroupInformation:

Re: EBADF: Bad file descriptor

Thanks I will look into logs to see if I see anything else… sanjay From: Colin McCabe mailto:cmcc...@alumni.cmu.edu>> Reply-To: "user@hadoop.apache.org" mailto:user@hadoop.apache.org>> Date: Wednesday, July 10, 2013 11:52 AM To: "user@hadoop.apache.org

RE: CompositeInputFormat

Sorry I should've specified that I need an example of CompositeInputFormat that uses the new API. The example linked below uses old API objects like JobConf. Any known examples of CompositeInputFormat using the new API? Thanks in advance, Andrew From: Jay Vyas [mailto:jayunit...@gmail.com] Sen

Re: CompositeInputFormat

Map Side joins will use the CompositeInputFormat. They will only really be worth doing if one data set is small, and the other is large. This is a good example : http://www.congiu.com/joins-in-hadoop-using-compositeinputformat/ the trick is to google for CompositeInputFormat.compose() :)

CompositeInputFormat

Hi, I want to perform a JOIN on two sets of data with Hadoop. I read that the class CompositeInputFormat can be used to perform joins on data, but I can't find any examples of how to do it. Could someone help me out? It would be much appreciated. :) Thanks in advance, Andrew

Re: copy files from ftp to hdfs in parallel, distcp failed

On 11 July 2013 06:27, Hao Ren wrote: > Hi, > > I am running a hdfs on Amazon EC2 > > Say, I have a ftp server where stores some data. > I just want to copy these data directly to hdfs in a parallel way (which > maybe more efficient). > > I think hadoop distcp is what I need. > http://hadoop.ap

Re: New Distributed Cache

Yeah Andrew.. there seems to be some problem with context.getCacheFiles() api which is returning null.. Path[] cachedFilePaths = context.getLocalCacheFiles(); // I am checking why it is deprecated... for (Path cachedFilePath : cachedFilePaths) { File cachedFile = new Fi

Re: Cloudera links and Document

Satish, the right alias for Cloudera Manager questions scm-us...@cloudera.org Thanks On Thu, Jul 11, 2013 at 9:20 AM, Suresh Srinivas wrote: > Sathish, this mailing list for Apache Hadoop related questions. Please > post questions related to other distributions to appropriate vendor's > mailin

How are 'PHYSICAL_MEMORY_BYTES' and 'VIRTUAL_MEMORY_BYTES' calculated?

Hello, I am wondering how memory counters 'PHYSICAL_MEMORY_BYTES' and 'VIRTUAL_MEMORY_BYTES' are calculated? They are peaks of memory usage or cumulative usage? Thanks for help,

Re: Cloudera links and Document

Sathish, this mailing list for Apache Hadoop related questions. Please post questions related to other distributions to appropriate vendor's mailing list. On Thu, Jul 11, 2013 at 6:28 AM, Sathish Kumar wrote: > Hi All, > > Can anyone help me the link or document that explain the below. > > How

RE: New Distributed Cache

So in my driver code, I try to store the file in the cache with this line of code: job.addCacheFile(new URI("file location")); Then in my Mapper code, I do this to try and access the cached file: URI[] localPaths = context.getCacheFiles(); File f = new File(localPaths[0]); However, I get a Nul

Re: Task failure in slave node

Than you, it resolved the problem. Funny, I don't remember that I copied mahout libs to n1 hadoop but there they are. Tervitades, Margus (Margusja) Roo +372 51 48 780 http://margus.roo.ee skype: margusja -BEGIN PUBLIC KEY- MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvbeg7LwEC2SCpAEewwpC3ajxE

Re: Cloudera links and Document

Hi, Go through the links. http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Cloudera-Manager-Managing-Clusters/cmmc_CM_architecture.html http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Cloudera-Manager-Managing-Clusters/cmmc_CM_architect

Re: Task failure in slave node

sorry for typo, mahout, not mahou. sent from mobile On Jul 11, 2013 9:40 PM, "Azuryy Yu" wrote: > hi, > > put all mahou jars under hadoop_home/lib, then restart cluster. > On Jul 11, 2013 8:45 PM, "Margusja" wrote: > >> Hi >> >> I have tow nodes: >> n1 (master, salve) and n2 (slave) >> >> aft

Re: Task failure in slave node

hi, put all mahou jars under hadoop_home/lib, then restart cluster. On Jul 11, 2013 8:45 PM, "Margusja" wrote: > Hi > > I have tow nodes: > n1 (master, salve) and n2 (slave) > > after set up I ran wordcount example and it worked fine: > [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-**examp

Cloudera links and Document

Hi All, Can anyone help me the link or document that explain the below. How Cloudera Manager works and handle the clusters (Agent and Master Server)? How the Cloudera Manager Process Flow works? Where can I locate Cloudera configuration files and explanation in brief? Regards Sathish

copy files from ftp to hdfs in parallel, distcp failed

Hi, I am running a hdfs on Amazon EC2 Say, I have a ftp server where stores some data. I just want to copy these data directly to hdfs in a parallel way (which maybe more efficient). I think hadoop distcp is what I need. But $ bin/hadoop distcp ftp://username:passwd@hostname/some/path/

Task failure in slave node

Hi I have tow nodes: n1 (master, salve) and n2 (slave) after set up I ran wordcount example and it worked fine: [hduser@n1 ~]$ hadoop jar /usr/local/hadoop/hadoop-examples-1.0.4.jar wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output 13/07/11 15:30:44 INFO input.FileInputFormat: Tota

API difference between hadoop branch1 and branch2?

Dear all, I have some applications used to run on hadoop-1.1.1, and now I'd like to try them out on hadoop-2.0. Code of these application will invoke both MapReduce and HDFS api, so I'd like to know what I need to modify if migrate to hadoop-2.0. Is there any material(doc/link) about the API diffe

Re: ConnectionException in container, happens only sometimes

Here are logs of RM and 2 NMs: RM (master-host): http://pastebin.com/q4qJP8Ld NM where AM ran (slave-1-host): http://pastebin.com/vSsz7mjG NM where slave container ran (slave-2-host): http://pastebin.com/NMFi6gRp The only related error I've found in them is the following (from RM logs): ... 2013