Size of intermediate data in the gridmix benchmark

[Vikas Ashok Patil]

Hello all,

If I scale down the size of input data for the gridmix2 benchmark to 100Mb,
what would be the maximum amount of intermediate data that would be
generated ?

Please let me know how could I figure it out before running the benchmark.

Thanks,
Vikas A Patil

mapred.jobtracker.retirejob.interval

[Manhee Jo]

Hi,

When I ran a job (containing some hundreds of thousands tasks) over our 
hadoop-0.19.2 cluster, I got OutOfMemoryError at JobTracker. Monitoring 
memory usage at the JobTracker with Ganglia, it looks like that memory space 
of the JobTracker is released every 24 hours, which is the default value of 
mapred.jobtracker.retirejob.interval.
What happens if I set mapred.jobtracker.retirejob.interval to any less 
values like 12 hours or even 0? Does it only control job retirement interval 
from JobTrackers memory? Any ohter side effects can you suppose?


Thanks.
Manhee

[OT] Real Time Open source solutions for aggregation and stream processing

[Ian Holsman]

firstly, my apologies for the off-topic message,
but I thought most people on this list would be knowledgeable and 
interested in this kind of thing.


We are looking to find a open source, scalable solution to do RT 
aggregation and stream processing (similar to what the 'hop' project 
http://code.google.com/p/hop/ set out to do) for large(ish) click-stream 
logs.


My first thought was something like esper, but in our testing it kind of 
hits the wall at around 10,000 rules per JVM.


I was wondering if any of you guys had some experiences in this area, 
and what your favorite toolsets are around this.


currently we are using cassandra and redis with home grown software to 
do the aggregation, but I'd love to use a common package if there is one.


and again.. apologies for the off-topic message and the x-posting.

regards
Ian

Re: Problems with HOD and HDFS

[Edward Capriolo]

On Tue, Jun 15, 2010 at 3:10 PM, Jason Stowe wrote:

> Hi David,
> The original HOD project was integrated with Condor (
> http://bit.ly/CondorProject), which Yahoo! was using to schedule clusters.
>
> A year or two ago, the Condor project in addition to being open-source w/o
> costs for licensing, created close integration with Hadoop (as does SGE),
> as
> presented by me at a prior Hadoop World, and the Condor team at Condor Week
> 2010:
> http://bit.ly/Condor_Hadoop_CondorWeek2010
>
> My company has solutions for deploying Hadoop Clusters on shared
> infrastructure using CycleServer and schedulers like Condor/SGE/etc. The
> general deployment strategy is to deploy head nodes (Name/Job Tracker),
> then
> execute nodes, and to be careful about how you deal with
> data/sizing/replication counts.
>
> If you're interested in this, please feel free to drop us a line at my
> e-mail or http://cyclecomputing.com/about/contact
>
> Thanks,
> Jason
>
>
> On Mon, Jun 14, 2010 at 7:45 PM, David Milne  wrote:
>
> > Unless I am missing something, the Fair Share and Capacity schedulers
> > sound like a solution to a different problem: aren't they for a
> > dedicated Hadoop cluster that needs to be shared by lots of people? I
> > have a general purpose cluster that needs to be shared by lots of
> > people. Only one of them (me) wants to run hadoop, and only wants to
> > run it  intermittently. I'm not concerned with data locality, as my
> > workflow is:
> >
> > 1) upload data I need to process to cluster
> > 2) run a chain of map-reduce tasks
> > 3) grab processed data from cluster
> > 4) clean up cluster
> >
> > Mesos sounds good, but I am definitely NOT brave about this. As I
> > said, I am just one user of the cluster among many. I would want to
> > stick with Torque and Maui for resource management.
> >
> > - Dave
> >
> > On Tue, Jun 15, 2010 at 12:37 AM, Amr Awadallah 
> wrote:
> > > Dave,
> > >
> > >  Yes, many others have the same situation, the recommended solution is
> > > either to use the Fair Share Scheduler or the Capacity Scheduler. These
> > > schedulers are much better than HOD since they take data locality into
> > > consideration (they don't just spin up 20 TT nodes on machines that
> have
> > > nothing to do with your data). They also don't lock down the nodes just
> > for
> > > you, so as TT are freed other jobs can use them immediately (as opposed
> > to
> > > no body can use them till your entire job is done).
> > >
> > >  Also, if you are brave and want to try something spanking new, then I
> > > recommend you reach out to the Mesos guys, they have a scheduler layer
> > under
> > > Hadoop that is data locality aware:
> > >
> > > http://mesos.berkeley.edu/
> > >
> > > -- amr
> > >
> > > On Sun, Jun 13, 2010 at 9:21 PM, David Milne 
> > wrote:
> > >
> > >> Ok, thanks Jeff.
> > >>
> > >> This is pretty surprising though. I would have thought many people
> > >> would be in my position, where they have to use Hadoop on a general
> > >> purpose cluster, and need it to play nice with a resource manager?
> > >> What do other people do in this position, if they don't use HOD?
> > >> Deprecated normally means there is a better alternative.
> > >>
> > >> - Dave
> > >>
> > >> On Mon, Jun 14, 2010 at 2:39 PM, Jeff Hammerbacher <
> ham...@cloudera.com
> > >
> > >> wrote:
> > >> > Hey Dave,
> > >> >
> > >> > I can't speak for the folks at Yahoo!, but from watching the JIRA, I
> > >> don't
> > >> > think HOD is actively used or developed anywhere these days. You're
> > >> > attempting to use a mostly deprecated project, and hence not
> receiving
> > >> any
> > >> > support on the mailing list.
> > >> >
> > >> > Thanks,
> > >> > Jeff
> > >> >
> > >> > On Sun, Jun 13, 2010 at 7:33 PM, David Milne 
> > >> wrote:
> > >> >
> > >> >> Anybody? I am completely stuck here. I have no idea who else I can
> > ask
> > >> >> or where I can go for more information. Is there somewhere specific
> > >> >> where I should be asking about HOD?
> > >> >>
> > >> >> Thank you,
> > >> >> Dave
> > >> >>
> > >> >> On Thu, Jun 10, 2010 at 2:56 PM, David Milne 
> > >> wrote:
> > >> >> > Hi there,
> > >> >> >
> > >> >> > I am trying to get Hadoop on Demand up and running, but am having
> > >> >> > problems with the ringmaster not being able to communicate with
> > HDFS.
> > >> >> >
> > >> >> > The output from the hod allocate command ends with this, with
> full
> > >> >> verbosity:
> > >> >> >
> > >> >> > [2010-06-10 14:40:22,650] CRITICAL/50 hadoop:298 - Failed to
> > retrieve
> > >> >> > 'hdfs' service address.
> > >> >> > [2010-06-10 14:40:22,654] DEBUG/10 hadoop:631 - Cleaning up
> cluster
> > id
> > >> >> > 34029.symphony.cs.waikato.ac.nz, as cluster could not be
> > allocated.
> > >> >> > [2010-06-10 14:40:22,655] DEBUG/10 hadoop:635 - Calling rm.stop()
> > >> >> > [2010-06-10 14:40:22,665] DEBUG/10 hadoop:637 - Returning from
> > >> rm.stop()
> > >> >> > [2010-06-10 14:40:22,666] CRITICAL/50 hod:401 - Cannot allocate
> > >> >> > cluster

Re: Hbase tutorial?

[rahulrinayat]

just write in thyis way
CREATE TABLE 'webtable ','mycolumn'
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/Hbase-tutorial-tp650605p896764.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Re: Jotracker java.lang.NumberFormatException

[ankit sharma]

Hi ,
 I got the mistake, It was unwanted newline in mapred-site.xml in
mapred.job.tracker.


On Tue, Jun 15, 2010 at 6:25 PM, ankit sharma wrote:

> Hi All ,
>  I have multinode cluster with 1 master (namenode,+ jobtracker) and
> 2 slavers (datanode + tasktracker).
>  I can start namenode and datanodes,but CANT start jobtracker.The
> log shows java.lang.NumberFormatException.
>
>
>
>  I will be greatfull if anybody can tell me what is the problem and
> why is this java execption being thrown?
>  here is the complete log , all the files value are
> attached.(master,slaves,core-site.xml...etc)
>
>
>
>  /
> 2010-06-15 17:05:12,679 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG:   host = centosxcat1/192.168.15.140
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.2
> STARTUP_MSG:   build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> /
> 2010-06-15 17:05:12,756 INFO org.apache.hadoop.mapred.JobTracker: Scheduler
> configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2010-06-15 17:05:12,768 FATAL org.apache.hadoop.mapred.JobTracker:
> java.lang.NumberFormatException: For input string: "54311
> "
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:481)
> at java.lang.Integer.parseInt(Integer.java:514)
> at
> org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:146)
> at
> org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:123)
> at
> org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:1807)
> at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1579)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
>
> 2010-06-15 17:05:12,769 INFO org.apache.hadoop.mapred.JobTracker:
> SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down JobTracker at centosxcat1/192.168.15.140
> /
>
> cat conf/master
> centosxcat1
>
> cat conf/salves
> aadityaxcat3
> linux-466z
>
> cat conf/core-site.xml
> 
> 
>
>  version="1.0"?>
> 
>
> 
>
> 
> 
> 
> dfs.name.dir
> 
> 
> /fsname
> 
> 
> 
> 
> dfs.data.dir
> 
> 
> /fsdata
> 
> 
> 
> 
> dfs.replication
> 
> 
> 2
> 
> 
> 
>
>
> cat conf/mapred-site.xml
> 
> 
>
> 
>
> 
> 
> 
> mapred.job.tracker
> 
> 
> centosxcat1:54311
> 
> 
> 
>
>
>
> 
> 
> fs.default.name
> hdfs://centosxcat1
> 
> 
>
>
>
> cat conf/hdfs-site.xml
>
>
>
>
>
>

exception related to logging (using latest sources)

[Asif Jan]

Hi

I am getting following exception when running map-reduce jobs.


java.lang.NullPointerException
	at  
org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:69)

at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:222)
at org.apache.hadoop.mapred.Child$4.run(Child.java:219)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
	at  
org 
.apache 
.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java: 
813)

at org.apache.hadoop.mapred.Child.main(Child.java:211)


I am using latest sources (0.22.0-snapshot) that I have built myself.

any ideas?

thanks

Re: Problems with HOD and HDFS

[Jason Stowe]

Hi David,
The original HOD project was integrated with Condor (
http://bit.ly/CondorProject), which Yahoo! was using to schedule clusters.

A year or two ago, the Condor project in addition to being open-source w/o
costs for licensing, created close integration with Hadoop (as does SGE), as
presented by me at a prior Hadoop World, and the Condor team at Condor Week
2010:
http://bit.ly/Condor_Hadoop_CondorWeek2010

My company has solutions for deploying Hadoop Clusters on shared
infrastructure using CycleServer and schedulers like Condor/SGE/etc. The
general deployment strategy is to deploy head nodes (Name/Job Tracker), then
execute nodes, and to be careful about how you deal with
data/sizing/replication counts.

If you're interested in this, please feel free to drop us a line at my
e-mail or http://cyclecomputing.com/about/contact

Thanks,
Jason


On Mon, Jun 14, 2010 at 7:45 PM, David Milne  wrote:

> Unless I am missing something, the Fair Share and Capacity schedulers
> sound like a solution to a different problem: aren't they for a
> dedicated Hadoop cluster that needs to be shared by lots of people? I
> have a general purpose cluster that needs to be shared by lots of
> people. Only one of them (me) wants to run hadoop, and only wants to
> run it  intermittently. I'm not concerned with data locality, as my
> workflow is:
>
> 1) upload data I need to process to cluster
> 2) run a chain of map-reduce tasks
> 3) grab processed data from cluster
> 4) clean up cluster
>
> Mesos sounds good, but I am definitely NOT brave about this. As I
> said, I am just one user of the cluster among many. I would want to
> stick with Torque and Maui for resource management.
>
> - Dave
>
> On Tue, Jun 15, 2010 at 12:37 AM, Amr Awadallah  wrote:
> > Dave,
> >
> >  Yes, many others have the same situation, the recommended solution is
> > either to use the Fair Share Scheduler or the Capacity Scheduler. These
> > schedulers are much better than HOD since they take data locality into
> > consideration (they don't just spin up 20 TT nodes on machines that have
> > nothing to do with your data). They also don't lock down the nodes just
> for
> > you, so as TT are freed other jobs can use them immediately (as opposed
> to
> > no body can use them till your entire job is done).
> >
> >  Also, if you are brave and want to try something spanking new, then I
> > recommend you reach out to the Mesos guys, they have a scheduler layer
> under
> > Hadoop that is data locality aware:
> >
> > http://mesos.berkeley.edu/
> >
> > -- amr
> >
> > On Sun, Jun 13, 2010 at 9:21 PM, David Milne 
> wrote:
> >
> >> Ok, thanks Jeff.
> >>
> >> This is pretty surprising though. I would have thought many people
> >> would be in my position, where they have to use Hadoop on a general
> >> purpose cluster, and need it to play nice with a resource manager?
> >> What do other people do in this position, if they don't use HOD?
> >> Deprecated normally means there is a better alternative.
> >>
> >> - Dave
> >>
> >> On Mon, Jun 14, 2010 at 2:39 PM, Jeff Hammerbacher  >
> >> wrote:
> >> > Hey Dave,
> >> >
> >> > I can't speak for the folks at Yahoo!, but from watching the JIRA, I
> >> don't
> >> > think HOD is actively used or developed anywhere these days. You're
> >> > attempting to use a mostly deprecated project, and hence not receiving
> >> any
> >> > support on the mailing list.
> >> >
> >> > Thanks,
> >> > Jeff
> >> >
> >> > On Sun, Jun 13, 2010 at 7:33 PM, David Milne 
> >> wrote:
> >> >
> >> >> Anybody? I am completely stuck here. I have no idea who else I can
> ask
> >> >> or where I can go for more information. Is there somewhere specific
> >> >> where I should be asking about HOD?
> >> >>
> >> >> Thank you,
> >> >> Dave
> >> >>
> >> >> On Thu, Jun 10, 2010 at 2:56 PM, David Milne 
> >> wrote:
> >> >> > Hi there,
> >> >> >
> >> >> > I am trying to get Hadoop on Demand up and running, but am having
> >> >> > problems with the ringmaster not being able to communicate with
> HDFS.
> >> >> >
> >> >> > The output from the hod allocate command ends with this, with full
> >> >> verbosity:
> >> >> >
> >> >> > [2010-06-10 14:40:22,650] CRITICAL/50 hadoop:298 - Failed to
> retrieve
> >> >> > 'hdfs' service address.
> >> >> > [2010-06-10 14:40:22,654] DEBUG/10 hadoop:631 - Cleaning up cluster
> id
> >> >> > 34029.symphony.cs.waikato.ac.nz, as cluster could not be
> allocated.
> >> >> > [2010-06-10 14:40:22,655] DEBUG/10 hadoop:635 - Calling rm.stop()
> >> >> > [2010-06-10 14:40:22,665] DEBUG/10 hadoop:637 - Returning from
> >> rm.stop()
> >> >> > [2010-06-10 14:40:22,666] CRITICAL/50 hod:401 - Cannot allocate
> >> >> > cluster /home/dmilne/hadoop/cluster
> >> >> > [2010-06-10 14:40:23,090] DEBUG/10 hod:597 - return code: 7
> >> >> >
> >> >> >
> >> >> > I've attached the hodrc file below, but briefly HOD is supposed to
> >> >> > provision an HDFS cluster as well as a Map/Reduce cluster, and
> seems
> >> >> > to be failing to do so. The ringmaster log looks like 

Re: Hadoop and IP on InfiniBand (IPoIB)

[Russell Brown]

FYI, Allen Wittnauer,

I'm using Linux not Solaris, but I'll pay attention to your comment 
about Solaris if I install Solaris on the cluster.  Thanks again for 
your helpful comments.


Russ

On 06/15/10 11:10 AM, Allen Wittenauer wrote:

On Jun 15, 2010, at 7:40 AM, Russell Brown wrote:

  

Thanks, Allen, for responding.

So, if I understand you correctly, the dfs.datanode.dns.interface and 
mapred.tasktracker.dns.interface options may be used to define inbound 
connections only?



Correct.  The daemons will bind to those interfaces and use those names as 
their 'official' connection in.

  

Concerning the OS configuration, my /etc/hosts files assign unique host names 
to the ethernet and IB interfaces.  However, even if I specify the IB host 
names in the masters and slaves files, communication still occurs via ethernet, 
not via IB.



BTW, are you doing this on Solaris or Linux?

Solaris is notorious for not honoring inbound and outbound interfaces. [In 
other words, just because the packet came in on bge0, that is no guarantee that 
the reply will go out on bge0 if another route is available.  Particularly 
frustrating with NFS and SunCluster.]

  

Your recommendation would therefore be to define IB instead of ethernet as the 
default network interface connection, right?



Yup.  Or at least give it a lower cost in the routing table.



--


Russell A. Brown|  Oracle
russ.br...@oracle.com   |  UMPK14-260
(650) 786-3011 (office) |  14 Network Circle
(650) 786-3453 (fax)|  Menlo Park, CA 94025

Re: Using wget to download file from HDFS

[Owen O'Malley]

On Jun 15, 2010, at 9:30 AM, Jaydeep Ayachit wrote:

Thanks, data node may not be known. Is it possible to direct url to  
namenode and namenode handling streaming by fetching data from  
various data nodes?


If you access the servlet on the NameNode, it will automatically  
redirect you to a data node that has some of the data on it. You  
certainly should not pick a random data node yourself.


Also note that in yahoo 0.20.104 or 0.22, you'll need a Kerberos  
ticket or delegation token to use the servlet.


-- Owen

Re: Hadoop and IP on InfiniBand (IPoIB)

[Allen Wittenauer]

On Jun 15, 2010, at 7:40 AM, Russell Brown wrote:

> Thanks, Allen, for responding.
> 
> So, if I understand you correctly, the dfs.datanode.dns.interface and 
> mapred.tasktracker.dns.interface options may be used to define inbound 
> connections only?

Correct.  The daemons will bind to those interfaces and use those names as 
their 'official' connection in.

> Concerning the OS configuration, my /etc/hosts files assign unique host names 
> to the ethernet and IB interfaces.  However, even if I specify the IB host 
> names in the masters and slaves files, communication still occurs via 
> ethernet, not via IB.

BTW, are you doing this on Solaris or Linux?

Solaris is notorious for not honoring inbound and outbound interfaces. [In 
other words, just because the packet came in on bge0, that is no guarantee that 
the reply will go out on bge0 if another route is available.  Particularly 
frustrating with NFS and SunCluster.]

> Your recommendation would therefore be to define IB instead of ethernet as 
> the default network interface connection, right?

Yup.  Or at least give it a lower cost in the routing table.

Re: How to use MapFile in mapreduce

[Harsh J]

Yes, your thought was right! Using SequenceFileInputFormat should work
fine (MapFile is just a specialization of it, a sorted one), so just
pass the input paths to it.

On Tue, Jun 15, 2010 at 10:43 PM, Asif Jan  wrote:
> Hi
>
> any pointers on how to use the MapFile with new mapreduce API.
>
> I did find the correspondinf output format e.g.
> org.apache.hadoop.mapreduce.lib.output.MapFileOutputFormat, but was not able
> to see how I can specify MapFileInputFormat ?  (naively I thought that
> org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
>  should work for MapFile as well)
>
> will I have to implement RecordReader in order to read from a MapFile ?
>
> Thanks
>
>
>
>
>
>
>
>
>
>



-- 
Harsh J
www.harshj.com

How to use MapFile in mapreduce

[Asif Jan]

Hi

any pointers on how to use the MapFile with new mapreduce API.

I did find the correspondinf output format e.g.  
org.apache.hadoop.mapreduce.lib.output.MapFileOutputFormat, but was  
not able to see how I can specify MapFileInputFormat ?  (naively I  
thought that  
org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;

 should work for MapFile as well)

will I have to implement RecordReader in order to read from a MapFile ?

Thanks

Re: Using wget to download file from HDFS

[Edward Capriolo]

On Tue, Jun 15, 2010 at 12:30 PM, Jaydeep Ayachit <
jaydeep_ayac...@persistent.co.in> wrote:

> Thanks, data node may not be known. Is it possible to direct url to
> namenode and namenode handling streaming by fetching data from various data
> nodes?
>
> Regards
> Jaydeep
>
> -Original Message-
> From: Harsh J [mailto:qwertyman...@gmail.com]
> Sent: Tuesday, June 15, 2010 9:38 PM
> To: common-user@hadoop.apache.org
> Subject: Re: Using wget to download file from HDFS
>
> Sure you can. A http download option is also provided in the DataNode
> web-interface (def-port:50075).
>
> Use the "streamFile" feature of the same. An example follows.
>
> If I have a file called 'results' lying as /user/hadoop/results, I'll do:
> wget
> http://hostname.top.dom:50075/streamFile?filename=/user/hadoop/results
> -Oresults
>
> This will get me the file data in the wget-local file 'results'
>
> On Tue, Jun 15, 2010 at 7:12 PM, Jaydeep Ayachit
>  wrote:
> > Hello,
> >
> > HDFS supports http read-only access to filesystem. Is it possible to use
> wget to download file using some url like
> > http://:http://%3cnamenode%3e:%3cweb> gui port>/..
> >
> > Thanks
> > Jaydeep
> >
> >
> > DISCLAIMER
> > ==
> > This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>
> DISCLAIMER
> ==
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>


To accomplish something like this: You have to use the name node web
interface and extract the names of datanodes from the html, then follow the
above process.

:::Edward reaches in his bag of tricks:::

Or you can kick up a webserver with tomcat to serve HDFS.

http://www.edwardcapriolo.com/wiki/en/Tomcat_Hadoop

Re: Jotracker java.lang.NumberFormatException

[Todd Lipcon]

Hi Ankit,

You need to trim your configuration variables so there is no extra
whitespace. eg foo, not:

foo


There's a patch up for this in many of the configs, but not sure if we got
mapred.job.tracker.

-Todd

On Tue, Jun 15, 2010 at 5:55 AM, ankit sharma wrote:

> Hi All ,
> I have multinode cluster with 1 master (namenode,+ jobtracker) and
> 2 slavers (datanode + tasktracker).
> I can start namenode and datanodes,but CANT start jobtracker.The
> log shows java.lang.NumberFormatException.
>
>
>
> I will be greatfull if anybody can tell me what is the problem and
> why is this java execption being thrown?
> here is the complete log , all the files value are
> attached.(master,slaves,core-site.xml...etc)
>
>
>
>  /
> 2010-06-15 17:05:12,679 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG:   host = centosxcat1/192.168.15.140
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.2
> STARTUP_MSG:   build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> /
> 2010-06-15 17:05:12,756 INFO org.apache.hadoop.mapred.JobTracker: Scheduler
> configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2010-06-15 17:05:12,768 FATAL org.apache.hadoop.mapred.JobTracker:
> java.lang.NumberFormatException: For input string: "54311
> "
>at
>
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>at java.lang.Integer.parseInt(Integer.java:481)
>at java.lang.Integer.parseInt(Integer.java:514)
>at
> org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:146)
>at
> org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:123)
>at
> org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:1807)
>at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1579)
>at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
>at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
>at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
>
> 2010-06-15 17:05:12,769 INFO org.apache.hadoop.mapred.JobTracker:
> SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down JobTracker at centosxcat1/192.168.15.140
> /
>
> cat conf/master
> centosxcat1
>
> cat conf/salves
> aadityaxcat3
> linux-466z
>
> cat conf/core-site.xml
> 
> 
>
>  version="1.0"?>
> 
>
> 
>
> 
> 
> 
> dfs.name.dir
> 
> 
> /fsname
> 
> 
> 
> 
> dfs.data.dir
> 
> 
> /fsdata
> 
> 
> 
> 
> dfs.replication
> 
> 
> 2
> 
> 
> 
>
>
> cat conf/mapred-site.xml
> 
> 
>
> 
>
> 
> 
> 
> mapred.job.tracker
> 
> 
> centosxcat1:54311
> 
> 
> 
>
>
>
> 
> 
> fs.default.name
> hdfs://centosxcat1
> 
> 
>
>
>
> cat conf/hdfs-site.xml
>



-- 
Todd Lipcon
Software Engineer, Cloudera

RE: Using wget to download file from HDFS

[Jaydeep Ayachit]

Thanks, data node may not be known. Is it possible to direct url to namenode 
and namenode handling streaming by fetching data from various data nodes?

Regards
Jaydeep

-Original Message-
From: Harsh J [mailto:qwertyman...@gmail.com]
Sent: Tuesday, June 15, 2010 9:38 PM
To: common-user@hadoop.apache.org
Subject: Re: Using wget to download file from HDFS

Sure you can. A http download option is also provided in the DataNode
web-interface (def-port:50075).

Use the "streamFile" feature of the same. An example follows.

If I have a file called 'results' lying as /user/hadoop/results, I'll do:
wget http://hostname.top.dom:50075/streamFile?filename=/user/hadoop/results
-O results

This will get me the file data in the wget-local file 'results'

On Tue, Jun 15, 2010 at 7:12 PM, Jaydeep Ayachit
 wrote:
> Hello,
>
> HDFS supports http read-only access to filesystem. Is it possible to use wget 
> to download file using some url like
> http://:http://%3cnamenode%3e:%3cweb> gui port>/..
>
> Thanks
> Jaydeep
>
>
> DISCLAIMER
> ==
> This e-mail may contain privileged and confidential information which is the 
> property of Persistent Systems Ltd. It is intended only for the use of the 
> individual or entity to which it is addressed. If you are not the intended 
> recipient, you are not authorized to read, retain, copy, print, distribute or 
> use this message. If you have received this communication in error, please 
> notify the sender and delete all copies of this message. Persistent Systems 
> Ltd. does not accept any liability for virus infected mails.
>



--
Harsh J
www.harshj.com

DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

Jotracker java.lang.NumberFormatException

[ankit sharma]

Hi All ,
 I have multinode cluster with 1 master (namenode,+ jobtracker) and
2 slavers (datanode + tasktracker).
 I can start namenode and datanodes,but CANT start jobtracker.The
log shows java.lang.NumberFormatException.



 I will be greatfull if anybody can tell me what is the problem and
why is this java execption being thrown?
 here is the complete log , all the files value are
attached.(master,slaves,core-site.xml...etc)



 /
2010-06-15 17:05:12,679 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/
STARTUP_MSG: Starting JobTracker
STARTUP_MSG:   host = centosxcat1/192.168.15.140
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
/
2010-06-15 17:05:12,756 INFO org.apache.hadoop.mapred.JobTracker: Scheduler
configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2010-06-15 17:05:12,768 FATAL org.apache.hadoop.mapred.JobTracker:
java.lang.NumberFormatException: For input string: "54311
"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:481)
at java.lang.Integer.parseInt(Integer.java:514)
at
org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:146)
at
org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:123)
at
org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:1807)
at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1579)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)

2010-06-15 17:05:12,769 INFO org.apache.hadoop.mapred.JobTracker:
SHUTDOWN_MSG:
/
SHUTDOWN_MSG: Shutting down JobTracker at centosxcat1/192.168.15.140
/

cat conf/master
centosxcat1

cat conf/salves
aadityaxcat3
linux-466z

cat conf/core-site.xml











dfs.name.dir


/fsname




dfs.data.dir


/fsdata




dfs.replication


2





cat conf/mapred-site.xml








mapred.job.tracker


centosxcat1:54311








fs.default.name
hdfs://centosxcat1





cat conf/hdfs-site.xml

Re: Using wget to download file from HDFS

[Harsh J]

Sure you can. A http download option is also provided in the DataNode
web-interface (def-port:50075).

Use the "streamFile" feature of the same. An example follows.

If I have a file called 'results' lying as /user/hadoop/results, I'll do:
wget http://hostname.top.dom:50075/streamFile?filename=/user/hadoop/results
-O results

This will get me the file data in the wget-local file 'results'

On Tue, Jun 15, 2010 at 7:12 PM, Jaydeep Ayachit
 wrote:
> Hello,
>
> HDFS supports http read-only access to filesystem. Is it possible to use wget 
> to download file using some url like
> http://:http://%3cnamenode%3e:%3cweb> gui port>/..
>
> Thanks
> Jaydeep
>
>
> DISCLAIMER
> ==
> This e-mail may contain privileged and confidential information which is the 
> property of Persistent Systems Ltd. It is intended only for the use of the 
> individual or entity to which it is addressed. If you are not the intended 
> recipient, you are not authorized to read, retain, copy, print, distribute or 
> use this message. If you have received this communication in error, please 
> notify the sender and delete all copies of this message. Persistent Systems 
> Ltd. does not accept any liability for virus infected mails.
>



-- 
Harsh J
www.harshj.com

Re: Hadoop and IP on InfiniBand (IPoIB)

[Russell Brown]

Thanks, Allen, for responding.

So, if I understand you correctly, the dfs.datanode.dns.interface and 
mapred.tasktracker.dns.interface options may be used to define inbound 
connections only?


Concerning the OS configuration, my /etc/hosts files assign unique host 
names to the ethernet and IB interfaces.  However, even if I specify the 
IB host names in the masters and slaves files, communication still 
occurs via ethernet, not via IB.


Your recommendation would therefore be to define IB instead of ethernet 
as the default network interface connection, right?


Thanks,

Russ


On 06/14/10 12:32 PM, Allen Wittenauer wrote:

On Jun 14, 2010, at 10:57 AM, Russell Brown wrote:

  

I'm a new user of Hadoop.  I have a Linux cluster with both gigabit ethernet 
and InfiniBand communications interfaces.  Could someone please tell me how to 
switch IP communication from ethernet (the default) to InfiniBand?  Thanks.




Hadoop will bind inbound connections via the interface settings in the various 
hadoop configuration files.  Outbound connections are unbound and based solely 
on OS configuration.  I filed a jira to fix this, but it is obviously low 
priority since few people run multi-nic boxes.  Best bet is to down the 
ethernet and up the IB, changing routing, etc, as necessary.



--


Russell A. Brown|  Oracle
russ.br...@oracle.com   |  UMPK14-260
(650) 786-3011 (office) |  14 Network Circle
(650) 786-3453 (fax)|  Menlo Park, CA 94025

Using wget to download file from HDFS

[Jaydeep Ayachit]

Hello,

HDFS supports http read-only access to filesystem. Is it possible to use wget 
to download file using some url like
http://:http://%3cnamenode%3e:%3cweb> gui port>/..

Thanks
Jaydeep


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

Re: Problems with HOD and HDFS

[Steve Loughran]

David Milne wrote:

Is there something else I could read about setting up short-lived
Hadoop clusters on virtual machines? I have no experience with VMs at
all. I see there is quite a bit of material about using them to get
Hadoop up and running with a psuedo-cluster on a single machine, but I
don't follow how this stretches out to using multiple machines
allocated by Torque.


My slides are up here
http://www.slideshare.net/steve_l/farming-hadoop-inthecloud

We've been bringing up hadoop in a virtual infrastructure, first you ask 
for the master node containing a NN, a JT and a DN with almost no 
storage (just enough for the filesystem to go live, so stop the JT 
blocking). If it comes up you then have a stable hostname for the 
filesystem which you can use for all the real worker nodes (DN + TT) you 
want.


Some nearby physicists are trying to get Hadoop to co-exist with the 
grid schedulers, I've added a feature request to make the reporting of 
task tracker slots something plugins can handle, so that you'd have a 
set of hadoop workers which could be used by the grid apps or by hadoop 
-with physical hadoop storage. When they were doing work scheduled out 
of hadoop, they'd report less availability to the Job Tracker, so not 
overload the machines.


Dan Templeton of Sun/Oracle has been working with getting Hadoop to 
coexist with his resource manager -he's worth contacting. Maybe we could 
persuade him to give public online talk on the topic.


-steve

Re: Problems with HOD and HDFS

[Vinod KV]

On Tuesday 15 June 2010 04:19 AM, David Milne wrote:

[2010-06-15 10:07:52,470] DEBUG/10 torque:147 - pbsdsh command:
/opt/torque-2.4.5/bin/pbsdsh
/home/dmilne/hadoop/hadoop-0.20.1/contrib/hod/bin/hodring
--hodring.tarball-retry-initial-time 1.0
--hodring.cmd-retry-initial-time 2.0 --hodring.cmd-retry-interval 2.0
--hodring.service-id 34350.symphony.cs.waikato.ac.nz
--hodring.temp-dir /scratch/local/dmilne/hod --hodring.http-port-range
8000-9000 --hodring.userid dmilne --hodring.java-home /opt/jdk1.6.0_20
--hodring.svcrgy-addr symphony.cs.waikato.ac.nz:36372
--hodring.download-addr h:t --hodring.tarball-retry-interval 3.0
--hodring.log-dir /scratch/local/dmilne/hod/log
--hodring.mapred-system-dir-root /mapredsystem
--hodring.xrs-port-range 32768-65536 --hodring.debug 4
--hodring.ringmaster-xrs-addr cn71:33771 --hodring.register
[2010-06-15 10:07:52,475] DEBUG/10 ringMaster:929 - Returned from runWorkers.

//chorus (many times)
   


Did you mean pbsdsh command itseld was printed many times above? That 
should not happen.


I previously thought hodrings could not start namenode but looks like 
hodrings themselves failed to startup. You can do two things:
 - See qstat output, log into the slave nodes where your job was 
supposed to start and see hodring logs there.
 - run the above hodring command yourselves directly on on these slave 
nodes for your job and see if it fails with some error.


+Vinod