Re: Standalone operation - file permission, Pseudo-Distributed operation - no output

[Jagat]

Hello

Can you please tell which version of Hadoop you are using and also

Does your error matches with below message?

Failed to set permissions of path:
file:/tmp/hadoop-jj/mapred/staging/jj-1931875024/.staging to 0700

Thanks
Jagat


On Thu, Mar 8, 2012 at 5:10 PM, madhu phatak  wrote:

> Hi,
> Just make sure both task tracker and data node is up. Go to localhost:50030
> and see is it shows no.of nodes equal to 1?
>
> On Thu, Feb 9, 2012 at 9:18 AM, Kyong-Ho Min  >wrote:
>
> > Hello,
> >
> > I am a hadoop newbie and I have 2 questions.
> >
> > I followed Hadoop standalone mode testing.
> > I got error message from Cygwin terminal  like file permission error.
> > I checked out mailing list and changed the part in
> RawLocalFileSystem.java
> > but not working.
> > Still I have file permission error in the directory:
> > c:/tmp/hadoop../mapred/staging...
> >
> >
> > I followed instruction about Pseudo-Distributed operation.
> > Ssh is OK and namenode -format is OK.
> > But it did not return any results and the processing is just halted.
> > The Cygwin console scripts are
> >
> > -
> > $ bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'
> > 12/02/09 14:25:44 INFO mapred.FileInputFormat: Total input paths to
> > process : 17
> > 12/02/09 14:25:44 INFO mapred.JobClient: Running job:
> job_201202091423_0001
> > 12/02/09 14:25:45 INFO mapred.JobClient:  map 0% reduce 0%
> > -
> >
> > Any help pls.
> > Thanks.
> >
> > Kyongho Min
> >
>
>
>
> --
> https://github.com/zinnia-phatak-dev/Nectar
>

Re: Very strange Java Collection behavior in Hadoop

[Jagat]

Hi Jay

Just follow this to install

http://jugnu-life.blogspot.in/2012/03/hadoop-installation-tutorial.html

The official tutorial at link below is also useful

http://hadoop.apache.org/common/docs/r1.0.1/single_node_setup.html

Thanks

Jagat

On Fri, Mar 23, 2012 at 12:08 PM, Jay Vyas  wrote:

> Hi guys : What the latest, simplest, best directions to get a tiny,
> psuedodistributed hadoop setup running on my ubuntu machine ?
>
> On Wed, Mar 21, 2012 at 5:14 PM,  wrote:
>
> > Owen,
> >
> > Is there interest in reverting hadoop-2399 in 0.23.x ?
> >
> > - Milind
> >
> > ---
> > Milind Bhandarkar
> > Greenplum Labs, EMC
> > (Disclaimer: Opinions expressed in this email are those of the author,
> and
> > do not necessarily represent the views of any organization, past or
> > present, the author might be affiliated with.)
> >
> >
> >
> > On 3/19/12 11:20 PM, "Owen O'Malley"  wrote:
> >
> > >On Mon, Mar 19, 2012 at 11:05 PM, madhu phatak 
> > >wrote:
> > >
> > >> Hi Owen O'Malley,
> > >>  Thank you for that Instant reply. It's working now. Can you explain
> me
> > >> what you mean by "input to reducer is reused" in little detail?
> > >
> > >
> > >Each time the statement "Text value = values.next();" is executed it
> > >always
> > >returns the same Text object with the contents of that object changed.
> > >When
> > >you add the Text to the list, you are adding a pointer to the same Text
> > >object. At the end you have 6 copies of the same pointer instead of 6
> > >different Text objects.
> > >
> > >The reason that I said it is my fault, is because I added the
> optimization
> > >that causes it. If you are interested in Hadoop archeology, it was
> > >HADOOP-2399 that made the change. I also did HADOOP-3522 to improve the
> > >documentation in the area.
> > >
> > >-- Owen
> >
> >
>
>
> --
> Jay Vyas
> MMSB/UCHC
>

Re: Accessing global Counters

[Jagat]

Hi

You can create your own counters like

enum CountFruits {
Apple,
Mango,
Banana
}


And in your mapper class when you see condition to increment , you can use
Reporter incrCounter method to do the same.

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/Reporter.html#incrCounter(java.lang.Enum,%20long)

e.g
// I saw Apple increment it by one
reporter.incrCounter(CountFruits.Apple,1);

Now you can access them using job.getCounters

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getCounters()

Hope this helps

Regards,

Jagat Singh


On Fri, Apr 20, 2012 at 9:43 PM, Gayatri Rao  wrote:

> Hi All,
>
> Is there a way for me to set global counters in Mapper and access them from
> reducer?
> Could you suggest how I can acheve this?
>
> Thanks
> Gayatri
>

Re: oceansync hadoop connection properties

[Jagat]

Hi

Can you tell how you started hadoop , those are locations where hadoop
namenode is running.

http://hadoop.apache.org/common/docs/current/single_node_setup.html

If you read the link above there we have detailed info about then and
hadoop install

If you are new to hadoop then you should not worry about opensync , just
play with hadoop and then go for third party tools.

Just let me know if you need help.

Regards,
Jagat Singh

---
Sent from Mobile , short and crisp.
On 22-Apr-2012 4:08 AM, "John Stein"  wrote:

> hello,
>
> I am fairly new to Hadoop and I am trying to figure out how to find the
> "full Name Node URI with port" and "full JobTracker URI with port" for
> usage with the new oceansync hadoop management software that came out.  The
> software is asking for two configuration properties and I am trying to
> figure out how I can find them.  This is what the config file looks like
> after rpm install.  I am able to launch the software but I cant figure out
> where to get the two properties.
>
>
> Preparing...###
> [100%]
> Installing OceanSync v2.0 to /opt/OceanSync/
>1:OceanSync  ###
> [100%]
> Installation of Ocean Sync v2.0 is completed!
> Please remember to configure Ocean Sync v2.0 by editing
> /opt/OceanSync/oceansync.conf with the correct Hadoop values
>
>
>  cat /opt/OceanSync/oceansync.conf
> # Ocean Sync v2.0 Configuration File
> # http://www.oceansync.com
>
> # Specify your full Name Node URI with port:
>
> hadoopURI=hdfs://oceansync.com:54310
>
> # Specify your full JobTracker URI with port:
>
> hadoopJobTracker=hdfs://oceansync.com:54311
>
> # Specify the full path to your hadoop binary for submitting MapReduce
> Jobs:
>
> hadoopBinary=/usr/hadoop-1.0.1/bin/hadoop
>
> # Logging Errors to oceansync.log:
>
> logErrors=true
>
> # Metrics Refresh Interval in miliseconds (1 = 10 seconds);
>
> metricDelay=1
>
> --
> John Stein
> Processing Engineer
> XTO Energy
>

Re: Text Analysis

[Jagat]

There are Api which you can use , offcourse they are third party.

---
Sent from Mobile , short and crisp.
On 25-Apr-2012 8:57 PM, "Robert Evans"  wrote:

> Hadoop itself is the core Map/Reduce and HDFS functionality.  The higher
> level algorithms like sentiment analysis are often done by others.
>  Cloudera has a video from HadoopWorld 2010 about it
>
>
> http://www.cloudera.com/resource/hw10_video_sentiment_analysis_powered_by_hadoop/
>
> And there are likely to be other tools like R that can help you out with
> it.  I am not really sure if mahout offers sentiment analysis or not, but
> you might want to look there too http://mahout.apache.org/
>
> --Bobby Evans
>
>
> On 4/25/12 7:50 AM, "karanveer.si...@barclays.com" <
> karanveer.si...@barclays.com> wrote:
>
> Hi,
>
> I wanted to know if there are any existing API's within Hadoop for us to
> do some text analysis like sentiment analysis, etc. OR are we to rely on
> tools like R, etc. for this.
>
>
> Regards,
> Karanveer
>
>
>
>
>
> This e-mail and any attachments are confidential and intended
> solely for the addressee and may also be privileged or exempt from
> disclosure under applicable law. If you are not the addressee, or
> have received this e-mail in error, please notify the sender
> immediately, delete it from your system and do not copy, disclose
> or otherwise act upon any part of this e-mail or its attachments.
>
> Internet communications are not guaranteed to be secure or
> virus-free.
> The Barclays Group does not accept responsibility for any loss
> arising from unauthorised access to, or interference with, any
> Internet communications by any third party, or from the
> transmission of any viruses. Replies to this e-mail may be
> monitored by the Barclays Group for operational or business
> reasons.
>
> Any opinion or other information in this e-mail or its attachments
> that does not relate to the business of the Barclays Group is
> personal to the sender and is not given or endorsed by the Barclays
> Group.
>
> Barclays Bank PLC. Registered in England and Wales (registered no.
> 1026167).
> Registered Office: 1 Churchill Place, London, E14 5HP, United
> Kingdom.
>
> Barclays Bank PLC is authorised and regulated by the Financial
> Services Authority.
>
>

Need to improve documentation for v 0.23.x ( v 2.x)

[Jagat]

Hello All,

As Apache Hadoop community is ready to release the next 2.0 alpha version
of Hadoop , i would like to bring attention towards need to make better
documentation of the tutorials and examples for the same.

Just one short example

See the Single Node Setup tutorials for v
1.xand
v
0.23,
you can say 0.23  author is in hurry with keeping all things in
assumption that reader knows everything what and where to do.

We should spend some time on documentation , with so many beautiful
features coming it would be great if you guys plan some special hackathon
meetings to improve its documentation , code examples so that people can
understand how to use them effectively.

At present only two people can understand 0.23 , those who wrote the code
and the other one is java compiler who is verifying its code :)

*Tom White , *I request if you are reading this message , please pick-up
your pen again to write Hadoop Definitive Guide edition 4th dedicated to
next release for greater benefit of community.

Thanks

Re: Need to improve documentation for v 0.23.x ( v 2.x)

[Jagat]

Hello Bobby,

Yes i will file couple of JIRAs and work on them in coming few days to
write how to setup things , include basics which were very good in old
documentation, yes there are many good things which we can take up from
1.0.2 documentation also.

Few days back Yahoo team was saying that they can donate documentation to
community which although old can be improved on to include within Apache.
May be we can get some update on confirmation for permission to use for the
same.

One of the project which is good in documentation ( wiki ) is Apache Mahout
we can learn from them , lots of extensible references , presentations ,
tutorials all at one place at wiki to refer.



On Mon, May 7, 2012 at 9:19 PM, Robert Evans  wrote:

> I agree that better documentation is almost always needed.  The problem is
> in finding the time to really make this happen.  If you or anyone else here
> wants to help out with this effort please feel free to file JIRAs and
> submit patches to improve the documentation.  Even if all the patch is, is
> a copy/paste of information from the 1.0.2 documentation that is still
> relevant for 2.0.
>
> --Bobby Evans
>
> On 5/4/12 2:21 PM, "Jagat"  wrote:
>
> Hello All,
>
> As Apache Hadoop community is ready to release the next 2.0 alpha version
> of Hadoop , i would like to bring attention towards need to make better
> documentation of the tutorials and examples for the same.
>
> Just one short example
>
> See the Single Node Setup tutorials for v
> 1.x<http://hadoop.apache.org/common/docs/r1.0.2/single_node_setup.html>and
> v
> 0.23<
> http://hadoop.apache.org/common/docs/r0.23.1/hadoop-yarn/hadoop-yarn-site/SingleCluster.html
> >,
> you can say 0.23  author is in hurry with keeping all things in
> assumption that reader knows everything what and where to do.
>
> We should spend some time on documentation , with so many beautiful
> features coming it would be great if you guys plan some special hackathon
> meetings to improve its documentation , code examples so that people can
> understand how to use them effectively.
>
> At present only two people can understand 0.23 , those who wrote the code
> and the other one is java compiler who is verifying its code :)
>
> *Tom White , *I request if you are reading this message , please pick-up
> your pen again to write Hadoop Definitive Guide edition 4th dedicated to
> next release for greater benefit of community.
>
> Thanks
>
>

Re: Hadoop in Eclipse

[Jagat]

Hello,

If you are looking to setup eclipse just to write mappers and reducers for
your code then

Create a eclipse java project
Download Apache bundle of Hadoop from Apache hadoop website
If you are using Hadoop 0.23 then go to share folder and add all those jars
e.g mapreduce , hdfs etc into project build path
If you are using Hadoop 1.x or 0,20 series then jars are present right
inside the home folder so you can directly add them to build path

Just run the main class which is your driver for your mapper and reducer
and remember to use job.setJarByClass method

Hope this helps.




On Tue, May 8, 2012 at 3:40 AM, Wilson Wayne - wwilso <
wayne.wil...@acxiom.com> wrote:

>  I’m new to Hadoop and am trying to get it setup in Eclipse.  I’m
> following the “Working with Hadoop under Eclipse” wiki to do this.  First
> let me make sure this will do what I am hoping it will do.  I’m not so much
> interested in modifying the Hadoop code itself (although I may in the not
> too distant future).  I’m more looking to be able to code and test mappers
> and reducers in Eclipse and leverage the debugging and other tools in
> Eclipse in that process.  If that’s not what the wiki is describing than
> let me know that and ignore my issue.  If there is a way to do this, would
> you please point me to where I can find the instructions.  If this is what
> the wiki is describing than I have an issue I need help with.
>
> ** **
>
> I’ve followed the directions on the Wiki and even worked through a couple
> of issues by searching the archive.  However, now I’ve hit an error that
> while I did find it in the archive, the fix that was presented is not
> fixing my issue.  After importing all the projects, I get a build error:
> “Project ‘MapReduceTools’ is missing required library:
> ‘lib/hadoop-core.jar’”.  I’ve added the M2_REPO variable as someone
> suggested in the archive where I found this error once.  That hasn’t fixed
> or even changed the error.  I’ve looked in my maven repository
> (c:\Users\wwilso\.m2\repository) and there is not a “Hadoop-core.jar”
> anywhere in the repository that windows explorer can find.  In fact,
> windows explorer can’t find a “Hadoop-core.jar” anywhere on my c:\ drive.
> 
>
> ** **
>
> I did my build using the directions from the wiki (got source using git in
> cygwin and then executing the two maven commands in cywgin).  I did use the
> patch for the “scripts/saveVersion.sh” not found error to resolve that
> issue.  That’s the only thing I have changed from the wiki or code.  Using
> Maven-3.0.4, ant-1.8.3, and protobuf-2.4.1.  Anyone have any ideas?  Thanks
> in advance.
>
> ** **
>
> ** **
>
> WAYNE WILSON
> Product Technical Consultant
> Acxiom Corporation
> EML   wayne.wil...@acxiom.com 
> TEL501.342.2870
> MBL   972.984.0268
> 5213 Rolling Rock Dr, McKinney, TX, 75070, USA
> WWW.ACXIOM.COM
>
> 
>
> [image: Friend Us on Facebook]   [image:
> Link Us on LinkedIn]
> [image: Follow Us on Twitter] 
>
> ** **
>
> [image: cid:image004.png@01CB84F1.26214350]
>   --
>
> The information contained in this communication is confidential, is
> intended only for the use of the recipient named above, and may be legally
> privileged. If the reader of this message is not the intended recipient,
> you are hereby notified that any dissemination, distribution or copying of
> this communication is strictly prohibited. If you have received this
> communication in error, please resend this communication to the sender and
> delete the original message or any copy of it from your computer system.
> Thank you.
>
> ** **
>

Re: Unable to work with Hadoop 1.0.1 using eclipse-indigo

[Jagat]

Hello Ravi

To create map reduce programs plugin is not mandatory.

Just download Hadoop
Create one Java project in Eclipse
Add jar files from Home folder of Hadoop ( from share folder in Hadoop 2.x
) to project class path
Create new mapper class and Reducer class , Driver class
Run it





On Thu, May 17, 2012 at 6:48 PM, Ravi Joshi  wrote:

> Hi, i recently downloaded and successfully installed hadoop-1.0.1 in my
> ubuntu 10.04 LTS. I have hadoop-1.0.1.tar.gz downloaded and now i want
> to design map-reduce application. As suggested by some blogs, first we
> should install eclipse plugin for hadoop, which is located inside
> contrib->eclipse plugin but in my hadoop-1.0.1.tar.gz, inside contrib
>  directory no eclipse plugin is found. Inside contrib directory only
> datajoin, failmon, gridmix, hdfsproxy, hod, index, streaming and vaidya
> directories are present. when i looked over src->contrib, i can find
> eclipse plugin directory but no jar file.
> I haven't work with hadoop
> under eclipse before, can somebody please explain me the plugin
> installation, so that i can start map-reduce development.
> Thanking you.
>
> -Ravi Joshi

Re: Unable to work with Hadoop 1.0.1 using eclipse-indigo

[Jagat]

Can you check why its saying

input path does not exist:
file:/home/hduser/Desktop/Eclipse_Workspace/K-Means
Algorithm/~/Desktop/input/doc

Re: Yahoo Hadoop Tutorial with new APIs?

[Jagat]

Hello Bobby,

Great news !!

Thanks for your efforts in handling those legal issues. I will assign
myself few JIRA's.

To start off we can take reference for dividing the documentation into same
modules as original Yahoo Tutorials and adding relevant features which have
been incorporated into new versions of Hadoop. I have also written few
tutorials for installing 2.0 series which i promised few days back.

Thank you once again.

Regards,

Jagat



On Mon, Jun 4, 2012 at 8:35 PM, Robert Evans  wrote:

> I am happy to announce that I was able to get the license on the Yahoo!
> Hadoop tutorial updated from Creative Commons Attribution 3.0 Unported
> License to Apache 2.0.  I have filed HADOOP-8477 <
> https://issues.apache.org/jira/browse/HADOOP-8477> to pull the tutorial
> into the Hadoop project, and to update it accordingly.  I am going to be
> very busy the next little while and I am hoping that those in the community
> that want this can help drive it and possibly break it down into subtasks
> to get the tutorial up to date.  I am very happy to help out, but like I
> said it may be a while before I am able to do much on this.
>
> --Bobby Evans
>
> On 4/4/12 4:43 PM, "Marcos Ortiz"  wrote:
>
>  Ok, Robert, I will be waiting for you then. There are many folks that use
> this tutorial, so I think this a good effort in favor of the Hadoop
> community.It would be nice
>  if Yahoo! donate this work, because, I have some ideas behind this, for
> example: to release a Spanish version of the tutorial.
>  Regards and best wishes
>
>  On 04/04/2012 05:29 PM, Robert Evans wrote:
> Re: Yahoo Hadoop Tutorial with new APIs? I am dropping the cross posts and
> leaving this on common-user with the others BCCed.
>
>  Marcos,
>
>  That is a great idea to be able to update the tutorial, especially if the
> community is interested in helping to do so.  We are looking into the best
> way to do this.  The idea right now is to donate this to the Hadoop project
> so that the community can keep it up to date, but we need some time to jump
> through all of the corporate hoops to get this to happen.  We have a lot
> going on right now, so if you don't see any progress on this please feel
> free to ping me and bug me about it.
>
>  --
>  Bobby Evans
>
>
>  On 4/4/12 8:15 AM, "Jagat Singh"  wrote:
>
>
> Hello Marcos
>
>  Yes , Yahoo tutorials are pretty old but still they explain the concepts
> of Map Reduce , HDFS beautifully. The way in which tutorials have been
> defined into sub sections , each builing on previous one is awesome. I
> remember when i started i was digged in there for many days. The tutorials
> are lagging now from new API point of view.
>
>  Lets have some documentation session one day , I would love to Volunteer
> to update those tutorials if people at Yahoo take input from outside world
> :)
>
>  Regards,
>
>  Jagat
>
>  - Original Message -
>  From: Marcos Ortiz
>  Sent: 04/04/12 08:32 AM
>  To: common-user@hadoop.apache.org, 'hdfs-u...@hadoop.apache.org <%
> 27hdfs-u...@hadoop.apache.org> ', mapreduce-u...@hadoop.apache.org
>  Subject: Yahoo Hadoop Tutorial with new APIs?
>
>  Regards to all the list.
>  There are many people that use the Hadoop Tutorial released by Yahoo at
> http://developer.yahoo.com/hadoop/tutorial/
> http://developer.yahoo.com/hadoop/tutorial/module4.html#chaining
>  The main issue here is that, this tutorial is written with the old APIs?
> (Hadoop 0.18 I think).
>  Is there a project for update this tutorial to the new APIs? to Hadoop
> 1.0.2 or YARN (Hadoop 0.23)
>
>  Best wishes
>  -- Marcos Luis Ortíz Valmaseda (@marcosluis2186) Data Engineer at UCI
> http://marcosluis2186.posterous.com
>  http://www.uci.cu/
>
>
>
>
>  <http://www.uci.cu/>
>
>
>

Re: Yahoo Hadoop Tutorial with new APIs?

[Jagat Singh]

Hello Marcos

 Yes , Yahoo tutorials are pretty old but still they explain the concepts of 
Map Reduce , HDFS beautifully. The way in which tutorials have been defined 
into sub sections , each builing on previous one is awesome. I remember when i 
started i was digged in there for many days. The tutorials are lagging now from 
new API point of view.

 Lets have some documentation session one day , I would love to Volunteer to 
update those tutorials if people at Yahoo take input from outside world :)

 Regards,

 Jagat

- Original Message -
From: Marcos Ortiz
Sent: 04/04/12 08:32 AM
To: common-user@hadoop.apache.org, 'hdfs-u...@hadoop.apache.org', 
mapreduce-u...@hadoop.apache.org
Subject: Yahoo Hadoop Tutorial with new APIs?

Regards to all the list.
 There are many people that use the Hadoop Tutorial released by Yahoo at 
http://developer.yahoo.com/hadoop/tutorial/ 
http://developer.yahoo.com/hadoop/tutorial/module4.html#chaining 
The main issue here is that, this tutorial is written with the old APIs? 
(Hadoop 0.18 I think).
 Is there a project for update this tutorial to the new APIs? to Hadoop 1.0.2 
or YARN (Hadoop 0.23)

 Best wishes
 -- Marcos Luis Ortíz Valmaseda (@marcosluis2186) Data Engineer at UCI 
http://marcosluis2186.posterous.com 
 http://www.uci.cu/

Re: MapReduce - Libjars

[Jagat Singh]

Hello Karan,

Did you read this article

http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/

You can have external jars in lib directory in your jar while packaging
your jar
or you can use 
DistributedCache<http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/filecache/DistributedCache.html>
.

The syntax for doing with libjars is something like this

hadoop jar myjar.jar com.driver.ClassName -libjars
/home/thirdparty/grearJar.jar

It would be available at ${mapred.local.dir}/taskTracker/archive/${user.name
}/distcache/

Read that Cloudera article it would give you idea about number of ways of
doing it.

Besides this one previous discussion can also be helpful.
http://search-hadoop.com/m/gKLlP1hMjp21

Regards,

Jagat Singh



On Thu, Jun 7, 2012 at 10:36 AM,  wrote:

> Hi,
>
> Within my map reduce programs; I am using an external JAVA library to help
> parse my raw files. When I am submitting my map reduce program, I am
> getting errors as the external class being referenced is not identified.
> Later, I explicitly specified the external jar being referenced with the
> "libjars" option. But I still continue to face problems as below. Any
> inputs to resolve the issue will be helpful.
>
> 12/06/06 13:36:37 ERROR security.UserGroupInformation:
> PriviledgedActionException as:hadoop
> cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not
> exist: hdfs://ldtdsrbd02.etf.barcapetf.com:8020/user/hadoop/-libjars
> Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException:
> Input path does not exist: hdfs://
> ldtdsrbd02.etf.barcapetf.com:8020/user/hadoop/-libjars
>
> Regards,
> Karan
>
>
>
>
> This e-mail and any attachments are confidential and intended
> solely for the addressee and may also be privileged or exempt from
> disclosure under applicable law. If you are not the addressee, or
> have received this e-mail in error, please notify the sender
> immediately, delete it from your system and do not copy, disclose
> or otherwise act upon any part of this e-mail or its attachments.
>
> Internet communications are not guaranteed to be secure or
> virus-free.
> The Barclays Group does not accept responsibility for any loss
> arising from unauthorised access to, or interference with, any
> Internet communications by any third party, or from the
> transmission of any viruses. Replies to this e-mail may be
> monitored by the Barclays Group for operational or business
> reasons.
>
> Any opinion or other information in this e-mail or its attachments
> that does not relate to the business of the Barclays Group is
> personal to the sender and is not given or endorsed by the Barclays
> Group.
>
> Barclays Bank PLC. Registered in England and Wales (registered no.
> 1026167).
> Registered Office: 1 Churchill Place, London, E14 5HP, United
> Kingdom.
>
> Barclays Bank PLC is authorised and regulated by the Financial
> Services Authority.
>

Re: Hadoop command not found:hdfs and yarn

[Jagat Singh]

Hello ,

Can you quickly review your hadoop install with below page may be you get
some hints to install.

http://jugnu-life.blogspot.in/2012/05/hadoop-20-install-tutorial-023x.html

The depreciated warning is correct as hadoop jobs have been divided now.

Regards,

Jagat Singh

On Fri, Jun 8, 2012 at 2:56 PM, Prajakta Kalmegh wrote:

> Hi
>
> I am trying to execute the following commands for setting up Hadoop:
> # Format the namenode
> hdfs namenode -format
> # Start the namenode
> hdfs namenode
> # Start a datanode
> hdfs datanode
>
> yarn resourcemanager
> yarn nodemanager
>
> It gives me a "Hadoop Command not found." error for all the commands. When
> I try to use "hadoop namenode -format" instead, it gives me a deprecated
> command warning. Can someone please tell me if I am missing including any
> env variables? I have included HADOOP_COMMON_HOME, HADOOP_HDFS_HOME,
> HADOOP_MAPRED_HOME, YARN_HOME, HADOOP_CONF_DIR, YARN_CONF_DIR,
> HADOOP_PREFIX in my path (apart from java etc).
>
> I am following the instructions for setting up Hadoop with Eclipse given
> in
> - http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment
> -
>
> http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn-site/SingleCluster.html
>
> Regards,
> Prajakta
>
>

Re: set the mapred.map.tasks.speculative.execution=false, but it is not useful.

[Jagat Singh]

Besides Speculative execution , Tasks can be attempted multiple times due
to failures.

So you can see 3 attempt there.



On Tue, Jun 12, 2012 at 5:08 PM, 林育智  wrote:

> hi all:
>   I set the mapred.map.tasks.speculative.execution=false,but in the
> userlogs,you could found 3 attempt map task's log. have i miss something?
> expect your help. thanks.
>

Re: Yahoo Hadoop Tutorial with new APIs?

[Jagat Singh]

Hi ,

Can you update this link in Jira which Robert created so that we can merge
what ever we can into code examples.

Thanks,

Jagat Singh

---
Sent from Mobile , short and crisp.
On 19-Jun-2012 8:34 AM, "JAGANADH G"  wrote:

> On Mon, Jun 4, 2012 at 8:35 PM, Robert Evans  wrote:
>
> > I am happy to announce that I was able to get the license on the Yahoo!
> > Hadoop tutorial updated from Creative Commons Attribution 3.0 Unported
> > License to Apache 2.0.  I have filed HADOOP-8477 <
> > https://issues.apache.org/jira/browse/HADOOP-8477> to pull the tutorial
> > into the Hadoop project, and to update it accordingly.  I am going to be
> > very busy the next little while and I am hoping that those in the
> community
> > that want this can help drive it and possibly break it down into subtasks
> > to get the tutorial up to date.  I am very happy to help out, but like I
> > said it may be a while before I am able to do much on this.
> >
> >
>
> Hi
>
> I Just started rewriting the code .
>
> The code is available at
> https://bitbucket.org/jaganadhg/yahoohadooptutorialcoderw
>
> Best regards
>
> --
> **
> JAGANADH G
> http://jaganadhg.in
> *ILUGCBE*
> http://ilugcbe.org.in
>

Re: Yahoo Hadoop Tutorial with new APIs?

[Jagat Singh]

Thank you,

Would you be willing to volunteer for editing some document also along with
code?

If you go through the links in Jira you can have idea about work done till
now and then you can join accordingly.

Just email me.

Thanks,

Jagat Singh

---
Sent from Mobile , short and crisp.
On 19-Jun-2012 9:15 AM, "JAGANADH G"  wrote:

> On Tue, Jun 19, 2012 at 8:44 AM, Jagat Singh  wrote:
>
> > Hi ,
> >
> > Can you update this link in Jira which Robert created so that we can
> merge
> > what ever we can into code examples.
> >
> >
>
> Hi Jagat
>
> Done
>
> Best regards
> --
> **
> JAGANADH G
> http://jaganadhg.in
> *ILUGCBE*
> http://ilugcbe.org.in
>

Re: Permission request from "Translation for education"

[Jagat Singh]

Hello,

The documentation is released under Apache 2.0

You are free to modify and translate.

Just go through the below *Redistribution *section also.

http://www.apache.org/licenses/LICENSE-2.0.html

Regards,

Jagat Singh

On Tue, Jun 19, 2012 at 10:04 AM,  wrote:

> Hey there,
>
> I am a student at the Department of Foreign Languages and at the same time
> a volunteer at an organization named “Translation for Education”. I love
> surfing on the Internet and being informed about the latest happenings
> around me. Unfortunately, most of my fellow citizens don’t know English, so
> that makes them feel somehow excluded from all this variety of useful info
> from the Internet.
> That was the reason that made me decide to start being a volunteer at such
> an NGO. So I created a blog, where I post translations of some of the texts
> that really caught my attention. Sometimes it’s a simple description of
> some organization, and sometimes it might be a narrowly specialized
> scientific article.
> Don’t you think that I do that without authors’ permission! I do ask them
> if they let me translate and then post the translation on my blog. And if
> they don’t, then they don’t.
> So I am asking you about the same, actually. Could you provide me the
> permission to translate the article on page http://hadoop.apache.org/**
> common/docs/stable/hdfs_user_**guide.html<http://hadoop.apache.org/common/docs/stable/hdfs_user_guide.html>into
>  , on condition that the translation is absolutely non-commercial and I
> will mention and credit you as the author of the article, and put the link
> to your source next to my translation too.
>
> Hoping for your understanding and waiting for your answer!
>
> Cheers,
>
>
>

Re: namenode starting problem

[Jagat Singh]

The quick solution to this is

Create *hadoop-env.sh* file in your configuration folder ( same place where
you have your hdfs-site.xml etc)

Add the following into it

export JAVA_HOME=/home/hadoop/software/java/jdk1.6.0_31

Just change the path above to correct JAVA_HOME

Just write back if it doesnt work.

Regards,

Jagat Singh


On Tue, Jun 19, 2012 at 5:52 PM, soham sardar wrote:

> I am using open jdk 6 and my java home is set as
>
> soham@XPS-L501X:~$ echo $JAVA_HOME
> /usr/lib/jvm/java-6-openjdk
>
> And the error i am getting during starting the nodes is
>
> hduser@XPS-L501X:/home/soham/cloudera/hadoop-2.0.0-cdh4.0.0/sbin$
> ./start-all.sh
>
> This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
> Starting namenodes on [localhost]
> localhost: Error: JAVA_HOME is not set and could not be found.
> localhost: Error: JAVA_HOME is not set and could not be found.
> Starting secondary namenodes [0.0.0.0]
> 0.0.0.0: Error: JAVA_HOME is not set and could not be found.
> starting yarn daemons
> starting resourcemanager, logging to
>
> /home/soham/cloudera/hadoop-2.0.0-cdh4.0.0/logs/yarn-hduser-resourcemanager-XPS-L501X.out
> localhost: Error: JAVA_HOME is not set and could not be found.
>
> hduser@XPS-L501X:/home/soham/cloudera/hadoop-2.0.0-cdh4.0.0/sbin$
> ./start-dfs.sh
> Starting namenodes on [localhost]
> localhost: Error: JAVA_HOME is not set and could not be found.
> localhost: Error: JAVA_HOME is not set and could not be found.
> Starting secondary namenodes [0.0.0.0]
> 0.0.0.0: Error: JAVA_HOME is not set and could not be found.
>
>
>
> can some one help me as to what is the error ??
> thanks in advance
>

Re: namenode startup error

[Jagat Singh]

Just try the steps which i mentioned in your earlier question.

And this would go.

Re: Error starting MRAppMaster

[Jagat Singh]

Seems you are using Pig with Hadoop 0.23 or 2.0 version.

Can you quickly recompile pig with 23 option and try this again.



On Thu, Jun 21, 2012 at 10:02 AM, Prajakta Kalmegh wrote:

> Hi
>
> I am getting the following error while trying to execute any example
> (wordcount, terasort etc):
> 12/06/21 09:52:39 INFO mapreduce.Job: Running job: job_1340251923324_0001
> 12/06/21 09:52:45 INFO mapreduce.Job: Job job_1340251923324_0001 running in
> uber mode : false
> 12/06/21 09:52:45 INFO mapreduce.Job:  map 0% reduce 0%
> 12/06/21 09:52:45 INFO mapreduce.Job: Job job_1340251923324_0001 failed
> with state FAILED due to: Application application_1340251923324_0001 failed
> 1 times due to AM Container for appattempt_1340251923324_0001_01 exited
> with  exitCode: 1 due to:
> .Failing this attempt.. Failing the application.
> 12/06/21 09:52:45 INFO mapreduce.Job: Counters: 0
> -
>
> *The contents of containers logs are pasted below:*
> 2012-06-21 09:52:43,856 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1340251923324_0001_01
> 2012-06-21 09:52:44,625 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.Error: Unresolved compilation problems:
> The method register(Class, EventHandler) in the type
> Dispatcher is not applicable for the arguments (Class,
> EventHandler)
>  org.apache.hadoop.mapreduce.jobhistory.EventType cannot be resolved to a
> type
>
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.init(MRAppMaster.java:261)
>  at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1049)
> at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>  at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1046)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1010)
> 2012-06-21 09:52:44,628 INFO [Thread-0]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a
> signal. Signaling RMCommunicator and JobHistoryEventHandler.
>
> --
>
> Any idea what would be going wrong?
>
> Regards,
> Prajakta
>