SQL analysis

[karanveer.singh]

We are looking at doing some initial analysis on SQL text info within the query 
runs to come up with some kind of path output to depict how various tables are 
linked to each other. For example. A 'view' might be a join from two table's 
top of the hierarchy and in turn might be creating some new tables, etc.

Any suggestions, thoughts, etc. on how can we approach this within the Hadoop 
space.



Regards,
Karan





This e-mail and any attachments are confidential and intended
solely for the addressee and may also be privileged or exempt from
disclosure under applicable law. If you are not the addressee, or
have received this e-mail in error, please notify the sender
immediately, delete it from your system and do not copy, disclose
or otherwise act upon any part of this e-mail or its attachments.

Internet communications are not guaranteed to be secure or
virus-free.
The Barclays Group does not accept responsibility for any loss
arising from unauthorised access to, or interference with, any
Internet communications by any third party, or from the
transmission of any viruses. Replies to this e-mail may be
monitored by the Barclays Group for operational or business
reasons.

Any opinion or other information in this e-mail or its attachments
that does not relate to the business of the Barclays Group is
personal to the sender and is not given or endorsed by the Barclays
Group.

Barclays Bank PLC. Registered in England and Wales (registered no.
1026167).
Registered Office: 1 Churchill Place, London, E14 5HP, United
Kingdom.

Barclays Bank PLC is authorised and regulated by the Financial
Services Authority.

Re: High load on datanode startup

[Todd Lipcon]

That's real weird..

If you can reproduce this after a reboot, I'd recommend letting the DN
run for a minute, and then capturing a "jstack " as well as
the output of "top -H -p  -b -n 5" and send it to the list.

What JVM/JDK are you using? What OS version?

-Todd


On Wed, May 9, 2012 at 11:57 PM, Darrell Taylor
 wrote:
> On Wed, May 9, 2012 at 10:52 PM, Raj Vishwanathan  wrote:
>
>> The picture either too small or too pixelated for my eyes :-)
>>
>
> There should be a zoom option in the top right of the page that allows you
> to view it full size
>
>
>>
>> Can you login to the box and send the output of top? If the system is
>> unresponsive, it has to be something more than an unbalanced hdfs cluster,
>> methinks.
>>
>
> Sorry, I'm unable to login to the box, it's completely unresponsive.
>
>
>>
>> Raj
>>
>>
>>
>> >
>> > From: Darrell Taylor 
>> >To: common-user@hadoop.apache.org; Raj Vishwanathan 
>> >Sent: Wednesday, May 9, 2012 2:40 PM
>> >Subject: Re: High load on datanode startup
>> >
>> >On Wed, May 9, 2012 at 10:23 PM, Raj Vishwanathan 
>> wrote:
>> >
>> >> When you say 'load', what do you mean? CPU load or something else?
>> >>
>> >
>> >I mean in the unix sense of load average, i.e. top would show a load of
>> >(currently) 376.
>> >
>> >Looking at Ganglia stats for the box it's not CPU load as such, the graphs
>> >shows actual CPU usage as 30%, but the number of running processes is
>> >simply growing in a linear manner - screen shot of ganglia page here :
>> >
>> >
>> https://picasaweb.google.com/lh/photo/Q0uFSzyLiriDuDnvyRUikXVR0iWwMibMfH0upnTwi28?feat=directlink
>> >
>> >
>> >
>> >>
>> >> Raj
>> >>
>> >>
>> >>
>> >> >
>> >> > From: Darrell Taylor 
>> >> >To: common-user@hadoop.apache.org
>> >> >Sent: Wednesday, May 9, 2012 9:52 AM
>> >> >Subject: High load on datanode startup
>> >> >
>> >> >Hi,
>> >> >
>> >> >I wonder if someone could give some pointers with a problem I'm having?
>> >> >
>> >> >I have a 7 machine cluster setup for testing and we have been pouring
>> data
>> >> >into it for a week without issue, have learnt several thing along the
>> way
>> >> >and solved all the problems up to now by searching online, but now I'm
>> >> >stuck.  One of the data nodes decided to have a load of 70+ this
>> morning,
>> >> >stopping datanode and tasktracker brought it back to normal, but every
>> >> time
>> >> >I start the datanode again the load shoots through the roof, and all I
>> get
>> >> >in the logs is :
>> >> >
>> >> >STARTUP_MSG: Starting DataNode
>> >> >
>> >> >
>> >> >STARTUP_MSG:   host = pl464/10.20.16.64
>> >> >
>> >> >
>> >> >STARTUP_MSG:   args = []
>> >> >
>> >> >
>> >> >STARTUP_MSG:   version = 0.20.2-cdh3u3
>> >> >
>> >> >
>> >> >STARTUP_MSG:   build =
>> >>
>> >>
>> >file:///data/1/tmp/nightly_2012-03-20_13-13-48_3/hadoop-0.20-0.20.2+923.197-1~squeeze
>> >> >-/
>> >> >
>> >> >
>> >> >2012-05-09 16:12:05,925 INFO
>> >> >org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
>> >> already
>> >> >set up for Hadoop, not re-installing.
>> >> >
>> >> >2012-05-09 16:12:06,139 INFO
>> >> >org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
>> >> already
>> >> >set up for Hadoop, not re-installing.
>> >> >
>> >> >Nothing else.
>> >> >
>> >> >The load seems to max out only 1 of the CPUs, but the machine becomes
>> >> >*very* unresponsive
>> >> >
>> >> >Anybody got any pointers of things I can try?
>> >> >
>> >> >Thanks
>> >> >Darrell.
>> >> >
>> >> >
>> >> >
>> >>
>> >
>> >
>> >
>>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: DataNodeRegistration problem

[Harsh J]

Fourie,

Good to know. Just for the record, do you remember the version of
OpenJDK that was installed? So that we can mark that as unusable in
our docs/wiki.

On Thu, May 10, 2012 at 12:22 PM, Fourie Joubert
 wrote:
> Hi
>
> Yes - that was indeed the problem...
>
> I cleaned up the Java's on all the nodes, did a clean reinstall of Sun
> jdk1.6.0_23 and the problem is gone.
>
> Many thanks and regards!
>
>
> Fourie
>
> On 05/09/2012 05:47 PM, Harsh J wrote:
>>
>> You may be hitting https://issues.apache.org/jira/browse/HDFS-1115?
>> Have you ensured Sun JDK is the only JDK available in the machines and
>> your services aren't using OpenJDK accidentally?
>>
>> On Wed, May 9, 2012 at 8:44 PM, Fourie Joubert
>>  wrote:
>>>
>>> Hi
>>>
>>> I am running Hadoop-1.0.1 with Sun jdk1.6.0_23.
>>>
>>> My system is a head node with 14 compute blades
>>>
>>> When trying to start hadoop, I get the following message in the logs for
>>> each data node:
>>>
>>>
>>> 2012-05-09 16:53:35,548 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>> DatanodeRegistration(137.215.75.201:50010,
>>> storageID=DS-2067460883-137.215.75.201-50010-1336575105195,
>>> infoPort=50075,
>>> ipcPort=50020):DataXceiver
>>>
>>> java.net.SocketException: Protocol not available
>>> ...
>>> ...
>>>
>>> The full log is shown below.
>>>
>>> I can't seem to get past this problem - any help or advice would be
>>> sincerely appreciated.
>>>
>>> Kindest regards!
>>>
>>> Fourie
>>>
>>>
>>>
>>>
>>>
>>> 2012-05-09 16:53:31,800 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>> /
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG:   host = wonko1/137.215.75.201
>>> STARTUP_MSG:   args = []
>>> STARTUP_MSG:   version = 1.0.1
>>> STARTUP_MSG:   build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1243785; compiled by 'hortonfo' on Tue Feb 14 08:15:38 UTC 2012
>>> /
>>> 2012-05-09 16:53:31,934 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig:
>>> loaded properties from hadoop-metrics2.properties
>>> 2012-05-09 16:53:31,945 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2012-05-09 16:53:31,946 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period
>>> at 10 second(s).
>>> 2012-05-09 16:53:31,946 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics
>>> system
>>> started
>>> 2012-05-09 16:53:32,022 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> ugi
>>> registered.
>>> 2012-05-09 16:53:32,232 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Registered
>>> FSDatasetStatusMBean
>>> 2012-05-09 16:53:32,242 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at
>>> 50010
>>> 2012-05-09 16:53:32,244 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is
>>> 1048576 bytes/s
>>> 2012-05-09 16:53:32,291 INFO org.mortbay.log: Logging to
>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>> org.mortbay.log.Slf4jLog
>>> 2012-05-09 16:53:32,347 INFO org.apache.hadoop.http.HttpServer: Added
>>> global
>>> filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>>> 2012-05-09 16:53:32,359 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: dfs.webhdfs.enabled =
>>> false
>>> 2012-05-09 16:53:32,359 INFO org.apache.hadoop.http.HttpServer: Port
>>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>>> -1.
>>> Opening the listener on 50075
>>> 2012-05-09 16:53:32,359 INFO org.apache.hadoop.http.HttpServer:
>>> listener.getLocalPort() returned 50075
>>> webServer.getConnectors()[0].getLocalPort() returned 50075
>>> 2012-05-09 16:53:32,360 INFO org.apache.hadoop.http.HttpServer: Jetty
>>> bound
>>> to port 50075
>>> 2012-05-09 16:53:32,360 INFO org.mortbay.log: jetty-6.1.26
>>> 2012-05-09 16:53:32,590 INFO org.mortbay.log: Started
>>> SelectChannelConnector@0.0.0.0:50075
>>> 2012-05-09 16:53:32,594 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> jvm
>>> registered.
>>> 2012-05-09 16:53:32,595 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> DataNode registered.
>>> 2012-05-09 16:53:32,614 INFO org.apache.hadoop.ipc.Server: Starting
>>> SocketReader
>>> 2012-05-09 16:53:32,616 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> RpcDetailedActivityForPort50020 registered.
>>> 2012-05-09 16:53:32,616 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> RpcActivityForPort50020 registered.
>>> 2012-05-09 16:53:32,618 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
>>> DatanodeRegistration(wonko1.bi.up.ac.za:50010,
>>> storageID=DS-2067460883-137.215.75.201-50010

job Killing itself upon some condition

[tousif]

Hi,

i have scheduled a map reduce job and i want this map reduce job to kill 
itself upon some condition. overall it should fail.  Here I'm not using  
run method of tool runner.

Re: Consistent Checksum error using SequenceFileInputFormat against /content & /parse_text folders output by Nutch.

[Ali Safdar Kureishy]

Actually, the ChecksumError happens every time, but it can happen at
different points of the execution...sometimes at the beginning, and
sometimes at the tail end of the map phase.

Hoping to hear from someone with a workaround...

Regards,
Safdar


On Thu, May 10, 2012 at 7:59 AM, Ali Safdar Kureishy <
safdar.kurei...@gmail.com> wrote:

> Hi Subbu!
>
> Thanks so much for this tip. Strangely, it doesn't seem to work for me ...
> I still get the checksum error (though it appears to happen later on in the
> job).
>
> Has this workaround always worked for you? I also tried using the
> setMaxMapperFailurePercentage() and setMaxReducerFailurePercentage()
> settings (set them to 20% each), but I still see this chekcsum error.
>
> Any thoughts/suggestions?
>
> Thanks again!
>
> Regards,
> Safdar
>
>
> On Wed, May 9, 2012 at 12:37 PM, Kasi Subrahmanyam  > wrote:
>
>> HI Ali,
>> I also faced this error when i ran the jobs either in local or in a
>> cluster.
>> I am able to solve this problem by removing the .crc file created in the
>> input folder for this job.
>> Please check that there is no .crc file in the input.
>> I hope this solves the problem.
>>
>> Thanks,
>> Subbu
>>
>>
>> On Wed, May 9, 2012 at 1:31 PM, Ali Safdar Kureishy <
>> safdar.kurei...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I've included both the Nutch and Hadoop mailing lists, since I don't
>>> know which one of the two is the root cause for this issue, and it might be
>>> possible to pursue a resolution from both sides.
>>>
>>> What I'm trying to do is to dump the contents of all the fetched pages
>>> from my nutch crawl -- about 600K of them. I've tried extracting this
>>> information initially from the */parse_text* folder, but I
>>> kept receiving the error below, so I switched over to the */content
>>> *folder, but BOTH of these *consistently *give me the following
>>> Checksum Error exception which fails the map-reduce job. At the very least
>>> I'm hoping to get some tip(s) on how to ignore this error and let my job
>>> complete.
>>>
>>> *org.apache.hadoop.fs.ChecksumException: Checksum Error
>>> at
>>> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:164)
>>> at
>>> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:101)
>>> at org.apache.hadoop.mapred.IFile$Reader.readData(IFile.java:328)
>>> at org.apache.hadoop.mapred.IFile$Reader.rejigData(IFile.java:358)
>>> at
>>> org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:342)
>>> at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:404)
>>> at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:220)
>>> at
>>> org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:330)
>>> at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350)
>>> at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156)
>>> at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:499)
>>> at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:381)
>>> at org.apache.hadoop.mapred.Merger.merge(Merger.java:77)
>>> at
>>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1522)
>>> at
>>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1154)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:359)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>> at
>>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
>>> *
>>>
>>> I'm using the *SequenceFileInputFormat* to read the data in each case.
>>>
>>> I have also attached the Hadoop output (checksum-error.txt). I have no
>>> idea how to ignore this error or to debug it. I've tried setting the
>>> boolean "*io.skip.checksum.errors*" property to *true* on the MapReduce
>>> Conf object, but it makes no difference. The error still happens
>>> consistently, so it seems like I'm either not setting the right property,
>>> or that it is being ignored by Hadoop? Since the error is thrown down in
>>> the internals of Hadoop, there doesn't seem to be any other way to ignore
>>> the error either, without changing Hadoop code (that I'm not able to do at
>>> this point). Is this a problem with the data that was output by Nutch? Or
>>> is this a bug with Hadoop? *Btw, I ran Nutch in local mode (without
>>> hadoop), and I'm running the Hadoop job (below) purely as an application
>>> from Eclipse (not via the bin/hadoop script).*
>>>
>>> Any help or pointers on how to dig further with this would be greatly
>>> appreciated. If there is any other way for me to ignore these checksum
>>> errors and let the job complete, do please share that with me as well.
>>>
>>> Here is the code for the reader job using MapReduce:
>>>
>>> package org.q.alt.sc.nutch.readerjobs;
>>>
>>> import java.io.IOException;
>>>
>>> import org.apache.hadoop.conf.Configured;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.Text;
>>> imp

Re: High load on datanode startup

[Darrell Taylor]

On Thu, May 10, 2012 at 9:33 AM, Todd Lipcon  wrote:

> That's real weird..
>
> If you can reproduce this after a reboot, I'd recommend letting the DN
> run for a minute, and then capturing a "jstack " as well as
> the output of "top -H -p  -b -n 5" and send it to the list.


What I did after the reboot this morning was to move the my dn, nn, and
mapred directories out of the the way, create a new one, formatted it, and
restarted the node, it's now happy.

I'll try moving the directories back later and do the jstack as you suggest.


>
> What JVM/JDK are you using? What OS version?
>

root@pl446:/# dpkg --get-selections | grep java
java-common install
libjaxp1.3-java install
libjaxp1.3-java-gcj install
libmysql-java   install
libxerces2-java install
libxerces2-java-gcj install
sun-java6-bin   install
sun-java6-javadbinstall
sun-java6-jdk   install
sun-java6-jre   install

root@pl446:/# java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

root@pl446:/# cat /etc/issue
Debian GNU/Linux 6.0 \n \l



>
> -Todd
>
>
> On Wed, May 9, 2012 at 11:57 PM, Darrell Taylor
>  wrote:
> > On Wed, May 9, 2012 at 10:52 PM, Raj Vishwanathan 
> wrote:
> >
> >> The picture either too small or too pixelated for my eyes :-)
> >>
> >
> > There should be a zoom option in the top right of the page that allows
> you
> > to view it full size
> >
> >
> >>
> >> Can you login to the box and send the output of top? If the system is
> >> unresponsive, it has to be something more than an unbalanced hdfs
> cluster,
> >> methinks.
> >>
> >
> > Sorry, I'm unable to login to the box, it's completely unresponsive.
> >
> >
> >>
> >> Raj
> >>
> >>
> >>
> >> >
> >> > From: Darrell Taylor 
> >> >To: common-user@hadoop.apache.org; Raj Vishwanathan  >
> >> >Sent: Wednesday, May 9, 2012 2:40 PM
> >> >Subject: Re: High load on datanode startup
> >> >
> >> >On Wed, May 9, 2012 at 10:23 PM, Raj Vishwanathan 
> >> wrote:
> >> >
> >> >> When you say 'load', what do you mean? CPU load or something else?
> >> >>
> >> >
> >> >I mean in the unix sense of load average, i.e. top would show a load of
> >> >(currently) 376.
> >> >
> >> >Looking at Ganglia stats for the box it's not CPU load as such, the
> graphs
> >> >shows actual CPU usage as 30%, but the number of running processes is
> >> >simply growing in a linear manner - screen shot of ganglia page here :
> >> >
> >> >
> >>
> https://picasaweb.google.com/lh/photo/Q0uFSzyLiriDuDnvyRUikXVR0iWwMibMfH0upnTwi28?feat=directlink
> >> >
> >> >
> >> >
> >> >>
> >> >> Raj
> >> >>
> >> >>
> >> >>
> >> >> >
> >> >> > From: Darrell Taylor 
> >> >> >To: common-user@hadoop.apache.org
> >> >> >Sent: Wednesday, May 9, 2012 9:52 AM
> >> >> >Subject: High load on datanode startup
> >> >> >
> >> >> >Hi,
> >> >> >
> >> >> >I wonder if someone could give some pointers with a problem I'm
> having?
> >> >> >
> >> >> >I have a 7 machine cluster setup for testing and we have been
> pouring
> >> data
> >> >> >into it for a week without issue, have learnt several thing along
> the
> >> way
> >> >> >and solved all the problems up to now by searching online, but now
> I'm
> >> >> >stuck.  One of the data nodes decided to have a load of 70+ this
> >> morning,
> >> >> >stopping datanode and tasktracker brought it back to normal, but
> every
> >> >> time
> >> >> >I start the datanode again the load shoots through the roof, and
> all I
> >> get
> >> >> >in the logs is :
> >> >> >
> >> >> >STARTUP_MSG: Starting DataNode
> >> >> >
> >> >> >
> >> >> >STARTUP_MSG:   host = pl464/10.20.16.64
> >> >> >
> >> >> >
> >> >> >STARTUP_MSG:   args = []
> >> >> >
> >> >> >
> >> >> >STARTUP_MSG:   version = 0.20.2-cdh3u3
> >> >> >
> >> >> >
> >> >> >STARTUP_MSG:   build =
> >> >>
> >> >>
> >>
> >file:///data/1/tmp/nightly_2012-03-20_13-13-48_3/hadoop-0.20-0.20.2+923.197-1~squeeze
> >> >> >-/
> >> >> >
> >> >> >
> >> >> >2012-05-09 16:12:05,925 INFO
> >> >> >org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
> >> >> already
> >> >> >set up for Hadoop, not re-installing.
> >> >> >
> >> >> >2012-05-09 16:12:06,139 INFO
> >> >> >org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
> >> >> already
> >> >> >set up for Hadoop, not re-installing.
> >> >> >
> >> >> >Nothing else.
> >> >> >
> >> >> >The load seems to max out only 1 of the CPUs, but the machine
> becomes
> >> >> >*very* unresponsive
> >> >> >
> >> >> >Anybody got any pointers of things I can try?
> >> >> >
> >> >> >Thanks
> >> >> >Darrel

RE: job Killing itself upon some condition

[Devaraj k]

Hi Tousif,

 You can kill the Running Job using the killJob() client API. 

If you want to kill the job itself, you can get the job id using task attempt 
id from map() or reduce() functions, and you can invoke the killJob() API based 
on your condition.


Thanks
Devaraj


From: tousif [tousif.pa...@gmail.com]
Sent: Thursday, May 10, 2012 3:15 PM
To: common-user@hadoop.apache.org
Subject: job Killing itself upon some condition

Hi,

i have scheduled a map reduce job and i want this map reduce job to kill
itself upon some condition. overall it should fail.  Here I'm not using
run method of tool runner.

Re: job Killing itself upon some condition

[tousif]

On Thursday 10 May 2012 04:25 PM, Devaraj k wrote:

Hi Tousif,

  You can kill the Running Job using the killJob() client API.

If you want to kill the job itself, you can get the job id using task attempt 
id from map() or reduce() functions, and you can invoke the killJob() API based 
on your condition.


Thanks
Devaraj


From: tousif [tousif.pa...@gmail.com]
Sent: Thursday, May 10, 2012 3:15 PM
To: common-user@hadoop.apache.org
Subject: job Killing itself upon some condition

Hi,

i have scheduled a map reduce job and i want this map reduce job to kill
itself upon some condition. overall it should fail.  Here I'm not using
run method of tool runner.

Thank you very much.

Re: SQL analysis

[Shi Yu]

It depends on your use case, for example, query only or you have 
requirement of real time insert and update.  The solutions can 
be different. 

You might need consider HBase, Cassandra or tools like Flume. 

RE: SQL analysis

[karanveer.singh]

Our focus as of now is on batch queries. And keen to explore on the approach to 
get to the path analysis. 

Regards,
Karanveer

-Original Message-
From: Shi Yu [mailto:sh...@uchicago.edu] 
Sent: 10 May 2012 17:02
To: common-user@hadoop.apache.org
Subject: Re: SQL analysis 

It depends on your use case, for example, query only or you have 
requirement of real time insert and update.  The solutions can 
be different. 

You might need consider HBase, Cassandra or tools like Flume. 
This e-mail and any attachments are confidential and intended
solely for the addressee and may also be privileged or exempt from
disclosure under applicable law. If you are not the addressee, or
have received this e-mail in error, please notify the sender
immediately, delete it from your system and do not copy, disclose
or otherwise act upon any part of this e-mail or its attachments.

Internet communications are not guaranteed to be secure or
virus-free.
The Barclays Group does not accept responsibility for any loss
arising from unauthorised access to, or interference with, any
Internet communications by any third party, or from the
transmission of any viruses. Replies to this e-mail may be
monitored by the Barclays Group for operational or business
reasons.

Any opinion or other information in this e-mail or its attachments
that does not relate to the business of the Barclays Group is
personal to the sender and is not given or endorsed by the Barclays
Group.

Barclays Bank PLC. Registered in England and Wales (registered no.
1026167).
Registered Office: 1 Churchill Place, London, E14 5HP, United
Kingdom.

Barclays Bank PLC is authorised and regulated by the Financial
Services Authority.

RE: SQL analysis

[Shi Yu]

Flume might be suitable for your case.

https://cwiki.apache.org/FLUME/

Shi 

RE: SQL analysis

[karanveer.singh]

I am more worried about the analysis assuming this data is in HDFS.


-Original Message-
From: Shi Yu [mailto:sh...@uchicago.edu] 
Sent: 10 May 2012 18:58
To: common-user@hadoop.apache.org
Subject: RE: SQL analysis 

Flume might be suitable for your case.

https://cwiki.apache.org/FLUME/

Shi 
This e-mail and any attachments are confidential and intended
solely for the addressee and may also be privileged or exempt from
disclosure under applicable law. If you are not the addressee, or
have received this e-mail in error, please notify the sender
immediately, delete it from your system and do not copy, disclose
or otherwise act upon any part of this e-mail or its attachments.

Internet communications are not guaranteed to be secure or
virus-free.
The Barclays Group does not accept responsibility for any loss
arising from unauthorised access to, or interference with, any
Internet communications by any third party, or from the
transmission of any viruses. Replies to this e-mail may be
monitored by the Barclays Group for operational or business
reasons.

Any opinion or other information in this e-mail or its attachments
that does not relate to the business of the Barclays Group is
personal to the sender and is not given or endorsed by the Barclays
Group.

Barclays Bank PLC. Registered in England and Wales (registered no.
1026167).
Registered Office: 1 Churchill Place, London, E14 5HP, United
Kingdom.

Barclays Bank PLC is authorised and regulated by the Financial
Services Authority.

How to start up datanode with kerberos?

[shixing]

Hi，all：
Now I want to setup the security with hbase by kerberos.
As I know, the hbase's ugi is based on the hadoop UserGroupInformation
without parameter "hadoop.job.ugi" after 0.20.2. So when I use the cdh3u3,
the ugi can be generated by two authentication : simple or kerberos.
Firstly I should setup the hdfs based on kerberos. After I setup KDC,
and configuration account for kerberos, I can't start datanode for this
message:

12/05/10 22:41:10 INFO security.UserGroupInformation: Login successful
for user shubao.sx/dwbasis130001.sqa.cm4.tbsite@tbsite.net using keytab
file /home/shubao.sx/hadoop-0.20.2-cdh3u3/conf/kadm5.keytab
12/05/10 22:41:10 ERROR datanode.DataNode: java.lang.RuntimeException:
Cannot start secure cluster without privileged resources. In a secure
cluster, the DataNode must be started from within jsvc. If using Cloudera
packages, please install the hadoop-0.20-sbin package.

For development purposes ONLY you may override this check by setting
dfs.datanode.require.secure.ports to false. *** THIS WILL OPEN A SECURITY
HOLE AND MUST NOT BE USED FOR A REAL CLUSTER ***.
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:331)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:305)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1606)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1546)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1564)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1690)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1707)

And I install the jsvc command to start the datanode as followed(May be it
is wrong?), there is no log or error:
/home/shubao.sx/commons-daemon-1.0.10/jsvc -Dproc_datanode -Xmx1000m
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote
-Dhadoop.log.dir=/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../logs
-Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/..
-Dhadoop.id.str=shubao.sx -Dhadoop.root.logger=INFO,console
-Djava.library.path=/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/native/Linux-amd64-64
-Dhadoop.policy.file=hadoop-policy.xml -classpath
/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../conf:/home/shubao.sx/java6_64/lib/tools.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../hadoop-core-0.20.2-cdh3u3.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/ant-contrib-1.0b3.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/aspectjrt-1.6.5.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/aspectjtools-1.6.5.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/commons-cli-1.2.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/commons-codec-1.4.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/commons-daemon-1.0.1.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/commons-el-1.0.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/commons-httpclient-3.1.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/commons-lang-2.4.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/commons-logging-1.0.4.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/commons-logging-api-1.0.4.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/commons-net-1.4.1.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/core-3.1.1.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/guava-r09-jarjar.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/hadoop-fairscheduler-0.20.2-cdh3u3.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/hsqldb-1.8.0.10.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/jackson-core-asl-1.5.2.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/jackson-mapper-asl-1.5.2.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/jasper-compiler-5.5.12.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/jasper-runtime-5.5.12.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/jets3t-0.6.1.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/jetty-6.1.26.cloudera.1.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/jetty-util-6.1.26.cloudera.1.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/jsch-0.1.42.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/junit-4.5.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/kfs-0.2.2.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/log4j-1.2.15.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/mockito-all-1.8.2.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/oro-2.0.8.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/servlet-api-2.5-20081211.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/slf4j-api-1.4.3.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/slf4j-log4j12-1.4.3.jar:/home/shubao.sx/hadoop-0.20.2-cdh3u3/bin/../lib/xml

Re: SQL analysis

[Shi Yu]

If the analysis you mention is to create "view" of multiple tables. Once 
your data is sorted by the keys in HDFS. You could try Map Side join or 
Reducer Side join in Hadoop to generate the "view" of your data (same 
keys of multiple data sets are combined). There are many code samples 
web, play it around might help.


If you want further analysis like Business Intelligence, then you need 
to train various models.




On 5/10/2012 8:30 AM, karanveer.si...@barclays.com wrote:

I am more worried about the analysis assuming this data is in HDFS.


-Original Message-
From: Shi Yu [mailto:sh...@uchicago.edu]
Sent: 10 May 2012 18:58
To: common-user@hadoop.apache.org
Subject: RE: SQL analysis

Flume might be suitable for your case.

https://cwiki.apache.org/FLUME/

Shi
This e-mail and any attachments are confidential and intended
solely for the addressee and may also be privileged or exempt from
disclosure under applicable law. If you are not the addressee, or
have received this e-mail in error, please notify the sender
immediately, delete it from your system and do not copy, disclose
or otherwise act upon any part of this e-mail or its attachments.

Internet communications are not guaranteed to be secure or
virus-free.
The Barclays Group does not accept responsibility for any loss
arising from unauthorised access to, or interference with, any
Internet communications by any third party, or from the
transmission of any viruses. Replies to this e-mail may be
monitored by the Barclays Group for operational or business
reasons.

Any opinion or other information in this e-mail or its attachments
that does not relate to the business of the Barclays Group is
personal to the sender and is not given or endorsed by the Barclays
Group.

Barclays Bank PLC. Registered in England and Wales (registered no.
1026167).
Registered Office: 1 Churchill Place, London, E14 5HP, United
Kingdom.

Barclays Bank PLC is authorised and regulated by the Financial
Services Authority.

Re: loading a class at job runtime

[Pat Ferrel]

Thanks, that's where I started but #3, which has several methods I 
tried, did not work and the reference claims they are not currently 
supported in cloudera's implementation. Not sure why so I don't know if 
it's a hadoop thing or not.


I ended up putting the classes in the job jar (#2 of cloudera ref), 
which isn't ideal since I'm using mahout's jobs and I really didn't want 
to rebuild that all the time but oh well it works.


On 5/8/12 11:02 PM, JunYong Li wrote:

hope useful to you:
http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/

2012/5/9 Pat Ferrel


I specify a class at runtime via a parameter to a job. I used to put a jar
with the possible class choices in $HADOOP_HOME/lib on all machines (using
only one in a psuedo cluster for this example) then restart my cluster and
run my job. Then I upgraded from 0.20.203 to 0.20.205, which may have
nothing to do with the problem but there it is.

In any case the job cannot find the class in $HADOOP_HOME/lib. I checked
the jar and its in the right place and has my classes in it.

Did I miss something? Is there a better way?

Sorry ahead of time, this must be a repeat of a repeat but the above is
the best advice I can find.

Re: High load on datanode startup

[Raj Vishwanathan]

Darrell

Are the new dn,nn and mapred directories on the same physical disk? Nothing on 
NFS , correct?

Could you be having some hardware issue? Any clue in /var/log/messages or dmesg?

A non responsive system indicates a CPU that is really busy either doing 
something or waiting for something and the fact that it happens only on some 
nodes indicates a local problem.

Raj



>
> From: Darrell Taylor 
>To: common-user@hadoop.apache.org 
>Cc: Raj Vishwanathan  
>Sent: Thursday, May 10, 2012 3:57 AM
>Subject: Re: High load on datanode startup
> 
>On Thu, May 10, 2012 at 9:33 AM, Todd Lipcon  wrote:
>
>> That's real weird..
>>
>> If you can reproduce this after a reboot, I'd recommend letting the DN
>> run for a minute, and then capturing a "jstack " as well as
>> the output of "top -H -p  -b -n 5" and send it to the list.
>
>
>What I did after the reboot this morning was to move the my dn, nn, and
>mapred directories out of the the way, create a new one, formatted it, and
>restarted the node, it's now happy.
>
>I'll try moving the directories back later and do the jstack as you suggest.
>
>
>>
>> What JVM/JDK are you using? What OS version?
>>
>
>root@pl446:/# dpkg --get-selections | grep java
>java-common                                     install
>libjaxp1.3-java                                 install
>libjaxp1.3-java-gcj                             install
>libmysql-java                                   install
>libxerces2-java                                 install
>libxerces2-java-gcj                             install
>sun-java6-bin                                   install
>sun-java6-javadb                                install
>sun-java6-jdk                                   install
>sun-java6-jre                                   install
>
>root@pl446:/# java -version
>java version "1.6.0_26"
>Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
>Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
>
>root@pl446:/# cat /etc/issue
>Debian GNU/Linux 6.0 \n \l
>
>
>
>>
>> -Todd
>>
>>
>> On Wed, May 9, 2012 at 11:57 PM, Darrell Taylor
>>  wrote:
>> > On Wed, May 9, 2012 at 10:52 PM, Raj Vishwanathan 
>> wrote:
>> >
>> >> The picture either too small or too pixelated for my eyes :-)
>> >>
>> >
>> > There should be a zoom option in the top right of the page that allows
>> you
>> > to view it full size
>> >
>> >
>> >>
>> >> Can you login to the box and send the output of top? If the system is
>> >> unresponsive, it has to be something more than an unbalanced hdfs
>> cluster,
>> >> methinks.
>> >>
>> >
>> > Sorry, I'm unable to login to the box, it's completely unresponsive.
>> >
>> >
>> >>
>> >> Raj
>> >>
>> >>
>> >>
>> >> >
>> >> > From: Darrell Taylor 
>> >> >To: common-user@hadoop.apache.org; Raj Vishwanathan > >
>> >> >Sent: Wednesday, May 9, 2012 2:40 PM
>> >> >Subject: Re: High load on datanode startup
>> >> >
>> >> >On Wed, May 9, 2012 at 10:23 PM, Raj Vishwanathan 
>> >> wrote:
>> >> >
>> >> >> When you say 'load', what do you mean? CPU load or something else?
>> >> >>
>> >> >
>> >> >I mean in the unix sense of load average, i.e. top would show a load of
>> >> >(currently) 376.
>> >> >
>> >> >Looking at Ganglia stats for the box it's not CPU load as such, the
>> graphs
>> >> >shows actual CPU usage as 30%, but the number of running processes is
>> >> >simply growing in a linear manner - screen shot of ganglia page here :
>> >> >
>> >> >
>> >>
>> https://picasaweb.google.com/lh/photo/Q0uFSzyLiriDuDnvyRUikXVR0iWwMibMfH0upnTwi28?feat=directlink
>> >> >
>> >> >
>> >> >
>> >> >>
>> >> >> Raj
>> >> >>
>> >> >>
>> >> >>
>> >> >> >
>> >> >> > From: Darrell Taylor 
>> >> >> >To: common-user@hadoop.apache.org
>> >> >> >Sent: Wednesday, May 9, 2012 9:52 AM
>> >> >> >Subject: High load on datanode startup
>> >> >> >
>> >> >> >Hi,
>> >> >> >
>> >> >> >I wonder if someone could give some pointers with a problem I'm
>> having?
>> >> >> >
>> >> >> >I have a 7 machine cluster setup for testing and we have been
>> pouring
>> >> data
>> >> >> >into it for a week without issue, have learnt several thing along
>> the
>> >> way
>> >> >> >and solved all the problems up to now by searching online, but now
>> I'm
>> >> >> >stuck.  One of the data nodes decided to have a load of 70+ this
>> >> morning,
>> >> >> >stopping datanode and tasktracker brought it back to normal, but
>> every
>> >> >> time
>> >> >> >I start the datanode again the load shoots through the roof, and
>> all I
>> >> get
>> >> >> >in the logs is :
>> >> >> >
>> >> >> >STARTUP_MSG: Starting DataNode
>> >> >> >
>> >> >> >
>> >> >> >STARTUP_MSG:   host = pl464/10.20.16.64
>> >> >> >
>> >> >> >
>> >> >> >STARTUP_MSG:   args = []
>> >> >> >
>> >> >> >
>> >> >> >STARTUP_MSG:   version = 0.20.2-cdh3u3
>> >> >> >
>> >> >> >
>> >> >> >STARTUP_MSG:   build =
>> >> >>
>> >> >>
>> >>
>> >file:///data/1/tmp/nightly_2012-03-20_13-13-48_3/hadoop-0.20-0.20.2+

Re: reducer out of memory?

[Yang]

thanks, let me try this


On Wed, May 9, 2012 at 11:27 PM, Zizon Qiu  wrote:
> try setting a lower value for mapred.job.shuffle.input.buffer.percent .
> the reducer used it to decide whether use in-memory shuffle.
> the default value is 0.7,meaning 70% of the "memory" are used as shuffle
> buffer.
>
> On Thu, May 10, 2012 at 2:50 AM, Yang  wrote:
>
>> it seems that if I put too many records into the same mapper output
>> key, all these records are grouped into one key one one reducer,
>>
>> then the reducer became out of memory.
>>
>>
>> but the reducer interface is:
>>
>>       public void reduce(K key, Iterator values,
>>                          OutputCollector output,
>>                          Reporter reporter)
>>
>>
>> so  all the values belonging to the key can be iterated, so
>> theoretically they can be iterated from disk, and does not have to be
>> in memory at the same time,
>> so why am I getting out of heap error? is there some param I could
>> tune (apart from -Xmx since my box is ultimately bounded in memory
>> capacity)
>>
>> thanks
>> Yang
>>

Re: reducer out of memory?

[Yang]

thanks, let me run more of this with the settings provided later in
this thread and provide the details

On Wed, May 9, 2012 at 10:12 PM, Harsh J  wrote:
> Can you share your job details (or a sample reducer code) and also
> share your exact error?
>
> If you are holding reducer provided values/keys in memory in your
> implementation, it can easily cause an OOME if not handled properly.
> The reducer by itself does read the values off a sorted file on the
> disk and doesn't cache the whole group in memory.
>
> On Thu, May 10, 2012 at 12:20 AM, Yang  wrote:
>> it seems that if I put too many records into the same mapper output
>> key, all these records are grouped into one key one one reducer,
>>
>> then the reducer became out of memory.
>>
>>
>> but the reducer interface is:
>>
>>       public void reduce(K key, Iterator values,
>>                          OutputCollector output,
>>                          Reporter reporter)
>>
>>
>> so  all the values belonging to the key can be iterated, so
>> theoretically they can be iterated from disk, and does not have to be
>> in memory at the same time,
>> so why am I getting out of heap error? is there some param I could
>> tune (apart from -Xmx since my box is ultimately bounded in memory
>> capacity)
>>
>> thanks
>> Yang
>
>
>
> --
> Harsh J

Exception closing file

[Aleksandr Elbakyan]

Hello,

I am running the job, when in two similar size clusters, in one it is running 
fine on other gives fallowing exceptions.

I checked the safe mode is off. 

Not sure what do I need fix to avoid this?

2012-05-10 00:38:02,637 ERROR org.apache.hadoop.hdfs.DFSClient: Exception 
closing file /user/_temporary/_attempt_201204251940_0258_r_07_1/part-7 
: 
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Could not complete 
write to file 
/user/_temporary/_attempt_201204251940_0258_r_07_1/part-7 by 
DFSClient_attempt_201204251940_0258_r_07_1
at org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:497)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:966)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:962)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:960)

org.apache.hadoop.ipc.RemoteException: java.io.IOException: Could not complete 
write to file 
/user/_temporary/_attempt_201204251940_0258_r_07_1/part-7 by 
DFSClient_attempt_201204251940_0258_r_07_1
at org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:497)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:966)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:962)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:960)

at org.apache.hadoop.ipc.Client.call(Client.java:740)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.complete(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3280)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3204)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.close(DFSClient.java:1034)
at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:241)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:269)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:1531)
at 
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:1547)



Regards.

Re: Monitoring Hadoop Cluster

[Manu S]

Cool :) Thanks for your quick reply.


On Fri, May 11, 2012 at 11:07 AM, v v  wrote:

> Ganglia won't send alerts, but it is a very good monitoring tool for
> viewing the metrics (hadoop has support for emitting metrics to
> ganglia).
>
> If you need alerting (eg, dfs namespace > 90%) then you will have to
> use nagios or something similar to it.
>
> I would say, you need both.
>
> On May 11, 10:31 am, Manu S  wrote:
> > Hi All,
> >
> > Which is the best monitoring tool for Hadoop cluster monitoring? Ganglia
> or
> > Nagios?
> >
> > Thanks,
> > Manu S
>

Re: Monitoring Hadoop Cluster

[JunYong Li]

Each has its own merits.
http://developer.yahoo.com/hadoop/tutorial/module7.html#monitoring

2012/5/11 Manu S 

> Hi All,
>
> Which is the best monitoring tool for Hadoop cluster monitoring? Ganglia or
> Nagios?
>
> Thanks,
> Manu S
>



-- 
Regards
Junyong

Re: Exception closing file

[JunYong Li]

The client is done writing data to the given filename, and would like to
complete it. The function returns whether the file has been closed
successfully. If the function returns false, the caller should try again. A
call to complete() will not return true until all the file's blocks have
been replicated the minimum number of times. Thus, DataNode failures may
cause a client to call complete() several times before succeeding.

so you should check datanodes have errors in log.

2012/5/11 Aleksandr Elbakyan 

> Hello,
>
> I am running the job, when in two similar size clusters, in one it is
> running fine on other gives fallowing exceptions.
>
> I checked the safe mode is off.
>
> Not sure what do I need fix to avoid this?
>
> 2012-05-10 00:38:02,637 ERROR org.apache.hadoop.hdfs.DFSClient: Exception
> closing file
> /user/_temporary/_attempt_201204251940_0258_r_07_1/part-7 :
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Could not
> complete write to file
> /user/_temporary/_attempt_201204251940_0258_r_07_1/part-7 by
> DFSClient_attempt_201204251940_0258_r_07_1
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:497)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:966)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:962)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:960)
>
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Could not
> complete write to file
> /user/_temporary/_attempt_201204251940_0258_r_07_1/part-7 by
> DFSClient_attempt_201204251940_0258_r_07_1
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:497)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:966)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:962)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:960)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:740)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy1.complete(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy1.complete(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3280)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3204)
> at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.close(DFSClient.java:1034)
> at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:241)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:269)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:1531)
> at
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:1547)
>
>
>
> Regards.
>
>


-- 
Regards
Junyong

Re: loading a class at job runtime

[JunYong Li]

i have encounter CLassNotFound mant times and the reason is the class dose
really not exits in the classpath.
the computer can't go wrong, so calm down, look it over

2012/5/11 Pat Ferrel 

> Thanks, that's where I started but #3, which has several methods I tried,
> did not work and the reference claims they are not currently supported in
> cloudera's implementation. Not sure why so I don't know if it's a hadoop
> thing or not.
>
> I ended up putting the classes in the job jar (#2 of cloudera ref), which
> isn't ideal since I'm using mahout's jobs and I really didn't want to
> rebuild that all the time but oh well it works.
>
> On 5/8/12 11:02 PM, JunYong Li wrote:
>
>> hope useful to you:
>> http://www.cloudera.com/blog/**2011/01/how-to-include-third-**
>> party-libraries-in-your-map-**reduce-job/
>>
>> 2012/5/9 Pat Ferrel
>>
>>  I specify a class at runtime via a parameter to a job. I used to put a
>>> jar
>>> with the possible class choices in $HADOOP_HOME/lib on all machines
>>> (using
>>> only one in a psuedo cluster for this example) then restart my cluster
>>> and
>>> run my job. Then I upgraded from 0.20.203 to 0.20.205, which may have
>>> nothing to do with the problem but there it is.
>>>
>>> In any case the job cannot find the class in $HADOOP_HOME/lib. I checked
>>> the jar and its in the right place and has my classes in it.
>>>
>>> Did I miss something? Is there a better way?
>>>
>>> Sorry ahead of time, this must be a repeat of a repeat but the above is
>>> the best advice I can find.
>>>
>>>
>>>
>>>
>>


-- 
Regards
Junyong

Re: Monitoring Hadoop Cluster

[Manu S]

Thanks a lot Junyong

On Fri, May 11, 2012 at 11:15 AM, JunYong Li  wrote:

> Each has its own merits.
> http://developer.yahoo.com/hadoop/tutorial/module7.html#monitoring
>
> 2012/5/11 Manu S 
>
> > Hi All,
> >
> > Which is the best monitoring tool for Hadoop cluster monitoring? Ganglia
> or
> > Nagios?
> >
> > Thanks,
> > Manu S
> >
>
>
>
> --
> Regards
> Junyong
>