RE: Debugging YARN AM

[Devaraj k]

Hi Curtis,

 "yarn.app.mapreduce.am.command-opts" configuration is specific to 
MRAppMaster. It is not applicable for DistributedShell AM.

If you want to dump out debug information then you can make use of the debug 
option of DistributedShell application. If you want to debug by connecting 
remotely, you need to update the DS application code accordingly.

Thanks
Devaraj K

From: Curtis Ullerich [mailto:curtisuller...@gmail.com]
Sent: 18 June 2013 08:19
To: user@hadoop.apache.org
Subject: Debugging YARN AM

Hi all,

I can successfully debug the MapReduce ApplicationMaster in standalone mode by 
launching the pi estimator example with this command:

hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi 
"-Dyarn.app.mapreduce.am.command-opts=-Xdebug 
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10

and then attaching a debugger to port 8000 using Eclipse. This doesn't work 
with the DistributedShell example, presumably because it's not configurable 
with yarn.app.mapreduce.am.command as it's not MapReduce. Looking in 
yarn-default.xml, I don't see an equivalent parameter. For learning purposes, 
how can I debug the DistributedShell example (and other AMs)?

Thanks!

Curtis

Re: Environment variable representing classpath for AM launch

[Rahul Bhattacharjee]

Thanks Devraj,

Yes , the default yarn site has this property already , I have not
overridden this. The $CLASSPATH environment should have the proper
classpath set with the locations from this property.I was under the
impression that setting $CLASSPATH to the environment  of application
launch context should be enough.Its not working for me. Will look more into
it.






On Tue, Jun 18, 2013 at 10:50 AM, Devaraj k  wrote:

>  Hi Rahul,
>
> ** **
>
> You can make use of the below configuration to set up the
> launch context for your application master. 
>
> ** **
>
>   
>
>   
>
>   
>
> CLASSPATH for YARN applications. A comma-separated list**
> **
>
> of CLASSPATH entries
>
>  yarn.application.classpath
>
>  
> $HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
> 
>
>   
>
> ** **
>
> Thanks
>
> Devaraj K
>
> ** **
>
> *From:* Rahul Bhattacharjee [mailto:rahul.rec@gmail.com]
> *Sent:* 18 June 2013 10:16
> *To:* user@hadoop.apache.org
> *Subject:* Environment variable representing classpath for AM launch
>
> ** **
>
> Hi,
>
> Is ther's any environment variable (available in all nodes of an Yarn
> cluster) which represents a java classpath containing all the core jars of
> yarn. I was thinking to use that variable to setup the environment where to
> run the application master.
>
> Thanks,
>
> Rahul
>

Submitting jobs programmatically (even from Windows, IDE...)

[Vjeran Marcinko]

Hi,

 

As a newcomer to Hadoop, I really struggled in the beginning to set things
up with running jobs from my IDE, and after I dove deep into how things
works, I discovered there are plenty of things that one has to take care,
and many parameters that can influence one's mind on how to best deploy the
jobs. Most of them I described in my blog with some additional utility code
that can help many to submit their jobs, together with any necessary 3rd
party libs:

 

http://pannoniancoder.blogspot.com/2013/06/submitting-hadoop-jobs-programmat
ically.html

 

Moreover, I develop on Windows (don't ask me why), so one more thing to
worry about since Hadoop has not been developed with that OS in mind the
first place.

 

Regards,

Vjeran

 

 

About eclipse plugin

[Ramya S]

Hi All,
 
Does hadoop-2.0.5-alpha-eclipse-plugin.jar is available from any site>?
 
Ramya

RE: Environment variable representing classpath for AM launch

[Devaraj k]

Hi Rahul,

You can make use of the below configuration to set up the 
launch context for your application master.

  

  
CLASSPATH for YARN applications. A comma-separated list
of CLASSPATH entries
 yarn.application.classpath
 
$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
  

Thanks
Devaraj K

From: Rahul Bhattacharjee [mailto:rahul.rec@gmail.com]
Sent: 18 June 2013 10:16
To: user@hadoop.apache.org
Subject: Environment variable representing classpath for AM launch

Hi,
Is ther's any environment variable (available in all nodes of an Yarn cluster) 
which represents a java classpath containing all the core jars of yarn. I was 
thinking to use that variable to setup the environment where to run the 
application master.
Thanks,
Rahul

Environment variable representing classpath for AM launch

[Rahul Bhattacharjee]

Hi,

Is ther's any environment variable (available in all nodes of an Yarn
cluster) which represents a java classpath containing all the core jars of
yarn. I was thinking to use that variable to setup the environment where to
run the application master.

Thanks,
Rahul

Debugging YARN AM

[Curtis Ullerich]

Hi all,

I can successfully debug the MapReduce ApplicationMaster in standalone mode
by launching the pi estimator example with this command:

hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi
"-Dyarn.app.mapreduce.am.command-opts=-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000" 10 10

and then attaching a debugger to port 8000 using Eclipse. This doesn't work
with the DistributedShell example, presumably because it's not configurable
with yarn.app.mapreduce.am.command as it's not MapReduce. Looking in
yarn-default.xml, I don't see an equivalent parameter. For learning
purposes, how can I debug the DistributedShell example (and other AMs)?

Thanks!

Curtis

Re: Setting up secure Hadoop using whosso

[Harsh J]

I've not used it but http://whosso.com/?page_id=92 suggests they did
go the plugin auth route.

On Mon, Jun 17, 2013 at 7:53 PM, Tom Tran  wrote:
> Hello All,
>
> I am planning to setup a secure Hadoop cluster and found this product whosso
> (www.whosso.com) - which sets up secure Hadoop cluster without requiring
> Kerberos and Ldap.
> We can plugin any login mechanism. Has anyone used it ?
>
> Tom



-- 
Harsh J

RE: Namenode memory usage

[Brahma Reddy Battula]

Can you take heapdump and check ..? Here you can check which objects are using 
how much..



Command : jmap -histo:live namenodepid


From: Personal [nitin.mo...@gmail.com]
Sent: Tuesday, June 18, 2013 7:20 AM
To: user@hadoop.apache.org
Subject: Re: Namenode memory usage


  E Lego



On Mon, Jun 17, 2013 at 1:04 PM, Patai Sangbutsarakum 
mailto:silvianhad...@gmail.com>> wrote:

Hi Hadoopers,

My dedicated Namenode box has 48G of memory, 40G is allocated for NN. HEAP

This is from 50070/dfshealth.jsp
28540193 files and directories, 32324098 blocks = 60864291 total. Heap Size is 
23.34 GB / 38.54 GB (60%)

The Heap is fluctuating between less than 20G up to almost 100%


However, from top command Residence size is constantly at 39G not matter how 
low of the memory usage in defshealth.jsp page
  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
 4628 apps  19   0 40.4g  39g  22m S 132.9 83.9  44821:08 
/usr/java/jdk/jre/bin/java -Dproc_namenode -Xmx4m


Is this the time to upgrade the ram to the namenode box?

I remember the easy rule of thumb is 150 bye of every 1M for blocks+file+dir, 
so 60864291 * 150byte is around 9G. I just don't understand why 40G seems to be 
used up.?
Please educate..

Hope this make sense
P

Re: Unsubscribe

[Michael Aro]

Hi,

I assume from this post you are trying to unsubscribe from the list. Please
send an email to 'user-unsubscr...@hadoop.apache.org' without the quotes to
do so.

Cheers, Mike.


On Mon, Jun 17, 2013 at 12:42 PM, Knollhoff, David <
dknollh...@languageline.com> wrote:

> **
>
> David A Knollhoff
> ***Director Computer Operations and Network Services*
> ***LanguageLine Solutions*
> Phone/Fax:
> E-mail: *u...@languageline.com* 
>
>  [image: Picture (Device Independent Bitmap)]
>  [image:
> Picture (Device Independent Bitmap)]   [image:
> Picture (Device Independent Bitmap)]
>  [image: Picture
> (Device Independent Bitmap)]
>
> NOTE:  This e-mail is confidential and is intended only for the
> recipient(s) listed.  Unauthorized use or disclosure of this e-mail or any
> of the information in it is strictly prohibited.  If you are not a listed
> recipient or someone authorized to receive e-mail on behalf of a listed
> recipient, please reply to the sender that the e-mail was misdirected and
> delete the e-mail.  Thank you.
>
>
<><><><>

Re: Namenode memory usage

[Personal]

E Lego

On Mon, Jun 17, 2013 at 1:04 PM, Patai Sangbutsarakum
 wrote:

> Hi Hadoopers,
> My dedicated Namenode box has 48G of memory, 40G is allocated for NN. HEAP
> This is from 50070/dfshealth.jsp
> *28540193 files and directories, 32324098 blocks = 60864291 total. Heap
> Size is 23.34 GB / 38.54 GB (60%) *
> *The Heap is fluctuating between less than 20G up to almost 100%*
> *
> *
> *
> *
> However, from top command Residence size is constantly at 39G not matter
> how low of the memory usage in defshealth.jsp page
>   PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
>  4628 apps  19   0 40.4g  39g  22m S 132.9 83.9  44821:08
> /usr/java/jdk/jre/bin/java -Dproc_namenode -Xmx4m
> Is this the time to upgrade the ram to the namenode box?
> I remember the easy rule of thumb is 150 bye of every 1M for
> blocks+file+dir, so *60864291 * 150byte is around 9G. I just don't
> understand why 40G seems to be used up.?*
> *Please educate..*
> *
> *
> *Hope this make sense*
> *P*

Re: Namenode memory usage

[Patai Sangbutsarakum]

Chris, Thanks a million; it's the huge relieve.
My next action is turn on the GC and verify if it ever goes to full GC a
lot.

I do appreciate your help.


On Mon, Jun 17, 2013 at 2:56 PM, Chris Nauroth wrote:

> Hello Patai,
>
> The numbers you see there (23.34 GB / 38.54 GB) are the JVM total memory /
> max memory.
>
> The max memory is always going to be equivalent to your -Xmx setting
> (4m).  This is the maximum amount of memory that the JVM will attempt
> to allocate from the OS.
>
> The total memory is the amount of memory that the JVM has allocated right
> now.  This value starts at the value you specified for -Xms (or a low
> default if -Xms is unspecified).  Then, the JVM allocates memory lazily
> throughout the lifetime of the process.  Over time, you'll see the total
> memory gradually grow as needed, eventually stopping at the value of max
> memory.  For the JVMs I've worked with, total memory never goes down (the
> JVM doesn't return memory during the process lifetime), but I believe this
> part is implementation-specific, so you might see different behavior on a
> different JVM.
>
> Relating this back to your original question, I don't think these numbers
> alone strongly indicate a need to upgrade RAM.  If total memory is 23GB,
> then it hasn't yet attempted to use the full 40GB that you've deployed.  If
> you're concerned about this though, you can gather more detailed
> information by enabling GC logging on the process.  If you see a lot of
> full GCs, and it appears that there is still very little memory remaining
> after full GC, then that's a stronger indicator that the process needs more
> RAM.
>
> Hope this helps,
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Mon, Jun 17, 2013 at 1:03 PM, Patai Sangbutsarakum <
> silvianhad...@gmail.com> wrote:
>
>> Hi Hadoopers,
>>
>> My dedicated Namenode box has 48G of memory, 40G is allocated for NN. HEAP
>>
>> This is from 50070/dfshealth.jsp
>> *28540193 files and directories, 32324098 blocks = 60864291 total. Heap
>> Size is 23.34 GB / 38.54 GB (60%) *
>>
>> *The Heap is fluctuating between less than 20G up to almost 100%*
>> *
>> *
>> *
>> *
>> However, from top command Residence size is constantly at 39G not matter
>> how low of the memory usage in defshealth.jsp page
>>   PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
>>
>>  4628 apps  19   0 40.4g  39g  22m S 132.9 83.9  44821:08
>> /usr/java/jdk/jre/bin/java -Dproc_namenode -Xmx4m
>>
>>
>> Is this the time to upgrade the ram to the namenode box?
>>
>> I remember the easy rule of thumb is 150 bye of every 1M for
>> blocks+file+dir, so *60864291 * 150byte is around 9G. I just don't
>> understand why 40G seems to be used up.?*
>> *Please educate..*
>> *
>> *
>> *Hope this make sense*
>> *P*
>>
>
>

Re: how to get the mapreduce code which was pig/hive script translated to?

[Bharati]

If I want to access and read the map classes that have been created where do I 
look for them.


Thanks,
Bharati

Sent from my iPad

On Jun 16, 2013, at 4:06 PM, Lance Norskog  wrote:

> Both Pig and Hive have an 'explain plan' command that prints a schematic 
> version. This might make it easier to see what M/R algorithms are used. 
> Mostly the data goes through single-threaded transforms inside a mapper or 
> reducer. 
> 
> https://cwiki.apache.org/Hive/languagemanual-explain.html
> 
> On 06/16/2013 09:17 AM, Edward Capriolo wrote:
>> Hive serializes the entire plan into an XML file if you set the log 4j 
>> settings to debug you should get the locations to the files itgenerates 
>> before launching the job.
>> 
>> 
>> On Sun, Jun 16, 2013 at 11:08 AM, Harsh J  wrote:
>>> This is a question for the Hive/Pig lists to answer best.
>>> 
>>> Note though that they only compile a plan, not the code. The code is
>>> available already, the compiled plan just structures the execution
>>> flow. If you take a look at the sources, you'll find the bits and
>>> pieces that get linked together depending on the operation.
>>> 
>>> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军  wrote:
>>> > hi,all
>>> >
>>> >  how to get the mapreduce code which was pig/hive script translated 
>>> > to?
>>> >
>>> > Best regards,
>>> > wanghj
>>> >
>>> 
>>> 
>>> 
>>> --
>>> Harsh J
>> 
> 
Fortigate Filtered

Re: Namenode memory usage

[Chris Nauroth]

Hello Patai,

The numbers you see there (23.34 GB / 38.54 GB) are the JVM total memory /
max memory.

The max memory is always going to be equivalent to your -Xmx setting
(4m).  This is the maximum amount of memory that the JVM will attempt
to allocate from the OS.

The total memory is the amount of memory that the JVM has allocated right
now.  This value starts at the value you specified for -Xms (or a low
default if -Xms is unspecified).  Then, the JVM allocates memory lazily
throughout the lifetime of the process.  Over time, you'll see the total
memory gradually grow as needed, eventually stopping at the value of max
memory.  For the JVMs I've worked with, total memory never goes down (the
JVM doesn't return memory during the process lifetime), but I believe this
part is implementation-specific, so you might see different behavior on a
different JVM.

Relating this back to your original question, I don't think these numbers
alone strongly indicate a need to upgrade RAM.  If total memory is 23GB,
then it hasn't yet attempted to use the full 40GB that you've deployed.  If
you're concerned about this though, you can gather more detailed
information by enabling GC logging on the process.  If you see a lot of
full GCs, and it appears that there is still very little memory remaining
after full GC, then that's a stronger indicator that the process needs more
RAM.

Hope this helps,

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Mon, Jun 17, 2013 at 1:03 PM, Patai Sangbutsarakum <
silvianhad...@gmail.com> wrote:

> Hi Hadoopers,
>
> My dedicated Namenode box has 48G of memory, 40G is allocated for NN. HEAP
>
> This is from 50070/dfshealth.jsp
> *28540193 files and directories, 32324098 blocks = 60864291 total. Heap
> Size is 23.34 GB / 38.54 GB (60%) *
>
> *The Heap is fluctuating between less than 20G up to almost 100%*
> *
> *
> *
> *
> However, from top command Residence size is constantly at 39G not matter
> how low of the memory usage in defshealth.jsp page
>   PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
>
>  4628 apps  19   0 40.4g  39g  22m S 132.9 83.9  44821:08
> /usr/java/jdk/jre/bin/java -Dproc_namenode -Xmx4m
>
>
> Is this the time to upgrade the ram to the namenode box?
>
> I remember the easy rule of thumb is 150 bye of every 1M for
> blocks+file+dir, so *60864291 * 150byte is around 9G. I just don't
> understand why 40G seems to be used up.?*
> *Please educate..*
> *
> *
> *Hope this make sense*
> *P*
>

Namenode memory usage

[Patai Sangbutsarakum]

Hi Hadoopers,

My dedicated Namenode box has 48G of memory, 40G is allocated for NN. HEAP

This is from 50070/dfshealth.jsp
*28540193 files and directories, 32324098 blocks = 60864291 total. Heap
Size is 23.34 GB / 38.54 GB (60%) *

*The Heap is fluctuating between less than 20G up to almost 100%*
*
*
*
*
However, from top command Residence size is constantly at 39G not matter
how low of the memory usage in defshealth.jsp page
  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND

 4628 apps  19   0 40.4g  39g  22m S 132.9 83.9  44821:08
/usr/java/jdk/jre/bin/java -Dproc_namenode -Xmx4m


Is this the time to upgrade the ram to the namenode box?

I remember the easy rule of thumb is 150 bye of every 1M for
blocks+file+dir, so *60864291 * 150byte is around 9G. I just don't
understand why 40G seems to be used up.?*
*Please educate..*
*
*
*Hope this make sense*
*P*

Unsubscribe

[Knollhoff, David]

David A Knollhoff
Director Computer Operations and Network Services
LanguageLine Solutions
Phone/Fax: 
E-mail: u...@languageline.com

 <>
  
<>  
<>
  <> 

NOTE:  This e-mail is confidential and is intended only for the
recipient(s) listed.  Unauthorized use or disclosure of this e-mail or
any of the information in it is strictly prohibited.  If you are not a
listed recipient or someone authorized to receive e-mail on behalf of a
listed recipient, please reply to the sender that the e-mail was
misdirected and delete the e-mail.  Thank you.



<><><><>

Re: Eclipse plug-in for Hadoop

[Srimanth Gunturi]

Hi Omkar,
You are right that the Eclipse plugins have to be built from source
(remote cluster in your instance).
It is like that because the Eclipse plugin has HDFS/MR client code
compiled directly into it.

I was unable to find a central update-site/website where Eclipse
plugins were made available.
Regards,
Srimanth





On Mon, Jun 17, 2013 at 2:27 AM, Omkar Joshi
 wrote:
> Hi,
>
>
>
> I have Hadoop 1.0.4 version running on a remote cluster running of Ubuntu
> 12.04 Precise. I have only the command-line access to the cluster machines
> via Putty etc.
>
>
>
> Currently, any MR etc. code that I write using Eclipse running on my local
> Windows machine, I have to build and ship the jar file to the remote cluster
> and execute it.
>
>
>
> Now, I wish to add a plug-in on my local Eclipse which will allow me to
> execute my MR etc. codes on the remote cluster(alike a Java program, a web
> application).
>
>
>
> While I’m facing issues in building the jar from src/contrib/eclipse-plugin
> directory, is my assumption, itself in the first place, correct – build the
> eclipse plug-in jar on the remote cluster machine, ship it to the local
> eclipse and use it to run code on the same remote cluster?
>
>
>
> Regards,
>
> Omkar Joshi
>
>
>
>
> 
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"

Re: Unsubscribe

[Devaraj k]

Hi,

   You need to send a mail to user-unsubscr...@hadoop.apache.org for
unsubscribing from this list.

http://hadoop.apache.org/mailing_lists.html#User


Thanks
Devaraj K
On 6/17/13, Manny Vazquez  wrote:
>
>
> Manuel Vazquez, BI Administrator, IS Engineering
> Apollo Group | Apollo Tech
> 4035 S Riverpoint Parkway | MS: CF-L205| Phoenix, AZ 85040
> Office: 602-557-6979 | Cell: 602-317-1690
> email: manny.vazq...@apollogrp.edu
>
> P Please consider the environment before printing this email.
> http://www.apollogrp.edu
>
>
> 
> This message is private and confidential. If you have received it in error,
> please notify the sender and remove it from your system.
>
>

Unsubscribe

[Manny Vazquez]

Manuel Vazquez, BI Administrator, IS Engineering
Apollo Group | Apollo Tech
4035 S Riverpoint Parkway | MS: CF-L205| Phoenix, AZ 85040
Office: 602-557-6979 | Cell: 602-317-1690
email: manny.vazq...@apollogrp.edu

P Please consider the environment before printing this email.
http://www.apollogrp.edu



This message is private and confidential. If you have received it in error, 
please notify the sender and remove it from your system.

Re: Hadoop database ecosystem overview

[Ted Yu]

John:

The following was one recent case study:

http://nosql.mypopescu.com/post/34305991777/ycsb-benchmark-results-for-cassandra-hbase-mongodb
Cheers

On Mon, Jun 17, 2013 at 7:22 AM, John Lilley wrote:

>  I’d like to find a web site or some slides that clearly delineate the
> “databases” of the Hadoop ecosystem and what they are each good at.  If we
> look at HBase, Hive, and Cassandra (and others?) is it easy to
> differentiate them based on:
>
> **· **Transactional vs batch
>
> **· **Full database operations (insert, delete, update) vs
> query-only vs append-only
>
> **· **Managed storage vs open/flat/file storage
>
> Also it would be good to see how these stack up against GreenPlum and
> MongoDb
>
> Thanks
>
> john 
>
> ** **
>

Setting up secure Hadoop using whosso

[Tom Tran]

Hello All,

I am planning to setup a secure Hadoop cluster and found this product
whosso (www.whosso.com) - which sets up secure Hadoop cluster without
requiring Kerberos and Ldap.
We can plugin any login mechanism. Has anyone used it ?

Tom

Hadoop database ecosystem overview

[John Lilley]

I'd like to find a web site or some slides that clearly delineate the 
"databases" of the Hadoop ecosystem and what they are each good at.  If we look 
at HBase, Hive, and Cassandra (and others?) is it easy to differentiate them 
based on:

* Transactional vs batch

* Full database operations (insert, delete, update) vs query-only vs 
append-only

* Managed storage vs open/flat/file storage
Also it would be good to see how these stack up against GreenPlum and MongoDb
Thanks
john

Re: how to close hadoop when tmp files were cleared

[Azuryy Yu]

Yes Harsh, It's my fault.


On Mon, Jun 17, 2013 at 5:09 PM, Harsh J  wrote:

> Hey Azuryy,
>
> The -9 (SIGKILL) is unnecessary and isn't recommended unless its
> unresponsive. The SIGTERM has an additional benefit of running any
> necessary shutdown handling procedures, but SIGKILL is instant death.
>
> On Mon, Jun 17, 2013 at 2:34 PM, Azuryy Yu  wrote:
> > ps aux|grep java  , you can find pid, then just 'kill -9' to stop the
> Hadoop
> > process.
> >
> >
> > On Mon, Jun 17, 2013 at 4:34 PM, Harsh J  wrote:
> >>
> >> Just send the processes a SIGTERM signal (regular kill). Its what the
> >> script does anyway. Ensure to change the PID directory before the next
> >> restart though.
> >>
> >> On Mon, Jun 17, 2013 at 1:09 PM,   wrote:
> >> >
> >> > Hi，
> >> >
> >> > My hadoop cluster has been running for a period of time.  Now i want
> to
> >> > close it for some system changes. But the command "bin/stop-all.sh"
> >> > shows
> >> > "no jobtracker to stop","no tasktracker to stop","no namenode to stop"
> >> > and
> >> > "no datanode to stop". I use "jps" got nothing but jps itself.
> However,
> >> > hadoop is indeed running.I think it may be some tmp files about hadoop
> >> > had
> >> > been cleared by operation system. Could someone tell me how to stop
> >> > hadoop
> >> > in case of no data files breaks ?
> >> > Any guidance would be greatly appreciated. Thanks!
> >> >
> >> > Jeff
> >> >
> >> >
> >> > 
> >> > ZTE Information Security Notice: The information contained in this
> mail
> >> > (and
> >> > any attachment transmitted herewith) is privileged and confidential
> and
> >> > is
> >> > intended for the exclusive use of the addressee(s).  If you are not an
> >> > intended recipient, any disclosure, reproduction, distribution or
> other
> >> > dissemination or use of the information contained is strictly
> >> > prohibited.
> >> > If you have received this mail in error, please delete it and notify
> us
> >> > immediately.
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Eclipse plug-in for Hadoop

[Omkar Joshi]

Hi,

I have Hadoop 1.0.4 version running on a remote cluster running of Ubuntu 12.04 
Precise. I have only the command-line access to the cluster machines via Putty 
etc.

Currently, any MR etc. code that I write using Eclipse running on my local 
Windows machine, I have to build and ship the jar file to the remote cluster 
and execute it.

Now, I wish to add a plug-in on my local Eclipse which will allow me to execute 
my MR etc. codes on the remote cluster(alike a Java program, a web application).

While I'm facing issues in building the jar from src/contrib/eclipse-plugin 
directory, is my assumption, itself in the first place, correct - build the 
eclipse plug-in jar on the remote cluster machine, ship it to the local eclipse 
and use it to run code on the same remote cluster?

Regards,
Omkar Joshi



The contents of this e-mail and any attachment(s) may contain confidential or 
privileged information for the intended recipient(s). Unintended recipients are 
prohibited from taking action on the basis of information in this e-mail and 
using or disseminating the information, and must notify the sender and delete 
it from their system. L&T Infotech will not accept responsibility or liability 
for the accuracy or completeness of, or the presence of any virus or disabling 
code in this e-mail"

答复: Re: how to close hadoop when tmp files were cleared

[zhang . heng16]

Thanks all. Now i'm using "kill pid" to stop it.






Harsh J  
2013-06-17 17:09
请答复 给
user@hadoop.apache.org


收件人
"" 
抄送

主题
Re: how to close hadoop when tmp files were cleared






Hey Azuryy,

The -9 (SIGKILL) is unnecessary and isn't recommended unless its
unresponsive. The SIGTERM has an additional benefit of running any
necessary shutdown handling procedures, but SIGKILL is instant death.

On Mon, Jun 17, 2013 at 2:34 PM, Azuryy Yu  wrote:
> ps aux|grep java  , you can find pid, then just 'kill -9' to stop the 
Hadoop
> process.
>
>
> On Mon, Jun 17, 2013 at 4:34 PM, Harsh J  wrote:
>>
>> Just send the processes a SIGTERM signal (regular kill). Its what the
>> script does anyway. Ensure to change the PID directory before the next
>> restart though.
>>
>> On Mon, Jun 17, 2013 at 1:09 PM,   wrote:
>> >
>> > Hi，
>> >
>> > My hadoop cluster has been running for a period of time.  Now i want 
to
>> > close it for some system changes. But the command "bin/stop-all.sh"
>> > shows
>> > "no jobtracker to stop","no tasktracker to stop","no namenode to 
stop"
>> > and
>> > "no datanode to stop". I use "jps" got nothing but jps itself. 
However,
>> > hadoop is indeed running.I think it may be some tmp files about 
hadoop
>> > had
>> > been cleared by operation system. Could someone tell me how to stop
>> > hadoop
>> > in case of no data files breaks ?
>> > Any guidance would be greatly appreciated. Thanks!
>> >
>> > Jeff
>> >
>> >
>> > 
>> > ZTE Information Security Notice: The information contained in this 
mail
>> > (and
>> > any attachment transmitted herewith) is privileged and confidential 
and
>> > is
>> > intended for the exclusive use of the addressee(s).  If you are not 
an
>> > intended recipient, any disclosure, reproduction, distribution or 
other
>> > dissemination or use of the information contained is strictly
>> > prohibited.
>> > If you have received this mail in error, please delete it and notify 
us
>> > immediately.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J



ZTE Information Security Notice: The information contained in this mail (and 
any attachment transmitted herewith) is privileged and confidential and is 
intended for the exclusive use of the addressee(s).  If you are not an intended 
recipient, any disclosure, reproduction, distribution or other dissemination or 
use of the information contained is strictly prohibited.  If you have received 
this mail in error, please delete it and notify us immediately.

Re: how to close hadoop when tmp files were cleared

[Harsh J]

Hey Azuryy,

The -9 (SIGKILL) is unnecessary and isn't recommended unless its
unresponsive. The SIGTERM has an additional benefit of running any
necessary shutdown handling procedures, but SIGKILL is instant death.

On Mon, Jun 17, 2013 at 2:34 PM, Azuryy Yu  wrote:
> ps aux|grep java  , you can find pid, then just 'kill -9' to stop the Hadoop
> process.
>
>
> On Mon, Jun 17, 2013 at 4:34 PM, Harsh J  wrote:
>>
>> Just send the processes a SIGTERM signal (regular kill). Its what the
>> script does anyway. Ensure to change the PID directory before the next
>> restart though.
>>
>> On Mon, Jun 17, 2013 at 1:09 PM,   wrote:
>> >
>> > Hi，
>> >
>> > My hadoop cluster has been running for a period of time.  Now i want to
>> > close it for some system changes. But the command "bin/stop-all.sh"
>> > shows
>> > "no jobtracker to stop","no tasktracker to stop","no namenode to stop"
>> > and
>> > "no datanode to stop". I use "jps" got nothing but jps itself. However,
>> > hadoop is indeed running.I think it may be some tmp files about hadoop
>> > had
>> > been cleared by operation system. Could someone tell me how to stop
>> > hadoop
>> > in case of no data files breaks ?
>> > Any guidance would be greatly appreciated. Thanks!
>> >
>> > Jeff
>> >
>> >
>> > 
>> > ZTE Information Security Notice: The information contained in this mail
>> > (and
>> > any attachment transmitted herewith) is privileged and confidential and
>> > is
>> > intended for the exclusive use of the addressee(s).  If you are not an
>> > intended recipient, any disclosure, reproduction, distribution or other
>> > dissemination or use of the information contained is strictly
>> > prohibited.
>> > If you have received this mail in error, please delete it and notify us
>> > immediately.
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: how to close hadoop when tmp files were cleared

[Azuryy Yu]

ps aux|grep java  , you can find pid, then just 'kill -9' to stop the
Hadoop process.


On Mon, Jun 17, 2013 at 4:34 PM, Harsh J  wrote:

> Just send the processes a SIGTERM signal (regular kill). Its what the
> script does anyway. Ensure to change the PID directory before the next
> restart though.
>
> On Mon, Jun 17, 2013 at 1:09 PM,   wrote:
> >
> > Hi，
> >
> > My hadoop cluster has been running for a period of time.  Now i want to
> > close it for some system changes. But the command "bin/stop-all.sh" shows
> > "no jobtracker to stop","no tasktracker to stop","no namenode to stop"
> and
> > "no datanode to stop". I use "jps" got nothing but jps itself. However,
> > hadoop is indeed running.I think it may be some tmp files about hadoop
> had
> > been cleared by operation system. Could someone tell me how to stop
> hadoop
> > in case of no data files breaks ?
> > Any guidance would be greatly appreciated. Thanks!
> >
> > Jeff
> >
> >
> > 
> > ZTE Information Security Notice: The information contained in this mail
> (and
> > any attachment transmitted herewith) is privileged and confidential and
> is
> > intended for the exclusive use of the addressee(s).  If you are not an
> > intended recipient, any disclosure, reproduction, distribution or other
> > dissemination or use of the information contained is strictly prohibited.
> > If you have received this mail in error, please delete it and notify us
> > immediately.
> >
> >
>
>
>
> --
> Harsh J
>

Re: how to close hadoop when tmp files were cleared

[Harsh J]

Just send the processes a SIGTERM signal (regular kill). Its what the
script does anyway. Ensure to change the PID directory before the next
restart though.

On Mon, Jun 17, 2013 at 1:09 PM,   wrote:
>
> Hi，
>
> My hadoop cluster has been running for a period of time.  Now i want to
> close it for some system changes. But the command "bin/stop-all.sh" shows
> "no jobtracker to stop","no tasktracker to stop","no namenode to stop" and
> "no datanode to stop". I use "jps" got nothing but jps itself. However,
> hadoop is indeed running.I think it may be some tmp files about hadoop had
> been cleared by operation system. Could someone tell me how to stop hadoop
> in case of no data files breaks ?
> Any guidance would be greatly appreciated. Thanks!
>
> Jeff
>
>
> 
> ZTE Information Security Notice: The information contained in this mail (and
> any attachment transmitted herewith) is privileged and confidential and is
> intended for the exclusive use of the addressee(s).  If you are not an
> intended recipient, any disclosure, reproduction, distribution or other
> dissemination or use of the information contained is strictly prohibited.
> If you have received this mail in error, please delete it and notify us
> immediately.
>
>



-- 
Harsh J

答复: Re: how to close hadoop when tmp files were cleared

[zhang . heng16]

Yes, i use the same account named "hadoop".

Using "ps aux|grep java" could find hadoop is running, i forgot to change 
the "HADOOP_PID_DIR" parameter, and operation system must cleared the pid 
files.




Azuryy Yu  
2013-06-17 15:51
请答复 给
user@hadoop.apache.org


收件人
user@hadoop.apache.org
抄送

主题
Re: how to close hadoop when tmp files were cleared






you must using the same account that start your cluster. 


On Mon, Jun 17, 2013 at 3:39 PM,  wrote:

Hi， 

My hadoop cluster has been running for a period of time.  Now i want to 
close it for some system changes. But the command "bin/stop-all.sh" shows 
"no jobtracker to stop","no tasktracker to stop","no namenode to stop" and 
"no datanode to stop". I use "jps" got nothing but jps itself. However, 
 hadoop is indeed running.I think it may be some tmp files about hadoop 
had been cleared by operation system. Could someone tell me how to stop 
hadoop in case of no data files breaks ? 
Any guidance would be greatly appreciated. Thanks! 

Jeff 



ZTE Information Security Notice: The information contained in this mail 
(and any attachment transmitted herewith) is privileged and confidential 
and is intended for the exclusive use of the addressee(s).  If you are not 
an intended recipient, any disclosure, reproduction, distribution or other 
dissemination or use of the information contained is strictly prohibited. 
If you have received this mail in error, please delete it and notify us 
immediately.






ZTE Information Security Notice: The information contained in this mail (and 
any attachment transmitted herewith) is privileged and confidential and is 
intended for the exclusive use of the addressee(s).  If you are not an intended 
recipient, any disclosure, reproduction, distribution or other dissemination or 
use of the information contained is strictly prohibited.  If you have received 
this mail in error, please delete it and notify us immediately.

Re: how to close hadoop when tmp files were cleared

[Azuryy Yu]

you must using the same account that start your cluster.


On Mon, Jun 17, 2013 at 3:39 PM,  wrote:

>
> Hi，
>
> My hadoop cluster has been running for a period of time.  Now i want to
> close it for some system changes. But the command "bin/stop-all.sh" shows
> "no jobtracker to stop","no tasktracker to stop","no namenode to stop" and
> "no datanode to stop". I use "jps" got nothing but jps itself. However,
>  hadoop is indeed running.I think it may be some tmp files about hadoop had
> been cleared by operation system. Could someone tell me how to stop hadoop
> in case of no data files breaks ?
> Any guidance would be greatly appreciated. Thanks!
>
> Jeff
>
>
> 
> ZTE Information Security Notice: The information contained in this mail (and 
> any attachment transmitted herewith) is privileged and confidential and is 
> intended for the exclusive use of the addressee(s).  If you are not an 
> intended recipient, any disclosure, reproduction, distribution or other 
> dissemination or use of the information contained is strictly prohibited.  If 
> you have received this mail in error, please delete it and notify us 
> immediately.
>
>
>
>

how to close hadoop when tmp files were cleared

[zhang . heng16]

Hi，

My hadoop cluster has been running for a period of time.  Now i want to 
close it for some system changes. But the command "bin/stop-all.sh" shows 
"no jobtracker to stop","no tasktracker to stop","no namenode to stop" and 
"no datanode to stop". I use "jps" got nothing but jps itself. However, 
hadoop is indeed running.I think it may be some tmp files about hadoop had 
been cleared by operation system. Could someone tell me how to stop hadoop 
in case of no data files breaks ?
Any guidance would be greatly appreciated. Thanks!

Jeff


ZTE Information Security Notice: The information contained in this mail (and 
any attachment transmitted herewith) is privileged and confidential and is 
intended for the exclusive use of the addressee(s).  If you are not an intended 
recipient, any disclosure, reproduction, distribution or other dissemination or 
use of the information contained is strictly prohibited.  If you have received 
this mail in error, please delete it and notify us immediately.