Spark UI and running spark-submit with --master yarn

2015-03-02 Thread Anupama Joshi 
Hi ,


   1.  When I run my application with --master yarn-cluster or --master
   yarn --deploy-mode cluster , I can not  the spark UI at the  location --
   masternode:4040Even if I am running the job , I can not see teh SPARK UI.
   2. When I run with --master yarn --deploy-mode client  -- I see the
   Spark UI but I cannot see my job  running.

When I run spark-submit with --master local[*] , I see the spark UI , my
job everything (Thats great)

Do I need to do some settings to see the UI?

Thanks

-AJ

Re: Spark UI and running spark-submit with --master yarn

2015-03-02 Thread Anupama Joshi 
Hi Marcelo,
Thanks for the quick reply.
I have a EMR cluster and I am running the spark-submit on the master node
in the cluster.
When I start the spark-submit , I see
15/03/02 23:48:33 INFO client.RMProxy: Connecting to ResourceManager at /
172.31.43.254:9022
But If I try that URL or the use the external DNS
ec2-52-10-234-111.us-west-2.compute.amazonaws.com:9022
it does not work
What am I missing here ?
Thanks a lot for the help
-AJ


On Mon, Mar 2, 2015 at 3:50 PM, Marcelo Vanzin van...@cloudera.com wrote:

 What are you calling masternode? In yarn-cluster mode, the driver
 is running somewhere in your cluster, not on the machine where you run
 spark-submit.

 The easiest way to get to the Spark UI when using Yarn is to use the
 Yarn RM's web UI. That will give you a link to the application's UI
 regardless of whether it's running on client or cluster mode.

 On Mon, Mar 2, 2015 at 3:39 PM, Anupama Joshi anupama.jo...@gmail.com
 wrote:
  Hi ,
 
   When I run my application with --master yarn-cluster or --master yarn
  --deploy-mode cluster , I can not  the spark UI at the  location --
  masternode:4040Even if I am running the job , I can not see teh SPARK
 UI.
  When I run with --master yarn --deploy-mode client  -- I see the Spark UI
  but I cannot see my job  running.
 
  When I run spark-submit with --master local[*] , I see the spark UI , my
 job
  everything (Thats great)
 
  Do I need to do some settings to see the UI?
 
  Thanks
 
  -AJ
 
 
 
 
 
 



 --
 Marcelo

sorting output of join operation

2015-02-23 Thread Anupama Joshi 
Hi ,
 To simplify my problem -
I have 2 files from which I reading words.
the o/p is like
file 1
aaa 4
bbb 6
ddd 3

file 2
ddd 2
bbb 6
ttt 5

if I do file1.join(file2)
I get (ddd(3,2)
bbb(6,6)

If I want to sort the output by the number of occurances of the word i file1
. How do I achive that.
Any help would be appreciated.
Thanks
AJ