How to get driver id?

[Xingwei Yang]

Hi Guys:

  I want to kill an application but I could not find the driver id of
the application from web ui. Is there any way to get it from command line?
Thanks

-- 
Sincerely Yours
Xingwei Yang
https://sites.google.com/site/xingweiyang1223/

Transfer from RDD to JavaRDD

[Xingwei Yang]

I use Spark in Java.

I want to access the vectors of RowMatrix M, thus I use M.rows(), which is
a RDDVector

I want to transform it to JavaRDDVector, I used the following command;

 JavaRDDVector data = JavaRDD.fromRDD(M.rows(),
scala.reflect.ClassTag$.MODULE$.apply(Vector.class);

However, it shows a error like this:

The method fromRDD(RDDT, ClassTagT) in the type JavaRDD is not
applicable for the arguments (RDDVector, ClassTagObject)

Is there anything wrong with the method? Thanks a lot.


-- 
Sincerely Yours
Xingwei Yang
https://sites.google.com/site/xingweiyang1223/

spark-ec2 Web UI Problem

[Xingwei Yang]

Hi Guys:

I have succsefully installed apache-spark on Amazon ec2 using spark-ec2
command and I could login to the master node.

Here is the installation message:
RSYNC'ing /etc/ganglia to slaves...
ec2-54-148-197-89.us-west-2.compute.amazonaws.com
Shutting down GANGLIA gmond:
   [FAILED]
Starting GANGLIA gmond:[  OK  ]
Shutting down GANGLIA gmond:   [FAILED]
Starting GANGLIA gmond:[  OK  ]
Connection to ec2-54-148-197-89.us-west-2.compute.amazonaws.com closed.
Shutting down GANGLIA gmetad:  [FAILED]
Starting GANGLIA gmetad:   [  OK  ]
Stopping httpd:[FAILED]
Starting httpd:[  OK  ]
Connection to ec2-54-148-248-162.us-west-2.compute.amazonaws.com closed.
Spark standalone cluster started at
http://ec2-54-148-248-162.us-west-2.compute.amazonaws.com:8080
Ganglia started at
http://ec2-54-148-248-162.us-west-2.compute.amazonaws.com:5080/ganglia



However, I could not open the web UI. I have checked the security group,
which allows the connection to the port 8080. I could not figure out how to
solve it.

Any sueggestion is appreciated. Thanks a lot.


-- 
Sincerely Yours
Xingwei Yang
https://sites.google.com/site/xingweiyang1223/

Unable to run applications on clusters on EC2

[Xingwei Yang]

I think it is related to my previous questions, but I separate them. In my
previous question, I could not connect to WebUI even though I could log
into the cluster without any problem.

Also, I tried lynx localhost:8080 and I could get the information about the
cluster;

I could also user spark-submit to submit job locally by setting master to
localhost


However, I could not submit the job to the cluster master and I get the
error like this:

14/12/04 22:14:39 WARN scheduler.TaskSchedulerImpl: Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
registered and have sufficient memory
14/12/04 22:14:42 INFO client.AppClient$ClientActor: Connecting to master
spark://ec2-54-149-92-187.us-west-2.compute.amazonaws.com:7070...
14/12/04 22:14:42 WARN client.AppClient$ClientActor: Could not connect to
akka.tcp://
sparkmas...@ec2-54-149-92-187.us-west-2.compute.amazonaws.com:7070:
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://
sparkmas...@ec2-54-149-92-187.us-west-2.compute.amazonaws.com:7070]
14/12/04 22:14:42 WARN client.AppClient$ClientActor: Could not connect to
akka.tcp://
sparkmas...@ec2-54-149-92-187.us-west-2.compute.amazonaws.com:7070:
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://
sparkmas...@ec2-54-149-92-187.us-west-2.compute.amazonaws.com:7070]
14/12/04 22:14:42 WARN client.AppClient$ClientActor: Could not connect to
akka.tcp://
sparkmas...@ec2-54-149-92-187.us-west-2.compute.amazonaws.com:7070:
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://
sparkmas...@ec2-54-149-92-187.us-west-2.compute.amazonaws.com:7070]
14/12/04 22:14:42 WARN client.AppClient$ClientActor: Could not connect to
akka.tcp://
sparkmas...@ec2-54-149-92-187.us-west-2.compute.amazonaws.com:7070:
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://
sparkmas...@ec2-54-149-92-187.us-west-2.compute.amazonaws.com:7070]


Please let me know if you any any clue about it. Thanks a lot.

-- 
Sincerely Yours
Xingwei Yang
https://sites.google.com/site/xingweiyang1223/