Re: running spark program on intellij connecting to remote master for cluster

2017-05-11 Thread s t
Hello David, Let me make it more clear; * There is not any spark installed on windows laptop, just the intellij and the related dependencies. * SparkLauncher is good starting point for submitting a job programatically but i am not sure if my current problem is related with job

Re: running spark program on intellij connecting to remote master for cluster

2017-05-10 Thread David Kaczynski
Do you have Spark installed locally on your laptop with IntelliJ? Are you using the SparkLauncher class or your local spark-submit script? A while back, I was trying to submit a spark job from my local workstation to a remote cluster using the SparkLauncher class, but I didn't actually have

running spark program on intellij connecting to remote master for cluster

2017-05-10 Thread s t
Hello, I am trying to run spark code from my laptop with intellij. I have cluster of 2 nodes and a master. When i start the program from intellij it gets error of some missing classes. I am aware that some jars need to be distributed to the workers but do not know if it is possible