Re: Spark Yarn Cluster with Reference File

2016-09-23 Thread ayan guha
You may try copying the file to same location on all nodes and try to read from that place On 24 Sep 2016 00:20, "ABHISHEK" wrote: > I have tried with hdfs/tmp location but it didn't work. Same error. > > On 23 Sep 2016 19:37, "Aditya"

Re: Spark Yarn Cluster with Reference File

2016-09-23 Thread ABHISHEK
I have tried with hdfs/tmp location but it didn't work. Same error. On 23 Sep 2016 19:37, "Aditya" wrote: > Hi Abhishek, > > Try below spark submit. > spark-submit --master yarn --deploy-mode cluster --files hdfs:// > abc.com:8020/tmp/abc.drl --class

Re: Spark Yarn Cluster with Reference File

2016-09-23 Thread Aditya
Hi Abhishek, Try below spark submit. spark-submit --master yarn --deploy-mode cluster --files hdfs://abc.com:8020/tmp/abc.drl --class com.abc.StartMain abc-0.0.1-SNAPSHOT-jar-with-dependencies.jar abc.drl On Friday 23

Re: Spark Yarn Cluster with Reference File

2016-09-23 Thread ABHISHEK
Thanks for your response Aditya and Steve. Steve: I have tried specifying both /tmp/filename in hdfs and local path but it didn't work. You may be write that Kie session is configured to access files from Local path. I have attached code here for your reference and if you find some thing wrong,

Re: Spark Yarn Cluster with Reference File

2016-09-23 Thread Steve Loughran
On 23 Sep 2016, at 08:33, ABHISHEK > wrote: at java.lang.Thread.run(Thread.java:745) Caused by: java.io.FileNotFoundException: hdfs:/abc.com:8020/user/abhietc/abc.drl (No such file or directory)

Re: Spark Yarn Cluster with Reference File

2016-09-23 Thread Aditya
Hi Abhishek, From your spark-submit it seems your passing the file as a parameter to the driver program. So now it depends what exactly you are doing with that parameter. Using --files option it will be available to all the worker nodes but if in your code if you are referencing using the

Spark Yarn Cluster with Reference File

2016-09-23 Thread ABHISHEK
Hello there, I have Spark Application which refer to an external file ‘abc.drl’ and having unstructured data. Application is able to find this reference file if I run app in Local mode but in Yarn with Cluster mode, it is not able to find the file in the specified path. I tried with both local