Re: Spark-submit and multiple files

2015-03-20 Thread Guillaume Charhon
Hi Davies, I am already using --py-files. The system does use the other file. The error I am getting is not trivial. Please check the error log. On Thu, Mar 19, 2015 at 8:03 PM, Davies Liu dav...@databricks.com wrote: You could submit additional Python source via --py-files , for example:

Re: Spark-submit and multiple files

2015-03-20 Thread Davies Liu
You MUST put --py-files BEFORE main.py, as mentioned in another threads. On Fri, Mar 20, 2015 at 1:47 AM, Guillaume Charhon guilla...@databerries.com wrote: Hi Davies, I am already using --py-files. The system does use the other file. The error I am getting is not trivial. Please check the

Re: Spark-submit and multiple files

2015-03-20 Thread Petar Zecevic
I tried your program in yarn-client mode and it worked with no exception. This is the command I used: spark-submit --master yarn-client --py-files work.py main.py (Spark 1.2.1) On 20.3.2015. 9:47, Guillaume Charhon wrote: Hi Davies, I am already using --py-files. The system does use the

Re: Spark-submit and multiple files

2015-03-19 Thread Davies Liu
You could submit additional Python source via --py-files , for example: $ bin/spark-submit --py-files work.py main.py On Tue, Mar 17, 2015 at 3:29 AM, poiuytrez guilla...@databerries.com wrote: Hello guys, I am having a hard time to understand how spark-submit behave with multiple files. I