Thanks! That worked! :) And to read the files, I used pyspark.SparkFiles module.
On Thu, Jun 16, 2016 at 7:12 AM, Sun Rui <sunrise_...@163.com> wrote: > have you tried > --files <your h5 file>? > > On Jun 15, 2016, at 18:50, ar7 <ashraag...@gmail.com> wrote: > > > > I am using PySpark 1.6.1 for my spark application. I have additional > modules > > which I am loading using the argument --py-files. I also have a h5 file > > which I need to access from one of the modules for initializing the > > ApolloNet. > > > > Is there any way I could access those files from the modules if I put > them > > in the same archive? I tried this approach but it was throwing an error > > because the files are not there in every worker. I can think of one > solution > > which is copying the file to each of the workers but I want to know if > there > > are better ways to do it? > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Adding-h5-files-in-a-zip-to-use-with-PySpark-tp27173.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > For additional commands, e-mail: user-h...@spark.apache.org > > > > > -- Regards, Ashwin Raaghav
