This is a long standing bug in Spark. –jars and –files doesn’t work in
Standalone mode
https://issues.apache.org/jira/browse/SPARK-4160
From: Marius <m.die0...@gmail.com>
Date: Wednesday, May 9, 2018 at 3:51 AM
To: "user@spark.apache.org" <user@spark.apache.org>
Subject: Spark 2.3.0 --files vs. addFile()
Hey,
i am using Spark to distribute the execution of a binary tool and to do some
further calculation further down stream. I want to distribute the binary tool
using either the --files or the addFile option from spark to make it available
on each worker node. However although he tells my that he added the file:
2018-05-09 07:42:19 INFO SparkContext:54 - Added file s3a://executables/blastp
at s3a://executables/foo with timestamp 1525851739972
2018-05-09 07:42:20 INFO Utils:54 - Fetching s3a://executables/foo to
/tmp/spark-54931ea6-b3d6-419b-997b-a498da898b77/userFiles-5e4b66e5-de4a-4420-a641-4453b9ea2ead/fetchFileTemp3437582648265876247.tmp
However when i want to execute the tool using pipe it does not work. I
currently assume that the file is only downloaded to the master node. However i
am not sure if i misunderstood the concept of adding files in spark or if i did
something wrong.
I am getting the path with Sparkfiles.get(). It does work but the bin is not
there.
This is my call:
spark-submit \
--class de.jlu.bioinfsys.sparkBlast.run.Run \
--master $master \
--jars ${awsPath},${awsJavaSDK} \
--files
s3a://database/a.a.z,s3a://database/a.a.y,s3a://database/a.a.x,s3a://executables/tool
\
--conf spark.executor.extraClassPath=${awsPath}:${awsJavaSDK} \
--conf spark.driver.extraClassPath=${awsPath}:${awsJavaSDK} \
--conf
spark.hadoop.fs.s3a.endpoint=https://s3.computational.bio.uni-giessen.de/<https://urldefense.proofpoint.com/v2/url?u=https-3A__s3.computational.bio.uni-2Dgiessen.de_&d=DwMDaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=F2RNeGILvLdBxn7RJ4effes_QFIiEsoVM2rPi9qX1DKow5HQSjq0_WhIW109SXQ4&m=X4cHBr7qC1lO1X6gj37AszPDf02jI96W88e-9rVvaFw&s=eN8XVAxUFlLI0UTs2kV6eTGXhM6c8OJMJMWaa5rlK2I&e=>
\
--conf spark.hadoop.fs.s3a.access.key=$s3Access \
--conf spark.hadoop.fs.s3a.secret.key=$s3Secret \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
${execJarPath}
I am using Spark v 2.3.0 along with scala in Standalone cluster node with three
workers.
Cheers
Marius
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to
Capital One and/or its affiliates and may only be used solely in performance of
work or services for Capital One. The information transmitted herewith is
intended only for use by the individual or entity to which it is addressed. If
the reader of this message is not the intended recipient, you are hereby
notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is
strictly prohibited. If you have received this communication in error, please
contact the sender and delete the material from your computer.
