thanks Marcelo. are these files distributed through hdfs?
Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 17 July 2017 at 18:46, Marcelo Vanzin <van...@cloudera.com> wrote: > The YARN backend distributes all files and jars you submit with your > application. > > On Mon, Jul 17, 2017 at 10:45 AM, Mich Talebzadeh > <mich.talebza...@gmail.com> wrote: > > thanks guys. > > > > just to clarify let us assume i am doing spark-submit as below: > > > > ${SPARK_HOME}/bin/spark-submit \ > > --packages ${PACKAGES} \ > > --driver-memory 2G \ > > --num-executors 2 \ > > --executor-memory 2G \ > > --executor-cores 2 \ > > --master yarn \ > > --deploy-mode client \ > > --conf "${SCHEDULER}" \ > > --conf "${EXTRAJAVAOPTIONS}" \ > > --jars ${JARS} \ > > --class "${FILE_NAME}" \ > > --conf "${SPARKUIPORT}" \ > > --conf "${SPARKDRIVERPORT}" \ > > --conf "${SPARKFILESERVERPORT}" \ > > --conf "${SPARKBLOCKMANAGERPORT}" \ > > --conf "${SPARKKRYOSERIALIZERBUFFERMAX}" \ > > ${JAR_FILE} > > > > The ${JAR_FILE} is the one. As I understand Spark should distribute that > > ${JAR_FILE} to each container? > > > > Also --jars ${JARS} are the list of normal jar files that need to exist > in > > the same directory on each executor node? > > > > cheers, > > > > > > > > Dr Mich Talebzadeh > > > > > > > > LinkedIn > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd > OABUrV8Pw > > > > > > > > http://talebzadehmich.wordpress.com > > > > > > Disclaimer: Use it at your own risk. Any and all responsibility for any > > loss, damage or destruction of data or any other property which may arise > > from relying on this email's technical content is explicitly disclaimed. > The > > author will in no case be liable for any monetary damages arising from > such > > loss, damage or destruction. > > > > > > > > > > On 17 July 2017 at 18:18, ayan guha <guha.a...@gmail.com> wrote: > >> > >> Hi Mitch > >> > >> your jar file can be anywhere in the file system, including hdfs. > >> > >> If using yarn, preferably use cluster mode in terms of deployment. > >> > >> Yarn will distribute the jar to each container. > >> > >> Best > >> Ayan > >> > >> On Tue, 18 Jul 2017 at 2:17 am, Marcelo Vanzin <van...@cloudera.com> > >> wrote: > >>> > >>> Spark distributes your application jar for you. > >>> > >>> On Mon, Jul 17, 2017 at 8:41 AM, Mich Talebzadeh > >>> <mich.talebza...@gmail.com> wrote: > >>> > hi guys, > >>> > > >>> > > >>> > an uber/fat jar file has been created to run with spark in CDH yarc > >>> > client > >>> > mode. > >>> > > >>> > As usual job is submitted to the edge node. > >>> > > >>> > does the jar file has to be placed in the same directory ewith spark > is > >>> > running in the cluster to make it work? > >>> > > >>> > Also what will happen if say out of 9 nodes running spark, 3 have not > >>> > got > >>> > the jar file. will that job fail or it will carry on on the fremaing > 6 > >>> > nodes > >>> > that have that jar file? > >>> > > >>> > thanks > >>> > > >>> > Dr Mich Talebzadeh > >>> > > >>> > > >>> > > >>> > LinkedIn > >>> > > >>> > https://www.linkedin.com/profile/view?id= > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >>> > > >>> > > >>> > > >>> > http://talebzadehmich.wordpress.com > >>> > > >>> > > >>> > Disclaimer: Use it at your own risk. Any and all responsibility for > any > >>> > loss, damage or destruction of data or any other property which may > >>> > arise > >>> > from relying on this email's technical content is explicitly > >>> > disclaimed. The > >>> > author will in no case be liable for any monetary damages arising > from > >>> > such > >>> > loss, damage or destruction. > >>> > > >>> > > >>> > >>> > >>> > >>> -- > >>> Marcelo > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >>> > >> -- > >> Best Regards, > >> Ayan Guha > > > > > > > > -- > Marcelo >
