driver crashesneed to find out why driver keeps crashing

Dear Apache Spark community, My spark driver crashes and logs does not gives enough explanation of why it happens: INFO [2019-10-21 16:33:37,045] ({pool-6-thread-7} SchedulerFactory.java[jobStarted]:109) - Job 20190926-163704_913596201 started by scheduler interpreter_2100843352 DEBUG

good materiala to learn apache spark

. Is there any material? Thank you very much Manuel Sopena Ballesteros | Big data Engineer Garvan Institute of Medical Research The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010 T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel...@garvan.org.au<mailto:manuel...@garvan.org

update LD_LIBRARY_PATH when running apache job in a YARN cluster

Is there a way to specify the LD_LIBRARY_PATH in the spark-submit command or in the config file? Manuel Sopena Ballesteros | Big data Engineer Garvan Institute of Medical Research The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010 T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E

RE: spark-submit can find python?

cache/application_1512016123441_0045/container_1512016123441_0045_02_01/tmp/1516059780057-0/bin/python? Who copies it and from where? And what do I need to do in order to make my spark-submit job to run? Thank you Manuel From: Manuel Sopena Ballesteros Sent: Tuesday, January 16, 2018 1

spark-submit can find python?

irectory /tmp/spark-888af623-c81d-4ff1-ac8a-15f25112cc4a QUESTION: Why spark/yarn can't find this file /d0/hadoop/yarn/local/usercache/mansop/appcache/application_1512016123441_0032/container_1512016123441_0032_02_01/tmp/1515989862748-0/bin/python? Who copies it and from where? And what do I ne

learning Spark

performance possible. Any suggestion? Thank you very much Manuel Sopena Ballesteros | Systems Engineer Garvan Institute of Medical Research The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010 T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel...@garvan.org.au