In your spark-env, did you set LD_LIBRARY_PATH ? Cheers
On Wed, Nov 25, 2015 at 7:32 AM, Oriol López Massaguer < oriol.lo...@gmail.com> wrote: > Hello; > > I'm trying to use a native library in Spark. > > I was using a simple standalone cluster with one master and worker. > > According to the documentation I edited the spark-defautls.conf by setting: > > spark.driver.extraClassPath=/opt/eTOX_spark/lib/org.RDKit.jar > spark.driver.extraLibraryPath=/opt/eTOX_spark/lib/ > spark.executor.extraLibraryPath=/opt/eTOX_spark/lib/ > > In the path /opt/eTOX_spark/lib/ there are 3 so files wich are wrapped in > org.RDKit.jar. > > But when I try so submit a job that uses the native library I get: > > Exception in thread "main" java.lang.UnsatisfiedLinkError: > org.RDKit.RDKFuncsJNI.RWMol_MolFromSmiles__SWIG_3(Ljava/lang/String;)J > at org.RDKit.RDKFuncsJNI.RWMol_MolFromSmiles__SWIG_3(Native Method) > at org.RDKit.RWMol.MolFromSmiles(RWMol.java:426) > at models.spark.sources.eTOX_DB$.main(eTOX.scala:54) > at models.spark.sources.eTOX_DB.main(eTOX.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:727) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > I use the submit.sh with the following parameters: > > /opt/spark/bin/spark-submit --verbose --class > "models.spark.sources.eTOX_DB" --master > spark://localhost.localdomain:7077 > target/scala-2.10/etox_spark_2.10-1.0.jar > > the full output is: > > Using properties file: /opt/spark/conf/spark-defaults.conf > Adding default property: spark.driver.extraLibraryPath=/opt/eTOX_spark/lib/ > Adding default property: > spark.driver.extraClassPath=/opt/eTOX_spark/lib/org.RDKit.jar > Adding default property: > spark.executor.extraLibraryPath=/opt/eTOX_spark/lib/ > Parsed arguments: > master spark://localhost.localdomain:7077 > deployMode null > executorMemory null > executorCores null > totalExecutorCores null > propertiesFile /opt/spark/conf/spark-defaults.conf > driverMemory null > driverCores null > driverExtraClassPath /opt/eTOX_spark/lib/org.RDKit.jar > driverExtraLibraryPath /opt/eTOX_spark/lib/ > driverExtraJavaOptions null > supervise false > queue null > numExecutors null > files null > pyFiles null > archives null > mainClass models.spark.sources.eTOX_DB > primaryResource > file:/opt/eTOX_spark/target/scala-2.10/etox_spark_2.10-1.0.jar > name models.spark.sources.eTOX_DB > childArgs [] > jars null > packages null > packagesExclusions null > repositories null > verbose true > > Spark properties used, including those specified through > --conf and those from the properties file > /opt/spark/conf/spark-defaults.conf: > spark.executor.extraLibraryPath -> /opt/eTOX_spark/lib/ > spark.driver.extraLibraryPath -> /opt/eTOX_spark/lib/ > spark.driver.extraClassPath -> /opt/eTOX_spark/lib/org.RDKit.jar > > > Main class: > models.spark.sources.eTOX_DB > Arguments: > > System properties: > spark.executor.extraLibraryPath -> /opt/eTOX_spark/lib/ > spark.driver.extraLibraryPath -> /opt/eTOX_spark/lib/ > SPARK_SUBMIT -> true > spark.app.name -> models.spark.sources.eTOX_DB > spark.jars -> > file:/opt/eTOX_spark/target/scala-2.10/etox_spark_2.10-1.0.jar > spark.submit.deployMode -> client > spark.master -> spark://localhost.localdomain:7077 > spark.driver.extraClassPath -> /opt/eTOX_spark/lib/org.RDKit.jar > Classpath elements: > file:/opt/eTOX_spark/target/scala-2.10/etox_spark_2.10-1.0.jar > > > Buffer(/opt/jdk1.8.0_45/jre/lib/amd64/libzip.so) > Loading libraries > Buffer(/opt/jdk1.8.0_45/jre/lib/amd64/libzip.so, /opt/eTOX_spark/lib/ > libboost_thread.1.48.0.so, /opt/eTOX_spark/lib/libboost_system.1.48.0.so, > /opt/eTOX_spark/lib/libGraphMolWrap.so) > Loading libraries > Using Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties > 15/11/25 16:27:32 INFO SparkContext: Running Spark version 1.6.0-SNAPSHOT > 15/11/25 16:27:33 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 15/11/25 16:27:33 WARN Utils: Your hostname, localhost.localdomain > resolves to a loopback address: 127.0.0.1; using 10.0.2.15 instead (on > interface enp0s3) > 15/11/25 16:27:33 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to > another address > 15/11/25 16:27:33 INFO SecurityManager: Changing view acls to: user > 15/11/25 16:27:33 INFO SecurityManager: Changing modify acls to: user > 15/11/25 16:27:33 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(user); users > with modify permissions: Set(user) > 15/11/25 16:27:34 INFO Utils: Successfully started service 'sparkDriver' > on port 35799. > 15/11/25 16:27:34 INFO Slf4jLogger: Slf4jLogger started > 15/11/25 16:27:34 INFO Remoting: Starting remoting > 15/11/25 16:27:34 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkDriverActorSystem@10.0.2.15:48344] > 15/11/25 16:27:34 INFO Utils: Successfully started service > 'sparkDriverActorSystem' on port 48344. > 15/11/25 16:27:34 INFO SparkEnv: Registering MapOutputTracker > 15/11/25 16:27:34 INFO SparkEnv: Registering BlockManagerMaster > 15/11/25 16:27:34 INFO DiskBlockManager: Created local directory at > /tmp/blockmgr-ffbf1759-2e79-4ecd-a0d8-5d7f28d3c132 > 15/11/25 16:27:34 INFO MemoryStore: MemoryStore started with capacity > 736.1 MB > 15/11/25 16:27:34 INFO HttpFileServer: HTTP File server directory is > /tmp/spark-3f269496-c3aa-4bbc-8955-1ab88a179420/httpd-4ea1a9f9-2ce6-41b3-82ef-96e9094f5aa4 > 15/11/25 16:27:34 INFO HttpServer: Starting HTTP Server > 15/11/25 16:27:35 INFO Utils: Successfully started service 'HTTP file > server' on port 50803. > 15/11/25 16:27:35 INFO SparkEnv: Registering OutputCommitCoordinator > 15/11/25 16:27:35 INFO Utils: Successfully started service 'SparkUI' on > port 4040. > 15/11/25 16:27:35 INFO SparkUI: Started SparkUI at http://10.0.2.15:4040 > 15/11/25 16:27:35 INFO SparkContext: Added JAR > file:/opt/eTOX_spark/target/scala-2.10/etox_spark_2.10-1.0.jar at > http://10.0.2.15:50803/jars/etox_spark_2.10-1.0.jar with timestamp > 1448465255275 > 15/11/25 16:27:35 INFO AppClient$ClientEndpoint: Connecting to master > spark://localhost.localdomain:7077... > 15/11/25 16:27:35 INFO SparkDeploySchedulerBackend: Connected to Spark > cluster with app ID app-20151125162735-0003 > 15/11/25 16:27:35 INFO AppClient$ClientEndpoint: Executor added: > app-20151125162735-0003/0 on worker-20151125161159-10.0.2.15-48408 ( > 10.0.2.15:48408) with 4 cores > 15/11/25 16:27:35 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20151125162735-0003/0 on hostPort 10.0.2.15:48408 with 4 cores, > 1024.0 MB RAM > 15/11/25 16:27:35 INFO Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 35769. > 15/11/25 16:27:35 INFO NettyBlockTransferService: Server created on 35769 > 15/11/25 16:27:35 INFO BlockManagerMaster: Trying to register BlockManager > 15/11/25 16:27:35 INFO BlockManagerMasterEndpoint: Registering block > manager 10.0.2.15:35769 with 736.1 MB RAM, BlockManagerId(driver, > 10.0.2.15, 35769) > 15/11/25 16:27:35 INFO AppClient$ClientEndpoint: Executor updated: > app-20151125162735-0003/0 is now LOADING > 15/11/25 16:27:35 INFO AppClient$ClientEndpoint: Executor updated: > app-20151125162735-0003/0 is now RUNNING > 15/11/25 16:27:35 INFO BlockManagerMaster: Registered BlockManager > 15/11/25 16:27:35 INFO SparkDeploySchedulerBackend: SchedulerBackend is > ready for scheduling beginning after reached minRegisteredResourcesRatio: > 0.0 > 15/11/25 16:27:36 INFO ParquetRelation: Listing > file:/opt/etox_reports_2015_1/spark_data/etox/allfindings.parquet on driver > 15/11/25 16:27:37 INFO SparkContext: Starting job: load at eTOX.scala:52 > 15/11/25 16:27:37 INFO DAGScheduler: Got job 0 (load at eTOX.scala:52) > with 2 output partitions > 15/11/25 16:27:37 INFO DAGScheduler: Final stage: ResultStage 0 (load at > eTOX.scala:52) > 15/11/25 16:27:37 INFO DAGScheduler: Parents of final stage: List() > 15/11/25 16:27:37 INFO DAGScheduler: Missing parents: List() > 15/11/25 16:27:37 INFO DAGScheduler: Submitting ResultStage 0 > (MapPartitionsRDD[1] at load at eTOX.scala:52), which has no missing parents > 15/11/25 16:27:37 INFO MemoryStore: Ensuring 1048576 bytes of free space > for block broadcast_0(free: 771883008, max: 771883008) > 15/11/25 16:27:37 INFO MemoryStore: Ensuring 63000 bytes of free space for > block broadcast_0(free: 771883008, max: 771883008) > 15/11/25 16:27:37 INFO MemoryStore: Block broadcast_0 stored as values in > memory (estimated size 61.5 KB, free 61.5 KB) > 15/11/25 16:27:37 INFO MemoryStore: Ensuring 21090 bytes of free space for > block broadcast_0_piece0(free: 771820008, max: 771883008) > 15/11/25 16:27:37 INFO MemoryStore: Block broadcast_0_piece0 stored as > bytes in memory (estimated size 20.6 KB, free 82.1 KB) > 15/11/25 16:27:37 INFO BlockManagerInfo: Added broadcast_0_piece0 in > memory on 10.0.2.15:35769 (size: 20.6 KB, free: 736.1 MB) > 15/11/25 16:27:37 INFO SparkContext: Created broadcast 0 from broadcast at > DAGScheduler.scala:1003 > 15/11/25 16:27:37 INFO DAGScheduler: Submitting 2 missing tasks from > ResultStage 0 (MapPartitionsRDD[1] at load at eTOX.scala:52) > 15/11/25 16:27:37 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks > 15/11/25 16:27:39 INFO SparkDeploySchedulerBackend: Registered executor > NettyRpcEndpointRef(null) (10.0.2.15:50472) with ID 0 > 15/11/25 16:27:39 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID > 0, 10.0.2.15, partition 0,PROCESS_LOCAL, 2137 bytes) > 15/11/25 16:27:39 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID > 1, 10.0.2.15, partition 1,PROCESS_LOCAL, 2285 bytes) > 15/11/25 16:27:39 INFO BlockManagerMasterEndpoint: Registering block > manager 10.0.2.15:39373 with 736.1 MB RAM, BlockManagerId(0, 10.0.2.15, > 39373) > 15/11/25 16:27:39 INFO BlockManagerInfo: Added broadcast_0_piece0 in > memory on 10.0.2.15:39373 (size: 20.6 KB, free: 736.1 MB) > 15/11/25 16:27:40 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID > 0) in 1097 ms on 10.0.2.15 (1/2) > 15/11/25 16:27:42 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID > 1) in 3050 ms on 10.0.2.15 (2/2) > 15/11/25 16:27:42 INFO DAGScheduler: ResultStage 0 (load at eTOX.scala:52) > finished in 4.739 s > 15/11/25 16:27:42 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks > have all completed, from pool > 15/11/25 16:27:42 INFO DAGScheduler: Job 0 finished: load at > eTOX.scala:52, took 5.157662 s > 15/11/25 16:27:42 INFO MemoryStore: Ensuring 1048576 bytes of free space > for block broadcast_1(free: 771798918, max: 771883008) > 15/11/25 16:27:42 INFO MemoryStore: Ensuring 63280 bytes of free space for > block broadcast_1(free: 771798918, max: 771883008) > 15/11/25 16:27:42 INFO MemoryStore: Block broadcast_1 stored as values in > memory (estimated size 61.8 KB, free 143.9 KB) > 15/11/25 16:27:42 INFO MemoryStore: Ensuring 19788 bytes of free space for > block broadcast_1_piece0(free: 771735638, max: 771883008) > 15/11/25 16:27:42 INFO MemoryStore: Block broadcast_1_piece0 stored as > bytes in memory (estimated size 19.3 KB, free 163.2 KB) > 15/11/25 16:27:42 INFO BlockManagerInfo: Added broadcast_1_piece0 in > memory on 10.0.2.15:35769 (size: 19.3 KB, free: 736.1 MB) > 15/11/25 16:27:42 INFO SparkContext: Created broadcast 1 from count at > eTOX.scala:53 > 15/11/25 16:27:42 INFO deprecation: mapred.min.split.size is deprecated. > Instead, use mapreduce.input.fileinputformat.split.minsize > 15/11/25 16:27:43 INFO ParquetRelation: Reading Parquet file(s) from > file:/opt/etox_reports_2015_1/spark_data/etox/allfindings.parquet/part-r-00000-a0018ffe-9aaf-4446-a718-79097835d08d.gz.parquet > 15/11/25 16:27:43 INFO SparkContext: Starting job: count at eTOX.scala:53 > 15/11/25 16:27:43 INFO DAGScheduler: Registering RDD 4 (count at > eTOX.scala:53) > 15/11/25 16:27:43 INFO DAGScheduler: Got job 1 (count at eTOX.scala:53) > with 1 output partitions > 15/11/25 16:27:43 INFO DAGScheduler: Final stage: ResultStage 2 (count at > eTOX.scala:53) > 15/11/25 16:27:43 INFO DAGScheduler: Parents of final stage: > List(ShuffleMapStage 1) > 15/11/25 16:27:43 INFO DAGScheduler: Missing parents: List(ShuffleMapStage > 1) > 15/11/25 16:27:43 INFO DAGScheduler: Submitting ShuffleMapStage 1 > (MapPartitionsRDD[4] at count at eTOX.scala:53), which has no missing > parents > 15/11/25 16:27:43 INFO MemoryStore: Ensuring 1048576 bytes of free space > for block broadcast_2(free: 771715850, max: 771883008) > 15/11/25 16:27:43 INFO MemoryStore: Ensuring 12840 bytes of free space for > block broadcast_2(free: 771715850, max: 771883008) > 15/11/25 16:27:43 INFO MemoryStore: Block broadcast_2 stored as values in > memory (estimated size 12.5 KB, free 175.8 KB) > 15/11/25 16:27:43 INFO MemoryStore: Ensuring 6175 bytes of free space for > block broadcast_2_piece0(free: 771703010, max: 771883008) > 15/11/25 16:27:43 INFO MemoryStore: Block broadcast_2_piece0 stored as > bytes in memory (estimated size 6.0 KB, free 181.8 KB) > 15/11/25 16:27:43 INFO BlockManagerInfo: Added broadcast_2_piece0 in > memory on 10.0.2.15:35769 (size: 6.0 KB, free: 736.1 MB) > 15/11/25 16:27:43 INFO SparkContext: Created broadcast 2 from broadcast at > DAGScheduler.scala:1003 > 15/11/25 16:27:43 INFO DAGScheduler: Submitting 1 missing tasks from > ShuffleMapStage 1 (MapPartitionsRDD[4] at count at eTOX.scala:53) > 15/11/25 16:27:43 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks > 15/11/25 16:27:43 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID > 2, 10.0.2.15, partition 0,PROCESS_LOCAL, 2300 bytes) > 15/11/25 16:27:43 INFO BlockManagerInfo: Added broadcast_2_piece0 in > memory on 10.0.2.15:39373 (size: 6.0 KB, free: 736.1 MB) > 15/11/25 16:27:43 INFO BlockManagerInfo: Added broadcast_1_piece0 in > memory on 10.0.2.15:39373 (size: 19.3 KB, free: 736.1 MB) > 15/11/25 16:27:44 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID > 2) in 1000 ms on 10.0.2.15 (1/1) > 15/11/25 16:27:44 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks > have all completed, from pool > 15/11/25 16:27:44 INFO DAGScheduler: ShuffleMapStage 1 (count at > eTOX.scala:53) finished in 1.003 s > 15/11/25 16:27:44 INFO DAGScheduler: looking for newly runnable stages > 15/11/25 16:27:44 INFO DAGScheduler: running: Set() > 15/11/25 16:27:44 INFO DAGScheduler: waiting: Set(ResultStage 2) > 15/11/25 16:27:44 INFO DAGScheduler: failed: Set() > 15/11/25 16:27:44 INFO DAGScheduler: Submitting ResultStage 2 > (MapPartitionsRDD[7] at count at eTOX.scala:53), which has no missing > parents > 15/11/25 16:27:44 INFO MemoryStore: Ensuring 1048576 bytes of free space > for block broadcast_3(free: 771696835, max: 771883008) > 15/11/25 16:27:44 INFO MemoryStore: Ensuring 14032 bytes of free space for > block broadcast_3(free: 771696835, max: 771883008) > 15/11/25 16:27:44 INFO MemoryStore: Block broadcast_3 stored as values in > memory (estimated size 13.7 KB, free 195.5 KB) > 15/11/25 16:27:44 INFO MemoryStore: Ensuring 6698 bytes of free space for > block broadcast_3_piece0(free: 771682803, max: 771883008) > 15/11/25 16:27:44 INFO MemoryStore: Block broadcast_3_piece0 stored as > bytes in memory (estimated size 6.5 KB, free 202.1 KB) > 15/11/25 16:27:44 INFO BlockManagerInfo: Added broadcast_3_piece0 in > memory on 10.0.2.15:35769 (size: 6.5 KB, free: 736.1 MB) > 15/11/25 16:27:44 INFO SparkContext: Created broadcast 3 from broadcast at > DAGScheduler.scala:1003 > 15/11/25 16:27:44 INFO DAGScheduler: Submitting 1 missing tasks from > ResultStage 2 (MapPartitionsRDD[7] at count at eTOX.scala:53) > 15/11/25 16:27:44 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks > 15/11/25 16:27:44 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID > 3, 10.0.2.15, partition 0,NODE_LOCAL, 2060 bytes) > 15/11/25 16:27:44 INFO BlockManagerInfo: Added broadcast_3_piece0 in > memory on 10.0.2.15:39373 (size: 6.5 KB, free: 736.1 MB) > 15/11/25 16:27:44 INFO MapOutputTrackerMasterEndpoint: Asked to send map > output locations for shuffle 0 to 10.0.2.15:50472 > 15/11/25 16:27:44 INFO MapOutputTrackerMaster: Size of output statuses for > shuffle 0 is 138 bytes > 15/11/25 16:27:44 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID > 3) in 190 ms on 10.0.2.15 (1/1) > 15/11/25 16:27:44 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks > have all completed, from pool > 15/11/25 16:27:44 INFO DAGScheduler: ResultStage 2 (count at > eTOX.scala:53) finished in 0.195 s > 15/11/25 16:27:44 INFO DAGScheduler: Job 1 finished: count at > eTOX.scala:53, took 1.249107 s > Num Findings: 2775141 > Exception in thread "main" java.lang.UnsatisfiedLinkError: > org.RDKit.RDKFuncsJNI.RWMol_MolFromSmiles__SWIG_3(Ljava/lang/String;)J > at org.RDKit.RDKFuncsJNI.RWMol_MolFromSmiles__SWIG_3(Native Method) > at org.RDKit.RWMol.MolFromSmiles(RWMol.java:426) > at models.spark.sources.eTOX_DB$.main(eTOX.scala:54) > at models.spark.sources.eTOX_DB.main(eTOX.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:727) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > 15/11/25 16:27:44 INFO SparkContext: Invoking stop() from shutdown hook > 15/11/25 16:27:44 INFO BlockManagerInfo: Removed broadcast_3_piece0 on > 10.0.2.15:35769 in memory (size: 6.5 KB, free: 736.1 MB) > 15/11/25 16:27:44 INFO BlockManagerInfo: Removed broadcast_3_piece0 on > 10.0.2.15:39373 in memory (size: 6.5 KB, free: 736.1 MB) > 15/11/25 16:27:44 INFO ContextCleaner: Cleaned accumulator 12 > 15/11/25 16:27:44 INFO BlockManagerInfo: Removed broadcast_2_piece0 on > 10.0.2.15:35769 in memory (size: 6.0 KB, free: 736.1 MB) > 15/11/25 16:27:44 INFO BlockManagerInfo: Removed broadcast_2_piece0 on > 10.0.2.15:39373 in memory (size: 6.0 KB, free: 736.1 MB) > 15/11/25 16:27:44 INFO SparkUI: Stopped Spark web UI at > http://10.0.2.15:4040 > 15/11/25 16:27:44 INFO ContextCleaner: Cleaned accumulator 11 > 15/11/25 16:27:44 INFO ContextCleaner: Cleaned shuffle 0 > 15/11/25 16:27:44 INFO DAGScheduler: Stopping DAGScheduler > 15/11/25 16:27:44 INFO SparkDeploySchedulerBackend: Shutting down all > executors > 15/11/25 16:27:44 INFO SparkDeploySchedulerBackend: Asking each executor > to shut down > 15/11/25 16:27:44 INFO MapOutputTrackerMasterEndpoint: > MapOutputTrackerMasterEndpoint stopped! > 15/11/25 16:27:44 INFO MemoryStore: MemoryStore cleared > 15/11/25 16:27:44 INFO BlockManager: BlockManager stopped > 15/11/25 16:27:44 INFO BlockManagerMaster: BlockManagerMaster stopped > 15/11/25 16:27:44 INFO > OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: > OutputCommitCoordinator stopped! > 15/11/25 16:27:44 INFO RemoteActorRefProvider$RemotingTerminator: Shutting > down remote daemon. > 15/11/25 16:27:44 INFO SparkContext: Successfully stopped SparkContext > 15/11/25 16:27:44 INFO ShutdownHookManager: Shutdown hook called > 15/11/25 16:27:44 INFO RemoteActorRefProvider$RemotingTerminator: Remote > daemon shut down; proceeding with flushing remote transports. > 15/11/25 16:27:44 INFO ShutdownHookManager: Deleting directory > /tmp/spark-3f269496-c3aa-4bbc-8955-1ab88a179420 > > > > Oriol. >
