spark-submit is a script which calls spark-class script. Can you output the
command that spark-class runs (say, by putting set -x before the very last
line?). You should see the java command that is being run. The scripts do
some parameter setting so it's possible you're missing something. It seems
to me you think your worker memory is 2G but the executor is clearly
launched with -Xms512M" "-Xmx512M"...so that's all you'd get.
On Mon, Sep 8, 2014 at 10:16 AM, ericacm <eric...@gmail.com> wrote:
> Dear all:
>
> I am a brand new Spark user trying out the SimpleApp from the Quick Start
> page.
>
> Here is the code:
>
> object SimpleApp {
> def main(args: Array[String]) {
> val logFile = "/dev/spark-1.0.2-bin-hadoop2/README.md" // Should be
> some
> file on your system
> val conf = new SparkConf()
> .setAppName("Simple Application")
> .set("spark.executor.memory", "512m")
> .setMaster("spark://myhost.local:7077")
>
>
> .setJars(Seq("/spark-experiments/target/spark-experiments-1.0-SNAPSHOT.jar"))
> val sc = new SparkContext(conf)
> try {
> val logData = sc.textFile(logFile, 2).cache()
> val numAs = logData.filter(line => line.contains("a")).count()
> val numBs = logData.filter(line => line.contains("b")).count()
> println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
> } finally {
> sc.stop()
> }
> }
> }
>
> I am using Spark 1.0.2 and Scala 2.10.4. In spark-env.sh I have
> SPARK_WORKER_MEMORY=2g.
>
> I am trying to run this as a standalone Java app in my IDE.
>
> Note that this code *does* work when I either
> - Change the master to "local" (works running from IDE)
> - Run it using spark-submit
>
> The application/driver log is:
>
> 14/09/08 10:03:55 INFO spark.SecurityManager: Changing view acls to: eric
> 14/09/08 10:03:55 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(eric)
> 14/09/08 10:03:56 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 14/09/08 10:03:56 INFO Remoting: Starting remoting
> 14/09/08 10:03:56 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://spark@10.0.1.5:61645]
> 14/09/08 10:03:56 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://spark@10.0.1.5:61645]
> 14/09/08 10:03:56 INFO spark.SparkEnv: Registering MapOutputTracker
> 14/09/08 10:03:56 INFO spark.SparkEnv: Registering BlockManagerMaster
> 14/09/08 10:03:56 INFO storage.DiskBlockManager: Created local directory at
>
> /var/folders/j1/5rzyf1x97q9_7gj3mdc79t3c0000gn/T/spark-local-20140908100356-2496
> 14/09/08 10:03:56 INFO storage.MemoryStore: MemoryStore started with
> capacity 279.5 MB.
> 14/09/08 10:03:56 INFO network.ConnectionManager: Bound socket to port
> 61646
> with id = ConnectionManagerId(10.0.1.5,61646)
> 14/09/08 10:03:56 INFO storage.BlockManagerMaster: Trying to register
> BlockManager
> 14/09/08 10:03:56 INFO storage.BlockManagerInfo: Registering block manager
> 10.0.1.5:61646 with 279.5 MB RAM
> 14/09/08 10:03:56 INFO storage.BlockManagerMaster: Registered BlockManager
> 14/09/08 10:03:56 INFO spark.HttpServer: Starting HTTP Server
> 14/09/08 10:03:57 INFO server.Server: jetty-8.1.14.v20131031
> 14/09/08 10:03:57 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:61647
> 14/09/08 10:03:57 INFO broadcast.HttpBroadcast: Broadcast server started at
> http://10.0.1.5:61647
> 14/09/08 10:03:57 INFO spark.HttpFileServer: HTTP File server directory is
>
> /var/folders/j1/5rzyf1x97q9_7gj3mdc79t3c0000gn/T/spark-d5637279-5caa-4c14-a00f-650f1dd915bc
> 14/09/08 10:03:57 INFO spark.HttpServer: Starting HTTP Server
> 14/09/08 10:03:57 INFO server.Server: jetty-8.1.14.v20131031
> 14/09/08 10:03:57 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:61648
> 14/09/08 10:03:57 INFO server.Server: jetty-8.1.14.v20131031
> 14/09/08 10:03:57 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:4040
> 14/09/08 10:03:57 INFO ui.SparkUI: Started SparkUI at http://10.0.1.5:4040
> 2014-09-08 10:03:57.567 java[58736:1703] Unable to load realm info from
> SCDynamicStore
> 14/09/08 10:03:57 INFO spark.SparkContext: Added JAR
> /spark-experiments/target/spark-experiments-1.0-SNAPSHOT.jar at
> http://10.0.1.5:61648/jars/spark-experiments-1.0-SNAPSHOT.jar with
> timestamp
> 1410185037723
> 14/09/08 10:03:57 INFO client.AppClient$ClientActor: Connecting to master
> spark://myhost.local:7077...
> 14/09/08 10:03:57 INFO storage.MemoryStore: ensureFreeSpace(32960) called
> with curMem=0, maxMem=293063884
> 14/09/08 10:03:57 INFO storage.MemoryStore: Block broadcast_0 stored as
> values to memory (estimated size 32.2 KB, free 279.5 MB)
> 14/09/08 10:03:58 INFO cluster.SparkDeploySchedulerBackend: Connected to
> Spark cluster with app ID app-20140908100358-0002
> 14/09/08 10:03:58 INFO client.AppClient$ClientActor: Executor added:
> app-20140908100358-0002/0 on worker-20140908100129-10.0.1.5-61526
> (10.0.1.5:61526) with 8 cores
> 14/09/08 10:03:58 INFO cluster.SparkDeploySchedulerBackend: Granted
> executor
> ID app-20140908100358-0002/0 on hostPort 10.0.1.5:61526 with 8 cores,
> 512.0
> MB RAM
> 14/09/08 10:03:58 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/09/08 10:03:58 WARN snappy.LoadSnappy: Snappy native library not loaded
> 14/09/08 10:03:58 INFO mapred.FileInputFormat: Total input paths to process
> : 1
> 14/09/08 10:03:58 INFO client.AppClient$ClientActor: Executor updated:
> app-20140908100358-0002/0 is now RUNNING
> 14/09/08 10:03:58 INFO spark.SparkContext: Starting job: count at
> SimpleApp.scala:16
> 14/09/08 10:03:58 INFO scheduler.DAGScheduler: Got job 0 (count at
> SimpleApp.scala:16) with 2 output partitions (allowLocal=false)
> 14/09/08 10:03:58 INFO scheduler.DAGScheduler: Final stage: Stage 0(count
> at
> SimpleApp.scala:16)
> 14/09/08 10:03:58 INFO scheduler.DAGScheduler: Parents of final stage:
> List()
> 14/09/08 10:03:58 INFO scheduler.DAGScheduler: Missing parents: List()
> 14/09/08 10:03:58 INFO scheduler.DAGScheduler: Submitting Stage 0
> (FilteredRDD[2] at filter at SimpleApp.scala:16), which has no missing
> parents
> 14/09/08 10:03:58 INFO scheduler.DAGScheduler: Submitting 2 missing tasks
> from Stage 0 (FilteredRDD[2] at filter at SimpleApp.scala:16)
> 14/09/08 10:03:58 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0
> with
> 2 tasks
> 14/09/08 10:04:00 INFO cluster.SparkDeploySchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@10.0.1.5:61650/user/Executor#281350807]
> with
> ID 0
> 14/09/08 10:04:01 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID
> 0 on executor 0: 10.0.1.5 (PROCESS_LOCAL)
> 14/09/08 10:04:01 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as
> 1919 bytes in 2 ms
> 14/09/08 10:04:01 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID
> 1 on executor 0: 10.0.1.5 (PROCESS_LOCAL)
> 14/09/08 10:04:01 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as
> 1919 bytes in 0 ms
> 14/09/08 10:04:01 INFO storage.BlockManagerInfo: Registering block manager
> 10.0.1.5:61655 with 294.4 MB RAM
> 14/09/08 10:04:02 INFO client.AppClient$ClientActor: Executor updated:
> app-20140908100358-0002/0 is now EXITED (Command exited with code 52)
> 14/09/08 10:04:02 INFO cluster.SparkDeploySchedulerBackend: Executor 0
> disconnected, so removing it
> 14/09/08 10:04:02 INFO cluster.SparkDeploySchedulerBackend: Executor
> app-20140908100358-0002/0 removed: Command exited with code 52
> 14/09/08 10:04:02 ERROR scheduler.TaskSchedulerImpl: Lost executor 0 on
> 10.0.1.5: remote Akka client disassociated
> 14/09/08 10:04:02 INFO scheduler.TaskSetManager: Re-queueing tasks for 0
> from TaskSet 0.0
> 14/09/08 10:04:02 WARN scheduler.TaskSetManager: Lost TID 1 (task 0.0:1)
> 14/09/08 10:04:02 WARN scheduler.TaskSetManager: Lost TID 0 (task 0.0:0)
> 14/09/08 10:04:02 INFO client.AppClient$ClientActor: Executor added:
> app-20140908100358-0002/1 on worker-20140908100129-10.0.1.5-61526
> (10.0.1.5:61526) with 8 cores
> 14/09/08 10:04:02 INFO cluster.SparkDeploySchedulerBackend: Granted
> executor
> ID app-20140908100358-0002/1 on hostPort 10.0.1.5:61526 with 8 cores,
> 512.0
> MB RAM
> 14/09/08 10:04:02 INFO scheduler.DAGScheduler: Executor lost: 0 (epoch 0)
> 14/09/08 10:04:02 INFO client.AppClient$ClientActor: Executor updated:
> app-20140908100358-0002/1 is now RUNNING
> 14/09/08 10:04:02 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 0 from BlockManagerMaster.
> 14/09/08 10:04:02 INFO storage.BlockManagerMaster: Removed 0 successfully
> in
> removeExecutor
> 14/09/08 10:04:05 INFO cluster.SparkDeploySchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@10.0.1.5:61668/user/Executor#162441524]
> with
> ID 1
> 14/09/08 10:04:05 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID
> 2 on executor 1: 10.0.1.5 (PROCESS_LOCAL)
> 14/09/08 10:04:05 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as
> 1919 bytes in 0 ms
> 14/09/08 10:04:05 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID
> 3 on executor 1: 10.0.1.5 (PROCESS_LOCAL)
> 14/09/08 10:04:05 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as
> 1919 bytes in 0 ms
> 14/09/08 10:04:05 INFO storage.BlockManagerInfo: Registering block manager
> 10.0.1.5:61673 with 294.4 MB RAM
> 14/09/08 10:04:06 INFO cluster.SparkDeploySchedulerBackend: Executor 1
> disconnected, so removing it
> 14/09/08 10:04:06 ERROR scheduler.TaskSchedulerImpl: Lost executor 1 on
> 10.0.1.5: remote Akka client disassociated
> 14/09/08 10:04:06 INFO scheduler.TaskSetManager: Re-queueing tasks for 1
> from TaskSet 0.0
> 14/09/08 10:04:06 WARN scheduler.TaskSetManager: Lost TID 2 (task 0.0:0)
> 14/09/08 10:04:06 WARN scheduler.TaskSetManager: Lost TID 3 (task 0.0:1)
> 14/09/08 10:04:06 INFO scheduler.DAGScheduler: Executor lost: 1 (epoch 1)
> 14/09/08 10:04:06 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 1 from BlockManagerMaster.
> 14/09/08 10:04:06 INFO storage.BlockManagerMaster: Removed 1 successfully
> in
> removeExecutor
> 14/09/08 10:04:06 INFO client.AppClient$ClientActor: Executor updated:
> app-20140908100358-0002/1 is now EXITED (Command exited with code 52)
> 14/09/08 10:04:06 INFO cluster.SparkDeploySchedulerBackend: Executor
> app-20140908100358-0002/1 removed: Command exited with code 52
> 14/09/08 10:04:06 INFO client.AppClient$ClientActor: Executor added:
> app-20140908100358-0002/2 on worker-20140908100129-10.0.1.5-61526
> (10.0.1.5:61526) with 8 cores
> 14/09/08 10:04:06 INFO cluster.SparkDeploySchedulerBackend: Granted
> executor
> ID app-20140908100358-0002/2 on hostPort 10.0.1.5:61526 with 8 cores,
> 512.0
> MB RAM
> 14/09/08 10:04:06 INFO client.AppClient$ClientActor: Executor updated:
> app-20140908100358-0002/2 is now RUNNING
> 14/09/08 10:04:09 INFO cluster.SparkDeploySchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@10.0.1.5:61687/user/Executor#-1094963901]
> with ID 2
> 14/09/08 10:04:09 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID
> 4 on executor 2: 10.0.1.5 (PROCESS_LOCAL)
> 14/09/08 10:04:09 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as
> 1919 bytes in 0 ms
> 14/09/08 10:04:09 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID
> 5 on executor 2: 10.0.1.5 (PROCESS_LOCAL)
> 14/09/08 10:04:09 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as
> 1919 bytes in 1 ms
> 14/09/08 10:04:10 INFO storage.BlockManagerInfo: Registering block manager
> 10.0.1.5:61692 with 294.4 MB RAM
> 14/09/08 10:04:11 INFO cluster.SparkDeploySchedulerBackend: Executor 2
> disconnected, so removing it
> 14/09/08 10:04:11 ERROR scheduler.TaskSchedulerImpl: Lost executor 2 on
> 10.0.1.5: remote Akka client disassociated
> 14/09/08 10:04:11 INFO scheduler.TaskSetManager: Re-queueing tasks for 2
> from TaskSet 0.0
> 14/09/08 10:04:11 WARN scheduler.TaskSetManager: Lost TID 5 (task 0.0:0)
> 14/09/08 10:04:11 WARN scheduler.TaskSetManager: Lost TID 4 (task 0.0:1)
> 14/09/08 10:04:11 INFO scheduler.DAGScheduler: Executor lost: 2 (epoch 2)
> 14/09/08 10:04:11 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 2 from BlockManagerMaster.
> 14/09/08 10:04:11 INFO storage.BlockManagerMaster: Removed 2 successfully
> in
> removeExecutor
> 14/09/08 10:04:11 INFO client.AppClient$ClientActor: Executor updated:
> app-20140908100358-0002/2 is now EXITED (Command exited with code 52)
> 14/09/08 10:04:11 INFO cluster.SparkDeploySchedulerBackend: Executor
> app-20140908100358-0002/2 removed: Command exited with code 52
> 14/09/08 10:04:11 INFO client.AppClient$ClientActor: Executor added:
> app-20140908100358-0002/3 on worker-20140908100129-10.0.1.5-61526
> (10.0.1.5:61526) with 8 cores
> 14/09/08 10:04:11 INFO cluster.SparkDeploySchedulerBackend: Granted
> executor
> ID app-20140908100358-0002/3 on hostPort 10.0.1.5:61526 with 8 cores,
> 512.0
> MB RAM
> 14/09/08 10:04:11 INFO client.AppClient$ClientActor: Executor updated:
> app-20140908100358-0002/3 is now RUNNING
> 14/09/08 10:04:14 INFO cluster.SparkDeploySchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@10.0.1.5:61705/user/Executor#-460934085]
> with
> ID 3
> 14/09/08 10:04:14 INFO scheduler.TaskSetManager: Starting task 0.0:1 as TID
> 6 on executor 3: 10.0.1.5 (PROCESS_LOCAL)
> 14/09/08 10:04:14 INFO scheduler.TaskSetManager: Serialized task 0.0:1 as
> 1919 bytes in 0 ms
> 14/09/08 10:04:14 INFO scheduler.TaskSetManager: Starting task 0.0:0 as TID
> 7 on executor 3: 10.0.1.5 (PROCESS_LOCAL)
> 14/09/08 10:04:14 INFO scheduler.TaskSetManager: Serialized task 0.0:0 as
> 1919 bytes in 1 ms
> 14/09/08 10:04:14 INFO storage.BlockManagerInfo: Registering block manager
> 10.0.1.5:61710 with 294.4 MB RAM
> 14/09/08 10:04:15 INFO cluster.SparkDeploySchedulerBackend: Executor 3
> disconnected, so removing it
> 14/09/08 10:04:15 ERROR scheduler.TaskSchedulerImpl: Lost executor 3 on
> 10.0.1.5: remote Akka client disassociated
> 14/09/08 10:04:15 INFO scheduler.TaskSetManager: Re-queueing tasks for 3
> from TaskSet 0.0
> 14/09/08 10:04:15 WARN scheduler.TaskSetManager: Lost TID 7 (task 0.0:0)
> 14/09/08 10:04:15 ERROR scheduler.TaskSetManager: Task 0.0:0 failed 4
> times;
> aborting job
> 14/09/08 10:04:15 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0,
> whose tasks have all completed, from pool
> 14/09/08 10:04:15 INFO scheduler.TaskSchedulerImpl: Cancelling stage 0
> 14/09/08 10:04:15 INFO scheduler.DAGScheduler: Failed to run count at
> SimpleApp.scala:16
> 14/09/08 10:04:15 INFO client.AppClient$ClientActor: Executor updated:
> app-20140908100358-0002/3 is now EXITED (Command exited with code 52)
> 14/09/08 10:04:15 INFO cluster.SparkDeploySchedulerBackend: Executor
> app-20140908100358-0002/3 removed: Command exited with code 52
> 14/09/08 10:04:15 INFO client.AppClient$ClientActor: Executor added:
> app-20140908100358-0002/4 on worker-20140908100129-10.0.1.5-61526
> (10.0.1.5:61526) with 8 cores
> 14/09/08 10:04:15 INFO cluster.SparkDeploySchedulerBackend: Granted
> executor
> ID app-20140908100358-0002/4 on hostPort 10.0.1.5:61526 with 8 cores,
> 512.0
> MB RAM
> 14/09/08 10:04:15 INFO scheduler.DAGScheduler: Executor lost: 3 (epoch 3)
> 14/09/08 10:04:15 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 3 from BlockManagerMaster.
> 14/09/08 10:04:15 INFO storage.BlockManagerMaster: Removed 3 successfully
> in
> removeExecutor
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/metrics/json,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/stages/stage/kill,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/static,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/executors/json,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/executors,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/environment/json,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/environment,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/storage/rdd/json,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/storage/rdd,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/storage/json,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/storage,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/stages/pool/json,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/stages/pool,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/stages/stage/json,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/stages/stage,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/stages/json,null}
> 14/09/08 10:04:15 INFO handler.ContextHandler: stopped
> o.e.j.s.ServletContextHandler{/stages,null}
> 14/09/08 10:04:15 INFO client.AppClient$ClientActor: Executor updated:
> app-20140908100358-0002/4 is now RUNNING
> 14/09/08 10:04:15 INFO ui.SparkUI: Stopped Spark web UI at
> http://10.0.1.5:4040
> 14/09/08 10:04:15 INFO scheduler.DAGScheduler: Stopping DAGScheduler
> 14/09/08 10:04:15 INFO cluster.SparkDeploySchedulerBackend: Shutting down
> all executors
> 14/09/08 10:04:15 INFO cluster.SparkDeploySchedulerBackend: Asking each
> executor to shut down
> 14/09/08 10:04:16 INFO spark.MapOutputTrackerMasterActor:
> MapOutputTrackerActor stopped!
> 14/09/08 10:04:16 INFO network.ConnectionManager: Selector thread was
> interrupted!
> 14/09/08 10:04:16 INFO network.ConnectionManager: ConnectionManager stopped
> 14/09/08 10:04:16 INFO storage.MemoryStore: MemoryStore cleared
> 14/09/08 10:04:16 INFO storage.BlockManager: BlockManager stopped
> 14/09/08 10:04:16 INFO storage.BlockManagerMasterActor: Stopping
> BlockManagerMaster
> 14/09/08 10:04:17 INFO storage.BlockManagerMaster: BlockManagerMaster
> stopped
> 14/09/08 10:04:17 INFO spark.SparkContext: Successfully stopped
> SparkContext
> Exception in thread "main" org.apache.spark.SparkException: Job aborted due
> to stage failure: Task 0.0:0 failed 4 times, most recent failure: TID 7 on
> host 10.0.1.5 failed for unknown reason
> Driver stacktrace:
> at
> org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1049)
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1033)
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1031)
> at
>
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1031)
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635)
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635)
> at scala.Option.foreach(Option.scala:236)
> at
>
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:635)
> at
>
> org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1234)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
> at akka.actor.ActorCell.invoke(ActorCell.scala:456)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> at
>
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
>
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 14/09/08 10:04:17 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Shutting down remote daemon.
>
> The stderr log from the executor is:
>
> Spark Executor Command:
> "/Library/Java/JavaVirtualMachines/jdk1.7.0_13.jdk/Contents/Home/bin/java"
> "-cp"
>
> "::/dev/spark-1.0.2-bin-hadoop2/conf:/dev/spark-1.0.2-bin-hadoop2/lib/spark-assembly-1.0.2-hadoop2.2.0.jar:/dev/spark-1.0.2-bin-hadoop2/lib/datanucleus-api-jdo-3.2.1.jar:/Users/dev/spark-1.0.2-bin-hadoop2/lib/datanucleus-core-3.2.2.jar:/dev/spark-1.0.2-bin-hadoop2/lib/datanucleus-rdbms-3.2.1.jar"
> "-XX:MaxPermSize=128m" "-Xms512M" "-Xmx512M"
> "org.apache.spark.executor.CoarseGrainedExecutorBackend"
> "akka.tcp://spark@10.0.1.5:61645/user/CoarseGrainedScheduler" "1"
> "10.0.1.5"
> "8" "akka.tcp://sparkWorker@10.0.1.5:61526/user/Worker"
> "app-20140908100358-0002"
> ========================================
>
> 2014-09-08 10:04:04.202 java[58774:1703] Unable to load realm info from
> SCDynamicStore
> 14/09/08 10:04:04 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/09/08 10:04:04 INFO SecurityManager: Changing view acls to: eric
> 14/09/08 10:04:04 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(eric)
> 14/09/08 10:04:05 INFO Slf4jLogger: Slf4jLogger started
> 14/09/08 10:04:05 INFO Remoting: Starting remoting
> 14/09/08 10:04:05 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkExecutor@10.0.1.5:61668]
> 14/09/08 10:04:05 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkExecutor@10.0.1.5:61668]
> 14/09/08 10:04:05 INFO CoarseGrainedExecutorBackend: Connecting to driver:
> akka.tcp://spark@10.0.1.5:61645/user/CoarseGrainedScheduler
> 14/09/08 10:04:05 INFO WorkerWatcher: Connecting to worker
> akka.tcp://sparkWorker@10.0.1.5:61526/user/Worker
> 14/09/08 10:04:05 INFO WorkerWatcher: Successfully connected to
> akka.tcp://sparkWorker@10.0.1.5:61526/user/Worker
> 14/09/08 10:04:05 INFO CoarseGrainedExecutorBackend: Successfully
> registered
> with driver
> 14/09/08 10:04:05 INFO SecurityManager: Changing view acls to: eric
> 14/09/08 10:04:05 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(eric)
> 14/09/08 10:04:05 INFO Slf4jLogger: Slf4jLogger started
> 14/09/08 10:04:05 INFO Remoting: Starting remoting
> 14/09/08 10:04:05 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://spark@10.0.1.5:61671]
> 14/09/08 10:04:05 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://spark@10.0.1.5:61671]
> 14/09/08 10:04:05 INFO SparkEnv: Connecting to MapOutputTracker:
> akka.tcp://spark@10.0.1.5:61645/user/MapOutputTracker
> 14/09/08 10:04:05 INFO SparkEnv: Connecting to BlockManagerMaster:
> akka.tcp://spark@10.0.1.5:61645/user/BlockManagerMaster
> 14/09/08 10:04:05 INFO DiskBlockManager: Created local directory at
>
> /var/folders/j1/5rzyf1x97q9_7gj3mdc79t3c0000gn/T/spark-local-20140908100405-509f
> 14/09/08 10:04:05 INFO MemoryStore: MemoryStore started with capacity 294.4
> MB.
> 14/09/08 10:04:05 INFO ConnectionManager: Bound socket to port 61673 with
> id
> = ConnectionManagerId(10.0.1.5,61673)
> 14/09/08 10:04:05 INFO BlockManagerMaster: Trying to register BlockManager
> 14/09/08 10:04:05 INFO BlockManagerMaster: Registered BlockManager
> 14/09/08 10:04:05 INFO HttpFileServer: HTTP File server directory is
>
> /var/folders/j1/5rzyf1x97q9_7gj3mdc79t3c0000gn/T/spark-3bb0c7fb-be2f-4efe-9dde-bcf8349a962a
> 14/09/08 10:04:05 INFO HttpServer: Starting HTTP Server
> 14/09/08 10:04:05 INFO CoarseGrainedExecutorBackend: Got assigned task 2
> 14/09/08 10:04:06 INFO CoarseGrainedExecutorBackend: Got assigned task 3
> 14/09/08 10:04:06 INFO Executor: Running task ID 2
> 14/09/08 10:04:06 INFO Executor: Running task ID 3
> 14/09/08 10:04:06 INFO Executor: Fetching
> http://10.0.1.5:61648/jars/spark-experiments-1.0-SNAPSHOT.jar with
> timestamp
> 1410185037723
> 14/09/08 10:04:06 INFO Utils: Fetching
> http://10.0.1.5:61648/jars/spark-experiments-1.0-SNAPSHOT.jar to
>
> /var/folders/j1/5rzyf1x97q9_7gj3mdc79t3c0000gn/T/fetchFileTemp4194990807172315371.tmp
> 14/09/08 10:04:06 INFO Executor: Adding
>
> file:/dev/spark-1.0.2-bin-hadoop2/work/app-20140908100358-0002/1/./spark-experiments-1.0-SNAPSHOT.jar
> to class loader
> 14/09/08 10:04:06 INFO HttpBroadcast: Started reading broadcast variable 0
> 14/09/08 10:04:06 ERROR Executor: Exception in task ID 2
> java.lang.OutOfMemoryError: Java heap space
> at
>
> org.apache.hadoop.io.WritableUtils.readCompressedStringArray(WritableUtils.java:183)
> at
> org.apache.hadoop.conf.Configuration.readFields(Configuration.java:2378)
> at
> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:285)
> at
> org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:77)
> at
>
> org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:42)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1872)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
> at
>
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
> at
> org.apache.spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala:205)
> at
> org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala:89)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1872)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> 14/09/08 10:04:06 INFO HttpBroadcast: Started reading broadcast variable 0
> 14/09/08 10:04:06 ERROR ExecutorUncaughtExceptionHandler: Uncaught
> exception
> in thread Thread[Executor task launch worker-0,5,main]
> java.lang.OutOfMemoryError: Java heap space
> at
>
> org.apache.hadoop.io.WritableUtils.readCompressedStringArray(WritableUtils.java:183)
> at
> org.apache.hadoop.conf.Configuration.readFields(Configuration.java:2378)
> at
> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:285)
> at
> org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:77)
> at
>
> org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:42)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1872)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
> at
>
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
> at
> org.apache.spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala:205)
> at
> org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala:89)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1872)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
>
> It seems like a serialization problem because there is plenty of heap space
> (and it works with spark-submit).
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Cannot-run-SimpleApp-as-regular-Java-app-tp13695.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
