Can you try adding sc.stop at the end of your program? looks like its
having a hard-time closing off sparkcontext.
Thanks
Best Regards
On Tue, Jul 7, 2015 at 4:08 PM, Hafsa Asif <hafsa.a...@matchinguu.com>
wrote:
> Hi,
>
> I run the following simple Java spark standalone app with maven command
> "exec:java -Dexec.mainClass=SimpleApp"
>
> public class SimpleApp {
> public static void main(String[] args) {
> System.out.println("Reading and Connecting with Spark.....");
> try {
> String logFile = "/home/asif/spark-1.4.0/README.md"; // Should
> be some file on your system
> SparkConf conf = new SparkConf().setAppName("Simple
> Application").setMaster("local");
> JavaSparkContext sc = new JavaSparkContext(conf);
> JavaRDD<String> logData = sc.textFile(logFile).cache();
>
> long numAs = logData.filter(new Function<String, Boolean>() {
> public Boolean call(String s) { return s.contains("a"); }
> }).count();
>
> long numBs = logData.filter(new Function<String, Boolean>() {
> public Boolean call(String s) { return s.contains("b"); }
> }).count();
>
> System.out.println("Lines with a: " + numAs + ", lines with b:
> "
> + numBs);
> }
> catch(Exception e){
> System.out.println ("Error in connecting with Spark");
> }
> }
> }
>
> Well, it builds successfully and also giving results but with thread
> exception. What is the reason of the thread exception and how to solve it
> in standalone mode because in spark shell with spark commit command, it is
> running fine.
> Log trace is:
>
> [INFO] Scanning for projects...
> [INFO]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Building standAloneSparkApp 1.0-SNAPSHOT
> [INFO]
> ------------------------------------------------------------------------
> [INFO]
> [INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ standAloneSparkApp
> ---
> Reading and Connecting with Spark.....
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 15/07/07 03:28:34 INFO SparkContext: Running Spark version 1.4.0
> 15/07/07 03:28:34 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 15/07/07 03:28:34 WARN Utils: Your hostname, ubuntu resolves to a loopback
> address: 127.0.1.1; using 192.168.116.133 instead (on interface eth0)
> 15/07/07 03:28:34 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
> another address
> 15/07/07 03:28:34 INFO SecurityManager: Changing view acls to: asif
> 15/07/07 03:28:34 INFO SecurityManager: Changing modify acls to: asif
> 15/07/07 03:28:34 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(asif); users
> with modify permissions: Set(asif)
> 15/07/07 03:28:35 INFO Slf4jLogger: Slf4jLogger started
> 15/07/07 03:28:36 INFO Remoting: Starting remoting
> 15/07/07 03:28:36 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@192.168.116.133:34863]
> 15/07/07 03:28:36 INFO Utils: Successfully started service 'sparkDriver' on
> port 34863.
> 15/07/07 03:28:36 INFO SparkEnv: Registering MapOutputTracker
> 15/07/07 03:28:36 INFO SparkEnv: Registering BlockManagerMaster
> 15/07/07 03:28:36 INFO DiskBlockManager: Created local directory at
>
> /tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca/blockmgr-422d1fcb-bda4-4a2a-93a6-4d49c28cdf28
> 15/07/07 03:28:36 INFO MemoryStore: MemoryStore started with capacity 534.5
> MB
> 15/07/07 03:28:36 INFO HttpFileServer: HTTP File server directory is
>
> /tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca/httpd-6f024c0f-60c4-413b-bb29-eee93e697651
> 15/07/07 03:28:36 INFO HttpServer: Starting HTTP Server
> 15/07/07 03:28:36 INFO Utils: Successfully started service 'HTTP file
> server' on port 46189.
> 15/07/07 03:28:37 INFO SparkEnv: Registering OutputCommitCoordinator
> 15/07/07 03:28:37 INFO Utils: Successfully started service 'SparkUI' on
> port
> 4040.
> 15/07/07 03:28:37 INFO SparkUI: Started SparkUI at
> http://192.168.116.133:4040
> 15/07/07 03:28:37 INFO Executor: Starting executor ID driver on host
> localhost
> 15/07/07 03:28:38 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36884.
> 15/07/07 03:28:38 INFO NettyBlockTransferService: Server created on 36884
> 15/07/07 03:28:38 INFO BlockManagerMaster: Trying to register BlockManager
> 15/07/07 03:28:38 INFO BlockManagerMasterEndpoint: Registering block
> manager
> localhost:36884 with 534.5 MB RAM, BlockManagerId(driver, localhost, 36884)
> 15/07/07 03:28:38 INFO BlockManagerMaster: Registered BlockManager
> 15/07/07 03:28:39 INFO MemoryStore: ensureFreeSpace(110248) called with
> curMem=0, maxMem=560497950
> 15/07/07 03:28:39 INFO MemoryStore: Block broadcast_0 stored as values in
> memory (estimated size 107.7 KB, free 534.4 MB)
> 15/07/07 03:28:40 INFO MemoryStore: ensureFreeSpace(10090) called with
> curMem=110248, maxMem=560497950
> 15/07/07 03:28:40 INFO MemoryStore: Block broadcast_0_piece0 stored as
> bytes
> in memory (estimated size 9.9 KB, free 534.4 MB)
> 15/07/07 03:28:40 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
> on localhost:36884 (size: 9.9 KB, free: 534.5 MB)
> 15/07/07 03:28:40 INFO SparkContext: Created broadcast 0 from textFile at
> SimpleApp.java:19
> 15/07/07 03:28:40 INFO FileInputFormat: Total input paths to process : 1
> 15/07/07 03:28:40 INFO SparkContext: Starting job: count at
> SimpleApp.java:23
> 15/07/07 03:28:40 INFO DAGScheduler: Got job 0 (count at SimpleApp.java:23)
> with 1 output partitions (allowLocal=false)
> 15/07/07 03:28:40 INFO DAGScheduler: Final stage: ResultStage 0(count at
> SimpleApp.java:23)
> 15/07/07 03:28:40 INFO DAGScheduler: Parents of final stage: List()
> 15/07/07 03:28:40 INFO DAGScheduler: Missing parents: List()
> 15/07/07 03:28:40 INFO DAGScheduler: Submitting ResultStage 0
> (MapPartitionsRDD[2] at filter at SimpleApp.java:21), which has no missing
> parents
> 15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(3280) called with
> curMem=120338, maxMem=560497950
> 15/07/07 03:28:41 INFO MemoryStore: Block broadcast_1 stored as values in
> memory (estimated size 3.2 KB, free 534.4 MB)
> 15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(1933) called with
> curMem=123618, maxMem=560497950
> 15/07/07 03:28:41 INFO MemoryStore: Block broadcast_1_piece0 stored as
> bytes
> in memory (estimated size 1933.0 B, free 534.4 MB)
> 15/07/07 03:28:41 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory
> on localhost:36884 (size: 1933.0 B, free: 534.5 MB)
> 15/07/07 03:28:41 INFO SparkContext: Created broadcast 1 from broadcast at
> DAGScheduler.scala:874
> 15/07/07 03:28:41 INFO DAGScheduler: Submitting 1 missing tasks from
> ResultStage 0 (MapPartitionsRDD[2] at filter at SimpleApp.java:21)
> 15/07/07 03:28:41 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
> 15/07/07 03:28:41 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID
> 0, localhost, PROCESS_LOCAL, 1410 bytes)
> 15/07/07 03:28:41 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
> 15/07/07 03:28:41 INFO CacheManager: Partition rdd_1_0 not found, computing
> it
> 15/07/07 03:28:41 INFO HadoopRDD: Input split:
> file:/home/asif/spark-1.4.0/README.md:0+3624
> 15/07/07 03:28:41 INFO deprecation: mapred.tip.id is deprecated. Instead,
> use mapreduce.task.id
> 15/07/07 03:28:41 INFO deprecation: mapred.task.id is deprecated. Instead,
> use mapreduce.task.attempt.id
> 15/07/07 03:28:41 INFO deprecation: mapred.task.is.map is deprecated.
> Instead, use mapreduce.task.ismap
> 15/07/07 03:28:41 INFO deprecation: mapred.task.partition is deprecated.
> Instead, use mapreduce.task.partition
> 15/07/07 03:28:41 INFO deprecation: mapred.job.id is deprecated. Instead,
> use mapreduce.job.id
> 15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(11576) called with
> curMem=125551, maxMem=560497950
> 15/07/07 03:28:41 INFO MemoryStore: Block rdd_1_0 stored as values in
> memory
> (estimated size 11.3 KB, free 534.4 MB)
> 15/07/07 03:28:41 INFO BlockManagerInfo: Added rdd_1_0 in memory on
> localhost:36884 (size: 11.3 KB, free: 534.5 MB)
> 15/07/07 03:28:41 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0).
> 2410 bytes result sent to driver
> 15/07/07 03:28:41 INFO DAGScheduler: ResultStage 0 (count at
> SimpleApp.java:23) finished in 0.542 s
> 15/07/07 03:28:41 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID
> 0) in 520 ms on localhost (1/1)
> 15/07/07 03:28:41 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
> have all completed, from pool
> 15/07/07 03:28:41 INFO DAGScheduler: Job 0 finished: count at
> SimpleApp.java:23, took 0.918158 s
> 15/07/07 03:28:41 INFO SparkContext: Starting job: count at
> SimpleApp.java:27
> 15/07/07 03:28:41 INFO DAGScheduler: Got job 1 (count at SimpleApp.java:27)
> with 1 output partitions (allowLocal=false)
> 15/07/07 03:28:41 INFO DAGScheduler: Final stage: ResultStage 1(count at
> SimpleApp.java:27)
> 15/07/07 03:28:41 INFO DAGScheduler: Parents of final stage: List()
> 15/07/07 03:28:41 INFO DAGScheduler: Missing parents: List()
> 15/07/07 03:28:41 INFO DAGScheduler: Submitting ResultStage 1
> (MapPartitionsRDD[3] at filter at SimpleApp.java:25), which has no missing
> parents
> 15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(3280) called with
> curMem=137127, maxMem=560497950
> 15/07/07 03:28:41 INFO MemoryStore: Block broadcast_2 stored as values in
> memory (estimated size 3.2 KB, free 534.4 MB)
> 15/07/07 03:28:41 INFO MemoryStore: ensureFreeSpace(1933) called with
> curMem=140407, maxMem=560497950
> 15/07/07 03:28:41 INFO MemoryStore: Block broadcast_2_piece0 stored as
> bytes
> in memory (estimated size 1933.0 B, free 534.4 MB)
> 15/07/07 03:28:41 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory
> on localhost:36884 (size: 1933.0 B, free: 534.5 MB)
> 15/07/07 03:28:41 INFO SparkContext: Created broadcast 2 from broadcast at
> DAGScheduler.scala:874
> 15/07/07 03:28:41 INFO DAGScheduler: Submitting 1 missing tasks from
> ResultStage 1 (MapPartitionsRDD[3] at filter at SimpleApp.java:25)
> 15/07/07 03:28:41 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
> 15/07/07 03:28:41 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID
> 1, localhost, PROCESS_LOCAL, 1410 bytes)
> 15/07/07 03:28:41 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
> 15/07/07 03:28:41 INFO BlockManager: Found block rdd_1_0 locally
> 15/07/07 03:28:41 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1).
> 1830 bytes result sent to driver
> 15/07/07 03:28:41 INFO DAGScheduler: ResultStage 1 (count at
> SimpleApp.java:27) finished in 0.020 s
> 15/07/07 03:28:41 INFO DAGScheduler: Job 1 finished: count at
> SimpleApp.java:27, took 0.070602 s
> Lines with a: 60, lines with b: 29
> 15/07/07 03:28:41 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID
> 1) in 24 ms on localhost (1/1)
> 15/07/07 03:28:41 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks
> have all completed, from pool
> 15/07/07 03:28:41 ERROR ContextCleaner: Error in cleaning thread
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> at
>
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:157)
> at
> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1215)
> at
> org.apache.spark.ContextCleaner.org
> $apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:154)
> at
> org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:67)
> 15/07/07 03:28:41 ERROR Utils: uncaught error in thread SparkListenerBus,
> stopping SparkContext
> java.lang.InterruptedException
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
> at
>
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:65)
> at
> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1215)
> at
>
> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
> 15/07/07 03:28:42 INFO SparkUI: Stopped Spark web UI at
> http://192.168.116.133:4040
> 15/07/07 03:28:42 INFO DAGScheduler: Stopping DAGScheduler
> [WARNING] thread Thread[sparkDriver-scheduler-1,5,SimpleApp] was
> interrupted
> but is still alive after waiting at least 14999msecs
> [WARNING] thread Thread[sparkDriver-scheduler-1,5,SimpleApp] will linger
> despite being asked to die via interruption
> [WARNING] thread
> Thread[sparkDriver-akka.actor.default-dispatcher-3,5,SimpleApp] will linger
> despite being asked to die via interruption
> [WARNING] thread
> Thread[sparkDriver-akka.actor.default-dispatcher-4,5,SimpleApp] will linger
> despite being asked to die via interruption
> [WARNING] thread
> Thread[sparkDriver-akka.actor.default-dispatcher-5,5,SimpleApp] will linger
> despite being asked to die via interruption
> [WARNING] thread
> Thread[sparkDriver-akka.actor.default-dispatcher-6,5,SimpleApp] will linger
> despite being asked to die via interruption
> [WARNING] thread
> Thread[sparkDriver-akka.remote.default-remote-dispatcher-7,5,SimpleApp]
> will
> linger despite being asked to die via interruption
> [WARNING] thread Thread[New I/O worker #1,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] thread Thread[New I/O worker #2,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] thread Thread[New I/O boss #3,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] thread Thread[New I/O worker #4,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] thread Thread[New I/O worker #5,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] thread Thread[New I/O server boss #6,5,SimpleApp] will linger
> despite being asked to die via interruption
> [WARNING] thread
> Thread[sparkDriver-akka.remote.default-remote-dispatcher-15,5,SimpleApp]
> will linger despite being asked to die via interruption
> [WARNING] thread Thread[MAP_OUTPUT_TRACKER cleanup timer,5,SimpleApp] will
> linger despite being asked to die via interruption
> [WARNING] thread Thread[BLOCK_MANAGER cleanup timer,5,SimpleApp] will
> linger
> despite being asked to die via interruption
> [WARNING] thread Thread[BROADCAST_VARS cleanup timer,5,SimpleApp] will
> linger despite being asked to die via interruption
> [WARNING] thread Thread[qtp1611265991-33 Acceptor0
> SocketConnector@0.0.0.0:46189,5,SimpleApp] will linger despite being asked
> to die via interruption
> [WARNING] thread Thread[heartbeat-receiver-event-loop-thread,5,SimpleApp]
> will linger despite being asked to die via interruption
> [WARNING] thread Thread[shuffle-server-0,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] thread Thread[SparkListenerBus,5,SimpleApp] will linger despite
> being asked to die via interruption
> [WARNING] NOTE: 20 thread(s) did not finish despite being asked to via
> interruption. This is not a problem with exec:java, it is a problem with
> the
> running code. Although not serious, it should be remedied.
> [WARNING] Couldn't destroy threadgroup
>
> org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=SimpleApp,maxpri=10]
> java.lang.IllegalThreadStateException
> at java.lang.ThreadGroup.destroy(ThreadGroup.java:778)
> at
> org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:328)
> at
>
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> at
>
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
> at
>
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
>
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
>
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> at
>
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
> at
>
> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
> at
>
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
>
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
> at
>
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 26.998s
> [INFO] Finished at: Tue Jul 07 03:28:56 PDT 2015
> [INFO] Final Memory: 20M/60M
> [INFO]
> ------------------------------------------------------------------------
> 15/07/07 03:28:57 INFO DiskBlockManager: Shutdown hook called
> 15/07/07 03:28:57 INFO Utils: path =
>
> /tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca/blockmgr-422d1fcb-bda4-4a2a-93a6-4d49c28cdf28,
> already present as root for deletion.
> 15/07/07 03:28:57 INFO Utils: Shutdown hook called
> 15/07/07 03:28:57 INFO Utils: Deleting directory
>
> /tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca/userFiles-3d0c9f72-b6f0-431f-9ca4-c82d040411c7
> 15/07/07 03:28:57 INFO Utils: Deleting directory
> /tmp/spark-b8a40d9c-d470-404e-bcd7-5763791ffeca
>
> Process finished with exit code 0
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-solve-ThreadException-in-Apache-Spark-standalone-Java-Application-tp23671.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
