Help in understanding Exchange in Spark UI

[Dhruv Singla]

Hey Team

I've posted a question of StackOverflow. The link is -
https://stackoverflow.com/questions/78644118/understanding-exchange-in-spark-ui

I haven't got any responses yet. If possible could you please look into it?
If you need me to write the question in the mailing list, I can do that as
well.

Thanks & Regards
Dhruv

Re: [Spark on SBT] Executor just keeps running

[Dhruv Singla]

You can reproduce the behavior in ordinary Scala code if you keep reduce in
an object outside the main method. Hope it might help

On Mon, Apr 17, 2023 at 10:22 PM Dhruv Singla  wrote:

> Hi Team
>I was trying to run spark using `sbt console` on the terminal. I am
> able to build the project successfully using build.sbt and the following
> piece of code runs fine on IntelliJ. The only issue I am facing while
> running the same on terminal is that the Executor keeps running and is not
> able to complete the task. I don't know if it is not able to get the
> resources it wants or something else is stopping it.
> Here's the Code
> ```
>
> import org.apache.spark._val sc = new SparkContext("local[1]", 
> "SimpleProg")val nums = sc.parallelize(List(1, 2, 3, 4));
> println(nums.reduce((a, b) => a - b))
>
> ```
>
> I've attached a file that contains the errors that show up when I manually
> stop the program using `Ctrl+C`.
>

[Spark on SBT] Executor just keeps running

[Dhruv Singla]

Hi Team
   I was trying to run spark using `sbt console` on the terminal. I am
able to build the project successfully using build.sbt and the following
piece of code runs fine on IntelliJ. The only issue I am facing while
running the same on terminal is that the Executor keeps running and is not
able to complete the task. I don't know if it is not able to get the
resources it wants or something else is stopping it.
Here's the Code
```

import org.apache.spark._val sc = new SparkContext("local[1]",
"SimpleProg")val nums = sc.parallelize(List(1, 2, 3, 4));
println(nums.reduce((a, b) => a - b))

```

I've attached a file that contains the errors that show up when I manually
stop the program using `Ctrl+C`.
23/04/16 19:04:39 INFO SparkContext: Starting job: reduce at :17
23/04/16 19:04:39 INFO DAGScheduler: Got job 2 (reduce at :17) with 1 
output partitions
23/04/16 19:04:39 INFO DAGScheduler: Final stage: ResultStage 2 (reduce at 
:17)
23/04/16 19:04:39 INFO DAGScheduler: Parents of final stage: List()
23/04/16 19:04:39 INFO DAGScheduler: Missing parents: List()
23/04/16 19:04:39 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 
192.168.0.120:50005 in memory (size: 2.3 KiB, free: 434.4 MiB)
23/04/16 19:04:39 INFO DAGScheduler: Submitting ResultStage 2 
(ParallelCollectionRDD[0] at parallelize at :16), which has no missing 
parents
23/04/16 19:04:39 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 
192.168.0.120:50005 in memory (size: 1279.0 B, free: 434.4 MiB)
23/04/16 19:04:39 INFO MemoryStore: Block broadcast_3 stored as values in 
memory (estimated size 2.2 KiB, free 434.3 MiB)
23/04/16 19:04:39 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in 
memory (estimated size 1389.0 B, free 434.3 MiB)
23/04/16 19:04:39 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 
192.168.0.120:50005 (size: 1389.0 B, free: 434.4 MiB)
23/04/16 19:04:39 INFO SparkContext: Created broadcast 3 from broadcast at 
DAGScheduler.scala:1200
23/04/16 19:04:39 INFO DAGScheduler: Submitting 1 missing tasks from 
ResultStage 2 (ParallelCollectionRDD[0] at parallelize at :16) (first 
15 
tasks are for partitions Vector(0))
23/04/16 19:04:39 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
23/04/16 19:04:39 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, 
192.168.0.120, executor driver, partition 0, PROCESS_LOCAL, 7290 bytes)
23/04/16 19:04:39 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)

[warn] Canceling execution...
[warn] Run canceled.
23/04/16 19:05:30 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 2)
java.lang.NoClassDefFoundError: Could not initialize class 
$line9.$read$$iw$$iw$$iw$$iw$
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
java.base/java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:237)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
java.base/java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1250)
at 
java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2096)
at 
java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594)
at 
java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:1993)
at 
java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1588)
at 
java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2355)
at 
java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2249)
at 
java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2087)
at 
java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594)
at 
java.base/java.io.ObjectInputStream.readArray(ObjectInputStream.java:1993)
at 
java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1588)
at 
java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2355)
at 
java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2249)
at 
java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2087)
at 
java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594)
at 
java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2355)
at