sagarlakshmipathy commented on issue #4924: URL: https://github.com/apache/incubator-gluten/issues/4924#issuecomment-1989783821
Just to add, Spark 3.2 (both v1.1.1 and v1.1.0) fails as well. ` ./spark-3.2.4-bin-hadoop3.2/bin/spark-shell --master yarn --deploy-mode client --jars https://github.com/apache/incubator-gluten/releases/download/v1.1.0/gluten-velox-bundle-spark3.2_2.12-1.1.0.jar --conf spark.plugins=io.glutenproject.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=10g --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager ` ``` Exception in thread "main" java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894) at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:422) at org.apache.spark.executor.YarnCoarseGrainedExecutorBackend$.main(YarnCoarseGrainedExecutorBackend.scala:81) at org.apache.spark.executor.YarnCoarseGrainedExecutorBackend.main(YarnCoarseGrainedExecutorBackend.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.shuffle.sort.ColumnarShuffleManager at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:216) at org.apache.spark.util.Utils$.instantiateSerializerOrShuffleManager(Utils.scala:2671) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:315) at org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:207) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$7(CoarseGrainedExecutorBackend.scala:471) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) ... 4 more [2024-03-12 01:59:05.500]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : 2024-03-12 01:59:04,092 INFO executor.CoarseGrainedExecutorBackend: Started daemon with process name: 15010@ip-10-0-93-24 2024-03-12 01:59:04,098 INFO util.SignalUtils: Registering signal handler for TERM 2024-03-12 01:59:04,100 INFO util.SignalUtils: Registering signal handler for HUP 2024-03-12 01:59:04,100 INFO util.SignalUtils: Registering signal handler for INT 2024-03-12 01:59:04,508 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2024-03-12 01:59:04,622 INFO spark.SecurityManager: Changing view acls to: yarn,hadoop 2024-03-12 01:59:04,622 INFO spark.SecurityManager: Changing modify acls to: yarn,hadoop 2024-03-12 01:59:04,623 INFO spark.SecurityManager: Changing view acls groups to: 2024-03-12 01:59:04,624 INFO spark.SecurityManager: Changing modify acls groups to: 2024-03-12 01:59:04,624 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set() 2024-03-12 01:59:05,025 INFO client.TransportClientFactory: Successfully created connection to ip-10-0-98-19.us-west-2.compute.internal/10.0.98.19:44481 after 70 ms (0 ms spent in bootstraps) 2024-03-12 01:59:05,119 INFO spark.SecurityManager: Changing view acls to: yarn,hadoop 2024-03-12 01:59:05,119 INFO spark.SecurityManager: Changing modify acls to: yarn,hadoop 2024-03-12 01:59:05,119 INFO spark.SecurityManager: Changing view acls groups to: 2024-03-12 01:59:05,120 INFO spark.SecurityManager: Changing modify acls groups to: 2024-03-12 01:59:05,120 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set() 2024-03-12 01:59:05,169 INFO client.TransportClientFactory: Successfully created connection to ip-10-0-98-19.us-west-2.compute.internal/10.0.98.19:44481 after 1 ms (0 ms spent in bootstraps) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
