dossett commented on pull request #29231: URL: https://github.com/apache/spark/pull/29231#issuecomment-710061815
Running on GCP's dataproc 2.0: ``` aniskodedossett@dossett-delta-w-0:~$ spark-sql --version Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 3.0.1 /_/ Using Scala version 2.12.12, OpenJDK 64-Bit Server VM, 1.8.0_265 Branch HEAD Compiled by user on 2020-09-17T10:01:54Z Revision 0aee93de8ef2a90403093b91843de9777b7ab5ef Url https://bigdataoss-internal.googlesource.com/third_party/apache/spark Type --help for more information. ``` I'm playing with the Databricks Delta Lake a simple `vacuum` command fails with a long stack trace with this at the bottom: ``` Caused by: java.lang.NoSuchFieldError: numNonEmptyBlocks at org.apache.spark.scheduler.HighlyCompressedMapStatus.<init>(MapStatus.scala:174) at org.apache.spark.scheduler.HighlyCompressedMapStatus$.apply(MapStatus.scala:269) at org.apache.spark.scheduler.MapStatus$.apply(MapStatus.scala:70) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:71) at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` If this is helpful I'm happy to provide more information. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org