[
https://issues.apache.org/jira/browse/SPARK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tanel Kiis updated SPARK-35296:
-------------------------------
Description:
I hit this assertion error when using dataset.observe:
{code}
{code}
was:
I hit this assertion error when using dataset.observe:
{code}
java.lang.AssertionError: assertion failed
at scala.Predef$.assert(Predef.scala:208)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
org.apache.spark.sql.execution.AggregatingAccumulator.setState(AggregatingAccumulator.scala:204)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
org.apache.spark.sql.execution.CollectMetricsExec.$anonfun$doExecute$2(CollectMetricsExec.scala:72)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
org.apache.spark.sql.execution.CollectMetricsExec.$anonfun$doExecute$2$adapted(CollectMetricsExec.scala:71)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
org.apache.spark.TaskContext$$anon$1.onTaskCompletion(TaskContext.scala:125)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
org.apache.spark.TaskContextImpl.$anonfun$markTaskCompleted$1(TaskContextImpl.scala:124)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
org.apache.spark.TaskContextImpl.$anonfun$markTaskCompleted$1$adapted(TaskContextImpl.scala:124)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
org.apache.spark.TaskContextImpl.$anonfun$invokeListeners$1(TaskContextImpl.scala:137)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
org.apache.spark.TaskContextImpl.$anonfun$invokeListeners$1$adapted(TaskContextImpl.scala:135)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:135)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:124)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at org.apache.spark.scheduler.Task.run(Task.scala:147)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
~[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
[bell-hadoop-utils-container-1.1-20210503.142959-520-jar-with-dependencies.jar:1.1-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
{code}
> Dataset.observe fails with an assertion
> ---------------------------------------
>
> Key: SPARK-35296
> URL: https://issues.apache.org/jira/browse/SPARK-35296
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.1
> Reporter: Tanel Kiis
> Priority: Major
>
> I hit this assertion error when using dataset.observe:
> {code}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
