Apologies. Issue is seen after we upgraded from Spark 3.1 to Spark 3.3. The
same query runs fine on Spark 3.1.
Omit the Spark version mentioned in email subject earlier.
Anup
Error trace:
query_result.explain(extended=True)\n File
\"…/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py\"
raise Py4JJavaError(\npy4j.protocol.Py4JJavaError: An error occurred while
calling z:org.apache.spark.sql.api.python.PythonSQLUtils.explainString.\n:
java.lang.IllegalStateException: You hit a query analyzer bug. Please report
your query to Spark user mailing list.\n\tat
org.apache.spark.sql.execution.SparkStrategies$Aggregation$.apply(SparkStrategies.scala:516)\n\tat
org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$1(QueryPlanner.scala:63)\n\tat
scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)\n\tat
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)\n\tat
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491)\n\tat
org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)\n\tat
org.apache.spark.sql.execution.SparkStrategies.plan(SparkStrategies.scala:72)\n\tat
org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$3(QueryPlanner.scala:78)\n\tat
scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:196)\n\tat
scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:194)\n\tat
scala.collection.Iterator.foreach(Iterator.scala:943)\n\tat
scala.collection.Iterator.foreach$(Iterator.scala:943)\n\tat
scala.collection.AbstractIterator.foreach(Iterator.scala:1431)\n\tat
scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:199)\n\tat
scala.collect...
From: "Sharma, Anup"
Date: Tuesday, February 20, 2024 at 4:58 PM
To: "user@spark.apache.org"
Cc: "Thinderu, Shalini"
Subject: Spark 4.0 Query Analyzer Bug Report
Hi Spark team,
We ran into a dataframe issue after upgrading from spark 3.1 to 4.
query_result.explain(extended=True)\n File
\"…/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py\"
raise Py4JJavaError(\npy4j.protocol.Py4JJavaError: An error occurred while
calling z:org.apache.spark.sql.api.python.PythonSQLUtils.explainString.\n:
java.lang.IllegalStateException: You hit a query analyzer bug. Please report
your query to Spark user mailing list.\n\tat
org.apache.spark.sql.execution.SparkStrategies$Aggregation$.apply(SparkStrategies.scala:516)\n\tat
org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$1(QueryPlanner.scala:63)\n\tat
scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)\n\tat
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)\n\tat
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491)\n\tat
org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:93)\n\tat
org.apache.spark.sql.execution.SparkStrategies.plan(SparkStrategies.scala:72)\n\tat
org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$3(QueryPlanner.scala:78)\n\tat
scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:196)\n\tat
scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:194)\n\tat
scala.collection.Iterator.foreach(Iterator.scala:943)\n\tat
scala.collection.Iterator.foreach$(Iterator.scala:943)\n\tat
scala.collection.AbstractIterator.foreach(Iterator.scala:1431)\n\tat
scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:199)\n\tat
scala.collect...
Could you please let us know if this is already being looked at?
Thanks,
Anup