Indeed valid points raised including the potential typo in the new spark
version. I suggest, in the meantime, you should look for the so called
alternative debugging methods
-
- Simpler explain(), try basic explain() or explain("extended"). This
might provide a less detailed, but
Do you mean Spark 3.4? 4.0 is very much not released yet.
Also it would help if you could share your query & more of the logs leading
up to the error.
On Tue, Feb 20, 2024 at 3:07 PM Sharma, Anup
wrote:
> Hi Spark team,
>
>
>
> We ran into a dataframe issue after upgrading from spark 3.1 to 4.
Hi Spark team,
We ran into a dataframe issue after upgrading from spark 3.1 to 4.
query_result.explain(extended=True)\n File
\"…/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py\"
raise Py4JJavaError(\npy4j.protocol.Py4JJavaError: An error occurred while
calling
