[jira] [Created] (SPARK-48390) SparkListenerBus not sending tableName details in logical plan for spark versions 3.4.2 and above
Mayur Madnani created SPARK-48390:
-
Summary: SparkListenerBus not sending tableName details in logical
plan for spark versions 3.4.2 and above
Key: SPARK-48390
URL: https://issues.apache.org/jira/browse/SPARK-48390
Project: Spark
Issue Type: Bug
Components: Spark Core, SQL
Affects Versions: 3.4.3, 3.5.1, 3.5.0, 3.4.2, 3.5.2
Reporter: Mayur Madnani
In OpenLineage, via SparkEventListener a logical plan event is received and by
parsing it the frameworks deduces Input/Output table names to create a lineage.
The issue is that in spark versions 3.4.2 and above (tested and reproducible in
3.4.2 & 3.5.0) the logical plan event sent by spark core is partial and is
missing the tableName property which was been sent in earlier versions (working
in spark 3.3.4).
+_Note: This issue is only encountered in drop table events._+
For a drop table event, see below the logical plan in different spark versions
*Spark 3.3.4*
{code:java}
[
{
"class": "org.apache.spark.sql.execution.command.DropTableCommand",
"num-children": 0,
"tableName":
{ "product-class": "org.apache.spark.sql.catalyst.TableIdentifier", "table":
"drop_table_test", "database": "default" }
,
"ifExists": false,
"isView": false,
"purge": false
}
]
{code}
*Spark 3.4.2*
{code:java}
[
{ "class": "org.apache.spark.sql.catalyst.plans.logical.DropTable",
"num-children": 1, "child": 0, "ifExists": false, "purge": false }
,
{ "class": "org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier",
"num-children": 0, "catalog": null, "identifier": null }
]
{code}
More details in referenced issue here:
[https://github.com/OpenLineage/OpenLineage/issues/2716]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48092) Spark images are based of Python 3.8 which is soon EOL
[ https://issues.apache.org/jira/browse/SPARK-48092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayur Madnani updated SPARK-48092: -- Description: Python 3.8 will be EOL in Oct 2024 and all the Spark docker images are based on Python 3.8 as of now. I am proposing to use Python 3.10 as default. Let me know if I can pick this up to make the changes in [spark-docker|[https://github.com/apache/spark-docker]] was: Python 3.8 will be EOL in Oct 2024 and all the Spark docker images are based on Python 3.8 as of now. I am proposing to use Python 3.10 as default. Let me know if I can pick this up to make the changes in [spark-docker|[https://github.com/apache/spark-docker]] !Screenshot 2024-05-02 at 21.00.18.png! > Spark images are based of Python 3.8 which is soon EOL > -- > > Key: SPARK-48092 > URL: https://issues.apache.org/jira/browse/SPARK-48092 > Project: Spark > Issue Type: Bug > Components: Spark Docker >Affects Versions: 3.4.2, 3.4.0, 3.4.1, 3.5.0, 3.5.1 >Reporter: Mayur Madnani >Priority: Blocker > Attachments: Screenshot 2024-05-02 at 21.00.18.png, Screenshot > 2024-05-02 at 21.00.48.png > > > Python 3.8 will be EOL in Oct 2024 and all the Spark docker images are based > on Python 3.8 as of now. > I am proposing to use Python 3.10 as default. Let me know if I can pick this > up to make the changes in > [spark-docker|[https://github.com/apache/spark-docker]] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48092) Spark images are based of Python 3.8 which is soon EOL
[ https://issues.apache.org/jira/browse/SPARK-48092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayur Madnani updated SPARK-48092: -- Attachment: Screenshot 2024-05-02 at 21.00.18.png Screenshot 2024-05-02 at 21.00.48.png > Spark images are based of Python 3.8 which is soon EOL > -- > > Key: SPARK-48092 > URL: https://issues.apache.org/jira/browse/SPARK-48092 > Project: Spark > Issue Type: Bug > Components: Spark Docker >Affects Versions: 3.4.2, 3.4.0, 3.4.1, 3.5.0, 3.5.1 >Reporter: Mayur Madnani >Priority: Blocker > Attachments: Screenshot 2024-05-02 at 21.00.18.png, Screenshot > 2024-05-02 at 21.00.48.png > > > Python 3.8 will be EOL in Oct 2024 and all the Spark docker images are based > on Python 3.8 as of now. > I am proposing to use Python 3.10 as default. Let me know if I can pick this > up to make the changes in > [spark-docker|[https://github.com/apache/spark-docker]] > > !image-2024-05-02-14-45-18-492.png! > !image-2024-05-02-14-44-43-423.png! > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48092) Spark images are based of Python 3.8 which is soon EOL
[ https://issues.apache.org/jira/browse/SPARK-48092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayur Madnani updated SPARK-48092: -- Description: Python 3.8 will be EOL in Oct 2024 and all the Spark docker images are based on Python 3.8 as of now. I am proposing to use Python 3.10 as default. Let me know if I can pick this up to make the changes in [spark-docker|[https://github.com/apache/spark-docker]] !Screenshot 2024-05-02 at 21.00.18.png! was: Python 3.8 will be EOL in Oct 2024 and all the Spark docker images are based on Python 3.8 as of now. I am proposing to use Python 3.10 as default. Let me know if I can pick this up to make the changes in [spark-docker|[https://github.com/apache/spark-docker]] !image-2024-05-02-14-45-18-492.png! !image-2024-05-02-14-44-43-423.png! > Spark images are based of Python 3.8 which is soon EOL > -- > > Key: SPARK-48092 > URL: https://issues.apache.org/jira/browse/SPARK-48092 > Project: Spark > Issue Type: Bug > Components: Spark Docker >Affects Versions: 3.4.2, 3.4.0, 3.4.1, 3.5.0, 3.5.1 >Reporter: Mayur Madnani >Priority: Blocker > Attachments: Screenshot 2024-05-02 at 21.00.18.png, Screenshot > 2024-05-02 at 21.00.48.png > > > Python 3.8 will be EOL in Oct 2024 and all the Spark docker images are based > on Python 3.8 as of now. > I am proposing to use Python 3.10 as default. Let me know if I can pick this > up to make the changes in > [spark-docker|[https://github.com/apache/spark-docker]] > > !Screenshot 2024-05-02 at 21.00.18.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48092) Spark images are based of Python 3.8 which is soon EOL
Mayur Madnani created SPARK-48092: - Summary: Spark images are based of Python 3.8 which is soon EOL Key: SPARK-48092 URL: https://issues.apache.org/jira/browse/SPARK-48092 Project: Spark Issue Type: Bug Components: Spark Docker Affects Versions: 3.5.1, 3.5.0, 3.4.1, 3.4.0, 3.4.2 Reporter: Mayur Madnani Python 3.8 will be EOL in Oct 2024 and all the Spark docker images are based on Python 3.8 as of now. I am proposing to use Python 3.10 as default. Let me know if I can pick this up to make the changes in [spark-docker|[https://github.com/apache/spark-docker]] !image-2024-05-02-14-45-18-492.png! !image-2024-05-02-14-44-43-423.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
