The job failed when we upgraded from spark 3.3.1 to spark3.4.1

2023-11-14 Thread Hanyu Huang
The version our job originally ran was spark 3.3.1 and Apache Iceberg to 1.2.0, But since we upgraded to spark3.4.1 and Apache Iceberg to 1.3.1, jobs started to fail frequently, We tried to upgrade only iceberg without upgrading spark, and the job did not report an error. Detailed description:

Re: Okio Vulnerability in Spark 3.4.1

2023-11-14 Thread Bjørn Jørgensen
FYI I have opened Update okio to version 1.17.6 for this now. tor. 31. aug. 2023 kl. 21:18 skrev Sean Owen : > It's a dependency of some other HTTP library. Use mvn dependency:tree to > see where it comes from. It may be more

Why create/drop/alter/rename partition does not post listener event in ExternalCatalogWithListener?

2023-11-14 Thread 李响
Dear Spark Community: In ExternalCatalogWithListener , I see postToAll() is called for create/drop/alter/rename database/table/function to post