Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

2023-08-31 Thread Wenchen Fan
Sorry for the last-minute bug report, but we found a regression in 3.5: the SQL INSERT command without a column list fills missing columns with NULL while Spark 3.4 does not allow it. According to the SQL standard, this shouldn't be allowed and thus a regression in 3.5. The fix has been merged

Re: [DISCUSS] Updating documentation hosted for EOL and maintenance releases

2023-08-31 Thread Matei Zaharia
It would be great to do this IMO, because there are often usability and formatting fixes needed to docs over time, and people naturally search for docs from their *deployed* version of the project — not the latest version, hoping that it also applies to their release. For example, right now

Re: [DISCUSS] SPIP: Python Stored Procedures

2023-08-31 Thread Mich Talebzadeh
I concur with the view point raised by @Sean Owen While this might introduce some challenges related to compatibility and environment issues, it is not fundamentally different from how the users currently import and use common code in Python. The main difference is that now this shared code would

Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

2023-08-31 Thread Ian Manning
+1 (non-binding) Using Spark Core, Spark SQL, Structured Streaming. On Tue, Aug 29, 2023 at 8:12 PM Yuanjian Li wrote: > Please vote on releasing the following candidate(RC3) as Apache Spark > version 3.5.0. > > The vote is open until 11:59pm Pacific time Aug 31st and passes if a > majority +1

Re: [DISCUSS] SPIP: Python Stored Procedures

2023-08-31 Thread Sean Owen
I think you're talking past Hyukjin here. I think the response is: none of that is managed by Pyspark now, and this proposal does not change that. Your current interpreter and environment is used to execute the stored procedure, which is just Python code. It's on you to bring an environment that

Re: [DISCUSS] SPIP: Python Stored Procedures

2023-08-31 Thread Mich Talebzadeh
These are my initial thoughts: As usual your mileage varies. Depending on the use case, introducing support for stored procedures (SP) in Spark SQL with Python as the procedural language *Pros* - Can potentially provide more flexibility and capabilities in the respective SQL workflows. We

Re: [DISCUSS] SPIP: Python Stored Procedures

2023-08-31 Thread Mich Talebzadeh
Thanks Allison! Mich Talebzadeh, Distinguished Technologist, Solutions Architect & Engineer London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any