Re: [DISCUSS] SPIP: Python Stored Procedures

Thanks Alison for your explanation. 1. As a matter of interest, what does "sessionCatalog.resolveProcedure" do? Does it recompile the stored procedure (SP)? 2. If the SP makes a reference to an underlying table and table schema is changed. then by definition that SP compiled plan will

Re: [DISCUSS] SPIP: Python Stored Procedures

Hi Mich, Thank you for your comments! I've left some comments on the SPIP, but let's continue the discussion here. You've highlighted the potential advantages of Python stored procedures, and I'd like to emphasize two important aspects: 1. *Versatility*: Integrating Python into SQL provides

Re: [DISCUSS] SPIP: Python Stored Procedures

I have noticed an worthy discussion in the SPIP comments regarding the definition of "stored procedure" in the context of Spark, and I believe it is an important point to address. To provide some historical context, Sybase , a

Re: [DISCUSS] SPIP: Python Stored Procedures

I concur with the view point raised by @Sean Owen While this might introduce some challenges related to compatibility and environment issues, it is not fundamentally different from how the users currently import and use common code in Python. The main difference is that now this shared code would

Re: [DISCUSS] SPIP: Python Stored Procedures

I think you're talking past Hyukjin here. I think the response is: none of that is managed by Pyspark now, and this proposal does not change that. Your current interpreter and environment is used to execute the stored procedure, which is just Python code. It's on you to bring an environment that

Re: [DISCUSS] SPIP: Python Stored Procedures

These are my initial thoughts: As usual your mileage varies. Depending on the use case, introducing support for stored procedures (SP) in Spark SQL with Python as the procedural language *Pros* - Can potentially provide more flexibility and capabilities in the respective SQL workflows. We

Re: [DISCUSS] SPIP: Python Stored Procedures

Thanks Allison! Mich Talebzadeh, Distinguished Technologist, Solutions Architect & Engineer London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any

Re: [DISCUSS] SPIP: Python Stored Procedures

> Which Python version will run that stored procedure? > > All Python versions supported in PySpark > Where in stored procedure defines the exact python version which will run the code? That was the question. > How to manage external dependencies? > > Existing way we have >

Re: [DISCUSS] SPIP: Python Stored Procedures

Which Python version will run that stored procedure? All Python versions supported in PySpark How to manage external dependencies? Existing way we have https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html . In fact, this will use the external dependencies within your

Re: [DISCUSS] SPIP: Python Stored Procedures

-1 Great idea to ignore the experience of others and copy bad practices back for nothing. If you are familiar with Python ecosystem then you should answer the questions: 1. Which Python version will run that stored procedure? 2. How to manage external dependencies? 3. How to test it via a common

Re: [DISCUSS] SPIP: Python Stored Procedures

+1 we should have this .. a lot of other projects and DBMSes have this too, and we currently don't have a way to handle them within Apache Spark. Disclaimer: I am the shepherd of this SPIP. On Thu, 31 Aug 2023 at 09:31, Allison Wang wrote: > Hi Mich, > > I've updated the permissions on the

Re: [DISCUSS] SPIP: Python Stored Procedures

Hi Mich, I've updated the permissions on the document. Please feel free to leave comments. Thanks, Allison On Wed, Aug 30, 2023 at 3:44 PM Mich Talebzadeh wrote: > Hi, > > Great. Please allow edit access on SPIP or ability to comment. > > Thanks > > Mich Talebzadeh, > Distinguished

Re: [DISCUSS] SPIP: Python Stored Procedures

Hi Mich, I've updated the permissions on the document. Please feel free to leave comments. Thanks, Allison On Wed, Aug 30, 2023 at 3:44 PM Mich Talebzadeh wrote: > Hi, > > Great. Please allow edit access on SPIP or ability to comment. > > Thanks > > Mich Talebzadeh, > Distinguished

Re: [DISCUSS] SPIP: Python Stored Procedures

Hi, Great. Please allow edit access on SPIP or ability to comment. Thanks Mich Talebzadeh, Distinguished Technologist, Solutions Architect & Engineer London United Kingdom view my Linkedin profile

[DISCUSS] SPIP: Python Stored Procedures

Hi all, I would like to start a discussion on “Python Stored Procedures". This proposal aims to extend Spark SQL by introducing support for stored procedures, starting with Python as the procedural language. This will enable users to run complex logic using Python within their SQL workflows and