+1 On Thu, 19 Mar 2026 at 06:42, Szehon Ho <[email protected]> wrote:
> +1 (non binding) > > Thanks > Szehon > > On Wed, Mar 18, 2026 at 2:17 PM Yicong Huang <[email protected]> > wrote: > >> +1 (non-binding) >> >> >> On Wed, Mar 18, 2026 at 1:49 PM DB Tsai <[email protected]> wrote: >> >>> +1 >>> >>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >>> >>> On Mar 18, 2026, at 12:24 PM, Chao Sun <[email protected]> wrote: >>> >>> +1 (binding) >>> >>> On Wed, Mar 18, 2026 at 12:00 PM Herman van Hovell via dev < >>> [email protected]> wrote: >>> >>>> +1 >>>> >>>> On Wed, Mar 18, 2026 at 2:57 PM Daniel Tenedorio <[email protected]> >>>> wrote: >>>> >>>>> +1 (non-binding) >>>>> >>>>> Thank you for your persistence working on this proposal and figuring >>>>> out the details. >>>>> >>>>> On 2026/03/18 09:32:40 Haiyang Sun via dev wrote: >>>>> > Hi Spark devs, >>>>> > >>>>> > I would like to call for *a new vote following the previous attempt* >>>>> for the >>>>> > *SPIP: Language-Agnostic UDF Execution Protocol for Spark *after >>>>> addressing >>>>> > comments and providing a supplementary design document for worker >>>>> > specification. >>>>> > >>>>> > The SPIP proposes a structured, language-agnostic framework for >>>>> running >>>>> > user-defined functions (UDFs) in Spark across multiple programming >>>>> languages >>>>> > >>>>> > Today, Spark Connect allows users to write queries from multiple >>>>> languages, >>>>> > but support for user-defined functions remains incomplete. In >>>>> practice, >>>>> > only Scala, Java, Python have working support, and this relies on >>>>> > language-specific mechanisms that do not generalize well to other >>>>> languages >>>>> > such as Go <https://github.com/apache/spark-connect-go> / Rust >>>>> > <https://github.com/apache/spark-connect-rust> / Swift >>>>> > <https://github.com/apache/spark-connect-swift> / TypeScript >>>>> > <https://github.com/BaldrVivaldelli/ts-spark-connector> where UDF >>>>> support >>>>> > is currently unavailable. In addition, there are legacy limitations >>>>> in the >>>>> > existing PySpark worker implementation that make it difficult to >>>>> evolve the >>>>> > system or extend it to new languages. >>>>> > >>>>> > The proposal introduces two related components: >>>>> > >>>>> > >>>>> > 1. >>>>> > >>>>> > *A unified UDF execution protocol* >>>>> > >>>>> > The proposal defines a structured API and execution protocol for >>>>> running >>>>> > UDFs outside the Spark executor process and communicating with >>>>> Spark via >>>>> > inter-process communication (IPC). This protocol enables Spark to >>>>> interact >>>>> > with external UDF workers in a consistent and extensible way, >>>>> regardless of >>>>> > the implementation language. >>>>> > 2. >>>>> > >>>>> > *A worker specification for provisioning and lifecycle >>>>> management.* >>>>> > >>>>> > To support multi-language execution environments, the proposal >>>>> also >>>>> > introduces a worker specification describing how UDF workers can >>>>> be >>>>> > installed, started, connected to, and terminated. This document >>>>> complements >>>>> > the SPIP by outlining how workers can be provisioned and managed >>>>> in a >>>>> > consistent way. >>>>> > >>>>> > Note that this SPIP can help enable UDF support for languages that >>>>> > currently do not support UDFs. For languages that already have UDF >>>>> > implementations (especially Python), the goal is not to replace >>>>> existing >>>>> > implementations immediately, but to provide a framework that may >>>>> allow them >>>>> > to gradually evolve toward more language-agnostic abstractions over >>>>> time. >>>>> > >>>>> > More details can be found in the SPIP document and the supplementary >>>>> design >>>>> > for worker specification: >>>>> > >>>>> > SPIP: >>>>> > >>>>> https://docs.google.com/document/d/19Whzq127QxVt2Luk0EClgaDtcpBsFUp67NcVdKKyPF8 >>>>> > >>>>> > Worker specification design document: >>>>> > >>>>> https://docs.google.com/document/d/1Dx9NqHRNuUpatH9DYoFF9cmvUl2fqHT4Rjbyw4EGLHs >>>>> > >>>>> > Discussion Thread: >>>>> > https://lists.apache.org/thread/9t4svsnd71j7sb4r4scf2xhh8dvp3b43 >>>>> > >>>>> > Previous vote and discussion thread: >>>>> > https://lists.apache.org/thread/81xghrfwvopp274rgyxfthsstb2xmkz1 >>>>> > >>>>> > *Please vote on adopting this proposal.* >>>>> > >>>>> > [ ] +1: Accept the proposal as an official SPIP >>>>> > >>>>> > [ ] +0: No opinion >>>>> > >>>>> > [ ] -1: Disapprove (please explain why) >>>>> > >>>>> > The vote will remain open for *at least 72 hours. * >>>>> > >>>>> > Thanks to everyone who participated in the discussion and provided >>>>> valuable >>>>> > feedback! >>>>> > >>>>> > >>>>> > Best regards, >>>>> > >>>>> > Haiyang >>>>> > >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe e-mail: [email protected] >>>>> >>>>> >>>
