+1 (non-binding)
On Wed, Mar 18, 2026 at 1:49 PM DB Tsai <[email protected]> wrote: > +1 > > DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > > On Mar 18, 2026, at 12:24 PM, Chao Sun <[email protected]> wrote: > > +1 (binding) > > On Wed, Mar 18, 2026 at 12:00 PM Herman van Hovell via dev < > [email protected]> wrote: > >> +1 >> >> On Wed, Mar 18, 2026 at 2:57 PM Daniel Tenedorio <[email protected]> >> wrote: >> >>> +1 (non-binding) >>> >>> Thank you for your persistence working on this proposal and figuring out >>> the details. >>> >>> On 2026/03/18 09:32:40 Haiyang Sun via dev wrote: >>> > Hi Spark devs, >>> > >>> > I would like to call for *a new vote following the previous attempt* >>> for the >>> > *SPIP: Language-Agnostic UDF Execution Protocol for Spark *after >>> addressing >>> > comments and providing a supplementary design document for worker >>> > specification. >>> > >>> > The SPIP proposes a structured, language-agnostic framework for running >>> > user-defined functions (UDFs) in Spark across multiple programming >>> languages >>> > >>> > Today, Spark Connect allows users to write queries from multiple >>> languages, >>> > but support for user-defined functions remains incomplete. In practice, >>> > only Scala, Java, Python have working support, and this relies on >>> > language-specific mechanisms that do not generalize well to other >>> languages >>> > such as Go <https://github.com/apache/spark-connect-go> / Rust >>> > <https://github.com/apache/spark-connect-rust> / Swift >>> > <https://github.com/apache/spark-connect-swift> / TypeScript >>> > <https://github.com/BaldrVivaldelli/ts-spark-connector> where UDF >>> support >>> > is currently unavailable. In addition, there are legacy limitations in >>> the >>> > existing PySpark worker implementation that make it difficult to >>> evolve the >>> > system or extend it to new languages. >>> > >>> > The proposal introduces two related components: >>> > >>> > >>> > 1. >>> > >>> > *A unified UDF execution protocol* >>> > >>> > The proposal defines a structured API and execution protocol for >>> running >>> > UDFs outside the Spark executor process and communicating with >>> Spark via >>> > inter-process communication (IPC). This protocol enables Spark to >>> interact >>> > with external UDF workers in a consistent and extensible way, >>> regardless of >>> > the implementation language. >>> > 2. >>> > >>> > *A worker specification for provisioning and lifecycle management.* >>> > >>> > To support multi-language execution environments, the proposal also >>> > introduces a worker specification describing how UDF workers can be >>> > installed, started, connected to, and terminated. This document >>> complements >>> > the SPIP by outlining how workers can be provisioned and managed in >>> a >>> > consistent way. >>> > >>> > Note that this SPIP can help enable UDF support for languages that >>> > currently do not support UDFs. For languages that already have UDF >>> > implementations (especially Python), the goal is not to replace >>> existing >>> > implementations immediately, but to provide a framework that may allow >>> them >>> > to gradually evolve toward more language-agnostic abstractions over >>> time. >>> > >>> > More details can be found in the SPIP document and the supplementary >>> design >>> > for worker specification: >>> > >>> > SPIP: >>> > >>> https://docs.google.com/document/d/19Whzq127QxVt2Luk0EClgaDtcpBsFUp67NcVdKKyPF8 >>> > >>> > Worker specification design document: >>> > >>> https://docs.google.com/document/d/1Dx9NqHRNuUpatH9DYoFF9cmvUl2fqHT4Rjbyw4EGLHs >>> > >>> > Discussion Thread: >>> > https://lists.apache.org/thread/9t4svsnd71j7sb4r4scf2xhh8dvp3b43 >>> > >>> > Previous vote and discussion thread: >>> > https://lists.apache.org/thread/81xghrfwvopp274rgyxfthsstb2xmkz1 >>> > >>> > *Please vote on adopting this proposal.* >>> > >>> > [ ] +1: Accept the proposal as an official SPIP >>> > >>> > [ ] +0: No opinion >>> > >>> > [ ] -1: Disapprove (please explain why) >>> > >>> > The vote will remain open for *at least 72 hours. * >>> > >>> > Thanks to everyone who participated in the discussion and provided >>> valuable >>> > feedback! >>> > >>> > >>> > Best regards, >>> > >>> > Haiyang >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: [email protected] >>> >>> >
