Thanks Haiyang for driving this. +1 from my side too. On Wed, 18 Mar 2026 at 03:57, Holden Karau <[email protected]> wrote:
> This is great! I think we should maybe call a new vote but with the worker > spec I’m now much more comfortable with the design. > > Twitter: https://twitter.com/holdenkarau > Fight Health Insurance: https://www.fighthealthinsurance.com/ > <https://www.fighthealthinsurance.com/?q=hk_email> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > Pronouns: she/her > > > On Tue, Mar 17, 2026 at 9:11 AM Haiyang Sun <[email protected]> > wrote: > >> Hi Holden, >> >> >> As promised, here is the additional document >> <https://docs.google.com/document/d/1Dx9NqHRNuUpatH9DYoFF9cmvUl2fqHT4Rjbyw4EGLHs/edit?tab=t.0#heading=h.4h01j4b8rjzv> >> describing the proposed worker specification design, which I have also >> attached to the JIRA ticket: SPARK-55278 >> <https://issues.apache.org/jira/browse/SPARK-55278>. >> >> As discussed, I hope this helps clarify the concerns raised earlier. >> >> Thank you for your feedback. >> >> Best, >> >> Haiyang >> >> On Thu, Feb 26, 2026 at 10:30 PM Holden Karau <[email protected]> >> wrote: >> >>> Looking forward to these additional docs :) >>> >>> Twitter: https://twitter.com/holdenkarau >>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>> <https://www.fighthealthinsurance.com/?q=hk_email> >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> Pronouns: she/her >>> >>> >>> On Thu, Feb 26, 2026 at 12:50 PM Haiyang Sun via dev < >>> [email protected]> wrote: >>> >>>> Hi Holden, >>>> >>>> Thanks again for the detailed comments and suggestions. >>>> >>>> I’ve responded inline in the document and will revise the SPIP to make >>>> several areas more explicit. For visibility, here is a short summary: >>>> >>>> 1) Security (new IPC mechanism) >>>> >>>> We will add a dedicated security section. Overall, this should not be >>>> worse than the current socket-based implementation. Moving to gRPC may >>>> actually improve our position by leveraging existing ecosystem support for >>>> TLS, authentication, interceptors, and observability — which are harder to >>>> standardize correctly on top of a raw socket protocol. >>>> >>>> 2) Performance assumptions >>>> >>>> Agreed — we should back claims with systematic benchmarking. We have an >>>> early gRPC prototype with preliminary results comparable to the current >>>> socket path, but we will avoid strong claims until properly benchmarked. >>>> The existing Python/Scala paths will remain, and any default switch would >>>> only happen after meeting explicit performance goals. >>>> >>>> 3) Fallback / migration strategy >>>> >>>> We will make this explicit in the SPIP. The plan is to separate the >>>> transport layer from UDF processing logic in worker.py, allowing gRPC and >>>> socket to share the same execution logic. This enables safe fallback and >>>> reduces long-term dual-maintenance overhead. >>>> >>>> 4) Worker specification >>>> >>>> We do have a more detailed design and can publish it as a supporting >>>> document. The SPIP will clarify the expected structure and required >>>> metadata without going too deep into implementation detail. >>>> >>>> 5) Dependency management >>>> >>>> This will be defined in the worker specification. Each language >>>> implementation defines its dependency requirements, and clusters are >>>> expected to provision environments accordingly (as is already the case for >>>> Python today). >>>> >>>> 6) Unified query planning concerns >>>> >>>> The intent is not to force identical planning behavior across >>>> languages. The worker specification can expose metadata (e.g., pipelining >>>> support, concurrency, memory characteristics, data format constraints), >>>> allowing the planner to remain flexible and language-aware without >>>> hardcoding per-language rules. >>>> >>>> 7) Inter-UDF pipelining >>>> >>>> Pipelining is supported by the protocol design (similar to PySpark). >>>> The init message can declare multiple UDFs and define chaining and input >>>> mappings. Whether a language supports this can be expressed in worker >>>> metadata so planning can respect it. >>>> >>>> Hopefully this addresses the main concerns. I’ll update the SPIP to >>>> reflect these clarifications more explicitly. >>>> >>>> Thanks again for the thoughtful review. >>>> >>>> >>>>
