[ 
https://issues.apache.org/jira/browse/SPARK-60131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved SPARK-60131.
----------------------------------
    Fix Version/s: 4.4.0
       Resolution: Fixed

Issue resolved by pull request 60304
[https://github.com/apache/spark/pull/60304]

> Removing an unregistered Scala Connect streaming listener stops the shared bus
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-60131
>                 URL: https://issues.apache.org/jira/browse/SPARK-60131
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect, Structured Streaming
>    Affects Versions: 4.0.0, 5.0.0
>            Reporter: Vitthal
>            Assignee: Vitthal
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> Problem
> In the Scala Spark Connect client, removing a streaming listener that was 
> never registered can stop event delivery for another registered listener. The 
> removal code checks whether one listener remains, but does not check whether 
> it is removing that listener before stopping the shared bus.
> How to reproduce
> 1. Register listener A with spark.streams.addListener(A).
> 2. Call spark.streams.removeListener(B), where B is a different listener that 
> was never registered.
> 3. The client sends a command to remove the shared server-side listener bus 
> and interrupts its event worker.
> Expected behavior
> Removing B should do nothing. Listener A should remain registered and 
> continue receiving events. This also matches the membership check already 
> used by Python Connect.
> Why this fix is needed
> Removing an unrelated listener should not disable event handling for a 
> registered listener. The server cannot prevent this mistake because the 
> removal command addresses the shared bus, rather than an individual client 
> listener.
> Proposed change and testing
> Check that the listener is registered before stopping the shared bus. Keep 
> the existing behavior for removing a registered listener, including the last 
> listener.
> A focused test against the production removal method fails before the fix 
> because it records an unwanted bus removal command. After the fix, that test, 
> the registered-listener removal test, and SparkSessionSuite pass: 12 tests in 
> total. Module style checks also pass. The existing live-server streaming 
> listener test also fails before the fix because the termination event is not 
> delivered, and passes after the fix. It checks progress and termination 
> events for both listener interface variants. No tests were ignored in either 
> live run.
> Related issues
> SPARK-59847 and SPARK-59623 concern Python Connect deadlocks when removing 
> the last registered listener. This issue concerns removing an unregistered 
> listener in the Scala client, so the trigger and fix are different.
> SPARK-58000 concerns Scala listener-test flakiness caused by interrupted 
> worker cleanup. It is related lifecycle context, but does not address this 
> missing membership check.
> Affected versions
> The same removal path is present in the Spark 4.0.0 source. The regression 
> was run on the current 5.0.0-SNAPSHOT master checkout; no released-version 
> runtime test was run.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to