[
https://issues.apache.org/jira/browse/YUNIKORN-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dale Richardson updated YUNIKORN-3364:
--------------------------------------
Description:
Follow-up to YUNIKORN-3357 (PR #1124); burns down the leakcheck exemption
events.(*EventStreaming).CreateEventStream.func1.
{{EventStreaming.CreateEventStream}} forwards events with bare blocking sends
({{{}consumer <- event{}}} in the history-replay loop and in the select body,
{{event_streaming.go}} ~lines 109 and 127). When a consumer stops reading (a
disconnected or stalled REST event-stream client), the forwarder blocks on the
send *outside* its select and can no longer observe {{stop}} or
{{{}e.stopCh{}}}. The slow-consumer eviction path then closes both channels,
but closing a channel cannot unblock a sender: the goroutine, its 1000-event
consumer buffer and its {{seen}} map are pinned for the life of the process —
exactly the case the eviction logic exists to handle.
Proposed fix: make both sends selectable:
{code:java}
select {
case consumer <- event:
case <-stop:
close(consumer)
return
case <-e.stopCh:
close(consumer)
return
}
{code}
A second, test-only cause shares this top frame (a test that creates a stream
and never calls {{{}RemoveStream{}}}); fix that alongside, then delete the
exemption in {{{}pkg/common/leakcheck/leakcheck.go{}}}.
was:
Follow-up to YUNIKORN-3357 (PR #1124); burns down the leakcheck exemption
events.(*EventStreaming).CreateEventStream.func1.
{{EventStreaming.CreateEventStream}} forwards events with bare blocking sends
({{consumer <- event}} in the history-replay loop and in the select body,
{{event_streaming.go}} ~lines 109 and 127). When a consumer stops reading (a
disconnected or stalled REST event-stream client), the forwarder blocks on the
send *outside* its select and can no longer observe {{stop}} or {{e.stopCh}}.
The slow-consumer eviction path then closes both channels, but closing a
channel cannot unblock a sender: the goroutine, its 1000-event consumer buffer
and its {{seen}} map are pinned for the life of the process — exactly the case
the eviction logic exists to handle.
Proposed fix: make both sends selectable:
{code}
select {
case consumer <- event:
case <-stop:
close(consumer)
return
case <-e.stopCh:
close(consumer)
return
}
{code}
A second, test-only cause shares this top frame (a test that creates a stream
and never calls {{RemoveStream}}); fix that alongside, then delete the
exemption in {{pkg/common/leakcheck/leakcheck.go}}.
> Event stream forwarder leaks permanently when a consumer stalls;
> slow-consumer eviction cannot release it
> ---------------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3364
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3364
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Dale Richardson
> Priority: Major
> Labels: newbie
>
> Follow-up to YUNIKORN-3357 (PR #1124); burns down the leakcheck exemption
> events.(*EventStreaming).CreateEventStream.func1.
> {{EventStreaming.CreateEventStream}} forwards events with bare blocking sends
> ({{{}consumer <- event{}}} in the history-replay loop and in the select body,
> {{event_streaming.go}} ~lines 109 and 127). When a consumer stops reading (a
> disconnected or stalled REST event-stream client), the forwarder blocks on
> the send *outside* its select and can no longer observe {{stop}} or
> {{{}e.stopCh{}}}. The slow-consumer eviction path then closes both channels,
> but closing a channel cannot unblock a sender: the goroutine, its 1000-event
> consumer buffer and its {{seen}} map are pinned for the life of the process —
> exactly the case the eviction logic exists to handle.
> Proposed fix: make both sends selectable:
> {code:java}
> select {
> case consumer <- event:
> case <-stop:
> close(consumer)
> return
> case <-e.stopCh:
> close(consumer)
> return
> }
> {code}
> A second, test-only cause shares this top frame (a test that creates a stream
> and never calls {{{}RemoveStream{}}}); fix that alongside, then delete the
> exemption in {{{}pkg/common/leakcheck/leakcheck.go{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
