[
https://issues.apache.org/jira/browse/YUNIKORN-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dale Richardson updated YUNIKORN-3363:
--------------------------------------
Description:
Follow-up to YUNIKORN-3357 (goleak adoption, PR #1124); burns down leakcheck
exemptions events.(*EventSystemImpl).StartServiceWithPublisher.func1 and
events.(*eventPublisher).start.func1.
{{events.Init()}} registers a configmap callback ({{{}event_system.go{}}}) that
{{EventSystemImpl.Stop()}} never removes via
{{configs.RemoveConfigMapCallback}} — compare {{{}health_checker.go{}}}, which
deregisters correctly. Any configuration reload after {{Stop()}} runs
{{reloadConfig() -> restart()}} and resurrects an event system that nobody
holds a reference to; its handler ({{{}StartServiceWithPublisher.func1{}}}) and
publisher ({{{}eventPublisher.start.func1{}}}) goroutines then run for the life
of the process.
Aggravator: {{eventSystemId}} is {{{}fmt.Sprintf("event-system-%d",
time.Now().Unix()){}}}, so repeated {{Init()}} calls collide on the callback
key within a second and accumulate across seconds.
Proposed fix:
* deregister the callback in {{Stop()}} (or re-register in
{{{}StartServiceWithPublisher{}}})
* use a unique id (uuid) instead of a unix-seconds timestamp
* delete the two events exemptions in {{pkg/common/leakcheck/leakcheck.go}}
update: the mechanism (Stop() doesn't deregister the callback) is real, but
harmless while Stop() only ever coincides with process exit; the resurrection
impact is latent, appears only if core becomes restartable in-process.
was:
Follow-up to YUNIKORN-3357 (goleak adoption, PR #1124); burns down leakcheck
exemptions events.(*EventSystemImpl).StartServiceWithPublisher.func1 and
events.(*eventPublisher).start.func1.
{{events.Init()}} registers a configmap callback ({{{}event_system.go{}}}) that
{{EventSystemImpl.Stop()}} never removes via
{{configs.RemoveConfigMapCallback}} — compare {{{}health_checker.go{}}}, which
deregisters correctly. Any configuration reload after {{Stop()}} runs
{{reloadConfig() -> restart()}} and resurrects an event system that nobody
holds a reference to; its handler ({{{}StartServiceWithPublisher.func1{}}}) and
publisher ({{{}eventPublisher.start.func1{}}}) goroutines then run for the life
of the process.
Aggravator: {{eventSystemId}} is {{{}fmt.Sprintf("event-system-%d",
time.Now().Unix()){}}}, so repeated {{Init()}} calls collide on the callback
key within a second and accumulate across seconds.
Proposed fix:
* deregister the callback in {{Stop()}} (or re-register in
{{{}StartServiceWithPublisher{}}})
* use a unique id (uuid) instead of a unix-seconds timestamp
* delete the two events exemptions in {{pkg/common/leakcheck/leakcheck.go}}
Priority: Minor (was: Major)
> Event system restarts after Stop(): configmap callback never deregistered,
> leaking handler and publisher goroutines
> -------------------------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3363
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3363
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Dale Richardson
> Assignee: weichen lai
> Priority: Minor
> Labels: newbie
>
> Follow-up to YUNIKORN-3357 (goleak adoption, PR #1124); burns down leakcheck
> exemptions events.(*EventSystemImpl).StartServiceWithPublisher.func1 and
> events.(*eventPublisher).start.func1.
> {{events.Init()}} registers a configmap callback ({{{}event_system.go{}}})
> that {{EventSystemImpl.Stop()}} never removes via
> {{configs.RemoveConfigMapCallback}} — compare {{{}health_checker.go{}}},
> which deregisters correctly. Any configuration reload after {{Stop()}} runs
> {{reloadConfig() -> restart()}} and resurrects an event system that nobody
> holds a reference to; its handler ({{{}StartServiceWithPublisher.func1{}}})
> and publisher ({{{}eventPublisher.start.func1{}}}) goroutines then run for
> the life of the process.
> Aggravator: {{eventSystemId}} is {{{}fmt.Sprintf("event-system-%d",
> time.Now().Unix()){}}}, so repeated {{Init()}} calls collide on the callback
> key within a second and accumulate across seconds.
> Proposed fix:
> * deregister the callback in {{Stop()}} (or re-register in
> {{{}StartServiceWithPublisher{}}})
> * use a unique id (uuid) instead of a unix-seconds timestamp
> * delete the two events exemptions in {{pkg/common/leakcheck/leakcheck.go}}
> update: the mechanism (Stop() doesn't deregister the callback) is real, but
> harmless while Stop() only ever coincides with process exit; the resurrection
> impact is latent, appears only if core becomes restartable in-process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
