Ruifeng Zheng created SPARK-60807:
-------------------------------------
Summary: Restore offloaded ML model summaries during fetch
Key: SPARK-60807
URL: https://issues.apache.org/jira/browse/SPARK-60807
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Ruifeng Zheng
Fetching a Spark Connect ML training-summary attribute can fail with
CONNECT_ML.MODEL_SUMMARY_LOST after model offloading.
The Python client recovers by sending CreateSummary and then retrying FETCH
once. The in-memory ML cache uses soft values, so an entry can be evicted again
between those RPCs. The retry then reloads a model without its summary.
Relation FETCH has a related gap: it restores the summary and then performs
another cache lookup through AttributeHelper.
A deterministic reproduction is to evict a fitted model before each command
FETCH, or configure the server ML cache with a zero in-memory size limit.
Summary access should succeed while the model remains available on disk.
Restore the summary and evaluate the attribute on the same strongly referenced
model instance within one FETCH request. The command should carry the summary
dataset, as relation FETCH already does, while retaining compatibility with
older clients and servers.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
