mkincaid commented on PR #56584: URL: https://github.com/apache/spark/pull/56584#issuecomment-5019108163
> Hi @mkincaid , instead of modify the `SizeEstimator` which may affect all callsites, I think overriding the `estimatedSize` with allowed fields list for each ml implimentation should be more safe and precise. E.g. in the linearSVC model, we only take the paramap and coefficient vector into account. Thanks @zhengruifeng. What do you think about a combination of the above as first choice + also set `estimatedSize = 0` on `SparkSession` and `SparkContext`? I think that’s a good backstop for cases where we don’t have a good model-specific override to make sure we keep the bloated session/context out. When I tested there seemed to be something like 10-12 affected models. I can patch `StringIndexer`, but for the time being various other models will have the same issue, so I think a general fix is also helpful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
