Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/19560
@gatorsmile @dongjoon-hyun
Thanks a lot for looking into this.
This pr aims to avoid OOM if metastore fails to update table properties
after the data is already produced. With the config in this pr enabled, we
check the size on filesystem only when `totalSize` is below
`spark.sql.autoBroadcastJoinThreshold`, so I think the cost can be acceptable.
Yes, the storage can be other filesystems. I refined the name. Please take
a look again when you have time.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
