Yesheng Ma created SPARK-53623:
----------------------------------
Summary: O(N^2) operation in readLargeTableProp in
CatalogColumnStat
Key: SPARK-53623
URL: https://issues.apache.org/jira/browse/SPARK-53623
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.0.1
Reporter: Yesheng Ma
The CatalogColumnStat.readLargeTableProp
[https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala#L766-L784]
is an O(N) operation. Considering a table can have a lot of table properties,
this effectively becomes an O(N^2) operation, which can be very slow for tables
with a lot of table properties.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
