[
https://issues.apache.org/jira/browse/SPARK-47336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-47336:
-----------------------------------
Labels: pull-request-available (was: )
> Provide to PySpark a functionality to get estimated size of DataFrame in bytes
> ------------------------------------------------------------------------------
>
> Key: SPARK-47336
> URL: https://issues.apache.org/jira/browse/SPARK-47336
> Project: Spark
> Issue Type: New Feature
> Components: Connect, PySpark
> Affects Versions: 4.0.0
> Reporter: Semyon Sinchenko
> Priority: Minor
> Labels: pull-request-available
>
> Something equal to
> sessionState().executePlan(...).optimizedPlan().stats().sizeInBytes() in
> JVM-Spark. It may be done via simple call of `_jsparkSession` in a regular
> PySpark and via a plugin for Spark Connect.
>
> This functionality is useful when one need to check a possibility of
> broadcast join without modifying global broadcast threshold.
>
> The function in PySpark API may looks like:
> `DataFrame.estimate_size_in_bytes() -> float` or
> `DataFrame.estimateSizeInBytes() -> float`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
