[
https://issues.apache.org/jira/browse/SPARK-28562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-28562:
---------------------------------
Priority: Minor (was: Critical)
> PySpark profiling is not understandable
> ---------------------------------------
>
> Key: SPARK-28562
> URL: https://issues.apache.org/jira/browse/SPARK-28562
> Project: Spark
> Issue Type: Question
> Components: Optimizer
> Affects Versions: 2.4.0
> Reporter: Albertus Kelvin
> Priority: Minor
>
> I was profiling code in PySpark. What I did was set the
> "spark.python.profile" in the config to "true". I also made a simple method
> consisting of several dataframe operations, such as "withColumn" and "join".
> Here's the code sample:
> {code:python}
> def join_df(df, df1):
> df = df.withColumn('rowa', F.lit(100))
> df = df.withColumn('rowb', df['rowa'] * F.lit(100))
>
> joined_df = df.join(df1,'rowid',how='left')
> return joined_df
> {code}
> However, after the driver exits, the output of the profiler was not
> understandable because there were no my filename and the corresponding
> methods. All exists was Spark's built-in files and methods, such as "rdd.py",
> "worker.py", and "serializers.py".
> The question is, how to show all of my methods that become the bottlenecks?
> For example, using the above code sample, I'd like to know the time needed
> for "withColumn" and "join" operation.
> Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
