[jira] [Commented] (SPARK-40281) Memory Profiler on Executors
[ https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17631214#comment-17631214 ] Apache Spark commented on SPARK-40281: -- User 'xinrong-meng' has created a pull request for this issue: https://github.com/apache/spark/pull/38584 > Memory Profiler on Executors > > > Key: SPARK-40281 > URL: https://issues.apache.org/jira/browse/SPARK-40281 > Project: Spark > Issue Type: New Feature > Components: PySpark >Affects Versions: 3.4.0 >Reporter: Xinrong Meng >Priority: Major > > Profiling is critical to performance engineering. Memory consumption is a key > indicator of how efficient a PySpark program is. There is an existing effort > on memory profiling of Python programs, Memory Profiler > ([https://pypi.org/project/memory-profiler/).|https://pypi.org/project/memory-profiler/] > PySpark applications run as independent sets of processes on a cluster, > coordinated by the SparkContext object in the driver program. On the driver > side, PySpark is a regular Python process, thus, we can profile it as a > normal Python program using Memory Profiler. > However, on the executors side, we are missing such memory profiler. Since > executors are distributed on different nodes in the cluster, we need to > aggregate profiles. Furthermore, Python worker processes are spawned per > executor for the Python/Pandas UDF execution, which makes the memory > profiling more intricate. > The ticket proposes to implement a Memory Profiler on Executors. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40281) Memory Profiler on Executors
[ https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17631213#comment-17631213 ] Apache Spark commented on SPARK-40281: -- User 'xinrong-meng' has created a pull request for this issue: https://github.com/apache/spark/pull/38584 > Memory Profiler on Executors > > > Key: SPARK-40281 > URL: https://issues.apache.org/jira/browse/SPARK-40281 > Project: Spark > Issue Type: New Feature > Components: PySpark >Affects Versions: 3.4.0 >Reporter: Xinrong Meng >Priority: Major > > Profiling is critical to performance engineering. Memory consumption is a key > indicator of how efficient a PySpark program is. There is an existing effort > on memory profiling of Python programs, Memory Profiler > ([https://pypi.org/project/memory-profiler/).|https://pypi.org/project/memory-profiler/] > PySpark applications run as independent sets of processes on a cluster, > coordinated by the SparkContext object in the driver program. On the driver > side, PySpark is a regular Python process, thus, we can profile it as a > normal Python program using Memory Profiler. > However, on the executors side, we are missing such memory profiler. Since > executors are distributed on different nodes in the cluster, we need to > aggregate profiles. Furthermore, Python worker processes are spawned per > executor for the Python/Pandas UDF execution, which makes the memory > profiling more intricate. > The ticket proposes to implement a Memory Profiler on Executors. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40281) Memory Profiler on Executors
[ https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630120#comment-17630120 ] Xinrong Meng commented on SPARK-40281: -- Thanks [~alfiewdavidson] for the feedback! I am currently working on that and will keep you updated. :) > Memory Profiler on Executors > > > Key: SPARK-40281 > URL: https://issues.apache.org/jira/browse/SPARK-40281 > Project: Spark > Issue Type: New Feature > Components: PySpark >Affects Versions: 3.4.0 >Reporter: Xinrong Meng >Priority: Major > > Profiling is critical to performance engineering. Memory consumption is a key > indicator of how efficient a PySpark program is. There is an existing effort > on memory profiling of Python programs, Memory Profiler > ([https://pypi.org/project/memory-profiler/).|https://pypi.org/project/memory-profiler/] > PySpark applications run as independent sets of processes on a cluster, > coordinated by the SparkContext object in the driver program. On the driver > side, PySpark is a regular Python process, thus, we can profile it as a > normal Python program using Memory Profiler. > However, on the executors side, we are missing such memory profiler. Since > executors are distributed on different nodes in the cluster, we need to > aggregate profiles. Furthermore, Python worker processes are spawned per > executor for the Python/Pandas UDF execution, which makes the memory > profiling more intricate. > The ticket proposes to implement a Memory Profiler on Executors. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40281) Memory Profiler on Executors
[ https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630080#comment-17630080 ] Alfred Davidson commented on SPARK-40281: - +1 in general this would be good regardless of access/usage pattern. Also having the option to have this exposed in the UI would be a bonus and will consolidate metrics (I can imagine many doing their own thing). I look forward to seeing what’s next > Memory Profiler on Executors > > > Key: SPARK-40281 > URL: https://issues.apache.org/jira/browse/SPARK-40281 > Project: Spark > Issue Type: New Feature > Components: PySpark >Affects Versions: 3.4.0 >Reporter: Xinrong Meng >Priority: Major > > Profiling is critical to performance engineering. Memory consumption is a key > indicator of how efficient a PySpark program is. There is an existing effort > on memory profiling of Python programs, Memory Profiler > ([https://pypi.org/project/memory-profiler/).|https://pypi.org/project/memory-profiler/] > PySpark applications run as independent sets of processes on a cluster, > coordinated by the SparkContext object in the driver program. On the driver > side, PySpark is a regular Python process, thus, we can profile it as a > normal Python program using Memory Profiler. > However, on the executors side, we are missing such memory profiler. Since > executors are distributed on different nodes in the cluster, we need to > aggregate profiles. Furthermore, Python worker processes are spawned per > executor for the Python/Pandas UDF execution, which makes the memory > profiling more intricate. > The ticket proposes to implement a Memory Profiler on Executors. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
