[ https://issues.apache.org/jira/browse/SPARK-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen updated SPARK-14091: ------------------------------- Summary: Improve performance of SparkContext.getCallSite() (was: Consider improving performance of SparkContext.getCallSite()) > Improve performance of SparkContext.getCallSite() > ------------------------------------------------- > > Key: SPARK-14091 > URL: https://issues.apache.org/jira/browse/SPARK-14091 > Project: Spark > Issue Type: Bug > Components: Spark Core > Reporter: Rajesh Balamohan > Assignee: Rajesh Balamohan > > Currently SparkContext.getCallSite() makes a call to Utils.getCallSite(). > {noformat} > private[spark] def getCallSite(): CallSite = { > val callSite = Utils.getCallSite() > CallSite( > > Option(getLocalProperty(CallSite.SHORT_FORM)).getOrElse(callSite.shortForm), > > Option(getLocalProperty(CallSite.LONG_FORM)).getOrElse(callSite.longForm) > ) > } > {noformat} > However, in some places utils.withDummyCallSite(sc) is invoked to avoid > expensive threaddumps within getCallSite(). But Utils.getCallSite() is > evaluated earlier causing threaddumps to be computed. This would impact when > lots of RDDs are created (e.g spends close to 3-7 seconds when 1000+ are RDDs > are present, which can have significant impact when entire query runtime is > in the order of 10-20 seconds) > Creating this jira to consider evaluating getCallSite only when needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org