[ https://issues.apache.org/jira/browse/SPARK-14351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Manoj Kumar updated SPARK-14351: -------------------------------- Comment: was deleted (was: Here are my thoughts: Also ccing [~sethah] since he has seen this part of the codebase quite a few times to get some ideas. Right now, a copy of size *statsSize* is created that is sliced from *allStats* for every instantiation of *ImpurityCalculator*. The reason to do this is because there are calls to *add* and *subtract* that modify the *ImpurityCalculator* inplace. However the calls to *add* or *subtract* are very less as compared to the class instantiations. (roughly called one time for every 2 times *ImpurityCalculator* is instantiated. I see two alternatives. 1. Pass the view directly to the *ImpurityCalculator* and make a copy whenever *add* or *subtract* is called. 2. Pass *allStats*, *offset*, *offset + statsSize* to the *impurityCalculator* and make a copy of *allStats* whenever *add* or *subtract* is called. Both will involve making *stats* a def, which would provide a copy whenever it is being called. The first one is more more favourable because the size of *allStats* is huge. WDYT?) > Optimize ImpurityAggregator for decision trees > ---------------------------------------------- > > Key: SPARK-14351 > URL: https://issues.apache.org/jira/browse/SPARK-14351 > Project: Spark > Issue Type: Improvement > Components: ML > Reporter: Joseph K. Bradley > > {{RandomForest.binsToBestSplit}} currently takes a large amount of time. > Based on some quick profiling, I believe a big chunk of this is spent in > {{ImpurityAggregator.getCalculator}} (which seems to make unnecessary Array > copies) and {{RandomForest.calculateImpurityStats}}. > This JIRA is for: > * Doing more profiling to confirm that unnecessary time is being spent in > some of these methods. > * Optimizing the implementation > * Profiling again to confirm the speedups > Local profiling for large enough examples should suffice, especially since > the optimizations should not need to change the amount of data communicated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org