[jira] [Commented] (SPARK-21972) Allow users to control input data persistence in ML Estimators via a handlePersistence ml.Param
[
https://issues.apache.org/jira/browse/SPARK-21972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609296#comment-16609296
]
Wenchen Fan commented on SPARK-21972:
-
I'm removing the target version, since we are not going to merge it to 2.4
> Allow users to control input data persistence in ML Estimators via a
> handlePersistence ml.Param
> ---
>
> Key: SPARK-21972
> URL: https://issues.apache.org/jira/browse/SPARK-21972
> Project: Spark
> Issue Type: Improvement
> Components: ML, MLlib
>Affects Versions: 2.2.0
>Reporter: Siddharth Murching
>Priority: Major
>
> Several Spark ML algorithms (LogisticRegression, LinearRegression, KMeans,
> etc) call {{cache()}} on uncached input datasets to improve performance.
> Unfortunately, these algorithms a) check input persistence inaccurately
> ([SPARK-18608|https://issues.apache.org/jira/browse/SPARK-18608]) and b)
> check the persistence level of the input dataset but not any of its parents.
> These issues can result in unwanted double-caching of input data & degraded
> performance (see
> [SPARK-21799|https://issues.apache.org/jira/browse/SPARK-21799]).
> This ticket proposes adding a boolean {{handlePersistence}} param
> (org.apache.spark.ml.param) so that users can specify whether an ML algorithm
> should try to cache un-cached input data. {{handlePersistence}} will be
> {{true}} by default, corresponding to existing behavior (always persisting
> uncached input), but users can achieve finer-grained control over input
> persistence by setting {{handlePersistence}} to {{false}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21972) Allow users to control input data persistence in ML Estimators via a handlePersistence ml.Param
[
https://issues.apache.org/jira/browse/SPARK-21972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180129#comment-16180129
]
zhengruifeng commented on SPARK-21972:
--
[~josephkb] I added {{preprocess}} & {{postprocess}} for alg-specific
processing, however only {{GeneralizedLinearRegression}} has extra column
{{offsetCol}} to deal with, so I will merge them into {{fit()}}. I will update
the PR ASAP.
> Allow users to control input data persistence in ML Estimators via a
> handlePersistence ml.Param
> ---
>
> Key: SPARK-21972
> URL: https://issues.apache.org/jira/browse/SPARK-21972
> Project: Spark
> Issue Type: Improvement
> Components: ML, MLlib
>Affects Versions: 2.2.0
>Reporter: Siddharth Murching
>
> Several Spark ML algorithms (LogisticRegression, LinearRegression, KMeans,
> etc) call {{cache()}} on uncached input datasets to improve performance.
> Unfortunately, these algorithms a) check input persistence inaccurately
> ([SPARK-18608|https://issues.apache.org/jira/browse/SPARK-18608]) and b)
> check the persistence level of the input dataset but not any of its parents.
> These issues can result in unwanted double-caching of input data & degraded
> performance (see
> [SPARK-21799|https://issues.apache.org/jira/browse/SPARK-21799]).
> This ticket proposes adding a boolean {{handlePersistence}} param
> (org.apache.spark.ml.param) so that users can specify whether an ML algorithm
> should try to cache un-cached input data. {{handlePersistence}} will be
> {{true}} by default, corresponding to existing behavior (always persisting
> uncached input), but users can achieve finer-grained control over input
> persistence by setting {{handlePersistence}} to {{false}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21972) Allow users to control input data persistence in ML Estimators via a handlePersistence ml.Param
[
https://issues.apache.org/jira/browse/SPARK-21972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179436#comment-16179436
]
Joseph K. Bradley commented on SPARK-21972:
---
Definitely, as @zhengruifeng said, let's keep the correction to the current
caching so that 2.2.1 will fix the regression from Spark 2.1 -> 2.2.0.
@zhengruifeng Why do we need to have preprocess/train/postprocess for
Predictor? Can't that be done in fit() anyways?
Also, let's not worry about little optimizations such as not caching for
maxIter = 0 or 1; that sounds like unnecessary optimization since no one should
really use maxIter = 0 or 1 anyways.
> Allow users to control input data persistence in ML Estimators via a
> handlePersistence ml.Param
> ---
>
> Key: SPARK-21972
> URL: https://issues.apache.org/jira/browse/SPARK-21972
> Project: Spark
> Issue Type: Improvement
> Components: ML, MLlib
>Affects Versions: 2.2.0
>Reporter: Siddharth Murching
>
> Several Spark ML algorithms (LogisticRegression, LinearRegression, KMeans,
> etc) call {{cache()}} on uncached input datasets to improve performance.
> Unfortunately, these algorithms a) check input persistence inaccurately
> ([SPARK-18608|https://issues.apache.org/jira/browse/SPARK-18608]) and b)
> check the persistence level of the input dataset but not any of its parents.
> These issues can result in unwanted double-caching of input data & degraded
> performance (see
> [SPARK-21799|https://issues.apache.org/jira/browse/SPARK-21799]).
> This ticket proposes adding a boolean {{handlePersistence}} param
> (org.apache.spark.ml.param) so that users can specify whether an ML algorithm
> should try to cache un-cached input data. {{handlePersistence}} will be
> {{true}} by default, corresponding to existing behavior (always persisting
> uncached input), but users can achieve finer-grained control over input
> persistence by setting {{handlePersistence}} to {{false}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21972) Allow users to control input data persistence in ML Estimators via a handlePersistence ml.Param
[
https://issues.apache.org/jira/browse/SPARK-21972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178572#comment-16178572
]
zhengruifeng commented on SPARK-21972:
--
[~WeichenXu123] you solution is reasonable. however I think we then need to set
handlePersistence to true by default to avoid perfermance regression (like
SPARK-21799), this is what [~josephkb] suggested not doing.
> Allow users to control input data persistence in ML Estimators via a
> handlePersistence ml.Param
> ---
>
> Key: SPARK-21972
> URL: https://issues.apache.org/jira/browse/SPARK-21972
> Project: Spark
> Issue Type: Improvement
> Components: ML, MLlib
>Affects Versions: 2.2.0
>Reporter: Siddharth Murching
>
> Several Spark ML algorithms (LogisticRegression, LinearRegression, KMeans,
> etc) call {{cache()}} on uncached input datasets to improve performance.
> Unfortunately, these algorithms a) check input persistence inaccurately
> ([SPARK-18608|https://issues.apache.org/jira/browse/SPARK-18608]) and b)
> check the persistence level of the input dataset but not any of its parents.
> These issues can result in unwanted double-caching of input data & degraded
> performance (see
> [SPARK-21799|https://issues.apache.org/jira/browse/SPARK-21799]).
> This ticket proposes adding a boolean {{handlePersistence}} param
> (org.apache.spark.ml.param) so that users can specify whether an ML algorithm
> should try to cache un-cached input data. {{handlePersistence}} will be
> {{true}} by default, corresponding to existing behavior (always persisting
> uncached input), but users can achieve finer-grained control over input
> persistence by setting {{handlePersistence}} to {{false}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21972) Allow users to control input data persistence in ML Estimators via a handlePersistence ml.Param
[
https://issues.apache.org/jira/browse/SPARK-21972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171773#comment-16171773
]
Weichen Xu commented on SPARK-21972:
[~podongfeng] [~Siddharth Murching] I have a simpler proposal: remove all
auto-checking caching code in algos such as LiR/LoR, whether cache or not only
depend on the param. (Because currently these checking is totally useless,
remove them don't change anything actually.) So that it can avoid big change
in the interface for now.
> Allow users to control input data persistence in ML Estimators via a
> handlePersistence ml.Param
> ---
>
> Key: SPARK-21972
> URL: https://issues.apache.org/jira/browse/SPARK-21972
> Project: Spark
> Issue Type: Improvement
> Components: ML, MLlib
>Affects Versions: 2.2.0
>Reporter: Siddharth Murching
>
> Several Spark ML algorithms (LogisticRegression, LinearRegression, KMeans,
> etc) call {{cache()}} on uncached input datasets to improve performance.
> Unfortunately, these algorithms a) check input persistence inaccurately
> ([SPARK-18608|https://issues.apache.org/jira/browse/SPARK-18608]) and b)
> check the persistence level of the input dataset but not any of its parents.
> These issues can result in unwanted double-caching of input data & degraded
> performance (see
> [SPARK-21799|https://issues.apache.org/jira/browse/SPARK-21799]).
> This ticket proposes adding a boolean {{handlePersistence}} param
> (org.apache.spark.ml.param) so that users can specify whether an ML algorithm
> should try to cache un-cached input data. {{handlePersistence}} will be
> {{true}} by default, corresponding to existing behavior (always persisting
> uncached input), but users can achieve finer-grained control over input
> persistence by setting {{handlePersistence}} to {{false}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21972) Allow users to control input data persistence in ML Estimators via a handlePersistence ml.Param
[
https://issues.apache.org/jira/browse/SPARK-21972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171342#comment-16171342
]
zhengruifeng commented on SPARK-21972:
--
Since persistence handling is very algorithm-dependent, I perfer this solution:
1, add method {preprocess(dataset: Dataset[_]): DataFrame} in {Predictor}, and
call it before {train()), we do casting, columns selection and data persistence
(if necessary) in it.
2, add method {postprocess}, and call it after {train()), we unpersist the
intermediate dataframe if needed.
3, for specific purpose, we can override it, like in current PR,
{GeneralizedLinearRegression} override {preprocess} to add a column selection
and casting for {offsetCol}.
I personally think that, to handle persistence, we can use more information:
for examples, for algs containing param {maxIter}, if {maxIter} is set 0 or 1,
the input dataset may be unnecsssary to cache.
> Allow users to control input data persistence in ML Estimators via a
> handlePersistence ml.Param
> ---
>
> Key: SPARK-21972
> URL: https://issues.apache.org/jira/browse/SPARK-21972
> Project: Spark
> Issue Type: Improvement
> Components: ML, MLlib
>Affects Versions: 2.2.0
>Reporter: Siddharth Murching
>
> Several Spark ML algorithms (LogisticRegression, LinearRegression, KMeans,
> etc) call {{cache()}} on uncached input datasets to improve performance.
> Unfortunately, these algorithms a) check input persistence inaccurately
> ([SPARK-18608|https://issues.apache.org/jira/browse/SPARK-18608]) and b)
> check the persistence level of the input dataset but not any of its parents.
> These issues can result in unwanted double-caching of input data & degraded
> performance (see
> [SPARK-21799|https://issues.apache.org/jira/browse/SPARK-21799]).
> This ticket proposes adding a boolean {{handlePersistence}} param
> (org.apache.spark.ml.param) so that users can specify whether an ML algorithm
> should try to cache un-cached input data. {{handlePersistence}} will be
> {{true}} by default, corresponding to existing behavior (always persisting
> uncached input), but users can achieve finer-grained control over input
> persistence by setting {{handlePersistence}} to {{false}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21972) Allow users to control input data persistence in ML Estimators via a handlePersistence ml.Param
[
https://issues.apache.org/jira/browse/SPARK-21972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162007#comment-16162007
]
Joseph K. Bradley commented on SPARK-21972:
---
The issue (a) does not really conflict with or affect this JIRA; it can be
fixed separately.
Also, I recommend we *not* set handlePersistence to true by default in the
shared param. Algorithms should set the default individually since the right
behavior is very algorithm-dependent.
> Allow users to control input data persistence in ML Estimators via a
> handlePersistence ml.Param
> ---
>
> Key: SPARK-21972
> URL: https://issues.apache.org/jira/browse/SPARK-21972
> Project: Spark
> Issue Type: Improvement
> Components: ML, MLlib
>Affects Versions: 2.2.0
>Reporter: Siddharth Murching
>
> Several Spark ML algorithms (LogisticRegression, LinearRegression, KMeans,
> etc) call {{cache()}} on uncached input datasets to improve performance.
> Unfortunately, these algorithms a) check input persistence inaccurately
> ([SPARK-18608|https://issues.apache.org/jira/browse/SPARK-18608]) and b)
> check the persistence level of the input dataset but not any of its parents.
> These issues can result in unwanted double-caching of input data & degraded
> performance (see
> [SPARK-21799|https://issues.apache.org/jira/browse/SPARK-21799]).
> This ticket proposes adding a boolean {{handlePersistence}} param
> (org.apache.spark.ml.param) so that users can specify whether an ML algorithm
> should try to cache un-cached input data. {{handlePersistence}} will be
> {{true}} by default, corresponding to existing behavior (always persisting
> uncached input), but users can achieve finer-grained control over input
> persistence by setting {{handlePersistence}} to {{false}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21972) Allow users to control input data persistence in ML Estimators via a handlePersistence ml.Param
[
https://issues.apache.org/jira/browse/SPARK-21972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161634#comment-16161634
]
Siddharth Murching commented on SPARK-21972:
Link to old PR containing work on this:
https://github.com/apache/spark/pull/17014
> Allow users to control input data persistence in ML Estimators via a
> handlePersistence ml.Param
> ---
>
> Key: SPARK-21972
> URL: https://issues.apache.org/jira/browse/SPARK-21972
> Project: Spark
> Issue Type: Improvement
> Components: ML, MLlib
>Affects Versions: 2.2.0
>Reporter: Siddharth Murching
>
> Several Spark ML algorithms (LogisticRegression, LinearRegression, KMeans,
> etc) call {{cache()}} on uncached input datasets to improve performance.
> Unfortunately, these algorithms a) check input persistence inaccurately
> ([SPARK-18608|https://issues.apache.org/jira/browse/SPARK-18608]) and b)
> check the persistence level of the input dataset but not any of its parents.
> These issues can result in unwanted double-caching of input data & degraded
> performance (see
> [SPARK-21799|https://issues.apache.org/jira/browse/SPARK-21799]).
> This ticket proposes adding a boolean {{handlePersistence}} param
> (org.apache.spark.ml.param) so that users can specify whether an ML algorithm
> should try to cache un-cached input data. {{handlePersistence}} will be
> {{true}} by default, corresponding to existing behavior (always persisting
> uncached input), but users can achieve finer-grained control over input
> persistence by setting {{handlePersistence}} to {{false}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21972) Allow users to control input data persistence in ML Estimators via a handlePersistence ml.Param
[
https://issues.apache.org/jira/browse/SPARK-21972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160758#comment-16160758
]
Apache Spark commented on SPARK-21972:
--
User 'zhengruifeng' has created a pull request for this issue:
https://github.com/apache/spark/pull/19186
> Allow users to control input data persistence in ML Estimators via a
> handlePersistence ml.Param
> ---
>
> Key: SPARK-21972
> URL: https://issues.apache.org/jira/browse/SPARK-21972
> Project: Spark
> Issue Type: Improvement
> Components: ML, MLlib
>Affects Versions: 2.2.0
>Reporter: Siddharth Murching
>
> Several Spark ML algorithms (LogisticRegression, LinearRegression, KMeans,
> etc) call {{cache()}} on uncached input datasets to improve performance.
> Unfortunately, these algorithms a) check input persistence inaccurately
> ([SPARK-18608|https://issues.apache.org/jira/browse/SPARK-18608]) and b)
> check the persistence level of the input dataset but not any of its parents.
> These issues can result in unwanted double-caching of input data & degraded
> performance (see
> [SPARK-21799|https://issues.apache.org/jira/browse/SPARK-21799]).
> This ticket proposes adding a boolean {{handlePersistence}} param
> (org.apache.spark.ml.param) so that users can specify whether an ML algorithm
> should try to cache un-cached input data. {{handlePersistence}} will be
> {{true}} by default, corresponding to existing behavior (always persisting
> uncached input), but users can achieve finer-grained control over input
> persistence by setting {{handlePersistence}} to {{false}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21972) Allow users to control input data persistence in ML Estimators via a handlePersistence ml.Param
[
https://issues.apache.org/jira/browse/SPARK-21972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160624#comment-16160624
]
Siddharth Murching commented on SPARK-21972:
Work has already begun on this in this PR:
[https://github.com/apache/spark/pull/17014|https://github.com/apache/spark/pull/17014]
> Allow users to control input data persistence in ML Estimators via a
> handlePersistence ml.Param
> ---
>
> Key: SPARK-21972
> URL: https://issues.apache.org/jira/browse/SPARK-21972
> Project: Spark
> Issue Type: Improvement
> Components: ML, MLlib
>Affects Versions: 2.2.0
>Reporter: Siddharth Murching
>
> Several Spark ML algorithms (LogisticRegression, LinearRegression, KMeans,
> etc) call {{cache()}} on uncached input datasets to improve performance.
> Unfortunately, these algorithms a) check input persistence inaccurately (see
> [SPARK-18608|https://issues.apache.org/jira/browse/SPARK-18608]) and b) check
> the persistence level of the input dataset but not any of its parents. These
> issues can result in unwanted double-caching of input data & degraded
> performance (see
> [SPARK-21799|https://issues.apache.org/jira/browse/SPARK-21799]).
> This ticket proposes adding a boolean {{handlePersistence}} param
> (org.apache.spark.ml.param) so that users can specify whether an ML algorithm
> should try to cache un-cached input data. {{handlePersistence}} will be
> {{true}} by default, corresponding to existing behavior (always persisting
> uncached input), but users can achieve finer-grained control over input
> persistence by setting {{handlePersistence}} to {{false}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
