[
https://issues.apache.org/jira/browse/SPARK-12098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15085075#comment-15085075
]
Xusen Yin commented on SPARK-12098:
-----------------------------------
[~josephkb] I understand that. I'll try to publish it as a Spark package first.
> Cross validator with multi-arm bandit search
> --------------------------------------------
>
> Key: SPARK-12098
> URL: https://issues.apache.org/jira/browse/SPARK-12098
> Project: Spark
> Issue Type: New Feature
> Components: ML, MLlib
> Reporter: Xusen Yin
>
> The classic cross-validation requires all inner classifiers iterate to a
> fixed number of iterations, or until convergence states. It is costly
> especially in the massive data scenario. According to the paper
> Non-stochastic Best Arm Identification and Hyperparameter Optimization
> (http://arxiv.org/pdf/1502.07943v1.pdf), we can see a promising way to reduce
> the amount of total iterations of cross-validation with multi-armed bandit
> search.
> The multi-armed bandit search for cross-validation (bandit search for short)
> requires warm-start of ml algorithms, and fine-grained control of the inner
> behavior of the corss validator.
> Since there are bunch of algorithms of bandit search to find the best
> parameter set, we intent to provide only a few of them in the beginning to
> reduce the test/perf-test work and make it more stable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
