[
https://issues.apache.org/jira/browse/SPARK-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306032#comment-14306032
]
Sean Owen commented on SPARK-4587:
----------------------------------
True; you could also store N separate PMML models! At least, the question of
how to divvy up one huge model data set is different from how to represent it.
One way or the other you have to keep track of distributed model data. the only
thing I'd really suggest is trying to get the PMML machinery working first, and
seeing how far that can be leveraged before building new mechanisms.
Importing what you export seems like a good, bright line to draw. The big
question will be what to do with features or settings you don't support: if the
model says use a certain distance metric that isn't supported in k-means, do
you ignore it or fail?
> Model export/import
> -------------------
>
> Key: SPARK-4587
> URL: https://issues.apache.org/jira/browse/SPARK-4587
> Project: Spark
> Issue Type: New Feature
> Components: ML, MLlib
> Reporter: Xiangrui Meng
> Assignee: Joseph K. Bradley
> Priority: Critical
>
> This is an umbrella JIRA for one of the most requested features on the user
> mailing list. Model export/import can be done via Java serialization. But it
> doesn't work for models stored distributively, e.g., ALS and LDA. Ideally, we
> should provide save/load methods to every model. PMML is an option but it has
> its limitations. There are couple things we need to discuss: 1) data format,
> 2) how to preserve partitioning, 3) data compatibility between versions and
> language APIs, etc.
> UPDATE: [Design doc for model import/export |
> https://docs.google.com/document/d/1kABFz1ssKJxLGMkboreSl3-I2CdLAOjNh5IQCrnDN3g/edit?usp=sharing]
> This document sketches machine learning model import/export plans, including
> goals, an API, and development plans.
> The design doc proposes:
> * Support our own Spark-specific format.
> ** This is needed to (a) support distributed models and (b) get model
> import/export support into Spark quickly (while avoiding new dependencies).
> * Also support PMML
> ** This is needed since it is the only thing approaching an industry standard.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
