[
https://issues.apache.org/jira/browse/SPARK-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367041#comment-15367041
]
Apache Spark commented on SPARK-16431:
--------------------------------------
User 'husseinhazimeh' has created a pull request for this issue:
https://github.com/apache/spark/pull/14101
> Add a unified method that accepts single instances to feature transformers
> and predictors
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-16431
> URL: https://issues.apache.org/jira/browse/SPARK-16431
> Project: Spark
> Issue Type: Improvement
> Components: ML
> Reporter: Hussein Hazimeh
> Priority: Minor
>
> Current transformers in spark.ml can only operate on DataFrames and don't
> have a method that accepts single instances. A typical transformer has a
> User-Defined Function (udf) in its *transform* method which includes a set of
> operations on the features of a single instance:
> {code}val column_operation = udf {operations on single instance}{code}
> Adding a new method that operates directly on single instances (e.g. called
> *transformInstance*) and using it in the udf instead can be useful:
> {code}def transformInstance(features: featureType): OutputType = {operations
> on single instance}
> val column_operation = udf {transformInstance}{code}
> Predictors also don’t have a public method that does predictions on single
> instances. *transformInstance* can be easily added to predictors by acting as
> a wrapper for the internal method predict (which takes features as input).
> This simple change has (at least) three benefits.
> # Providing a low-latency transformation/prediction method to support machine
> learning applications that require real-time predictions. The current
> *transform* method has a relatively high latency when transforming single
> instances or small batches due to the overhead introduced by DataFrame
> operations. I measured the latency required to classify a single instance in
> the 20 Newsgroups dataset using the current *transform* method and the
> proposed *transformInstance*. The ML pipeline contains a tokenizer, stopword
> remover, TF hasher, IDF, scaler, and Logisitc Regression. The table below
> shows the latency percentiles in milliseconds after measuring the time to
> classify 700 documents.
> ||Transformation Method||P50||P90||P99||Max||
> |*transform*|31.44|39.43|67.75|126.97|
> |*transformInstance*|0.16|0.38|1.16|3.2|
> *transformInstance* is 200 times faster on average and can classify a
> document in less than a millisecond. By profiling the code of *transform*,
> it turns out that every transformer in the pipeline wastes 5 milliseconds on
> average in DataFrame-related operations when transforming a single instance.
> This implies that the latency increases linearly with the pipeline size which
> can be problematic.
> # Increasing code readability and allowing easier debugging as operations on
> rows are now combined into a function that can be tested independently of the
> higher-level *transform* method.
> # Adding flexibility to create new models: for example, check this
> [comment|https://github.com/apache/spark/pull/8883#issuecomment-215559305] on
> supporting new ensemble methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
