Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21525#discussion_r215461652
--- Diff: mllib/src/main/scala/org/apache/spark/ml/Transformer.scala ---
@@ -116,10 +116,17 @@ abstract class UnaryTransformer[IN, OUT, T <:
UnaryTransformer[IN, OUT, T]]
StructType(outputFields)
}
+ /**
+ * Returns [[Metadata]] to be attached to the output column.
+ */
+ protected def outputMetadata(outputSchema: StructType, dataset:
Dataset[_]): Metadata =
+ Metadata.empty
+
override def transform(dataset: Dataset[_]): DataFrame = {
- transformSchema(dataset.schema, logging = true)
+ val outputSchema = transformSchema(dataset.schema, logging = true)
val transformUDF = udf(this.createTransformFunc, outputDataType)
- dataset.withColumn($(outputCol), transformUDF(dataset($(inputCol))))
+ val metadata = outputMetadata(outputSchema, dataset)
--- End diff --
`HashingTF` is an example that the metadata is created in `transformSchema`
and attached to `outputSchema`. So my question is, do we need an extra API
`outputMetadata` to do this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
