Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16351#discussion_r93230247
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -215,84 +215,133 @@ private[csv] object CSVInferSchema {
}
private[csv] object CSVTypeCast {
+ // A `ValueConverter` is responsible for converting the given value to a
desired type.
+ private type ValueConverter = String => Any
/**
- * Casts given string datum to specified type.
- * Currently we do not support complex types (ArrayType, MapType,
StructType).
+ * Create converters which cast each given string datum to each
specified type in given schema.
+ * Currently, we do not support complex types (`ArrayType`, `MapType`,
`StructType`).
*
- * For string types, this is simply the datum. For other types.
+ * For string types, this is simply the datum.
+ * For other types, this is converted into the value according to the
type.
* For other nullable types, returns null if it is null or equals to the
value specified
* in `nullValue` option.
*
- * @param datum string value
- * @param name field name in schema.
- * @param castType data type to cast `datum` into.
- * @param nullable nullability for the field.
+ * @param schema schema that contains data types to cast the given value
into.
* @param options CSV options.
*/
- def castTo(
+ private[sql] def makeConverters(
--- End diff --
Ops, I can remove this access modifier. Will remove it soon and the one
below too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
