[
https://issues.apache.org/jira/browse/SPARK-14081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin resolved SPARK-14081.
---------------------------------
Resolution: Fixed
Assignee: Travis Crawford
Fix Version/s: 2.0.0
> DataFrameNaFunctions fill should not convert float fields to double
> -------------------------------------------------------------------
>
> Key: SPARK-14081
> URL: https://issues.apache.org/jira/browse/SPARK-14081
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.1
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Fix For: 2.0.0
>
>
> [DataFrameNaFunctions|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala]
> provides useful function for dealing with null values in a DataFrame.
> Currently it changes FloatType columns to DoubleType when zero filling. Spark
> should preserve the column data type.
> In the following example, notice how `zeroFilledDF` has its `floatField`
> converted from float to double.
> {code}
> scala> :paste
> // Entering paste mode (ctrl-D to finish)
> import org.apache.spark.sql._
> import org.apache.spark.sql.types._
> val schema = StructType(Seq(
> StructField("intField", IntegerType),
> StructField("longField", LongType),
> StructField("floatField", FloatType),
> StructField("doubleField", DoubleType)))
> val rdd = sc.parallelize(Seq(Row(1,1L,1f,1d), Row(null,null,null,null)))
> val df = sqlContext.createDataFrame(rdd, schema)
> val zeroFilledDF = df.na.fill(0)
> // Exiting paste mode, now interpreting.
> import org.apache.spark.sql._
> import org.apache.spark.sql.types._
> schema: org.apache.spark.sql.types.StructType =
> StructType(StructField(intField,IntegerType,true),
> StructField(longField,LongType,true), StructField(floatField,FloatType,true),
> StructField(doubleField,DoubleType,true))
> rdd: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] =
> ParallelCollectionRDD[2] at parallelize at <console>:48
> df: org.apache.spark.sql.DataFrame = [intField: int, longField: bigint,
> floatField: float, doubleField: double]
> zeroFilledDF: org.apache.spark.sql.DataFrame = [intField: int, longField:
> bigint, floatField: double, doubleField: double]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
