eaton created SPARK-26567: ----------------------------- Summary: Should we align CSV query results with hive text query results: an int field, if the input value is 1.0, hive text query results is 1, CSV query results is null Key: SPARK-26567 URL: https://issues.apache.org/jira/browse/SPARK-26567 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 2.4.0 Reporter: eaton
If we want to be consistent, we can modify the makeConverter function in UnivocityParser, but the performance may get worse.The modified code is as follows: {code:java} // code placeholder {code} def makeConverter( name: String, dataType: DataType, nullable: Boolean = true, options: CSVOptions): ValueConverter = dataType match { case _: ByteType => (d: String) => nullSafeDatum(d, name, nullable, options)(_.toDouble.intValue().toByte) case _: ShortType => (d: String) => nullSafeDatum(d, name, nullable, options)(_.toDouble.intValue().toShort) case _: IntegerType => (d: String) => nullSafeDatum(d, name, nullable, options)(_.toDouble.intValue()) case _: LongType => (d: String) => nullSafeDatum(d, name, nullable, options)(_.toDouble.intValue().toLong) -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org