Hi Yin, posted here because I think it's a bug.
So, it will return null and I can get a nullpointerexception, as I was
getting. Is this really the expected behavior? Never seen something
returning null in other Scala tools that I used.
Regards,
2015-09-14 18:54 GMT-03:00 Yin Huai <yh...@databricks.com>:
> btw, move it to user list.
>
> On Mon, Sep 14, 2015 at 2:54 PM, Yin Huai <yh...@databricks.com> wrote:
>
>> A scale of 10 means that there are 10 digits at the right of the decimal
>> point. If you also have precision 10, the range of your data will be [0, 1)
>> and casting "10.5" to DecimalType(10, 10) will return null, which is
>> expected.
>>
>> On Mon, Sep 14, 2015 at 1:42 PM, Dirceu Semighini Filho <
>> dirceu.semigh...@gmail.com> wrote:
>>
>>> Hi all,
>>> I'm moving from spark 1.4 to 1.5, and one of my tests is failing.
>>> It seems that there was some changes in org.apache.spark.sql.types.
>>> DecimalType
>>>
>>> This ugly code is a little sample to reproduce the error, don't use it
>>> into your project.
>>>
>>> test("spark test") {
>>> val file =
>>> context.sparkContext().textFile(s"${defaultFilePath}Test.csv").map(f =>
>>> Row.fromSeq({
>>> val values = f.split(",")
>>>
>>> Seq(values.head.toString.toInt,values.tail.head.toString.toInt,BigDecimal(values.tail.tail.head),
>>> values.tail.tail.tail.head)}))
>>>
>>> val structType = StructType(Seq(StructField("id", IntegerType, false),
>>> StructField("int2", IntegerType, false), StructField("double",
>>>
>>> DecimalType(10,10), false),
>>>
>>>
>>> StructField("str2", StringType, false)))
>>>
>>> val df = context.sqlContext.createDataFrame(file,structType)
>>> df.first
>>> }
>>>
>>> The content of the file is:
>>>
>>> 1,5,10.5,va
>>> 2,1,0.1,vb
>>> 3,8,10.0,vc
>>>
>>> The problem resides in DecimalType, before 1.5 the scala wasn't
>>> required. Now when using DecimalType(12,10) it works fine, but using
>>> DecimalType(10,10) the Decimal values
>>> 10.5 became null, and the 0.1 works.
>>>
>>> Is there anybody working with DecimalType for 1.5.1?
>>>
>>> Regards,
>>> Dirceu
>>>
>>>
>>
>
