A scale of 10 means that there are 10 digits at the right of the decimal
point. If you also have precision 10, the range of your data will be [0, 1)
and casting "10.5" to DecimalType(10, 10) will return null, which is
expected.
On Mon, Sep 14, 2015 at 1:42 PM, Dirceu Semighini Filho <
dirceu.semigh...@gmail.com> wrote:
> Hi all,
> I'm moving from spark 1.4 to 1.5, and one of my tests is failing.
> It seems that there was some changes in org.apache.spark.sql.types.
> DecimalType
>
> This ugly code is a little sample to reproduce the error, don't use it
> into your project.
>
> test("spark test") {
> val file =
> context.sparkContext().textFile(s"${defaultFilePath}Test.csv").map(f =>
> Row.fromSeq({
> val values = f.split(",")
>
> Seq(values.head.toString.toInt,values.tail.head.toString.toInt,BigDecimal(values.tail.tail.head),
> values.tail.tail.tail.head)}))
>
> val structType = StructType(Seq(StructField("id", IntegerType, false),
> StructField("int2", IntegerType, false), StructField("double",
>
> DecimalType(10,10), false),
>
>
> StructField("str2", StringType, false)))
>
> val df = context.sqlContext.createDataFrame(file,structType)
> df.first
> }
>
> The content of the file is:
>
> 1,5,10.5,va
> 2,1,0.1,vb
> 3,8,10.0,vc
>
> The problem resides in DecimalType, before 1.5 the scala wasn't required.
> Now when using DecimalType(12,10) it works fine, but using
> DecimalType(10,10) the Decimal values
> 10.5 became null, and the 0.1 works.
>
> Is there anybody working with DecimalType for 1.5.1?
>
> Regards,
> Dirceu
>
>
