Looks like what you observed is due to the following code in Decimal.scala :
def set(decimal: BigDecimal, precision: Int, scale: Int): Decimal = {
this.decimalVal = decimal.setScale(scale, ROUND_HALF_UP)
require(
decimalVal.precision <= precision,
s"Decimal precision ${decimalVal.precision} exceeds max precision
$precision")
You can construct BigDecimal, use the following method on the instance:
http://docs.oracle.com/javase/7/docs/api/java/math/BigDecimal.html#setScale(int,%20java.math.RoundingMode)
and pass to set(decimal: BigDecimal)
FYI
On Thu, Dec 3, 2015 at 9:46 AM, Philip Dodds <philip.do...@gmail.com> wrote:
> I'm not sure if there is a way around this just looking for advice,
>
> I create a dataframe from some decimals with a specific precision and
> scale, then when I look at the dataframe it has defaulted the precision
> and scale back again.
>
> Is there a way to retain the precision and scale when doing a toDF()
>
> example code:
>
> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
> import sqlContext.implicits._
> import org.apache.spark.sql.types.Decimal
>
> val a = new Decimal().set(BigDecimal(50),14,4)
> val b = new Decimal().set(BigDecimal(50),14,4)
>
> val data = Seq.fill(5) {
> (a,b)
> }
>
> val trades = data.toDF()
>
> trades.printSchema()
>
> the result of this code would show
>
> root
> |-- _1: decimal(38,18) (nullable = true)
> |-- _2: decimal(38,18) (nullable = true)
>
> sqlContext: org.apache.spark.sql.SQLContext =
> org.apache.spark.sql.SQLContext@3a1a48f7
> import sqlContext.implicits._
> import org.apache.spark.sql.types.Decimal
> a: org.apache.spark.sql.types.Decimal = 50.000000000000000000
> b: org.apache.spark.sql.types.Decimal = 50.000000000000000000
> data: Seq[(org.apache.spark.sql.types.Decimal,
> org.apache.spark.sql.types.Decimal)] =
> List((50.000000000000000000,50.000000000000000000),
> (50.000000000000000000,50.000000000000000000),
> (50.000000000000000000,50.000000000000000000),
> (50.000000000000000000,50.000000000000000000),
> (50.000000000000000000,50.000000000000000000))
> trades: org.apache.spark.sql.DataFrame = [_1: decimal(38,18), _2:
> decimal(38,18)]
>
>
> Any advice would be brilliant
>
>
> Thanks
>
> P
>
>
> --
> Philip Dodds
>
> philip.do...@gmail.com
> @philipdodds
>
>
