Did a little more digging and it appears it was just the way I constructed
the Decimal
It works if you do
val data = Seq.fill(5) {
Trade(Decimal(BigDecimal(5),38,20), Decimal(BigDecimal(5),38,20))
}
On Thu, Dec 3, 2015 at 8:58 AM, Philip Dodds <philip.do...@gmail.com> wrote:
> Opened https://issues.apache.org/jira/browse/SPARK-12128
>
> Thanks
>
> P
>
> On Thu, Dec 3, 2015 at 8:51 AM, Philip Dodds <philip.do...@gmail.com>
> wrote:
>
>> I'll open up a JIRA for it, it appears to work when you use a literal
>> number but not when it is coming from the same dataframe
>>
>> Thanks!
>>
>> P
>>
>> On Thu, Dec 3, 2015 at 1:52 AM, Sahil Sareen <sareen...@gmail.com> wrote:
>>
>>> +1 looks like a bug
>>>
>>> I think referencing trades() twice in multiplication is broken,
>>>
>>> scala> trades.select(trades("quantity")*trades("quantity")).show
>>>
>>> +---------------------+
>>> |(quantity * quantity)|
>>> +---------------------+
>>> | null|
>>> | null|
>>>
>>> scala> sqlContext.sql("select price*price as PP from trades").show
>>>
>>> +----+
>>> | PP|
>>> +----+
>>> |null|
>>> |null|
>>>
>>>
>>> -Sahil
>>>
>>> On Thu, Dec 3, 2015 at 12:02 PM, Akhil Das <ak...@sigmoidanalytics.com>
>>> wrote:
>>>
>>>> Not quiet sure whats happening, but its not an issue with
>>>> multiplication i guess as the following query worked for me:
>>>>
>>>> trades.select(trades("price")*9.5).show
>>>> +-------------+
>>>> |(price * 9.5)|
>>>> +-------------+
>>>> | 199.5|
>>>> | 228.0|
>>>> | 190.0|
>>>> | 199.5|
>>>> | 190.0|
>>>> | 256.5|
>>>> | 218.5|
>>>> | 275.5|
>>>> | 218.5|
>>>> ......
>>>> ......
>>>>
>>>>
>>>> Could it be with the precision? ccing dev list, may be you can open up
>>>> a jira for this as it seems to be a bug.
>>>>
>>>> Thanks
>>>> Best Regards
>>>>
>>>> On Mon, Nov 30, 2015 at 12:41 AM, Philip Dodds <philip.do...@gmail.com>
>>>> wrote:
>>>>
>>>>> I hit a weird issue when I tried to multiply to decimals in a select
>>>>> (either in scala or as SQL), and Im assuming I must be missing the point.
>>>>>
>>>>> The issue is fairly easy to recreate with something like the following:
>>>>>
>>>>>
>>>>> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>>>>> import sqlContext.implicits._
>>>>> import org.apache.spark.sql.types.Decimal
>>>>>
>>>>> case class Trade(quantity: Decimal,price: Decimal)
>>>>>
>>>>> val data = Seq.fill(100) {
>>>>> val price = Decimal(20+scala.util.Random.nextInt(10))
>>>>> val quantity = Decimal(20+scala.util.Random.nextInt(10))
>>>>>
>>>>> Trade(quantity, price)
>>>>> }
>>>>>
>>>>> val trades = sc.parallelize(data).toDF()
>>>>> trades.registerTempTable("trades")
>>>>>
>>>>> trades.select(trades("price")*trades("quantity")).show
>>>>>
>>>>> sqlContext.sql("select
>>>>> price/quantity,price*quantity,price+quantity,price-quantity from
>>>>> trades").show
>>>>>
>>>>> The odd part is if you run it you will see that the addition/division
>>>>> and subtraction works but the multiplication returns a null.
>>>>>
>>>>> Tested on 1.5.1/1.5.2 (Scala 2.10 and 2.11)
>>>>>
>>>>> ie.
>>>>>
>>>>> +------------------+
>>>>>
>>>>> |(price * quantity)|
>>>>>
>>>>> +------------------+
>>>>>
>>>>> | null|
>>>>>
>>>>> | null|
>>>>>
>>>>> | null|
>>>>>
>>>>> | null|
>>>>>
>>>>> | null|
>>>>>
>>>>> +------------------+
>>>>>
>>>>>
>>>>> +--------------------+----+--------------------+--------------------+
>>>>>
>>>>> | _c0| _c1| _c2| _c3|
>>>>>
>>>>> +--------------------+----+--------------------+--------------------+
>>>>>
>>>>> |0.952380952380952381|null|41.00000000000000...|-1.00000000000000...|
>>>>>
>>>>> |1.380952380952380952|null|50.00000000000000...|8.000000000000000000|
>>>>>
>>>>> |1.272727272727272727|null|50.00000000000000...|6.000000000000000000|
>>>>>
>>>>> |0.833333333333333333|null|44.00000000000000...|-4.00000000000000...|
>>>>>
>>>>> |1.000000000000000000|null|58.00000000000000...| 0E-18|
>>>>>
>>>>> +--------------------+----+--------------------+--------------------+
>>>>>
>>>>>
>>>>> Just keen to know what I did wrong?
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>> P
>>>>>
>>>>> --
>>>>> Philip Dodds
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Philip Dodds
>>
>> philip.do...@gmail.com
>> @philipdodds
>>
>>
>
>
> --
> Philip Dodds
>
> philip.do...@gmail.com
> @philipdodds
>
>
--
Philip Dodds
philip.do...@gmail.com
@philipdodds
