Opened https://issues.apache.org/jira/browse/SPARK-12128
Thanks
P
On Thu, Dec 3, 2015 at 8:51 AM, Philip Dodds <philip.do...@gmail.com> wrote:
> I'll open up a JIRA for it, it appears to work when you use a literal
> number but not when it is coming from the same dataframe
>
> Thanks!
>
> P
>
> On Thu, Dec 3, 2015 at 1:52 AM, Sahil Sareen <sareen...@gmail.com> wrote:
>
>> +1 looks like a bug
>>
>> I think referencing trades() twice in multiplication is broken,
>>
>> scala> trades.select(trades("quantity")*trades("quantity")).show
>>
>> +---------------------+
>> |(quantity * quantity)|
>> +---------------------+
>> | null|
>> | null|
>>
>> scala> sqlContext.sql("select price*price as PP from trades").show
>>
>> +----+
>> | PP|
>> +----+
>> |null|
>> |null|
>>
>>
>> -Sahil
>>
>> On Thu, Dec 3, 2015 at 12:02 PM, Akhil Das <ak...@sigmoidanalytics.com>
>> wrote:
>>
>>> Not quiet sure whats happening, but its not an issue with multiplication
>>> i guess as the following query worked for me:
>>>
>>> trades.select(trades("price")*9.5).show
>>> +-------------+
>>> |(price * 9.5)|
>>> +-------------+
>>> | 199.5|
>>> | 228.0|
>>> | 190.0|
>>> | 199.5|
>>> | 190.0|
>>> | 256.5|
>>> | 218.5|
>>> | 275.5|
>>> | 218.5|
>>> ......
>>> ......
>>>
>>>
>>> Could it be with the precision? ccing dev list, may be you can open up a
>>> jira for this as it seems to be a bug.
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Mon, Nov 30, 2015 at 12:41 AM, Philip Dodds <philip.do...@gmail.com>
>>> wrote:
>>>
>>>> I hit a weird issue when I tried to multiply to decimals in a select
>>>> (either in scala or as SQL), and Im assuming I must be missing the point.
>>>>
>>>> The issue is fairly easy to recreate with something like the following:
>>>>
>>>>
>>>> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>>>> import sqlContext.implicits._
>>>> import org.apache.spark.sql.types.Decimal
>>>>
>>>> case class Trade(quantity: Decimal,price: Decimal)
>>>>
>>>> val data = Seq.fill(100) {
>>>> val price = Decimal(20+scala.util.Random.nextInt(10))
>>>> val quantity = Decimal(20+scala.util.Random.nextInt(10))
>>>>
>>>> Trade(quantity, price)
>>>> }
>>>>
>>>> val trades = sc.parallelize(data).toDF()
>>>> trades.registerTempTable("trades")
>>>>
>>>> trades.select(trades("price")*trades("quantity")).show
>>>>
>>>> sqlContext.sql("select
>>>> price/quantity,price*quantity,price+quantity,price-quantity from
>>>> trades").show
>>>>
>>>> The odd part is if you run it you will see that the addition/division
>>>> and subtraction works but the multiplication returns a null.
>>>>
>>>> Tested on 1.5.1/1.5.2 (Scala 2.10 and 2.11)
>>>>
>>>> ie.
>>>>
>>>> +------------------+
>>>>
>>>> |(price * quantity)|
>>>>
>>>> +------------------+
>>>>
>>>> | null|
>>>>
>>>> | null|
>>>>
>>>> | null|
>>>>
>>>> | null|
>>>>
>>>> | null|
>>>>
>>>> +------------------+
>>>>
>>>>
>>>> +--------------------+----+--------------------+--------------------+
>>>>
>>>> | _c0| _c1| _c2| _c3|
>>>>
>>>> +--------------------+----+--------------------+--------------------+
>>>>
>>>> |0.952380952380952381|null|41.00000000000000...|-1.00000000000000...|
>>>>
>>>> |1.380952380952380952|null|50.00000000000000...|8.000000000000000000|
>>>>
>>>> |1.272727272727272727|null|50.00000000000000...|6.000000000000000000|
>>>>
>>>> |0.833333333333333333|null|44.00000000000000...|-4.00000000000000...|
>>>>
>>>> |1.000000000000000000|null|58.00000000000000...| 0E-18|
>>>>
>>>> +--------------------+----+--------------------+--------------------+
>>>>
>>>>
>>>> Just keen to know what I did wrong?
>>>>
>>>>
>>>> Cheers
>>>>
>>>> P
>>>>
>>>> --
>>>> Philip Dodds
>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> Philip Dodds
>
> philip.do...@gmail.com
> @philipdodds
>
>
--
Philip Dodds
philip.do...@gmail.com
@philipdodds
