I'll open up a JIRA for it, it appears to work when you use a literal
number but not when it is coming from the same dataframe
Thanks!
P
On Thu, Dec 3, 2015 at 1:52 AM, Sahil Sareen <sareen...@gmail.com> wrote:
> +1 looks like a bug
>
> I think referencing trades() twice in multiplication is broken,
>
> scala> trades.select(trades("quantity")*trades("quantity")).show
>
> +---------------------+
> |(quantity * quantity)|
> +---------------------+
> | null|
> | null|
>
> scala> sqlContext.sql("select price*price as PP from trades").show
>
> +----+
> | PP|
> +----+
> |null|
> |null|
>
>
> -Sahil
>
> On Thu, Dec 3, 2015 at 12:02 PM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> Not quiet sure whats happening, but its not an issue with multiplication
>> i guess as the following query worked for me:
>>
>> trades.select(trades("price")*9.5).show
>> +-------------+
>> |(price * 9.5)|
>> +-------------+
>> | 199.5|
>> | 228.0|
>> | 190.0|
>> | 199.5|
>> | 190.0|
>> | 256.5|
>> | 218.5|
>> | 275.5|
>> | 218.5|
>> ......
>> ......
>>
>>
>> Could it be with the precision? ccing dev list, may be you can open up a
>> jira for this as it seems to be a bug.
>>
>> Thanks
>> Best Regards
>>
>> On Mon, Nov 30, 2015 at 12:41 AM, Philip Dodds <philip.do...@gmail.com>
>> wrote:
>>
>>> I hit a weird issue when I tried to multiply to decimals in a select
>>> (either in scala or as SQL), and Im assuming I must be missing the point.
>>>
>>> The issue is fairly easy to recreate with something like the following:
>>>
>>>
>>> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>>> import sqlContext.implicits._
>>> import org.apache.spark.sql.types.Decimal
>>>
>>> case class Trade(quantity: Decimal,price: Decimal)
>>>
>>> val data = Seq.fill(100) {
>>> val price = Decimal(20+scala.util.Random.nextInt(10))
>>> val quantity = Decimal(20+scala.util.Random.nextInt(10))
>>>
>>> Trade(quantity, price)
>>> }
>>>
>>> val trades = sc.parallelize(data).toDF()
>>> trades.registerTempTable("trades")
>>>
>>> trades.select(trades("price")*trades("quantity")).show
>>>
>>> sqlContext.sql("select
>>> price/quantity,price*quantity,price+quantity,price-quantity from
>>> trades").show
>>>
>>> The odd part is if you run it you will see that the addition/division
>>> and subtraction works but the multiplication returns a null.
>>>
>>> Tested on 1.5.1/1.5.2 (Scala 2.10 and 2.11)
>>>
>>> ie.
>>>
>>> +------------------+
>>>
>>> |(price * quantity)|
>>>
>>> +------------------+
>>>
>>> | null|
>>>
>>> | null|
>>>
>>> | null|
>>>
>>> | null|
>>>
>>> | null|
>>>
>>> +------------------+
>>>
>>>
>>> +--------------------+----+--------------------+--------------------+
>>>
>>> | _c0| _c1| _c2| _c3|
>>>
>>> +--------------------+----+--------------------+--------------------+
>>>
>>> |0.952380952380952381|null|41.00000000000000...|-1.00000000000000...|
>>>
>>> |1.380952380952380952|null|50.00000000000000...|8.000000000000000000|
>>>
>>> |1.272727272727272727|null|50.00000000000000...|6.000000000000000000|
>>>
>>> |0.833333333333333333|null|44.00000000000000...|-4.00000000000000...|
>>>
>>> |1.000000000000000000|null|58.00000000000000...| 0E-18|
>>>
>>> +--------------------+----+--------------------+--------------------+
>>>
>>>
>>> Just keen to know what I did wrong?
>>>
>>>
>>> Cheers
>>>
>>> P
>>>
>>> --
>>> Philip Dodds
>>>
>>>
>>>
>>
>
--
Philip Dodds
philip.do...@gmail.com
@philipdodds
