Aggregators#composing-aggregators
> .
>
> Best,
> Anastasios
>
> On Mon, Dec 18, 2017 at 10:38 AM, Julien CHAMP <jch...@tellmeplus.com>
> wrote:
>
>> I've been looking for several solutions but I can't find something
>> efficient to compute many window function efficie
I've been looking for several solutions but I can't find something
efficient to compute many window function efficiently ( optimized
computation or efficient parallelism )
Am I the only one interested by this ?
Regards,
Julien
Le ven. 15 déc. 2017 à 21:34, Julien CHAMP <jch...@tellmeplus.
May be I should consider something like impala ?
Le ven. 15 déc. 2017 à 11:32, Julien CHAMP <jch...@tellmeplus.com> a écrit :
> Hi Spark Community members !
>
> I want to do several ( from 1 to 10) aggregate functions using window
> functions on something like 100 columns.
&g
quot;col").over(tw))
Is not really efficient :/
It seems that it iterates on the whole column for each aggregation ? Am I
right ?
Is there a way to compute all the required operations on a columns with a
single pass ?
Event better, to compute all the required operations on ALL columns with a
si
e Chebaane
>
>
> 2017-07-18 18:21 GMT+02:00 Julien CHAMP <jch...@tellmeplus.com>:
>
>> Hi Radhwane !
>>
>> I've tested both your solutions using dataframe or spark sql... and in
>> both cases spark is stucked :/
>> Did you test the code that you g
s _value
> FROM df a CROSS JOIN df b
> ON b.timestamp >= a.timestamp - 200000L and b.timestamp <= a.timestamp
> ) c
> GROUP BY c.id, c.timestamp, c.value ORDER BY c.timestamp
>
>
> This must be also possible also on Spark Streaming however don't expect hig
ps://issues.apache.org/jira/browse/SPARK-19451 ) when working with Long
values !!! So I can't use this
So my question is ( of course ) how can I resolve my problem ?
If I use spark streaming I will face the same issue ?
I'll be glad to discuss this problem with you, feel free to answer :)
Re
ps://issues.apache.org/jira/browse/SPARK-19451 ) when working with Long
values !!! So I can't use this
So my question is ( of course ) how can I resolve my problem ?
If I use spark streaming I will face the same issue ?
I'll be glad to discuss this problem with you, feel free to answer :)
Regard
