Github user tilumi commented on the issue:
https://github.com/apache/spark/pull/14129
I optimized the 'codeGenWithArrayAggBufferNumericHistogram' algorithm and
the benchmark result is:
|(rows,
Github user tilumi commented on the issue:
https://github.com/apache/spark/pull/14129
I finally come out a optimal solution. Since the number of attributes in
aggregation buffer is linear to generated code size & code generation time, I
instead use just one attribute with array type.
Github user tilumi commented on the issue:
https://github.com/apache/spark/pull/14129
I'll improve it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user tilumi commented on the issue:
https://github.com/apache/spark/pull/14129
According to the result, I figured out the performance of the codegen
version is bounded by the number of bins since the large generated code for
array creation.
---
If your project is set up
Github user tilumi commented on the issue:
https://github.com/apache/spark/pull/14129
I Implement 3 kinds of histogram_numeric and the result is
(10, 100)).map((pair) => {
---
If your project is set up for it, you can reply to this email and have your
reply
Github user tilumi commented on the issue:
https://github.com/apache/spark/pull/14129
I think the cause may be it'll use sortAggregation instead of
hashaggregation if I use array as an buffer
---
If your project is set up for it, you can reply to this email and have your
reply
Github user tilumi commented on the issue:
https://github.com/apache/spark/pull/14129
May I ask why an array in DeclarativeAggregate is not the most efficient??
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user tilumi commented on the issue:
https://github.com/apache/spark/pull/14129
Ok, I'll change the implementation to non-array version and run benchmark
on both versions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14129
**[Test build #3178 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3178/consoleFull)**
for PR 14129 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14129
**[Test build #3178 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3178/consoleFull)**
for PR 14129 at commit
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/14129
@tilumi could show some benchmarks for this? I think that this will have
some performance problems. Using an array in a DeclarativeAggregate is not the
most efficient.
---
If your project is
11 matches
Mail list logo
