[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12345 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-210261676 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55886/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-210261671 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-210261256 **[Test build #55886 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55886/consoleFull)** for PR 12345 at commit [`9b5ee1d`](https://github.com/apache/spark/commit/9b5ee1d9d648ae454cc0749047528509a032c7ab). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-210249105 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-210237677 **[Test build #55886 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55886/consoleFull)** for PR 12345 at commit [`9b5ee1d`](https://github.com/apache/spark/commit/9b5ee1d9d648ae454cc0749047528509a032c7ab). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-210236961 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55869/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-210236958 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-210236603 **[Test build #55869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55869/consoleFull)** for PR 12345 at commit [`ec66a54`](https://github.com/apache/spark/commit/ec66a5404d07da359763e4e8f8af909d045d3eb1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/12345#discussion_r59814608 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregate.scala --- @@ -261,7 +263,12 @@ case class TungstenAggregate( .map(_.asInstanceOf[DeclarativeAggregate]) private val bufferSchema = StructType.fromAttributes(aggregateBufferAttributes) - // The name for HashMap + // The name for Vectorized HashMap + private var vectorizedHashMapTerm: String = _ + private var isVectorizedHashMapEnabled: Boolean = sqlContext.conf.columnarAggregateMapEnabled && +(modes.contains(Partial) || modes.contains(PartialMerge)) --- End diff -- I don't think this is the correct logic. I think modes.forall( is Partial) is what we want. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/12345#discussion_r59814567
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregate.scala
---
@@ -437,17 +444,21 @@ case class TungstenAggregate(
val initAgg = ctx.freshName("initAgg")
ctx.addMutableState("boolean", initAgg, s"$initAgg = false;")
-// create AggregateHashMap
-val isAggregateHashMapEnabled: Boolean = false
-val isAggregateHashMapSupported: Boolean =
+// We currently only enable vectorized hashmap for long key/value types
+isVectorizedHashMapEnabled = isVectorizedHashMapEnabled &&
--- End diff --
I think it's weird that you have the check split between here and line 268.
maybe combine all these checks to 268 and make isVecdtorizedEnabled a val.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-210208658 **[Test build #55869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55869/consoleFull)** for PR 12345 at commit [`ec66a54`](https://github.com/apache/spark/commit/ec66a5404d07da359763e4e8f8af909d045d3eb1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209791378 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209791385 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55797/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209790828 **[Test build #55797 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55797/consoleFull)** for PR 12345 at commit [`041c001`](https://github.com/apache/spark/commit/041c001957999de8d383a10c0b7126c62f98ad9b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user sameeragarwal commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209771906 @nongli thanks, comments addressed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209771837 **[Test build #55797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55797/consoleFull)** for PR 12345 at commit [`041c001`](https://github.com/apache/spark/commit/041c001957999de8d383a10c0b7126c62f98ad9b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/12345#discussion_r59650089 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregate.scala --- @@ -533,56 +578,104 @@ case class TungstenAggregate( val inputAttr = aggregateBufferAttributes ++ child.output ctx.currentVars = new Array[ExprCode](aggregateBufferAttributes.length) ++ input + +ctx.INPUT_ROW = aggregateRow +// TODO: support subexpression elimination +val aggregateRowEvals = updateExpr.map(BindReferences.bindReference(_, inputAttr).gen(ctx)) --- End diff -- This code needs some more comments to unerstand the high level parts. It's not obviuos why you do this twice, once with ctx.INPUT_ROW = aggregateRow and once again with buffer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/12345#discussion_r59649964 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ColumnarAggMapCodeGenerator.scala --- @@ -21,10 +21,10 @@ import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext import org.apache.spark.sql.types.StructType /** - * This is a helper object to generate an append-only single-key/single value aggregate hash - * map that can act as a 'cache' for extremely fast key-value lookups while evaluating aggregates - * (and fall back to the `BytesToBytesMap` if a given key isn't found). This is 'codegened' in - * TungstenAggregate to speed up aggregates w/ key. + * This is a helper class to generate an append-only aggregate hash map that can act as a 'cache' + * for extremely fast key-value lookups while evaluating aggregates (and fall back to the + * `BytesToBytesMap` if a given key isn't found). This is 'codegened' in TungstenAggregate to speed + * up aggregates w/ key. --- End diff -- You chose to have this not handle null keys right? Comment that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/12345#discussion_r59649860 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregate.scala --- @@ -533,56 +578,104 @@ case class TungstenAggregate( val inputAttr = aggregateBufferAttributes ++ child.output ctx.currentVars = new Array[ExprCode](aggregateBufferAttributes.length) ++ input + +ctx.INPUT_ROW = aggregateRow +// TODO: support subexpression elimination --- End diff -- let's remove this todo in both places, not sure what it means anymore. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/12345#discussion_r59649827 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregate.scala --- @@ -513,8 +555,11 @@ case class TungstenAggregate( ctx.currentVars = input val keyCode = GenerateUnsafeProjection.createCode( --- End diff -- I think we can do better with the variable names to improve readability. keyCode, groupByKeys, key all sound too similar. Add row or unsafe row to the keys that have been collected to an unsafe row or differentiate them somehow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/12345#discussion_r59649610
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregate.scala
---
@@ -484,13 +497,42 @@ case class TungstenAggregate(
// so `copyResult` should be reset to `false`.
ctx.copyResult = false
+def outputFromGeneratedMap: Option[String] = {
+ if (isAggregateHashMapEnabled) {
+val row = ctx.freshName("aggregateHashMapRow")
--- End diff --
Comment at a high level what this is doing. Something like. " Iterate over
the aggregate rows and convert them from ColumnarBatch.Row to UnsafeRow"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/12345#discussion_r59649417
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ColumnarAggMapCodeGenerator.scala
---
@@ -65,27 +69,43 @@ class ColumnarAggMapCodeGenerator(
.mkString("\n")};
""".stripMargin
+val generatedAggBufferSchema: String =
+ s"""
+ |new org.apache.spark.sql.types.StructType()
+ |${bufferSchema.map(key =>
+s""".add("${key.name}",
org.apache.spark.sql.types.DataTypes.${key.dataType})""")
+.mkString("\n")};
+ """.stripMargin
+
s"""
| private org.apache.spark.sql.execution.vectorized.ColumnarBatch
batch;
+ | private org.apache.spark.sql.execution.vectorized.ColumnarBatch
aggregateBufferBatch;
| private int[] buckets;
| private int numBuckets;
| private int maxSteps;
| private int numRows = 0;
| private org.apache.spark.sql.types.StructType schema =
$generatedSchema
+ | private org.apache.spark.sql.types.StructType
aggregateBufferSchema =
+ |$generatedAggBufferSchema
|
- | public $generatedClassName(int capacity, double loadFactor, int
maxSteps) {
- |assert (capacity > 0 && ((capacity & (capacity - 1)) == 0));
- |this.maxSteps = maxSteps;
- |numBuckets = (int) (capacity / loadFactor);
+ | public $generatedClassName() {
+ |// TODO: These should be generated based on the schema
+ |int DEFAULT_CAPACITY = 1 << 16;
+ |double DEFAULT_LOAD_FACTOR = 0.25;
+ |int DEFAULT_MAX_STEPS = 2;
+ |assert (DEFAULT_CAPACITY > 0 && ((DEFAULT_CAPACITY &
(DEFAULT_CAPACITY - 1)) == 0));
+ |this.maxSteps = DEFAULT_MAX_STEPS;
+ |numBuckets = (int) (DEFAULT_CAPACITY / DEFAULT_LOAD_FACTOR);
|batch =
org.apache.spark.sql.execution.vectorized.ColumnarBatch.allocate(schema,
- | org.apache.spark.memory.MemoryMode.ON_HEAP, capacity);
+ | org.apache.spark.memory.MemoryMode.ON_HEAP,
DEFAULT_CAPACITY);
+ |aggregateBufferBatch =
org.apache.spark.sql.execution.vectorized.ColumnarBatch.allocate(
+ | aggregateBufferSchema,
org.apache.spark.memory.MemoryMode.ON_HEAP, DEFAULT_CAPACITY);
--- End diff --
Let's leave a TODO to fix this. There should be a nicer way to get a
projection of a batch instead of this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user sameeragarwal commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209696691 @nongli just did! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209695875 @sameeragarwal have you updated the generated code? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209694646 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209694650 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55753/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209694476 **[Test build #55753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55753/consoleFull)** for PR 12345 at commit [`0ca0db1`](https://github.com/apache/spark/commit/0ca0db17130bb6a1e59cdbc699f5abd946821d44). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209693606 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55750/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209693602 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209693429 **[Test build #55750 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55750/consoleFull)** for PR 12345 at commit [`ececd57`](https://github.com/apache/spark/commit/ececd5770e0c5410cf3da463f3b52db467d1f5ca). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user sameeragarwal commented on a diff in the pull request:
https://github.com/apache/spark/pull/12345#discussion_r59637760
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/BenchmarkWholeStageCodegen.scala
---
@@ -153,16 +153,36 @@ class BenchmarkWholeStageCodegen extends
SparkFunSuite {
ignore("aggregate with keys") {
val N = 20 << 20
-runBenchmark("Aggregate w keys", N) {
- sqlContext.range(N).selectExpr("(id & 65535) as
k").groupBy("k").sum().collect()
+val benchmark = new Benchmark("Aggregate w keys", N)
--- End diff --
Extended `testFallbackStartsAt` to now support 2 offsets (for aggregate
hash map and bytes to bytes map) and modified `AggregationQuerySuite` such that
all existing tests test for all new codepaths.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user sameeragarwal commented on a diff in the pull request:
https://github.com/apache/spark/pull/12345#discussion_r59637455
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ColumnarAggMapCodeGenerator.scala
---
@@ -103,7 +119,7 @@ class ColumnarAggMapCodeGenerator(
s"""
|// TODO: Improve this hash function
|private long hash($groupingKeySignature) {
- | return ${groupingKeys.map(_._2).mkString(" ^ ")};
+ | return ${groupingKeys.map(_._2).mkString(" | ")};
--- End diff --
No particular reason, was planning to do this as part of a separate small
PR (along with the benchmarks). Please let me know if you'd prefer it here
instead
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209675767 **[Test build #55753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55753/consoleFull)** for PR 12345 at commit [`0ca0db1`](https://github.com/apache/spark/commit/0ca0db17130bb6a1e59cdbc699f5abd946821d44). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209671390 **[Test build #55750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55750/consoleFull)** for PR 12345 at commit [`ececd57`](https://github.com/apache/spark/commit/ececd5770e0c5410cf3da463f3b52db467d1f5ca). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209622374 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55735/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209622188 **[Test build #55735 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55735/consoleFull)** for PR 12345 at commit [`fc6b8cb`](https://github.com/apache/spark/commit/fc6b8cb337e11d3d92f0f13828b8a0b85e30929c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209622370 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209593060 **[Test build #55735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55735/consoleFull)** for PR 12345 at commit [`fc6b8cb`](https://github.com/apache/spark/commit/fc6b8cb337e11d3d92f0f13828b8a0b85e30929c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209290930 I don't think this works if we have NULL keys. This is kind of annoying, let's think about that tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209290367 /* 120 */ batch.column(1).putLong(numRows, 0); I don't think this is right. You should initialize the agg exprs to NULL --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/12345#discussion_r59506077
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ColumnarAggMapCodeGenerator.scala
---
@@ -65,27 +65,43 @@ class ColumnarAggMapCodeGenerator(
.mkString("\n")};
""".stripMargin
+val generatedAggBufferSchema: String =
+ s"""
+ |new org.apache.spark.sql.types.StructType()
+ |${bufferSchema.map(key =>
+s""".add("${key.name}",
org.apache.spark.sql.types.DataTypes.${key.dataType})""")
+.mkString("\n")};
+ """.stripMargin
+
s"""
- | private org.apache.spark.sql.execution.vectorized.ColumnarBatch
batch;
+ | public org.apache.spark.sql.execution.vectorized.ColumnarBatch
batch;
--- End diff --
Can we not make either of these public. Also, have a comment that explains
why you have two batches their relation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/12345#discussion_r59505993
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/BenchmarkWholeStageCodegen.scala
---
@@ -153,16 +153,36 @@ class BenchmarkWholeStageCodegen extends
SparkFunSuite {
ignore("aggregate with keys") {
val N = 20 << 20
-runBenchmark("Aggregate w keys", N) {
- sqlContext.range(N).selectExpr("(id & 65535) as
k").groupBy("k").sum().collect()
+val benchmark = new Benchmark("Aggregate w keys", N)
--- End diff --
Do we have tests that exercise the correctness here? It should group by a
varying number of keys to test the paths you addd.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/12345#discussion_r59505821
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ColumnarAggMapCodeGenerator.scala
---
@@ -103,7 +119,7 @@ class ColumnarAggMapCodeGenerator(
s"""
|// TODO: Improve this hash function
|private long hash($groupingKeySignature) {
- | return ${groupingKeys.map(_._2).mkString(" ^ ")};
+ | return ${groupingKeys.map(_._2).mkString(" | ")};
--- End diff --
Any reason not do implmeent the h = h * 37 + v hash function?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209210452 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209210454 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55678/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209210278 **[Test build #55678 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55678/consoleFull)** for PR 12345 at commit [`c2fc385`](https://github.com/apache/spark/commit/c2fc38584dd073036a1f04f7cd7da9fcf50739e8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209209164 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55677/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209209159 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209208765 **[Test build #55677 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55677/consoleFull)** for PR 12345 at commit [`4ee5687`](https://github.com/apache/spark/commit/4ee56873764d62efdaf8c47cb74aa399f2194fde). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209192126 **[Test build #55678 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55678/consoleFull)** for PR 12345 at commit [`c2fc385`](https://github.com/apache/spark/commit/c2fc38584dd073036a1f04f7cd7da9fcf50739e8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user sameeragarwal commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209190745 cc @nongli --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12345#issuecomment-209190140 **[Test build #55677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55677/consoleFull)** for PR 12345 at commit [`4ee5687`](https://github.com/apache/spark/commit/4ee56873764d62efdaf8c47cb74aa399f2194fde). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
Github user sameeragarwal commented on the pull request:
https://github.com/apache/spark/pull/12345#issuecomment-209190135
Generate code for a query of the form `sqlContext.range(N).selectExpr("(id
& 65535) as k").groupBy("k").sum().collect()` looks like:
```java
/* 009 */ final class GeneratedIterator extends
org.apache.spark.sql.execution.BufferedRowIterator {
/* 010 */ private Object[] references;
/* 011 */ private boolean agg_initAgg;
/* 012 */ private agg_GeneratedAggregateHashMap agg_aggregateHashMap;
/* 013 */ private
java.util.Iterator
agg_genMapIter;
/* 014 */ private
org.apache.spark.sql.execution.aggregate.TungstenAggregate agg_plan;
/* 015 */ private
org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap agg_hashMap;
/* 016 */ private org.apache.spark.sql.execution.UnsafeKVExternalSorter
agg_sorter;
/* 017 */ private org.apache.spark.unsafe.KVIterator agg_mapIter;
/* 018 */ private org.apache.spark.sql.execution.metric.LongSQLMetric
range_numOutputRows;
/* 019 */ private
org.apache.spark.sql.execution.metric.LongSQLMetricValue range_metricValue;
/* 020 */ private boolean range_initRange;
/* 021 */ private long range_partitionEnd;
/* 022 */ private long range_number;
/* 023 */ private boolean range_overflow;
/* 024 */ private scala.collection.Iterator range_input;
/* 025 */ private UnsafeRow range_result;
/* 026 */ private
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder range_holder;
/* 027 */ private
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
range_rowWriter;
/* 028 */ private UnsafeRow project_result;
/* 029 */ private
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder project_holder;
/* 030 */ private
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
project_rowWriter;
/* 031 */ private UnsafeRow agg_result;
/* 032 */ private
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder agg_holder;
/* 033 */ private
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter agg_rowWriter;
/* 034 */ private
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowJoiner
agg_unsafeRowJoiner;
/* 035 */ private org.apache.spark.sql.execution.metric.LongSQLMetric
wholestagecodegen_numOutputRows;
/* 036 */ private
org.apache.spark.sql.execution.metric.LongSQLMetricValue
wholestagecodegen_metricValue;
/* 037 */ private UnsafeRow wholestagecodegen_result;
/* 038 */ private
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder
wholestagecodegen_holder;
/* 039 */ private
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
wholestagecodegen_rowWriter;
/* 040 */
/* 041 */ public GeneratedIterator(Object[] references) {
/* 042 */ this.references = references;
/* 043 */ }
/* 044 */
/* 045 */ public void init(int index, scala.collection.Iterator inputs[])
{
/* 046 */ partitionIndex = index;
/* 047 */ agg_initAgg = false;
/* 048 */ agg_aggregateHashMap = new agg_GeneratedAggregateHashMap();
/* 049 */
/* 050 */ this.agg_plan =
(org.apache.spark.sql.execution.aggregate.TungstenAggregate) references[0];
/* 051 */
/* 052 */ this.range_numOutputRows =
(org.apache.spark.sql.execution.metric.LongSQLMetric) references[1];
/* 053 */ range_metricValue =
(org.apache.spark.sql.execution.metric.LongSQLMetricValue)
range_numOutputRows.localValue();
/* 054 */ range_initRange = false;
/* 055 */ range_partitionEnd = 0L;
/* 056 */ range_number = 0L;
/* 057 */ range_overflow = false;
/* 058 */ range_input = inputs[0];
/* 059 */ range_result = new UnsafeRow(1);
/* 060 */ this.range_holder = new
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(range_result, 0);
/* 061 */ this.range_rowWriter = new
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(range_holder,
1);
/* 062 */ project_result = new UnsafeRow(1);
/* 063 */ this.project_holder = new
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_result,
0);
/* 064 */ this.project_rowWriter = new
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(project_holder,
1);
/* 065 */ agg_result = new UnsafeRow(1);
/* 066 */ this.agg_holder = new
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(agg_result, 0);
/* 067 */ this.agg_rowWriter = new
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(agg_holder,
1);
/* 068 */ agg_unsafeRowJoiner = agg_plan.createUnsafeJoiner();
/* 069 */ this.wholestagecodegen_numOutputRows =
(org.apache.spark.sql.execution.metric.LongSQLMetric) references[2];
/* 070 */ wholestagecodegen
[GitHub] spark pull request: [SPARK-14447][SQL] Speed up TungstenAggregate ...
GitHub user sameeragarwal opened a pull request: https://github.com/apache/spark/pull/12345 [SPARK-14447][SQL] Speed up TungstenAggregate w/ keys using AggregateHashMap ## What changes were proposed in this pull request? This patch speeds up group-by aggregates by around 3-5x by leveraging an in-memory `AggregateHashMap` (please see https://github.com/apache/spark/pull/12161), an append-only aggregate hash map that can act as a 'cache' for extremely fast key-value lookups while evaluating aggregates (and fall back to the `BytesToBytesMap` if a given key isn't found). Architecturally, it is backed by a power-of-2-sized array for index lookups and a columnar batch that stores the key-value pairs. The index lookups in the array rely on linear probing (with a small number of maximum tries) and use an inexpensive hash function which makes it really efficient for a majority of lookups. However, using linear probing and an inexpensive hash function also makes it less robust as compared to the `BytesToBytesMap` (especially for a large number of keys or even for certain distribution of keys) and requires us to fall back on the latter for correctness. ## How was this patch tested? Java HotSpot(TM) 64-Bit Server VM 1.8.0_73-b02 on Mac OS X 10.11.4 Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz Aggregate w keys: Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative --- codegen = F 2124 / 2204 9.9 101.3 1.0X codegen = T hashmap = F 1198 / 1364 17.5 57.1 1.8X codegen = T hashmap = T 369 / 600 56.8 17.6 5.8X You can merge this pull request into a Git repository by running: $ git pull https://github.com/sameeragarwal/spark tungsten-aggregate-integration Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12345.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12345 commit 7c158bd137f057453d17ef360906e5be90bf5004 Author: Sameer Agarwal Date: 2016-03-31T21:15:34Z [SPARK-14394] commit ebaea6a87b704afedd47bdd2dd17c92c3ffc6e8e Author: Sameer Agarwal Date: 2016-04-07T00:37:08Z Integrating AggregateHashMap for Aggregates with Group By commit cee7e65b3cf7569b4e46941158f164c2130c3981 Author: Sameer Agarwal Date: 2016-04-12T17:33:42Z Add SQLConf commit 8c9e17a1d40e3014e39b1d04f3a458aa129784f8 Author: Sameer Agarwal Date: 2016-04-12T23:01:03Z 20ns commit 3379294b76d91a55dbe86e31efb9812c8d37768c Author: Sameer Agarwal Date: 2016-04-12T23:18:36Z generated code commit 4ee56873764d62efdaf8c47cb74aa399f2194fde Author: Sameer Agarwal Date: 2016-04-13T01:23:27Z benchmark --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
