[GitHub] [spark] sigmod commented on a change in pull request #32298: [SPARK-34079][SQL] Merge non-correlated scalar subqueries to multi-column scalar subqueries for better reuse
sigmod commented on a change in pull request #32298:
URL: https://github.com/apache/spark/pull/32298#discussion_r628712826
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LeafNode,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.{MULTI_SCALAR_SUBQUERY,
SCALAR_SUBQUERY}
+
+/**
+ * This rule tries to merge multiple non-correlated [[ScalarSubquery]]s into a
+ * [[MultiScalarSubquery]] to compute multiple scalar values once.
+ *
+ * The process is the following:
+ * - While traversing through the plan each [[ScalarSubquery]] plan is tried
to merge into the cache
+ * of already seen subquery plans. If merge is possible then cache is
updated with the merged
+ * subquery plan, if not then the new subquery plan is added to the cache.
+ * - The original [[ScalarSubquery]] expression is replaced to a reference
pointing to its cached
+ * version in this form:
`GetStructField(MultiScalarSubquery(SubqueryReference(...)))`.
+ * - A second traversal checks if a [[SubqueryReference]] is pointing to a
subquery plan that
+ * returns multiple values and either replaces only [[SubqueryReference]] to
the cached plan or
+ * restores the whole expression to its original [[ScalarSubquery]] form.
+ * - [[ReuseSubquery]] rule makes sure that merged subqueries are computed
once.
+ *
+ * Eg. the following query:
+ *
+ * SELECT
+ * (SELECT avg(a) FROM t GROUP BY b),
+ * (SELECT sum(b) FROM t GROUP BY b)
+ *
+ * is optimized from:
+ *
+ * Project [scalar-subquery#231 [] AS scalarsubquery()#241,
+ * scalar-subquery#232 [] AS scalarsubquery()#242L]
+ * : :- Aggregate [b#234], [avg(a#233) AS avg(a)#236]
+ * : : +- Relation default.t[a#233,b#234] parquet
+ * : +- Aggregate [b#240], [sum(b#240) AS sum(b)#238L]
+ * : +- Project [b#240]
+ * :+- Relation default.t[a#239,b#240] parquet
Review comment:
>> In this PR the new MergeScalarSubqueries rule runs in a separate
batch
>> after column pruning, close to the end of optimization.
>> This is by design to make sure no subsequent rule changes the structure
I don't see how we can guarantee that. I think the hidden, inter-rule
dependency can add complexities for future development and maintenance.
For instance, someone could implement a new Strategy that internally calls
ColumnPruning after exploring one logical plan alternatives. By the time such a
Strategy is implemented, the authors wouldn't be aware of the fact that
ColumnPruning should *not* be called after MergeScalarSubqueries. When they
find that's an issue, they would then have to either (a) add some hacks in the
Aggregate to mark that MergeScalarSubqueries has been applied and hence
ColumnPruning should not go through it, or (b) re-implement the rule per my
proposal (1).
I'm wondering whether we can pursue (2) for now, if it meets your need. It's
less ambitious but may address most of your issue?
>> Add a performance improvement to 1. so as to physical plan a merged
>> subquery only once. This is your (1) basically.
The performance was not my initial concern, but rather, we'd better make
MergeScalarSubqueries self-contained and does not depend on *an assumption that
could be changed*.
>> But I don't think we need lateral views
Sub-querying over arrays are important use cases, for which we don't want to
de-correlate. In this case, a subquery is more like an ordinary expression and
should be evaluated within the operator node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
[GitHub] [spark] sigmod commented on a change in pull request #32298: [SPARK-34079][SQL] Merge non-correlated scalar subqueries to multi-column scalar subqueries for better reuse
sigmod commented on a change in pull request #32298:
URL: https://github.com/apache/spark/pull/32298#discussion_r628712826
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LeafNode,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.{MULTI_SCALAR_SUBQUERY,
SCALAR_SUBQUERY}
+
+/**
+ * This rule tries to merge multiple non-correlated [[ScalarSubquery]]s into a
+ * [[MultiScalarSubquery]] to compute multiple scalar values once.
+ *
+ * The process is the following:
+ * - While traversing through the plan each [[ScalarSubquery]] plan is tried
to merge into the cache
+ * of already seen subquery plans. If merge is possible then cache is
updated with the merged
+ * subquery plan, if not then the new subquery plan is added to the cache.
+ * - The original [[ScalarSubquery]] expression is replaced to a reference
pointing to its cached
+ * version in this form:
`GetStructField(MultiScalarSubquery(SubqueryReference(...)))`.
+ * - A second traversal checks if a [[SubqueryReference]] is pointing to a
subquery plan that
+ * returns multiple values and either replaces only [[SubqueryReference]] to
the cached plan or
+ * restores the whole expression to its original [[ScalarSubquery]] form.
+ * - [[ReuseSubquery]] rule makes sure that merged subqueries are computed
once.
+ *
+ * Eg. the following query:
+ *
+ * SELECT
+ * (SELECT avg(a) FROM t GROUP BY b),
+ * (SELECT sum(b) FROM t GROUP BY b)
+ *
+ * is optimized from:
+ *
+ * Project [scalar-subquery#231 [] AS scalarsubquery()#241,
+ * scalar-subquery#232 [] AS scalarsubquery()#242L]
+ * : :- Aggregate [b#234], [avg(a#233) AS avg(a)#236]
+ * : : +- Relation default.t[a#233,b#234] parquet
+ * : +- Aggregate [b#240], [sum(b#240) AS sum(b)#238L]
+ * : +- Project [b#240]
+ * :+- Relation default.t[a#239,b#240] parquet
Review comment:
>> In this PR the new MergeScalarSubqueries rule runs in a separate
batch
>> after column pruning, close to the end of optimization.
>> This is by design to make sure no subsequent rule changes the structure
I don't see how we can guarantee that. I think the hidden, inter-rule
dependency can add complexities for future development and maintenance.
For instance, someone could implement a new Strategy that internally calls
ColumnPruning after exploring one logical plan alternative. By the time such a
Strategy is implemented, the authors wouldn't be aware of the fact that
ColumnPruning should *not* be called after MergeScalarSubqueries. When they
find that's an issue, they would then have to either (a) add some hacks in the
Aggregate to mark that MergeScalarSubqueries has been applied and hence
ColumnPruning should not go through it, or (b) re-implement the rule per my
proposal (1).
I'm wondering whether we can pursue (2) for now, if it meets your need. It's
less ambitious but may address most of your issue?
>> Add a performance improvement to 1. so as to physical plan a merged
>> subquery only once. This is your (1) basically.
The performance was not my initial concern, but rather, we'd better make
MergeScalarSubqueries self-contained and does not depend on *an assumption that
could be changed*.
>> But I don't think we need lateral views
Sub-querying over arrays are important use cases, for which we don't want to
de-correlate. In this case, a subquery is more like an ordinary expression and
should be evaluated within the operator node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
--
[GitHub] [spark] SparkQA commented on pull request #32474: [SPARK-35347][SQL] Use MethodUtils for looking up methods in Invoke and StaticInvoke
SparkQA commented on pull request #32474: URL: https://github.com/apache/spark/pull/32474#issuecomment-835158249 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42802/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
AmplabJenkins commented on pull request #32031: URL: https://github.com/apache/spark/pull/32031#issuecomment-835159459 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138276/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27432: [SPARK-28325][SQL] Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
AmplabJenkins commented on pull request #27432: URL: https://github.com/apache/spark/pull/27432#issuecomment-835159461 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42801/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
SparkQA commented on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-835159651 **[Test build #138281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138281/testReport)** for PR 32470 at commit [`c67a801`](https://github.com/apache/spark/commit/c67a8013d6f848e7ee0227e50f5fec499d8648dc). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
AmplabJenkins removed a comment on pull request #32031: URL: https://github.com/apache/spark/pull/32031#issuecomment-835159459 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138276/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27432: [SPARK-28325][SQL] Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
AmplabJenkins removed a comment on pull request #27432: URL: https://github.com/apache/spark/pull/27432#issuecomment-835159461 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42801/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32298: [SPARK-34079][SQL] Merge non-correlated scalar subqueries to multi-column scalar subqueries for better reuse
sigmod commented on a change in pull request #32298:
URL: https://github.com/apache/spark/pull/32298#discussion_r628712826
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LeafNode,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.{MULTI_SCALAR_SUBQUERY,
SCALAR_SUBQUERY}
+
+/**
+ * This rule tries to merge multiple non-correlated [[ScalarSubquery]]s into a
+ * [[MultiScalarSubquery]] to compute multiple scalar values once.
+ *
+ * The process is the following:
+ * - While traversing through the plan each [[ScalarSubquery]] plan is tried
to merge into the cache
+ * of already seen subquery plans. If merge is possible then cache is
updated with the merged
+ * subquery plan, if not then the new subquery plan is added to the cache.
+ * - The original [[ScalarSubquery]] expression is replaced to a reference
pointing to its cached
+ * version in this form:
`GetStructField(MultiScalarSubquery(SubqueryReference(...)))`.
+ * - A second traversal checks if a [[SubqueryReference]] is pointing to a
subquery plan that
+ * returns multiple values and either replaces only [[SubqueryReference]] to
the cached plan or
+ * restores the whole expression to its original [[ScalarSubquery]] form.
+ * - [[ReuseSubquery]] rule makes sure that merged subqueries are computed
once.
+ *
+ * Eg. the following query:
+ *
+ * SELECT
+ * (SELECT avg(a) FROM t GROUP BY b),
+ * (SELECT sum(b) FROM t GROUP BY b)
+ *
+ * is optimized from:
+ *
+ * Project [scalar-subquery#231 [] AS scalarsubquery()#241,
+ * scalar-subquery#232 [] AS scalarsubquery()#242L]
+ * : :- Aggregate [b#234], [avg(a#233) AS avg(a)#236]
+ * : : +- Relation default.t[a#233,b#234] parquet
+ * : +- Aggregate [b#240], [sum(b#240) AS sum(b)#238L]
+ * : +- Project [b#240]
+ * :+- Relation default.t[a#239,b#240] parquet
Review comment:
>> In this PR the new MergeScalarSubqueries rule runs in a separate
batch
>> after column pruning, close to the end of optimization.
>> This is by design to make sure no subsequent rule changes the structure
I don't see how we can guarantee that. I think the hidden, inter-rule
dependency can add complexities for future development and maintenance.
For instance, someone could implement a new Strategy that internally calls
ColumnPruning after exploring one logical plan alternative. By the time such a
Strategy is implemented, the authors wouldn't be aware of the fact that
ColumnPruning should *not* be called after MergeScalarSubqueries.
- First of all, such issues can hardly be detected by the author's new
unit/query tests, as an effective test has to have both effective patterns to
trigger the Strategy and for MergeScalarSubqueries. However, such a case can
happen in prod traffic;
- Second, if they do find that's an issue, they would then have to either
(a) add some hacks in the Aggregate to mark that MergeScalarSubqueries has been
applied and hence ColumnPruning should not go through it, or (b) re-implement
the rule per my proposal (1).
I'm wondering whether we can pursue (2) for now, if it meets your need. It's
less ambitious but may address most of your issue? If you indeed have to
extract subqueries over the entire tree, I don't have a clean approach in mind
other than (1).
>> Add a performance improvement to 1. so as to physical plan a merged
>> subquery only once. This is your (1) basically.
The performance was not my initial concern, but rather, we'd better make
MergeScalarSubqueries self-contained and does not depend on *an assumption that
could be changed*.
>> But I don't think we need lateral views
Sub-querying over arrays are important use cases, for which we don't want to
de-correlate. In this case, a subqu
[GitHub] [spark] SparkQA commented on pull request #32474: [SPARK-35347][SQL] Use MethodUtils for looking up methods in Invoke and StaticInvoke
SparkQA commented on pull request #32474: URL: https://github.com/apache/spark/pull/32474#issuecomment-835160882 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42802/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32474: [SPARK-35347][SQL] Use MethodUtils for looking up methods in Invoke and StaticInvoke
AmplabJenkins commented on pull request #32474: URL: https://github.com/apache/spark/pull/32474#issuecomment-835160949 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42802/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32298: [SPARK-34079][SQL] Merge non-correlated scalar subqueries to multi-column scalar subqueries for better reuse
sigmod commented on a change in pull request #32298:
URL: https://github.com/apache/spark/pull/32298#discussion_r628712826
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LeafNode,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.{MULTI_SCALAR_SUBQUERY,
SCALAR_SUBQUERY}
+
+/**
+ * This rule tries to merge multiple non-correlated [[ScalarSubquery]]s into a
+ * [[MultiScalarSubquery]] to compute multiple scalar values once.
+ *
+ * The process is the following:
+ * - While traversing through the plan each [[ScalarSubquery]] plan is tried
to merge into the cache
+ * of already seen subquery plans. If merge is possible then cache is
updated with the merged
+ * subquery plan, if not then the new subquery plan is added to the cache.
+ * - The original [[ScalarSubquery]] expression is replaced to a reference
pointing to its cached
+ * version in this form:
`GetStructField(MultiScalarSubquery(SubqueryReference(...)))`.
+ * - A second traversal checks if a [[SubqueryReference]] is pointing to a
subquery plan that
+ * returns multiple values and either replaces only [[SubqueryReference]] to
the cached plan or
+ * restores the whole expression to its original [[ScalarSubquery]] form.
+ * - [[ReuseSubquery]] rule makes sure that merged subqueries are computed
once.
+ *
+ * Eg. the following query:
+ *
+ * SELECT
+ * (SELECT avg(a) FROM t GROUP BY b),
+ * (SELECT sum(b) FROM t GROUP BY b)
+ *
+ * is optimized from:
+ *
+ * Project [scalar-subquery#231 [] AS scalarsubquery()#241,
+ * scalar-subquery#232 [] AS scalarsubquery()#242L]
+ * : :- Aggregate [b#234], [avg(a#233) AS avg(a)#236]
+ * : : +- Relation default.t[a#233,b#234] parquet
+ * : +- Aggregate [b#240], [sum(b#240) AS sum(b)#238L]
+ * : +- Project [b#240]
+ * :+- Relation default.t[a#239,b#240] parquet
Review comment:
>> In this PR the new MergeScalarSubqueries rule runs in a separate
batch
>> after column pruning, close to the end of optimization.
>> This is by design to make sure no subsequent rule changes the structure
I don't see how we can guarantee that. I think the hidden, inter-rule
dependency can add complexities for future development and maintenance.
For instance, someone could implement a new Strategy that internally calls
ColumnPruning after exploring one logical plan alternative. By the time such a
Strategy is implemented, the authors wouldn't be aware of the fact that
ColumnPruning should *not* be called after MergeScalarSubqueries.
- First of all, such issues can hardly be detected by the author's new
unit/query tests, as an effective test has to have both effective patterns to
trigger the Strategy and for MergeScalarSubqueries. However, such a case can
happen in prod traffic;
- Second, if they do find that's an issue, they would then have to either
(a) add some hacks in the Aggregate to mark that MergeScalarSubqueries has been
applied and hence ColumnPruning should not go through it, or (b) re-implement
MergeScalarSubqueries per my proposal (1).
I'm wondering whether we can pursue (2) for now, if it meets your need. It's
less ambitious but may address most of your issue? If you indeed have to
extract subqueries over the entire tree, I don't have a clean approach in mind
other than (1).
>> Add a performance improvement to 1. so as to physical plan a merged
>> subquery only once. This is your (1) basically.
The performance was not my initial concern, but rather, we'd better make
MergeScalarSubqueries self-contained and does not depend on *an assumption that
could be changed*.
>> But I don't think we need lateral views
Sub-querying over arrays are important use cases, for which we don't want to
de-correlate. In this
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32474: [SPARK-35347][SQL] Use MethodUtils for looking up methods in Invoke and StaticInvoke
AmplabJenkins removed a comment on pull request #32474: URL: https://github.com/apache/spark/pull/32474#issuecomment-835160949 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42802/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32298: [SPARK-34079][SQL] Merge non-correlated scalar subqueries to multi-column scalar subqueries for better reuse
sigmod commented on a change in pull request #32298:
URL: https://github.com/apache/spark/pull/32298#discussion_r628712826
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LeafNode,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.{MULTI_SCALAR_SUBQUERY,
SCALAR_SUBQUERY}
+
+/**
+ * This rule tries to merge multiple non-correlated [[ScalarSubquery]]s into a
+ * [[MultiScalarSubquery]] to compute multiple scalar values once.
+ *
+ * The process is the following:
+ * - While traversing through the plan each [[ScalarSubquery]] plan is tried
to merge into the cache
+ * of already seen subquery plans. If merge is possible then cache is
updated with the merged
+ * subquery plan, if not then the new subquery plan is added to the cache.
+ * - The original [[ScalarSubquery]] expression is replaced to a reference
pointing to its cached
+ * version in this form:
`GetStructField(MultiScalarSubquery(SubqueryReference(...)))`.
+ * - A second traversal checks if a [[SubqueryReference]] is pointing to a
subquery plan that
+ * returns multiple values and either replaces only [[SubqueryReference]] to
the cached plan or
+ * restores the whole expression to its original [[ScalarSubquery]] form.
+ * - [[ReuseSubquery]] rule makes sure that merged subqueries are computed
once.
+ *
+ * Eg. the following query:
+ *
+ * SELECT
+ * (SELECT avg(a) FROM t GROUP BY b),
+ * (SELECT sum(b) FROM t GROUP BY b)
+ *
+ * is optimized from:
+ *
+ * Project [scalar-subquery#231 [] AS scalarsubquery()#241,
+ * scalar-subquery#232 [] AS scalarsubquery()#242L]
+ * : :- Aggregate [b#234], [avg(a#233) AS avg(a)#236]
+ * : : +- Relation default.t[a#233,b#234] parquet
+ * : +- Aggregate [b#240], [sum(b#240) AS sum(b)#238L]
+ * : +- Project [b#240]
+ * :+- Relation default.t[a#239,b#240] parquet
Review comment:
>> In this PR the new MergeScalarSubqueries rule runs in a separate
batch
>> after column pruning, close to the end of optimization.
>> This is by design to make sure no subsequent rule changes the structure
I don't see how we can guarantee that. I think the hidden, inter-rule
dependency can add complexities for future development and maintenance.
For instance, someone could implement a new Strategy that internally calls
ColumnPruning after exploring one logical plan alternative. By the time such a
Strategy is implemented, the authors wouldn't be aware of the fact that
ColumnPruning should *not* be called after MergeScalarSubqueries.
- First of all, such issues can hardly be detected by the author's new
unit/query tests, as an effective test has to have both effective patterns to
trigger the Strategy and for MergeScalarSubqueries. However, such a case can
happen in prod traffic;
- Second, if they do find that's an issue, they would then have to either
(a) add some hacks in the Aggregate to mark that MergeScalarSubqueries has been
applied and hence ColumnPruning should not go through it, or (b) re-implement
MergeScalarSubqueries per my proposal (1).
I'm wondering whether we can pursue (2) for now, if it meets your need. It's
less ambitious but may address most of your issue? If you indeed have to
extract subqueries over the entire tree, I don't have a clean approach in mind
other than (1).
>> Add a performance improvement to 1. so as to physical plan a merged
>> subquery only once. This is your (1) basically.
The performance was not my initial concern, but rather, we'd better make
MergeScalarSubqueries self-contained and does not depend on *an assumption that
could later be changed*.
>> But I don't think we need lateral views
Sub-querying over arrays are important use cases, for which we don't want to
de-correlate. In
[GitHub] [spark] sigmod commented on a change in pull request #32298: [SPARK-34079][SQL] Merge non-correlated scalar subqueries to multi-column scalar subqueries for better reuse
sigmod commented on a change in pull request #32298:
URL: https://github.com/apache/spark/pull/32298#discussion_r628712826
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LeafNode,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.{MULTI_SCALAR_SUBQUERY,
SCALAR_SUBQUERY}
+
+/**
+ * This rule tries to merge multiple non-correlated [[ScalarSubquery]]s into a
+ * [[MultiScalarSubquery]] to compute multiple scalar values once.
+ *
+ * The process is the following:
+ * - While traversing through the plan each [[ScalarSubquery]] plan is tried
to merge into the cache
+ * of already seen subquery plans. If merge is possible then cache is
updated with the merged
+ * subquery plan, if not then the new subquery plan is added to the cache.
+ * - The original [[ScalarSubquery]] expression is replaced to a reference
pointing to its cached
+ * version in this form:
`GetStructField(MultiScalarSubquery(SubqueryReference(...)))`.
+ * - A second traversal checks if a [[SubqueryReference]] is pointing to a
subquery plan that
+ * returns multiple values and either replaces only [[SubqueryReference]] to
the cached plan or
+ * restores the whole expression to its original [[ScalarSubquery]] form.
+ * - [[ReuseSubquery]] rule makes sure that merged subqueries are computed
once.
+ *
+ * Eg. the following query:
+ *
+ * SELECT
+ * (SELECT avg(a) FROM t GROUP BY b),
+ * (SELECT sum(b) FROM t GROUP BY b)
+ *
+ * is optimized from:
+ *
+ * Project [scalar-subquery#231 [] AS scalarsubquery()#241,
+ * scalar-subquery#232 [] AS scalarsubquery()#242L]
+ * : :- Aggregate [b#234], [avg(a#233) AS avg(a)#236]
+ * : : +- Relation default.t[a#233,b#234] parquet
+ * : +- Aggregate [b#240], [sum(b#240) AS sum(b)#238L]
+ * : +- Project [b#240]
+ * :+- Relation default.t[a#239,b#240] parquet
Review comment:
>> In this PR the new MergeScalarSubqueries rule runs in a separate
batch
>> after column pruning, close to the end of optimization.
>> This is by design to make sure no subsequent rule changes the structure
I don't see how we can guarantee that. I think the hidden, inter-rule
dependency can add complexities for future development and maintenance.
For instance, someone could implement a new Strategy that internally calls
ColumnPruning after exploring one logical plan alternative. By the time such a
Strategy is implemented, the authors wouldn't be aware of the fact that
ColumnPruning should *not* be called after MergeScalarSubqueries.
- First of all, such issues can hardly be detected by the author's new
unit/query tests, as an effective test has to have both effective patterns to
trigger the Strategy and for MergeScalarSubqueries. However, such a case can
happen in prod traffic;
- Second, if they do find that's an issue, they would then have to either
(a) add some hacks in the Aggregate to mark that MergeScalarSubqueries has been
applied and hence ColumnPruning should not go through it, or (b) re-implement
MergeScalarSubqueries per my proposal (1).
I'm wondering whether we can pursue (2) for now, if it meets your need. It's
less ambitious but may address most of your issue? If you indeed have to
extract subqueries over the entire tree, I don't have a clean approach in mind
other than (1).
>> Add a performance improvement to 1. so as to physical plan a merged
>> subquery only once. This is your (1) basically.
The performance was not my initial concern, but rather, I think we'd better
make MergeScalarSubqueries self-contained and does not depend on *an assumption
that could later be changed*.
>> But I don't think we need lateral views
Sub-querying over arrays are important use cases, for which we don't want to
de-corre
[GitHub] [spark] sunchao commented on pull request #32354: [SPARK-35232][SQL] Nested column pruning should retain column metadata
sunchao commented on pull request #32354: URL: https://github.com/apache/spark/pull/32354#issuecomment-835163544 Thanks @viirya for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32473: [SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark
SparkQA commented on pull request #32473: URL: https://github.com/apache/spark/pull/32473#issuecomment-835165918 **[Test build #138272 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138272/testReport)** for PR 32473 at commit [`1bb6675`](https://github.com/apache/spark/commit/1bb6675f6e09e9ca844ec021e310316584bdf876). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32473: [SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark
SparkQA removed a comment on pull request #32473: URL: https://github.com/apache/spark/pull/32473#issuecomment-835030476 **[Test build #138272 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138272/testReport)** for PR 32473 at commit [`1bb6675`](https://github.com/apache/spark/commit/1bb6675f6e09e9ca844ec021e310316584bdf876). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32473: [SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark
AmplabJenkins commented on pull request #32473: URL: https://github.com/apache/spark/pull/32473#issuecomment-835166794 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138272/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32473: [SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark
AmplabJenkins removed a comment on pull request #32473: URL: https://github.com/apache/spark/pull/32473#issuecomment-835166794 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138272/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32298: [SPARK-34079][SQL] Merge non-correlated scalar subqueries to multi-column scalar subqueries for better reuse
sigmod commented on a change in pull request #32298:
URL: https://github.com/apache/spark/pull/32298#discussion_r628712826
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/MergeScalarSubqueries.scala
##
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LeafNode,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.{MULTI_SCALAR_SUBQUERY,
SCALAR_SUBQUERY}
+
+/**
+ * This rule tries to merge multiple non-correlated [[ScalarSubquery]]s into a
+ * [[MultiScalarSubquery]] to compute multiple scalar values once.
+ *
+ * The process is the following:
+ * - While traversing through the plan each [[ScalarSubquery]] plan is tried
to merge into the cache
+ * of already seen subquery plans. If merge is possible then cache is
updated with the merged
+ * subquery plan, if not then the new subquery plan is added to the cache.
+ * - The original [[ScalarSubquery]] expression is replaced to a reference
pointing to its cached
+ * version in this form:
`GetStructField(MultiScalarSubquery(SubqueryReference(...)))`.
+ * - A second traversal checks if a [[SubqueryReference]] is pointing to a
subquery plan that
+ * returns multiple values and either replaces only [[SubqueryReference]] to
the cached plan or
+ * restores the whole expression to its original [[ScalarSubquery]] form.
+ * - [[ReuseSubquery]] rule makes sure that merged subqueries are computed
once.
+ *
+ * Eg. the following query:
+ *
+ * SELECT
+ * (SELECT avg(a) FROM t GROUP BY b),
+ * (SELECT sum(b) FROM t GROUP BY b)
+ *
+ * is optimized from:
+ *
+ * Project [scalar-subquery#231 [] AS scalarsubquery()#241,
+ * scalar-subquery#232 [] AS scalarsubquery()#242L]
+ * : :- Aggregate [b#234], [avg(a#233) AS avg(a)#236]
+ * : : +- Relation default.t[a#233,b#234] parquet
+ * : +- Aggregate [b#240], [sum(b#240) AS sum(b)#238L]
+ * : +- Project [b#240]
+ * :+- Relation default.t[a#239,b#240] parquet
Review comment:
>> In this PR the new MergeScalarSubqueries rule runs in a separate
batch
>> after column pruning, close to the end of optimization.
>> This is by design to make sure no subsequent rule changes the structure
I don't see how we can guarantee that. I think the hidden, inter-rule
dependency can add complexities for future development and maintenance.
For instance, someone could implement a new Strategy that internally calls
ColumnPruning after exploring one logical plan alternative. By the time such a
Strategy is implemented, the authors wouldn't be aware of the fact that
ColumnPruning should *not* be called after MergeScalarSubqueries.
- First of all, such issues can hardly be detected by the author's new
unit/query tests, as an effective test has to have both effective patterns to
trigger the Strategy and MergeScalarSubqueries. However, such a case can happen
in prod traffic;
- Second, if they do find that's an issue, they would then have to either
(a) add some hacks in the Aggregate to mark that MergeScalarSubqueries has been
applied and hence ColumnPruning should not go through it, or (b) re-implement
MergeScalarSubqueries per my proposal (1).
I'm wondering whether we can pursue (2) for now, if it meets your need. It's
less ambitious but may address most of your issue? If you indeed have to
extract subqueries over the entire tree, I don't have a clean approach in mind
other than (1).
>> Add a performance improvement to 1. so as to physical plan a merged
>> subquery only once. This is your (1) basically.
The performance was not my initial concern, but rather, I think we'd better
make MergeScalarSubqueries self-contained and does not depend on *an assumption
that could later be changed*.
>> But I don't think we need lateral views
Sub-querying over arrays are important use cases, for which we don't want to
de-correlate
[GitHub] [spark] AngersZhuuuu opened a new pull request #32475: Spark 34775
AngersZh opened a new pull request #32475: URL: https://github.com/apache/spark/pull/32475 ### What changes were proposed in this pull request? This is a followup from #31691. Push down limit through Window when the partitionSpec of all window functions is not empty and the same order is used. Push down limit through Window when the partitionSpec of all window functions is not empty ### Why are the changes needed? Improve query performance. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added UT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] c21 opened a new pull request #32476: [SPARK-35349][SQL] Add code-gen for left/right outer sort merge join
c21 opened a new pull request #32476:
URL: https://github.com/apache/spark/pull/32476
### What changes were proposed in this pull request?
This PR is to add code-gen support for LEFT OUTER / RIGHT OUTER sort merge
join. Currently sort merge join only supports inner join type
(https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala#L374
). There's no fundamental reason why we cannot support code-gen for other join
types. Here we add code-gen for LEFT OUTER / RIGHT OUTER join. Will submit
followup PRs to add LEFT SEMI, LEFT ANTI and FULL OUTER code-gen separately.
The change is to extend current sort merge join logic to work with LEFT
OUTER and RIGHT OUTER (should work with LEFT SEMI/ANTI as well, but FULL OUTER
join needs some other more code change). Replace left/right with
streamed/buffered to make code extendable to other join types besides inner
join.
Example query:
```
val df1 = spark.range(10).select($"id".as("k1"), $"id".as("k3"))
val df2 = spark.range(4).select($"id".as("k2"), $"id".as("k4"))
df1.join(df2.hint("SHUFFLE_MERGE"), $"k1" === $"k2" && $"k3" + 1 < $"k4",
"left_outer").explain("codegen")
```
Example generated code:
```
== Subtree 5 / 5 (maxMethodCodeSize:396; maxConstantPoolSize:159(0.24%
used); numInnerClasses:0) ==
*(5) SortMergeJoin [k1#2L], [k2#8L], LeftOuter, ((k3#3L + 1) < k4#9L)
:- *(2) Sort [k1#2L ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(k1#2L, 5), ENSURE_REQUIREMENTS, [id=#26]
: +- *(1) Project [id#0L AS k1#2L, id#0L AS k3#3L]
:+- *(1) Range (0, 10, step=1, splits=2)
+- *(4) Sort [k2#8L ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(k2#8L, 5), ENSURE_REQUIREMENTS, [id=#32]
+- *(3) Project [id#6L AS k2#8L, id#6L AS k4#9L]
+- *(3) Range (0, 4, step=1, splits=2)
Generated code:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIteratorForCodegenStage5(references);
/* 003 */ }
/* 004 */
/* 005 */ // codegenStageId=5
/* 006 */ final class GeneratedIteratorForCodegenStage5 extends
org.apache.spark.sql.execution.BufferedRowIterator {
/* 007 */ private Object[] references;
/* 008 */ private scala.collection.Iterator[] inputs;
/* 009 */ private scala.collection.Iterator smj_streamedInput_0;
/* 010 */ private scala.collection.Iterator smj_bufferedInput_0;
/* 011 */ private InternalRow smj_streamedRow_0;
/* 012 */ private InternalRow smj_bufferedRow_0;
/* 013 */ private long smj_value_2;
/* 014 */ private
org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray smj_matches_0;
/* 015 */ private long smj_value_3;
/* 016 */ private
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[]
smj_mutableStateArray_0 = new
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[1];
/* 017 */
/* 018 */ public GeneratedIteratorForCodegenStage5(Object[] references) {
/* 019 */ this.references = references;
/* 020 */ }
/* 021 */
/* 022 */ public void init(int index, scala.collection.Iterator[] inputs) {
/* 023 */ partitionIndex = index;
/* 024 */ this.inputs = inputs;
/* 025 */ smj_streamedInput_0 = inputs[0];
/* 026 */ smj_bufferedInput_0 = inputs[1];
/* 027 */
/* 028 */ smj_matches_0 = new
org.apache.spark.sql.execution.ExternalAppendOnlyUnsafeRowArray(2147483632,
2147483647);
/* 029 */ smj_mutableStateArray_0[0] = new
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(4, 0);
/* 030 */
/* 031 */ }
/* 032 */
/* 033 */ private boolean findNextJoinRows(
/* 034 */ scala.collection.Iterator streamedIter,
/* 035 */ scala.collection.Iterator bufferedIter) {
/* 036 */ smj_streamedRow_0 = null;
/* 037 */ int comp = 0;
/* 038 */ while (smj_streamedRow_0 == null) {
/* 039 */ if (!streamedIter.hasNext()) return false;
/* 040 */ smj_streamedRow_0 = (InternalRow) streamedIter.next();
/* 041 */ long smj_value_0 = smj_streamedRow_0.getLong(0);
/* 042 */ if (false) {
/* 043 */ if (!smj_matches_0.isEmpty()) {
/* 044 */ smj_matches_0.clear();
/* 045 */ }
/* 046 */ return false;
/* 047 */
/* 048 */ }
/* 049 */ if (!smj_matches_0.isEmpty()) {
/* 050 */ comp = 0;
/* 051 */ if (comp == 0) {
/* 052 */ comp = (smj_value_0 > smj_value_3 ? 1 : smj_value_0 <
smj_value_3 ? -1 : 0);
/* 053 */ }
/* 054 */
/* 055 */ if (comp == 0) {
/* 056 */ return true;
/* 057 */ }
/* 058 */ smj_matches_0.clear();
/* 059 */ }
/* 060 */
/* 061 */ do {
/* 062 */ if (smj_bufferedRow_0 == null) {
/* 063 *
[GitHub] [spark] SparkQA commented on pull request #27432: [SPARK-28325][SQL] Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
SparkQA commented on pull request #27432: URL: https://github.com/apache/spark/pull/27432#issuecomment-835171715 **[Test build #138279 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138279/testReport)** for PR 27432 at commit [`e4805a8`](https://github.com/apache/spark/commit/e4805a82db49628ee535ae53f8e5b4de43a21053). * This patch **fails PySpark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #27432: [SPARK-28325][SQL] Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
SparkQA removed a comment on pull request #27432: URL: https://github.com/apache/spark/pull/27432#issuecomment-835107611 **[Test build #138279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138279/testReport)** for PR 27432 at commit [`e4805a8`](https://github.com/apache/spark/commit/e4805a82db49628ee535ae53f8e5b4de43a21053). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] c21 commented on pull request #32476: [SPARK-35349][SQL] Add code-gen for left/right outer sort merge join
c21 commented on pull request #32476: URL: https://github.com/apache/spark/pull/32476#issuecomment-835172528 cc @cloud-fan and @maropu could you help take a look when you have time? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32444: [SPARK-35111][SPARK-35112][SQL][FOLLOWUP] Support Cast string to year-month interval
SparkQA commented on pull request #32444: URL: https://github.com/apache/spark/pull/32444#issuecomment-835181688 **[Test build #138274 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138274/testReport)** for PR 32444 at commit [`8569d86`](https://github.com/apache/spark/commit/8569d86a7c9b7867f69e40165618dda3389dc0ca). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32444: [SPARK-35111][SPARK-35112][SQL][FOLLOWUP] Support Cast string to year-month interval
SparkQA removed a comment on pull request #32444: URL: https://github.com/apache/spark/pull/32444#issuecomment-835030582 **[Test build #138274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138274/testReport)** for PR 32444 at commit [`8569d86`](https://github.com/apache/spark/commit/8569d86a7c9b7867f69e40165618dda3389dc0ca). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
SparkQA commented on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-835187278 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32465: [SPARK-35331][SQL] Support resolving missing attrs for distribute/cluster by/repartition hint
SparkQA commented on pull request #32465: URL: https://github.com/apache/spark/pull/32465#issuecomment-835189993 **[Test build #138273 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138273/testReport)** for PR 32465 at commit [`2ce0fb6`](https://github.com/apache/spark/commit/2ce0fb690ea3123233a8714402367ddc12346907). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32465: [SPARK-35331][SQL] Support resolving missing attrs for distribute/cluster by/repartition hint
SparkQA removed a comment on pull request #32465: URL: https://github.com/apache/spark/pull/32465#issuecomment-835030515 **[Test build #138273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138273/testReport)** for PR 32465 at commit [`2ce0fb6`](https://github.com/apache/spark/commit/2ce0fb690ea3123233a8714402367ddc12346907). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #27432: [SPARK-28325][SQL] Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
AmplabJenkins commented on pull request #27432: URL: https://github.com/apache/spark/pull/27432#issuecomment-835193649 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138279/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32444: [SPARK-35111][SPARK-35112][SQL][FOLLOWUP] Support Cast string to year-month interval
AmplabJenkins commented on pull request #32444: URL: https://github.com/apache/spark/pull/32444#issuecomment-835193651 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138274/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32465: [SPARK-35331][SQL] Support resolving missing attrs for distribute/cluster by/repartition hint
AmplabJenkins commented on pull request #32465: URL: https://github.com/apache/spark/pull/32465#issuecomment-835193650 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138273/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
AmplabJenkins commented on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-835193655 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42803/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
AmplabJenkins removed a comment on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-835193655 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42803/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32444: [SPARK-35111][SPARK-35112][SQL][FOLLOWUP] Support Cast string to year-month interval
AmplabJenkins removed a comment on pull request #32444: URL: https://github.com/apache/spark/pull/32444#issuecomment-835193651 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138274/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32465: [SPARK-35331][SQL] Support resolving missing attrs for distribute/cluster by/repartition hint
AmplabJenkins removed a comment on pull request #32465: URL: https://github.com/apache/spark/pull/32465#issuecomment-835193650 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138273/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #27432: [SPARK-28325][SQL] Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
AmplabJenkins removed a comment on pull request #27432: URL: https://github.com/apache/spark/pull/27432#issuecomment-835193649 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138279/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32475: [SPARK-34775][SQL] Push down limit through window when partitionSpec is not empty
SparkQA commented on pull request #32475: URL: https://github.com/apache/spark/pull/32475#issuecomment-835194836 **[Test build #138283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138283/testReport)** for PR 32475 at commit [`c4f8220`](https://github.com/apache/spark/commit/c4f8220b35ca3fb6133d2707d1ce5131a7cdb67a). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32476: [SPARK-35349][SQL] Add code-gen for left/right outer sort merge join
SparkQA commented on pull request #32476: URL: https://github.com/apache/spark/pull/32476#issuecomment-835194769 **[Test build #138282 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138282/testReport)** for PR 32476 at commit [`95e56b8`](https://github.com/apache/spark/commit/95e56b8cbbeee90ce04103f5771a22adbb64be84). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #30138: [SPARK-33075][SQL] Enable auto bucketed scan by default (disable only for cached query)
maropu commented on a change in pull request #30138:
URL: https://github.com/apache/spark/pull/30138#discussion_r628719198
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanHelper.scala
##
@@ -139,15 +138,21 @@ trait AdaptiveSparkPlanHelper {
}
/**
- * Returns a cloned [[SparkSession]] with adaptive execution disabled, or
the original
- * [[SparkSession]] if its adaptive execution is already disabled.
+ * Returns a cloned [[SparkSession]] with all specified configurations
disabled, or
+ * the original [[SparkSession]] if all configurations are already disabled.
*/
- def getOrCloneSessionWithAqeOff[T](session: SparkSession): SparkSession = {
-if (!session.sessionState.conf.adaptiveExecutionEnabled) {
+ def getOrCloneSessionWithConfigsOff(
Review comment:
Could you move this discussion into
https://issues.apache.org/jira/browse/SPARK-35332 ? I thinks the jira ticket is
related to this topic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #30138: [SPARK-33075][SQL] Enable auto bucketed scan by default (disable only for cached query)
maropu commented on a change in pull request #30138:
URL: https://github.com/apache/spark/pull/30138#discussion_r628719198
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanHelper.scala
##
@@ -139,15 +138,21 @@ trait AdaptiveSparkPlanHelper {
}
/**
- * Returns a cloned [[SparkSession]] with adaptive execution disabled, or
the original
- * [[SparkSession]] if its adaptive execution is already disabled.
+ * Returns a cloned [[SparkSession]] with all specified configurations
disabled, or
+ * the original [[SparkSession]] if all configurations are already disabled.
*/
- def getOrCloneSessionWithAqeOff[T](session: SparkSession): SparkSession = {
-if (!session.sessionState.conf.adaptiveExecutionEnabled) {
+ def getOrCloneSessionWithConfigsOff(
Review comment:
Could you join the discussion in
https://issues.apache.org/jira/browse/SPARK-35332 ? I thinks the jira ticket is
related to this topic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32475: [SPARK-34775][SQL] Push down limit through window when partitionSpec is not empty
SparkQA commented on pull request #32475: URL: https://github.com/apache/spark/pull/32475#issuecomment-835198057 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32475: [SPARK-34775][SQL] Push down limit through window when partitionSpec is not empty
AmplabJenkins commented on pull request #32475: URL: https://github.com/apache/spark/pull/32475#issuecomment-835198095 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42804/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32475: [SPARK-34775][SQL] Push down limit through window when partitionSpec is not empty
AmplabJenkins removed a comment on pull request #32475: URL: https://github.com/apache/spark/pull/32475#issuecomment-835198095 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42804/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer opened a new pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
beliefer opened a new pull request #32477:
URL: https://github.com/apache/spark/pull/32477
### What changes were proposed in this pull request?
`ANSI SQL: SIMILAR TO ... ESCAPE` is very useful.
There are some mainstream database support the syntax.
**PostgreSQL**:
https://www.postgresql.org/docs/current/functions-matching.html#FUNCTIONS-SIMILARTO-REGEXP
**Redshift**:
https://docs.aws.amazon.com/redshift/latest/dg/pattern-matching-conditions-similar-to.html
**Sybase**:
http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.help.sqlanywhere.12.0.0/dbreference/like-regexp-similarto.html
**Firebird**:
http://firebirdsql.org/file/documentation/html/en/refdocs/fblangref25/firebird-25-language-reference.html#fblangref25-commons-predsiimilarto
This util supports the following pattern-matching metacharacters:
Operator | Description
-- | --
% | Matches any sequence of zero or more characters.
_ | Matches any single character.
\| | Denotes alternation (either of two alternatives).
\* | Repeat the previous item zero or more times.
\+ | Repeat the previous item one or more times.
? | Repeat the previous item zero or one time.
{m} | Repeat the previous item exactly m times.
{m,} | Repeat the previous item m or more times.
{m,n} | Repeat the previous item at least m and not more than n times.
() | Parentheses group items into a single logical item.
[...] | A bracket expression specifies a character class, just as in POSIX
regular expressions.
### Why are the changes needed?
`ANSI SQL: SIMILAR TO ... ESCAPE` is very useful.
### Does this PR introduce _any_ user-facing change?
Yes, a new feature.
### How was this patch tested?
New tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhongyu09 commented on pull request #31269: [SPARK-33933][SQL] Materialize BroadcastQueryStage first to try to avoid broadcast timeout in AQE
zhongyu09 commented on pull request #31269: URL: https://github.com/apache/spark/pull/31269#issuecomment-835203722 Hi @cloud-fan @HyukjinKwon @viirya, I have some follow up to resolved the issue completely, it is better to reuse the JIRA SPARK-33933 or create a new JIRA? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
SparkQA commented on pull request #32477: URL: https://github.com/apache/spark/pull/32477#issuecomment-835205122 **[Test build #138284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138284/testReport)** for PR 32477 at commit [`1363fb5`](https://github.com/apache/spark/commit/1363fb5c473a9d3f15d829f029432e8f8a1ab30d). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
SparkQA commented on pull request #32477: URL: https://github.com/apache/spark/pull/32477#issuecomment-835206382 **[Test build #138284 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138284/testReport)** for PR 32477 at commit [`1363fb5`](https://github.com/apache/spark/commit/1363fb5c473a9d3f15d829f029432e8f8a1ab30d). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
SparkQA removed a comment on pull request #32477: URL: https://github.com/apache/spark/pull/32477#issuecomment-835205122 **[Test build #138284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138284/testReport)** for PR 32477 at commit [`1363fb5`](https://github.com/apache/spark/commit/1363fb5c473a9d3f15d829f029432e8f8a1ab30d). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
SparkQA commented on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-835207742 **[Test build #138281 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138281/testReport)** for PR 32470 at commit [`c67a801`](https://github.com/apache/spark/commit/c67a8013d6f848e7ee0227e50f5fec499d8648dc). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
SparkQA removed a comment on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-835159651 **[Test build #138281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138281/testReport)** for PR 32470 at commit [`c67a801`](https://github.com/apache/spark/commit/c67a8013d6f848e7ee0227e50f5fec499d8648dc). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #32199: [SPARK-35100][ML] Refactor AFT - support virtual centering
zhengruifeng commented on pull request #32199: URL: https://github.com/apache/spark/pull/32199#issuecomment-835214415 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32476: [SPARK-35349][SQL] Add code-gen for left/right outer sort merge join
SparkQA commented on pull request #32476: URL: https://github.com/apache/spark/pull/32476#issuecomment-835219552 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42805/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32476: [SPARK-35349][SQL] Add code-gen for left/right outer sort merge join
SparkQA commented on pull request #32476: URL: https://github.com/apache/spark/pull/32476#issuecomment-835221920 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42805/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32442: [SPARK-35283][SQL] Support query some DDL with CTES
SparkQA commented on pull request #32442: URL: https://github.com/apache/spark/pull/32442#issuecomment-835225367 **[Test build #138275 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138275/testReport)** for PR 32442 at commit [`afed495`](https://github.com/apache/spark/commit/afed4959158b637a7eac58b382a2ad629e755860). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32442: [SPARK-35283][SQL] Support query some DDL with CTES
SparkQA removed a comment on pull request #32442: URL: https://github.com/apache/spark/pull/32442#issuecomment-835069839 **[Test build #138275 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138275/testReport)** for PR 32442 at commit [`afed495`](https://github.com/apache/spark/commit/afed4959158b637a7eac58b382a2ad629e755860). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #32350: [SPARK-35231][SQL] logical.Range override maxRowsPerPartition
zhengruifeng commented on a change in pull request #32350: URL: https://github.com/apache/spark/pull/32350#discussion_r622693358 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala ## @@ -161,6 +162,7 @@ case class Filter(condition: Expression, child: LogicalPlan) override def output: Seq[Attribute] = child.output override def maxRows: Option[Long] = child.maxRows + override def maxRowsPerPartition: Option[Long] = child.maxRowsPerPartition Review comment: this change is not needed for the added test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
AmplabJenkins commented on pull request #32477: URL: https://github.com/apache/spark/pull/32477#issuecomment-835228682 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138284/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32442: [SPARK-35283][SQL] Support query some DDL with CTES
AmplabJenkins commented on pull request #32442: URL: https://github.com/apache/spark/pull/32442#issuecomment-835228686 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138275/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #19393: [SPARK-21644][SQL] LocalLimit.maxRows is defined incorrectly
zhengruifeng commented on a change in pull request #19393:
URL: https://github.com/apache/spark/pull/19393#discussion_r628724926
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##
@@ -200,6 +203,17 @@ case class Union(children: Seq[LogicalPlan]) extends
LogicalPlan {
}
}
+ /**
+ * Note the definition has assumption about how union is implemented
physically.
+ */
+ override def maxRowsPerPartition: Option[Long] = {
+if (children.exists(_.maxRowsPerPartition.isEmpty)) {
+ None
+} else {
+ Some(children.flatMap(_.maxRowsPerPartition).sum)
Review comment:
I guess here should be
`Some(children.flatMap(_.maxRowsPerPartition).max)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32476: [SPARK-35349][SQL] Add code-gen for left/right outer sort merge join
AmplabJenkins commented on pull request #32476: URL: https://github.com/apache/spark/pull/32476#issuecomment-835228684 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42805/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
AmplabJenkins commented on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-835228683 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138281/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32442: [SPARK-35283][SQL] Support query some DDL with CTES
AmplabJenkins removed a comment on pull request #32442: URL: https://github.com/apache/spark/pull/32442#issuecomment-835228686 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138275/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32476: [SPARK-35349][SQL] Add code-gen for left/right outer sort merge join
AmplabJenkins removed a comment on pull request #32476: URL: https://github.com/apache/spark/pull/32476#issuecomment-835228684 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42805/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
AmplabJenkins removed a comment on pull request #32470: URL: https://github.com/apache/spark/pull/32470#issuecomment-835228683 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138281/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
AmplabJenkins removed a comment on pull request #32477: URL: https://github.com/apache/spark/pull/32477#issuecomment-835228682 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138284/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
SparkQA commented on pull request #32477: URL: https://github.com/apache/spark/pull/32477#issuecomment-835230828 **[Test build #138285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138285/testReport)** for PR 32477 at commit [`cd8d9f7`](https://github.com/apache/spark/commit/cd8d9f7dc08721b77274ea21f38d3302b49f3dea). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32199: [SPARK-35100][ML] Refactor AFT - support virtual centering
SparkQA commented on pull request #32199: URL: https://github.com/apache/spark/pull/32199#issuecomment-835231150 **[Test build #138286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138286/testReport)** for PR 32199 at commit [`e8658cc`](https://github.com/apache/spark/commit/e8658cc1da49a02fbdf0b0db6d10d010a980d55b). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31847: [SPARK-34755][SQL] Support the utils for transform number format
SparkQA commented on pull request #31847: URL: https://github.com/apache/spark/pull/31847#issuecomment-835231358 **[Test build #138287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138287/testReport)** for PR 31847 at commit [`88d814e`](https://github.com/apache/spark/commit/88d814eb923ec24405fbcb3abcbaa701599acfa8). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
SparkQA commented on pull request #32477: URL: https://github.com/apache/spark/pull/32477#issuecomment-835231674 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
AmplabJenkins commented on pull request #32477: URL: https://github.com/apache/spark/pull/32477#issuecomment-835231716 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42806/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
AmplabJenkins removed a comment on pull request #32477: URL: https://github.com/apache/spark/pull/32477#issuecomment-835231716 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42806/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32199: [SPARK-35100][ML] Refactor AFT - support virtual centering
SparkQA commented on pull request #32199: URL: https://github.com/apache/spark/pull/32199#issuecomment-835253282 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31847: [SPARK-34755][SQL] Support the utils for transform number format
SparkQA commented on pull request #31847: URL: https://github.com/apache/spark/pull/31847#issuecomment-835254619 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
SparkQA commented on pull request #32477: URL: https://github.com/apache/spark/pull/32477#issuecomment-835256998 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42807/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32199: [SPARK-35100][ML] Refactor AFT - support virtual centering
AmplabJenkins commented on pull request #32199: URL: https://github.com/apache/spark/pull/32199#issuecomment-835262724 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42808/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31847: [SPARK-34755][SQL] Support the utils for transform number format
AmplabJenkins commented on pull request #31847: URL: https://github.com/apache/spark/pull/31847#issuecomment-835262728 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42809/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32199: [SPARK-35100][ML] Refactor AFT - support virtual centering
AmplabJenkins removed a comment on pull request #32199: URL: https://github.com/apache/spark/pull/32199#issuecomment-835262724 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42808/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31847: [SPARK-34755][SQL] Support the utils for transform number format
AmplabJenkins removed a comment on pull request #31847: URL: https://github.com/apache/spark/pull/31847#issuecomment-835262728 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42809/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32465: [SPARK-35331][SQL] Support resolving missing attrs for distribute/cluster by/repartition hint
SparkQA commented on pull request #32465: URL: https://github.com/apache/spark/pull/32465#issuecomment-835263700 **[Test build #138288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138288/testReport)** for PR 32465 at commit [`5163262`](https://github.com/apache/spark/commit/51632625e9fb3f09800112fe121de1dcae1dacee). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32464: [SPARK-35062][SQL] Group exception messages in sql/streaming
SparkQA commented on pull request #32464: URL: https://github.com/apache/spark/pull/32464#issuecomment-835264165 **[Test build #138278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138278/testReport)** for PR 32464 at commit [`bab4787`](https://github.com/apache/spark/commit/bab478798fa903894b184e5111dfad7d9cc50ad0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
SparkQA commented on pull request #32477: URL: https://github.com/apache/spark/pull/32477#issuecomment-835264449 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42807/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
AmplabJenkins commented on pull request #32477: URL: https://github.com/apache/spark/pull/32477#issuecomment-835264467 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42807/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32464: [SPARK-35062][SQL] Group exception messages in sql/streaming
SparkQA removed a comment on pull request #32464: URL: https://github.com/apache/spark/pull/32464#issuecomment-835102357 **[Test build #138278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138278/testReport)** for PR 32464 at commit [`bab4787`](https://github.com/apache/spark/commit/bab478798fa903894b184e5111dfad7d9cc50ad0). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32477: [SPARK-35348][SQL] Support the utils for escapse the regex for ANSI SQL: SIMILAR TO … ESCAPE syntax
AmplabJenkins removed a comment on pull request #32477: URL: https://github.com/apache/spark/pull/32477#issuecomment-835264467 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42807/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32464: [SPARK-35062][SQL] Group exception messages in sql/streaming
AmplabJenkins commented on pull request #32464: URL: https://github.com/apache/spark/pull/32464#issuecomment-835265304 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138278/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32464: [SPARK-35062][SQL] Group exception messages in sql/streaming
AmplabJenkins removed a comment on pull request #32464: URL: https://github.com/apache/spark/pull/32464#issuecomment-835265304 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138278/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32199: [SPARK-35100][ML] Refactor AFT - support virtual centering
SparkQA commented on pull request #32199: URL: https://github.com/apache/spark/pull/32199#issuecomment-835265968 **[Test build #138286 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138286/testReport)** for PR 32199 at commit [`e8658cc`](https://github.com/apache/spark/commit/e8658cc1da49a02fbdf0b0db6d10d010a980d55b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32199: [SPARK-35100][ML] Refactor AFT - support virtual centering
SparkQA removed a comment on pull request #32199: URL: https://github.com/apache/spark/pull/32199#issuecomment-835231150 **[Test build #138286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138286/testReport)** for PR 32199 at commit [`e8658cc`](https://github.com/apache/spark/commit/e8658cc1da49a02fbdf0b0db6d10d010a980d55b). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32199: [SPARK-35100][ML] Refactor AFT - support virtual centering
AmplabJenkins commented on pull request #32199: URL: https://github.com/apache/spark/pull/32199#issuecomment-835266408 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138286/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32199: [SPARK-35100][ML] Refactor AFT - support virtual centering
AmplabJenkins removed a comment on pull request #32199: URL: https://github.com/apache/spark/pull/32199#issuecomment-835266408 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138286/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #32464: [SPARK-35062][SQL] Group exception messages in sql/streaming
beliefer commented on pull request #32464: URL: https://github.com/apache/spark/pull/32464#issuecomment-835269616 ping @allisonwang-db -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer opened a new pull request #32478: [SPARK-35063][SQL] Group exception messages in sql/catalyst
beliefer opened a new pull request #32478: URL: https://github.com/apache/spark/pull/32478 ### What changes were proposed in this pull request? This PR group exception messages in `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst`. ### Why are the changes needed? It will largely help with standardization of error messages and its maintenance. ### Does this PR introduce _any_ user-facing change? No. Error messages remain unchanged. ### How was this patch tested? No new tests - pass all original tests to make sure it doesn't break any existing behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32478: [SPARK-35063][SQL] Group exception messages in sql/catalyst
SparkQA commented on pull request #32478: URL: https://github.com/apache/spark/pull/32478#issuecomment-835271558 **[Test build #138289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138289/testReport)** for PR 32478 at commit [`5967c33`](https://github.com/apache/spark/commit/5967c33c8dc375691e324cffa8a540ff73031799). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32478: [SPARK-35063][SQL] Group exception messages in sql/catalyst
SparkQA commented on pull request #32478: URL: https://github.com/apache/spark/pull/32478#issuecomment-835272549 **[Test build #138289 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138289/testReport)** for PR 32478 at commit [`5967c33`](https://github.com/apache/spark/commit/5967c33c8dc375691e324cffa8a540ff73031799). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32478: [SPARK-35063][SQL] Group exception messages in sql/catalyst
AmplabJenkins commented on pull request #32478: URL: https://github.com/apache/spark/pull/32478#issuecomment-835272571 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138289/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32478: [SPARK-35063][SQL] Group exception messages in sql/catalyst
SparkQA removed a comment on pull request #32478: URL: https://github.com/apache/spark/pull/32478#issuecomment-835271558 **[Test build #138289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138289/testReport)** for PR 32478 at commit [`5967c33`](https://github.com/apache/spark/commit/5967c33c8dc375691e324cffa8a540ff73031799). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32478: [SPARK-35063][SQL] Group exception messages in sql/catalyst
AmplabJenkins removed a comment on pull request #32478: URL: https://github.com/apache/spark/pull/32478#issuecomment-835272571 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138289/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
