This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 7d7c991 [SPARK-36788][SQL] Change log level of AQE for non-supported
plans from warning to debug
7d7c991 is described below
commit 7d7c9915bb5ba59bed54201dfb32bd61ebd7259d
Author: Hyukjin Kwon <gurwls...@apache.org>
AuthorDate: Fri Sep 17 12:01:43 2021 +0900
[SPARK-36788][SQL] Change log level of AQE for non-supported plans from
warning to debug
### What changes were proposed in this pull request?
This PR suppresses the warnings for plans where AQE is not supported.
Currently we show the warnings such as:
```
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query: Sort
[a#324881 DESC NULLS FIRST], true, 23
+- Scan ExistingRDD[a#324881]
```
for every plan that AQE is not supported.
### Why are the changes needed?
It's too noisy now. Below is the example of `SortSuite` run:
```
14:51:40.675 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query: Sort
[a#324881 DESC NULLS FIRST], true, 23
+- Scan ExistingRDD[a#324881]
.
[info] - sorting on DayTimeIntervalType(0,1) with nullable=true,
sortOrder=List('a DESC NULLS FIRST) (785 milliseconds)
14:51:41.416 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query:
ReferenceSort [a#324884 ASC NULLS FIRST], true
+- Scan ExistingRDD[a#324884]
.
14:51:41.467 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query: Sort
[a#324884 ASC NULLS FIRST], true, 23
+- Scan ExistingRDD[a#324884]
.
[info] - sorting on DayTimeIntervalType(0,1) with nullable=false,
sortOrder=List('a ASC NULLS FIRST) (796 milliseconds)
14:51:42.210 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query:
ReferenceSort [a#324887 ASC NULLS LAST], true
+- Scan ExistingRDD[a#324887]
.
14:51:42.259 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query: Sort
[a#324887 ASC NULLS LAST], true, 23
+- Scan ExistingRDD[a#324887]
.
[info] - sorting on DayTimeIntervalType(0,1) with nullable=false,
sortOrder=List('a ASC NULLS LAST) (797 milliseconds)
14:51:43.009 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query:
ReferenceSort [a#324890 DESC NULLS LAST], true
+- Scan ExistingRDD[a#324890]
.
14:51:43.061 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query: Sort
[a#324890 DESC NULLS LAST], true, 23
+- Scan ExistingRDD[a#324890]
.
[info] - sorting on DayTimeIntervalType(0,1) with nullable=false,
sortOrder=List('a DESC NULLS LAST) (848 milliseconds)
14:51:43.857 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query:
ReferenceSort [a#324893 DESC NULLS FIRST], true
+- Scan ExistingRDD[a#324893]
.
14:51:43.903 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query: Sort
[a#324893 DESC NULLS FIRST], true, 23
+- Scan ExistingRDD[a#324893]
.
[info] - sorting on DayTimeIntervalType(0,1) with nullable=false,
sortOrder=List('a DESC NULLS FIRST) (827 milliseconds)
14:51:44.682 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query:
ReferenceSort [a#324896 ASC NULLS FIRST], true
+- Scan ExistingRDD[a#324896]
.
14:51:44.748 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query: Sort
[a#324896 ASC NULLS FIRST], true, 23
+- Scan ExistingRDD[a#324896]
.
[info] - sorting on YearMonthIntervalType(0,1) with nullable=true,
sortOrder=List('a ASC NULLS FIRST) (565 milliseconds)
14:51:45.248 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query:
ReferenceSort [a#324899 ASC NULLS LAST], true
+- Scan ExistingRDD[a#324899]
.
14:51:45.312 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query: Sort
[a#324899 ASC NULLS LAST], true, 23
+- Scan ExistingRDD[a#324899]
.
[info] - sorting on YearMonthIntervalType(0,1) with nullable=true,
sortOrder=List('a ASC NULLS LAST) (591 milliseconds)
14:51:45.841 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query:
ReferenceSort [a#324902 DESC NULLS LAST], true
+- Scan ExistingRDD[a#324902]
.
14:51:45.905 WARN
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan:
spark.sql.adaptive.enabled is enabled but is not supported for query: Sort
[a#324902 DESC NULLS LAST], true, 23
+- Scan ExistingRDD[a#324902]
.
```
### Does this PR introduce _any_ user-facing change?
Yes, it will show less warnings to users. Note that AQE is enabled by
default from Spark 3.2, see SPARK-33679
### How was this patch tested?
Manually tested via unittests.
Closes #34026 from HyukjinKwon/minor-log-level.
Authored-by: Hyukjin Kwon <gurwls...@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
(cherry picked from commit 917d7dad4dcdbeac5094899fa9b7fffc67376cec)
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
.../apache/spark/sql/execution/adaptive/InsertAdaptiveSparkPlan.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/InsertAdaptiveSparkPlan.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/InsertAdaptiveSparkPlan.scala
index c92878c..68042d8 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/InsertAdaptiveSparkPlan.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/InsertAdaptiveSparkPlan.scala
@@ -68,7 +68,7 @@ case class InsertAdaptiveSparkPlan(
plan
}
} else {
- logWarning(s"${SQLConf.ADAPTIVE_EXECUTION_ENABLED.key} is enabled " +
+ logDebug(s"${SQLConf.ADAPTIVE_EXECUTION_ENABLED.key} is enabled " +
s"but is not supported for query: $plan.")
plan
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
