[spark] branch master updated: [MINOR][SQL] Document AnalysisOnlyCommand
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9bebf4b36cd [MINOR][SQL] Document AnalysisOnlyCommand
9bebf4b36cd is described below
commit 9bebf4b36cdfc85e2ae53c448ea9fc1bf1f0f843
Author: Reynold Xin
AuthorDate: Mon Dec 26 13:31:12 2022 +0800
[MINOR][SQL] Document AnalysisOnlyCommand
### What changes were proposed in this pull request?
This patch adds more documentation to AnalysisOnlyCommand, since it's not
obvious why it's needed or how it works.
### Why are the changes needed?
See above.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
N/A
Closes #39199 from rxin/AnalysisOnlyCommand_comment.
Authored-by: Reynold Xin
Signed-off-by: Wenchen Fan
---
.../org/apache/spark/sql/catalyst/plans/logical/Command.scala | 10 +-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Command.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Command.scala
index 8e23c2ea0e2..fc9eb5d03e4 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Command.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Command.scala
@@ -44,7 +44,15 @@ trait BinaryCommand extends Command with
BinaryLike[LogicalPlan]
/**
* A logical node that can be used for a command that requires its children to
be only analyzed,
- * but not optimized.
+ * but not optimized. An example would be "create view": we don't need to
optimize the view subtree
+ * because we will just store the entire view text as is in the catalog.
+ *
+ * The way we do this is by setting the children to empty once the subtree is
analyzed. This will
+ * prevent the optimizer (or the analyzer from that point on) from traversing
into the children.
+ *
+ * There's a corresponding rule
+ * [[org.apache.spark.sql.catalyst.analysis.Analyzer.HandleSpecialCommand]]
that marks these
+ * commands analyzed.
*/
trait AnalysisOnlyCommand extends Command {
val isAnalyzed: Boolean
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-41334][CONNECT][PYTHON][FOLLOWUP] Add the missing `UNSPECIFIED` in enum
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b5036bd6de5 [SPARK-41334][CONNECT][PYTHON][FOLLOWUP] Add the missing
`UNSPECIFIED` in enum
b5036bd6de5 is described below
commit b5036bd6de52cf8569f5b4dd6835f522ed22a269
Author: Ruifeng Zheng
AuthorDate: Sun Dec 25 19:53:06 2022 +0900
[SPARK-41334][CONNECT][PYTHON][FOLLOWUP] Add the missing `UNSPECIFIED` in
enum
### What changes were proposed in this pull request?
add `SORT_DIRECTION_UNSPECIFIED ` and `SORT_NULLS_UNSPECIFIED ` removed in
https://github.com/apache/spark/pull/39090
### Why are the changes needed?
this is required in [proto style guide
](https://developers.google.com/protocol-buffers/docs/style):
> The zero value enum should have the suffix UNSPECIFIED.
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
existing UT
Closes #39208 from zhengruifeng/connect_sort_order_followup.
Authored-by: Ruifeng Zheng
Signed-off-by: Hyukjin Kwon
---
.../main/protobuf/spark/connect/expressions.proto | 10 ++--
.../pyspark/sql/connect/proto/expressions_pb2.py | 56 +++---
.../pyspark/sql/connect/proto/expressions_pb2.pyi | 20
3 files changed, 46 insertions(+), 40 deletions(-)
diff --git
a/connector/connect/common/src/main/protobuf/spark/connect/expressions.proto
b/connector/connect/common/src/main/protobuf/spark/connect/expressions.proto
index 4062e20d332..65cf9291a2f 100644
--- a/connector/connect/common/src/main/protobuf/spark/connect/expressions.proto
+++ b/connector/connect/common/src/main/protobuf/spark/connect/expressions.proto
@@ -113,13 +113,15 @@ message Expression {
NullOrdering null_ordering = 3;
enum SortDirection {
- SORT_DIRECTION_ASCENDING = 0;
- SORT_DIRECTION_DESCENDING = 1;
+ SORT_DIRECTION_UNSPECIFIED = 0;
+ SORT_DIRECTION_ASCENDING = 1;
+ SORT_DIRECTION_DESCENDING = 2;
}
enum NullOrdering {
- SORT_NULLS_FIRST = 0;
- SORT_NULLS_LAST = 1;
+ SORT_NULLS_UNSPECIFIED = 0;
+ SORT_NULLS_FIRST = 1;
+ SORT_NULLS_LAST = 2;
}
}
diff --git a/python/pyspark/sql/connect/proto/expressions_pb2.py
b/python/pyspark/sql/connect/proto/expressions_pb2.py
index 3b948b71a0e..9e5b887348c 100644
--- a/python/pyspark/sql/connect/proto/expressions_pb2.py
+++ b/python/pyspark/sql/connect/proto/expressions_pb2.py
@@ -33,7 +33,7 @@ from pyspark.sql.connect.proto import types_pb2 as
spark_dot_connect_dot_types__
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
-
b'\n\x1fspark/connect/expressions.proto\x12\rspark.connect\x1a\x19spark/connect/types.proto"\xeb\x1d\n\nExpression\x12=\n\x07literal\x18\x01
\x01(\x0b\x32!.spark.connect.Expression.LiteralH\x00R\x07literal\x12\x62\n\x14unresolved_attribute\x18\x02
\x01(\x0b\x32-.spark.connect.Expression.UnresolvedAttributeH\x00R\x13unresolvedAttribute\x12_\n\x13unresolved_function\x18\x03
\x01(\x0b\x32,.spark.connect.Expression.UnresolvedFunctionH\x00R\x12unresolvedFunction\x12Y\n\x11\x65xpression_st
[...]
+
b'\n\x1fspark/connect/expressions.proto\x12\rspark.connect\x1a\x19spark/connect/types.proto"\xa7\x1e\n\nExpression\x12=\n\x07literal\x18\x01
\x01(\x0b\x32!.spark.connect.Expression.LiteralH\x00R\x07literal\x12\x62\n\x14unresolved_attribute\x18\x02
\x01(\x0b\x32-.spark.connect.Expression.UnresolvedAttributeH\x00R\x13unresolvedAttribute\x12_\n\x13unresolved_function\x18\x03
\x01(\x0b\x32,.spark.connect.Expression.UnresolvedFunctionH\x00R\x12unresolvedFunction\x12Y\n\x11\x65xpression_st
[...]
)
@@ -226,7 +226,7 @@ if _descriptor._USE_C_DESCRIPTORS == False:
DESCRIPTOR._options = None
DESCRIPTOR._serialized_options =
b"\n\036org.apache.spark.connect.protoP\001"
_EXPRESSION._serialized_start = 78
-_EXPRESSION._serialized_end = 3897
+_EXPRESSION._serialized_end = 3957
_EXPRESSION_WINDOW._serialized_start = 943
_EXPRESSION_WINDOW._serialized_end = 1726
_EXPRESSION_WINDOW_WINDOWFRAME._serialized_start = 1233
@@ -236,31 +236,31 @@ if _descriptor._USE_C_DESCRIPTORS == False:
_EXPRESSION_WINDOW_WINDOWFRAME_FRAMETYPE._serialized_start = 1647
_EXPRESSION_WINDOW_WINDOWFRAME_FRAMETYPE._serialized_end = 1726
_EXPRESSION_SORTORDER._serialized_start = 1729
-_EXPRESSION_SORTORDER._serialized_end = 2094
+_EXPRESSION_SORTORDER._serialized_end = 2154
_EXPRESSION_SORTORDER_SORTDIRECTION._serialized_start = 1959
-_EXPRESSION_SORTORDER_SORTDIRECTION._serialized_end = 2035
-_EXPRESSION_SORTORDER_NULLORDERING._serialized_start = 2037
-_EXPRESSION_SORTORDER_NULLORDERING._serialized_end = 2094
-_EXPRESSION_CAST._serialized_start = 2097
-_EXPRESSION_CAST._serialized_end = 2242
-
[spark] branch master updated (4a887888159 -> 8c4a8d0c97b)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 4a887888159 [SPARK-41702][CONNECT][PYTHON] Add invalid column ops add 8c4a8d0c97b [SPARK-4][CONNECT][PYTHON][FOLLOWUP] Rename `numRows` in proto to follow proto style No new revisions were added by this update. Summary of changes: .../main/protobuf/spark/connect/relations.proto| 2 +- python/pyspark/sql/connect/dataframe.py| 2 +- python/pyspark/sql/connect/plan.py | 10 ++-- python/pyspark/sql/connect/proto/relations_pb2.py | 60 +++--- python/pyspark/sql/connect/proto/relations_pb2.pyi | 10 ++-- 5 files changed, 42 insertions(+), 42 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (eb3f38ea4b5 -> 4a887888159)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from eb3f38ea4b5 [SPARK-41701][CONNECT][PYTHON] Make column op support `decimal` add 4a887888159 [SPARK-41702][CONNECT][PYTHON] Add invalid column ops No new revisions were added by this update. Summary of changes: python/pyspark/sql/connect/column.py | 15 ++ .../sql/tests/connect/test_connect_column.py | 32 ++ 2 files changed, 47 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d5865d0c085 -> eb3f38ea4b5)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from d5865d0c085 [SPARK-41700][CONNECT][PYTHON] Remove `FunctionBuilder` add eb3f38ea4b5 [SPARK-41701][CONNECT][PYTHON] Make column op support `decimal` No new revisions were added by this update. Summary of changes: python/pyspark/sql/connect/column.py | 9 +++-- .../sql/tests/connect/test_connect_column.py | 39 ++ 2 files changed, 46 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
