[spark] branch master updated: [SPARK-32234][FOLLOWUP][SQL] Update the description of utility method
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 99f33ec [SPARK-32234][FOLLOWUP][SQL] Update the description of
utility method
99f33ec is described below
commit 99f33ec30f04bb0f7b09c3c2abfc5d5b6af50599
Author: SaurabhChawla
AuthorDate: Mon Jul 27 08:14:02 2020 +
[SPARK-32234][FOLLOWUP][SQL] Update the description of utility method
### What changes were proposed in this pull request?
As the part of this PR https://github.com/apache/spark/pull/29045 added the
helper method. This PR is the FOLLOWUP PR to update the description of helper
method.
### Why are the changes needed?
For better readability and understanding of the code
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Since its only change of updating the description , So ran the Spark shell
Closes #29232 from SaurabhChawla100/SPARK-32234-Desc.
Authored-by: SaurabhChawla
Signed-off-by: Wenchen Fan
---
.../spark/sql/execution/datasources/orc/OrcUtils.scala | 14 ++
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
index e102539..072e670 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
@@ -207,10 +207,16 @@ object OrcUtils extends Logging {
}
/**
- * @return Returns the result schema string based on the canPruneCols flag.
- * resultSchemaString will be created using resultsSchema in case of
- * canPruneCols is true and for canPruneCols as false value
- * resultSchemaString will be created using the actual dataSchema.
+ * Returns the result schema to read from ORC file. In addition, It sets
+ * the schema string to 'orc.mapred.input.schema' so ORC reader can use
later.
+ *
+ * @param canPruneCols Flag to decide whether pruned cols schema is send to
resultSchema
+ * or to send the entire dataSchema to resultSchema.
+ * @param dataSchema Schema of the orc files.
+ * @param resultSchema Result data schema created after pruning cols.
+ * @param partitionSchema Schema of partitions.
+ * @param conf Hadoop Configuration.
+ * @return Returns the result schema as string.
*/
def orcResultSchemaString(
canPruneCols: Boolean,
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bfa5d57 -> 99f33ec)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bfa5d57 [SPARK-32452][R][SQL] Bump up the minimum Arrow version as 1.0.0 in SparkR add 99f33ec [SPARK-32234][FOLLOWUP][SQL] Update the description of utility method No new revisions were added by this update. Summary of changes: .../spark/sql/execution/datasources/orc/OrcUtils.scala | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bfa5d57 -> 99f33ec)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bfa5d57 [SPARK-32452][R][SQL] Bump up the minimum Arrow version as 1.0.0 in SparkR add 99f33ec [SPARK-32234][FOLLOWUP][SQL] Update the description of utility method No new revisions were added by this update. Summary of changes: .../spark/sql/execution/datasources/orc/OrcUtils.scala | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bfa5d57 -> 99f33ec)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bfa5d57 [SPARK-32452][R][SQL] Bump up the minimum Arrow version as 1.0.0 in SparkR add 99f33ec [SPARK-32234][FOLLOWUP][SQL] Update the description of utility method No new revisions were added by this update. Summary of changes: .../spark/sql/execution/datasources/orc/OrcUtils.scala | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bfa5d57 -> 99f33ec)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bfa5d57 [SPARK-32452][R][SQL] Bump up the minimum Arrow version as 1.0.0 in SparkR add 99f33ec [SPARK-32234][FOLLOWUP][SQL] Update the description of utility method No new revisions were added by this update. Summary of changes: .../spark/sql/execution/datasources/orc/OrcUtils.scala | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (99f33ec -> 6ab29b3)
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 99f33ec [SPARK-32234][FOLLOWUP][SQL] Update the description of
utility method
add 6ab29b3 [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign
the documentation base
No new revisions were added by this update.
Summary of changes:
.github/workflows/master.yml | 8 +-
.gitignore | 1 +
LICENSE| 2 +-
dev/create-release/spark-rm/Dockerfile | 5 +-
dev/lint-python| 18 +-
dev/requirements.txt | 1 +
dev/tox.ini| 2 +-
docs/README.md | 7 +-
docs/_plugins/copy_api_dirs.rb | 4 +-
docs/img/spark-logo-reverse.png| Bin 0 -> 21547 bytes
python/docs/Makefile | 4 +-
python/docs/_static/pyspark.css| 90
python/docs/_static/pyspark.js | 99
python/docs/_templates/layout.html | 6 -
python/docs/index.rst | 53 --
python/docs/make2.bat | 4 +-
python/docs/pyspark.ml.rst | 122 -
python/docs/pyspark.mllib.rst | 99
python/docs/pyspark.resource.rst | 11 -
python/docs/pyspark.rst| 21 -
python/docs/pyspark.sql.rst| 37 --
python/docs/pyspark.streaming.rst | 17 -
python/docs/{ => source}/_static/copybutton.js | 0
python/docs/source/_static/css/pyspark.css | 94
python/docs/source/_templates/class_with_docs.rst | 79 +++
python/docs/{ => source}/conf.py | 51 +-
python/docs/source/development/index.rst | 21 +
python/docs/source/getting_started/index.rst | 22 +
python/docs/source/index.rst | 32 ++
python/docs/source/migration_guide/index.rst | 22 +
python/docs/source/reference/index.rst | 34 ++
python/docs/source/reference/pyspark.ml.rst| 363 ++
python/docs/source/reference/pyspark.mllib.rst | 253 ++
python/docs/source/reference/pyspark.resource.rst | 38 ++
python/docs/source/reference/pyspark.rst | 275 +++
python/docs/source/reference/pyspark.sql.rst | 542 +
python/docs/source/reference/pyspark.ss.rst| 90
python/docs/source/reference/pyspark.streaming.rst | 130 +
python/docs/source/user_guide/index.rst| 22 +
python/pyspark/sql/functions.py| 4 +
python/pyspark/sql/readwriter.py | 17 +
python/pyspark/sql/streaming.py| 13 +
42 files changed, 2129 insertions(+), 584 deletions(-)
create mode 100644 docs/img/spark-logo-reverse.png
delete mode 100644 python/docs/_static/pyspark.css
delete mode 100644 python/docs/_static/pyspark.js
delete mode 100644 python/docs/_templates/layout.html
delete mode 100644 python/docs/index.rst
delete mode 100644 python/docs/pyspark.ml.rst
delete mode 100644 python/docs/pyspark.mllib.rst
delete mode 100644 python/docs/pyspark.resource.rst
delete mode 100644 python/docs/pyspark.rst
delete mode 100644 python/docs/pyspark.sql.rst
delete mode 100644 python/docs/pyspark.streaming.rst
rename python/docs/{ => source}/_static/copybutton.js (100%)
create mode 100644 python/docs/source/_static/css/pyspark.css
create mode 100644 python/docs/source/_templates/class_with_docs.rst
rename python/docs/{ => source}/conf.py (89%)
create mode 100644 python/docs/source/development/index.rst
create mode 100644 python/docs/source/getting_started/index.rst
create mode 100644 python/docs/source/index.rst
create mode 100644 python/docs/source/migration_guide/index.rst
create mode 100644 python/docs/source/reference/index.rst
create mode 100644 python/docs/source/reference/pyspark.ml.rst
create mode 100644 python/docs/source/reference/pyspark.mllib.rst
create mode 100644 python/docs/source/reference/pyspark.resource.rst
create mode 100644 python/docs/source/reference/pyspark.rst
create mode 100644 python/docs/source/reference/pyspark.sql.rst
create mode 100644 python/docs/source/reference/pyspark.ss.rst
create mode 100644 python/docs/source/reference/pyspark.streaming.rst
create mode 100644 python/docs/source/user_guide/index.rst
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (99f33ec -> 6ab29b3)
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 99f33ec [SPARK-32234][FOLLOWUP][SQL] Update the description of
utility method
add 6ab29b3 [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign
the documentation base
No new revisions were added by this update.
Summary of changes:
.github/workflows/master.yml | 8 +-
.gitignore | 1 +
LICENSE| 2 +-
dev/create-release/spark-rm/Dockerfile | 5 +-
dev/lint-python| 18 +-
dev/requirements.txt | 1 +
dev/tox.ini| 2 +-
docs/README.md | 7 +-
docs/_plugins/copy_api_dirs.rb | 4 +-
docs/img/spark-logo-reverse.png| Bin 0 -> 21547 bytes
python/docs/Makefile | 4 +-
python/docs/_static/pyspark.css| 90
python/docs/_static/pyspark.js | 99
python/docs/_templates/layout.html | 6 -
python/docs/index.rst | 53 --
python/docs/make2.bat | 4 +-
python/docs/pyspark.ml.rst | 122 -
python/docs/pyspark.mllib.rst | 99
python/docs/pyspark.resource.rst | 11 -
python/docs/pyspark.rst| 21 -
python/docs/pyspark.sql.rst| 37 --
python/docs/pyspark.streaming.rst | 17 -
python/docs/{ => source}/_static/copybutton.js | 0
python/docs/source/_static/css/pyspark.css | 94
python/docs/source/_templates/class_with_docs.rst | 79 +++
python/docs/{ => source}/conf.py | 51 +-
python/docs/source/development/index.rst | 21 +
python/docs/source/getting_started/index.rst | 22 +
python/docs/source/index.rst | 32 ++
python/docs/source/migration_guide/index.rst | 22 +
python/docs/source/reference/index.rst | 34 ++
python/docs/source/reference/pyspark.ml.rst| 363 ++
python/docs/source/reference/pyspark.mllib.rst | 253 ++
python/docs/source/reference/pyspark.resource.rst | 38 ++
python/docs/source/reference/pyspark.rst | 275 +++
python/docs/source/reference/pyspark.sql.rst | 542 +
python/docs/source/reference/pyspark.ss.rst| 90
python/docs/source/reference/pyspark.streaming.rst | 130 +
python/docs/source/user_guide/index.rst| 22 +
python/pyspark/sql/functions.py| 4 +
python/pyspark/sql/readwriter.py | 17 +
python/pyspark/sql/streaming.py| 13 +
42 files changed, 2129 insertions(+), 584 deletions(-)
create mode 100644 docs/img/spark-logo-reverse.png
delete mode 100644 python/docs/_static/pyspark.css
delete mode 100644 python/docs/_static/pyspark.js
delete mode 100644 python/docs/_templates/layout.html
delete mode 100644 python/docs/index.rst
delete mode 100644 python/docs/pyspark.ml.rst
delete mode 100644 python/docs/pyspark.mllib.rst
delete mode 100644 python/docs/pyspark.resource.rst
delete mode 100644 python/docs/pyspark.rst
delete mode 100644 python/docs/pyspark.sql.rst
delete mode 100644 python/docs/pyspark.streaming.rst
rename python/docs/{ => source}/_static/copybutton.js (100%)
create mode 100644 python/docs/source/_static/css/pyspark.css
create mode 100644 python/docs/source/_templates/class_with_docs.rst
rename python/docs/{ => source}/conf.py (89%)
create mode 100644 python/docs/source/development/index.rst
create mode 100644 python/docs/source/getting_started/index.rst
create mode 100644 python/docs/source/index.rst
create mode 100644 python/docs/source/migration_guide/index.rst
create mode 100644 python/docs/source/reference/index.rst
create mode 100644 python/docs/source/reference/pyspark.ml.rst
create mode 100644 python/docs/source/reference/pyspark.mllib.rst
create mode 100644 python/docs/source/reference/pyspark.resource.rst
create mode 100644 python/docs/source/reference/pyspark.rst
create mode 100644 python/docs/source/reference/pyspark.sql.rst
create mode 100644 python/docs/source/reference/pyspark.ss.rst
create mode 100644 python/docs/source/reference/pyspark.streaming.rst
create mode 100644 python/docs/source/user_guide/index.rst
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (99f33ec -> 6ab29b3)
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 99f33ec [SPARK-32234][FOLLOWUP][SQL] Update the description of
utility method
add 6ab29b3 [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign
the documentation base
No new revisions were added by this update.
Summary of changes:
.github/workflows/master.yml | 8 +-
.gitignore | 1 +
LICENSE| 2 +-
dev/create-release/spark-rm/Dockerfile | 5 +-
dev/lint-python| 18 +-
dev/requirements.txt | 1 +
dev/tox.ini| 2 +-
docs/README.md | 7 +-
docs/_plugins/copy_api_dirs.rb | 4 +-
docs/img/spark-logo-reverse.png| Bin 0 -> 21547 bytes
python/docs/Makefile | 4 +-
python/docs/_static/pyspark.css| 90
python/docs/_static/pyspark.js | 99
python/docs/_templates/layout.html | 6 -
python/docs/index.rst | 53 --
python/docs/make2.bat | 4 +-
python/docs/pyspark.ml.rst | 122 -
python/docs/pyspark.mllib.rst | 99
python/docs/pyspark.resource.rst | 11 -
python/docs/pyspark.rst| 21 -
python/docs/pyspark.sql.rst| 37 --
python/docs/pyspark.streaming.rst | 17 -
python/docs/{ => source}/_static/copybutton.js | 0
python/docs/source/_static/css/pyspark.css | 94
python/docs/source/_templates/class_with_docs.rst | 79 +++
python/docs/{ => source}/conf.py | 51 +-
python/docs/source/development/index.rst | 21 +
python/docs/source/getting_started/index.rst | 22 +
python/docs/source/index.rst | 32 ++
python/docs/source/migration_guide/index.rst | 22 +
python/docs/source/reference/index.rst | 34 ++
python/docs/source/reference/pyspark.ml.rst| 363 ++
python/docs/source/reference/pyspark.mllib.rst | 253 ++
python/docs/source/reference/pyspark.resource.rst | 38 ++
python/docs/source/reference/pyspark.rst | 275 +++
python/docs/source/reference/pyspark.sql.rst | 542 +
python/docs/source/reference/pyspark.ss.rst| 90
python/docs/source/reference/pyspark.streaming.rst | 130 +
python/docs/source/user_guide/index.rst| 22 +
python/pyspark/sql/functions.py| 4 +
python/pyspark/sql/readwriter.py | 17 +
python/pyspark/sql/streaming.py| 13 +
42 files changed, 2129 insertions(+), 584 deletions(-)
create mode 100644 docs/img/spark-logo-reverse.png
delete mode 100644 python/docs/_static/pyspark.css
delete mode 100644 python/docs/_static/pyspark.js
delete mode 100644 python/docs/_templates/layout.html
delete mode 100644 python/docs/index.rst
delete mode 100644 python/docs/pyspark.ml.rst
delete mode 100644 python/docs/pyspark.mllib.rst
delete mode 100644 python/docs/pyspark.resource.rst
delete mode 100644 python/docs/pyspark.rst
delete mode 100644 python/docs/pyspark.sql.rst
delete mode 100644 python/docs/pyspark.streaming.rst
rename python/docs/{ => source}/_static/copybutton.js (100%)
create mode 100644 python/docs/source/_static/css/pyspark.css
create mode 100644 python/docs/source/_templates/class_with_docs.rst
rename python/docs/{ => source}/conf.py (89%)
create mode 100644 python/docs/source/development/index.rst
create mode 100644 python/docs/source/getting_started/index.rst
create mode 100644 python/docs/source/index.rst
create mode 100644 python/docs/source/migration_guide/index.rst
create mode 100644 python/docs/source/reference/index.rst
create mode 100644 python/docs/source/reference/pyspark.ml.rst
create mode 100644 python/docs/source/reference/pyspark.mllib.rst
create mode 100644 python/docs/source/reference/pyspark.resource.rst
create mode 100644 python/docs/source/reference/pyspark.rst
create mode 100644 python/docs/source/reference/pyspark.sql.rst
create mode 100644 python/docs/source/reference/pyspark.ss.rst
create mode 100644 python/docs/source/reference/pyspark.streaming.rst
create mode 100644 python/docs/source/user_guide/index.rst
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (99f33ec -> 6ab29b3)
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 99f33ec [SPARK-32234][FOLLOWUP][SQL] Update the description of
utility method
add 6ab29b3 [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign
the documentation base
No new revisions were added by this update.
Summary of changes:
.github/workflows/master.yml | 8 +-
.gitignore | 1 +
LICENSE| 2 +-
dev/create-release/spark-rm/Dockerfile | 5 +-
dev/lint-python| 18 +-
dev/requirements.txt | 1 +
dev/tox.ini| 2 +-
docs/README.md | 7 +-
docs/_plugins/copy_api_dirs.rb | 4 +-
docs/img/spark-logo-reverse.png| Bin 0 -> 21547 bytes
python/docs/Makefile | 4 +-
python/docs/_static/pyspark.css| 90
python/docs/_static/pyspark.js | 99
python/docs/_templates/layout.html | 6 -
python/docs/index.rst | 53 --
python/docs/make2.bat | 4 +-
python/docs/pyspark.ml.rst | 122 -
python/docs/pyspark.mllib.rst | 99
python/docs/pyspark.resource.rst | 11 -
python/docs/pyspark.rst| 21 -
python/docs/pyspark.sql.rst| 37 --
python/docs/pyspark.streaming.rst | 17 -
python/docs/{ => source}/_static/copybutton.js | 0
python/docs/source/_static/css/pyspark.css | 94
python/docs/source/_templates/class_with_docs.rst | 79 +++
python/docs/{ => source}/conf.py | 51 +-
python/docs/source/development/index.rst | 21 +
python/docs/source/getting_started/index.rst | 22 +
python/docs/source/index.rst | 32 ++
python/docs/source/migration_guide/index.rst | 22 +
python/docs/source/reference/index.rst | 34 ++
python/docs/source/reference/pyspark.ml.rst| 363 ++
python/docs/source/reference/pyspark.mllib.rst | 253 ++
python/docs/source/reference/pyspark.resource.rst | 38 ++
python/docs/source/reference/pyspark.rst | 275 +++
python/docs/source/reference/pyspark.sql.rst | 542 +
python/docs/source/reference/pyspark.ss.rst| 90
python/docs/source/reference/pyspark.streaming.rst | 130 +
python/docs/source/user_guide/index.rst| 22 +
python/pyspark/sql/functions.py| 4 +
python/pyspark/sql/readwriter.py | 17 +
python/pyspark/sql/streaming.py| 13 +
42 files changed, 2129 insertions(+), 584 deletions(-)
create mode 100644 docs/img/spark-logo-reverse.png
delete mode 100644 python/docs/_static/pyspark.css
delete mode 100644 python/docs/_static/pyspark.js
delete mode 100644 python/docs/_templates/layout.html
delete mode 100644 python/docs/index.rst
delete mode 100644 python/docs/pyspark.ml.rst
delete mode 100644 python/docs/pyspark.mllib.rst
delete mode 100644 python/docs/pyspark.resource.rst
delete mode 100644 python/docs/pyspark.rst
delete mode 100644 python/docs/pyspark.sql.rst
delete mode 100644 python/docs/pyspark.streaming.rst
rename python/docs/{ => source}/_static/copybutton.js (100%)
create mode 100644 python/docs/source/_static/css/pyspark.css
create mode 100644 python/docs/source/_templates/class_with_docs.rst
rename python/docs/{ => source}/conf.py (89%)
create mode 100644 python/docs/source/development/index.rst
create mode 100644 python/docs/source/getting_started/index.rst
create mode 100644 python/docs/source/index.rst
create mode 100644 python/docs/source/migration_guide/index.rst
create mode 100644 python/docs/source/reference/index.rst
create mode 100644 python/docs/source/reference/pyspark.ml.rst
create mode 100644 python/docs/source/reference/pyspark.mllib.rst
create mode 100644 python/docs/source/reference/pyspark.resource.rst
create mode 100644 python/docs/source/reference/pyspark.rst
create mode 100644 python/docs/source/reference/pyspark.sql.rst
create mode 100644 python/docs/source/reference/pyspark.ss.rst
create mode 100644 python/docs/source/reference/pyspark.streaming.rst
create mode 100644 python/docs/source/user_guide/index.rst
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (99f33ec -> 6ab29b3)
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 99f33ec [SPARK-32234][FOLLOWUP][SQL] Update the description of
utility method
add 6ab29b3 [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign
the documentation base
No new revisions were added by this update.
Summary of changes:
.github/workflows/master.yml | 8 +-
.gitignore | 1 +
LICENSE| 2 +-
dev/create-release/spark-rm/Dockerfile | 5 +-
dev/lint-python| 18 +-
dev/requirements.txt | 1 +
dev/tox.ini| 2 +-
docs/README.md | 7 +-
docs/_plugins/copy_api_dirs.rb | 4 +-
docs/img/spark-logo-reverse.png| Bin 0 -> 21547 bytes
python/docs/Makefile | 4 +-
python/docs/_static/pyspark.css| 90
python/docs/_static/pyspark.js | 99
python/docs/_templates/layout.html | 6 -
python/docs/index.rst | 53 --
python/docs/make2.bat | 4 +-
python/docs/pyspark.ml.rst | 122 -
python/docs/pyspark.mllib.rst | 99
python/docs/pyspark.resource.rst | 11 -
python/docs/pyspark.rst| 21 -
python/docs/pyspark.sql.rst| 37 --
python/docs/pyspark.streaming.rst | 17 -
python/docs/{ => source}/_static/copybutton.js | 0
python/docs/source/_static/css/pyspark.css | 94
python/docs/source/_templates/class_with_docs.rst | 79 +++
python/docs/{ => source}/conf.py | 51 +-
python/docs/source/development/index.rst | 21 +
python/docs/source/getting_started/index.rst | 22 +
python/docs/source/index.rst | 32 ++
python/docs/source/migration_guide/index.rst | 22 +
python/docs/source/reference/index.rst | 34 ++
python/docs/source/reference/pyspark.ml.rst| 363 ++
python/docs/source/reference/pyspark.mllib.rst | 253 ++
python/docs/source/reference/pyspark.resource.rst | 38 ++
python/docs/source/reference/pyspark.rst | 275 +++
python/docs/source/reference/pyspark.sql.rst | 542 +
python/docs/source/reference/pyspark.ss.rst| 90
python/docs/source/reference/pyspark.streaming.rst | 130 +
python/docs/source/user_guide/index.rst| 22 +
python/pyspark/sql/functions.py| 4 +
python/pyspark/sql/readwriter.py | 17 +
python/pyspark/sql/streaming.py| 13 +
42 files changed, 2129 insertions(+), 584 deletions(-)
create mode 100644 docs/img/spark-logo-reverse.png
delete mode 100644 python/docs/_static/pyspark.css
delete mode 100644 python/docs/_static/pyspark.js
delete mode 100644 python/docs/_templates/layout.html
delete mode 100644 python/docs/index.rst
delete mode 100644 python/docs/pyspark.ml.rst
delete mode 100644 python/docs/pyspark.mllib.rst
delete mode 100644 python/docs/pyspark.resource.rst
delete mode 100644 python/docs/pyspark.rst
delete mode 100644 python/docs/pyspark.sql.rst
delete mode 100644 python/docs/pyspark.streaming.rst
rename python/docs/{ => source}/_static/copybutton.js (100%)
create mode 100644 python/docs/source/_static/css/pyspark.css
create mode 100644 python/docs/source/_templates/class_with_docs.rst
rename python/docs/{ => source}/conf.py (89%)
create mode 100644 python/docs/source/development/index.rst
create mode 100644 python/docs/source/getting_started/index.rst
create mode 100644 python/docs/source/index.rst
create mode 100644 python/docs/source/migration_guide/index.rst
create mode 100644 python/docs/source/reference/index.rst
create mode 100644 python/docs/source/reference/pyspark.ml.rst
create mode 100644 python/docs/source/reference/pyspark.mllib.rst
create mode 100644 python/docs/source/reference/pyspark.resource.rst
create mode 100644 python/docs/source/reference/pyspark.rst
create mode 100644 python/docs/source/reference/pyspark.sql.rst
create mode 100644 python/docs/source/reference/pyspark.ss.rst
create mode 100644 python/docs/source/reference/pyspark.streaming.rst
create mode 100644 python/docs/source/user_guide/index.rst
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-32435][PYTHON] Remove heapq3 port from Python 3
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new a82aee0 [SPARK-32435][PYTHON] Remove heapq3 port from Python 3 a82aee0 is described below commit a82aee044127825ffefa0ed09b0ae5b987b9dd21 Author: HyukjinKwon AuthorDate: Mon Jul 27 20:10:13 2020 +0900 [SPARK-32435][PYTHON] Remove heapq3 port from Python 3 ### What changes were proposed in this pull request? This PR removes the manual port of `heapq3.py` introduced from SPARK-3073. The main reason of this was to support Python 2.6 and 2.7 because Python 2's `heapq.merge()` doesn't not support `key` and `reverse`. See - https://docs.python.org/2/library/heapq.html#heapq.merge in Python 2 - https://docs.python.org/3.8/library/heapq.html#heapq.merge in Python 3 Since we dropped the Python 2 at SPARK-32138, we can remove this away. ### Why are the changes needed? To remove unnecessary codes. Also, we can leverage bug fixes made in Python 3.x at `heapq`. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Existing tests should cover. I locally ran and verified: ```bash ./python/run-tests --python-executable=python3 --testname="pyspark.tests.test_shuffle" ./python/run-tests --python-executable=python3 --testname="pyspark.shuffle ExternalSorter" ./python/run-tests --python-executable=python3 --testname="pyspark.tests.test_rdd RDDTests.test_external_group_by_key" ``` Closes #29229 from HyukjinKwon/SPARK-32435. Authored-by: HyukjinKwon Signed-off-by: HyukjinKwon --- LICENSE | 1 - LICENSE-binary| 6 - dev/.rat-excludes | 1 - dev/tox.ini | 2 +- licenses-binary/LICENSE-heapq.txt | 280 licenses/LICENSE-heapq.txt| 49 --- python/pylintrc | 2 +- python/pyspark/heapq3.py | 890 -- python/pyspark/shuffle.py | 6 +- 9 files changed, 5 insertions(+), 1232 deletions(-) diff --git a/LICENSE b/LICENSE index 8cec4f5..df6bed1 100644 --- a/LICENSE +++ b/LICENSE @@ -222,7 +222,6 @@ external/spark-ganglia-lgpl/src/main/java/com/codahale/metrics/ganglia/GangliaRe Python Software Foundation License -- -pyspark/heapq3.py python/docs/source/_static/copybutton.js BSD 3-Clause diff --git a/LICENSE-binary b/LICENSE-binary index b50da6b..d363661 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -557,12 +557,6 @@ jakarta.ws.rs:jakarta.ws.rs-api https://github.com/eclipse-ee4j/jaxrs-api org.glassfish.hk2.external:jakarta.inject -Python Software Foundation License --- - -pyspark/heapq3.py - - Public Domain - diff --git a/dev/.rat-excludes b/dev/.rat-excludes index db6a4ce..3889dc9 100644 --- a/dev/.rat-excludes +++ b/dev/.rat-excludes @@ -49,7 +49,6 @@ jsonFormatter.min.js .*log pyspark-coverage-site/* cloudpickle/* -heapq3.py join.py SparkExprTyper.scala SparkILoop.scala diff --git a/dev/tox.ini b/dev/tox.ini index e25595a..5bf27d1 100644 --- a/dev/tox.ini +++ b/dev/tox.ini @@ -16,4 +16,4 @@ [pycodestyle] ignore=E226,E241,E305,E402,E722,E731,E741,W503,W504 max-line-length=100 -exclude=python/pyspark/cloudpickle/*.py,heapq3.py,shared.py,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/* +exclude=python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/* diff --git a/licenses-binary/LICENSE-heapq.txt b/licenses-binary/LICENSE-heapq.txt deleted file mode 100644 index 0c4c4b9..000 --- a/licenses-binary/LICENSE-heapq.txt +++ /dev/null @@ -1,280 +0,0 @@ - -# A. HISTORY OF THE SOFTWARE -# == -# -# Python was created in the early 1990s by Guido van Rossum at Stichting -# Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands -# as a successor of a language called ABC. Guido remains Python's -# principal author, although it includes many contributions from others. -# -# In 1995, Guido continued his work on Python at the Corporation for -# National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) -# in Reston, Virginia where he released several versions of the -# software. -# -# In May 2000, Guido and the Python core development team moved to -# BeOpen.com to form the BeOpen PythonLabs team. In October of the same -# year, the PythonLabs team moved to Digital Creations (now Zope -# Corporation, see http://www.zope.com). In 2001, the Python Software -# Foundation (PSF, see http://www.python.org/psf/) was formed, a -# non-profit organization created specifically to own Python-rel
[spark] branch master updated (6ab29b3 -> a82aee0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6ab29b3 [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base add a82aee0 [SPARK-32435][PYTHON] Remove heapq3 port from Python 3 No new revisions were added by this update. Summary of changes: LICENSE | 1 - LICENSE-binary| 6 - dev/.rat-excludes | 1 - dev/tox.ini | 2 +- licenses-binary/LICENSE-heapq.txt | 280 licenses/LICENSE-heapq.txt| 49 --- python/pylintrc | 2 +- python/pyspark/heapq3.py | 890 -- python/pyspark/shuffle.py | 6 +- 9 files changed, 5 insertions(+), 1232 deletions(-) delete mode 100644 licenses-binary/LICENSE-heapq.txt delete mode 100644 licenses/LICENSE-heapq.txt delete mode 100644 python/pyspark/heapq3.py - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6ab29b3 -> a82aee0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6ab29b3 [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base add a82aee0 [SPARK-32435][PYTHON] Remove heapq3 port from Python 3 No new revisions were added by this update. Summary of changes: LICENSE | 1 - LICENSE-binary| 6 - dev/.rat-excludes | 1 - dev/tox.ini | 2 +- licenses-binary/LICENSE-heapq.txt | 280 licenses/LICENSE-heapq.txt| 49 --- python/pylintrc | 2 +- python/pyspark/heapq3.py | 890 -- python/pyspark/shuffle.py | 6 +- 9 files changed, 5 insertions(+), 1232 deletions(-) delete mode 100644 licenses-binary/LICENSE-heapq.txt delete mode 100644 licenses/LICENSE-heapq.txt delete mode 100644 python/pyspark/heapq3.py - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6ab29b3 -> a82aee0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6ab29b3 [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base add a82aee0 [SPARK-32435][PYTHON] Remove heapq3 port from Python 3 No new revisions were added by this update. Summary of changes: LICENSE | 1 - LICENSE-binary| 6 - dev/.rat-excludes | 1 - dev/tox.ini | 2 +- licenses-binary/LICENSE-heapq.txt | 280 licenses/LICENSE-heapq.txt| 49 --- python/pylintrc | 2 +- python/pyspark/heapq3.py | 890 -- python/pyspark/shuffle.py | 6 +- 9 files changed, 5 insertions(+), 1232 deletions(-) delete mode 100644 licenses-binary/LICENSE-heapq.txt delete mode 100644 licenses/LICENSE-heapq.txt delete mode 100644 python/pyspark/heapq3.py - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6ab29b3 -> a82aee0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6ab29b3 [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base add a82aee0 [SPARK-32435][PYTHON] Remove heapq3 port from Python 3 No new revisions were added by this update. Summary of changes: LICENSE | 1 - LICENSE-binary| 6 - dev/.rat-excludes | 1 - dev/tox.ini | 2 +- licenses-binary/LICENSE-heapq.txt | 280 licenses/LICENSE-heapq.txt| 49 --- python/pylintrc | 2 +- python/pyspark/heapq3.py | 890 -- python/pyspark/shuffle.py | 6 +- 9 files changed, 5 insertions(+), 1232 deletions(-) delete mode 100644 licenses-binary/LICENSE-heapq.txt delete mode 100644 licenses/LICENSE-heapq.txt delete mode 100644 python/pyspark/heapq3.py - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-30794][CORE] Stage Level scheduling: Add ability to set off heap memory
This is an automated email from the ASF dual-hosted git repository.
tgraves pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 998086c [SPARK-30794][CORE] Stage Level scheduling: Add ability to
set off heap memory
998086c is described below
commit 998086c9a179692b2687bc9a104dbbb35f5a44e2
Author: Warren Zhu
AuthorDate: Mon Jul 27 08:16:13 2020 -0500
[SPARK-30794][CORE] Stage Level scheduling: Add ability to set off heap
memory
### What changes were proposed in this pull request?
Support set off heap memory in `ExecutorResourceRequests`
### Why are the changes needed?
Support stage level scheduling
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added UT in `ResourceProfileSuite` and `DAGSchedulerSuite`
Closes #28972 from warrenzhu25/30794.
Authored-by: Warren Zhu
Signed-off-by: Thomas Graves
---
.../spark/resource/ExecutorResourceRequests.scala | 14 ++
.../apache/spark/resource/ResourceProfile.scala| 8 +++-
.../spark/resource/ResourceProfileSuite.scala | 50 +++---
.../apache/spark/scheduler/DAGSchedulerSuite.scala | 5 ++-
python/pyspark/resource/requests.py| 9
python/pyspark/resource/tests/test_resources.py| 5 ++-
.../apache/spark/deploy/yarn/YarnAllocator.scala | 3 +-
.../spark/deploy/yarn/YarnSparkHadoopUtil.scala| 23 --
8 files changed, 102 insertions(+), 15 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/resource/ExecutorResourceRequests.scala
b/core/src/main/scala/org/apache/spark/resource/ExecutorResourceRequests.scala
index 9da6ffb..654afa0 100644
---
a/core/src/main/scala/org/apache/spark/resource/ExecutorResourceRequests.scala
+++
b/core/src/main/scala/org/apache/spark/resource/ExecutorResourceRequests.scala
@@ -55,6 +55,20 @@ class ExecutorResourceRequests() extends Serializable {
}
/**
+ * Specify off heap memory. The value specified will be converted to MiB.
+ * This value only take effect when MEMORY_OFFHEAP_ENABLED is true.
+ *
+ * @param amount Amount of memory. In the same format as JVM memory strings
(e.g. 512m, 2g).
+ * Default unit is MiB if not specified.
+ */
+ def offHeapMemory(amount: String): this.type = {
+val amountMiB = JavaUtils.byteStringAsMb(amount)
+val req = new ExecutorResourceRequest(OFFHEAP_MEM, amountMiB)
+_executorResources.put(OFFHEAP_MEM, req)
+this
+ }
+
+ /**
* Specify overhead memory. The value specified will be converted to MiB.
*
* @param amount Amount of memory. In the same format as JVM memory strings
(e.g. 512m, 2g).
diff --git
a/core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
b/core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
index f56ea69..8a37670 100644
--- a/core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
+++ b/core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
@@ -243,13 +243,15 @@ object ResourceProfile extends Logging {
// task resources
val CPUS = "cpus"
// Executor resources
+ // Make sure add new executor resource in below allSupportedExecutorResources
val CORES = "cores"
val MEMORY = "memory"
+ val OFFHEAP_MEM = "offHeap"
val OVERHEAD_MEM = "memoryOverhead"
val PYSPARK_MEM = "pyspark.memory"
// all supported spark executor resources (minus the custom resources like
GPUs/FPGAs)
- val allSupportedExecutorResources = Seq(CORES, MEMORY, OVERHEAD_MEM,
PYSPARK_MEM)
+ val allSupportedExecutorResources = Seq(CORES, MEMORY, OVERHEAD_MEM,
PYSPARK_MEM, OFFHEAP_MEM)
val UNKNOWN_RESOURCE_PROFILE_ID = -1
val DEFAULT_RESOURCE_PROFILE_ID = 0
@@ -295,6 +297,10 @@ object ResourceProfile extends Logging {
ereqs.memory(conf.get(EXECUTOR_MEMORY).toString)
conf.get(EXECUTOR_MEMORY_OVERHEAD).map(mem =>
ereqs.memoryOverhead(mem.toString))
conf.get(PYSPARK_EXECUTOR_MEMORY).map(mem =>
ereqs.pysparkMemory(mem.toString))
+if (conf.get(MEMORY_OFFHEAP_ENABLED)) {
+ // Explicitly add suffix b as default unit of offHeapMemory is Mib
+ ereqs.offHeapMemory(conf.get(MEMORY_OFFHEAP_SIZE).toString + "b")
+}
val execReq = ResourceUtils.parseAllResourceRequests(conf,
SPARK_EXECUTOR_PREFIX)
execReq.foreach { req =>
val name = req.id.resourceName
diff --git
a/core/src/test/scala/org/apache/spark/resource/ResourceProfileSuite.scala
b/core/src/test/scala/org/apache/spark/resource/ResourceProfileSuite.scala
index 29d3ef1..d0479ca 100644
--- a/core/src/test/scala/org/apache/spark/resource/ResourceProfileSuite.scala
+++ b/core/src/test/scala/org/apache/spark/resource/ResourceProfileSuite.scala
@@ -18,7 +18,7 @@
package org.apache.spark.resource
import org.apache.spark.{SparkConf, SparkFunSuite}
-import org.apa
[spark] branch master updated (998086c -> ea58e52)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 998086c [SPARK-30794][CORE] Stage Level scheduling: Add ability to set off heap memory add ea58e52 [SPARK-32434][CORE][FOLLOW-UP] Fix load-spark-env.cmd to be able to run in Windows properly No new revisions were added by this update. Summary of changes: bin/load-spark-env.cmd | 23 --- 1 file changed, 12 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (998086c -> ea58e52)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 998086c [SPARK-30794][CORE] Stage Level scheduling: Add ability to set off heap memory add ea58e52 [SPARK-32434][CORE][FOLLOW-UP] Fix load-spark-env.cmd to be able to run in Windows properly No new revisions were added by this update. Summary of changes: bin/load-spark-env.cmd | 23 --- 1 file changed, 12 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (998086c -> ea58e52)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 998086c [SPARK-30794][CORE] Stage Level scheduling: Add ability to set off heap memory add ea58e52 [SPARK-32434][CORE][FOLLOW-UP] Fix load-spark-env.cmd to be able to run in Windows properly No new revisions were added by this update. Summary of changes: bin/load-spark-env.cmd | 23 --- 1 file changed, 12 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (998086c -> ea58e52)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 998086c [SPARK-30794][CORE] Stage Level scheduling: Add ability to set off heap memory add ea58e52 [SPARK-32434][CORE][FOLLOW-UP] Fix load-spark-env.cmd to be able to run in Windows properly No new revisions were added by this update. Summary of changes: bin/load-spark-env.cmd | 23 --- 1 file changed, 12 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (998086c -> ea58e52)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 998086c [SPARK-30794][CORE] Stage Level scheduling: Add ability to set off heap memory add ea58e52 [SPARK-32434][CORE][FOLLOW-UP] Fix load-spark-env.cmd to be able to run in Windows properly No new revisions were added by this update. Summary of changes: bin/load-spark-env.cmd | 23 --- 1 file changed, 12 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ea58e52 -> 548b7db)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ea58e52 [SPARK-32434][CORE][FOLLOW-UP] Fix load-spark-env.cmd to be able to run in Windows properly add 548b7db [SPARK-32420][SQL] Add handling for unique key in non-codegen hash join No new revisions were added by this update. Summary of changes: .../spark/sql/execution/joins/HashJoin.scala | 154 ++--- .../sql/execution/joins/ExistenceJoinSuite.scala | 10 +- .../spark/sql/execution/joins/OuterJoinSuite.scala | 63 - 3 files changed, 177 insertions(+), 50 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ea58e52 -> 548b7db)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ea58e52 [SPARK-32434][CORE][FOLLOW-UP] Fix load-spark-env.cmd to be able to run in Windows properly add 548b7db [SPARK-32420][SQL] Add handling for unique key in non-codegen hash join No new revisions were added by this update. Summary of changes: .../spark/sql/execution/joins/HashJoin.scala | 154 ++--- .../sql/execution/joins/ExistenceJoinSuite.scala | 10 +- .../spark/sql/execution/joins/OuterJoinSuite.scala | 63 - 3 files changed, 177 insertions(+), 50 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (548b7db -> d315ebf)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 548b7db [SPARK-32420][SQL] Add handling for unique key in non-codegen hash join add d315ebf [SPARK-32424][SQL] Fix silent data change for timestamp parsing if overflow happens No new revisions were added by this update. Summary of changes: docs/sql-ref-datetime-pattern.md | 2 +- .../catalyst/util/DateTimeFormatterHelper.scala| 4 +- .../sql/catalyst/util/TimestampFormatter.scala | 3 +- .../catalyst/util/TimestampFormatterSuite.scala| 10 +++ .../inputs/datetime-formatting-invalid.sql | 4 +- .../sql-tests/inputs/datetime-formatting.sql | 2 +- .../sql-tests/inputs/datetime-parsing-invalid.sql | 9 ++ .../sql-tests/inputs/datetime-parsing.sql | 28 +++ .../results/datetime-formatting-invalid.sql.out| 4 +- .../results/datetime-formatting-legacy.sql.out | 18 ++-- .../sql-tests/results/datetime-formatting.sql.out | 18 ++-- .../results/datetime-parsing-invalid.sql.out | 55 +++- .../results/datetime-parsing-legacy.sql.out| 98 +- .../sql-tests/results/datetime-parsing.sql.out | 98 +- 14 files changed, 323 insertions(+), 30 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ea58e52 -> 548b7db)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ea58e52 [SPARK-32434][CORE][FOLLOW-UP] Fix load-spark-env.cmd to be able to run in Windows properly add 548b7db [SPARK-32420][SQL] Add handling for unique key in non-codegen hash join No new revisions were added by this update. Summary of changes: .../spark/sql/execution/joins/HashJoin.scala | 154 ++--- .../sql/execution/joins/ExistenceJoinSuite.scala | 10 +- .../spark/sql/execution/joins/OuterJoinSuite.scala | 63 - 3 files changed, 177 insertions(+), 50 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (548b7db -> d315ebf)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 548b7db [SPARK-32420][SQL] Add handling for unique key in non-codegen hash join add d315ebf [SPARK-32424][SQL] Fix silent data change for timestamp parsing if overflow happens No new revisions were added by this update. Summary of changes: docs/sql-ref-datetime-pattern.md | 2 +- .../catalyst/util/DateTimeFormatterHelper.scala| 4 +- .../sql/catalyst/util/TimestampFormatter.scala | 3 +- .../catalyst/util/TimestampFormatterSuite.scala| 10 +++ .../inputs/datetime-formatting-invalid.sql | 4 +- .../sql-tests/inputs/datetime-formatting.sql | 2 +- .../sql-tests/inputs/datetime-parsing-invalid.sql | 9 ++ .../sql-tests/inputs/datetime-parsing.sql | 28 +++ .../results/datetime-formatting-invalid.sql.out| 4 +- .../results/datetime-formatting-legacy.sql.out | 18 ++-- .../sql-tests/results/datetime-formatting.sql.out | 18 ++-- .../results/datetime-parsing-invalid.sql.out | 55 +++- .../results/datetime-parsing-legacy.sql.out| 98 +- .../sql-tests/results/datetime-parsing.sql.out | 98 +- 14 files changed, 323 insertions(+), 30 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ea58e52 -> 548b7db)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ea58e52 [SPARK-32434][CORE][FOLLOW-UP] Fix load-spark-env.cmd to be able to run in Windows properly add 548b7db [SPARK-32420][SQL] Add handling for unique key in non-codegen hash join No new revisions were added by this update. Summary of changes: .../spark/sql/execution/joins/HashJoin.scala | 154 ++--- .../sql/execution/joins/ExistenceJoinSuite.scala | 10 +- .../spark/sql/execution/joins/OuterJoinSuite.scala | 63 - 3 files changed, 177 insertions(+), 50 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (548b7db -> d315ebf)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 548b7db [SPARK-32420][SQL] Add handling for unique key in non-codegen hash join add d315ebf [SPARK-32424][SQL] Fix silent data change for timestamp parsing if overflow happens No new revisions were added by this update. Summary of changes: docs/sql-ref-datetime-pattern.md | 2 +- .../catalyst/util/DateTimeFormatterHelper.scala| 4 +- .../sql/catalyst/util/TimestampFormatter.scala | 3 +- .../catalyst/util/TimestampFormatterSuite.scala| 10 +++ .../inputs/datetime-formatting-invalid.sql | 4 +- .../sql-tests/inputs/datetime-formatting.sql | 2 +- .../sql-tests/inputs/datetime-parsing-invalid.sql | 9 ++ .../sql-tests/inputs/datetime-parsing.sql | 28 +++ .../results/datetime-formatting-invalid.sql.out| 4 +- .../results/datetime-formatting-legacy.sql.out | 18 ++-- .../sql-tests/results/datetime-formatting.sql.out | 18 ++-- .../results/datetime-parsing-invalid.sql.out | 55 +++- .../results/datetime-parsing-legacy.sql.out| 98 +- .../sql-tests/results/datetime-parsing.sql.out | 98 +- 14 files changed, 323 insertions(+), 30 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ea58e52 -> 548b7db)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ea58e52 [SPARK-32434][CORE][FOLLOW-UP] Fix load-spark-env.cmd to be able to run in Windows properly add 548b7db [SPARK-32420][SQL] Add handling for unique key in non-codegen hash join No new revisions were added by this update. Summary of changes: .../spark/sql/execution/joins/HashJoin.scala | 154 ++--- .../sql/execution/joins/ExistenceJoinSuite.scala | 10 +- .../spark/sql/execution/joins/OuterJoinSuite.scala | 63 - 3 files changed, 177 insertions(+), 50 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (548b7db -> d315ebf)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 548b7db [SPARK-32420][SQL] Add handling for unique key in non-codegen hash join add d315ebf [SPARK-32424][SQL] Fix silent data change for timestamp parsing if overflow happens No new revisions were added by this update. Summary of changes: docs/sql-ref-datetime-pattern.md | 2 +- .../catalyst/util/DateTimeFormatterHelper.scala| 4 +- .../sql/catalyst/util/TimestampFormatter.scala | 3 +- .../catalyst/util/TimestampFormatterSuite.scala| 10 +++ .../inputs/datetime-formatting-invalid.sql | 4 +- .../sql-tests/inputs/datetime-formatting.sql | 2 +- .../sql-tests/inputs/datetime-parsing-invalid.sql | 9 ++ .../sql-tests/inputs/datetime-parsing.sql | 28 +++ .../results/datetime-formatting-invalid.sql.out| 4 +- .../results/datetime-formatting-legacy.sql.out | 18 ++-- .../sql-tests/results/datetime-formatting.sql.out | 18 ++-- .../results/datetime-parsing-invalid.sql.out | 55 +++- .../results/datetime-parsing-legacy.sql.out| 98 +- .../sql-tests/results/datetime-parsing.sql.out | 98 +- 14 files changed, 323 insertions(+), 30 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (548b7db -> d315ebf)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 548b7db [SPARK-32420][SQL] Add handling for unique key in non-codegen hash join add d315ebf [SPARK-32424][SQL] Fix silent data change for timestamp parsing if overflow happens No new revisions were added by this update. Summary of changes: docs/sql-ref-datetime-pattern.md | 2 +- .../catalyst/util/DateTimeFormatterHelper.scala| 4 +- .../sql/catalyst/util/TimestampFormatter.scala | 3 +- .../catalyst/util/TimestampFormatterSuite.scala| 10 +++ .../inputs/datetime-formatting-invalid.sql | 4 +- .../sql-tests/inputs/datetime-formatting.sql | 2 +- .../sql-tests/inputs/datetime-parsing-invalid.sql | 9 ++ .../sql-tests/inputs/datetime-parsing.sql | 28 +++ .../results/datetime-formatting-invalid.sql.out| 4 +- .../results/datetime-formatting-legacy.sql.out | 18 ++-- .../sql-tests/results/datetime-formatting.sql.out | 18 ++-- .../results/datetime-parsing-invalid.sql.out | 55 +++- .../results/datetime-parsing-legacy.sql.out| 98 +- .../sql-tests/results/datetime-parsing.sql.out | 98 +- 14 files changed, 323 insertions(+), 30 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d315ebf -> c114066)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d315ebf [SPARK-32424][SQL] Fix silent data change for timestamp parsing if overflow happens add c114066 [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/TestUtils.scala | 8 +++- core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala | 6 -- .../scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala | 3 ++- 3 files changed, 13 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d315ebf -> c114066)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d315ebf [SPARK-32424][SQL] Fix silent data change for timestamp parsing if overflow happens add c114066 [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/TestUtils.scala | 8 +++- core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala | 6 -- .../scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala | 3 ++- 3 files changed, 13 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c114066 -> f7542d3)
This is an automated email from the ASF dual-hosted git repository. huaxingao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c114066 [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable add f7542d3 [SPARK-32457][ML] logParam thresholds in DT/GBT/FM/LR/MLP No new revisions were added by this update. Summary of changes: .../org/apache/spark/ml/classification/DecisionTreeClassifier.scala | 2 +- .../scala/org/apache/spark/ml/classification/FMClassifier.scala | 6 ++ .../scala/org/apache/spark/ml/classification/GBTClassifier.scala| 2 +- .../org/apache/spark/ml/classification/LogisticRegression.scala | 4 ++-- .../spark/ml/classification/MultilayerPerceptronClassifier.scala| 2 +- 5 files changed, 7 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d315ebf -> c114066)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d315ebf [SPARK-32424][SQL] Fix silent data change for timestamp parsing if overflow happens add c114066 [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/TestUtils.scala | 8 +++- core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala | 6 -- .../scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala | 3 ++- 3 files changed, 13 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c114066 -> f7542d3)
This is an automated email from the ASF dual-hosted git repository. huaxingao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c114066 [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable add f7542d3 [SPARK-32457][ML] logParam thresholds in DT/GBT/FM/LR/MLP No new revisions were added by this update. Summary of changes: .../org/apache/spark/ml/classification/DecisionTreeClassifier.scala | 2 +- .../scala/org/apache/spark/ml/classification/FMClassifier.scala | 6 ++ .../scala/org/apache/spark/ml/classification/GBTClassifier.scala| 2 +- .../org/apache/spark/ml/classification/LogisticRegression.scala | 4 ++-- .../spark/ml/classification/MultilayerPerceptronClassifier.scala| 2 +- 5 files changed, 7 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d315ebf -> c114066)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d315ebf [SPARK-32424][SQL] Fix silent data change for timestamp parsing if overflow happens add c114066 [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/TestUtils.scala | 8 +++- core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala | 6 -- .../scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala | 3 ++- 3 files changed, 13 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c114066 -> f7542d3)
This is an automated email from the ASF dual-hosted git repository. huaxingao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c114066 [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable add f7542d3 [SPARK-32457][ML] logParam thresholds in DT/GBT/FM/LR/MLP No new revisions were added by this update. Summary of changes: .../org/apache/spark/ml/classification/DecisionTreeClassifier.scala | 2 +- .../scala/org/apache/spark/ml/classification/FMClassifier.scala | 6 ++ .../scala/org/apache/spark/ml/classification/GBTClassifier.scala| 2 +- .../org/apache/spark/ml/classification/LogisticRegression.scala | 4 ++-- .../spark/ml/classification/MultilayerPerceptronClassifier.scala| 2 +- 5 files changed, 7 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d315ebf -> c114066)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d315ebf [SPARK-32424][SQL] Fix silent data change for timestamp parsing if overflow happens add c114066 [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/TestUtils.scala | 8 +++- core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala | 6 -- .../scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala | 3 ++- 3 files changed, 13 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c114066 -> f7542d3)
This is an automated email from the ASF dual-hosted git repository. huaxingao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c114066 [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable add f7542d3 [SPARK-32457][ML] logParam thresholds in DT/GBT/FM/LR/MLP No new revisions were added by this update. Summary of changes: .../org/apache/spark/ml/classification/DecisionTreeClassifier.scala | 2 +- .../scala/org/apache/spark/ml/classification/FMClassifier.scala | 6 ++ .../scala/org/apache/spark/ml/classification/GBTClassifier.scala| 2 +- .../org/apache/spark/ml/classification/LogisticRegression.scala | 4 ++-- .../spark/ml/classification/MultilayerPerceptronClassifier.scala| 2 +- 5 files changed, 7 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c114066 -> f7542d3)
This is an automated email from the ASF dual-hosted git repository. huaxingao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c114066 [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable add f7542d3 [SPARK-32457][ML] logParam thresholds in DT/GBT/FM/LR/MLP No new revisions were added by this update. Summary of changes: .../org/apache/spark/ml/classification/DecisionTreeClassifier.scala | 2 +- .../scala/org/apache/spark/ml/classification/FMClassifier.scala | 6 ++ .../scala/org/apache/spark/ml/classification/GBTClassifier.scala| 2 +- .../org/apache/spark/ml/classification/LogisticRegression.scala | 4 ++-- .../spark/ml/classification/MultilayerPerceptronClassifier.scala| 2 +- 5 files changed, 7 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f7542d3 -> 8de4333)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f7542d3 [SPARK-32457][ML] logParam thresholds in DT/GBT/FM/LR/MLP add 8de4333 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs No new revisions were added by this update. Summary of changes: docs/_data/menu-sql.yaml | 6 + docs/sql-ref-syntax-ddl-create-table-hiveformat.md | 94 +++- docs/sql-ref-syntax-qry-select-case.md | 109 ++ docs/sql-ref-syntax-qry-select-clusterby.md| 3 + docs/sql-ref-syntax-qry-select-distribute-by.md| 3 + docs/sql-ref-syntax-qry-select-groupby.md | 27 + docs/sql-ref-syntax-qry-select-having.md | 3 + docs/sql-ref-syntax-qry-select-lateral-view.md | 125 + docs/sql-ref-syntax-qry-select-limit.md| 3 + docs/sql-ref-syntax-qry-select-orderby.md | 3 + docs/sql-ref-syntax-qry-select-pivot.md| 101 + docs/sql-ref-syntax-qry-select-sortby.md | 3 + docs/sql-ref-syntax-qry-select-where.md| 3 + docs/sql-ref-syntax-qry-select.md | 56 + docs/sql-ref-syntax-qry.md | 3 + docs/sql-ref-syntax.md | 3 + 16 files changed, 520 insertions(+), 25 deletions(-) create mode 100644 docs/sql-ref-syntax-qry-select-case.md create mode 100644 docs/sql-ref-syntax-qry-select-lateral-view.md create mode 100644 docs/sql-ref-syntax-qry-select-pivot.md - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f7542d3 -> 8de4333)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f7542d3 [SPARK-32457][ML] logParam thresholds in DT/GBT/FM/LR/MLP add 8de4333 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs No new revisions were added by this update. Summary of changes: docs/_data/menu-sql.yaml | 6 + docs/sql-ref-syntax-ddl-create-table-hiveformat.md | 94 +++- docs/sql-ref-syntax-qry-select-case.md | 109 ++ docs/sql-ref-syntax-qry-select-clusterby.md| 3 + docs/sql-ref-syntax-qry-select-distribute-by.md| 3 + docs/sql-ref-syntax-qry-select-groupby.md | 27 + docs/sql-ref-syntax-qry-select-having.md | 3 + docs/sql-ref-syntax-qry-select-lateral-view.md | 125 + docs/sql-ref-syntax-qry-select-limit.md| 3 + docs/sql-ref-syntax-qry-select-orderby.md | 3 + docs/sql-ref-syntax-qry-select-pivot.md| 101 + docs/sql-ref-syntax-qry-select-sortby.md | 3 + docs/sql-ref-syntax-qry-select-where.md| 3 + docs/sql-ref-syntax-qry-select.md | 56 + docs/sql-ref-syntax-qry.md | 3 + docs/sql-ref-syntax.md | 3 + 16 files changed, 520 insertions(+), 25 deletions(-) create mode 100644 docs/sql-ref-syntax-qry-select-case.md create mode 100644 docs/sql-ref-syntax-qry-select-lateral-view.md create mode 100644 docs/sql-ref-syntax-qry-select-pivot.md - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 6ed93c3 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs 6ed93c3 is described below commit 6ed93c3e86c60323328b44cab45faa9ae3050dab Author: GuoPhilipse AuthorDate: Tue Jul 28 09:41:53 2020 +0900 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs ### What changes were proposed in this pull request? update sql-ref docs, the following key words will be added in this PR. CASE/ELSE WHEN/THEN MAP KEYS TERMINATED BY NULL DEFINED AS LINES TERMINATED BY ESCAPED BY COLLECTION ITEMS TERMINATED BY PIVOT LATERAL VIEW OUTER? ROW FORMAT SERDE ROW FORMAT DELIMITED FIELDS TERMINATED BY IGNORE NULLS FIRST LAST ### Why are the changes needed? let more users know the sql key words usage ### Does this PR introduce _any_ user-facing change?     ### How was this patch tested? No Closes #29056 from GuoPhilipse/add-missing-keywords. Lead-authored-by: GuoPhilipse Co-authored-by: GuoPhilipse <46367746+guophili...@users.noreply.github.com> Signed-off-by: Takeshi Yamamuro (cherry picked from commit 8de43338be879f0cfeebca328dbbcfd1e5bd70da) Signed-off-by: Takeshi Yamamuro --- docs/_data/menu-sql.yaml | 6 + docs/sql-ref-syntax-ddl-create-table-hiveformat.md | 94 +++- docs/sql-ref-syntax-qry-select-case.md | 109 ++ docs/sql-ref-syntax-qry-select-clusterby.md| 3 + docs/sql-ref-syntax-qry-select-distribute-by.md| 3 + docs/sql-ref-syntax-qry-select-groupby.md | 27 + docs/sql-ref-syntax-qry-select-having.md | 3 + docs/sql-ref-syntax-qry-select-lateral-view.md | 125 + docs/sql-ref-syntax-qry-select-limit.md| 3 + docs/sql-ref-syntax-qry-select-orderby.md | 3 + docs/sql-ref-syntax-qry-select-pivot.md| 101 + docs/sql-ref-syntax-qry-select-sortby.md | 3 + docs/sql-ref-syntax-qry-select-where.md| 3 + docs/sql-ref-syntax-qry-select.md | 56 + docs/sql-ref-syntax-qry.md | 3 + docs/sql-ref-syntax.md | 3 + 16 files changed, 520 insertions(+), 25 deletions(-) diff --git a/docs/_data/menu-sql.yaml b/docs/_data/menu-sql.yaml index eea657e..22fae0c 100644 --- a/docs/_data/menu-sql.yaml +++ b/docs/_data/menu-sql.yaml @@ -187,6 +187,12 @@ url: sql-ref-syntax-qry-select-tvf.html - text: Window Function url: sql-ref-syntax-qry-select-window.html +- text: CASE Clause + url: sql-ref-syntax-qry-select-case.html +- text: LATERAL VIEW Clause + url: sql-ref-syntax-qry-select-lateral-view.html +- text: PIVOT Clause + url: sql-ref-syntax-qry-select-pivot.html - text: EXPLAIN url: sql-ref-syntax-qry-explain.html - text: Auxiliary Statements diff --git a/docs/sql-ref-syntax-ddl-create-table-hiveformat.md b/docs/sql-ref-syntax-ddl-create-table-hiveformat.md index 38f8856..7bf847d 100644 --- a/docs/sql-ref-syntax-ddl-create-table-hiveformat.md +++ b/docs/sql-ref-syntax-ddl-create-table-hiveformat.md @@ -36,6 +36,14 @@ CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ] table_identifier [ LOCATION path ] [ TBLPROPERTIES ( key1=val1, key2=val2, ... ) ] [ AS select_statement ] + +row_format: +: SERDE serde_class [ WITH SERDEPROPERTIES (k1=v1, k2=v2, ... ) ] +| DELIMITED [ FIELDS TERMINATED BY fields_termiated_char [ ESCAPED BY escaped_char ] ] +[ COLLECTION ITEMS TERMINATED BY collection_items_termiated_char ] +[ MAP KEYS TERMINATED BY map_key_termiated_char ] +[ LINES TERMINATED BY row_termiated_char ] +[ NULL DEFINED AS null_char ] ``` Note that, the clauses between the columns definition clause and the AS SELECT clause can come in @@ -51,15 +59,55 @@ as any order. For example, you can write COMMENT table_comment after TBLPROPERTI * **EXTERNAL** -Table is defined using the path provided as LOCATION, does not use default loc
[spark] branch master updated (f7542d3 -> 8de4333)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f7542d3 [SPARK-32457][ML] logParam thresholds in DT/GBT/FM/LR/MLP add 8de4333 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs No new revisions were added by this update. Summary of changes: docs/_data/menu-sql.yaml | 6 + docs/sql-ref-syntax-ddl-create-table-hiveformat.md | 94 +++- docs/sql-ref-syntax-qry-select-case.md | 109 ++ docs/sql-ref-syntax-qry-select-clusterby.md| 3 + docs/sql-ref-syntax-qry-select-distribute-by.md| 3 + docs/sql-ref-syntax-qry-select-groupby.md | 27 + docs/sql-ref-syntax-qry-select-having.md | 3 + docs/sql-ref-syntax-qry-select-lateral-view.md | 125 + docs/sql-ref-syntax-qry-select-limit.md| 3 + docs/sql-ref-syntax-qry-select-orderby.md | 3 + docs/sql-ref-syntax-qry-select-pivot.md| 101 + docs/sql-ref-syntax-qry-select-sortby.md | 3 + docs/sql-ref-syntax-qry-select-where.md| 3 + docs/sql-ref-syntax-qry-select.md | 56 + docs/sql-ref-syntax-qry.md | 3 + docs/sql-ref-syntax.md | 3 + 16 files changed, 520 insertions(+), 25 deletions(-) create mode 100644 docs/sql-ref-syntax-qry-select-case.md create mode 100644 docs/sql-ref-syntax-qry-select-lateral-view.md create mode 100644 docs/sql-ref-syntax-qry-select-pivot.md - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 6ed93c3 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs 6ed93c3 is described below commit 6ed93c3e86c60323328b44cab45faa9ae3050dab Author: GuoPhilipse AuthorDate: Tue Jul 28 09:41:53 2020 +0900 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs ### What changes were proposed in this pull request? update sql-ref docs, the following key words will be added in this PR. CASE/ELSE WHEN/THEN MAP KEYS TERMINATED BY NULL DEFINED AS LINES TERMINATED BY ESCAPED BY COLLECTION ITEMS TERMINATED BY PIVOT LATERAL VIEW OUTER? ROW FORMAT SERDE ROW FORMAT DELIMITED FIELDS TERMINATED BY IGNORE NULLS FIRST LAST ### Why are the changes needed? let more users know the sql key words usage ### Does this PR introduce _any_ user-facing change?     ### How was this patch tested? No Closes #29056 from GuoPhilipse/add-missing-keywords. Lead-authored-by: GuoPhilipse Co-authored-by: GuoPhilipse <46367746+guophili...@users.noreply.github.com> Signed-off-by: Takeshi Yamamuro (cherry picked from commit 8de43338be879f0cfeebca328dbbcfd1e5bd70da) Signed-off-by: Takeshi Yamamuro --- docs/_data/menu-sql.yaml | 6 + docs/sql-ref-syntax-ddl-create-table-hiveformat.md | 94 +++- docs/sql-ref-syntax-qry-select-case.md | 109 ++ docs/sql-ref-syntax-qry-select-clusterby.md| 3 + docs/sql-ref-syntax-qry-select-distribute-by.md| 3 + docs/sql-ref-syntax-qry-select-groupby.md | 27 + docs/sql-ref-syntax-qry-select-having.md | 3 + docs/sql-ref-syntax-qry-select-lateral-view.md | 125 + docs/sql-ref-syntax-qry-select-limit.md| 3 + docs/sql-ref-syntax-qry-select-orderby.md | 3 + docs/sql-ref-syntax-qry-select-pivot.md| 101 + docs/sql-ref-syntax-qry-select-sortby.md | 3 + docs/sql-ref-syntax-qry-select-where.md| 3 + docs/sql-ref-syntax-qry-select.md | 56 + docs/sql-ref-syntax-qry.md | 3 + docs/sql-ref-syntax.md | 3 + 16 files changed, 520 insertions(+), 25 deletions(-) diff --git a/docs/_data/menu-sql.yaml b/docs/_data/menu-sql.yaml index eea657e..22fae0c 100644 --- a/docs/_data/menu-sql.yaml +++ b/docs/_data/menu-sql.yaml @@ -187,6 +187,12 @@ url: sql-ref-syntax-qry-select-tvf.html - text: Window Function url: sql-ref-syntax-qry-select-window.html +- text: CASE Clause + url: sql-ref-syntax-qry-select-case.html +- text: LATERAL VIEW Clause + url: sql-ref-syntax-qry-select-lateral-view.html +- text: PIVOT Clause + url: sql-ref-syntax-qry-select-pivot.html - text: EXPLAIN url: sql-ref-syntax-qry-explain.html - text: Auxiliary Statements diff --git a/docs/sql-ref-syntax-ddl-create-table-hiveformat.md b/docs/sql-ref-syntax-ddl-create-table-hiveformat.md index 38f8856..7bf847d 100644 --- a/docs/sql-ref-syntax-ddl-create-table-hiveformat.md +++ b/docs/sql-ref-syntax-ddl-create-table-hiveformat.md @@ -36,6 +36,14 @@ CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ] table_identifier [ LOCATION path ] [ TBLPROPERTIES ( key1=val1, key2=val2, ... ) ] [ AS select_statement ] + +row_format: +: SERDE serde_class [ WITH SERDEPROPERTIES (k1=v1, k2=v2, ... ) ] +| DELIMITED [ FIELDS TERMINATED BY fields_termiated_char [ ESCAPED BY escaped_char ] ] +[ COLLECTION ITEMS TERMINATED BY collection_items_termiated_char ] +[ MAP KEYS TERMINATED BY map_key_termiated_char ] +[ LINES TERMINATED BY row_termiated_char ] +[ NULL DEFINED AS null_char ] ``` Note that, the clauses between the columns definition clause and the AS SELECT clause can come in @@ -51,15 +59,55 @@ as any order. For example, you can write COMMENT table_comment after TBLPROPERTI * **EXTERNAL** -Table is defined using the path provided as LOCATION, does not use default loc
[spark] branch master updated (f7542d3 -> 8de4333)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f7542d3 [SPARK-32457][ML] logParam thresholds in DT/GBT/FM/LR/MLP add 8de4333 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs No new revisions were added by this update. Summary of changes: docs/_data/menu-sql.yaml | 6 + docs/sql-ref-syntax-ddl-create-table-hiveformat.md | 94 +++- docs/sql-ref-syntax-qry-select-case.md | 109 ++ docs/sql-ref-syntax-qry-select-clusterby.md| 3 + docs/sql-ref-syntax-qry-select-distribute-by.md| 3 + docs/sql-ref-syntax-qry-select-groupby.md | 27 + docs/sql-ref-syntax-qry-select-having.md | 3 + docs/sql-ref-syntax-qry-select-lateral-view.md | 125 + docs/sql-ref-syntax-qry-select-limit.md| 3 + docs/sql-ref-syntax-qry-select-orderby.md | 3 + docs/sql-ref-syntax-qry-select-pivot.md| 101 + docs/sql-ref-syntax-qry-select-sortby.md | 3 + docs/sql-ref-syntax-qry-select-where.md| 3 + docs/sql-ref-syntax-qry-select.md | 56 + docs/sql-ref-syntax-qry.md | 3 + docs/sql-ref-syntax.md | 3 + 16 files changed, 520 insertions(+), 25 deletions(-) create mode 100644 docs/sql-ref-syntax-qry-select-case.md create mode 100644 docs/sql-ref-syntax-qry-select-lateral-view.md create mode 100644 docs/sql-ref-syntax-qry-select-pivot.md - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 6ed93c3 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs 6ed93c3 is described below commit 6ed93c3e86c60323328b44cab45faa9ae3050dab Author: GuoPhilipse AuthorDate: Tue Jul 28 09:41:53 2020 +0900 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs ### What changes were proposed in this pull request? update sql-ref docs, the following key words will be added in this PR. CASE/ELSE WHEN/THEN MAP KEYS TERMINATED BY NULL DEFINED AS LINES TERMINATED BY ESCAPED BY COLLECTION ITEMS TERMINATED BY PIVOT LATERAL VIEW OUTER? ROW FORMAT SERDE ROW FORMAT DELIMITED FIELDS TERMINATED BY IGNORE NULLS FIRST LAST ### Why are the changes needed? let more users know the sql key words usage ### Does this PR introduce _any_ user-facing change?     ### How was this patch tested? No Closes #29056 from GuoPhilipse/add-missing-keywords. Lead-authored-by: GuoPhilipse Co-authored-by: GuoPhilipse <46367746+guophili...@users.noreply.github.com> Signed-off-by: Takeshi Yamamuro (cherry picked from commit 8de43338be879f0cfeebca328dbbcfd1e5bd70da) Signed-off-by: Takeshi Yamamuro --- docs/_data/menu-sql.yaml | 6 + docs/sql-ref-syntax-ddl-create-table-hiveformat.md | 94 +++- docs/sql-ref-syntax-qry-select-case.md | 109 ++ docs/sql-ref-syntax-qry-select-clusterby.md| 3 + docs/sql-ref-syntax-qry-select-distribute-by.md| 3 + docs/sql-ref-syntax-qry-select-groupby.md | 27 + docs/sql-ref-syntax-qry-select-having.md | 3 + docs/sql-ref-syntax-qry-select-lateral-view.md | 125 + docs/sql-ref-syntax-qry-select-limit.md| 3 + docs/sql-ref-syntax-qry-select-orderby.md | 3 + docs/sql-ref-syntax-qry-select-pivot.md| 101 + docs/sql-ref-syntax-qry-select-sortby.md | 3 + docs/sql-ref-syntax-qry-select-where.md| 3 + docs/sql-ref-syntax-qry-select.md | 56 + docs/sql-ref-syntax-qry.md | 3 + docs/sql-ref-syntax.md | 3 + 16 files changed, 520 insertions(+), 25 deletions(-) diff --git a/docs/_data/menu-sql.yaml b/docs/_data/menu-sql.yaml index eea657e..22fae0c 100644 --- a/docs/_data/menu-sql.yaml +++ b/docs/_data/menu-sql.yaml @@ -187,6 +187,12 @@ url: sql-ref-syntax-qry-select-tvf.html - text: Window Function url: sql-ref-syntax-qry-select-window.html +- text: CASE Clause + url: sql-ref-syntax-qry-select-case.html +- text: LATERAL VIEW Clause + url: sql-ref-syntax-qry-select-lateral-view.html +- text: PIVOT Clause + url: sql-ref-syntax-qry-select-pivot.html - text: EXPLAIN url: sql-ref-syntax-qry-explain.html - text: Auxiliary Statements diff --git a/docs/sql-ref-syntax-ddl-create-table-hiveformat.md b/docs/sql-ref-syntax-ddl-create-table-hiveformat.md index 38f8856..7bf847d 100644 --- a/docs/sql-ref-syntax-ddl-create-table-hiveformat.md +++ b/docs/sql-ref-syntax-ddl-create-table-hiveformat.md @@ -36,6 +36,14 @@ CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ] table_identifier [ LOCATION path ] [ TBLPROPERTIES ( key1=val1, key2=val2, ... ) ] [ AS select_statement ] + +row_format: +: SERDE serde_class [ WITH SERDEPROPERTIES (k1=v1, k2=v2, ... ) ] +| DELIMITED [ FIELDS TERMINATED BY fields_termiated_char [ ESCAPED BY escaped_char ] ] +[ COLLECTION ITEMS TERMINATED BY collection_items_termiated_char ] +[ MAP KEYS TERMINATED BY map_key_termiated_char ] +[ LINES TERMINATED BY row_termiated_char ] +[ NULL DEFINED AS null_char ] ``` Note that, the clauses between the columns definition clause and the AS SELECT clause can come in @@ -51,15 +59,55 @@ as any order. For example, you can write COMMENT table_comment after TBLPROPERTI * **EXTERNAL** -Table is defined using the path provided as LOCATION, does not use default loc
[spark] branch master updated (f7542d3 -> 8de4333)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f7542d3 [SPARK-32457][ML] logParam thresholds in DT/GBT/FM/LR/MLP add 8de4333 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs No new revisions were added by this update. Summary of changes: docs/_data/menu-sql.yaml | 6 + docs/sql-ref-syntax-ddl-create-table-hiveformat.md | 94 +++- docs/sql-ref-syntax-qry-select-case.md | 109 ++ docs/sql-ref-syntax-qry-select-clusterby.md| 3 + docs/sql-ref-syntax-qry-select-distribute-by.md| 3 + docs/sql-ref-syntax-qry-select-groupby.md | 27 + docs/sql-ref-syntax-qry-select-having.md | 3 + docs/sql-ref-syntax-qry-select-lateral-view.md | 125 + docs/sql-ref-syntax-qry-select-limit.md| 3 + docs/sql-ref-syntax-qry-select-orderby.md | 3 + docs/sql-ref-syntax-qry-select-pivot.md| 101 + docs/sql-ref-syntax-qry-select-sortby.md | 3 + docs/sql-ref-syntax-qry-select-where.md| 3 + docs/sql-ref-syntax-qry-select.md | 56 + docs/sql-ref-syntax-qry.md | 3 + docs/sql-ref-syntax.md | 3 + 16 files changed, 520 insertions(+), 25 deletions(-) create mode 100644 docs/sql-ref-syntax-qry-select-case.md create mode 100644 docs/sql-ref-syntax-qry-select-lateral-view.md create mode 100644 docs/sql-ref-syntax-qry-select-pivot.md - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 6ed93c3 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs 6ed93c3 is described below commit 6ed93c3e86c60323328b44cab45faa9ae3050dab Author: GuoPhilipse AuthorDate: Tue Jul 28 09:41:53 2020 +0900 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs ### What changes were proposed in this pull request? update sql-ref docs, the following key words will be added in this PR. CASE/ELSE WHEN/THEN MAP KEYS TERMINATED BY NULL DEFINED AS LINES TERMINATED BY ESCAPED BY COLLECTION ITEMS TERMINATED BY PIVOT LATERAL VIEW OUTER? ROW FORMAT SERDE ROW FORMAT DELIMITED FIELDS TERMINATED BY IGNORE NULLS FIRST LAST ### Why are the changes needed? let more users know the sql key words usage ### Does this PR introduce _any_ user-facing change?     ### How was this patch tested? No Closes #29056 from GuoPhilipse/add-missing-keywords. Lead-authored-by: GuoPhilipse Co-authored-by: GuoPhilipse <46367746+guophili...@users.noreply.github.com> Signed-off-by: Takeshi Yamamuro (cherry picked from commit 8de43338be879f0cfeebca328dbbcfd1e5bd70da) Signed-off-by: Takeshi Yamamuro --- docs/_data/menu-sql.yaml | 6 + docs/sql-ref-syntax-ddl-create-table-hiveformat.md | 94 +++- docs/sql-ref-syntax-qry-select-case.md | 109 ++ docs/sql-ref-syntax-qry-select-clusterby.md| 3 + docs/sql-ref-syntax-qry-select-distribute-by.md| 3 + docs/sql-ref-syntax-qry-select-groupby.md | 27 + docs/sql-ref-syntax-qry-select-having.md | 3 + docs/sql-ref-syntax-qry-select-lateral-view.md | 125 + docs/sql-ref-syntax-qry-select-limit.md| 3 + docs/sql-ref-syntax-qry-select-orderby.md | 3 + docs/sql-ref-syntax-qry-select-pivot.md| 101 + docs/sql-ref-syntax-qry-select-sortby.md | 3 + docs/sql-ref-syntax-qry-select-where.md| 3 + docs/sql-ref-syntax-qry-select.md | 56 + docs/sql-ref-syntax-qry.md | 3 + docs/sql-ref-syntax.md | 3 + 16 files changed, 520 insertions(+), 25 deletions(-) diff --git a/docs/_data/menu-sql.yaml b/docs/_data/menu-sql.yaml index eea657e..22fae0c 100644 --- a/docs/_data/menu-sql.yaml +++ b/docs/_data/menu-sql.yaml @@ -187,6 +187,12 @@ url: sql-ref-syntax-qry-select-tvf.html - text: Window Function url: sql-ref-syntax-qry-select-window.html +- text: CASE Clause + url: sql-ref-syntax-qry-select-case.html +- text: LATERAL VIEW Clause + url: sql-ref-syntax-qry-select-lateral-view.html +- text: PIVOT Clause + url: sql-ref-syntax-qry-select-pivot.html - text: EXPLAIN url: sql-ref-syntax-qry-explain.html - text: Auxiliary Statements diff --git a/docs/sql-ref-syntax-ddl-create-table-hiveformat.md b/docs/sql-ref-syntax-ddl-create-table-hiveformat.md index 38f8856..7bf847d 100644 --- a/docs/sql-ref-syntax-ddl-create-table-hiveformat.md +++ b/docs/sql-ref-syntax-ddl-create-table-hiveformat.md @@ -36,6 +36,14 @@ CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ] table_identifier [ LOCATION path ] [ TBLPROPERTIES ( key1=val1, key2=val2, ... ) ] [ AS select_statement ] + +row_format: +: SERDE serde_class [ WITH SERDEPROPERTIES (k1=v1, k2=v2, ... ) ] +| DELIMITED [ FIELDS TERMINATED BY fields_termiated_char [ ESCAPED BY escaped_char ] ] +[ COLLECTION ITEMS TERMINATED BY collection_items_termiated_char ] +[ MAP KEYS TERMINATED BY map_key_termiated_char ] +[ LINES TERMINATED BY row_termiated_char ] +[ NULL DEFINED AS null_char ] ``` Note that, the clauses between the columns definition clause and the AS SELECT clause can come in @@ -51,15 +59,55 @@ as any order. For example, you can write COMMENT table_comment after TBLPROPERTI * **EXTERNAL** -Table is defined using the path provided as LOCATION, does not use default loc
[spark] branch branch-3.0 updated: [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 6ed93c3 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs 6ed93c3 is described below commit 6ed93c3e86c60323328b44cab45faa9ae3050dab Author: GuoPhilipse AuthorDate: Tue Jul 28 09:41:53 2020 +0900 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs ### What changes were proposed in this pull request? update sql-ref docs, the following key words will be added in this PR. CASE/ELSE WHEN/THEN MAP KEYS TERMINATED BY NULL DEFINED AS LINES TERMINATED BY ESCAPED BY COLLECTION ITEMS TERMINATED BY PIVOT LATERAL VIEW OUTER? ROW FORMAT SERDE ROW FORMAT DELIMITED FIELDS TERMINATED BY IGNORE NULLS FIRST LAST ### Why are the changes needed? let more users know the sql key words usage ### Does this PR introduce _any_ user-facing change?     ### How was this patch tested? No Closes #29056 from GuoPhilipse/add-missing-keywords. Lead-authored-by: GuoPhilipse Co-authored-by: GuoPhilipse <46367746+guophili...@users.noreply.github.com> Signed-off-by: Takeshi Yamamuro (cherry picked from commit 8de43338be879f0cfeebca328dbbcfd1e5bd70da) Signed-off-by: Takeshi Yamamuro --- docs/_data/menu-sql.yaml | 6 + docs/sql-ref-syntax-ddl-create-table-hiveformat.md | 94 +++- docs/sql-ref-syntax-qry-select-case.md | 109 ++ docs/sql-ref-syntax-qry-select-clusterby.md| 3 + docs/sql-ref-syntax-qry-select-distribute-by.md| 3 + docs/sql-ref-syntax-qry-select-groupby.md | 27 + docs/sql-ref-syntax-qry-select-having.md | 3 + docs/sql-ref-syntax-qry-select-lateral-view.md | 125 + docs/sql-ref-syntax-qry-select-limit.md| 3 + docs/sql-ref-syntax-qry-select-orderby.md | 3 + docs/sql-ref-syntax-qry-select-pivot.md| 101 + docs/sql-ref-syntax-qry-select-sortby.md | 3 + docs/sql-ref-syntax-qry-select-where.md| 3 + docs/sql-ref-syntax-qry-select.md | 56 + docs/sql-ref-syntax-qry.md | 3 + docs/sql-ref-syntax.md | 3 + 16 files changed, 520 insertions(+), 25 deletions(-) diff --git a/docs/_data/menu-sql.yaml b/docs/_data/menu-sql.yaml index eea657e..22fae0c 100644 --- a/docs/_data/menu-sql.yaml +++ b/docs/_data/menu-sql.yaml @@ -187,6 +187,12 @@ url: sql-ref-syntax-qry-select-tvf.html - text: Window Function url: sql-ref-syntax-qry-select-window.html +- text: CASE Clause + url: sql-ref-syntax-qry-select-case.html +- text: LATERAL VIEW Clause + url: sql-ref-syntax-qry-select-lateral-view.html +- text: PIVOT Clause + url: sql-ref-syntax-qry-select-pivot.html - text: EXPLAIN url: sql-ref-syntax-qry-explain.html - text: Auxiliary Statements diff --git a/docs/sql-ref-syntax-ddl-create-table-hiveformat.md b/docs/sql-ref-syntax-ddl-create-table-hiveformat.md index 38f8856..7bf847d 100644 --- a/docs/sql-ref-syntax-ddl-create-table-hiveformat.md +++ b/docs/sql-ref-syntax-ddl-create-table-hiveformat.md @@ -36,6 +36,14 @@ CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ] table_identifier [ LOCATION path ] [ TBLPROPERTIES ( key1=val1, key2=val2, ... ) ] [ AS select_statement ] + +row_format: +: SERDE serde_class [ WITH SERDEPROPERTIES (k1=v1, k2=v2, ... ) ] +| DELIMITED [ FIELDS TERMINATED BY fields_termiated_char [ ESCAPED BY escaped_char ] ] +[ COLLECTION ITEMS TERMINATED BY collection_items_termiated_char ] +[ MAP KEYS TERMINATED BY map_key_termiated_char ] +[ LINES TERMINATED BY row_termiated_char ] +[ NULL DEFINED AS null_char ] ``` Note that, the clauses between the columns definition clause and the AS SELECT clause can come in @@ -51,15 +59,55 @@ as any order. For example, you can write COMMENT table_comment after TBLPROPERTI * **EXTERNAL** -Table is defined using the path provided as LOCATION, does not use default loc
[spark] branch master updated (8de4333 -> 8323c8e)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8de4333 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs add 8323c8e [SPARK-32059][SQL] Allow nested schema pruning thru window/sort plans No new revisions were added by this update. Summary of changes: .../catalyst/optimizer/NestedColumnAliasing.scala | 16 +++ .../optimizer/NestedColumnAliasingSuite.scala | 140 - .../execution/datasources/SchemaPruningSuite.scala | 63 ++ 3 files changed, 217 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8de4333 -> 8323c8e)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8de4333 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs add 8323c8e [SPARK-32059][SQL] Allow nested schema pruning thru window/sort plans No new revisions were added by this update. Summary of changes: .../catalyst/optimizer/NestedColumnAliasing.scala | 16 +++ .../optimizer/NestedColumnAliasingSuite.scala | 140 - .../execution/datasources/SchemaPruningSuite.scala | 63 ++ 3 files changed, 217 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8de4333 -> 8323c8e)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8de4333 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs add 8323c8e [SPARK-32059][SQL] Allow nested schema pruning thru window/sort plans No new revisions were added by this update. Summary of changes: .../catalyst/optimizer/NestedColumnAliasing.scala | 16 +++ .../optimizer/NestedColumnAliasingSuite.scala | 140 - .../execution/datasources/SchemaPruningSuite.scala | 63 ++ 3 files changed, 217 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8de4333 -> 8323c8e)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8de4333 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs add 8323c8e [SPARK-32059][SQL] Allow nested schema pruning thru window/sort plans No new revisions were added by this update. Summary of changes: .../catalyst/optimizer/NestedColumnAliasing.scala | 16 +++ .../optimizer/NestedColumnAliasingSuite.scala | 140 - .../execution/datasources/SchemaPruningSuite.scala | 63 ++ 3 files changed, 217 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8de4333 -> 8323c8e)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8de4333 [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs add 8323c8e [SPARK-32059][SQL] Allow nested schema pruning thru window/sort plans No new revisions were added by this update. Summary of changes: .../catalyst/optimizer/NestedColumnAliasing.scala | 16 +++ .../optimizer/NestedColumnAliasingSuite.scala | 140 - .../execution/datasources/SchemaPruningSuite.scala | 63 ++ 3 files changed, 217 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8323c8e -> 77f2ca6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8323c8e [SPARK-32059][SQL] Allow nested schema pruning thru window/sort plans add 77f2ca6 [MINOR][PYTHON] Fix spacing in error message No new revisions were added by this update. Summary of changes: python/pyspark/worker.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8323c8e -> 77f2ca6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8323c8e [SPARK-32059][SQL] Allow nested schema pruning thru window/sort plans add 77f2ca6 [MINOR][PYTHON] Fix spacing in error message No new revisions were added by this update. Summary of changes: python/pyspark/worker.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [MINOR][PYTHON] Fix spacing in error message
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new c1421d0 [MINOR][PYTHON] Fix spacing in error message
c1421d0 is described below
commit c1421d0da16d841a1b3be3c18e77b12adb407ec1
Author: Shantanu <12621235+hauntsani...@users.noreply.github.com>
AuthorDate: Tue Jul 28 11:22:18 2020 +0900
[MINOR][PYTHON] Fix spacing in error message
### What changes were proposed in this pull request?
Fixes spacing in an error message
### Why are the changes needed?
Makes error messages easier to read
### Does this PR introduce _any_ user-facing change?
Yes, it changes the error message
### How was this patch tested?
This patch doesn't affect any logic, so existing tests should cover it
Closes #29264 from hauntsaninja/patch-1.
Authored-by: Shantanu <12621235+hauntsani...@users.noreply.github.com>
Signed-off-by: HyukjinKwon
(cherry picked from commit 77f2ca6cced1c723d1c2e6082a1534f6436c6d2a)
Signed-off-by: HyukjinKwon
---
python/pyspark/worker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index 953b468..4b51046 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -261,7 +261,7 @@ def main(infile, outfile):
version = utf8_deserializer.loads(infile)
if version != "%d.%d" % sys.version_info[:2]:
raise Exception(("Python in worker has different version %s than
that in " +
- "driver %s, PySpark cannot run with different
minor versions." +
+ "driver %s, PySpark cannot run with different
minor versions. " +
"Please check environment variables
PYSPARK_PYTHON and " +
"PYSPARK_DRIVER_PYTHON are correctly set.") %
("%d.%d" % sys.version_info[:2], version))
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][PYTHON] Fix spacing in error message
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new b35b3eb [MINOR][PYTHON] Fix spacing in error message
b35b3eb is described below
commit b35b3ebdb9b39a30d8e261bbf0a80482c149c2a8
Author: Shantanu <12621235+hauntsani...@users.noreply.github.com>
AuthorDate: Tue Jul 28 11:22:18 2020 +0900
[MINOR][PYTHON] Fix spacing in error message
### What changes were proposed in this pull request?
Fixes spacing in an error message
### Why are the changes needed?
Makes error messages easier to read
### Does this PR introduce _any_ user-facing change?
Yes, it changes the error message
### How was this patch tested?
This patch doesn't affect any logic, so existing tests should cover it
Closes #29264 from hauntsaninja/patch-1.
Authored-by: Shantanu <12621235+hauntsani...@users.noreply.github.com>
Signed-off-by: HyukjinKwon
(cherry picked from commit 77f2ca6cced1c723d1c2e6082a1534f6436c6d2a)
Signed-off-by: HyukjinKwon
---
python/pyspark/worker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index 5f4a8a2..814f796 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -471,7 +471,7 @@ def main(infile, outfile):
version = utf8_deserializer.loads(infile)
if version != "%d.%d" % sys.version_info[:2]:
raise Exception(("Python in worker has different version %s than
that in " +
- "driver %s, PySpark cannot run with different
minor versions." +
+ "driver %s, PySpark cannot run with different
minor versions. " +
"Please check environment variables
PYSPARK_PYTHON and " +
"PYSPARK_DRIVER_PYTHON are correctly set.") %
("%d.%d" % sys.version_info[:2], version))
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8323c8e -> 77f2ca6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8323c8e [SPARK-32059][SQL] Allow nested schema pruning thru window/sort plans add 77f2ca6 [MINOR][PYTHON] Fix spacing in error message No new revisions were added by this update. Summary of changes: python/pyspark/worker.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][PYTHON] Fix spacing in error message
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new b35b3eb [MINOR][PYTHON] Fix spacing in error message
b35b3eb is described below
commit b35b3ebdb9b39a30d8e261bbf0a80482c149c2a8
Author: Shantanu <12621235+hauntsani...@users.noreply.github.com>
AuthorDate: Tue Jul 28 11:22:18 2020 +0900
[MINOR][PYTHON] Fix spacing in error message
### What changes were proposed in this pull request?
Fixes spacing in an error message
### Why are the changes needed?
Makes error messages easier to read
### Does this PR introduce _any_ user-facing change?
Yes, it changes the error message
### How was this patch tested?
This patch doesn't affect any logic, so existing tests should cover it
Closes #29264 from hauntsaninja/patch-1.
Authored-by: Shantanu <12621235+hauntsani...@users.noreply.github.com>
Signed-off-by: HyukjinKwon
(cherry picked from commit 77f2ca6cced1c723d1c2e6082a1534f6436c6d2a)
Signed-off-by: HyukjinKwon
---
python/pyspark/worker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index 5f4a8a2..814f796 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -471,7 +471,7 @@ def main(infile, outfile):
version = utf8_deserializer.loads(infile)
if version != "%d.%d" % sys.version_info[:2]:
raise Exception(("Python in worker has different version %s than
that in " +
- "driver %s, PySpark cannot run with different
minor versions." +
+ "driver %s, PySpark cannot run with different
minor versions. " +
"Please check environment variables
PYSPARK_PYTHON and " +
"PYSPARK_DRIVER_PYTHON are correctly set.") %
("%d.%d" % sys.version_info[:2], version))
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [MINOR][PYTHON] Fix spacing in error message
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new c1421d0 [MINOR][PYTHON] Fix spacing in error message
c1421d0 is described below
commit c1421d0da16d841a1b3be3c18e77b12adb407ec1
Author: Shantanu <12621235+hauntsani...@users.noreply.github.com>
AuthorDate: Tue Jul 28 11:22:18 2020 +0900
[MINOR][PYTHON] Fix spacing in error message
### What changes were proposed in this pull request?
Fixes spacing in an error message
### Why are the changes needed?
Makes error messages easier to read
### Does this PR introduce _any_ user-facing change?
Yes, it changes the error message
### How was this patch tested?
This patch doesn't affect any logic, so existing tests should cover it
Closes #29264 from hauntsaninja/patch-1.
Authored-by: Shantanu <12621235+hauntsani...@users.noreply.github.com>
Signed-off-by: HyukjinKwon
(cherry picked from commit 77f2ca6cced1c723d1c2e6082a1534f6436c6d2a)
Signed-off-by: HyukjinKwon
---
python/pyspark/worker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index 953b468..4b51046 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -261,7 +261,7 @@ def main(infile, outfile):
version = utf8_deserializer.loads(infile)
if version != "%d.%d" % sys.version_info[:2]:
raise Exception(("Python in worker has different version %s than
that in " +
- "driver %s, PySpark cannot run with different
minor versions." +
+ "driver %s, PySpark cannot run with different
minor versions. " +
"Please check environment variables
PYSPARK_PYTHON and " +
"PYSPARK_DRIVER_PYTHON are correctly set.") %
("%d.%d" % sys.version_info[:2], version))
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8323c8e -> 77f2ca6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8323c8e [SPARK-32059][SQL] Allow nested schema pruning thru window/sort plans add 77f2ca6 [MINOR][PYTHON] Fix spacing in error message No new revisions were added by this update. Summary of changes: python/pyspark/worker.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][PYTHON] Fix spacing in error message
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new b35b3eb [MINOR][PYTHON] Fix spacing in error message
b35b3eb is described below
commit b35b3ebdb9b39a30d8e261bbf0a80482c149c2a8
Author: Shantanu <12621235+hauntsani...@users.noreply.github.com>
AuthorDate: Tue Jul 28 11:22:18 2020 +0900
[MINOR][PYTHON] Fix spacing in error message
### What changes were proposed in this pull request?
Fixes spacing in an error message
### Why are the changes needed?
Makes error messages easier to read
### Does this PR introduce _any_ user-facing change?
Yes, it changes the error message
### How was this patch tested?
This patch doesn't affect any logic, so existing tests should cover it
Closes #29264 from hauntsaninja/patch-1.
Authored-by: Shantanu <12621235+hauntsani...@users.noreply.github.com>
Signed-off-by: HyukjinKwon
(cherry picked from commit 77f2ca6cced1c723d1c2e6082a1534f6436c6d2a)
Signed-off-by: HyukjinKwon
---
python/pyspark/worker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index 5f4a8a2..814f796 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -471,7 +471,7 @@ def main(infile, outfile):
version = utf8_deserializer.loads(infile)
if version != "%d.%d" % sys.version_info[:2]:
raise Exception(("Python in worker has different version %s than
that in " +
- "driver %s, PySpark cannot run with different
minor versions." +
+ "driver %s, PySpark cannot run with different
minor versions. " +
"Please check environment variables
PYSPARK_PYTHON and " +
"PYSPARK_DRIVER_PYTHON are correctly set.") %
("%d.%d" % sys.version_info[:2], version))
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [MINOR][PYTHON] Fix spacing in error message
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new c1421d0 [MINOR][PYTHON] Fix spacing in error message
c1421d0 is described below
commit c1421d0da16d841a1b3be3c18e77b12adb407ec1
Author: Shantanu <12621235+hauntsani...@users.noreply.github.com>
AuthorDate: Tue Jul 28 11:22:18 2020 +0900
[MINOR][PYTHON] Fix spacing in error message
### What changes were proposed in this pull request?
Fixes spacing in an error message
### Why are the changes needed?
Makes error messages easier to read
### Does this PR introduce _any_ user-facing change?
Yes, it changes the error message
### How was this patch tested?
This patch doesn't affect any logic, so existing tests should cover it
Closes #29264 from hauntsaninja/patch-1.
Authored-by: Shantanu <12621235+hauntsani...@users.noreply.github.com>
Signed-off-by: HyukjinKwon
(cherry picked from commit 77f2ca6cced1c723d1c2e6082a1534f6436c6d2a)
Signed-off-by: HyukjinKwon
---
python/pyspark/worker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index 953b468..4b51046 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -261,7 +261,7 @@ def main(infile, outfile):
version = utf8_deserializer.loads(infile)
if version != "%d.%d" % sys.version_info[:2]:
raise Exception(("Python in worker has different version %s than
that in " +
- "driver %s, PySpark cannot run with different
minor versions." +
+ "driver %s, PySpark cannot run with different
minor versions. " +
"Please check environment variables
PYSPARK_PYTHON and " +
"PYSPARK_DRIVER_PYTHON are correctly set.") %
("%d.%d" % sys.version_info[:2], version))
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8323c8e -> 77f2ca6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8323c8e [SPARK-32059][SQL] Allow nested schema pruning thru window/sort plans add 77f2ca6 [MINOR][PYTHON] Fix spacing in error message No new revisions were added by this update. Summary of changes: python/pyspark/worker.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][PYTHON] Fix spacing in error message
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new b35b3eb [MINOR][PYTHON] Fix spacing in error message
b35b3eb is described below
commit b35b3ebdb9b39a30d8e261bbf0a80482c149c2a8
Author: Shantanu <12621235+hauntsani...@users.noreply.github.com>
AuthorDate: Tue Jul 28 11:22:18 2020 +0900
[MINOR][PYTHON] Fix spacing in error message
### What changes were proposed in this pull request?
Fixes spacing in an error message
### Why are the changes needed?
Makes error messages easier to read
### Does this PR introduce _any_ user-facing change?
Yes, it changes the error message
### How was this patch tested?
This patch doesn't affect any logic, so existing tests should cover it
Closes #29264 from hauntsaninja/patch-1.
Authored-by: Shantanu <12621235+hauntsani...@users.noreply.github.com>
Signed-off-by: HyukjinKwon
(cherry picked from commit 77f2ca6cced1c723d1c2e6082a1534f6436c6d2a)
Signed-off-by: HyukjinKwon
---
python/pyspark/worker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index 5f4a8a2..814f796 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -471,7 +471,7 @@ def main(infile, outfile):
version = utf8_deserializer.loads(infile)
if version != "%d.%d" % sys.version_info[:2]:
raise Exception(("Python in worker has different version %s than
that in " +
- "driver %s, PySpark cannot run with different
minor versions." +
+ "driver %s, PySpark cannot run with different
minor versions. " +
"Please check environment variables
PYSPARK_PYTHON and " +
"PYSPARK_DRIVER_PYTHON are correctly set.") %
("%d.%d" % sys.version_info[:2], version))
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [MINOR][PYTHON] Fix spacing in error message
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new c1421d0 [MINOR][PYTHON] Fix spacing in error message
c1421d0 is described below
commit c1421d0da16d841a1b3be3c18e77b12adb407ec1
Author: Shantanu <12621235+hauntsani...@users.noreply.github.com>
AuthorDate: Tue Jul 28 11:22:18 2020 +0900
[MINOR][PYTHON] Fix spacing in error message
### What changes were proposed in this pull request?
Fixes spacing in an error message
### Why are the changes needed?
Makes error messages easier to read
### Does this PR introduce _any_ user-facing change?
Yes, it changes the error message
### How was this patch tested?
This patch doesn't affect any logic, so existing tests should cover it
Closes #29264 from hauntsaninja/patch-1.
Authored-by: Shantanu <12621235+hauntsani...@users.noreply.github.com>
Signed-off-by: HyukjinKwon
(cherry picked from commit 77f2ca6cced1c723d1c2e6082a1534f6436c6d2a)
Signed-off-by: HyukjinKwon
---
python/pyspark/worker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index 953b468..4b51046 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -261,7 +261,7 @@ def main(infile, outfile):
version = utf8_deserializer.loads(infile)
if version != "%d.%d" % sys.version_info[:2]:
raise Exception(("Python in worker has different version %s than
that in " +
- "driver %s, PySpark cannot run with different
minor versions." +
+ "driver %s, PySpark cannot run with different
minor versions. " +
"Please check environment variables
PYSPARK_PYTHON and " +
"PYSPARK_DRIVER_PYTHON are correctly set.") %
("%d.%d" % sys.version_info[:2], version))
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][PYTHON] Fix spacing in error message
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new b35b3eb [MINOR][PYTHON] Fix spacing in error message
b35b3eb is described below
commit b35b3ebdb9b39a30d8e261bbf0a80482c149c2a8
Author: Shantanu <12621235+hauntsani...@users.noreply.github.com>
AuthorDate: Tue Jul 28 11:22:18 2020 +0900
[MINOR][PYTHON] Fix spacing in error message
### What changes were proposed in this pull request?
Fixes spacing in an error message
### Why are the changes needed?
Makes error messages easier to read
### Does this PR introduce _any_ user-facing change?
Yes, it changes the error message
### How was this patch tested?
This patch doesn't affect any logic, so existing tests should cover it
Closes #29264 from hauntsaninja/patch-1.
Authored-by: Shantanu <12621235+hauntsani...@users.noreply.github.com>
Signed-off-by: HyukjinKwon
(cherry picked from commit 77f2ca6cced1c723d1c2e6082a1534f6436c6d2a)
Signed-off-by: HyukjinKwon
---
python/pyspark/worker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index 5f4a8a2..814f796 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -471,7 +471,7 @@ def main(infile, outfile):
version = utf8_deserializer.loads(infile)
if version != "%d.%d" % sys.version_info[:2]:
raise Exception(("Python in worker has different version %s than
that in " +
- "driver %s, PySpark cannot run with different
minor versions." +
+ "driver %s, PySpark cannot run with different
minor versions. " +
"Please check environment variables
PYSPARK_PYTHON and " +
"PYSPARK_DRIVER_PYTHON are correctly set.") %
("%d.%d" % sys.version_info[:2], version))
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [MINOR][PYTHON] Fix spacing in error message
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new c1421d0 [MINOR][PYTHON] Fix spacing in error message
c1421d0 is described below
commit c1421d0da16d841a1b3be3c18e77b12adb407ec1
Author: Shantanu <12621235+hauntsani...@users.noreply.github.com>
AuthorDate: Tue Jul 28 11:22:18 2020 +0900
[MINOR][PYTHON] Fix spacing in error message
### What changes were proposed in this pull request?
Fixes spacing in an error message
### Why are the changes needed?
Makes error messages easier to read
### Does this PR introduce _any_ user-facing change?
Yes, it changes the error message
### How was this patch tested?
This patch doesn't affect any logic, so existing tests should cover it
Closes #29264 from hauntsaninja/patch-1.
Authored-by: Shantanu <12621235+hauntsani...@users.noreply.github.com>
Signed-off-by: HyukjinKwon
(cherry picked from commit 77f2ca6cced1c723d1c2e6082a1534f6436c6d2a)
Signed-off-by: HyukjinKwon
---
python/pyspark/worker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index 953b468..4b51046 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -261,7 +261,7 @@ def main(infile, outfile):
version = utf8_deserializer.loads(infile)
if version != "%d.%d" % sys.version_info[:2]:
raise Exception(("Python in worker has different version %s than
that in " +
- "driver %s, PySpark cannot run with different
minor versions." +
+ "driver %s, PySpark cannot run with different
minor versions. " +
"Please check environment variables
PYSPARK_PYTHON and " +
"PYSPARK_DRIVER_PYTHON are correctly set.") %
("%d.%d" % sys.version_info[:2], version))
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (77f2ca6 -> 44a5258)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 77f2ca6 [MINOR][PYTHON] Fix spacing in error message add 44a5258 [SPARK-31525][SQL] Return an empty list for df.head() when df is empty No new revisions were added by this update. Summary of changes: docs/pyspark-migration-guide.md | 3 +++ python/pyspark/sql/dataframe.py | 5 +++-- 2 files changed, 6 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (77f2ca6 -> 44a5258)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 77f2ca6 [MINOR][PYTHON] Fix spacing in error message add 44a5258 [SPARK-31525][SQL] Return an empty list for df.head() when df is empty No new revisions were added by this update. Summary of changes: docs/pyspark-migration-guide.md | 3 +++ python/pyspark/sql/dataframe.py | 5 +++-- 2 files changed, 6 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (77f2ca6 -> 44a5258)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 77f2ca6 [MINOR][PYTHON] Fix spacing in error message add 44a5258 [SPARK-31525][SQL] Return an empty list for df.head() when df is empty No new revisions were added by this update. Summary of changes: docs/pyspark-migration-guide.md | 3 +++ python/pyspark/sql/dataframe.py | 5 +++-- 2 files changed, 6 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (77f2ca6 -> 44a5258)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 77f2ca6 [MINOR][PYTHON] Fix spacing in error message add 44a5258 [SPARK-31525][SQL] Return an empty list for df.head() when df is empty No new revisions were added by this update. Summary of changes: docs/pyspark-migration-guide.md | 3 +++ python/pyspark/sql/dataframe.py | 5 +++-- 2 files changed, 6 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (77f2ca6 -> 44a5258)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 77f2ca6 [MINOR][PYTHON] Fix spacing in error message add 44a5258 [SPARK-31525][SQL] Return an empty list for df.head() when df is empty No new revisions were added by this update. Summary of changes: docs/pyspark-migration-guide.md | 3 +++ python/pyspark/sql/dataframe.py | 5 +++-- 2 files changed, 6 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (44a5258 -> 12b9787)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 44a5258 [SPARK-31525][SQL] Return an empty list for df.head() when df is empty add 12b9787 [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/planning/patterns.scala | 36 +++ .../org/apache/spark/sql/internal/SQLConf.scala| 14 +++ .../spark/sql/execution/SparkStrategies.scala | 4 + .../adaptive/LogicalQueryStageStrategy.scala | 10 +- .../dynamicpruning/PlanDynamicPruningFilters.scala | 4 +- .../execution/joins/BroadcastHashJoinExec.scala| 82 ++-- .../spark/sql/execution/joins/HashedRelation.scala | 71 -- .../resources/sql-tests/inputs/group-by-filter.sql | 3 + .../inputs/subquery/in-subquery/in-basic.sql | 3 + .../inputs/subquery/in-subquery/in-having.sql | 3 + .../inputs/subquery/in-subquery/in-joins.sql | 3 + .../inputs/subquery/in-subquery/in-limit.sql | 3 + .../subquery/in-subquery/in-multiple-columns.sql | 3 + .../inputs/subquery/in-subquery/in-order-by.sql| 3 + .../inputs/subquery/in-subquery/in-with-cte.sql| 3 + .../inputs/subquery/in-subquery/nested-not-in.sql | 3 + .../inputs/subquery/in-subquery/not-in-joins.sql | 3 + .../not-in-unit-tests-multi-column-literal.sql | 3 + .../in-subquery/not-in-unit-tests-multi-column.sql | 3 + .../not-in-unit-tests-single-column-literal.sql| 3 + .../not-in-unit-tests-single-column.sql| 3 + .../inputs/subquery/in-subquery/simple-in.sql | 3 + .../sql-tests/results/explain-aqe.sql.out | 8 +- .../resources/sql-tests/results/explain.sql.out| 8 +- .../spark/sql/DynamicPartitionPruningSuite.scala | 2 +- .../scala/org/apache/spark/sql/JoinSuite.scala | 41 .../scala/org/apache/spark/sql/SubquerySuite.scala | 107 + .../sql/execution/WholeStageCodegenSuite.scala | 4 +- .../spark/sql/execution/debug/DebuggingSuite.scala | 3 +- .../sql/execution/joins/HashedRelationSuite.scala | 1 + 30 files changed, 409 insertions(+), 31 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (44a5258 -> 12b9787)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 44a5258 [SPARK-31525][SQL] Return an empty list for df.head() when df is empty add 12b9787 [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/planning/patterns.scala | 36 +++ .../org/apache/spark/sql/internal/SQLConf.scala| 14 +++ .../spark/sql/execution/SparkStrategies.scala | 4 + .../adaptive/LogicalQueryStageStrategy.scala | 10 +- .../dynamicpruning/PlanDynamicPruningFilters.scala | 4 +- .../execution/joins/BroadcastHashJoinExec.scala| 82 ++-- .../spark/sql/execution/joins/HashedRelation.scala | 71 -- .../resources/sql-tests/inputs/group-by-filter.sql | 3 + .../inputs/subquery/in-subquery/in-basic.sql | 3 + .../inputs/subquery/in-subquery/in-having.sql | 3 + .../inputs/subquery/in-subquery/in-joins.sql | 3 + .../inputs/subquery/in-subquery/in-limit.sql | 3 + .../subquery/in-subquery/in-multiple-columns.sql | 3 + .../inputs/subquery/in-subquery/in-order-by.sql| 3 + .../inputs/subquery/in-subquery/in-with-cte.sql| 3 + .../inputs/subquery/in-subquery/nested-not-in.sql | 3 + .../inputs/subquery/in-subquery/not-in-joins.sql | 3 + .../not-in-unit-tests-multi-column-literal.sql | 3 + .../in-subquery/not-in-unit-tests-multi-column.sql | 3 + .../not-in-unit-tests-single-column-literal.sql| 3 + .../not-in-unit-tests-single-column.sql| 3 + .../inputs/subquery/in-subquery/simple-in.sql | 3 + .../sql-tests/results/explain-aqe.sql.out | 8 +- .../resources/sql-tests/results/explain.sql.out| 8 +- .../spark/sql/DynamicPartitionPruningSuite.scala | 2 +- .../scala/org/apache/spark/sql/JoinSuite.scala | 41 .../scala/org/apache/spark/sql/SubquerySuite.scala | 107 + .../sql/execution/WholeStageCodegenSuite.scala | 4 +- .../spark/sql/execution/debug/DebuggingSuite.scala | 3 +- .../sql/execution/joins/HashedRelationSuite.scala | 1 + 30 files changed, 409 insertions(+), 31 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (44a5258 -> 12b9787)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 44a5258 [SPARK-31525][SQL] Return an empty list for df.head() when df is empty add 12b9787 [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/planning/patterns.scala | 36 +++ .../org/apache/spark/sql/internal/SQLConf.scala| 14 +++ .../spark/sql/execution/SparkStrategies.scala | 4 + .../adaptive/LogicalQueryStageStrategy.scala | 10 +- .../dynamicpruning/PlanDynamicPruningFilters.scala | 4 +- .../execution/joins/BroadcastHashJoinExec.scala| 82 ++-- .../spark/sql/execution/joins/HashedRelation.scala | 71 -- .../resources/sql-tests/inputs/group-by-filter.sql | 3 + .../inputs/subquery/in-subquery/in-basic.sql | 3 + .../inputs/subquery/in-subquery/in-having.sql | 3 + .../inputs/subquery/in-subquery/in-joins.sql | 3 + .../inputs/subquery/in-subquery/in-limit.sql | 3 + .../subquery/in-subquery/in-multiple-columns.sql | 3 + .../inputs/subquery/in-subquery/in-order-by.sql| 3 + .../inputs/subquery/in-subquery/in-with-cte.sql| 3 + .../inputs/subquery/in-subquery/nested-not-in.sql | 3 + .../inputs/subquery/in-subquery/not-in-joins.sql | 3 + .../not-in-unit-tests-multi-column-literal.sql | 3 + .../in-subquery/not-in-unit-tests-multi-column.sql | 3 + .../not-in-unit-tests-single-column-literal.sql| 3 + .../not-in-unit-tests-single-column.sql| 3 + .../inputs/subquery/in-subquery/simple-in.sql | 3 + .../sql-tests/results/explain-aqe.sql.out | 8 +- .../resources/sql-tests/results/explain.sql.out| 8 +- .../spark/sql/DynamicPartitionPruningSuite.scala | 2 +- .../scala/org/apache/spark/sql/JoinSuite.scala | 41 .../scala/org/apache/spark/sql/SubquerySuite.scala | 107 + .../sql/execution/WholeStageCodegenSuite.scala | 4 +- .../spark/sql/execution/debug/DebuggingSuite.scala | 3 +- .../sql/execution/joins/HashedRelationSuite.scala | 1 + 30 files changed, 409 insertions(+), 31 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (44a5258 -> 12b9787)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 44a5258 [SPARK-31525][SQL] Return an empty list for df.head() when df is empty add 12b9787 [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/planning/patterns.scala | 36 +++ .../org/apache/spark/sql/internal/SQLConf.scala| 14 +++ .../spark/sql/execution/SparkStrategies.scala | 4 + .../adaptive/LogicalQueryStageStrategy.scala | 10 +- .../dynamicpruning/PlanDynamicPruningFilters.scala | 4 +- .../execution/joins/BroadcastHashJoinExec.scala| 82 ++-- .../spark/sql/execution/joins/HashedRelation.scala | 71 -- .../resources/sql-tests/inputs/group-by-filter.sql | 3 + .../inputs/subquery/in-subquery/in-basic.sql | 3 + .../inputs/subquery/in-subquery/in-having.sql | 3 + .../inputs/subquery/in-subquery/in-joins.sql | 3 + .../inputs/subquery/in-subquery/in-limit.sql | 3 + .../subquery/in-subquery/in-multiple-columns.sql | 3 + .../inputs/subquery/in-subquery/in-order-by.sql| 3 + .../inputs/subquery/in-subquery/in-with-cte.sql| 3 + .../inputs/subquery/in-subquery/nested-not-in.sql | 3 + .../inputs/subquery/in-subquery/not-in-joins.sql | 3 + .../not-in-unit-tests-multi-column-literal.sql | 3 + .../in-subquery/not-in-unit-tests-multi-column.sql | 3 + .../not-in-unit-tests-single-column-literal.sql| 3 + .../not-in-unit-tests-single-column.sql| 3 + .../inputs/subquery/in-subquery/simple-in.sql | 3 + .../sql-tests/results/explain-aqe.sql.out | 8 +- .../resources/sql-tests/results/explain.sql.out| 8 +- .../spark/sql/DynamicPartitionPruningSuite.scala | 2 +- .../scala/org/apache/spark/sql/JoinSuite.scala | 41 .../scala/org/apache/spark/sql/SubquerySuite.scala | 107 + .../sql/execution/WholeStageCodegenSuite.scala | 4 +- .../spark/sql/execution/debug/DebuggingSuite.scala | 3 +- .../sql/execution/joins/HashedRelationSuite.scala | 1 + 30 files changed, 409 insertions(+), 31 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (44a5258 -> 12b9787)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 44a5258 [SPARK-31525][SQL] Return an empty list for df.head() when df is empty add 12b9787 [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/planning/patterns.scala | 36 +++ .../org/apache/spark/sql/internal/SQLConf.scala| 14 +++ .../spark/sql/execution/SparkStrategies.scala | 4 + .../adaptive/LogicalQueryStageStrategy.scala | 10 +- .../dynamicpruning/PlanDynamicPruningFilters.scala | 4 +- .../execution/joins/BroadcastHashJoinExec.scala| 82 ++-- .../spark/sql/execution/joins/HashedRelation.scala | 71 -- .../resources/sql-tests/inputs/group-by-filter.sql | 3 + .../inputs/subquery/in-subquery/in-basic.sql | 3 + .../inputs/subquery/in-subquery/in-having.sql | 3 + .../inputs/subquery/in-subquery/in-joins.sql | 3 + .../inputs/subquery/in-subquery/in-limit.sql | 3 + .../subquery/in-subquery/in-multiple-columns.sql | 3 + .../inputs/subquery/in-subquery/in-order-by.sql| 3 + .../inputs/subquery/in-subquery/in-with-cte.sql| 3 + .../inputs/subquery/in-subquery/nested-not-in.sql | 3 + .../inputs/subquery/in-subquery/not-in-joins.sql | 3 + .../not-in-unit-tests-multi-column-literal.sql | 3 + .../in-subquery/not-in-unit-tests-multi-column.sql | 3 + .../not-in-unit-tests-single-column-literal.sql| 3 + .../not-in-unit-tests-single-column.sql| 3 + .../inputs/subquery/in-subquery/simple-in.sql | 3 + .../sql-tests/results/explain-aqe.sql.out | 8 +- .../resources/sql-tests/results/explain.sql.out| 8 +- .../spark/sql/DynamicPartitionPruningSuite.scala | 2 +- .../scala/org/apache/spark/sql/JoinSuite.scala | 41 .../scala/org/apache/spark/sql/SubquerySuite.scala | 107 + .../sql/execution/WholeStageCodegenSuite.scala | 4 +- .../spark/sql/execution/debug/DebuggingSuite.scala | 3 +- .../sql/execution/joins/HashedRelationSuite.scala | 1 + 30 files changed, 409 insertions(+), 31 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
