[spark] branch master updated (160c144 -> bce8d93)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 160c144 [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type add bce8d93 [SPARK-30765][SQL] Refine base operator abstraction code style No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/DataSourceScanExec.scala| 8 .../org/apache/spark/sql/execution/joins/HashJoin.scala| 14 +++--- .../spark/sql/execution/python/ArrowEvalPythonExec.scala | 4 ++-- .../spark/sql/execution/python/BatchEvalPythonExec.scala | 2 +- .../apache/spark/sql/execution/python/EvalPythonExec.scala | 5 +++-- .../spark/sql/execution/python/WindowInPandasExec.scala| 2 +- .../execution/streaming/FlatMapGroupsWithStateExec.scala | 2 +- .../org/apache/spark/sql/execution/window/WindowExec.scala | 2 +- .../apache/spark/sql/execution/window/WindowExecBase.scala | 9 - 9 files changed, 24 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (160c144 -> bce8d93)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 160c144 [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type add bce8d93 [SPARK-30765][SQL] Refine base operator abstraction code style No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/DataSourceScanExec.scala| 8 .../org/apache/spark/sql/execution/joins/HashJoin.scala| 14 +++--- .../spark/sql/execution/python/ArrowEvalPythonExec.scala | 4 ++-- .../spark/sql/execution/python/BatchEvalPythonExec.scala | 2 +- .../apache/spark/sql/execution/python/EvalPythonExec.scala | 5 +++-- .../spark/sql/execution/python/WindowInPandasExec.scala| 2 +- .../execution/streaming/FlatMapGroupsWithStateExec.scala | 2 +- .../org/apache/spark/sql/execution/window/WindowExec.scala | 2 +- .../apache/spark/sql/execution/window/WindowExecBase.scala | 9 - 9 files changed, 24 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c3f4946 [SPARK-30590][SQL] Untyped select API cannot take typed
column expression that needs input type
c3f4946 is described below
commit c3f494627b65e47c1cf7b7873fa28322a4cd0afa
Author: Liang-Chi Hsieh
AuthorDate: Thu Feb 27 14:09:07 2020 +0800
[SPARK-30590][SQL] Untyped select API cannot take typed column expression
that needs input type
### What changes were proposed in this pull request?
This patch proposes to throw clear analysis exception if untyped
`Dataset.select` takes typed column expression that needs input type.
### Why are the changes needed?
`Dataset` provides few typed `select` helper functions to select typed
column expressions. The maximum number of typed columns supported is 5. If
wanting to select more than 5 typed columns, it silently calls untyped
`Dataset.select` and can causes weird unresolved error, like:
```
org.apache.spark.sql.AnalysisException: unresolved operator 'Aggregate
[fooagg(FooAgg(1), None, None, None, input[0, int, false] AS value#114,
assertnotnull(cast(value#114 as int)), input[0, int, false] AS value#113,
IntegerType, IntegerType, false) AS foo_agg_1#116, fooagg(FooAgg(2), None,
None, None, input[0, int, false] AS value#119, assertnotnull(cast(value#119 as
int)), input[0, int, false] AS value#118, IntegerType, IntegerType, false) AS
foo_agg_2#121, fooagg(FooAgg(3), None, N [...]
'Aggregate [fooagg(FooAgg(1), None, None, None, input[0, int, false] AS
value#114, assertnotnull(cast(value#114 as int)), input[0, int, false] AS
value#113, IntegerType, IntegerType, false) AS foo_agg_1#116, fooagg(FooAgg(2),
None, None, None, input[0, int, false] AS value#119,
assertnotnull(cast(value#119 as int)), input[0, int, false] AS value#118,
IntegerType, IntegerType, false) AS foo_agg_2#121, fooagg(FooAgg(3), None,
None, None, input[0, int, false] AS value#124, assertnotnull( [...]
+- Project [_1#6 AS a#13, _2#7 AS b#14, _3#8 AS c#15, _4#9 AS d#16, _5#10
AS e#17, _6#11 AS F#18]
+- LocalRelation [_1#6, _2#7, _3#8, _4#9, _5#10, _6#11]
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:43)
at
org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:95)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$3.apply(CheckAnalysis.scala:431)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$3.apply(CheckAnalysis.scala:430)
at
org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:430)
```
However, to fully disallow typed columns as input to untyped `select` API
will break current usage like `count` that is a `TypedColumn` in `functions`.
In order to keep compatibility, we should allow current usage of certain
`TypedColumn`s as input to untyped `select` API. For the `TypedColumn`s that
will cause unresolved exception, we should explicitly let users know that they
are incorrectly calling untyped `select` with typed columns which need input
type.
### Does this PR introduce any user-facing change?
Yes, but this PR only refines the error message.
When users call `Dataset.select` API with typed column that needs input
type, an analysis exception will be thrown. Previously an unresolved error will
be thrown.
### How was this patch tested?
Unit tests.
Closes #27499 from viirya/SPARK-30590.
Lead-authored-by: Liang-Chi Hsieh
Co-authored-by: Liang-Chi Hsieh
Signed-off-by: Wenchen Fan
(cherry picked from commit 160c144baa82b7f76301b718694187cf939ca465)
Signed-off-by: Wenchen Fan
---
.../main/scala/org/apache/spark/sql/Dataset.scala | 21 ++-
.../apache/spark/sql/DatasetAggregatorSuite.scala | 24 ++
2 files changed, 44 insertions(+), 1 deletion(-)
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
index 42f3535..d85e23b 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
@@ -48,6 +48,7 @@ import
org.apache.spark.sql.catalyst.plans.physical.{Partitioning, PartitioningC
import org.apache.spark.sql.catalyst.trees.TreeNodeTag
import org.apache.spark.sql.catalyst.util.IntervalUtils
import org.apache.spark.sql.execution._
+import org.apache.spark.sql.execution.aggregate.TypedAggregateExpression
import org.apache.spark.sql.execution.arrow.{ArrowBatchStreamWriter,
ArrowConverters}
[spark] branch master updated (1515d45 -> 160c144)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1515d45 [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate add 160c144 [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/Dataset.scala | 21 ++- .../apache/spark/sql/DatasetAggregatorSuite.scala | 24 ++ 2 files changed, 44 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1515d45 -> 160c144)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1515d45 [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate add 160c144 [SPARK-30590][SQL] Untyped select API cannot take typed column expression that needs input type No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/Dataset.scala | 21 ++- .../apache/spark/sql/DatasetAggregatorSuite.scala | 24 ++ 2 files changed, 44 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new dcd2094 [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL
Boolean-Predicate
dcd2094 is described below
commit dcd2094371db90c2bfe5dcfdd7a01033db88727d
Author: beliefer
AuthorDate: Thu Feb 27 13:42:02 2020 +0800
[SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate
### What changes were proposed in this pull request?
This PR follows https://github.com/apache/spark/pull/25074 and improves the
implement.
### Why are the changes needed?
Improve code.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Exists UT
Closes #27699 from beliefer/improve-boolean-test.
Authored-by: beliefer
Signed-off-by: Wenchen Fan
(cherry picked from commit 1515d45b8db69de67cf61100f7cf4fa5c7cadbd4)
Signed-off-by: Wenchen Fan
---
.../sql/catalyst/expressions/predicates.scala | 60 --
.../spark/sql/catalyst/parser/AstBuilder.scala | 8 +--
.../sql/catalyst/expressions/PredicateSuite.scala | 32 +---
3 files changed, 6 insertions(+), 94 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
index bcd442a..bd190c3 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
@@ -927,66 +927,6 @@ case class GreaterThanOrEqual(left: Expression, right:
Expression)
protected override def nullSafeEval(input1: Any, input2: Any): Any =
ordering.gteq(input1, input2)
}
-trait BooleanTest extends UnaryExpression with Predicate with
ExpectsInputTypes {
-
- def boolValueForComparison: Boolean
- def boolValueWhenNull: Boolean
-
- override def nullable: Boolean = false
- override def inputTypes: Seq[DataType] = Seq(BooleanType)
-
- override def eval(input: InternalRow): Any = {
-val value = child.eval(input)
-Option(value) match {
- case None => boolValueWhenNull
- case other => if (boolValueWhenNull) {
-value == !boolValueForComparison
- } else {
-value == boolValueForComparison
- }
-}
- }
-
- override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
-val eval = child.genCode(ctx)
-ev.copy(code = code"""
- ${eval.code}
- ${CodeGenerator.javaType(dataType)} ${ev.value} =
${CodeGenerator.defaultValue(dataType)};
- if (${eval.isNull}) {
-${ev.value} = $boolValueWhenNull;
- } else if ($boolValueWhenNull) {
-${ev.value} = ${eval.value} == !$boolValueForComparison;
- } else {
-${ev.value} = ${eval.value} == $boolValueForComparison;
- }
- """, isNull = FalseLiteral)
- }
-}
-
-case class IsTrue(child: Expression) extends BooleanTest {
- override def boolValueForComparison: Boolean = true
- override def boolValueWhenNull: Boolean = false
- override def sql: String = s"(${child.sql} IS TRUE)"
-}
-
-case class IsNotTrue(child: Expression) extends BooleanTest {
- override def boolValueForComparison: Boolean = true
- override def boolValueWhenNull: Boolean = true
- override def sql: String = s"(${child.sql} IS NOT TRUE)"
-}
-
-case class IsFalse(child: Expression) extends BooleanTest {
- override def boolValueForComparison: Boolean = false
- override def boolValueWhenNull: Boolean = false
- override def sql: String = s"(${child.sql} IS FALSE)"
-}
-
-case class IsNotFalse(child: Expression) extends BooleanTest {
- override def boolValueForComparison: Boolean = false
- override def boolValueWhenNull: Boolean = true
- override def sql: String = s"(${child.sql} IS NOT FALSE)"
-}
-
/**
* IS UNKNOWN and IS NOT UNKNOWN are the same as IS NULL and IS NOT NULL,
respectively,
* except that the input expression must be of a boolean type.
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
index 2806fd4..fe72446 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
@@ -1414,12 +1414,12 @@ class AstBuilder(conf: SQLConf) extends
SqlBaseBaseVisitor[AnyRef] with Logging
case SqlBaseParser.NULL =>
IsNull(e)
case SqlBaseParser.TRUE => ctx.NOT match {
-case null => IsTrue(e)
-case _ => IsNotTrue(e)
+case null => EqualNullSafe(e, Literal(true))
+case _ => Not(EqualNullSafe(e, Literal(true)))
}
[spark] branch master updated (2b744fe -> 1515d45)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2b744fe [SPARK-30963][INFRA] Add GitHub Action job for document generation add 1515d45 [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/predicates.scala | 60 -- .../spark/sql/catalyst/parser/AstBuilder.scala | 8 +-- .../sql/catalyst/expressions/PredicateSuite.scala | 32 +--- 3 files changed, 6 insertions(+), 94 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2b744fe -> 1515d45)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2b744fe [SPARK-30963][INFRA] Add GitHub Action job for document generation add 1515d45 [SPARK-27924][SQL][FOLLOW-UP] Improve ANSI SQL Boolean-Predicate No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/predicates.scala | 60 -- .../spark/sql/catalyst/parser/AstBuilder.scala | 8 +-- .../sql/catalyst/expressions/PredicateSuite.scala | 32 +--- 3 files changed, 6 insertions(+), 94 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30963][INFRA] Add GitHub Action job for document generation
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c9463c6 [SPARK-30963][INFRA] Add GitHub Action job for document
generation
c9463c6 is described below
commit c9463c6f8e7e4ee0a0f6bf09618569ea5a84aa03
Author: Dongjoon Hyun
AuthorDate: Wed Feb 26 19:24:41 2020 -0800
[SPARK-30963][INFRA] Add GitHub Action job for document generation
### What changes were proposed in this pull request?
This PR aims to add a new `GitHub Action` job for document generation.
### Why are the changes needed?
We had better test the document generation in PR Builder.
-
https://lists.apache.org/thread.html/rd06a2154e853812652b8f7fa3c003746ed531b213c531517f055e1dc%40%3Cdev.spark.apache.org%3E
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Pass the GitHub Action in this PR.
Closes #27715 from dongjoon-hyun/SPARK-30963.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 2b744fe88547acc82a6178ff806b78056db7a025)
Signed-off-by: Dongjoon Hyun
---
.github/workflows/master.yml | 37 +
1 file changed, 37 insertions(+)
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index 0f80c88..33ea335 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -117,3 +117,40 @@ jobs:
run: ./R/install-dev.sh
- name: lint-r
run: ./dev/lint-r
+
+ docs:
+runs-on: ubuntu-latest
+name: Generate documents
+steps:
+- uses: actions/checkout@master
+- uses: actions/cache@v1
+ with:
+path: ~/.m2/repository
+key: docs-maven-repo-${{ hashFiles('**/pom.xml') }}
+restore-keys: |
+ docs-maven-repo-
+- uses: actions/setup-java@v1
+ with:
+java-version: '1.8'
+- uses: actions/setup-python@v1
+ with:
+python-version: '3.x'
+architecture: 'x64'
+- uses: actions/setup-ruby@v1
+ with:
+ruby-version: '2.7'
+- name: Install R
+ run: |
+echo 'deb https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/'
| sudo tee -a /etc/apt/sources.list
+curl -sL
"https://keyserver.ubuntu.com/pks/lookup?op=get=0xE298A3A825C0D65DFD57CBB651716619E084DAB9;
| sudo apt-key add
+sudo apt-get update
+sudo apt-get install -y r-base r-base-dev libcurl4-openssl-dev pandoc
+- name: Install packages
+ run: |
+pip install sphinx mkdocs numpy
+gem install jekyll jekyll-redirect-from rouge
+sudo Rscript -e "install.packages(c('curl', 'xml2', 'httr',
'devtools', 'testthat', 'knitr', 'rmarkdown', 'roxygen2', 'e1071', 'survival'),
repos='https://cloud.r-project.org/')"
+- name: Run jekyll build
+ run: |
+cd docs
+jekyll build
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (325bf56 -> 2b744fe)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 325bf56 [SPARK-30888][CORE][DOC] Add version information to the configuration of Network add 2b744fe [SPARK-30963][INFRA] Add GitHub Action job for document generation No new revisions were added by this update. Summary of changes: .github/workflows/master.yml | 37 + 1 file changed, 37 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30963][INFRA] Add GitHub Action job for document generation
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c9463c6 [SPARK-30963][INFRA] Add GitHub Action job for document
generation
c9463c6 is described below
commit c9463c6f8e7e4ee0a0f6bf09618569ea5a84aa03
Author: Dongjoon Hyun
AuthorDate: Wed Feb 26 19:24:41 2020 -0800
[SPARK-30963][INFRA] Add GitHub Action job for document generation
### What changes were proposed in this pull request?
This PR aims to add a new `GitHub Action` job for document generation.
### Why are the changes needed?
We had better test the document generation in PR Builder.
-
https://lists.apache.org/thread.html/rd06a2154e853812652b8f7fa3c003746ed531b213c531517f055e1dc%40%3Cdev.spark.apache.org%3E
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Pass the GitHub Action in this PR.
Closes #27715 from dongjoon-hyun/SPARK-30963.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 2b744fe88547acc82a6178ff806b78056db7a025)
Signed-off-by: Dongjoon Hyun
---
.github/workflows/master.yml | 37 +
1 file changed, 37 insertions(+)
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index 0f80c88..33ea335 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -117,3 +117,40 @@ jobs:
run: ./R/install-dev.sh
- name: lint-r
run: ./dev/lint-r
+
+ docs:
+runs-on: ubuntu-latest
+name: Generate documents
+steps:
+- uses: actions/checkout@master
+- uses: actions/cache@v1
+ with:
+path: ~/.m2/repository
+key: docs-maven-repo-${{ hashFiles('**/pom.xml') }}
+restore-keys: |
+ docs-maven-repo-
+- uses: actions/setup-java@v1
+ with:
+java-version: '1.8'
+- uses: actions/setup-python@v1
+ with:
+python-version: '3.x'
+architecture: 'x64'
+- uses: actions/setup-ruby@v1
+ with:
+ruby-version: '2.7'
+- name: Install R
+ run: |
+echo 'deb https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/'
| sudo tee -a /etc/apt/sources.list
+curl -sL
"https://keyserver.ubuntu.com/pks/lookup?op=get=0xE298A3A825C0D65DFD57CBB651716619E084DAB9;
| sudo apt-key add
+sudo apt-get update
+sudo apt-get install -y r-base r-base-dev libcurl4-openssl-dev pandoc
+- name: Install packages
+ run: |
+pip install sphinx mkdocs numpy
+gem install jekyll jekyll-redirect-from rouge
+sudo Rscript -e "install.packages(c('curl', 'xml2', 'httr',
'devtools', 'testthat', 'knitr', 'rmarkdown', 'roxygen2', 'e1071', 'survival'),
repos='https://cloud.r-project.org/')"
+- name: Run jekyll build
+ run: |
+cd docs
+jekyll build
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (325bf56 -> 2b744fe)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 325bf56 [SPARK-30888][CORE][DOC] Add version information to the configuration of Network add 2b744fe [SPARK-30963][INFRA] Add GitHub Action job for document generation No new revisions were added by this update. Summary of changes: .github/workflows/master.yml | 37 + 1 file changed, 37 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (cd3ef22 -> 325bf56)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from cd3ef22 [SPARK-30906][SQL][TESTS][FOLLOW-UP] Set the configuration against TestHive explicitly in HiveSerDeSuite add 325bf56 [SPARK-30888][CORE][DOC] Add version information to the configuration of Network No new revisions were added by this update. Summary of changes: .../org/apache/spark/internal/config/Network.scala | 14 ++ docs/configuration.md| 20 ++-- 2 files changed, 32 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30906][SQL][TESTS][FOLLOW-UP] Set the configuration against TestHive explicitly in HiveSerDeSuite
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 95df63c [SPARK-30906][SQL][TESTS][FOLLOW-UP] Set the configuration
against TestHive explicitly in HiveSerDeSuite
95df63c is described below
commit 95df63c45fd9c8159ddee00014023eda10df93dc
Author: HyukjinKwon
AuthorDate: Wed Feb 26 18:01:26 2020 -0800
[SPARK-30906][SQL][TESTS][FOLLOW-UP] Set the configuration against TestHive
explicitly in HiveSerDeSuite
### What changes were proposed in this pull request?
After https://github.com/apache/spark/pull/27659 (see
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/253/),
the tests below fail consistently, specifically in one job
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/
in Jenkins
```
org.apache.spark.sql.hive.execution.HiveSerDeSuite.Test the default
fileformat for Hive-serde tables
```
The profile is same as PR builder but seems it fails specifically in this
machine.
Several configurations used in `HiveSerDeSuite` are not being set
presumably due to the inconsistency between `SQLConf.get` and the active Spark
session described in the https://github.com/apache/spark/pull/27387, and as a
side effect of the cloned session at https://github.com/apache/spark/pull/27659.
This PR proposes to explicitly set the configuration against `TestHive` by
using `withExistingConf` at `withSQLConf`
### Why are the changes needed?
To make `spark-master-test-sbt-hadoop-2.7-hive-2.3` job pass.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Cannot reproduce in my local. Presumably it cannot be reproduced in the PR
builder. We should see if the tests pass at
`spark-master-test-sbt-hadoop-2.7-hive-2.3` job after this PR is merged.
Closes #27705 from HyukjinKwon/SPARK-30906.
Authored-by: HyukjinKwon
Signed-off-by: Gengliang Wang
(cherry picked from commit cd3ef2249fc6bd60e4ba62ee046dcf4115864206)
Signed-off-by: Gengliang Wang
---
.../scala/org/apache/spark/sql/hive/execution/HiveSerDeSuite.scala| 4
1 file changed, 4 insertions(+)
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSerDeSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSerDeSuite.scala
index 9a1190a..d2d3502 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSerDeSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSerDeSuite.scala
@@ -82,6 +82,10 @@ class HiveSerDeSuite extends HiveComparisonTest with
PlanTest with BeforeAndAfte
}.head
}
+ // Make sure we set the config values to TestHive.conf.
+ override def withSQLConf(pairs: (String, String)*)(f: => Unit): Unit =
+SQLConf.withExistingConf(TestHive.conf)(super.withSQLConf(pairs: _*)(f))
+
test("Test the default fileformat for Hive-serde tables") {
withSQLConf("hive.default.fileformat" -> "orc",
SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED.key -> "true") {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (825d3dc -> cd3ef22)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 825d3dc [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL add cd3ef22 [SPARK-30906][SQL][TESTS][FOLLOW-UP] Set the configuration against TestHive explicitly in HiveSerDeSuite No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/sql/hive/execution/HiveSerDeSuite.scala| 4 1 file changed, 4 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c285750 -> 825d3dc)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c285750 [SPARK-30909][CORE][DOC] Add version information to the configuration of Python add 825d3dc [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/internal/SQLConf.scala| 64 +- 1 file changed, 61 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c285750 -> 825d3dc)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c285750 [SPARK-30909][CORE][DOC] Add version information to the configuration of Python add 825d3dc [SPARK-30841][SQL][DOC] Add version information to the configuration of SQL No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/internal/SQLConf.scala| 64 +- 1 file changed, 61 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (776e21a -> c285750)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 776e21a [SPARK-30910][CORE][DOC] Add version information to the configuration of R add c285750 [SPARK-30909][CORE][DOC] Add version information to the configuration of Python No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/internal/config/Python.scala | 6 ++ docs/configuration.md | 6 -- 2 files changed, 10 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2c9db56 -> 776e21a)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2c9db56 [SPARK-30928][ML] Remove unnecessary MiMa excludes add 776e21a [SPARK-30910][CORE][DOC] Add version information to the configuration of R No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/internal/config/R.scala | 5 + docs/configuration.md| 8 +++- 2 files changed, 12 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2c9db56 -> 776e21a)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2c9db56 [SPARK-30928][ML] Remove unnecessary MiMa excludes add 776e21a [SPARK-30910][CORE][DOC] Add version information to the configuration of R No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/internal/config/R.scala | 5 + docs/configuration.md| 8 +++- 2 files changed, 12 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30928][ML] Remove unnecessary MiMa excludes
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 51fd2f9 [SPARK-30928][ML] Remove unnecessary MiMa excludes
51fd2f9 is described below
commit 51fd2f9e68db59cf73e70a4dc8fa927bb0151772
Author: Huaxin Gao
AuthorDate: Wed Feb 26 19:45:15 2020 -0600
[SPARK-30928][ML] Remove unnecessary MiMa excludes
### What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/SPARK-30928
remove unnecessary MiMa excludes
### Why are the changes needed?
When auditing binary incompatible changes for 3.0, I found several MiMa
excludes are not necessary, so remove these.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
run dev/mima to check
Closes #27696 from huaxingao/spark-mima.
Authored-by: Huaxin Gao
Signed-off-by: Sean Owen
(cherry picked from commit 2c9db5608d2d65181a690c32bd0c3e83107b2b9c)
Signed-off-by: Sean Owen
---
project/MimaExcludes.scala | 17 ++---
1 file changed, 2 insertions(+), 15 deletions(-)
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index 289ccb2..d69caa9 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -47,7 +47,7 @@ object MimaExcludes {
// [SPARK-29399][core] Remove old ExecutorPlugin interface.
ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ExecutorPlugin"),
-// [SPARK-][SQL][CORE][MLLIB] Remove more old deprecated items in Spark 3
+// [SPARK-28980][SQL][CORE][MLLIB] Remove more old deprecated items in
Spark 3
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.SQLContext.createExternalTable"),
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.createExternalTable"),
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.mllib.clustering.KMeans.train"),
@@ -79,6 +79,7 @@ object MimaExcludes {
// [SPARK-27090][CORE] Removing old LEGACY_DRIVER_IDENTIFIER ("")
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.SparkContext.LEGACY_DRIVER_IDENTIFIER"),
+
// [SPARK-25838] Remove formatVersion from Saveable
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.mllib.clustering.DistributedLDAModel.formatVersion"),
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.mllib.clustering.LocalLDAModel.formatVersion"),
@@ -231,13 +232,6 @@ object MimaExcludes {
ProblemFilters.exclude[FinalMethodProblem]("org.apache.spark.ml.feature.QuantileDiscretizer.relativeError"),
ProblemFilters.exclude[FinalMethodProblem]("org.apache.spark.ml.feature.QuantileDiscretizer.getRelativeError"),
-// [SPARK-25959] GBTClassifier picks wrong impurity stats on loading
-
ProblemFilters.exclude[InheritedNewAbstractMethodProblem]("org.apache.spark.ml.tree.HasVarianceImpurity.org$apache$spark$ml$tree$HasVarianceImpurity$_setter_$impurity_="),
-
ProblemFilters.exclude[InheritedNewAbstractMethodProblem]("org.apache.spark.ml.tree.HasVarianceImpurity.org$apache$spark$ml$tree$HasVarianceImpurity$_setter_$impurity_="),
-
ProblemFilters.exclude[InheritedNewAbstractMethodProblem]("org.apache.spark.ml.tree.HasVarianceImpurity.org$apache$spark$ml$tree$HasVarianceImpurity$_setter_$impurity_="),
-
ProblemFilters.exclude[InheritedNewAbstractMethodProblem]("org.apache.spark.ml.tree.HasVarianceImpurity.org$apache$spark$ml$tree$HasVarianceImpurity$_setter_$impurity_="),
-
ProblemFilters.exclude[InheritedNewAbstractMethodProblem]("org.apache.spark.ml.tree.HasVarianceImpurity.org$apache$spark$ml$tree$HasVarianceImpurity$_setter_$impurity_="),
-
// [SPARK-28968][ML] Add HasNumFeatures in the scala side
ProblemFilters.exclude[FinalMethodProblem]("org.apache.spark.ml.feature.FeatureHasher.getNumFeatures"),
ProblemFilters.exclude[FinalMethodProblem]("org.apache.spark.ml.feature.FeatureHasher.numFeatures"),
@@ -316,12 +310,6 @@ object MimaExcludes {
ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.feature.OneHotEncoderEstimator"),
ProblemFilters.exclude[MissingTypesProblem]("org.apache.spark.ml.feature.OneHotEncoder"),
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.feature.OneHotEncoder.transform"),
-
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.feature.OneHotEncoder.getInputCol"),
-
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.feature.OneHotEncoder.getOutputCol"),
-
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.feature.OneHotEncoder.inputCol"),
-
[spark] branch branch-3.0 updated: [SPARK-30928][ML] Remove unnecessary MiMa excludes
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 51fd2f9 [SPARK-30928][ML] Remove unnecessary MiMa excludes
51fd2f9 is described below
commit 51fd2f9e68db59cf73e70a4dc8fa927bb0151772
Author: Huaxin Gao
AuthorDate: Wed Feb 26 19:45:15 2020 -0600
[SPARK-30928][ML] Remove unnecessary MiMa excludes
### What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/SPARK-30928
remove unnecessary MiMa excludes
### Why are the changes needed?
When auditing binary incompatible changes for 3.0, I found several MiMa
excludes are not necessary, so remove these.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
run dev/mima to check
Closes #27696 from huaxingao/spark-mima.
Authored-by: Huaxin Gao
Signed-off-by: Sean Owen
(cherry picked from commit 2c9db5608d2d65181a690c32bd0c3e83107b2b9c)
Signed-off-by: Sean Owen
---
project/MimaExcludes.scala | 17 ++---
1 file changed, 2 insertions(+), 15 deletions(-)
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index 289ccb2..d69caa9 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -47,7 +47,7 @@ object MimaExcludes {
// [SPARK-29399][core] Remove old ExecutorPlugin interface.
ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ExecutorPlugin"),
-// [SPARK-][SQL][CORE][MLLIB] Remove more old deprecated items in Spark 3
+// [SPARK-28980][SQL][CORE][MLLIB] Remove more old deprecated items in
Spark 3
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.SQLContext.createExternalTable"),
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.createExternalTable"),
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.mllib.clustering.KMeans.train"),
@@ -79,6 +79,7 @@ object MimaExcludes {
// [SPARK-27090][CORE] Removing old LEGACY_DRIVER_IDENTIFIER ("")
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.SparkContext.LEGACY_DRIVER_IDENTIFIER"),
+
// [SPARK-25838] Remove formatVersion from Saveable
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.mllib.clustering.DistributedLDAModel.formatVersion"),
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.mllib.clustering.LocalLDAModel.formatVersion"),
@@ -231,13 +232,6 @@ object MimaExcludes {
ProblemFilters.exclude[FinalMethodProblem]("org.apache.spark.ml.feature.QuantileDiscretizer.relativeError"),
ProblemFilters.exclude[FinalMethodProblem]("org.apache.spark.ml.feature.QuantileDiscretizer.getRelativeError"),
-// [SPARK-25959] GBTClassifier picks wrong impurity stats on loading
-
ProblemFilters.exclude[InheritedNewAbstractMethodProblem]("org.apache.spark.ml.tree.HasVarianceImpurity.org$apache$spark$ml$tree$HasVarianceImpurity$_setter_$impurity_="),
-
ProblemFilters.exclude[InheritedNewAbstractMethodProblem]("org.apache.spark.ml.tree.HasVarianceImpurity.org$apache$spark$ml$tree$HasVarianceImpurity$_setter_$impurity_="),
-
ProblemFilters.exclude[InheritedNewAbstractMethodProblem]("org.apache.spark.ml.tree.HasVarianceImpurity.org$apache$spark$ml$tree$HasVarianceImpurity$_setter_$impurity_="),
-
ProblemFilters.exclude[InheritedNewAbstractMethodProblem]("org.apache.spark.ml.tree.HasVarianceImpurity.org$apache$spark$ml$tree$HasVarianceImpurity$_setter_$impurity_="),
-
ProblemFilters.exclude[InheritedNewAbstractMethodProblem]("org.apache.spark.ml.tree.HasVarianceImpurity.org$apache$spark$ml$tree$HasVarianceImpurity$_setter_$impurity_="),
-
// [SPARK-28968][ML] Add HasNumFeatures in the scala side
ProblemFilters.exclude[FinalMethodProblem]("org.apache.spark.ml.feature.FeatureHasher.getNumFeatures"),
ProblemFilters.exclude[FinalMethodProblem]("org.apache.spark.ml.feature.FeatureHasher.numFeatures"),
@@ -316,12 +310,6 @@ object MimaExcludes {
ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.ml.feature.OneHotEncoderEstimator"),
ProblemFilters.exclude[MissingTypesProblem]("org.apache.spark.ml.feature.OneHotEncoder"),
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.feature.OneHotEncoder.transform"),
-
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.feature.OneHotEncoder.getInputCol"),
-
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.feature.OneHotEncoder.getOutputCol"),
-
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.feature.OneHotEncoder.inputCol"),
-
[spark] branch master updated (a6026c8 -> 2c9db56)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a6026c8 [MINOR][BUILD] Fix make-distribution.sh to show usage without 'echo' cmd add 2c9db56 [SPARK-30928][ML] Remove unnecessary MiMa excludes No new revisions were added by this update. Summary of changes: project/MimaExcludes.scala | 17 ++--- 1 file changed, 2 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a6026c8 -> 2c9db56)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a6026c8 [MINOR][BUILD] Fix make-distribution.sh to show usage without 'echo' cmd add 2c9db56 [SPARK-30928][ML] Remove unnecessary MiMa excludes No new revisions were added by this update. Summary of changes: project/MimaExcludes.scala | 17 ++--- 1 file changed, 2 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [MINOR][BUILD] Fix make-distribution.sh to show usage without 'echo' cmd
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new b0a2c17 [MINOR][BUILD] Fix make-distribution.sh to show usage without
'echo' cmd
b0a2c17 is described below
commit b0a2c172fa6a54a6aa37d4ca9ab3bba4df6db1fe
Author: Kent Yao
AuthorDate: Wed Feb 26 14:40:32 2020 -0800
[MINOR][BUILD] Fix make-distribution.sh to show usage without 'echo' cmd
### What changes were proposed in this pull request?
turn off `x` mode and do not print the command while printing usage of
`make-distribution.sh`
### Why are the changes needed?
improve dev tools
### Does this PR introduce any user-facing change?
for only developers, clearer hints
after
```
./dev/make-distribution.sh --hel
+++ dirname ./dev/make-distribution.sh
++ cd ./dev/..
++ pwd
+ SPARK_HOME=/Users/kentyao/spark
+ DISTDIR=/Users/kentyao/spark/dist
+ MAKE_TGZ=false
+ MAKE_PIP=false
+ MAKE_R=false
+ NAME=none
+ MVN=/Users/kentyao/spark/build/mvn
+ (( 1 ))
+ case $1 in
+ echo 'Error: --hel is not supported'
Error: --hel is not supported
+ exit_with_usage
+ set +x
make-distribution.sh - tool for making binary distributions of Spark
usage:
make-distribution.sh [--name] [--tgz] [--pip] [--r] [--mvn ]
See Spark's "Building Spark" doc for correct Maven options.
```
before
```
+++ dirname ./dev/make-distribution.sh
++ cd ./dev/..
++ pwd
+ SPARK_HOME=/Users/kentyao/spark
+ DISTDIR=/Users/kentyao/spark/dist
+ MAKE_TGZ=false
+ MAKE_PIP=false
+ MAKE_R=false
+ NAME=none
+ MVN=/Users/kentyao/spark/build/mvn
+ (( 1 ))
+ case $1 in
+ echo 'Error: --hel is not supported'
Error: --hel is not supported
+ exit_with_usage
+ echo 'make-distribution.sh - tool for making binary distributions of
Spark'
make-distribution.sh - tool for making binary distributions of Spark
+ echo ''
+ echo usage:
usage:
+ cl_options='[--name] [--tgz] [--pip] [--r] [--mvn ]'
+ echo 'make-distribution.sh [--name] [--tgz] [--pip] [--r] [--mvn
] '
make-distribution.sh [--name] [--tgz] [--pip] [--r] [--mvn ]
+ echo 'See Spark'\''s "Building Spark" doc for correct Maven options.'
See Spark's "Building Spark" doc for correct Maven options.
+ echo ''
+ exit 1
```
### How was this patch tested?
manually
Closes #27706 from yaooqinn/build.
Authored-by: Kent Yao
Signed-off-by: Dongjoon Hyun
(cherry picked from commit a6026c830a582af75a0d95d18f7759922a086334)
Signed-off-by: Dongjoon Hyun
---
dev/make-distribution.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index 6227032..78b85bd 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -39,6 +39,7 @@ NAME=none
MVN="$SPARK_HOME/build/mvn"
function exit_with_usage {
+ set +x
echo "make-distribution.sh - tool for making binary distributions of Spark"
echo ""
echo "usage:"
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [MINOR][BUILD] Fix make-distribution.sh to show usage without 'echo' cmd
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new b0a2c17 [MINOR][BUILD] Fix make-distribution.sh to show usage without
'echo' cmd
b0a2c17 is described below
commit b0a2c172fa6a54a6aa37d4ca9ab3bba4df6db1fe
Author: Kent Yao
AuthorDate: Wed Feb 26 14:40:32 2020 -0800
[MINOR][BUILD] Fix make-distribution.sh to show usage without 'echo' cmd
### What changes were proposed in this pull request?
turn off `x` mode and do not print the command while printing usage of
`make-distribution.sh`
### Why are the changes needed?
improve dev tools
### Does this PR introduce any user-facing change?
for only developers, clearer hints
after
```
./dev/make-distribution.sh --hel
+++ dirname ./dev/make-distribution.sh
++ cd ./dev/..
++ pwd
+ SPARK_HOME=/Users/kentyao/spark
+ DISTDIR=/Users/kentyao/spark/dist
+ MAKE_TGZ=false
+ MAKE_PIP=false
+ MAKE_R=false
+ NAME=none
+ MVN=/Users/kentyao/spark/build/mvn
+ (( 1 ))
+ case $1 in
+ echo 'Error: --hel is not supported'
Error: --hel is not supported
+ exit_with_usage
+ set +x
make-distribution.sh - tool for making binary distributions of Spark
usage:
make-distribution.sh [--name] [--tgz] [--pip] [--r] [--mvn ]
See Spark's "Building Spark" doc for correct Maven options.
```
before
```
+++ dirname ./dev/make-distribution.sh
++ cd ./dev/..
++ pwd
+ SPARK_HOME=/Users/kentyao/spark
+ DISTDIR=/Users/kentyao/spark/dist
+ MAKE_TGZ=false
+ MAKE_PIP=false
+ MAKE_R=false
+ NAME=none
+ MVN=/Users/kentyao/spark/build/mvn
+ (( 1 ))
+ case $1 in
+ echo 'Error: --hel is not supported'
Error: --hel is not supported
+ exit_with_usage
+ echo 'make-distribution.sh - tool for making binary distributions of
Spark'
make-distribution.sh - tool for making binary distributions of Spark
+ echo ''
+ echo usage:
usage:
+ cl_options='[--name] [--tgz] [--pip] [--r] [--mvn ]'
+ echo 'make-distribution.sh [--name] [--tgz] [--pip] [--r] [--mvn
] '
make-distribution.sh [--name] [--tgz] [--pip] [--r] [--mvn ]
+ echo 'See Spark'\''s "Building Spark" doc for correct Maven options.'
See Spark's "Building Spark" doc for correct Maven options.
+ echo ''
+ exit 1
```
### How was this patch tested?
manually
Closes #27706 from yaooqinn/build.
Authored-by: Kent Yao
Signed-off-by: Dongjoon Hyun
(cherry picked from commit a6026c830a582af75a0d95d18f7759922a086334)
Signed-off-by: Dongjoon Hyun
---
dev/make-distribution.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index 6227032..78b85bd 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -39,6 +39,7 @@ NAME=none
MVN="$SPARK_HOME/build/mvn"
function exit_with_usage {
+ set +x
echo "make-distribution.sh - tool for making binary distributions of Spark"
echo ""
echo "usage:"
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][BUILD] Fix make-distribution.sh to show usage without 'echo' cmd
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 20e0e8d [MINOR][BUILD] Fix make-distribution.sh to show usage without
'echo' cmd
20e0e8d is described below
commit 20e0e8d596ddd9d612cd4e4d12ac1bef99851230
Author: Kent Yao
AuthorDate: Wed Feb 26 14:40:32 2020 -0800
[MINOR][BUILD] Fix make-distribution.sh to show usage without 'echo' cmd
### What changes were proposed in this pull request?
turn off `x` mode and do not print the command while printing usage of
`make-distribution.sh`
### Why are the changes needed?
improve dev tools
### Does this PR introduce any user-facing change?
for only developers, clearer hints
after
```
./dev/make-distribution.sh --hel
+++ dirname ./dev/make-distribution.sh
++ cd ./dev/..
++ pwd
+ SPARK_HOME=/Users/kentyao/spark
+ DISTDIR=/Users/kentyao/spark/dist
+ MAKE_TGZ=false
+ MAKE_PIP=false
+ MAKE_R=false
+ NAME=none
+ MVN=/Users/kentyao/spark/build/mvn
+ (( 1 ))
+ case $1 in
+ echo 'Error: --hel is not supported'
Error: --hel is not supported
+ exit_with_usage
+ set +x
make-distribution.sh - tool for making binary distributions of Spark
usage:
make-distribution.sh [--name] [--tgz] [--pip] [--r] [--mvn ]
See Spark's "Building Spark" doc for correct Maven options.
```
before
```
+++ dirname ./dev/make-distribution.sh
++ cd ./dev/..
++ pwd
+ SPARK_HOME=/Users/kentyao/spark
+ DISTDIR=/Users/kentyao/spark/dist
+ MAKE_TGZ=false
+ MAKE_PIP=false
+ MAKE_R=false
+ NAME=none
+ MVN=/Users/kentyao/spark/build/mvn
+ (( 1 ))
+ case $1 in
+ echo 'Error: --hel is not supported'
Error: --hel is not supported
+ exit_with_usage
+ echo 'make-distribution.sh - tool for making binary distributions of
Spark'
make-distribution.sh - tool for making binary distributions of Spark
+ echo ''
+ echo usage:
usage:
+ cl_options='[--name] [--tgz] [--pip] [--r] [--mvn ]'
+ echo 'make-distribution.sh [--name] [--tgz] [--pip] [--r] [--mvn
] '
make-distribution.sh [--name] [--tgz] [--pip] [--r] [--mvn ]
+ echo 'See Spark'\''s "Building Spark" doc for correct Maven options.'
See Spark's "Building Spark" doc for correct Maven options.
+ echo ''
+ exit 1
```
### How was this patch tested?
manually
Closes #27706 from yaooqinn/build.
Authored-by: Kent Yao
Signed-off-by: Dongjoon Hyun
(cherry picked from commit a6026c830a582af75a0d95d18f7759922a086334)
Signed-off-by: Dongjoon Hyun
---
dev/make-distribution.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index 0b30eec..9e767ce 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -39,6 +39,7 @@ NAME=none
MVN="$SPARK_HOME/build/mvn"
function exit_with_usage {
+ set +x
echo "make-distribution.sh - tool for making binary distributions of Spark"
echo ""
echo "usage:"
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (eced932 -> a6026c8)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from eced932 [SPARK-30918][SQL][FOLLOWUP] Fix typo in OptimizeSkewedJoin add a6026c8 [MINOR][BUILD] Fix make-distribution.sh to show usage without 'echo' cmd No new revisions were added by this update. Summary of changes: dev/make-distribution.sh | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][BUILD] Fix make-distribution.sh to show usage without 'echo' cmd
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 20e0e8d [MINOR][BUILD] Fix make-distribution.sh to show usage without
'echo' cmd
20e0e8d is described below
commit 20e0e8d596ddd9d612cd4e4d12ac1bef99851230
Author: Kent Yao
AuthorDate: Wed Feb 26 14:40:32 2020 -0800
[MINOR][BUILD] Fix make-distribution.sh to show usage without 'echo' cmd
### What changes were proposed in this pull request?
turn off `x` mode and do not print the command while printing usage of
`make-distribution.sh`
### Why are the changes needed?
improve dev tools
### Does this PR introduce any user-facing change?
for only developers, clearer hints
after
```
./dev/make-distribution.sh --hel
+++ dirname ./dev/make-distribution.sh
++ cd ./dev/..
++ pwd
+ SPARK_HOME=/Users/kentyao/spark
+ DISTDIR=/Users/kentyao/spark/dist
+ MAKE_TGZ=false
+ MAKE_PIP=false
+ MAKE_R=false
+ NAME=none
+ MVN=/Users/kentyao/spark/build/mvn
+ (( 1 ))
+ case $1 in
+ echo 'Error: --hel is not supported'
Error: --hel is not supported
+ exit_with_usage
+ set +x
make-distribution.sh - tool for making binary distributions of Spark
usage:
make-distribution.sh [--name] [--tgz] [--pip] [--r] [--mvn ]
See Spark's "Building Spark" doc for correct Maven options.
```
before
```
+++ dirname ./dev/make-distribution.sh
++ cd ./dev/..
++ pwd
+ SPARK_HOME=/Users/kentyao/spark
+ DISTDIR=/Users/kentyao/spark/dist
+ MAKE_TGZ=false
+ MAKE_PIP=false
+ MAKE_R=false
+ NAME=none
+ MVN=/Users/kentyao/spark/build/mvn
+ (( 1 ))
+ case $1 in
+ echo 'Error: --hel is not supported'
Error: --hel is not supported
+ exit_with_usage
+ echo 'make-distribution.sh - tool for making binary distributions of
Spark'
make-distribution.sh - tool for making binary distributions of Spark
+ echo ''
+ echo usage:
usage:
+ cl_options='[--name] [--tgz] [--pip] [--r] [--mvn ]'
+ echo 'make-distribution.sh [--name] [--tgz] [--pip] [--r] [--mvn
] '
make-distribution.sh [--name] [--tgz] [--pip] [--r] [--mvn ]
+ echo 'See Spark'\''s "Building Spark" doc for correct Maven options.'
See Spark's "Building Spark" doc for correct Maven options.
+ echo ''
+ exit 1
```
### How was this patch tested?
manually
Closes #27706 from yaooqinn/build.
Authored-by: Kent Yao
Signed-off-by: Dongjoon Hyun
(cherry picked from commit a6026c830a582af75a0d95d18f7759922a086334)
Signed-off-by: Dongjoon Hyun
---
dev/make-distribution.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index 0b30eec..9e767ce 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -39,6 +39,7 @@ NAME=none
MVN="$SPARK_HOME/build/mvn"
function exit_with_usage {
+ set +x
echo "make-distribution.sh - tool for making binary distributions of Spark"
echo ""
echo "usage:"
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-30759][SQL][3.0] Fix cache initialization in StringRegexExpression
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 56fa200 [SPARK-30759][SQL][3.0] Fix cache initialization in
StringRegexExpression
56fa200 is described below
commit 56fa200948d69396ce60b99256efd9da114bdb6c
Author: Maxim Gekk
AuthorDate: Wed Feb 26 14:25:58 2020 -0800
[SPARK-30759][SQL][3.0] Fix cache initialization in StringRegexExpression
In the PR, I propose to fix `cache` initialization in
`StringRegexExpression` by changing of expected value type in `case
Literal(value: String, StringType)` from `String` to `UTF8String`.
This is a backport of #27502 and #27547
Actually, the case doesn't work at all because `Literal`'s value has type
`UTF8String`, see
https://user-images.githubusercontent.com/1580697/74091681-0d4a2180-4acb-11ea-8a0d-7e8c65f4214e.png;>
No
Added new test by `RegexpExpressionsSuite`.
Closes #27713 from MaxGekk/str-regexp-foldable-pattern-backport.
Authored-by: Maxim Gekk
Signed-off-by: Dongjoon Hyun
(cherry picked from commit cfc48a8a3068972791410e8e36ff9cf1ba5af445)
Signed-off-by: Dongjoon Hyun
---
.../apache/spark/sql/catalyst/expressions/regexpExpressions.scala | 2 +-
.../spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala | 8
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
index 7086e4d..a271be5 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
@@ -40,7 +40,7 @@ abstract class StringRegexExpression extends BinaryExpression
// try cache the pattern for Literal
private lazy val cache: Pattern = right match {
-case x @ Literal(value: String, StringType) => compile(value)
+case Literal(value: UTF8String, StringType) => compile(value.toString)
case _ => null
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala
index 4c7a037..f7a97db 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala
@@ -244,4 +244,12 @@ class RegexpExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
checkEvaluation(StringSplit(s1, s2), null, row3)
}
+ test("SPARK-30759: cache initialization for literal patterns") {
+val expr = "A" like Literal.create("a", StringType)
+expr.eval()
+val cache = expr.getClass.getSuperclass
+ .getDeclaredFields.filter(_.getName.endsWith("cache")).head
+cache.setAccessible(true)
+
assert(cache.get(expr).asInstanceOf[java.util.regex.Pattern].pattern().contains("a"))
+ }
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (99b0cad -> cfc48a8)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 99b0cad [SPARK-30918][SQL][FOLLOWUP] Fix typo in OptimizeSkewedJoin add cfc48a8 [SPARK-30759][SQL][3.0] Fix cache initialization in StringRegexExpression No new revisions were added by this update. Summary of changes: .../apache/spark/sql/catalyst/expressions/regexpExpressions.scala | 2 +- .../spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala | 8 2 files changed, 9 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (d88557f -> 99b0cad)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from d88557f [SPARK-27619][SQL] MapType should be prohibited in hash expressions add 99b0cad [SPARK-30918][SQL][FOLLOWUP] Fix typo in OptimizeSkewedJoin No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala| 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (d88557f -> 99b0cad)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from d88557f [SPARK-27619][SQL] MapType should be prohibited in hash expressions add 99b0cad [SPARK-30918][SQL][FOLLOWUP] Fix typo in OptimizeSkewedJoin No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala| 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c913b9d -> eced932)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c913b9d [SPARK-27619][SQL] MapType should be prohibited in hash expressions add eced932 [SPARK-30918][SQL][FOLLOWUP] Fix typo in OptimizeSkewedJoin No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala| 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (d88557f -> 99b0cad)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from d88557f [SPARK-27619][SQL] MapType should be prohibited in hash expressions add 99b0cad [SPARK-30918][SQL][FOLLOWUP] Fix typo in OptimizeSkewedJoin No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala| 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c913b9d -> eced932)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c913b9d [SPARK-27619][SQL] MapType should be prohibited in hash expressions add eced932 [SPARK-30918][SQL][FOLLOWUP] Fix typo in OptimizeSkewedJoin No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala| 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-27619][SQL] MapType should be prohibited in hash expressions
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new d88557f [SPARK-27619][SQL] MapType should be prohibited in hash
expressions
d88557f is described below
commit d88557fb3e27c721d2b4838d8a0eff280c243939
Author: iRakson
AuthorDate: Thu Feb 27 01:48:12 2020 +0800
[SPARK-27619][SQL] MapType should be prohibited in hash expressions
### What changes were proposed in this pull request?
`hash()` and `xxhash64()` cannot be used on elements of `Maptype`. A new
configuration `spark.sql.legacy.useHashOnMapType` is introduced to allow users
to restore the previous behaviour.
When `spark.sql.legacy.useHashOnMapType` is set to false:
```
scala> spark.sql("select hash(map())");
org.apache.spark.sql.AnalysisException: cannot resolve 'hash(map())' due to
data type mismatch: input to function hash cannot contain elements of MapType;
line 1 pos 7;
'Project [unresolvedalias(hash(map(), 42), None)]
+- OneRowRelation
```
when `spark.sql.legacy.useHashOnMapType` is set to true :
```
scala> spark.sql("set spark.sql.legacy.useHashOnMapType=true");
res3: org.apache.spark.sql.DataFrame = [key: string, value: string]
scala> spark.sql("select hash(map())").first()
res4: org.apache.spark.sql.Row = [42]
```
### Why are the changes needed?
As discussed in Jira, SparkSql's map hashcodes depends on their order of
insertion which is not consistent with the normal scala behaviour which might
confuse users.
Code snippet from JIRA :
```
val a = spark.createDataset(Map(1->1, 2->2) :: Nil)
val b = spark.createDataset(Map(2->2, 1->1) :: Nil)
// Demonstration of how Scala Map equality is unaffected by insertion order:
assert(Map(1->1, 2->2).hashCode() == Map(2->2, 1->1).hashCode())
assert(Map(1->1, 2->2) == Map(2->2, 1->1))
assert(a.first() == b.first())
// In contrast, this will print two different hashcodes:
println(Seq(a, b).map(_.selectExpr("hash(*)").first()))
```
Also `MapType` is prohibited for aggregation / joins / equality comparisons
#7819 and set operations #17236.
### Does this PR introduce any user-facing change?
Yes. Now users cannot use hash functions on elements of `mapType`. To
restore the previous behaviour set `spark.sql.legacy.useHashOnMapType` to true.
### How was this patch tested?
UT added.
Closes #27580 from iRakson/SPARK-27619.
Authored-by: iRakson
Signed-off-by: Wenchen Fan
(cherry picked from commit c913b9d8b5c2e125c441f7fe923e333cd8ac2c2c)
Signed-off-by: Wenchen Fan
---
docs/sql-migration-guide.md | 2 ++
.../apache/spark/sql/catalyst/expressions/hash.scala | 15 ---
.../scala/org/apache/spark/sql/internal/SQLConf.scala | 6 ++
.../catalyst/expressions/HashExpressionsSuite.scala | 14 --
.../scala/org/apache/spark/sql/SQLQuerySuite.scala| 19 +++
5 files changed, 39 insertions(+), 17 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 7e52e69..7e0a536 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -224,6 +224,8 @@ license: |
- Since Spark 3.0, when casting string value to integral types(tinyint,
smallint, int and bigint), datetime types(date, timestamp and interval) and
boolean type, the leading and trailing whitespaces (<= ASCII 32) will be
trimmed before converted to these type values, e.g. `cast(' 1\t' as int)`
results `1`, `cast(' 1\t' as boolean)` results `true`, `cast('2019-10-10\t as
date)` results the date value `2019-10-10`. In Spark version 2.4 and earlier,
while casting string to integrals and b [...]
+ - Since Spark 3.0, An analysis exception will be thrown when hash
expressions are applied on elements of MapType. To restore the behavior before
Spark 3.0, set `spark.sql.legacy.useHashOnMapType` to true.
+
- Since Spark 3.0, numbers written in scientific notation(e.g. `1E2`) would
be parsed as Double. In Spark version 2.4 and earlier, they're parsed as
Decimal. To restore the behavior before Spark 3.0, you can set
`spark.sql.legacy.exponentLiteralAsDecimal.enabled` to `true`.
- Since Spark 3.0, we pad decimal numbers with trailing zeros to the scale
of the column for `spark-sql` interface, for example:
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
index d8eed27..5be7f54 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
+++
[spark] branch master updated (7330547 -> c913b9d)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7330547 [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables add c913b9d [SPARK-27619][SQL] MapType should be prohibited in hash expressions No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 2 ++ .../apache/spark/sql/catalyst/expressions/hash.scala | 15 --- .../scala/org/apache/spark/sql/internal/SQLConf.scala | 6 ++ .../catalyst/expressions/HashExpressionsSuite.scala | 14 -- .../scala/org/apache/spark/sql/SQLQuerySuite.scala| 19 +++ 5 files changed, 39 insertions(+), 17 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7330547 -> c913b9d)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7330547 [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables add c913b9d [SPARK-27619][SQL] MapType should be prohibited in hash expressions No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 2 ++ .../apache/spark/sql/catalyst/expressions/hash.scala | 15 --- .../scala/org/apache/spark/sql/internal/SQLConf.scala | 6 ++ .../catalyst/expressions/HashExpressionsSuite.scala | 14 -- .../scala/org/apache/spark/sql/SQLQuerySuite.scala| 19 +++ 5 files changed, 39 insertions(+), 17 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 0759e5e [SPARK-30782][SQL] Column resolution doesn't respect current
catalog/namespace for v2 tables
0759e5e is described below
commit 0759e5e7e58a689810d063188c598a5747096895
Author: Terry Kim
AuthorDate: Thu Feb 27 00:21:38 2020 +0800
[SPARK-30782][SQL] Column resolution doesn't respect current
catalog/namespace for v2 tables
### What changes were proposed in this pull request?
This PR proposes to fix an issue where qualified columns are not matched
for v2 tables if current catalog/namespace are used.
For v1 tables, you can currently perform the following:
```SQL
SELECT default.t.id FROM t;
```
For v2 tables, the following fails:
```SQL
USE testcat.ns1.ns2;
SELECT testcat.ns1.ns2.t.id FROM t;
org.apache.spark.sql.AnalysisException: cannot resolve
'`testcat.ns1.ns2.t.id`' given input columns: [t.id, t.point]; line 1 pos 7;
```
### Why are the changes needed?
It is a bug since qualified column names cannot match if current
catalog/namespace are used.
### Does this PR introduce any user-facing change?
Yes, now the following works:
```SQL
USE testcat.ns1.ns2;
SELECT testcat.ns1.ns2.t.id FROM t;
```
### How was this patch tested?
Added new tests
Closes #27532 from imback82/qualifed_col_respect_current.
Authored-by: Terry Kim
Signed-off-by: Wenchen Fan
(cherry picked from commit 73305475c10f1218bd2060e8575ab4072d0cc50b)
Signed-off-by: Wenchen Fan
---
.../spark/sql/catalyst/analysis/Analyzer.scala | 8 ++--
.../spark/sql/connector/DataSourceV2SQLSuite.scala | 54 +++---
2 files changed, 42 insertions(+), 20 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index aec7174..3d79799 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -807,8 +807,10 @@ class Analyzer(
def apply(plan: LogicalPlan): LogicalPlan =
ResolveTempViews(plan).resolveOperatorsUp {
case u: UnresolvedRelation =>
lookupV2Relation(u.multipartIdentifier)
- .map(SubqueryAlias(u.multipartIdentifier, _))
- .getOrElse(u)
+ .map { rel =>
+val ident = rel.identifier.get
+SubqueryAlias(rel.catalog.get.name +: ident.namespace :+
ident.name, rel)
+ }.getOrElse(u)
case u @ UnresolvedTable(NonSessionCatalogAndIdentifier(catalog, ident))
=>
CatalogV2Util.loadTable(catalog, ident)
@@ -933,7 +935,7 @@ class Analyzer(
v1SessionCatalog.getRelation(v1Table.v1Table)
case table =>
SubqueryAlias(
-identifier,
+ident.asMultipartIdentifier,
DataSourceV2Relation.create(table, Some(catalog), Some(ident)))
}
val key = catalog.name +: ident.namespace :+ ident.name
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
index f642114..4ff2093 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
@@ -685,12 +685,21 @@ class DataSourceV2SQLSuite
sql(s"CREATE TABLE $t (id bigint, point struct)
USING foo")
sql(s"INSERT INTO $t VALUES (1, (10, 20))")
- checkAnswer(
-sql(s"SELECT testcat.ns1.ns2.tbl.id, testcat.ns1.ns2.tbl.point.x FROM
$t"),
-Row(1, 10))
- checkAnswer(sql(s"SELECT ns1.ns2.tbl.id, ns1.ns2.tbl.point.x FROM $t"),
Row(1, 10))
- checkAnswer(sql(s"SELECT ns2.tbl.id, ns2.tbl.point.x FROM $t"), Row(1,
10))
- checkAnswer(sql(s"SELECT tbl.id, tbl.point.x FROM $t"), Row(1, 10))
+ def check(tbl: String): Unit = {
+checkAnswer(
+ sql(s"SELECT testcat.ns1.ns2.tbl.id, testcat.ns1.ns2.tbl.point.x
FROM $tbl"),
+ Row(1, 10))
+checkAnswer(sql(s"SELECT ns1.ns2.tbl.id, ns1.ns2.tbl.point.x FROM
$tbl"), Row(1, 10))
+checkAnswer(sql(s"SELECT ns2.tbl.id, ns2.tbl.point.x FROM $tbl"),
Row(1, 10))
+checkAnswer(sql(s"SELECT tbl.id, tbl.point.x FROM $tbl"), Row(1, 10))
+ }
+
+ // Test with qualified table name "testcat.ns1.ns2.tbl".
+ check(t)
+
+ // Test if current catalog and namespace is respected in column
resolution.
+ sql("USE testcat.ns1.ns2")
+
[spark] branch master updated (020b262 -> 7330547)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 020b262 [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite add 7330547 [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/analysis/Analyzer.scala | 8 ++-- .../spark/sql/connector/DataSourceV2SQLSuite.scala | 54 +++--- 2 files changed, 42 insertions(+), 20 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (020b262 -> 7330547)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 020b262 [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite add 7330547 [SPARK-30782][SQL] Column resolution doesn't respect current catalog/namespace for v2 tables No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/analysis/Analyzer.scala | 8 ++-- .../spark/sql/connector/DataSourceV2SQLSuite.scala | 54 +++--- 2 files changed, 42 insertions(+), 20 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 3e1795b [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration
against the current session explicitly in HiveShowCreateTableSuite
3e1795b is described below
commit 3e1795b4aeba5a627dc3e45888cefbcccd1c8941
Author: HyukjinKwon
AuthorDate: Wed Feb 26 20:48:43 2020 +0900
[SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the
current session explicitly in HiveShowCreateTableSuite
### What changes were proposed in this pull request?
After https://github.com/apache/spark/pull/27387 (see
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/202/),
the tests below fail consistently, specifically in one job
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/
in Jenkins
```
org.apache.spark.sql.hive.HiveShowCreateTableSuite.simple hive table
org.apache.spark.sql.hive.HiveShowCreateTableSuite.simple external hive
table
org.apache.spark.sql.hive.HiveShowCreateTableSuite.hive bucketing is
supported
```
The profile is same as PR builder but seems it fails specifically in this
machine. Seems the legacy configuration
`spark.sql.legacy.createHiveTableByDefault.enabled` is not being set due to the
inconsistency between `SQLConf.get` and the active Spark session as described
in the https://github.com/apache/spark/pull/27387.
This PR proposes to explicitly set the configuration against the session
used instead of `SQLConf.get`.
### Why are the changes needed?
To make `spark-master-test-sbt-hadoop-2.7-hive-2.3` job pass.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Cannot reproduce in my local. Presumably it cannot be reproduced in the PR
builder. We should see if the tests pass at
`spark-master-test-sbt-hadoop-2.7-hive-2.3` job after this PR is merged
Closes #27703 from HyukjinKwon/SPARK-30798-followup.
Authored-by: HyukjinKwon
Signed-off-by: HyukjinKwon
(cherry picked from commit 020b2622e597458b925d7227ed5f9fa269f2d391)
Signed-off-by: HyukjinKwon
---
.../org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala | 9 +
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala
index 99db1e3..50c9018 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.hive
import org.apache.spark.sql.{AnalysisException, ShowCreateTableSuite}
import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat,
CatalogTable}
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
import org.apache.spark.sql.hive.test.TestHiveSingleton
import org.apache.spark.sql.internal.{HiveSerDe, SQLConf}
@@ -30,12 +30,13 @@ class HiveShowCreateTableSuite extends ShowCreateTableSuite
with TestHiveSinglet
protected override def beforeAll(): Unit = {
super.beforeAll()
origCreateHiveTableConfig =
- SQLConf.get.getConf(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED)
-SQLConf.get.setConf(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED,
true)
+ spark.conf.get(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED)
+spark.conf.set(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED.key,
true)
}
protected override def afterAll(): Unit = {
-SQLConf.get.setConf(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED,
+spark.conf.set(
+ SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED.key,
origCreateHiveTableConfig)
super.afterAll()
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (28b8713 -> 020b262)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 28b8713 [SPARK-30950][BUILD] Setting version to 3.1.0-SNAPSHOT add 020b262 [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala | 9 + 1 file changed, 5 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 3e1795b [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration
against the current session explicitly in HiveShowCreateTableSuite
3e1795b is described below
commit 3e1795b4aeba5a627dc3e45888cefbcccd1c8941
Author: HyukjinKwon
AuthorDate: Wed Feb 26 20:48:43 2020 +0900
[SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the
current session explicitly in HiveShowCreateTableSuite
### What changes were proposed in this pull request?
After https://github.com/apache/spark/pull/27387 (see
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/202/),
the tests below fail consistently, specifically in one job
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/
in Jenkins
```
org.apache.spark.sql.hive.HiveShowCreateTableSuite.simple hive table
org.apache.spark.sql.hive.HiveShowCreateTableSuite.simple external hive
table
org.apache.spark.sql.hive.HiveShowCreateTableSuite.hive bucketing is
supported
```
The profile is same as PR builder but seems it fails specifically in this
machine. Seems the legacy configuration
`spark.sql.legacy.createHiveTableByDefault.enabled` is not being set due to the
inconsistency between `SQLConf.get` and the active Spark session as described
in the https://github.com/apache/spark/pull/27387.
This PR proposes to explicitly set the configuration against the session
used instead of `SQLConf.get`.
### Why are the changes needed?
To make `spark-master-test-sbt-hadoop-2.7-hive-2.3` job pass.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Cannot reproduce in my local. Presumably it cannot be reproduced in the PR
builder. We should see if the tests pass at
`spark-master-test-sbt-hadoop-2.7-hive-2.3` job after this PR is merged
Closes #27703 from HyukjinKwon/SPARK-30798-followup.
Authored-by: HyukjinKwon
Signed-off-by: HyukjinKwon
(cherry picked from commit 020b2622e597458b925d7227ed5f9fa269f2d391)
Signed-off-by: HyukjinKwon
---
.../org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala | 9 +
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala
index 99db1e3..50c9018 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.hive
import org.apache.spark.sql.{AnalysisException, ShowCreateTableSuite}
import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat,
CatalogTable}
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
import org.apache.spark.sql.hive.test.TestHiveSingleton
import org.apache.spark.sql.internal.{HiveSerDe, SQLConf}
@@ -30,12 +30,13 @@ class HiveShowCreateTableSuite extends ShowCreateTableSuite
with TestHiveSinglet
protected override def beforeAll(): Unit = {
super.beforeAll()
origCreateHiveTableConfig =
- SQLConf.get.getConf(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED)
-SQLConf.get.setConf(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED,
true)
+ spark.conf.get(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED)
+spark.conf.set(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED.key,
true)
}
protected override def afterAll(): Unit = {
-SQLConf.get.setConf(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED,
+spark.conf.set(
+ SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED.key,
origCreateHiveTableConfig)
super.afterAll()
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (28b8713 -> 020b262)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 28b8713 [SPARK-30950][BUILD] Setting version to 3.1.0-SNAPSHOT add 020b262 [SPARK-30798][SQL][TESTS][FOLLOW-UP] Set the configuration against the current session explicitly in HiveShowCreateTableSuite No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala | 9 + 1 file changed, 5 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
