[spark] branch branch-3.1 updated: [SPARK-34392][SQL] Support ZoneOffset +h:mm in DateTimeUtils. getZoneId
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new daeae50 [SPARK-34392][SQL] Support ZoneOffset +h:mm in DateTimeUtils.
getZoneId
daeae50 is described below
commit daeae5095a6202bfc7afa19cafde6c4b86a3613c
Author: ShiKai Wang
AuthorDate: Fri Feb 26 11:03:20 2021 -0600
[SPARK-34392][SQL] Support ZoneOffset +h:mm in DateTimeUtils. getZoneId
### What changes were proposed in this pull request?
To support +8:00 in Spark3 when execute sql
`select to_utc_timestamp("2020-02-07 16:00:00", "GMT+8:00")`
### Why are the changes needed?
+8:00 this format is supported in PostgreSQL,hive, presto, but not
supported in Spark3
https://issues.apache.org/jira/browse/SPARK-34392
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
unit test
Closes #31624 from Karl-WangSK/zone.
Lead-authored-by: ShiKai Wang
Co-authored-by: Karl-WangSK
Signed-off-by: Sean Owen
---
.../org/apache/spark/sql/catalyst/util/DateTimeUtils.scala | 5 -
.../apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala | 13 +
.../scala/org/apache/spark/sql/internal/SQLConfSuite.scala | 5 ++---
3 files changed, 19 insertions(+), 4 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
index 87cf3c9..89cb67c 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
@@ -50,7 +50,10 @@ object DateTimeUtils {
val TIMEZONE_OPTION = "timeZone"
- def getZoneId(timeZoneId: String): ZoneId = ZoneId.of(timeZoneId,
ZoneId.SHORT_IDS)
+ def getZoneId(timeZoneId: String): ZoneId = {
+// To support the (+|-)h:mm format because it was supported before Spark
3.0.
+ZoneId.of(timeZoneId.replaceFirst("(\\+|\\-)(\\d):", "$10$2:"),
ZoneId.SHORT_IDS)
+ }
def getTimeZone(timeZoneId: String): TimeZone =
TimeZone.getTimeZone(getZoneId(timeZoneId))
/**
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
index 3d841f3..fb2d511 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
@@ -471,6 +471,13 @@ class DateTimeUtilsSuite extends SparkFunSuite with
Matchers with SQLHelper {
test("2011-12-25 09:00:00.123456", JST.getId, "2011-12-25
18:00:00.123456")
test("2011-12-25 09:00:00.123456", LA.getId, "2011-12-25
01:00:00.123456")
test("2011-12-25 09:00:00.123456", "Asia/Shanghai", "2011-12-25
17:00:00.123456")
+test("2011-12-25 09:00:00.123456", "-7", "2011-12-25 02:00:00.123456")
+test("2011-12-25 09:00:00.123456", "+8:00", "2011-12-25
17:00:00.123456")
+test("2011-12-25 09:00:00.123456", "+8:00:00", "2011-12-25
17:00:00.123456")
+test("2011-12-25 09:00:00.123456", "+0800", "2011-12-25
17:00:00.123456")
+test("2011-12-25 09:00:00.123456", "-071020", "2011-12-25
01:49:40.123456")
+test("2011-12-25 09:00:00.123456", "-07:10:20", "2011-12-25
01:49:40.123456")
+
}
}
@@ -496,6 +503,12 @@ class DateTimeUtilsSuite extends SparkFunSuite with
Matchers with SQLHelper {
test("2011-12-25 18:00:00.123456", JST.getId, "2011-12-25
09:00:00.123456")
test("2011-12-25 01:00:00.123456", LA.getId, "2011-12-25
09:00:00.123456")
test("2011-12-25 17:00:00.123456", "Asia/Shanghai", "2011-12-25
09:00:00.123456")
+test("2011-12-25 02:00:00.123456", "-7", "2011-12-25 09:00:00.123456")
+test("2011-12-25 17:00:00.123456", "+8:00", "2011-12-25
09:00:00.123456")
+test("2011-12-25 17:00:00.123456", "+8:00:00", "2011-12-25
09:00:00.123456")
+test("2011-12-25 17:00:00.123456", "+0800", "2011-12-25
09:00:00.123456")
+test("2011-12-25 01:49:40.123456", "-071020", "2011-12-25
09:00:00.123456")
+test("
[spark-website] branch asf-site updated: Fix a sbt example for generating dependency graphs
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 0732fba Fix a sbt example for generating dependency graphs 0732fba is described below commit 0732fbafce4c92449c759a880d8b0885e243d5f3 Author: Takeshi Yamamuro AuthorDate: Sat Feb 27 07:23:49 2021 -0600 Fix a sbt example for generating dependency graphs This PR intends to fix an error below; ``` $./build/sbt dependency-tree [error] Not a valid command: dependency-tree [error] Not a valid project ID: dependency-tree [error] Expected ':' [error] Not a valid key: dependency-tree (similar: dependencyTree, dependencyOverrides, sbtDependency) [error] dependency-tree [error]^ ``` Author: Takeshi Yamamuro Closes #318 from maropu/DepTree. --- developer-tools.md| 2 +- site/developer-tools.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/developer-tools.md b/developer-tools.md index bf13ed3..3b929ac 100644 --- a/developer-tools.md +++ b/developer-tools.md @@ -374,7 +374,7 @@ $ git checkout origin/pr/112 -b new-branch ``` $ # sbt -$ build/sbt dependency-tree +$ build/sbt dependencyTree $ # Maven $ build/mvn -DskipTests install diff --git a/site/developer-tools.html b/site/developer-tools.html index 8b38f34..2f0d3bb 100644 --- a/site/developer-tools.html +++ b/site/developer-tools.html @@ -547,7 +547,7 @@ $ git checkout origin/pr/112 -b new-branch Generating Dependency Graphs $ # sbt -$ build/sbt dependency-tree +$ build/sbt dependencyTree $ # Maven $ build/mvn -DskipTests install - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Capture Jekyll document generation errors in the github action
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/spark-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6f878f5 Capture Jekyll document generation errors in the github action
6f878f5 is described below
commit 6f878f5f9a36f415de2136a17505e2a63e730b07
Author: attilapiros
AuthorDate: Fri Feb 26 16:34:45 2021 -0600
Capture Jekyll document generation errors in the github action
For testing purpose I have reverted a fix where a conflict was resolved
(temporary change).
Warnings are filtered out as these lines are coming form Ruby:
```
/var/lib/gems/2.7.0/gems/bundler-1.17.2/lib/bundler/rubygems_integration.rb:200:
warning: constant Gem::ConfigMap is deprecated
/var/lib/gems/2.7.0/gems/bundler-1.17.2/lib/bundler/rubygems_integration.rb:200:
warning: constant Gem::ConfigMap is deprecated
```
Author: attilapiros
Closes #316 from attilapiros/capture-jekyll-generation-errors.
---
.github/workflows/doc_gen.yml | 10 +-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/doc_gen.yml b/.github/workflows/doc_gen.yml
index 58c36b8..720b235 100644
--- a/.github/workflows/doc_gen.yml
+++ b/.github/workflows/doc_gen.yml
@@ -27,7 +27,15 @@ jobs:
run: |
export LC_ALL=C.UTF-8
export LANG=C.UTF-8
-bundle exec jekyll build
+OLD_IFS=$IFS
+IFS=
+GEN_ERRORS=$(bundle exec jekyll build 3>&2 2>&1 1>&3)
+if [ $(echo $GEN_ERRORS| grep -v -e '^$'| grep -c -v
"rubygems_integration") -ne 0 ]; then
+ echo "Error during document generation:"
+ echo $GEN_ERRORS
+ exit 1
+fi
+IFS=$OLD_IFS
CHANGED_FILE=( $(git ls-files --modified --other --exclude-standard
--directory | grep -v sitemap.xml) )
if [ ${#CHANGED_FILE[@]} -ne 0 ]; then
echo "Not all documentation was generated and/or not the right
Jekyll version was used! Modified / untracked files (excluding sitemap.xml):"
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8d68f3f -> 56e664c)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8d68f3f [MINOR] Add more known translations of contributors add 56e664c [SPARK-34392][SQL] Support ZoneOffset +h:mm in DateTimeUtils. getZoneId No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/catalyst/util/DateTimeUtils.scala | 5 - .../apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala | 13 + .../scala/org/apache/spark/sql/internal/SQLConfSuite.scala | 5 ++--- 3 files changed, 19 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Remove search-hadoop.com link
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new ce394da Remove search-hadoop.com link ce394da is described below commit ce394dab9e1a6cfdeaf85cf211c51d08fe10dfc0 Author: Sean Owen AuthorDate: Tue Feb 23 12:03:30 2021 -0600 Remove search-hadoop.com link Author: Sean Owen Closes #311 from srowen/SearchHadoop. --- contributing.md| 3 +-- site/contributing.html | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/contributing.md b/contributing.md index 5c68d98..43854b6 100644 --- a/contributing.md +++ b/contributing.md @@ -157,8 +157,7 @@ lists first, rather than consider filing a JIRA or proposing a change. When in d `u...@spark.apache.org` first about the possible change - Search the `u...@spark.apache.org` and `d...@spark.apache.org` mailing list archives for -related discussions. Use http://search-hadoop.com/?q=_project=Spark;>search-hadoop.com -or similar search tools. +related discussions. Often, the problem has been discussed before, with a resolution that doesn't require a code change, or recording what kinds of changes will not be accepted as a resolution. - Search JIRA for existing issues: diff --git a/site/contributing.html b/site/contributing.html index 0c99606..fc86824 100644 --- a/site/contributing.html +++ b/site/contributing.html @@ -362,8 +362,7 @@ lists first, rather than consider filing a JIRA or proposing a change. When in d u...@spark.apache.org first about the possible change Search the u...@spark.apache.org and d...@spark.apache.org mailing list archives for -related discussions. Use http://search-hadoop.com/?q=fc_project=Spark;>search-hadoop.com -or similar search tools. +related discussions. Often, the problem has been discussed before, with a resolution that doesnt require a code change, or recording what kinds of changes will not be accepted as a resolution. Search JIRA for existing issues: - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9942548 -> 94f9617)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9942548 [SPARK-34487][K8S][TESTS] Use the runtime Hadoop version in K8s IT add 94f9617 [SPARK-34129][SQL] Add table name to LogicalRelation.simpleString No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/datasources/LogicalRelation.scala | 3 ++- sql/core/src/test/resources/sql-tests/results/explain-aqe.sql.out | 4 ++-- sql/core/src/test/resources/sql-tests/results/explain-cbo.sql.out | 4 ++-- sql/core/src/test/resources/sql-tests/results/explain.sql.out | 4 ++-- .../org/apache/spark/sql/execution/QueryExecutionSuite.scala | 8 5 files changed, 16 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated: [SPARK-20977][CORE] Use a non-final field for the state of CollectionAccumulator
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 093d4e2 [SPARK-20977][CORE] Use a non-final field for the state of
CollectionAccumulator
093d4e2 is described below
commit 093d4e275ed7b39057020e60ac1daa8e89fb9ebe
Author: Gera Shegalov
AuthorDate: Sat Feb 20 20:57:14 2021 -0600
[SPARK-20977][CORE] Use a non-final field for the state of
CollectionAccumulator
This PR is a fix for the JLS 17.5.3 violation identified in
zsxwing's [19/Feb/19 11:47
comment](https://issues.apache.org/jira/browse/SPARK-20977?focusedCommentId=16772277=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16772277)
on the JIRA.
### What changes were proposed in this pull request?
- Use a var field to hold the state of the collection accumulator
### Why are the changes needed?
AccumulatorV2 auto-registration of accumulator during readObject doesn't
work with final fields that are post-processed outside readObject. As it stands
incompletely initialized objects are published to heartbeat thread. This leads
to sporadic exceptions knocking out executors which increases the cost of the
jobs. We observe such failures on a regular basis
https://github.com/NVIDIA/spark-rapids/issues/1522.
### Does this PR introduce _any_ user-facing change?
None
### How was this patch tested?
- this is a concurrency bug that is almost impossible to reproduce as a
quick unit test.
- By trial and error I crafted a command
https://github.com/NVIDIA/spark-rapids/pull/1688 that reproduces the issue on
my dev box several times per hour, with the first occurrence often within a few
minutes. After the patch, these Exceptions have not shown up after running
overnight for 10+ hours
- existing unit tests in *`AccumulatorV2Suite` and *`LiveEntitySuite`
Closes #31540 from gerashegalov/SPARK-20977.
Authored-by: Gera Shegalov
Signed-off-by: Sean Owen
(cherry picked from commit fadd0f5d9bff79cbd785631aa2962b9eda644ab8)
Signed-off-by: Sean Owen
---
.../org/apache/spark/util/AccumulatorV2.scala | 33 +-
1 file changed, 20 insertions(+), 13 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala
b/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala
index d5b3ce3..1453840 100644
--- a/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala
+++ b/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala
@@ -19,7 +19,7 @@ package org.apache.spark.util
import java.{lang => jl}
import java.io.ObjectInputStream
-import java.util.{ArrayList, Collections}
+import java.util.ArrayList
import java.util.concurrent.ConcurrentHashMap
import java.util.concurrent.atomic.AtomicLong
@@ -449,39 +449,46 @@ class DoubleAccumulator extends AccumulatorV2[jl.Double,
jl.Double] {
* @since 2.0.0
*/
class CollectionAccumulator[T] extends AccumulatorV2[T, java.util.List[T]] {
- private val _list: java.util.List[T] = Collections.synchronizedList(new
ArrayList[T]())
+ private var _list: java.util.List[T] = _
+
+ private def getOrCreate = {
+_list = Option(_list).getOrElse(new java.util.ArrayList[T]())
+_list
+ }
/**
* Returns false if this accumulator instance has any values in it.
*/
- override def isZero: Boolean = _list.isEmpty
+ override def isZero: Boolean = this.synchronized(getOrCreate.isEmpty)
override def copyAndReset(): CollectionAccumulator[T] = new
CollectionAccumulator
override def copy(): CollectionAccumulator[T] = {
val newAcc = new CollectionAccumulator[T]
-_list.synchronized {
- newAcc._list.addAll(_list)
+this.synchronized {
+ newAcc.getOrCreate.addAll(getOrCreate)
}
newAcc
}
- override def reset(): Unit = _list.clear()
+ override def reset(): Unit = this.synchronized {
+_list = null
+ }
- override def add(v: T): Unit = _list.add(v)
+ override def add(v: T): Unit = this.synchronized(getOrCreate.add(v))
override def merge(other: AccumulatorV2[T, java.util.List[T]]): Unit = other
match {
-case o: CollectionAccumulator[T] => _list.addAll(o.value)
+case o: CollectionAccumulator[T] =>
this.synchronized(getOrCreate.addAll(o.value))
case _ => throw new UnsupportedOperationException(
s"Cannot merge ${this.getClass.getName} with ${other.getClass.getName}")
}
- override def value: java.util.List[T] = _list.synchronized {
-java.util.Collections.unmodifiableList(new ArrayList[T](_list))
+ override def value: java.util.List[T] = this.synchronized {
+java.util.Collections.unmodifiableList(new ArrayList[T](getOrCreate))
}
- private[spark] def setValue(newValue: java.util.List[T]): Unit = {
-_list.cle
[spark] branch master updated (7de49a8 -> fadd0f5)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7de49a8 [SPARK-34481][SQL] Refactor dataframe reader/writer optionsWithPath logic add fadd0f5 [SPARK-20977][CORE] Use a non-final field for the state of CollectionAccumulator No new revisions were added by this update. Summary of changes: .../org/apache/spark/util/AccumulatorV2.scala | 33 +- 1 file changed, 20 insertions(+), 13 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-7768][CORE][SQL] Open UserDefinedType as a Developer API
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f78466d [SPARK-7768][CORE][SQL] Open UserDefinedType as a Developer
API
f78466d is described below
commit f78466dca6f0ddb1c979842f5a22e1a1e3b535bf
Author: Sean Owen
AuthorDate: Sat Feb 20 07:32:06 2021 -0600
[SPARK-7768][CORE][SQL] Open UserDefinedType as a Developer API
### What changes were proposed in this pull request?
UserDefinedType and UDTRegistration become public Developer APIs, not
package-private to Spark.
### Why are the changes needed?
This proposes to simply open up the UserDefinedType class as a developer
API. It was public in 1.x, but closed in 2.x for some possible redesign that
does not seem to have happened.
Other libraries have managed to define UDTs anyway by inserting shims into
the Spark namespace, and this evidently has worked OK. But package isolation in
Java 9+ breaks this.
The logic here is mostly: this is de facto a stable API, so can at least be
open to developers with the usual caveats about developer APIs.
Open questions:
- Is there in fact some important redesign that's needed before opening it?
The comment to this effect is from 2016
- Is this all that needs to be opened up? Like PythonUserDefinedType?
- Should any of this be kept package-private?
This was first proposed in https://github.com/apache/spark/pull/16478
though it was a larger change, but, the other API issues it was fixing seem to
have been addressed already (e.g. no need to return internal Spark types). It
was never really reviewed.
My hunch is that there isn't much downside, and some upside, to just
opening this as-is now.
### Does this PR introduce _any_ user-facing change?
UserDefinedType becomes visible to developers to subclass.
### How was this patch tested?
Existing tests; there is no change to the existing logic.
Closes #31461 from srowen/SPARK-7768.
Authored-by: Sean Owen
Signed-off-by: Sean Owen
---
mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala| 2 +-
mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala| 2 +-
.../main/scala/org/apache/spark/sql/types/UDTRegistration.scala| 3 ++-
.../main/scala/org/apache/spark/sql/types/UserDefinedType.scala| 7 +++
4 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala
b/mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala
index f4a8556..838b51a 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala
@@ -22,7 +22,7 @@ import
org.apache.spark.sql.catalyst.expressions.{GenericInternalRow, UnsafeArra
import org.apache.spark.sql.types._
/**
- * User-defined type for [[Matrix]] in [[mllib-local]] which allows easy
interaction with SQL
+ * User-defined type for [[Matrix]] in mllib-local which allows easy
interaction with SQL
* via [[org.apache.spark.sql.Dataset]].
*/
private[spark] class MatrixUDT extends UserDefinedType[Matrix] {
diff --git a/mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala
b/mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala
index 35bbaf5..8c81aca 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala
@@ -22,7 +22,7 @@ import
org.apache.spark.sql.catalyst.expressions.{GenericInternalRow, UnsafeArra
import org.apache.spark.sql.types._
/**
- * User-defined type for [[Vector]] in [[mllib-local]] which allows easy
interaction with SQL
+ * User-defined type for [[Vector]] in mllib-local which allows easy
interaction with SQL
* via [[org.apache.spark.sql.Dataset]].
*/
private[spark] class VectorUDT extends UserDefinedType[Vector] {
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/types/UDTRegistration.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/types/UDTRegistration.scala
index 20ec75c..f13651f 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/types/UDTRegistration.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/types/UDTRegistration.scala
@@ -20,6 +20,7 @@ package org.apache.spark.sql.types
import scala.collection.mutable
import org.apache.spark.SparkException
+import org.apache.spark.annotation.DeveloperApi
import org.apache.spark.internal.Logging
import org.apache.spark.util.Utils
@@ -29,7 +30,7 @@ import org.apache.spark.util.Utils
* However, by doing this, we add SparkSQL dependency on user classes. This
object provides
* alternative approach to register UDTs for user classes.
*/
-private[spark
[spark] branch master updated (ff5115c -> 2787328)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ff5115c [SPARK-33739][SQL] Jobs committed through the S3A Magic committer don't track bytes add 2787328 [SPARK-32703][SQL] Replace deprecated API calls from SpecificParquetRecordReaderBase No new revisions were added by this update. Summary of changes: .../parquet/SpecificParquetRecordReaderBase.java | 93 +++--- .../datasources/parquet/ParquetFileFormat.scala| 10 +-- .../v2/parquet/ParquetPartitionReaderFactory.scala | 18 ++--- 3 files changed, 36 insertions(+), 85 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f2e1468 -> e3b6e4a)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f2e1468 [SPARK-34428][BUILD] Update sbt version to 1.4.7 add e3b6e4a [SPARK-33434][PYTHON][DOCS] Added RuntimeConfig to PySpark docs No new revisions were added by this update. Summary of changes: python/docs/source/reference/pyspark.sql.rst | 11 +++ python/pyspark/sql/session.py| 4 2 files changed, 15 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e65b28c -> 37fe8c6)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e65b28c [SPARK-34352][SQL] Improve SQLQueryTestSuite so as could run on windows system add 37fe8c6 [SPARK-34395][SQL] Clean up unused code for code simplifications No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/StringExpressionsSuite.scala | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3b26bc2 -> 777d51e)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3b26bc2 [SPARK-34168][SQL] Support DPP in AQE when the join is Broadcast hash join at the beginning add 777d51e [SPARK-34374][SQL][DSTREAM] Use standard methods to extract keys or values from a Map No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/execution/datasources/rules.scala | 3 +-- .../apache/spark/streaming/scheduler/ReceiverSchedulingPolicy.scala| 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [MINOR][SQL][FOLLOW-UP] Add assertion to FixedLengthRowBasedKeyValueBatch
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d1131bc [MINOR][SQL][FOLLOW-UP] Add assertion to
FixedLengthRowBasedKeyValueBatch
d1131bc is described below
commit d1131bc85028ea0f78ac9ef73bba731080f1ff6a
Author: yliou
AuthorDate: Mon Feb 8 08:46:01 2021 -0600
[MINOR][SQL][FOLLOW-UP] Add assertion to FixedLengthRowBasedKeyValueBatch
### What changes were proposed in this pull request?
Adds an assert to `FixedLengthRowBasedKeyValueBatch#appendRow` method to
check the incoming vlen and klen by comparing them with the lengths stored as
member variables as followup to https://github.com/apache/spark/pull/30788
### Why are the changes needed?
Add assert statement to catch similar bugs in future.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Ran some tests locally, though not easy to test.
Closes #31447 from yliou/SPARK-33726-Assert.
Authored-by: yliou
Signed-off-by: Sean Owen
---
.../sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java | 2 ++
1 file changed, 2 insertions(+)
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java
index df52f9c..25400be 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java
@@ -46,6 +46,8 @@ public final class FixedLengthRowBasedKeyValueBatch extends
RowBasedKeyValueBatc
public UnsafeRow appendRow(Object kbase, long koff, int klen,
Object vbase, long voff, int vlen) {
// if run out of max supported rows or page size, return null
+assert(vlen == this.vlen);
+assert(klen == this.klen);
if (numRows >= capacity || page == null || page.size() - pageCursor <
recordLength) {
return null;
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c73f70b -> 178dc50)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c73f70b [SPARK-34141][SQL] Remove side effect from ExtractGenerator add 178dc50 [SPARK-34356][ML] OVR transform fix potential column conflict No new revisions were added by this update. Summary of changes: .../apache/spark/ml/classification/OneVsRest.scala | 58 -- .../spark/ml/classification/OneVsRestSuite.scala | 7 +++ 2 files changed, 28 insertions(+), 37 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated: [PYTHON][MINOR] Fix docstring of DataFrame.join
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 6a9 [PYTHON][MINOR] Fix docstring of DataFrame.join
6a9 is described below
commit 6a9a773ebffa74e3dd9bde3feec176e0af6e
Author: Xinrong Meng
AuthorDate: Sat Feb 6 09:08:49 2021 -0600
[PYTHON][MINOR] Fix docstring of DataFrame.join
### What changes were proposed in this pull request?
Fix docstring of PySpark `DataFrame.join`.
### Why are the changes needed?
For a better view of PySpark documentation.
### Does this PR introduce _any_ user-facing change?
No (only documentation changes).
### How was this patch tested?
Manual test.
From
To
Closes #31463 from xinrong-databricks/fixDoc.
Authored-by: Xinrong Meng
Signed-off-by: Sean Owen
(cherry picked from commit 747ad1809b4026aae4a7bedec2cac485bddcd5f2)
Signed-off-by: Sean Owen
---
python/pyspark/sql/dataframe.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/python/pyspark/sql/dataframe.py b/python/pyspark/sql/dataframe.py
index fe7d26d..6313474 100644
--- a/python/pyspark/sql/dataframe.py
+++ b/python/pyspark/sql/dataframe.py
@@ -1297,6 +1297,7 @@ class DataFrame(PandasMapOpsMixin, PandasConversionMixin):
Examples
The following performs a full outer join between ``df1`` and ``df2``.
+
>>> from pyspark.sql.functions import desc
>>> df.join(df2, df.name == df2.name, 'outer').select(df.name,
df2.height) \
.sort(desc("name")).collect()
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [PYTHON][MINOR] Fix docstring of DataFrame.join
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 747ad18 [PYTHON][MINOR] Fix docstring of DataFrame.join
747ad18 is described below
commit 747ad1809b4026aae4a7bedec2cac485bddcd5f2
Author: Xinrong Meng
AuthorDate: Sat Feb 6 09:08:49 2021 -0600
[PYTHON][MINOR] Fix docstring of DataFrame.join
### What changes were proposed in this pull request?
Fix docstring of PySpark `DataFrame.join`.
### Why are the changes needed?
For a better view of PySpark documentation.
### Does this PR introduce _any_ user-facing change?
No (only documentation changes).
### How was this patch tested?
Manual test.
From
To
Closes #31463 from xinrong-databricks/fixDoc.
Authored-by: Xinrong Meng
Signed-off-by: Sean Owen
---
python/pyspark/sql/dataframe.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/python/pyspark/sql/dataframe.py b/python/pyspark/sql/dataframe.py
index e09e87c6..4d606c6 100644
--- a/python/pyspark/sql/dataframe.py
+++ b/python/pyspark/sql/dataframe.py
@@ -1299,6 +1299,7 @@ class DataFrame(PandasMapOpsMixin, PandasConversionMixin):
Examples
The following performs a full outer join between ``df1`` and ``df2``.
+
>>> from pyspark.sql.functions import desc
>>> df.join(df2, df.name == df2.name, 'outer').select(df.name,
df2.height) \
.sort(desc("name")).collect()
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Remove not working link from third-party projects
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 6a5fc2c Remove not working link from third-party projects 6a5fc2c is described below commit 6a5fc2ccaa5ad648dc0b25575ff816c10e648bdf Author: Valentina AuthorDate: Fri Feb 5 13:25:07 2021 -0600 Remove not working link from third-party projects The referenced website with a blog post is not anymore available. Some other blog posts could replace this link, but I think the best solution might be to reference the git project, where the documentation is up to date. Author: Valentina Closes #302 from valan4ik/remove-sjs-blog-link. --- site/news/index.html | 5 +++-- site/third-party-projects.html | 4 +--- third-party-projects.md| 4 +--- 3 files changed, 5 insertions(+), 8 deletions(-) diff --git a/site/news/index.html b/site/news/index.html index 429e8ec..3c56c1a 100644 --- a/site/news/index.html +++ b/site/news/index.html @@ -208,8 +208,9 @@ January 7, 2021 The next official Spark release is Spark 3.1.1 instead of Spark 3.1.0. -There was an accident during Spark 3.1.0 RC1 preparation, -see http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Spark-3-1-0-RC1-td30524.html;>[VOTE] Release Spark 3.1.0 (RC1) in the Spark dev mailing list. +There was a technical issue during Spark 3.1.0 RC1 preparation, +see https://www.mail-archive.com/dev@spark.apache.org/msg27133.html;>[VOTE] Release Spark 3.1.0 (RC1) in the Spark dev mailing list. + diff --git a/site/third-party-projects.html b/site/third-party-projects.html index 79a35ce..b7f572b 100644 --- a/site/third-party-projects.html +++ b/site/third-party-projects.html @@ -220,9 +220,7 @@ Apache Spark. You can add a package as long as you have a GitHub repository. https://github.com/spark-jobserver/spark-jobserver;>REST Job Server for Apache Spark - -REST interface for managing and submitting Spark jobs on the same cluster -(see http://engineering.ooyala.com/blog/open-sourcing-our-spark-job-server;>blog post -for details) +REST interface for managing and submitting Spark jobs on the same cluster. http://mlbase.org/;>MLbase - Machine Learning research project on top of Spark https://mesos.apache.org/;>Apache Mesos - Cluster management system that supports running Spark diff --git a/third-party-projects.md b/third-party-projects.md index 993ac5c..95449ca 100644 --- a/third-party-projects.md +++ b/third-party-projects.md @@ -26,9 +26,7 @@ Apache Spark. You can add a package as long as you have a GitHub repository. Infrastructure Projects - https://github.com/spark-jobserver/spark-jobserver;>REST Job Server for Apache Spark - -REST interface for managing and submitting Spark jobs on the same cluster -(see http://engineering.ooyala.com/blog/open-sourcing-our-spark-job-server;>blog post -for details) +REST interface for managing and submitting Spark jobs on the same cluster. - http://mlbase.org/;>MLbase - Machine Learning research project on top of Spark - https://mesos.apache.org/;>Apache Mesos - Cluster management system that supports running Spark - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-34310][CORE][SQL] Replaces map and flatten with flatMap
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9db566a [SPARK-34310][CORE][SQL] Replaces map and flatten with flatMap
9db566a is described below
commit 9db566a8821c02427434c551ee6e4d2501563dfa
Author: yangjie01
AuthorDate: Mon Feb 1 08:21:35 2021 -0600
[SPARK-34310][CORE][SQL] Replaces map and flatten with flatMap
### What changes were proposed in this pull request?
Replaces `collection.map(f1).flatten(f2)` with `collection.flatMap` if
possible. it's semantically consistent, but looks simpler.
### Why are the changes needed?
Code Simpilefications.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass the Jenkins or GitHub Action
Closes #31416 from LuciferYang/SPARK-34310.
Authored-by: yangjie01
Signed-off-by: Sean Owen
---
core/src/main/scala/org/apache/spark/rdd/SubtractedRDD.scala | 2 +-
.../spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala | 2 +-
.../spark/sql/execution/datasources/HadoopFileLinesReaderSuite.scala | 4 ++--
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/rdd/SubtractedRDD.scala
b/core/src/main/scala/org/apache/spark/rdd/SubtractedRDD.scala
index d5a811d..1dec977 100644
--- a/core/src/main/scala/org/apache/spark/rdd/SubtractedRDD.scala
+++ b/core/src/main/scala/org/apache/spark/rdd/SubtractedRDD.scala
@@ -124,7 +124,7 @@ private[spark] class SubtractedRDD[K: ClassTag, V:
ClassTag, W: ClassTag](
integrate(0, t => getSeq(t._1) += t._2)
// the second dep is rdd2; remove all of its keys
integrate(1, t => map.remove(t._1))
-map.asScala.iterator.map(t => t._2.iterator.map((t._1, _))).flatten
+map.asScala.iterator.flatMap(t => t._2.iterator.map((t._1, _)))
}
override def clearDependencies(): Unit = {
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala
index aa2610d..188435d 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala
@@ -292,7 +292,7 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
// Final aggregate
val operators = expressions.map { e =>
val af = e.aggregateFunction
-val condition =
e.filter.map(distinctAggFilterAttrLookup.get(_)).flatten
+val condition = e.filter.flatMap(distinctAggFilterAttrLookup.get)
val naf = if (af.children.forall(_.foldable)) {
// If aggregateFunction's children are all foldable, we only put
the first child in
// distinctAggGroups. So here we only need to rewrite the first
child to
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/HadoopFileLinesReaderSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/HadoopFileLinesReaderSuite.scala
index 3c83388..771ddbd 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/HadoopFileLinesReaderSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/HadoopFileLinesReaderSuite.scala
@@ -36,13 +36,13 @@ class HadoopFileLinesReaderSuite extends SharedSparkSession
{
val delimOpt = delimiter.map(_.getBytes(StandardCharsets.UTF_8))
Files.write(path.toPath, text.getBytes(StandardCharsets.UTF_8))
-val lines = ranges.map { case (start, length) =>
+val lines = ranges.flatMap { case (start, length) =>
val file = PartitionedFile(InternalRow.empty, path.getCanonicalPath,
start, length)
val hadoopConf = conf.getOrElse(spark.sessionState.newHadoopConf())
val reader = new HadoopFileLinesReader(file, delimOpt, hadoopConf)
reader.map(_.toString)
-}.flatten
+}
lines
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (15445a8 -> 9ed0e3c)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 15445a8 [SPARK-34275][CORE][SQL][MLLIB] Replaces filter and size with count add 9ed0e3c [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/Dataset.scala | 16 + .../spark/sql/execution/stat/StatFunctions.scala | 4 .../org/apache/spark/sql/DataFrameSuite.scala | 28 ++ 3 files changed, 48 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (28131a7 -> 116f4ca)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 28131a7 [SPARK-34190][DOCS] Supplement the description for Python Package Management add 116f4ca [SPARK-34094][SQL] Extends StringTranslate to support unicode characters whose code point >= U+1 No new revisions were added by this update. Summary of changes: .../org/apache/spark/unsafe/types/UTF8String.java | 18 -- .../apache/spark/unsafe/types/UTF8StringSuite.java | 20 .../catalyst/expressions/stringExpressions.scala | 28 +++--- .../expressions/StringExpressionsSuite.scala | 7 ++ 4 files changed, 48 insertions(+), 25 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (dee596e -> d8cbef1)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from dee596e [SPARK-34027][SQL] Refresh cache in `ALTER TABLE .. RECOVER PARTITIONS` add d8cbef1 [SPARK-34093][ML] param maxDepth should check upper bound No new revisions were added by this update. Summary of changes: mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala | 5 +++-- python/pyspark/ml/tree.py | 3 ++- 2 files changed, 5 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-34123][WEB UI] optimize spark history summary page loading
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ebd8bc9 [SPARK-34123][WEB UI] optimize spark history summary page
loading
ebd8bc9 is described below
commit ebd8bc934de9d6aec53beb4ab60c998052038fad
Author: mohan3d
AuthorDate: Sun Jan 17 14:37:28 2021 -0600
[SPARK-34123][WEB UI] optimize spark history summary page loading
### What changes were proposed in this pull request?
Display history server entries using datatables instead of Mustache +
Datatables which proved to be faster and non-blocking for the webpage while
searching (using search bar in the page)
### Why are the changes needed?
Small changes in the attempts (entries) and removed part of HTML (Mustache
template).
### Does this PR introduce _any_ user-facing change?
Not very sure, but it's not supposed to change the way the page looks
rather it changes how entries are displayed.
### How was this patch tested?
Running test, since it's not adding new functionality.
Closes #31191 from mohan3d/feat/history-server-ui-optimization.
Lead-authored-by: mohan3d
Co-authored-by: Author: mohan3d
Signed-off-by: Sean Owen
---
.../spark/ui/static/historypage-template.html | 20
.../org/apache/spark/ui/static/historypage.js | 54 --
2 files changed, 41 insertions(+), 33 deletions(-)
diff --git
a/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
b/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
index 7e9927d..5427125 100644
---
a/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
+++
b/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
@@ -75,26 +75,6 @@
- {{#applications}}
-
- {{#attempts}}
- {{version}}
- {{id}}
- {{name}}
- {{#hasMultipleAttempts}}
- {{attemptId}}
- {{/hasMultipleAttempts}}
- {{startTime}}
- {{#showCompletedColumns}}
- {{endTime}}
- {{duration}}
- {{/showCompletedColumns}}
- {{sparkUser}}
- {{lastUpdated}}
- Download
- {{/attempts}}
-
- {{/applications}}
diff --git a/core/src/main/resources/org/apache/spark/ui/static/historypage.js
b/core/src/main/resources/org/apache/spark/ui/static/historypage.js
index 3a4c815..aa542a7 100644
--- a/core/src/main/resources/org/apache/spark/ui/static/historypage.js
+++ b/core/src/main/resources/org/apache/spark/ui/static/historypage.js
@@ -140,9 +140,13 @@ $(document).ready(function() {
(attempt.hasOwnProperty("attemptId") ? attempt["attemptId"] + "/"
: "") + "logs";
attempt["durationMillisec"] = attempt["duration"];
attempt["duration"] = formatDuration(attempt["duration"]);
- var hasAttemptId = attempt.hasOwnProperty("attemptId");
- var app_clone = {"id" : id, "name" : name, "version": version,
"hasAttemptId" : hasAttemptId, "attempts" : [attempt]};
- array.push(app_clone);
+ attempt["id"] = id;
+ attempt["name"] = name;
+ attempt["version"] = version;
+ attempt["attemptUrl"] = uiRoot + "/history/" + id + "/" +
+(attempt.hasOwnProperty("attemptId") ? attempt["attemptId"] + "/"
: "") + "jobs/";
+
+ array.push(attempt);
}
}
if(array.length < 20) {
@@ -165,17 +169,41 @@ $(document).ready(function() {
var completedColumnName = 'completed';
var durationColumnName = 'duration';
var conf = {
+ "data": array,
"columns": [
-{name: 'version'},
-{name: 'appId', type: "appid-numeric"},
-{name: 'appName'},
-{name: attemptIdColumnName},
-{name: startedColumnName},
-{name: completedColumnName},
-{name: durationColumnName, type: "title-numeric"},
-{name: 'user'},
-{name: 'lastUpdated'},
-{name: 'eventLog'},
+{name: 'version', data: 'version' },
+{
+ name: 'appId',
+ type: "appid-numeric",
+ data: 'id',
+ render: (id, type, row) => `${id}`
+},
+{name: 'appName', data: 'name' },
+{
+ name: attemptIdColumnName,
+ data: 'attemptId',
+ render: (attemptId, type, row) => (attemptId ? `${attemptId}` : '')
+
[spark] branch branch-3.1 updated: [SPARK-32598][SCHEDULER] Fix missing driver logs under UI App-Executors tab in standalone cluster mode
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new cbb20fd [SPARK-32598][SCHEDULER] Fix missing driver logs under UI
App-Executors tab in standalone cluster mode
cbb20fd is described below
commit cbb20fdad101984a83afebc4288611f4fdb543ac
Author: KevinSmile
AuthorDate: Fri Jan 15 09:01:26 2021 -0600
[SPARK-32598][SCHEDULER] Fix missing driver logs under UI App-Executors tab
in standalone cluster mode
### What changes were proposed in this pull request?
Fix [SPARK-32598] (missing driver logs under
UI-ApplicationDetails-Executors tab in standalone cluster mode) .
The direct bug is: the original author forgot to implement
`getDriverLogUrls` in `StandaloneSchedulerBackend`
https://github.com/apache/spark/blob/1de272f98d0ff22d0dd151797f22b8faf310963a/core/src/main/scala/org/apache/spark/scheduler/SchedulerBackend.scala#L70-L75
So we set DriverLogUrls as env in `DriverRunner`, and retrieve it at
`StandaloneSchedulerBackend`.
### Why are the changes needed?
Fix bug [SPARK-32598].
### Does this PR introduce _any_ user-facing change?
Yes. User will see driver logs (standalone cluster mode) under
UI-ApplicationDetails-Executors tab now.
Before:
After:
### How was this patch tested?
Re-check the real case in [SPARK-32598] and found this user-facing bug
fixed.
Closes #29644 from KevinSmile/kw-dev-master.
Authored-by: KevinSmile
Signed-off-by: Sean Owen
(cherry picked from commit c75c29dcaa9458a9ce0dd7a4d5fafbffb4b7f6a6)
Signed-off-by: Sean Owen
---
.../scala/org/apache/spark/deploy/worker/DriverRunner.scala | 11 +++
.../main/scala/org/apache/spark/deploy/worker/Worker.scala| 1 +
.../spark/scheduler/cluster/StandaloneSchedulerBackend.scala | 8
.../test/scala/org/apache/spark/deploy/DeployTestUtils.scala | 1 +
.../org/apache/spark/deploy/worker/DriverRunnerTest.scala | 3 ++-
5 files changed, 23 insertions(+), 1 deletion(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
b/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
index 4f9c497..1c3aa3a 100644
--- a/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
@@ -33,9 +33,11 @@ import org.apache.spark.deploy.master.DriverState
import org.apache.spark.deploy.master.DriverState.DriverState
import org.apache.spark.internal.Logging
import org.apache.spark.internal.config.{DRIVER_RESOURCES_FILE,
SPARK_DRIVER_PREFIX}
+import org.apache.spark.internal.config.UI.UI_REVERSE_PROXY
import org.apache.spark.internal.config.Worker.WORKER_DRIVER_TERMINATE_TIMEOUT
import org.apache.spark.resource.ResourceInformation
import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.ui.UIUtils
import org.apache.spark.util.{Clock, ShutdownHookManager, SystemClock, Utils}
/**
@@ -50,6 +52,7 @@ private[deploy] class DriverRunner(
val driverDesc: DriverDescription,
val worker: RpcEndpointRef,
val workerUrl: String,
+val workerWebUiUrl: String,
val securityManager: SecurityManager,
val resources: Map[String, ResourceInformation] = Map.empty)
extends Logging {
@@ -190,6 +193,14 @@ private[deploy] class DriverRunner(
val builder =
CommandUtils.buildProcessBuilder(driverDesc.command.copy(javaOpts = javaOpts),
securityManager, driverDesc.mem, sparkHome.getAbsolutePath,
substituteVariables)
+// add WebUI driver log url to environment
+val reverseProxy = conf.get(UI_REVERSE_PROXY)
+val workerUrlRef = UIUtils.makeHref(reverseProxy, driverId, workerWebUiUrl)
+builder.environment.put("SPARK_DRIVER_LOG_URL_STDOUT",
+ s"$workerUrlRef/logPage?driverId=$driverId=stdout")
+builder.environment.put("SPARK_DRIVER_LOG_URL_STDERR",
+ s"$workerUrlRef/logPage?driverId=$driverId=stderr")
+
runDriver(builder, driverDir, driverDesc.supervise)
}
diff --git a/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
b/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
index a6092f6..fd14c15 100755
--- a/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
@@ -651,6 +651,7 @@ private[deploy] class Worker(
driverDesc.copy(command =
Worker.maybeUpdateSSLSettings(driverDesc.command, conf)),
self,
[spark] branch master updated: [SPARK-32598][SCHEDULER] Fix missing driver logs under UI App-Executors tab in standalone cluster mode
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c75c29d [SPARK-32598][SCHEDULER] Fix missing driver logs under UI
App-Executors tab in standalone cluster mode
c75c29d is described below
commit c75c29dcaa9458a9ce0dd7a4d5fafbffb4b7f6a6
Author: KevinSmile
AuthorDate: Fri Jan 15 09:01:26 2021 -0600
[SPARK-32598][SCHEDULER] Fix missing driver logs under UI App-Executors tab
in standalone cluster mode
### What changes were proposed in this pull request?
Fix [SPARK-32598] (missing driver logs under
UI-ApplicationDetails-Executors tab in standalone cluster mode) .
The direct bug is: the original author forgot to implement
`getDriverLogUrls` in `StandaloneSchedulerBackend`
https://github.com/apache/spark/blob/1de272f98d0ff22d0dd151797f22b8faf310963a/core/src/main/scala/org/apache/spark/scheduler/SchedulerBackend.scala#L70-L75
So we set DriverLogUrls as env in `DriverRunner`, and retrieve it at
`StandaloneSchedulerBackend`.
### Why are the changes needed?
Fix bug [SPARK-32598].
### Does this PR introduce _any_ user-facing change?
Yes. User will see driver logs (standalone cluster mode) under
UI-ApplicationDetails-Executors tab now.
Before:
After:
### How was this patch tested?
Re-check the real case in [SPARK-32598] and found this user-facing bug
fixed.
Closes #29644 from KevinSmile/kw-dev-master.
Authored-by: KevinSmile
Signed-off-by: Sean Owen
---
.../scala/org/apache/spark/deploy/worker/DriverRunner.scala | 11 +++
.../main/scala/org/apache/spark/deploy/worker/Worker.scala| 1 +
.../spark/scheduler/cluster/StandaloneSchedulerBackend.scala | 8
.../test/scala/org/apache/spark/deploy/DeployTestUtils.scala | 1 +
.../org/apache/spark/deploy/worker/DriverRunnerTest.scala | 3 ++-
5 files changed, 23 insertions(+), 1 deletion(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
b/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
index 776d916..6945cb5 100644
--- a/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
@@ -33,9 +33,11 @@ import org.apache.spark.deploy.master.DriverState
import org.apache.spark.deploy.master.DriverState.DriverState
import org.apache.spark.internal.Logging
import org.apache.spark.internal.config.{DRIVER_RESOURCES_FILE,
SPARK_DRIVER_PREFIX}
+import org.apache.spark.internal.config.UI.UI_REVERSE_PROXY
import org.apache.spark.internal.config.Worker.WORKER_DRIVER_TERMINATE_TIMEOUT
import org.apache.spark.resource.ResourceInformation
import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.ui.UIUtils
import org.apache.spark.util.{Clock, ShutdownHookManager, SystemClock, Utils}
/**
@@ -50,6 +52,7 @@ private[deploy] class DriverRunner(
val driverDesc: DriverDescription,
val worker: RpcEndpointRef,
val workerUrl: String,
+val workerWebUiUrl: String,
val securityManager: SecurityManager,
val resources: Map[String, ResourceInformation] = Map.empty)
extends Logging {
@@ -189,6 +192,14 @@ private[deploy] class DriverRunner(
val builder =
CommandUtils.buildProcessBuilder(driverDesc.command.copy(javaOpts = javaOpts),
securityManager, driverDesc.mem, sparkHome.getAbsolutePath,
substituteVariables)
+// add WebUI driver log url to environment
+val reverseProxy = conf.get(UI_REVERSE_PROXY)
+val workerUrlRef = UIUtils.makeHref(reverseProxy, driverId, workerWebUiUrl)
+builder.environment.put("SPARK_DRIVER_LOG_URL_STDOUT",
+ s"$workerUrlRef/logPage?driverId=$driverId=stdout")
+builder.environment.put("SPARK_DRIVER_LOG_URL_STDERR",
+ s"$workerUrlRef/logPage?driverId=$driverId=stderr")
+
runDriver(builder, driverDir, driverDesc.supervise)
}
diff --git a/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
b/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
index a3c7375..cb36207 100755
--- a/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
@@ -652,6 +652,7 @@ private[deploy] class Worker(
driverDesc.copy(command =
Worker.maybeUpdateSSLSettings(driverDesc.command, conf)),
self,
workerUri,
+workerWebUiUrl,
securityMgr,
resources_)
drivers(driverId) = driv
[spark] branch branch-3.0 updated: [SPARK-32598][SCHEDULER] Fix missing driver logs under UI App-Executors tab in standalone cluster mode
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 70fa108 [SPARK-32598][SCHEDULER] Fix missing driver logs under UI
App-Executors tab in standalone cluster mode
70fa108 is described below
commit 70fa108494a13e939d0aafc0491e30488dbe1caa
Author: KevinSmile
AuthorDate: Fri Jan 15 09:01:26 2021 -0600
[SPARK-32598][SCHEDULER] Fix missing driver logs under UI App-Executors tab
in standalone cluster mode
### What changes were proposed in this pull request?
Fix [SPARK-32598] (missing driver logs under
UI-ApplicationDetails-Executors tab in standalone cluster mode) .
The direct bug is: the original author forgot to implement
`getDriverLogUrls` in `StandaloneSchedulerBackend`
https://github.com/apache/spark/blob/1de272f98d0ff22d0dd151797f22b8faf310963a/core/src/main/scala/org/apache/spark/scheduler/SchedulerBackend.scala#L70-L75
So we set DriverLogUrls as env in `DriverRunner`, and retrieve it at
`StandaloneSchedulerBackend`.
### Why are the changes needed?
Fix bug [SPARK-32598].
### Does this PR introduce _any_ user-facing change?
Yes. User will see driver logs (standalone cluster mode) under
UI-ApplicationDetails-Executors tab now.
Before:
After:
### How was this patch tested?
Re-check the real case in [SPARK-32598] and found this user-facing bug
fixed.
Closes #29644 from KevinSmile/kw-dev-master.
Authored-by: KevinSmile
Signed-off-by: Sean Owen
(cherry picked from commit c75c29dcaa9458a9ce0dd7a4d5fafbffb4b7f6a6)
Signed-off-by: Sean Owen
---
.../scala/org/apache/spark/deploy/worker/DriverRunner.scala | 11 +++
.../main/scala/org/apache/spark/deploy/worker/Worker.scala| 1 +
.../spark/scheduler/cluster/StandaloneSchedulerBackend.scala | 8
.../test/scala/org/apache/spark/deploy/DeployTestUtils.scala | 1 +
.../org/apache/spark/deploy/worker/DriverRunnerTest.scala | 3 ++-
5 files changed, 23 insertions(+), 1 deletion(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
b/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
index 53ec7b3..2caee18 100644
--- a/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala
@@ -33,9 +33,11 @@ import org.apache.spark.deploy.master.DriverState
import org.apache.spark.deploy.master.DriverState.DriverState
import org.apache.spark.internal.Logging
import org.apache.spark.internal.config.{DRIVER_RESOURCES_FILE,
SPARK_DRIVER_PREFIX}
+import org.apache.spark.internal.config.UI.UI_REVERSE_PROXY
import org.apache.spark.internal.config.Worker.WORKER_DRIVER_TERMINATE_TIMEOUT
import org.apache.spark.resource.ResourceInformation
import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.ui.UIUtils
import org.apache.spark.util.{Clock, ShutdownHookManager, SystemClock, Utils}
/**
@@ -50,6 +52,7 @@ private[deploy] class DriverRunner(
val driverDesc: DriverDescription,
val worker: RpcEndpointRef,
val workerUrl: String,
+val workerWebUiUrl: String,
val securityManager: SecurityManager,
val resources: Map[String, ResourceInformation] = Map.empty)
extends Logging {
@@ -190,6 +193,14 @@ private[deploy] class DriverRunner(
val builder =
CommandUtils.buildProcessBuilder(driverDesc.command.copy(javaOpts = javaOpts),
securityManager, driverDesc.mem, sparkHome.getAbsolutePath,
substituteVariables)
+// add WebUI driver log url to environment
+val reverseProxy = conf.get(UI_REVERSE_PROXY)
+val workerUrlRef = UIUtils.makeHref(reverseProxy, driverId, workerWebUiUrl)
+builder.environment.put("SPARK_DRIVER_LOG_URL_STDOUT",
+ s"$workerUrlRef/logPage?driverId=$driverId=stdout")
+builder.environment.put("SPARK_DRIVER_LOG_URL_STDERR",
+ s"$workerUrlRef/logPage?driverId=$driverId=stderr")
+
runDriver(builder, driverDir, driverDesc.supervise)
}
diff --git a/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
b/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
index 73cac880..08fdd0a 100755
--- a/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
@@ -628,6 +628,7 @@ private[deploy] class Worker(
driverDesc.copy(command =
Worker.maybeUpdateSSLSettings(driverDesc.command, conf)),
self,
[spark] branch master updated (6cd0092 -> 9e33d49)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6cd0092 Revert "[SPARK-34064][SQL] Cancel the running broadcast sub-jobs when SQL statement is cancelled" add 9e33d49 [SPARK-33346][CORE][SQL][MLLIB][DSTREAM][K8S] Change the never changed 'var' to 'val' No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 2 +- .../scala/org/apache/spark/scheduler/TaskSetManager.scala| 2 +- core/src/main/scala/org/apache/spark/status/LiveEntity.scala | 2 +- .../scala/org/apache/spark/util/collection/MedianHeap.scala | 4 ++-- .../src/test/scala/org/apache/spark/SharedSparkContext.scala | 2 +- core/src/test/scala/org/apache/spark/SparkConfSuite.scala| 2 +- core/src/test/scala/org/apache/spark/SparkContextSuite.scala | 6 +++--- .../apache/spark/deploy/history/ApplicationCacheSuite.scala | 2 +- .../spark/deploy/history/EventLogFileReadersSuite.scala | 2 +- .../scala/org/apache/spark/deploy/master/MasterSuite.scala | 2 +- .../spark/executor/CoarseGrainedExecutorBackendSuite.scala | 8 .../apache/spark/resource/ResourceProfileManagerSuite.scala | 2 +- .../scala/org/apache/spark/resource/ResourceUtilsSuite.scala | 2 +- .../spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala | 8 .../scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala | 2 +- .../org/apache/spark/scheduler/TaskSchedulerImplSuite.scala | 4 ++-- .../org/apache/spark/security/CryptoStreamUtilsSuite.scala | 4 ++-- .../org/apache/spark/status/ElementTrackingStoreSuite.scala | 4 ++-- .../org/apache/spark/storage/FlatmapIteratorSuite.scala | 4 ++-- .../scala/org/apache/spark/storage/MemoryStoreSuite.scala| 2 +- .../scala/org/apache/spark/util/ClosureCleanerSuite.scala| 12 ++-- .../org/apache/spark/util/random/RandomSamplerSuite.scala| 4 ++-- .../main/scala/org/apache/spark/examples/LocalKMeans.scala | 2 +- .../org/apache/spark/streaming/kafka010/KafkaRDDSuite.scala | 2 +- .../apache/spark/mllib/linalg/EigenValueDecomposition.scala | 2 +- .../spark/ml/clustering/PowerIterationClusteringSuite.scala | 4 ++-- .../apache/spark/sql/execution/joins/HashedRelation.scala| 2 +- .../sql/execution/streaming/sources/ForeachWriterTable.scala | 2 +- .../sql/streaming/test/DataStreamReaderWriterSuite.scala | 8 .../apache/spark/sql/test/DataFrameReaderWriterSuite.scala | 2 +- .../org/apache/spark/sql/hive/execution/HiveQuerySuite.scala | 2 +- .../org/apache/spark/streaming/LocalStreamingContext.scala | 2 +- .../scala/org/apache/spark/streaming/MasterFailureTest.scala | 2 +- .../apache/spark/streaming/ReceivedBlockHandlerSuite.scala | 2 +- .../spark/streaming/scheduler/ReceiverTrackerSuite.scala | 2 +- 35 files changed, 58 insertions(+), 58 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-33690][SQL][FOLLOWUP] Escape further meta-characters in showString
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b7da108 [SPARK-33690][SQL][FOLLOWUP] Escape further meta-characters
in showString
b7da108 is described below
commit b7da108cae2d354b972b0825e8b0bae1d5d300e5
Author: Kousuke Saruta
AuthorDate: Wed Jan 13 18:13:01 2021 -0600
[SPARK-33690][SQL][FOLLOWUP] Escape further meta-characters in showString
### What changes were proposed in this pull request?
This is a followup PR for SPARK-33690 (#30647) .
In addition to the original PR, this PR intends to escape the following
meta-characters in `Dataset#showString`.
* `\r` (carrige ret)
* `\f` (form feed)
* `\b` (backspace)
* `\u000B` (vertical tab)
* `\u0007` (bell)
### Why are the changes needed?
To avoid breaking the layout of `Dataset#showString`.
`\u0007` does not break the layout of `Dataset#showString` but it's noisy
(beeps for each row) so it should be also escaped.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Modified the existing tests.
I also build the documents and check the generated html for
`sql-migration-guide.md`.
Closes #31144 from sarutak/escape-metacharacters-in-getRows.
Authored-by: Kousuke Saruta
Signed-off-by: Sean Owen
---
docs/sql-migration-guide.md| 9 +++-
.../main/scala/org/apache/spark/sql/Dataset.scala | 14 +-
.../org/apache/spark/sql/DataFrameSuite.scala | 50 +++---
3 files changed, 46 insertions(+), 27 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 8cf1a9c..0245321 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -26,7 +26,14 @@ license: |
- In Spark 3.2, `spark.sql.adaptive.enabled` is enabled by default. To
restore the behavior before Spark 3.2, you can set `spark.sql.adaptive.enabled`
to `false`.
- - In Spark 3.2, the meta-characters `\n` and `\t` are escaped in the
`show()` action. In Spark 3.1 or earlier, the two metacharacters are output as
it is.
+ - In Spark 3.2, the following meta-characters are escaped in the `show()`
action. In Spark 3.1 or earlier, the following metacharacters are output as it
is.
+* `\n` (new line)
+* `\r` (carrige ret)
+* `\t` (horizontal tab)
+* `\f` (form feed)
+* `\b` (backspace)
+* `\u000B` (vertical tab)
+* `\u0007` (bell)
- In Spark 3.2, `ALTER TABLE .. RENAME TO PARTITION` throws
`PartitionAlreadyExistsException` instead of `AnalysisException` for tables
from Hive external when the target partition already exists.
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
index 5c27359..f959079 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
@@ -300,17 +300,27 @@ class Dataset[T] private[sql](
}
val data = newDf.select(castCols: _*).take(numRows + 1)
+def escapeMetaCharacters(str: String): String = {
+ str.replaceAll("\n", "n")
+.replaceAll("\r", "r")
+.replaceAll("\t", "t")
+.replaceAll("\f", "f")
+.replaceAll("\b", "b")
+.replaceAll("\u000B", "v")
+.replaceAll("\u0007", "a")
+}
+
// For array values, replace Seq and Array with square brackets
// For cells that are beyond `truncate` characters, replace it with the
// first `truncate-3` and "..."
-schema.fieldNames.toSeq +: data.map { row =>
+schema.fieldNames.map(escapeMetaCharacters).toSeq +: data.map { row =>
row.toSeq.map { cell =>
val str = cell match {
case null => "null"
case binary: Array[Byte] =>
binary.map("%02X".format(_)).mkString("[", " ", "]")
case _ =>
// Escapes meta-characters not to break the `showString` format
-cell.toString.replaceAll("\n", "n").replaceAll("\t", "t")
+escapeMetaCharacters(cell.toString)
}
if (truncate > 0 && str.length > truncate) {
// do not show ellipses for strings shorter than 4 characters.
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index d777cd4..13f8fa7 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql
[spark] branch master updated (8b1ba23 -> 62d8466)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8b1ba23 [SPARK-34068][CORE][SQL][MLLIB][GRAPHX] Remove redundant collection conversion add 62d8466 [SPARK-34051][SQL] Support 32-bit unicode escape in string literals No new revisions were added by this update. Summary of changes: .../apache/spark/sql/catalyst/parser/ParserUtils.scala | 18 +- .../spark/sql/catalyst/parser/ParserUtilsSuite.scala | 5 + 2 files changed, 22 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8c5fecd -> 8b1ba23)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8c5fecd [SPARK-34070][CORE][SQL] Replaces find and emptiness check with exists add 8b1ba23 [SPARK-34068][CORE][SQL][MLLIB][GRAPHX] Remove redundant collection conversion No new revisions were added by this update. Summary of changes: .../org/apache/spark/executor/ExecutorMetricsSource.scala | 2 +- .../org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala | 4 ++-- .../spark/streaming/kafka010/DirectKafkaInputDStream.scala | 2 +- .../main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala | 2 +- .../src/main/scala/org/apache/spark/ml/linalg/Vectors.scala| 2 +- .../apache/spark/mllib/clustering/GaussianMixtureModel.scala | 2 +- .../scala/org/apache/spark/mllib/clustering/LDAModel.scala | 8 .../src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala | 2 +- .../org/apache/spark/mllib/util/LinearDataGenerator.scala | 2 +- mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala | 2 +- .../scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala | 2 +- .../org/apache/spark/sql/catalyst/parser/AstBuilder.scala | 10 +- .../src/main/scala/org/apache/spark/sql/types/DataType.scala | 2 +- .../apache/spark/sql/execution/datasources/DataSource.scala| 2 +- .../org/apache/spark/sql/execution/python/EvaluatePython.scala | 2 +- .../apache/spark/sql/hive/thriftserver/ReflectionUtils.scala | 2 +- .../src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala | 2 +- 17 files changed, 25 insertions(+), 25 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bd5039f -> 8c5fecd)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bd5039f [SPARK-33741][CORE] Add min threshold time speculation config add 8c5fecd [SPARK-34070][CORE][SQL] Replaces find and emptiness check with exists No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/storage/BlockManagerDecommissioner.scala | 2 +- .../main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala | 2 +- .../scala/org/apache/spark/sql/execution/datasources/DataSource.scala | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-33991][CORE][WEBUI] Repair enumeration conversion error for AllJobsPage
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 1495ad8 [SPARK-33991][CORE][WEBUI] Repair enumeration conversion
error for AllJobsPage
1495ad8 is described below
commit 1495ad8c46197916527236331b57dce93aa3b8ec
Author: yikf <13468507...@163.com>
AuthorDate: Mon Jan 11 08:48:02 2021 -0600
[SPARK-33991][CORE][WEBUI] Repair enumeration conversion error for
AllJobsPage
### What changes were proposed in this pull request?
For `AllJobsPage `class, `AllJobsPage` gets the schedulingMode of
enumerated type by loading the `spark.scheduler.mode `configuration from
Sparkconf, but an enumeration conversion error occurs when I set the value of
this configuration to lowercase.
The reason for this problem is that the value of the `SchedulingMode
`enumeration class is uppercase, which occurs when I configure
`spark.scheduler.mode` to be lowercase.
I saw that the `#org.apache.spark.scheduler.TaskSchedulerImpl` class
convert the `spark. scheduler.mode` value to uppercase, so I think it should be
converted in `AllJobsPage `as well.
### Why are the changes needed?
An enumerated conversion error occurred with Spark when I set the value of
this configuration to lowercase.
### How was this patch tested?
Existing tests.
Closes #31015 from yikf/master.
Authored-by: yikf <13468507...@163.com>
Signed-off-by: Sean Owen
---
.../org/apache/spark/ui/jobs/AllJobsPage.scala | 6 --
.../scala/org/apache/spark/ui/UISeleniumSuite.scala | 21 +
2 files changed, 25 insertions(+), 2 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
b/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
index cfe15eb..94821e4 100644
--- a/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
@@ -19,7 +19,7 @@ package org.apache.spark.ui.jobs
import java.net.URLEncoder
import java.nio.charset.StandardCharsets.UTF_8
-import java.util.Date
+import java.util.{Date, Locale}
import javax.servlet.http.HttpServletRequest
import scala.collection.mutable.ListBuffer
@@ -277,15 +277,17 @@ private[ui] class AllJobsPage(parent: JobsTab, store:
AppStatusStore) extends We
s"${appSummary.numCompletedJobs}, only showing ${completedJobs.size}"
}
+// SPARK-33991 Avoid enumeration conversion error.
val schedulingMode = store.environmentInfo().sparkProperties.toMap
.get(SCHEDULER_MODE.key)
- .map { mode => SchedulingMode.withName(mode).toString }
+ .map { mode =>
SchedulingMode.withName(mode.toUpperCase(Locale.ROOT)).toString }
.getOrElse("Unknown")
val summary: NodeSeq =
+
User:
{parent.getSparkUser}
diff --git a/core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala
b/core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala
index d7caeaa..d10260e 100644
--- a/core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala
+++ b/core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala
@@ -123,6 +123,27 @@ class UISeleniumSuite extends SparkFunSuite with
WebBrowser with Matchers with B
sc
}
+ test("all jobs page should be rendered even though we configure the
scheduling mode to fair") {
+// Regression test for SPARK-33991
+val conf = Map("spark.scheduler.mode" -> "fair")
+withSpark(newSparkContext(additionalConfs = conf)) { sc =>
+ val rdd = sc.parallelize(0 to 100, 100).repartition(10).cache()
+ rdd.count()
+
+ eventually(timeout(5.seconds), interval(50.milliseconds)) {
+goToUi(sc, "/jobs")
+// The completed jobs table should have one row. The first row will be
the most recent job:
+val firstRow = find(cssSelector("tbody tr")).get.underlying
+val firstRowColumns = firstRow.findElements(By.tagName("td"))
+// if first row can get the id 0, then the page is rendered and the
scheduling mode is
+// displayed with no error when we visit http://localhost:4040/jobs/
even though
+// we configure the scheduling mode like spark.scheduler.mode=fair
+// instead of spark.scheculer.mode=FAIR
+firstRowColumns.get(0).getText should be ("0")
+ }
+}
+ }
+
test("effects of unpersist() / persist() should be reflected") {
// Regression test for SPARK-2527
withSpark(newSparkContext()) { sc =>
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (023eba2 -> 0781ed4)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 023eba2 [SPARK-33796][DOCS][FOLLOWUP] Tweak the width of left-menu of Spark SQL Guide add 0781ed4 [MINOR][SQL][TESTS] Fix the incorrect unicode escape test in ParserUtilsSuite No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/sql/catalyst/parser/ParserUtilsSuite.scala | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated: [SPARK-33796][DOCS][FOLLOWUP] Tweak the width of left-menu of Spark SQL Guide
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new d124af5 [SPARK-33796][DOCS][FOLLOWUP] Tweak the width of left-menu of
Spark SQL Guide
d124af5 is described below
commit d124af57ddcb3fb41cd36c6d56fb28c12e745979
Author: Kousuke Saruta
AuthorDate: Fri Jan 8 09:43:11 2021 -0600
[SPARK-33796][DOCS][FOLLOWUP] Tweak the width of left-menu of Spark SQL
Guide
### What changes were proposed in this pull request?
This PR tweaks the width of left-menu of Spark SQL Guide.
When I view the Spark SQL Guide with browsers on macOS, the title `Spark
SQL Guide` looks prettily.
But I often use Pop!_OS, an Ubuntu variant, and the title is overlapped
with browsers on it.
After this change, the title is no longer overlapped.
### Why are the changes needed?
For the pretty layout.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Built the document with `cd docs && SKIP_API=1 jekyll build` and confirmed
the layout.
Closes #31091 from sarutak/modify-layout-sparksql-guide.
Authored-by: Kousuke Saruta
Signed-off-by: Sean Owen
(cherry picked from commit 023eba2ad72f5119350c6c797808dadcfd1eaa19)
Signed-off-by: Sean Owen
---
docs/css/main.css | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/css/main.css b/docs/css/main.css
index 309ad7b..6710b6e 100755
--- a/docs/css/main.css
+++ b/docs/css/main.css
@@ -326,7 +326,7 @@ a.anchorjs-link:hover { text-decoration: none; }
border-left-width: 0px;
border-bottom-width: 0px;
margin-top: 0px;
- width: 210px;
+ width: 220px;
height: 80%;
float: left;
position: fixed;
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (157b72a -> 023eba2)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 157b72a [SPARK-33591][SQL] Recognize `null` in partition spec values add 023eba2 [SPARK-33796][DOCS][FOLLOWUP] Tweak the width of left-menu of Spark SQL Guide No new revisions were added by this update. Summary of changes: docs/css/main.css | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33398] Fix loading tree models prior to Spark 3.0
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 9f1bf4e [SPARK-33398] Fix loading tree models prior to Spark 3.0 9f1bf4e is described below commit 9f1bf4e47c4978be7d55cfadb7da6b7863942bc8 Author: Ruifeng Zheng AuthorDate: Sun Jan 3 11:52:46 2021 -0600 [SPARK-33398] Fix loading tree models prior to Spark 3.0 ### What changes were proposed in this pull request? In https://github.com/apache/spark/pull/21632/files#diff-0fdae8a6782091746ed20ea43f77b639f9c6a5f072dd2f600fcf9a7b37db4f47, a new field `rawCount` was added into `NodeData`, which cause that a tree model trained in 2.4 can not be loaded in 3.0/3.1/master; field `rawCount` is only used in training, and not used in `transform`/`predict`/`featureImportance`. So I just set it to -1L. ### Why are the changes needed? to support load old tree model in 3.0/3.1/master ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? added testsuites Closes #30889 from zhengruifeng/fix_tree_load. Authored-by: Ruifeng Zheng Signed-off-by: Sean Owen (cherry picked from commit 6b7527e381591bcd51be205853aea3e349893139) Signed-off-by: Sean Owen --- .../org/apache/spark/ml/tree/treeModels.scala | 48 ++--- .../ml-models/dtc-2.4.7/data/._SUCCESS.crc | Bin 0 -> 8 bytes ...-406c-894c-ca4eac67c690-c000.snappy.parquet.crc | Bin 0 -> 36 bytes .../resources/ml-models/dtc-2.4.7/data/_SUCCESS| 0 ...c890-406c-894c-ca4eac67c690-c000.snappy.parquet | Bin 0 -> 3242 bytes .../ml-models/dtc-2.4.7/metadata/._SUCCESS.crc | Bin 0 -> 8 bytes .../ml-models/dtc-2.4.7/metadata/.part-0.crc | Bin 0 -> 16 bytes .../ml-models/dtc-2.4.7/metadata/_SUCCESS | 0 .../ml-models/dtc-2.4.7/metadata/part-0| 1 + .../ml-models/dtr-2.4.7/data/._SUCCESS.crc | Bin 0 -> 8 bytes ...-4b3d-84af-d861adcb9ca8-c000.snappy.parquet.crc | Bin 0 -> 36 bytes .../resources/ml-models/dtr-2.4.7/data/_SUCCESS| 0 ...a437-4b3d-84af-d861adcb9ca8-c000.snappy.parquet | Bin 0 -> 3264 bytes .../ml-models/dtr-2.4.7/metadata/._SUCCESS.crc | Bin 0 -> 8 bytes .../ml-models/dtr-2.4.7/metadata/.part-0.crc | Bin 0 -> 12 bytes .../ml-models/dtr-2.4.7/metadata/_SUCCESS | 0 .../ml-models/dtr-2.4.7/metadata/part-0| 1 + .../ml-models/gbtc-2.4.7/data/._SUCCESS.crc| Bin 0 -> 8 bytes ...-41c7-91c0-6da8cc01fb43-c000.snappy.parquet.crc | Bin 0 -> 44 bytes .../resources/ml-models/gbtc-2.4.7/data/_SUCCESS | 0 ...c861-41c7-91c0-6da8cc01fb43-c000.snappy.parquet | Bin 0 -> 4542 bytes .../ml-models/gbtc-2.4.7/metadata/._SUCCESS.crc| Bin 0 -> 8 bytes .../ml-models/gbtc-2.4.7/metadata/.part-0.crc | Bin 0 -> 16 bytes .../ml-models/gbtc-2.4.7/metadata/_SUCCESS | 0 .../ml-models/gbtc-2.4.7/metadata/part-0 | 1 + .../gbtc-2.4.7/treesMetadata/._SUCCESS.crc | Bin 0 -> 8 bytes ...-4a90-813c-ddc394101e21-c000.snappy.parquet.crc | Bin 0 -> 36 bytes .../ml-models/gbtc-2.4.7/treesMetadata/_SUCCESS| 0 ...31e3-4a90-813c-ddc394101e21-c000.snappy.parquet | Bin 0 -> 3075 bytes .../ml-models/gbtr-2.4.7/data/._SUCCESS.crc| Bin 0 -> 8 bytes ...-4511-9aab-639288bfae6d-c000.snappy.parquet.crc | Bin 0 -> 40 bytes .../resources/ml-models/gbtr-2.4.7/data/_SUCCESS | 0 ...d346-4511-9aab-639288bfae6d-c000.snappy.parquet | Bin 0 -> 3740 bytes .../ml-models/gbtr-2.4.7/metadata/._SUCCESS.crc| Bin 0 -> 8 bytes .../ml-models/gbtr-2.4.7/metadata/.part-0.crc | Bin 0 -> 16 bytes .../ml-models/gbtr-2.4.7/metadata/_SUCCESS | 0 .../ml-models/gbtr-2.4.7/metadata/part-0 | 1 + .../gbtr-2.4.7/treesMetadata/._SUCCESS.crc | Bin 0 -> 8 bytes ...-4fd8-ad9c-4be239c2215a-c000.snappy.parquet.crc | Bin 0 -> 32 bytes .../ml-models/gbtr-2.4.7/treesMetadata/_SUCCESS| 0 ...87fe-4fd8-ad9c-4be239c2215a-c000.snappy.parquet | Bin 0 -> 3038 bytes .../ml-models/rfc-2.4.7/data/._SUCCESS.crc | Bin 0 -> 8 bytes ...-4485-b112-25b4b11c9009-c000.snappy.parquet.crc | Bin 0 -> 40 bytes .../resources/ml-models/rfc-2.4.7/data/_SUCCESS| 0 ...91f8-4485-b112-25b4b11c9009-c000.snappy.parquet | Bin 0 -> 3836 bytes .../ml-models/rfc-2.4.7/metadata/._SUCCESS.crc | Bin 0 -> 8 bytes .../ml-models/rfc-2.4.7/metadata/.part-0.crc | Bin 0 -> 16 bytes .../ml-models/rfc-2.4.7/metadata/_SUCCESS | 0 .../ml-models/rfc-2.4.7/metadata/part-0| 1 + .../rfc-2.4.7/treesMetadata/._SUCCESS.crc | Bin 0 -> 8 bytes ...-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet.crc | Bin 0 -> 36 b
[spark] branch branch-3.1 updated: [SPARK-33398] Fix loading tree models prior to Spark 3.0
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.1 by this push: new 1fa052f [SPARK-33398] Fix loading tree models prior to Spark 3.0 1fa052f is described below commit 1fa052fa589daa7d9e5218296cd2ef7143bae443 Author: Ruifeng Zheng AuthorDate: Sun Jan 3 11:52:46 2021 -0600 [SPARK-33398] Fix loading tree models prior to Spark 3.0 ### What changes were proposed in this pull request? In https://github.com/apache/spark/pull/21632/files#diff-0fdae8a6782091746ed20ea43f77b639f9c6a5f072dd2f600fcf9a7b37db4f47, a new field `rawCount` was added into `NodeData`, which cause that a tree model trained in 2.4 can not be loaded in 3.0/3.1/master; field `rawCount` is only used in training, and not used in `transform`/`predict`/`featureImportance`. So I just set it to -1L. ### Why are the changes needed? to support load old tree model in 3.0/3.1/master ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? added testsuites Closes #30889 from zhengruifeng/fix_tree_load. Authored-by: Ruifeng Zheng Signed-off-by: Sean Owen (cherry picked from commit 6b7527e381591bcd51be205853aea3e349893139) Signed-off-by: Sean Owen --- .../org/apache/spark/ml/tree/treeModels.scala | 48 ++--- .../ml-models/dtc-2.4.7/data/._SUCCESS.crc | Bin 0 -> 8 bytes ...-406c-894c-ca4eac67c690-c000.snappy.parquet.crc | Bin 0 -> 36 bytes .../resources/ml-models/dtc-2.4.7/data/_SUCCESS| 0 ...c890-406c-894c-ca4eac67c690-c000.snappy.parquet | Bin 0 -> 3242 bytes .../ml-models/dtc-2.4.7/metadata/._SUCCESS.crc | Bin 0 -> 8 bytes .../ml-models/dtc-2.4.7/metadata/.part-0.crc | Bin 0 -> 16 bytes .../ml-models/dtc-2.4.7/metadata/_SUCCESS | 0 .../ml-models/dtc-2.4.7/metadata/part-0| 1 + .../ml-models/dtr-2.4.7/data/._SUCCESS.crc | Bin 0 -> 8 bytes ...-4b3d-84af-d861adcb9ca8-c000.snappy.parquet.crc | Bin 0 -> 36 bytes .../resources/ml-models/dtr-2.4.7/data/_SUCCESS| 0 ...a437-4b3d-84af-d861adcb9ca8-c000.snappy.parquet | Bin 0 -> 3264 bytes .../ml-models/dtr-2.4.7/metadata/._SUCCESS.crc | Bin 0 -> 8 bytes .../ml-models/dtr-2.4.7/metadata/.part-0.crc | Bin 0 -> 12 bytes .../ml-models/dtr-2.4.7/metadata/_SUCCESS | 0 .../ml-models/dtr-2.4.7/metadata/part-0| 1 + .../ml-models/gbtc-2.4.7/data/._SUCCESS.crc| Bin 0 -> 8 bytes ...-41c7-91c0-6da8cc01fb43-c000.snappy.parquet.crc | Bin 0 -> 44 bytes .../resources/ml-models/gbtc-2.4.7/data/_SUCCESS | 0 ...c861-41c7-91c0-6da8cc01fb43-c000.snappy.parquet | Bin 0 -> 4542 bytes .../ml-models/gbtc-2.4.7/metadata/._SUCCESS.crc| Bin 0 -> 8 bytes .../ml-models/gbtc-2.4.7/metadata/.part-0.crc | Bin 0 -> 16 bytes .../ml-models/gbtc-2.4.7/metadata/_SUCCESS | 0 .../ml-models/gbtc-2.4.7/metadata/part-0 | 1 + .../gbtc-2.4.7/treesMetadata/._SUCCESS.crc | Bin 0 -> 8 bytes ...-4a90-813c-ddc394101e21-c000.snappy.parquet.crc | Bin 0 -> 36 bytes .../ml-models/gbtc-2.4.7/treesMetadata/_SUCCESS| 0 ...31e3-4a90-813c-ddc394101e21-c000.snappy.parquet | Bin 0 -> 3075 bytes .../ml-models/gbtr-2.4.7/data/._SUCCESS.crc| Bin 0 -> 8 bytes ...-4511-9aab-639288bfae6d-c000.snappy.parquet.crc | Bin 0 -> 40 bytes .../resources/ml-models/gbtr-2.4.7/data/_SUCCESS | 0 ...d346-4511-9aab-639288bfae6d-c000.snappy.parquet | Bin 0 -> 3740 bytes .../ml-models/gbtr-2.4.7/metadata/._SUCCESS.crc| Bin 0 -> 8 bytes .../ml-models/gbtr-2.4.7/metadata/.part-0.crc | Bin 0 -> 16 bytes .../ml-models/gbtr-2.4.7/metadata/_SUCCESS | 0 .../ml-models/gbtr-2.4.7/metadata/part-0 | 1 + .../gbtr-2.4.7/treesMetadata/._SUCCESS.crc | Bin 0 -> 8 bytes ...-4fd8-ad9c-4be239c2215a-c000.snappy.parquet.crc | Bin 0 -> 32 bytes .../ml-models/gbtr-2.4.7/treesMetadata/_SUCCESS| 0 ...87fe-4fd8-ad9c-4be239c2215a-c000.snappy.parquet | Bin 0 -> 3038 bytes .../ml-models/rfc-2.4.7/data/._SUCCESS.crc | Bin 0 -> 8 bytes ...-4485-b112-25b4b11c9009-c000.snappy.parquet.crc | Bin 0 -> 40 bytes .../resources/ml-models/rfc-2.4.7/data/_SUCCESS| 0 ...91f8-4485-b112-25b4b11c9009-c000.snappy.parquet | Bin 0 -> 3836 bytes .../ml-models/rfc-2.4.7/metadata/._SUCCESS.crc | Bin 0 -> 8 bytes .../ml-models/rfc-2.4.7/metadata/.part-0.crc | Bin 0 -> 16 bytes .../ml-models/rfc-2.4.7/metadata/_SUCCESS | 0 .../ml-models/rfc-2.4.7/metadata/part-0| 1 + .../rfc-2.4.7/treesMetadata/._SUCCESS.crc | Bin 0 -> 8 bytes ...-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet.crc | Bin 0 -> 36 b
[spark] branch master updated (963c60f -> 6b7527e)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 963c60f [SPARK-33955][SS] Add latest offsets to source progress add 6b7527e [SPARK-33398] Fix loading tree models prior to Spark 3.0 No new revisions were added by this update. Summary of changes: .../org/apache/spark/ml/tree/treeModels.scala | 48 ++--- .../metadata => dtc-2.4.7/data}/._SUCCESS.crc | Bin ...-406c-894c-ca4eac67c690-c000.snappy.parquet.crc | Bin 0 -> 36 bytes .../resources/ml-models/dtc-2.4.7/data/_SUCCESS| 0 ...c890-406c-894c-ca4eac67c690-c000.snappy.parquet | Bin 0 -> 3242 bytes .../metadata/._SUCCESS.crc | Bin .../ml-models/dtc-2.4.7/metadata/.part-0.crc | Bin 0 -> 16 bytes .../ml-models/dtc-2.4.7/metadata/_SUCCESS | 0 .../ml-models/dtc-2.4.7/metadata/part-0| 1 + .../metadata => dtr-2.4.7/data}/._SUCCESS.crc | Bin ...-4b3d-84af-d861adcb9ca8-c000.snappy.parquet.crc | Bin 0 -> 36 bytes .../resources/ml-models/dtr-2.4.7/data/_SUCCESS| 0 ...a437-4b3d-84af-d861adcb9ca8-c000.snappy.parquet | Bin 0 -> 3264 bytes .../metadata/._SUCCESS.crc | Bin .../ml-models/dtr-2.4.7/metadata/.part-0.crc | Bin 0 -> 12 bytes .../ml-models/dtr-2.4.7/metadata/_SUCCESS | 0 .../ml-models/dtr-2.4.7/metadata/part-0| 1 + .../metadata => gbtc-2.4.7/data}/._SUCCESS.crc | Bin ...-41c7-91c0-6da8cc01fb43-c000.snappy.parquet.crc | Bin 0 -> 44 bytes .../resources/ml-models/gbtc-2.4.7/data/_SUCCESS | 0 ...c861-41c7-91c0-6da8cc01fb43-c000.snappy.parquet | Bin 0 -> 4542 bytes .../metadata/._SUCCESS.crc | Bin .../ml-models/gbtc-2.4.7/metadata/.part-0.crc | Bin 0 -> 16 bytes .../ml-models/gbtc-2.4.7/metadata/_SUCCESS | 0 .../ml-models/gbtc-2.4.7/metadata/part-0 | 1 + .../treesMetadata}/._SUCCESS.crc | Bin ...-4a90-813c-ddc394101e21-c000.snappy.parquet.crc | Bin 0 -> 36 bytes .../ml-models/gbtc-2.4.7/treesMetadata/_SUCCESS| 0 ...31e3-4a90-813c-ddc394101e21-c000.snappy.parquet | Bin 0 -> 3075 bytes .../metadata => gbtr-2.4.7/data}/._SUCCESS.crc | Bin ...-4511-9aab-639288bfae6d-c000.snappy.parquet.crc | Bin 0 -> 40 bytes .../resources/ml-models/gbtr-2.4.7/data/_SUCCESS | 0 ...d346-4511-9aab-639288bfae6d-c000.snappy.parquet | Bin 0 -> 3740 bytes .../metadata/._SUCCESS.crc | Bin .../ml-models/gbtr-2.4.7/metadata/.part-0.crc | Bin 0 -> 16 bytes .../ml-models/gbtr-2.4.7/metadata/_SUCCESS | 0 .../ml-models/gbtr-2.4.7/metadata/part-0 | 1 + .../treesMetadata}/._SUCCESS.crc | Bin ...-4fd8-ad9c-4be239c2215a-c000.snappy.parquet.crc | Bin 0 -> 32 bytes .../ml-models/gbtr-2.4.7/treesMetadata/_SUCCESS| 0 ...87fe-4fd8-ad9c-4be239c2215a-c000.snappy.parquet | Bin 0 -> 3038 bytes .../metadata => rfc-2.4.7/data}/._SUCCESS.crc | Bin ...-4485-b112-25b4b11c9009-c000.snappy.parquet.crc | Bin 0 -> 40 bytes .../resources/ml-models/rfc-2.4.7/data/_SUCCESS| 0 ...91f8-4485-b112-25b4b11c9009-c000.snappy.parquet | Bin 0 -> 3836 bytes .../metadata/._SUCCESS.crc | Bin .../ml-models/rfc-2.4.7/metadata/.part-0.crc | Bin 0 -> 16 bytes .../ml-models/rfc-2.4.7/metadata/_SUCCESS | 0 .../ml-models/rfc-2.4.7/metadata/part-0| 1 + .../treesMetadata}/._SUCCESS.crc | Bin ...-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet.crc | Bin 0 -> 36 bytes .../ml-models/rfc-2.4.7/treesMetadata/_SUCCESS | 0 ...b666-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet | Bin 0 -> 3391 bytes .../metadata => rfr-2.4.7/data}/._SUCCESS.crc | Bin ...-40fc-b681-981caaeca996-c000.snappy.parquet.crc | Bin 0 -> 40 bytes .../resources/ml-models/rfr-2.4.7/data/_SUCCESS| 0 ...6edb-40fc-b681-981caaeca996-c000.snappy.parquet | Bin 0 -> 3797 bytes .../metadata/._SUCCESS.crc | Bin .../ml-models/rfr-2.4.7/metadata/.part-0.crc | Bin 0 -> 16 bytes .../ml-models/rfr-2.4.7/metadata/_SUCCESS | 0 .../ml-models/rfr-2.4.7/metadata/part-0| 1 + .../treesMetadata}/._SUCCESS.crc | Bin ...-447a-9b86-d95edaabcde8-c000.snappy.parquet.crc | Bin 0 -> 32 bytes .../ml-models/rfr-2.4.7/treesMetadata/_SUCCESS | 0 ...d349-447a-9b86-d95edaabcde8-c000.snappy.parquet | Bin 0 -> 3055 bytes .../DecisionTreeClassifierSuite.scala | 12 ++ .../ml/classification/GBTClassifierSuite.scala | 14 ++ .../MultilayerPerceptronClassifierSuite.scala | 2 +- .../RandomForestClassifierSuite.scala | 16 ++- .../apache/spark/ml/feature/HashingTFSuite.
[spark] branch master updated (f38265d -> 85de644)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f38265d [SPARK-33907][SQL] Only prune columns of from_json if parsing options is empty add 85de644 [SPARK-33804][CORE] Fix compilation warnings about 'view bounds are deprecated' No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/rdd/SequenceFileRDDFunctions.scala | 7 ++- core/src/main/scala/org/apache/spark/rdd/package.scala | 6 +- 2 files changed, 7 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated: [SPARK-33560][TEST-MAVEN][BUILD] Add "unused-import" check to Maven compilation process
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 1b3c519 [SPARK-33560][TEST-MAVEN][BUILD] Add "unused-import" check to
Maven compilation process
1b3c519 is described below
commit 1b3c519db0c67c32014327326a39a3c2e3c94218
Author: yangjie01
AuthorDate: Sat Dec 26 17:40:19 2020 -0600
[SPARK-33560][TEST-MAVEN][BUILD] Add "unused-import" check to Maven
compilation process
### What changes were proposed in this pull request?
Similar to SPARK-33441, this pr add `unused-import` check to Maven
compilation process. After this pr `unused-import` will trigger Maven
compilation error.
For Scala 2.13 profile, this pr also left TODO(SPARK-33499) similar to
SPARK-33441 because `scala.language.higherKinds` no longer needs to be imported
explicitly since Scala 2.13.1
### Why are the changes needed?
Let Maven build also check for unused imports as compilation error.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass the Jenkins or GitHub Action
- Local manual test：add an unused import intentionally to trigger maven
compilation error.
Closes #30784 from LuciferYang/SPARK-33560.
Authored-by: yangjie01
Signed-off-by: Sean Owen
(cherry picked from commit 37ae0a608670c660ba4c92b9ebb9cb9fb2bd67e6)
Signed-off-by: Sean Owen
---
pom.xml| 43 ++
.../sources/StreamingDataSourceV2Suite.scala | 2 +-
2 files changed, 44 insertions(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index 7e0bd6b..0241a02 100644
--- a/pom.xml
+++ b/pom.xml
@@ -2508,6 +2508,9 @@
-feature
-explaintypes
-target:jvm-1.8
+ -Xfatal-warnings
+ -Ywarn-unused:imports
+ -P:silencer:globalFilters=.*deprecated.*
-Xms1024m
@@ -2521,6 +2524,13 @@
${java.version}
-Xlint:all,-serial,-path,-try
+
+
+com.github.ghik
+silencer-plugin_${scala.version}
+1.6.0
+
+
@@ -3243,6 +3253,39 @@
+
+
+
+
+ net.alchim31.maven
+ scala-maven-plugin
+
+
+ -unchecked
+ -deprecation
+ -feature
+ -explaintypes
+ -target:jvm-1.8
+ -Wconf:cat=deprecation:wv,any:e
+
+ -Wconf:cat=scaladoc:wv
+ -Wconf:cat=lint-multiarg-infix:wv
+ -Wconf:cat=other-nullary-override:wv
+
-Wconf:cat=other-match-analysissite=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv
+
-Wconf:cat=other-pure-statementsite=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv
+
-Wconf:cat=other-pure-statementsite=org.apache.spark.scheduler.OutputCommitCoordinatorSuite.local
OutputCommitCoordinatorSuite.futureAction:wv
+
+
+
+
+
+
+
+
+
[spark] branch master updated: [SPARK-33560][TEST-MAVEN][BUILD] Add "unused-import" check to Maven compilation process
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 37ae0a6 [SPARK-33560][TEST-MAVEN][BUILD] Add "unused-import" check to
Maven compilation process
37ae0a6 is described below
commit 37ae0a608670c660ba4c92b9ebb9cb9fb2bd67e6
Author: yangjie01
AuthorDate: Sat Dec 26 17:40:19 2020 -0600
[SPARK-33560][TEST-MAVEN][BUILD] Add "unused-import" check to Maven
compilation process
### What changes were proposed in this pull request?
Similar to SPARK-33441, this pr add `unused-import` check to Maven
compilation process. After this pr `unused-import` will trigger Maven
compilation error.
For Scala 2.13 profile, this pr also left TODO(SPARK-33499) similar to
SPARK-33441 because `scala.language.higherKinds` no longer needs to be imported
explicitly since Scala 2.13.1
### Why are the changes needed?
Let Maven build also check for unused imports as compilation error.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass the Jenkins or GitHub Action
- Local manual test：add an unused import intentionally to trigger maven
compilation error.
Closes #30784 from LuciferYang/SPARK-33560.
Authored-by: yangjie01
Signed-off-by: Sean Owen
---
pom.xml| 43 ++
.../sources/StreamingDataSourceV2Suite.scala | 2 +-
2 files changed, 44 insertions(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index 4781f98..609c9fc 100644
--- a/pom.xml
+++ b/pom.xml
@@ -2508,6 +2508,9 @@
-feature
-explaintypes
-target:jvm-1.8
+ -Xfatal-warnings
+ -Ywarn-unused:imports
+ -P:silencer:globalFilters=.*deprecated.*
-Xms1024m
@@ -2521,6 +2524,13 @@
${java.version}
-Xlint:all,-serial,-path,-try
+
+
+com.github.ghik
+silencer-plugin_${scala.version}
+1.6.0
+
+
@@ -3243,6 +3253,39 @@
+
+
+
+
+ net.alchim31.maven
+ scala-maven-plugin
+
+
+ -unchecked
+ -deprecation
+ -feature
+ -explaintypes
+ -target:jvm-1.8
+ -Wconf:cat=deprecation:wv,any:e
+
+ -Wconf:cat=scaladoc:wv
+ -Wconf:cat=lint-multiarg-infix:wv
+ -Wconf:cat=other-nullary-override:wv
+
-Wconf:cat=other-match-analysissite=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv
+
-Wconf:cat=other-pure-statementsite=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv
+
-Wconf:cat=other-pure-statementsite=org.apache.spark.scheduler.OutputCommitCoordinatorSuite.local
OutputCommitCoordinatorSuite.futureAction:wv
+
+
+
+
+
+
+
+
+
[spark] branch master updated: [SPARK-33835][CORE] Refector AbstractCommandBuilder.buildJavaCommand: use firstNonEmpty
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 61881bb [SPARK-33835][CORE] Refector
AbstractCommandBuilder.buildJavaCommand: use firstNonEmpty
61881bb is described below
commit 61881bb6988aa0320b4bacfabbc0ee6f05f287cb
Author: offthewall123
AuthorDate: Wed Dec 23 20:01:53 2020 -0600
[SPARK-33835][CORE] Refector AbstractCommandBuilder.buildJavaCommand: use
firstNonEmpty
### What changes were proposed in this pull request?
refector AbstractCommandBuilder.buildJavaCommand: use firstNonEmpty
### Why are the changes needed?
For better code understanding, and firstNonEmpty can detect javaHome = "
", an empty string.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
End to End.
Closes #30831 from offthewall123/refector_AbstractCommandBuilder.
Authored-by: offthewall123
Signed-off-by: Sean Owen
---
.../org/apache/spark/launcher/AbstractCommandBuilder.java | 14 +-
1 file changed, 5 insertions(+), 9 deletions(-)
diff --git
a/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
b/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
index 778fd46..24ad9cb 100644
---
a/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
+++
b/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
@@ -92,17 +92,13 @@ abstract class AbstractCommandBuilder {
List buildJavaCommand(String extraClassPath) throws IOException {
List cmd = new ArrayList<>();
-String[] candidateJavaHomes = new String[] {
- javaHome,
+String firstJavaHome = firstNonEmpty(javaHome,
childEnv.get("JAVA_HOME"),
System.getenv("JAVA_HOME"),
- System.getProperty("java.home")
-};
-for (String javaHome : candidateJavaHomes) {
- if (javaHome != null) {
-cmd.add(join(File.separator, javaHome, "bin", "java"));
-break;
- }
+ System.getProperty("java.home"));
+
+if (firstJavaHome != null) {
+ cmd.add(join(File.separator, firstJavaHome, "bin", "java"));
}
// Load extra JAVA_OPTS from conf/java-opts, if it exists.
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-33756][SQL] Make BytesToBytesMap's MapIterator idempotent
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 9b8c193 [SPARK-33756][SQL] Make BytesToBytesMap's MapIterator
idempotent
9b8c193 is described below
commit 9b8c1930ed57117a70310d66eb1ab44f7ece9e0d
Author: Xianjin YE
AuthorDate: Sun Dec 20 08:51:17 2020 -0600
[SPARK-33756][SQL] Make BytesToBytesMap's MapIterator idempotent
### What changes were proposed in this pull request?
Make MapIterator of BytesToBytesMap `hasNext` method idempotent
### Why are the changes needed?
The `hasNext` maybe called multiple times, if not guarded, second call of
hasNext method after reaching the end of iterator will throw NoSuchElement
exception.
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
Update a unit test to cover this case.
Closes #30728 from advancedxy/SPARK-33756.
Authored-by: Xianjin YE
Signed-off-by: Sean Owen
(cherry picked from commit 13391683e7a863671d3d719dc81e20ec2a870725)
Signed-off-by: Sean Owen
---
.../main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java | 10 ++
.../apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java | 2 ++
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git
a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
index e2d258a..b38f845 100644
--- a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
+++ b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
@@ -408,10 +408,12 @@ public final class BytesToBytesMap extends MemoryConsumer
{
}
private void handleFailedDelete() {
- // remove the spill file from disk
- File file = spillWriters.removeFirst().getFile();
- if (file != null && file.exists() && !file.delete()) {
-logger.error("Was unable to delete spill file {}",
file.getAbsolutePath());
+ if (spillWriters.size() > 0) {
+// remove the spill file from disk
+File file = spillWriters.removeFirst().getFile();
+if (file != null && file.exists() && !file.delete()) {
+ logger.error("Was unable to delete spill file {}",
file.getAbsolutePath());
+}
}
}
}
diff --git
a/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
b/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
index 7741654..da11fd6 100644
---
a/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
+++
b/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
@@ -560,6 +560,8 @@ public abstract class AbstractBytesToBytesMapSuite {
iter2.next();
}
assertFalse(iter2.hasNext());
+ // calls hasNext twice deliberately, make sure it's idempotent
+ assertFalse(iter2.hasNext());
} finally {
map.free();
for (File spillFile : spillFilesCreated) {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33756][SQL] Make BytesToBytesMap's MapIterator idempotent
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new faf8dd5 [SPARK-33756][SQL] Make BytesToBytesMap's MapIterator
idempotent
faf8dd5 is described below
commit faf8dd575eb1a40b5bde91191baa4ae741269531
Author: Xianjin YE
AuthorDate: Sun Dec 20 08:51:17 2020 -0600
[SPARK-33756][SQL] Make BytesToBytesMap's MapIterator idempotent
### What changes were proposed in this pull request?
Make MapIterator of BytesToBytesMap `hasNext` method idempotent
### Why are the changes needed?
The `hasNext` maybe called multiple times, if not guarded, second call of
hasNext method after reaching the end of iterator will throw NoSuchElement
exception.
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
Update a unit test to cover this case.
Closes #30728 from advancedxy/SPARK-33756.
Authored-by: Xianjin YE
Signed-off-by: Sean Owen
(cherry picked from commit 13391683e7a863671d3d719dc81e20ec2a870725)
Signed-off-by: Sean Owen
---
.../main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java | 10 ++
.../apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java | 2 ++
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git
a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
index 4036856..888cc8c 100644
--- a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
+++ b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
@@ -393,10 +393,12 @@ public final class BytesToBytesMap extends MemoryConsumer
{
}
private void handleFailedDelete() {
- // remove the spill file from disk
- File file = spillWriters.removeFirst().getFile();
- if (file != null && file.exists() && !file.delete()) {
-logger.error("Was unable to delete spill file {}",
file.getAbsolutePath());
+ if (spillWriters.size() > 0) {
+// remove the spill file from disk
+File file = spillWriters.removeFirst().getFile();
+if (file != null && file.exists() && !file.delete()) {
+ logger.error("Was unable to delete spill file {}",
file.getAbsolutePath());
+}
}
}
}
diff --git
a/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
b/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
index 6e995a3..ab177ce 100644
---
a/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
+++
b/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
@@ -561,6 +561,8 @@ public abstract class AbstractBytesToBytesMapSuite {
iter2.next();
}
assertFalse(iter2.hasNext());
+ // calls hasNext twice deliberately, make sure it's idempotent
+ assertFalse(iter2.hasNext());
} finally {
map.free();
for (File spillFile : spillFilesCreated) {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated: [SPARK-33756][SQL] Make BytesToBytesMap's MapIterator idempotent
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 5124558 [SPARK-33756][SQL] Make BytesToBytesMap's MapIterator
idempotent
5124558 is described below
commit 5124558eb6ce561978231e3d9db9b86878e63ca9
Author: Xianjin YE
AuthorDate: Sun Dec 20 08:51:17 2020 -0600
[SPARK-33756][SQL] Make BytesToBytesMap's MapIterator idempotent
### What changes were proposed in this pull request?
Make MapIterator of BytesToBytesMap `hasNext` method idempotent
### Why are the changes needed?
The `hasNext` maybe called multiple times, if not guarded, second call of
hasNext method after reaching the end of iterator will throw NoSuchElement
exception.
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
Update a unit test to cover this case.
Closes #30728 from advancedxy/SPARK-33756.
Authored-by: Xianjin YE
Signed-off-by: Sean Owen
(cherry picked from commit 13391683e7a863671d3d719dc81e20ec2a870725)
Signed-off-by: Sean Owen
---
.../main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java | 10 ++
.../apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java | 2 ++
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git
a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
index d7940fc..f474c30 100644
--- a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
+++ b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
@@ -393,10 +393,12 @@ public final class BytesToBytesMap extends MemoryConsumer
{
}
private void handleFailedDelete() {
- // remove the spill file from disk
- File file = spillWriters.removeFirst().getFile();
- if (file != null && file.exists() && !file.delete()) {
-logger.error("Was unable to delete spill file {}",
file.getAbsolutePath());
+ if (spillWriters.size() > 0) {
+// remove the spill file from disk
+File file = spillWriters.removeFirst().getFile();
+if (file != null && file.exists() && !file.delete()) {
+ logger.error("Was unable to delete spill file {}",
file.getAbsolutePath());
+}
}
}
}
diff --git
a/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
b/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
index f4e952f..f35176a 100644
---
a/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
+++
b/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
@@ -576,6 +576,8 @@ public abstract class AbstractBytesToBytesMapSuite {
iter2.next();
}
assertFalse(iter2.hasNext());
+ // calls hasNext twice deliberately, make sure it's idempotent
+ assertFalse(iter2.hasNext());
} finally {
map.free();
for (File spillFile : spillFilesCreated) {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (df2314b -> 1339168)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from df2314b [SPARK-33852][SQL][TESTS] Use assertAnalysisError in HiveDDLSuite.scala add 1339168 [SPARK-33756][SQL] Make BytesToBytesMap's MapIterator idempotent No new revisions were added by this update. Summary of changes: .../main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java | 10 ++ .../apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java | 2 ++ 2 files changed, 8 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (06075d8 -> 37c4cd8)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 06075d8 [SPARK-33829][SQL] Renaming v2 tables should recreate the cache add 37c4cd8 [MINOR][DOCS] Fix typos in ScalaDocs for DataStreamWriter#foreachBatch No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/streaming/DataStreamWriter.scala | 14 -- 1 file changed, 8 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (de234ee -> 44563a0)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from de234ee [SPARK-33812][SQL] Split the histogram column stats when saving to hive metastore as table property add 44563a0 [SPARK-33518][ML] Improve performance of ML ALS recommendForAll by GEMV No new revisions were added by this update. Summary of changes: .../org/apache/spark/ml/recommendation/ALS.scala | 53 ++ 1 file changed, 33 insertions(+), 20 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (34e4d87 -> 8c81cf7)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 34e4d87 [SPARK-33774][UI][CORE] Back to Master" returns 500 error in Standalone cluster add 8c81cf7 [SPARK-22769] Do not log rpc post message error when sparkEnv is already stopped No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala | 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33774][UI][CORE] Back to Master" returns 500 error in Standalone cluster
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 99eb027 [SPARK-33774][UI][CORE] Back to Master" returns 500 error in
Standalone cluster
99eb027 is described below
commit 99eb02745e85a11dbbb1fa5de1167f07cd499797
Author: yi.wu
AuthorDate: Thu Dec 17 08:52:09 2020 -0600
[SPARK-33774][UI][CORE] Back to Master" returns 500 error in Standalone
cluster
### What changes were proposed in this pull request?
Initiate the `masterWebUiUrl` with the `webUi. webUrl` instead of the
`masterPublicAddress`.
### Why are the changes needed?
Since [SPARK-21642](https://issues.apache.org/jira/browse/SPARK-21642),
`WebUI` has changed from `localHostName` to `localCanonicalHostName` as the
hostname to set up the web UI. However, the `masterPublicAddress` is from
`RpcEnv`'s host address, which still uses `localHostName`. As a result, it
returns the wrong Master web URL to the Worker.
### Does this PR introduce _any_ user-facing change?
Yes, when users click "Back to Master" in the Worker page:
Before this PR:
https://user-images.githubusercontent.com/16397174/102057951-b9664280-3e29-11eb-8749-5ee293902bdf.png;>
After this PR:
(Return to the Master page successfully.)
### How was this patch tested?
Tested manually.
Closes #30759 from Ngone51/fix-back-to-master.
Authored-by: yi.wu
Signed-off-by: Sean Owen
(cherry picked from commit 34e4d87023535c086a0aa43fe194f794b41e09b7)
Signed-off-by: Sean Owen
---
core/src/main/scala/org/apache/spark/deploy/master/Master.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
index 3818a87..ab6cb30 100644
--- a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
@@ -143,7 +143,7 @@ private[deploy] class Master(
logInfo(s"Running Spark version ${org.apache.spark.SPARK_VERSION}")
webUi = new MasterWebUI(this, webUiPort)
webUi.bind()
-masterWebUiUrl = s"${webUi.scheme}$masterPublicAddress:${webUi.boundPort}"
+masterWebUiUrl = webUi.webUrl
if (reverseProxy) {
masterWebUiUrl =
conf.get(UI_REVERSE_PROXY_URL).orElse(Some(masterWebUiUrl)).get
webUi.addProxy()
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated: [SPARK-33774][UI][CORE] Back to Master" returns 500 error in Standalone cluster
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 919f964 [SPARK-33774][UI][CORE] Back to Master" returns 500 error in
Standalone cluster
919f964 is described below
commit 919f9648873dceabf0448792969276f940447c31
Author: yi.wu
AuthorDate: Thu Dec 17 08:52:09 2020 -0600
[SPARK-33774][UI][CORE] Back to Master" returns 500 error in Standalone
cluster
### What changes were proposed in this pull request?
Initiate the `masterWebUiUrl` with the `webUi. webUrl` instead of the
`masterPublicAddress`.
### Why are the changes needed?
Since [SPARK-21642](https://issues.apache.org/jira/browse/SPARK-21642),
`WebUI` has changed from `localHostName` to `localCanonicalHostName` as the
hostname to set up the web UI. However, the `masterPublicAddress` is from
`RpcEnv`'s host address, which still uses `localHostName`. As a result, it
returns the wrong Master web URL to the Worker.
### Does this PR introduce _any_ user-facing change?
Yes, when users click "Back to Master" in the Worker page:
Before this PR:
https://user-images.githubusercontent.com/16397174/102057951-b9664280-3e29-11eb-8749-5ee293902bdf.png;>
After this PR:
(Return to the Master page successfully.)
### How was this patch tested?
Tested manually.
Closes #30759 from Ngone51/fix-back-to-master.
Authored-by: yi.wu
Signed-off-by: Sean Owen
(cherry picked from commit 34e4d87023535c086a0aa43fe194f794b41e09b7)
Signed-off-by: Sean Owen
---
core/src/main/scala/org/apache/spark/deploy/master/Master.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
index cccd3da..9f1b36a 100644
--- a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
@@ -143,7 +143,7 @@ private[deploy] class Master(
logInfo(s"Running Spark version ${org.apache.spark.SPARK_VERSION}")
webUi = new MasterWebUI(this, webUiPort)
webUi.bind()
-masterWebUiUrl = s"${webUi.scheme}$masterPublicAddress:${webUi.boundPort}"
+masterWebUiUrl = webUi.webUrl
if (reverseProxy) {
val uiReverseProxyUrl =
conf.get(UI_REVERSE_PROXY_URL).map(_.stripSuffix("/"))
if (uiReverseProxyUrl.nonEmpty) {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (12b69cc -> 34e4d87)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 12b69cc [SPARK-26199][SPARK-31517][R] Fix strategy for handling ... names in mutate add 34e4d87 [SPARK-33774][UI][CORE] Back to Master" returns 500 error in Standalone cluster No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/deploy/master/Master.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c05ee06 -> a093d6f)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c05ee06 [SPARK-33685][SQL] Migrate DROP VIEW command to use UnresolvedView to resolve the identifier add a093d6f [MINOR] Spelling sql/core No new revisions were added by this update. Summary of changes: .../spark/sql/execution/ui/static/spark-sql-viz.js | 8 +++--- .../main/scala/org/apache/spark/sql/Dataset.scala | 10 +++ .../spark/sql/execution/DataSourceScanExec.scala | 6 ++-- .../apache/spark/sql/execution/ExplainUtils.scala | 8 +++--- .../ExternalAppendOnlyUnsafeRowArray.scala | 2 +- .../spark/sql/execution/SparkSqlParser.scala | 14 +- .../sql/execution/WholeStageCodegenExec.scala | 2 +- .../adaptive/AdaptiveSparkPlanHelper.scala | 2 +- .../command/InsertIntoDataSourceDirCommand.scala | 2 +- .../apache/spark/sql/execution/command/ddl.scala | 4 +-- .../spark/sql/execution/command/tables.scala | 2 +- .../sql/execution/datasources/DataSource.scala | 2 +- .../datasources/FileFormatDataWriter.scala | 14 +- .../execution/datasources/FileFormatWriter.scala | 2 +- .../execution/datasources/PartitioningUtils.scala | 2 +- .../datasources/v2/WriteToDataSourceV2Exec.scala | 2 +- .../spark/sql/execution/joins/HashedRelation.scala | 4 +-- .../sql/execution/python/ExtractPythonUDFs.scala | 6 ++-- .../streaming/CompactibleFileStreamLog.scala | 2 +- .../sql/execution/streaming/StreamExecution.scala | 2 +- .../state/FlatMapGroupsWithStateExecHelper.scala | 2 +- .../org/apache/spark/sql/internal/HiveSerDe.scala | 2 +- .../spark/sql/streaming/DataStreamWriter.scala | 4 +-- .../spark/sql/Java8DatasetAggregatorSuite.java | 16 +-- .../spark/sql/JavaDatasetAggregatorSuite.java | 24 .../inputs/ansi/decimalArithmeticOperations.sql| 2 +- .../sql-tests/inputs/postgreSQL/create_view.sql| 2 +- .../org/apache/spark/sql/CachedTableSuite.scala| 8 +++--- .../org/apache/spark/sql/DataFrameSuite.scala | 2 +- .../org/apache/spark/sql/DatasetCacheSuite.scala | 13 + .../apache/spark/sql/DatasetPrimitiveSuite.scala | 8 +++--- .../scala/org/apache/spark/sql/DatasetSuite.scala | 32 +++--- .../org/apache/spark/sql/DateFunctionsSuite.scala | 6 ++-- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 6 ++-- .../org/apache/spark/sql/SQLQueryTestSuite.scala | 10 +++ .../spark/sql/SparkSessionExtensionSuite.scala | 18 ++-- .../org/apache/spark/sql/TPCDSTableStats.scala | 2 +- .../spark/sql/connector/DataSourceV2SQLSuite.scala | 12 .../sql/execution/SQLWindowFunctionSuite.scala | 2 +- .../spark/sql/execution/SparkSqlParserSuite.scala | 2 +- .../sql/execution/WholeStageCodegenSuite.scala | 4 +-- .../adaptive/AdaptiveQueryExecSuite.scala | 8 +++--- .../sql/execution/arrow/ArrowConvertersSuite.scala | 2 +- .../spark/sql/execution/command/DDLSuite.scala | 12 .../execution/command/PlanResolutionSuite.scala| 16 +-- .../execution/datasources/DataSourceSuite.scala| 4 +-- .../execution/datasources/SchemaPruningSuite.scala | 8 +++--- .../parquet/ParquetInteroperabilitySuite.scala | 2 +- .../parquet/ParquetPartitionDiscoverySuite.scala | 4 +-- .../datasources/parquet/ParquetQuerySuite.scala| 4 +-- .../exchange/EnsureRequirementsSuite.scala | 2 +- .../sql/execution/metric/SQLMetricsSuite.scala | 2 +- .../execution/streaming/HDFSMetadataLogSuite.scala | 2 +- .../sql/execution/ui/SparkPlanInfoSuite.scala | 6 ++-- .../sql/internal/ExecutorSideSQLConfSuite.scala| 4 +-- .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 12 .../spark/sql/sources/BucketedReadSuite.scala | 18 ++-- .../sql/sources/CreateTableAsSelectSuite.scala | 2 +- .../apache/spark/sql/sources/TableScanSuite.scala | 6 ++-- .../sql/streaming/FileStreamSourceSuite.scala | 4 +-- .../apache/spark/sql/streaming/StreamSuite.scala | 8 +++--- .../streaming/test/DataStreamTableAPISuite.scala | 8 +++--- .../org/apache/spark/sql/test/SQLTestData.scala| 4 +-- .../apache/spark/sql/test/SharedSparkSession.scala | 2 +- 64 files changed, 208 insertions(+), 205 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [MINOR] Spelling sql not core
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c62b84a [MINOR] Spelling sql not core c62b84a is described below commit c62b84a0432e51fd10e628088ee311dc3be73d2f Author: Josh Soref AuthorDate: Mon Dec 7 08:40:29 2020 -0600 [MINOR] Spelling sql not core ### What changes were proposed in this pull request? This PR intends to fix typos in the sub-modules: * `sql/catalyst` * `sql/hive-thriftserver` * `sql/hive` Split per srowen https://github.com/apache/spark/pull/30323#issuecomment-728981618 NOTE: The misspellings have been reported at https://github.com/jsoref/spark/commit/706a726f87a0bbf5e31467fae9015218773db85b#commitcomment-44064356 ### Why are the changes needed? Misspelled words make it harder to read / understand content. ### Does this PR introduce _any_ user-facing change? There are various fixes to documentation, etc... ### How was this patch tested? No testing was performed Closes #30532 from jsoref/spelling-sql-not-core. Authored-by: Josh Soref Signed-off-by: Sean Owen --- sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala | 6 +++--- .../scala/org/apache/spark/sql/catalyst/StructFilters.scala | 2 +- .../org/apache/spark/sql/catalyst/analysis/Analyzer.scala| 2 +- .../apache/spark/sql/catalyst/analysis/CheckAnalysis.scala | 2 +- .../spark/sql/catalyst/analysis/StreamingJoinHelper.scala| 4 ++-- .../sql/catalyst/analysis/UpdateAttributeNullability.scala | 2 +- .../apache/spark/sql/catalyst/catalog/SessionCatalog.scala | 2 +- .../org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala | 2 +- .../spark/sql/catalyst/encoders/ExpressionEncoder.scala | 2 +- .../apache/spark/sql/catalyst/expressions/AliasHelper.scala | 2 +- .../org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala | 4 ++-- .../sql/catalyst/expressions/aggregate/Percentile.scala | 6 +++--- .../apache/spark/sql/catalyst/expressions/arithmetic.scala | 2 +- .../sql/catalyst/expressions/codegen/CodeGenerator.scala | 2 +- .../expressions/codegen/GenerateSafeProjection.scala | 2 +- .../org/apache/spark/sql/catalyst/expressions/hash.scala | 4 ++-- .../sql/catalyst/expressions/higherOrderFunctions.scala | 2 +- .../spark/sql/catalyst/expressions/jsonExpressions.scala | 12 ++-- .../spark/sql/catalyst/expressions/regexpExpressions.scala | 2 +- .../spark/sql/catalyst/expressions/windowExpressions.scala | 2 +- .../spark/sql/catalyst/optimizer/NestedColumnAliasing.scala | 2 +- .../org/apache/spark/sql/catalyst/optimizer/Optimizer.scala | 4 ++-- .../sql/catalyst/optimizer/PushDownLeftSemiAntiJoin.scala| 2 +- .../apache/spark/sql/catalyst/optimizer/expressions.scala| 2 +- .../org/apache/spark/sql/catalyst/optimizer/subquery.scala | 6 +++--- .../org/apache/spark/sql/catalyst/parser/ParserUtils.scala | 2 +- .../org/apache/spark/sql/catalyst/plans/QueryPlan.scala | 2 +- .../spark/sql/catalyst/plans/logical/LogicalPlan.scala | 2 +- .../apache/spark/sql/catalyst/plans/logical/PlanHelper.scala | 2 +- .../sql/catalyst/plans/logical/basicLogicalOperators.scala | 2 +- .../spark/sql/catalyst/plans/physical/partitioning.scala | 2 +- .../spark/sql/catalyst/util/DateTimeFormatterHelper.scala| 4 ++-- .../org/apache/spark/sql/catalyst/util/DateTimeUtils.scala | 2 +- .../apache/spark/sql/catalyst/util/QuantileSummaries.scala | 6 +++--- .../main/scala/org/apache/spark/sql/internal/SQLConf.scala | 6 +++--- .../scala/org/apache/spark/sql/RandomDataGenerator.scala | 6 +++--- .../apache/spark/sql/catalyst/analysis/AnalysisSuite.scala | 2 +- .../catalyst/analysis/ResolveGroupingAnalyticsSuite.scala| 4 ++-- .../spark/sql/catalyst/analysis/TypeCoercionSuite.scala | 2 +- .../sql/catalyst/analysis/UnsupportedOperationsSuite.scala | 2 +- .../spark/sql/catalyst/expressions/CodeGenerationSuite.scala | 4 ++-- .../spark/sql/catalyst/expressions/ComplexTypeSuite.scala| 4 ++-- .../catalyst/expressions/ConditionalExpressionSuite.scala| 4 ++-- .../sql/catalyst/expressions/ExpressionEvalHelper.scala | 2 +- .../sql/catalyst/expressions/ObjectExpressionsSuite.scala| 4 ++-- .../sql/catalyst/expressions/StringExpressionsSuite.scala| 2 +- .../sql/catalyst/expressions/aggregate/PercentileSuite.scala | 8 .../sql/catalyst/expressions/codegen/CodeBlockSuite.scala| 2 +- .../spark/sql/catalyst/optimizer/SetOperationSuite.scala | 8 .../apache/spark/sql/catalyst/parser/DDLParserSuite.scala| 2 +- .../spark/sql/catalyst/parser/DataTypeParserSuite.scala | 4 ++-- .../apache/spark/sql
[spark] branch master updated (35ded12 -> 13fd272)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 35ded12 [SPARK-33141][SQL] Capture SQL configs when creating permanent views add 13fd272 Spelling r common dev mlib external project streaming resource managers python No new revisions were added by this update. Summary of changes: R/CRAN_RELEASE.md | 2 +- R/install-dev.bat | 2 +- R/pkg/R/DataFrame.R| 6 +++--- R/pkg/R/RDD.R | 4 ++-- R/pkg/R/SQLContext.R | 2 +- R/pkg/R/WindowSpec.R | 4 ++-- R/pkg/R/column.R | 16 R/pkg/R/context.R | 4 ++-- R/pkg/R/deserialize.R | 2 +- R/pkg/R/functions.R| 4 ++-- R/pkg/R/install.R | 2 +- R/pkg/R/mllib_fpm.R| 2 +- R/pkg/R/mllib_tree.R | 4 ++-- R/pkg/R/mllib_utils.R | 2 +- R/pkg/R/pairRDD.R | 4 ++-- R/pkg/R/streaming.R| 2 +- R/pkg/R/types.R| 2 +- R/pkg/R/utils.R| 2 +- R/pkg/inst/worker/daemon.R | 4 ++-- R/pkg/inst/worker/worker.R | 8 R/pkg/tests/fulltests/test_Serde.R | 2 +- R/pkg/tests/fulltests/test_jvm_api.R | 6 +++--- R/pkg/tests/fulltests/test_sparkSQL.R | 6 +++--- R/pkg/tests/fulltests/test_utils.R | 2 +- R/pkg/vignettes/sparkr-vignettes.Rmd | 2 +- .../apache/spark/util/kvstore/LevelDBTypeInfo.java | 2 +- .../spark/network/client/TransportClient.java | 2 +- .../apache/spark/network/crypto/AuthEngine.java| 2 +- .../spark/network/crypto/AuthEngineSuite.java | 10 +- .../network/protocol/MessageWithHeaderSuite.java | 4 ++-- .../apache/spark/network/sasl/SparkSaslSuite.java | 16 .../server/OneForOneStreamManagerSuite.java| 2 +- .../network/util/TransportFrameDecoderSuite.java | 2 +- .../spark/network/shuffle/SimpleDownloadFile.java | 2 +- .../org/apache/spark/unsafe/types/UTF8String.java | 10 +- .../types/UTF8StringPropertyCheckSuite.scala | 6 +++--- dev/appveyor-guide.md | 12 ++-- dev/create-release/known_translations | 2 +- dev/create-release/release-build.sh| 2 +- dev/create-release/releaseutils.py | 6 +++--- dev/create-release/translate-contributors.py | 22 +++--- dev/github_jira_sync.py| 10 +- dev/run-tests-jenkins.py | 18 +- dev/run-tests.py | 6 +++--- dev/tests/pr_merge_ability.sh | 2 +- dev/tests/pr_public_classes.sh | 2 +- project/MimaExcludes.scala | 2 +- project/SparkBuild.scala | 6 +++--- python/docs/source/_static/css/pyspark.css | 2 +- .../docs/source/_templates/autosummary/class.rst | 2 +- python/docs/source/development/debugging.rst | 2 +- python/docs/source/development/testing.rst | 2 +- python/docs/source/getting_started/install.rst | 6 +++--- .../docs/source/getting_started/quickstart.ipynb | 4 ++-- python/docs/source/index.rst | 2 +- python/pyspark/__init__.pyi| 2 +- python/pyspark/cloudpickle/cloudpickle.py | 10 +- python/pyspark/cloudpickle/cloudpickle_fast.py | 10 +- python/pyspark/context.py | 4 ++-- python/pyspark/java_gateway.py | 2 +- python/pyspark/ml/feature.py | 2 +- python/pyspark/ml/regression.py| 2 +- python/pyspark/ml/regression.pyi | 2 +- python/pyspark/ml/tests/test_algorithms.py | 2 +- python/pyspark/ml/tests/test_image.py | 2 +- python/pyspark/mllib/clustering.py | 2 +- python/pyspark/mllib/evaluation.py | 4 ++-- python/pyspark/mllib/regression.py | 2 +- python/pyspark/mllib/stat/_statistics.py | 2 +- .../mllib/tests/test_streaming_algorithms.py | 2 +- python/pyspark/rdd.py | 4 ++-- python/pyspark/resource/requests.py| 4 ++-- python/pyspark/shuffle.py | 2 +- python/pyspark
[spark] branch master updated (47326ac -> 116b7b7)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 47326ac [SPARK-28704][SQL][TEST] Add back Skiped HiveExternalCatalogVersionsSuite in HiveSparkSubmitSuite at JDK9+ add 116b7b7 [SPARK-33466][ML][PYTHON] Imputer support mode(most_frequent) strategy No new revisions were added by this update. Summary of changes: .../org/apache/spark/ml/feature/Imputer.scala | 49 +++-- .../org/apache/spark/ml/feature/ImputerSuite.scala | 211 +++-- python/pyspark/ml/feature.py | 5 +- 3 files changed, 144 insertions(+), 121 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Add Data Mechanics to Powered By
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 2e33071 Add Data Mechanics to Powered By 2e33071 is described below commit 2e330710c855f4292cc066c60874a32385a60fb6 Author: Jean-Yves Stephan AuthorDate: Wed Nov 18 12:54:59 2020 -0600 Add Data Mechanics to Powered By Data Mechanics is a managed Spark platform that can be deployed on a Kubernetes cluster inside our customers cloud accounts. We'd love to be on the Powered By Spark page (along other Spark platforms). We contribute to open source projets in the Spark ecosystem (Spark on Kubernetes operator, Data Mechanics Delight). We also use Spark internally for our recommendation engine and logs processing. I tried to be objective / avoid marketing in the description, but I'm open to feedback on changing it. Thanks! Author: Jean-Yves Stephan Closes #299 from jystephan/datamechanics-poweredby. --- powered-by.md| 6 ++ site/powered-by.html | 9 + 2 files changed, 15 insertions(+) diff --git a/powered-by.md b/powered-by.md index 150d402..d314c88 100644 --- a/powered-by.md +++ b/powered-by.md @@ -88,6 +88,12 @@ and external data sources, driving holistic and actionable insights. - We provided a https://www.databricks.com/product;>cloud-optimized platform to run Spark and ML applications on Amazon Web Services and Azure, as well as a comprehensive https://databricks.com/training;>training program. +- https://www.datamechanics.co;>Data Mechanics + - Data Mechanics is a cloud-native Spark platform that can be deployed on a Kubernetes cluster +inside its customers AWS, GCP, or Azure cloud environments. + - Our focus is to make Spark easy-to-use and cost-effective for data engineering workloads. +We also develop the free, cross-platform, and partially open-source Spark monitoring tool +https://www.datamechanics.co/delight;>Data Mechanics Delight. - https://datapipelines.com;>Data Pipelines - Build and schedule ETL pipelines step-by-step via a simple no-code UI. - http://dianping.com;>Dianping.com diff --git a/site/powered-by.html b/site/powered-by.html index b12cf5f..8b93aaa 100644 --- a/site/powered-by.html +++ b/site/powered-by.html @@ -321,6 +321,15 @@ to run Spark and ML applications on Amazon Web Services and Azure, as well as a https://databricks.com/training;>training program. + https://www.datamechanics.co;>Data Mechanics + + Data Mechanics is a cloud-native Spark platform that can be deployed on a Kubernetes cluster +inside its customers AWS, GCP, or Azure cloud environments. + Our focus is to make Spark easy-to-use and cost-effective for data engineering workloads. +We also develop the free, cross-platform, and partially open-source Spark monitoring tool +https://www.datamechanics.co/delight;>Data Mechanics Delight. + + https://datapipelines.com;>Data Pipelines Build and schedule ETL pipelines step-by-step via a simple no-code UI. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Add Delight to third-party projects
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new ae8089e Add Delight to third-party projects ae8089e is described below commit ae8089ee233752422b02fb3cad7a50ed92ec71eb Author: Jean-Yves Stephan AuthorDate: Wed Nov 18 07:33:09 2020 -0600 Add Delight to third-party projects Data Mechanics Delight is a free, hosted, cross-platform monitoring tool for Apache Spark. It is powered by an open-source Spark agent which works on top of any type of Spark infrastructure (commercial or open-source, in the cloud or on-premise, etc). The project is live as of November 2020 and will get significant improvement over the following weeks -- right now it is a dashboard giving you access to the Spark UI (Spark History Server), but we will soon add new debugging screens with memory and CPU metrics to complement and improve on the Spark UI. We hope the Spark community will really benefit from this tool, and so we'd love to be listed on this page. Reference: - https://www.kdnuggets.com/2020/07/monitoring-apache-spark-better-ui.html - https://www.datamechanics.co/delight Author: Jean-Yves Stephan Closes #298 from jystephan/add-delight. --- site/third-party-projects.html | 2 ++ third-party-projects.md| 2 ++ 2 files changed, 4 insertions(+) diff --git a/site/third-party-projects.html b/site/third-party-projects.html index ceea89a..e97c063 100644 --- a/site/third-party-projects.html +++ b/site/third-party-projects.html @@ -268,8 +268,10 @@ transforming, and analyzing genomic data using Apache Spark Performance, Monitoring, and Debugging Tools for Spark + https://github.com/g1thubhub/phil_stopwatch;>Performance and debugging library - A library to analyze Spark and PySpark applications for improving performance and finding the cause of failures + https://www.datamechanics.co/delight;>Data Mechanics Delight - Delight is a free, hosted, cross-platform Spark UI alternative backed by an open-source Spark agent. It features new metrics and visualizations to simplify Spark monitoring and performance tuning. Additional Language Bindings diff --git a/third-party-projects.md b/third-party-projects.md index cf39b26..993ac5c 100644 --- a/third-party-projects.md +++ b/third-party-projects.md @@ -71,7 +71,9 @@ transforming, and analyzing genomic data using Apache Spark - http://rumbledb.org;>Rumble for Apache Spark - A JSONiq engine to query, with a functional language, large, nested, and heterogeneous JSON datasets that do not fit in dataframes. Performance, Monitoring, and Debugging Tools for Spark + - https://github.com/g1thubhub/phil_stopwatch;>Performance and debugging library - A library to analyze Spark and PySpark applications for improving performance and finding the cause of failures +- https://www.datamechanics.co/delight;>Data Mechanics Delight - Delight is a free, hosted, cross-platform Spark UI alternative backed by an open-source Spark agent. It features new metrics and visualizations to simplify Spark monitoring and performance tuning. Additional Language Bindings - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1090b1b -> 02fd52c)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1090b1b [SPARK-32860][DOCS][SQL] Updating documentation about map support in Encoders add 02fd52c [SPARK-33352][CORE][SQL][SS][MLLIB][AVRO][K8S] Fix procedure-like declaration compilation warnings in Scala 2.13 No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala | 2 +- core/src/main/scala/org/apache/spark/TaskEndReason.scala | 4 ++-- .../scala/org/apache/spark/executor/ExecutorMetrics.scala| 6 +++--- .../scala/org/apache/spark/rdd/InputFileBlockHolder.scala| 2 +- .../main/scala/org/apache/spark/rdd/LocalCheckpointRDD.scala | 2 +- .../main/scala/org/apache/spark/scheduler/MapStatus.scala| 2 +- .../scala/org/apache/spark/scheduler/ShuffleMapTask.scala| 2 +- .../spark/scheduler/cluster/StandaloneSchedulerBackend.scala | 3 ++- .../org/apache/spark/shuffle/FetchFailedException.scala | 2 +- .../apache/spark/storage/BlockManagerDecommissioner.scala| 2 +- .../main/scala/org/apache/spark/storage/StorageLevel.scala | 2 +- .../main/scala/org/apache/spark/storage/StorageUtils.scala | 2 +- .../scala/org/apache/spark/util/UninterruptibleThread.scala | 2 +- .../apache/spark/util/collection/ExternalAppendOnlyMap.scala | 2 +- .../spark/executor/CoarseGrainedExecutorBackendSuite.scala | 2 +- .../apache/spark/resource/ResourceProfileManagerSuite.scala | 4 ++-- .../org/apache/spark/resource/ResourceProfileSuite.scala | 4 ++-- .../spark/util/SparkUncaughtExceptionHandlerSuite.scala | 2 +- .../scala/org/apache/spark/sql/avro/AvroDeserializer.scala | 2 +- .../scala/org/apache/spark/sql/avro/AvroSerializer.scala | 2 +- .../apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala | 4 ++-- .../scala/org/apache/spark/metrics/sink/GangliaSink.scala| 6 +++--- .../src/main/scala/org/apache/spark/ml/stat/Summarizer.scala | 2 +- .../scala/org/apache/spark/mllib/feature/ChiSqSelector.scala | 2 +- .../org/apache/spark/mllib/feature/StandardScaler.scala | 2 +- .../org/apache/spark/mllib/tree/configuration/Strategy.scala | 4 ++-- .../org/apache/spark/deploy/k8s/integrationtest/Utils.scala | 6 +++--- .../spark/sql/catalyst/analysis/AlreadyExistException.scala | 2 +- .../apache/spark/sql/catalyst/catalog/SessionCatalog.scala | 4 ++-- .../spark/sql/catalyst/expressions/datetimeExpressions.scala | 2 +- .../src/main/scala/org/apache/spark/sql/SparkSession.scala | 2 +- .../sql/execution/ExternalAppendOnlyUnsafeRowArray.scala | 2 +- .../scala/org/apache/spark/sql/execution/command/views.scala | 2 +- .../execution/datasources/parquet/ParquetReadSupport.scala | 2 +- .../main/scala/org/apache/spark/sql/streaming/progress.scala | 2 +- .../scala/org/apache/spark/sql/test/TestSQLContext.scala | 4 ++-- .../spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala | 2 +- .../spark/sql/hive/thriftserver/HiveSessionImplSuite.scala | 2 +- .../test/scala/org/apache/spark/sql/hive/test/TestHive.scala | 12 ++-- 39 files changed, 58 insertions(+), 57 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1090b1b -> 02fd52c)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1090b1b [SPARK-32860][DOCS][SQL] Updating documentation about map support in Encoders add 02fd52c [SPARK-33352][CORE][SQL][SS][MLLIB][AVRO][K8S] Fix procedure-like declaration compilation warnings in Scala 2.13 No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala | 2 +- core/src/main/scala/org/apache/spark/TaskEndReason.scala | 4 ++-- .../scala/org/apache/spark/executor/ExecutorMetrics.scala| 6 +++--- .../scala/org/apache/spark/rdd/InputFileBlockHolder.scala| 2 +- .../main/scala/org/apache/spark/rdd/LocalCheckpointRDD.scala | 2 +- .../main/scala/org/apache/spark/scheduler/MapStatus.scala| 2 +- .../scala/org/apache/spark/scheduler/ShuffleMapTask.scala| 2 +- .../spark/scheduler/cluster/StandaloneSchedulerBackend.scala | 3 ++- .../org/apache/spark/shuffle/FetchFailedException.scala | 2 +- .../apache/spark/storage/BlockManagerDecommissioner.scala| 2 +- .../main/scala/org/apache/spark/storage/StorageLevel.scala | 2 +- .../main/scala/org/apache/spark/storage/StorageUtils.scala | 2 +- .../scala/org/apache/spark/util/UninterruptibleThread.scala | 2 +- .../apache/spark/util/collection/ExternalAppendOnlyMap.scala | 2 +- .../spark/executor/CoarseGrainedExecutorBackendSuite.scala | 2 +- .../apache/spark/resource/ResourceProfileManagerSuite.scala | 4 ++-- .../org/apache/spark/resource/ResourceProfileSuite.scala | 4 ++-- .../spark/util/SparkUncaughtExceptionHandlerSuite.scala | 2 +- .../scala/org/apache/spark/sql/avro/AvroDeserializer.scala | 2 +- .../scala/org/apache/spark/sql/avro/AvroSerializer.scala | 2 +- .../apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala | 4 ++-- .../scala/org/apache/spark/metrics/sink/GangliaSink.scala| 6 +++--- .../src/main/scala/org/apache/spark/ml/stat/Summarizer.scala | 2 +- .../scala/org/apache/spark/mllib/feature/ChiSqSelector.scala | 2 +- .../org/apache/spark/mllib/feature/StandardScaler.scala | 2 +- .../org/apache/spark/mllib/tree/configuration/Strategy.scala | 4 ++-- .../org/apache/spark/deploy/k8s/integrationtest/Utils.scala | 6 +++--- .../spark/sql/catalyst/analysis/AlreadyExistException.scala | 2 +- .../apache/spark/sql/catalyst/catalog/SessionCatalog.scala | 4 ++-- .../spark/sql/catalyst/expressions/datetimeExpressions.scala | 2 +- .../src/main/scala/org/apache/spark/sql/SparkSession.scala | 2 +- .../sql/execution/ExternalAppendOnlyUnsafeRowArray.scala | 2 +- .../scala/org/apache/spark/sql/execution/command/views.scala | 2 +- .../execution/datasources/parquet/ParquetReadSupport.scala | 2 +- .../main/scala/org/apache/spark/sql/streaming/progress.scala | 2 +- .../scala/org/apache/spark/sql/test/TestSQLContext.scala | 4 ++-- .../spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala | 2 +- .../spark/sql/hive/thriftserver/HiveSessionImplSuite.scala | 2 +- .../test/scala/org/apache/spark/sql/hive/test/TestHive.scala | 12 ++-- 39 files changed, 58 insertions(+), 57 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1090b1b -> 02fd52c)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1090b1b [SPARK-32860][DOCS][SQL] Updating documentation about map support in Encoders add 02fd52c [SPARK-33352][CORE][SQL][SS][MLLIB][AVRO][K8S] Fix procedure-like declaration compilation warnings in Scala 2.13 No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala | 2 +- core/src/main/scala/org/apache/spark/TaskEndReason.scala | 4 ++-- .../scala/org/apache/spark/executor/ExecutorMetrics.scala| 6 +++--- .../scala/org/apache/spark/rdd/InputFileBlockHolder.scala| 2 +- .../main/scala/org/apache/spark/rdd/LocalCheckpointRDD.scala | 2 +- .../main/scala/org/apache/spark/scheduler/MapStatus.scala| 2 +- .../scala/org/apache/spark/scheduler/ShuffleMapTask.scala| 2 +- .../spark/scheduler/cluster/StandaloneSchedulerBackend.scala | 3 ++- .../org/apache/spark/shuffle/FetchFailedException.scala | 2 +- .../apache/spark/storage/BlockManagerDecommissioner.scala| 2 +- .../main/scala/org/apache/spark/storage/StorageLevel.scala | 2 +- .../main/scala/org/apache/spark/storage/StorageUtils.scala | 2 +- .../scala/org/apache/spark/util/UninterruptibleThread.scala | 2 +- .../apache/spark/util/collection/ExternalAppendOnlyMap.scala | 2 +- .../spark/executor/CoarseGrainedExecutorBackendSuite.scala | 2 +- .../apache/spark/resource/ResourceProfileManagerSuite.scala | 4 ++-- .../org/apache/spark/resource/ResourceProfileSuite.scala | 4 ++-- .../spark/util/SparkUncaughtExceptionHandlerSuite.scala | 2 +- .../scala/org/apache/spark/sql/avro/AvroDeserializer.scala | 2 +- .../scala/org/apache/spark/sql/avro/AvroSerializer.scala | 2 +- .../apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala | 4 ++-- .../scala/org/apache/spark/metrics/sink/GangliaSink.scala| 6 +++--- .../src/main/scala/org/apache/spark/ml/stat/Summarizer.scala | 2 +- .../scala/org/apache/spark/mllib/feature/ChiSqSelector.scala | 2 +- .../org/apache/spark/mllib/feature/StandardScaler.scala | 2 +- .../org/apache/spark/mllib/tree/configuration/Strategy.scala | 4 ++-- .../org/apache/spark/deploy/k8s/integrationtest/Utils.scala | 6 +++--- .../spark/sql/catalyst/analysis/AlreadyExistException.scala | 2 +- .../apache/spark/sql/catalyst/catalog/SessionCatalog.scala | 4 ++-- .../spark/sql/catalyst/expressions/datetimeExpressions.scala | 2 +- .../src/main/scala/org/apache/spark/sql/SparkSession.scala | 2 +- .../sql/execution/ExternalAppendOnlyUnsafeRowArray.scala | 2 +- .../scala/org/apache/spark/sql/execution/command/views.scala | 2 +- .../execution/datasources/parquet/ParquetReadSupport.scala | 2 +- .../main/scala/org/apache/spark/sql/streaming/progress.scala | 2 +- .../scala/org/apache/spark/sql/test/TestSQLContext.scala | 4 ++-- .../spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala | 2 +- .../spark/sql/hive/thriftserver/HiveSessionImplSuite.scala | 2 +- .../test/scala/org/apache/spark/sql/hive/test/TestHive.scala | 12 ++-- 39 files changed, 58 insertions(+), 57 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1090b1b -> 02fd52c)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1090b1b [SPARK-32860][DOCS][SQL] Updating documentation about map support in Encoders add 02fd52c [SPARK-33352][CORE][SQL][SS][MLLIB][AVRO][K8S] Fix procedure-like declaration compilation warnings in Scala 2.13 No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala | 2 +- core/src/main/scala/org/apache/spark/TaskEndReason.scala | 4 ++-- .../scala/org/apache/spark/executor/ExecutorMetrics.scala| 6 +++--- .../scala/org/apache/spark/rdd/InputFileBlockHolder.scala| 2 +- .../main/scala/org/apache/spark/rdd/LocalCheckpointRDD.scala | 2 +- .../main/scala/org/apache/spark/scheduler/MapStatus.scala| 2 +- .../scala/org/apache/spark/scheduler/ShuffleMapTask.scala| 2 +- .../spark/scheduler/cluster/StandaloneSchedulerBackend.scala | 3 ++- .../org/apache/spark/shuffle/FetchFailedException.scala | 2 +- .../apache/spark/storage/BlockManagerDecommissioner.scala| 2 +- .../main/scala/org/apache/spark/storage/StorageLevel.scala | 2 +- .../main/scala/org/apache/spark/storage/StorageUtils.scala | 2 +- .../scala/org/apache/spark/util/UninterruptibleThread.scala | 2 +- .../apache/spark/util/collection/ExternalAppendOnlyMap.scala | 2 +- .../spark/executor/CoarseGrainedExecutorBackendSuite.scala | 2 +- .../apache/spark/resource/ResourceProfileManagerSuite.scala | 4 ++-- .../org/apache/spark/resource/ResourceProfileSuite.scala | 4 ++-- .../spark/util/SparkUncaughtExceptionHandlerSuite.scala | 2 +- .../scala/org/apache/spark/sql/avro/AvroDeserializer.scala | 2 +- .../scala/org/apache/spark/sql/avro/AvroSerializer.scala | 2 +- .../apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala | 4 ++-- .../scala/org/apache/spark/metrics/sink/GangliaSink.scala| 6 +++--- .../src/main/scala/org/apache/spark/ml/stat/Summarizer.scala | 2 +- .../scala/org/apache/spark/mllib/feature/ChiSqSelector.scala | 2 +- .../org/apache/spark/mllib/feature/StandardScaler.scala | 2 +- .../org/apache/spark/mllib/tree/configuration/Strategy.scala | 4 ++-- .../org/apache/spark/deploy/k8s/integrationtest/Utils.scala | 6 +++--- .../spark/sql/catalyst/analysis/AlreadyExistException.scala | 2 +- .../apache/spark/sql/catalyst/catalog/SessionCatalog.scala | 4 ++-- .../spark/sql/catalyst/expressions/datetimeExpressions.scala | 2 +- .../src/main/scala/org/apache/spark/sql/SparkSession.scala | 2 +- .../sql/execution/ExternalAppendOnlyUnsafeRowArray.scala | 2 +- .../scala/org/apache/spark/sql/execution/command/views.scala | 2 +- .../execution/datasources/parquet/ParquetReadSupport.scala | 2 +- .../main/scala/org/apache/spark/sql/streaming/progress.scala | 2 +- .../scala/org/apache/spark/sql/test/TestSQLContext.scala | 4 ++-- .../spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala | 2 +- .../spark/sql/hive/thriftserver/HiveSessionImplSuite.scala | 2 +- .../test/scala/org/apache/spark/sql/hive/test/TestHive.scala | 12 ++-- 39 files changed, 58 insertions(+), 57 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1090b1b -> 02fd52c)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1090b1b [SPARK-32860][DOCS][SQL] Updating documentation about map support in Encoders add 02fd52c [SPARK-33352][CORE][SQL][SS][MLLIB][AVRO][K8S] Fix procedure-like declaration compilation warnings in Scala 2.13 No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala | 2 +- core/src/main/scala/org/apache/spark/TaskEndReason.scala | 4 ++-- .../scala/org/apache/spark/executor/ExecutorMetrics.scala| 6 +++--- .../scala/org/apache/spark/rdd/InputFileBlockHolder.scala| 2 +- .../main/scala/org/apache/spark/rdd/LocalCheckpointRDD.scala | 2 +- .../main/scala/org/apache/spark/scheduler/MapStatus.scala| 2 +- .../scala/org/apache/spark/scheduler/ShuffleMapTask.scala| 2 +- .../spark/scheduler/cluster/StandaloneSchedulerBackend.scala | 3 ++- .../org/apache/spark/shuffle/FetchFailedException.scala | 2 +- .../apache/spark/storage/BlockManagerDecommissioner.scala| 2 +- .../main/scala/org/apache/spark/storage/StorageLevel.scala | 2 +- .../main/scala/org/apache/spark/storage/StorageUtils.scala | 2 +- .../scala/org/apache/spark/util/UninterruptibleThread.scala | 2 +- .../apache/spark/util/collection/ExternalAppendOnlyMap.scala | 2 +- .../spark/executor/CoarseGrainedExecutorBackendSuite.scala | 2 +- .../apache/spark/resource/ResourceProfileManagerSuite.scala | 4 ++-- .../org/apache/spark/resource/ResourceProfileSuite.scala | 4 ++-- .../spark/util/SparkUncaughtExceptionHandlerSuite.scala | 2 +- .../scala/org/apache/spark/sql/avro/AvroDeserializer.scala | 2 +- .../scala/org/apache/spark/sql/avro/AvroSerializer.scala | 2 +- .../apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala | 4 ++-- .../scala/org/apache/spark/metrics/sink/GangliaSink.scala| 6 +++--- .../src/main/scala/org/apache/spark/ml/stat/Summarizer.scala | 2 +- .../scala/org/apache/spark/mllib/feature/ChiSqSelector.scala | 2 +- .../org/apache/spark/mllib/feature/StandardScaler.scala | 2 +- .../org/apache/spark/mllib/tree/configuration/Strategy.scala | 4 ++-- .../org/apache/spark/deploy/k8s/integrationtest/Utils.scala | 6 +++--- .../spark/sql/catalyst/analysis/AlreadyExistException.scala | 2 +- .../apache/spark/sql/catalyst/catalog/SessionCatalog.scala | 4 ++-- .../spark/sql/catalyst/expressions/datetimeExpressions.scala | 2 +- .../src/main/scala/org/apache/spark/sql/SparkSession.scala | 2 +- .../sql/execution/ExternalAppendOnlyUnsafeRowArray.scala | 2 +- .../scala/org/apache/spark/sql/execution/command/views.scala | 2 +- .../execution/datasources/parquet/ParquetReadSupport.scala | 2 +- .../main/scala/org/apache/spark/sql/streaming/progress.scala | 2 +- .../scala/org/apache/spark/sql/test/TestSQLContext.scala | 4 ++-- .../spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala | 2 +- .../spark/sql/hive/thriftserver/HiveSessionImplSuite.scala | 2 +- .../test/scala/org/apache/spark/sql/hive/test/TestHive.scala | 12 ++-- 39 files changed, 58 insertions(+), 57 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Add geni to third party projects
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 804383c Add geni to third party projects 804383c is described below commit 804383c1b3beff208f3ae8386b0908136acae192 Author: anthony-khong AuthorDate: Sat Nov 7 08:11:37 2020 -0600 Add geni to third party projects I've tested it with `jekyll build` and `jekyll serve`. It works as expected. Author: anthony-khong Closes #296 from anthony-khong/add-geni-to-third-party-projects. --- site/third-party-projects.html | 1 + third-party-projects.md| 1 + 2 files changed, 2 insertions(+) diff --git a/site/third-party-projects.html b/site/third-party-projects.html index 068f70e..ceea89a 100644 --- a/site/third-party-projects.html +++ b/site/third-party-projects.html @@ -284,6 +284,7 @@ transforming, and analyzing genomic data using Apache Spark https://github.com/TheClimateCorporation/clj-spark;>clj-spark + https://github.com/zero-one-group/geni;>Geni - A Clojure dataframe library that runs on Apache Spark with a focus on optimizing the REPL experience. Groovy diff --git a/third-party-projects.md b/third-party-projects.md index 8f29bbb..cf39b26 100644 --- a/third-party-projects.md +++ b/third-party-projects.md @@ -82,6 +82,7 @@ transforming, and analyzing genomic data using Apache Spark Clojure - https://github.com/TheClimateCorporation/clj-spark;>clj-spark +- https://github.com/zero-one-group/geni;>Geni - A Clojure dataframe library that runs on Apache Spark with a focus on optimizing the REPL experience. Groovy - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Update organization
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 1f53dae Update organization 1f53dae is described below commit 1f53dae5affd0dda51e01d58ce15c9c26e55f0af Author: Felix Cheung <60766335+felixsafegr...@users.noreply.github.com> AuthorDate: Thu Nov 5 18:14:13 2020 -0600 Update organization Author: Felix Cheung <60766335+felixsafegr...@users.noreply.github.com> Closes #297 from felixsafegraph/asf-site. --- committers.md| 2 +- site/committers.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/committers.md b/committers.md index ac568cf..0b21f89 100644 --- a/committers.md +++ b/committers.md @@ -16,7 +16,7 @@ navigation: |Ryan Blue|Netflix| |Joseph Bradley|Databricks| |Matthew Cheah|Palantir| -|Felix Cheung|Uber| +|Felix Cheung|SafeGraph| |Mosharaf Chowdhury|University of Michigan, Ann Arbor| |Bryan Cutler|IBM| |Jason Dai|Intel| diff --git a/site/committers.html b/site/committers.html index 1ddfcd0..2a369e3 100644 --- a/site/committers.html +++ b/site/committers.html @@ -236,7 +236,7 @@ Felix Cheung - Uber + SafeGraph Mosharaf Chowdhury - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 55105a0 [SPARK-33284][WEB-UI] In the Storage UI page, clicking any
field to sort the table will cause the header content to be lost
55105a0 is described below
commit 55105a0784459331d5506eee9f37c2e655a2a6a0
Author: neko
AuthorDate: Tue Nov 3 08:49:52 2020 -0600
[SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort
the table will cause the header content to be lost
### What changes were proposed in this pull request?
In the old version of spark in the storage UI page, the sorting function is
normal, but sorting in the new version will cause the header content to be
lost, So I try to fix the bug.
### Why are the changes needed?
The header field of the table on the page is similar to the following,
**note that each th contains the span attribute**:
```html
Storage Level
.
```
Since [PR#26136](https://github.com/apache/spark/pull/26136), if the `th`
in the table itself contains the `span` attribute, the `span` will be deleted
directly after clicking the sort, and the original header content will be lost.
There are three problems in `sorttable.js`:
1. `sortrevind.class = "sorttable_sortrevind"` in
[sorttab.js#107](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L107)
and `sortfwdind.class = "sorttable_sortfwdind"` in
[sorttab.js#125](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L125)
sorttable_xx attribute should be assigned to`className` instead of `class`,
as javascript uses `rowlists[j].className.search` rather than
`rowlists[j].class.search` to determine whether the component has a sorting
flag or not.
2. `rowlists[j].className.search(/\sorttable_sortrevind\b/)` in
[sorttab.js#120](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L120)
was wrong. The original intention is to search whether `className` contains
the word `sorttable_sortrevind` , but the expression is wrong, it should be
`\bsorttable_sortrevind\b` instead of `\sorttable_sortrevind\b`
3. The if check statement in the following code snippet
([sorttab.js#141](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L141))
was wrong. **If the `search` function does not find the target, it will return
-1, but Boolean(-1) is actually equals true**. This statement will cause span
to be deleted even if it does not contain `sorttable_sortfwdind` or
`sorttable_sortrevind`.
```javascript
rowlists = this.parentNode.getElementsByTagName("span");
for (var j=0; j < rowlists.length; j++) {
if (rowlists[j].className.search(/\bsorttable_sortfwdind\b/)
||
rowlists[j].className.search(/\sorttable_sortrevind\b/) ) {
rowlists[j].parentNode.removeChild(rowlists[j]);
}
}
```
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
The manual test result of the ui page is as below:
Closes #30182 from akiyamaneko/ui_storage_sort_error.
Authored-by: neko
Signed-off-by: Sean Owen
(cherry picked from commit 56c623e98c54fdb4d47c9264ae1b282ecb2b7291)
Signed-off-by: Sean Owen
---
.../main/resources/org/apache/spark/ui/static/sorttable.js | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
b/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
index ecd580e..3f98a03 100644
--- a/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
+++ b/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
@@ -99,12 +99,12 @@ sorttable = {
'sorttable_sorted_reverse');
rowlists = this.parentNode.getElementsByTagName("span");
for (var j=0; j < rowlists.length; j++) {
-if (rowlists[j].className.search(/\bsorttable_sortfwdind\b/)) {
+if (rowlists[j].className.search(/\bsorttable_sortfwdind
[spark] branch branch-3.0 updated: [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 55105a0 [SPARK-33284][WEB-UI] In the Storage UI page, clicking any
field to sort the table will cause the header content to be lost
55105a0 is described below
commit 55105a0784459331d5506eee9f37c2e655a2a6a0
Author: neko
AuthorDate: Tue Nov 3 08:49:52 2020 -0600
[SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort
the table will cause the header content to be lost
### What changes were proposed in this pull request?
In the old version of spark in the storage UI page, the sorting function is
normal, but sorting in the new version will cause the header content to be
lost, So I try to fix the bug.
### Why are the changes needed?
The header field of the table on the page is similar to the following,
**note that each th contains the span attribute**:
```html
Storage Level
.
```
Since [PR#26136](https://github.com/apache/spark/pull/26136), if the `th`
in the table itself contains the `span` attribute, the `span` will be deleted
directly after clicking the sort, and the original header content will be lost.
There are three problems in `sorttable.js`:
1. `sortrevind.class = "sorttable_sortrevind"` in
[sorttab.js#107](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L107)
and `sortfwdind.class = "sorttable_sortfwdind"` in
[sorttab.js#125](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L125)
sorttable_xx attribute should be assigned to`className` instead of `class`,
as javascript uses `rowlists[j].className.search` rather than
`rowlists[j].class.search` to determine whether the component has a sorting
flag or not.
2. `rowlists[j].className.search(/\sorttable_sortrevind\b/)` in
[sorttab.js#120](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L120)
was wrong. The original intention is to search whether `className` contains
the word `sorttable_sortrevind` , but the expression is wrong, it should be
`\bsorttable_sortrevind\b` instead of `\sorttable_sortrevind\b`
3. The if check statement in the following code snippet
([sorttab.js#141](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L141))
was wrong. **If the `search` function does not find the target, it will return
-1, but Boolean(-1) is actually equals true**. This statement will cause span
to be deleted even if it does not contain `sorttable_sortfwdind` or
`sorttable_sortrevind`.
```javascript
rowlists = this.parentNode.getElementsByTagName("span");
for (var j=0; j < rowlists.length; j++) {
if (rowlists[j].className.search(/\bsorttable_sortfwdind\b/)
||
rowlists[j].className.search(/\sorttable_sortrevind\b/) ) {
rowlists[j].parentNode.removeChild(rowlists[j]);
}
}
```
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
The manual test result of the ui page is as below:
Closes #30182 from akiyamaneko/ui_storage_sort_error.
Authored-by: neko
Signed-off-by: Sean Owen
(cherry picked from commit 56c623e98c54fdb4d47c9264ae1b282ecb2b7291)
Signed-off-by: Sean Owen
---
.../main/resources/org/apache/spark/ui/static/sorttable.js | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
b/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
index ecd580e..3f98a03 100644
--- a/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
+++ b/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
@@ -99,12 +99,12 @@ sorttable = {
'sorttable_sorted_reverse');
rowlists = this.parentNode.getElementsByTagName("span");
for (var j=0; j < rowlists.length; j++) {
-if (rowlists[j].className.search(/\bsorttable_sortfwdind\b/)) {
+if (rowlists[j].className.search(/\bsorttable_sortfwdind
[spark] branch master updated (4c8ee88 -> 56c623e)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4c8ee88 [SPARK-33257][PYTHON][SQL] Support Column inputs in PySpark ordering functions (asc*, desc*) add 56c623e [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost No new revisions were added by this update. Summary of changes: .../main/resources/org/apache/spark/ui/static/sorttable.js | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 55105a0 [SPARK-33284][WEB-UI] In the Storage UI page, clicking any
field to sort the table will cause the header content to be lost
55105a0 is described below
commit 55105a0784459331d5506eee9f37c2e655a2a6a0
Author: neko
AuthorDate: Tue Nov 3 08:49:52 2020 -0600
[SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort
the table will cause the header content to be lost
### What changes were proposed in this pull request?
In the old version of spark in the storage UI page, the sorting function is
normal, but sorting in the new version will cause the header content to be
lost, So I try to fix the bug.
### Why are the changes needed?
The header field of the table on the page is similar to the following,
**note that each th contains the span attribute**:
```html
Storage Level
.
```
Since [PR#26136](https://github.com/apache/spark/pull/26136), if the `th`
in the table itself contains the `span` attribute, the `span` will be deleted
directly after clicking the sort, and the original header content will be lost.
There are three problems in `sorttable.js`:
1. `sortrevind.class = "sorttable_sortrevind"` in
[sorttab.js#107](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L107)
and `sortfwdind.class = "sorttable_sortfwdind"` in
[sorttab.js#125](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L125)
sorttable_xx attribute should be assigned to`className` instead of `class`,
as javascript uses `rowlists[j].className.search` rather than
`rowlists[j].class.search` to determine whether the component has a sorting
flag or not.
2. `rowlists[j].className.search(/\sorttable_sortrevind\b/)` in
[sorttab.js#120](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L120)
was wrong. The original intention is to search whether `className` contains
the word `sorttable_sortrevind` , but the expression is wrong, it should be
`\bsorttable_sortrevind\b` instead of `\sorttable_sortrevind\b`
3. The if check statement in the following code snippet
([sorttab.js#141](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L141))
was wrong. **If the `search` function does not find the target, it will return
-1, but Boolean(-1) is actually equals true**. This statement will cause span
to be deleted even if it does not contain `sorttable_sortfwdind` or
`sorttable_sortrevind`.
```javascript
rowlists = this.parentNode.getElementsByTagName("span");
for (var j=0; j < rowlists.length; j++) {
if (rowlists[j].className.search(/\bsorttable_sortfwdind\b/)
||
rowlists[j].className.search(/\sorttable_sortrevind\b/) ) {
rowlists[j].parentNode.removeChild(rowlists[j]);
}
}
```
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
The manual test result of the ui page is as below:
Closes #30182 from akiyamaneko/ui_storage_sort_error.
Authored-by: neko
Signed-off-by: Sean Owen
(cherry picked from commit 56c623e98c54fdb4d47c9264ae1b282ecb2b7291)
Signed-off-by: Sean Owen
---
.../main/resources/org/apache/spark/ui/static/sorttable.js | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
b/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
index ecd580e..3f98a03 100644
--- a/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
+++ b/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
@@ -99,12 +99,12 @@ sorttable = {
'sorttable_sorted_reverse');
rowlists = this.parentNode.getElementsByTagName("span");
for (var j=0; j < rowlists.length; j++) {
-if (rowlists[j].className.search(/\bsorttable_sortfwdind\b/)) {
+if (rowlists[j].className.search(/\bsorttable_sortfwdind
[spark] branch master updated (4c8ee88 -> 56c623e)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4c8ee88 [SPARK-33257][PYTHON][SQL] Support Column inputs in PySpark ordering functions (asc*, desc*) add 56c623e [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost No new revisions were added by this update. Summary of changes: .../main/resources/org/apache/spark/ui/static/sorttable.js | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 55105a0 [SPARK-33284][WEB-UI] In the Storage UI page, clicking any
field to sort the table will cause the header content to be lost
55105a0 is described below
commit 55105a0784459331d5506eee9f37c2e655a2a6a0
Author: neko
AuthorDate: Tue Nov 3 08:49:52 2020 -0600
[SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort
the table will cause the header content to be lost
### What changes were proposed in this pull request?
In the old version of spark in the storage UI page, the sorting function is
normal, but sorting in the new version will cause the header content to be
lost, So I try to fix the bug.
### Why are the changes needed?
The header field of the table on the page is similar to the following,
**note that each th contains the span attribute**:
```html
Storage Level
.
```
Since [PR#26136](https://github.com/apache/spark/pull/26136), if the `th`
in the table itself contains the `span` attribute, the `span` will be deleted
directly after clicking the sort, and the original header content will be lost.
There are three problems in `sorttable.js`:
1. `sortrevind.class = "sorttable_sortrevind"` in
[sorttab.js#107](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L107)
and `sortfwdind.class = "sorttable_sortfwdind"` in
[sorttab.js#125](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L125)
sorttable_xx attribute should be assigned to`className` instead of `class`,
as javascript uses `rowlists[j].className.search` rather than
`rowlists[j].class.search` to determine whether the component has a sorting
flag or not.
2. `rowlists[j].className.search(/\sorttable_sortrevind\b/)` in
[sorttab.js#120](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L120)
was wrong. The original intention is to search whether `className` contains
the word `sorttable_sortrevind` , but the expression is wrong, it should be
`\bsorttable_sortrevind\b` instead of `\sorttable_sortrevind\b`
3. The if check statement in the following code snippet
([sorttab.js#141](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L141))
was wrong. **If the `search` function does not find the target, it will return
-1, but Boolean(-1) is actually equals true**. This statement will cause span
to be deleted even if it does not contain `sorttable_sortfwdind` or
`sorttable_sortrevind`.
```javascript
rowlists = this.parentNode.getElementsByTagName("span");
for (var j=0; j < rowlists.length; j++) {
if (rowlists[j].className.search(/\bsorttable_sortfwdind\b/)
||
rowlists[j].className.search(/\sorttable_sortrevind\b/) ) {
rowlists[j].parentNode.removeChild(rowlists[j]);
}
}
```
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
The manual test result of the ui page is as below:
Closes #30182 from akiyamaneko/ui_storage_sort_error.
Authored-by: neko
Signed-off-by: Sean Owen
(cherry picked from commit 56c623e98c54fdb4d47c9264ae1b282ecb2b7291)
Signed-off-by: Sean Owen
---
.../main/resources/org/apache/spark/ui/static/sorttable.js | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
b/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
index ecd580e..3f98a03 100644
--- a/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
+++ b/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
@@ -99,12 +99,12 @@ sorttable = {
'sorttable_sorted_reverse');
rowlists = this.parentNode.getElementsByTagName("span");
for (var j=0; j < rowlists.length; j++) {
-if (rowlists[j].className.search(/\bsorttable_sortfwdind\b/)) {
+if (rowlists[j].className.search(/\bsorttable_sortfwdind
[spark] branch master updated (4c8ee88 -> 56c623e)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4c8ee88 [SPARK-33257][PYTHON][SQL] Support Column inputs in PySpark ordering functions (asc*, desc*) add 56c623e [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost No new revisions were added by this update. Summary of changes: .../main/resources/org/apache/spark/ui/static/sorttable.js | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 55105a0 [SPARK-33284][WEB-UI] In the Storage UI page, clicking any
field to sort the table will cause the header content to be lost
55105a0 is described below
commit 55105a0784459331d5506eee9f37c2e655a2a6a0
Author: neko
AuthorDate: Tue Nov 3 08:49:52 2020 -0600
[SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort
the table will cause the header content to be lost
### What changes were proposed in this pull request?
In the old version of spark in the storage UI page, the sorting function is
normal, but sorting in the new version will cause the header content to be
lost, So I try to fix the bug.
### Why are the changes needed?
The header field of the table on the page is similar to the following,
**note that each th contains the span attribute**:
```html
Storage Level
.
```
Since [PR#26136](https://github.com/apache/spark/pull/26136), if the `th`
in the table itself contains the `span` attribute, the `span` will be deleted
directly after clicking the sort, and the original header content will be lost.
There are three problems in `sorttable.js`:
1. `sortrevind.class = "sorttable_sortrevind"` in
[sorttab.js#107](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L107)
and `sortfwdind.class = "sorttable_sortfwdind"` in
[sorttab.js#125](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L125)
sorttable_xx attribute should be assigned to`className` instead of `class`,
as javascript uses `rowlists[j].className.search` rather than
`rowlists[j].class.search` to determine whether the component has a sorting
flag or not.
2. `rowlists[j].className.search(/\sorttable_sortrevind\b/)` in
[sorttab.js#120](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L120)
was wrong. The original intention is to search whether `className` contains
the word `sorttable_sortrevind` , but the expression is wrong, it should be
`\bsorttable_sortrevind\b` instead of `\sorttable_sortrevind\b`
3. The if check statement in the following code snippet
([sorttab.js#141](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L141))
was wrong. **If the `search` function does not find the target, it will return
-1, but Boolean(-1) is actually equals true**. This statement will cause span
to be deleted even if it does not contain `sorttable_sortfwdind` or
`sorttable_sortrevind`.
```javascript
rowlists = this.parentNode.getElementsByTagName("span");
for (var j=0; j < rowlists.length; j++) {
if (rowlists[j].className.search(/\bsorttable_sortfwdind\b/)
||
rowlists[j].className.search(/\sorttable_sortrevind\b/) ) {
rowlists[j].parentNode.removeChild(rowlists[j]);
}
}
```
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
The manual test result of the ui page is as below:
Closes #30182 from akiyamaneko/ui_storage_sort_error.
Authored-by: neko
Signed-off-by: Sean Owen
(cherry picked from commit 56c623e98c54fdb4d47c9264ae1b282ecb2b7291)
Signed-off-by: Sean Owen
---
.../main/resources/org/apache/spark/ui/static/sorttable.js | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
b/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
index ecd580e..3f98a03 100644
--- a/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
+++ b/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
@@ -99,12 +99,12 @@ sorttable = {
'sorttable_sorted_reverse');
rowlists = this.parentNode.getElementsByTagName("span");
for (var j=0; j < rowlists.length; j++) {
-if (rowlists[j].className.search(/\bsorttable_sortfwdind\b/)) {
+if (rowlists[j].className.search(/\bsorttable_sortfwdind
[spark] branch master updated (4c8ee88 -> 56c623e)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4c8ee88 [SPARK-33257][PYTHON][SQL] Support Column inputs in PySpark ordering functions (asc*, desc*) add 56c623e [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost No new revisions were added by this update. Summary of changes: .../main/resources/org/apache/spark/ui/static/sorttable.js | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 56c623e [SPARK-33284][WEB-UI] In the Storage UI page, clicking any
field to sort the table will cause the header content to be lost
56c623e is described below
commit 56c623e98c54fdb4d47c9264ae1b282ecb2b7291
Author: neko
AuthorDate: Tue Nov 3 08:49:52 2020 -0600
[SPARK-33284][WEB-UI] In the Storage UI page, clicking any field to sort
the table will cause the header content to be lost
### What changes were proposed in this pull request?
In the old version of spark in the storage UI page, the sorting function is
normal, but sorting in the new version will cause the header content to be
lost, So I try to fix the bug.
### Why are the changes needed?
The header field of the table on the page is similar to the following,
**note that each th contains the span attribute**:
```html
Storage Level
.
```
Since [PR#26136](https://github.com/apache/spark/pull/26136), if the `th`
in the table itself contains the `span` attribute, the `span` will be deleted
directly after clicking the sort, and the original header content will be lost.
There are three problems in `sorttable.js`:
1. `sortrevind.class = "sorttable_sortrevind"` in
[sorttab.js#107](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L107)
and `sortfwdind.class = "sorttable_sortfwdind"` in
[sorttab.js#125](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L125)
sorttable_xx attribute should be assigned to`className` instead of `class`,
as javascript uses `rowlists[j].className.search` rather than
`rowlists[j].class.search` to determine whether the component has a sorting
flag or not.
2. `rowlists[j].className.search(/\sorttable_sortrevind\b/)` in
[sorttab.js#120](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L120)
was wrong. The original intention is to search whether `className` contains
the word `sorttable_sortrevind` , but the expression is wrong, it should be
`\bsorttable_sortrevind\b` instead of `\sorttable_sortrevind\b`
3. The if check statement in the following code snippet
([sorttab.js#141](https://github.com/apache/spark/blob/9d5e48ea95d1c3017a51ff69584f32a18901b2b5/core/src/main/resources/org/apache/spark/ui/static/sorttable.js#L141))
was wrong. **If the `search` function does not find the target, it will return
-1, but Boolean(-1) is actually equals true**. This statement will cause span
to be deleted even if it does not contain `sorttable_sortfwdind` or
`sorttable_sortrevind`.
```javascript
rowlists = this.parentNode.getElementsByTagName("span");
for (var j=0; j < rowlists.length; j++) {
if (rowlists[j].className.search(/\bsorttable_sortfwdind\b/)
||
rowlists[j].className.search(/\sorttable_sortrevind\b/) ) {
rowlists[j].parentNode.removeChild(rowlists[j]);
}
}
```
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
The manual test result of the ui page is as below:
Closes #30182 from akiyamaneko/ui_storage_sort_error.
Authored-by: neko
Signed-off-by: Sean Owen
---
.../main/resources/org/apache/spark/ui/static/sorttable.js | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
b/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
index ecd580e..3f98a03 100644
--- a/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
+++ b/core/src/main/resources/org/apache/spark/ui/static/sorttable.js
@@ -99,12 +99,12 @@ sorttable = {
'sorttable_sorted_reverse');
rowlists = this.parentNode.getElementsByTagName("span");
for (var j=0; j < rowlists.length; j++) {
-if (rowlists[j].className.search(/\bsorttable_sortfwdind\b/)) {
+if (rowlists[j].className.search(/\bsorttable_sortfwdind\b/)
!= -1) {
rowlists[j].parentNode.removeChild(rowlists[j]);
}
}
[spark] branch master updated (dcb08204 -> 618695b)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from dcb08204 [SPARK-32785][SQL][DOCS][FOLLOWUP] Update migaration guide for incomplete interval literals add 618695b [SPARK-33111][ML][FOLLOW-UP] aft transform optimization - predictQuantiles No new revisions were added by this update. Summary of changes: .../ml/regression/AFTSurvivalRegression.scala | 42 -- .../ml/regression/AFTSurvivalRegressionSuite.scala | 2 +- 2 files changed, 25 insertions(+), 19 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (dcb08204 -> 618695b)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from dcb08204 [SPARK-32785][SQL][DOCS][FOLLOWUP] Update migaration guide for incomplete interval literals add 618695b [SPARK-33111][ML][FOLLOW-UP] aft transform optimization - predictQuantiles No new revisions were added by this update. Summary of changes: .../ml/regression/AFTSurvivalRegression.scala | 42 -- .../ml/regression/AFTSurvivalRegressionSuite.scala | 2 +- 2 files changed, 25 insertions(+), 19 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (dcb08204 -> 618695b)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from dcb08204 [SPARK-32785][SQL][DOCS][FOLLOWUP] Update migaration guide for incomplete interval literals add 618695b [SPARK-33111][ML][FOLLOW-UP] aft transform optimization - predictQuantiles No new revisions were added by this update. Summary of changes: .../ml/regression/AFTSurvivalRegression.scala | 42 -- .../ml/regression/AFTSurvivalRegressionSuite.scala | 2 +- 2 files changed, 25 insertions(+), 19 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (dcb08204 -> 618695b)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from dcb08204 [SPARK-32785][SQL][DOCS][FOLLOWUP] Update migaration guide for incomplete interval literals add 618695b [SPARK-33111][ML][FOLLOW-UP] aft transform optimization - predictQuantiles No new revisions were added by this update. Summary of changes: .../ml/regression/AFTSurvivalRegression.scala | 42 -- .../ml/regression/AFTSurvivalRegressionSuite.scala | 2 +- 2 files changed, 25 insertions(+), 19 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (dcb08204 -> 618695b)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from dcb08204 [SPARK-32785][SQL][DOCS][FOLLOWUP] Update migaration guide for incomplete interval literals add 618695b [SPARK-33111][ML][FOLLOW-UP] aft transform optimization - predictQuantiles No new revisions were added by this update. Summary of changes: .../ml/regression/AFTSurvivalRegression.scala | 42 -- .../ml/regression/AFTSurvivalRegressionSuite.scala | 2 +- 2 files changed, 25 insertions(+), 19 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Replace test-only to testOnly in Developer tools page
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new fe3e503 Replace test-only to testOnly in Developer tools page fe3e503 is described below commit fe3e5037d2eef83da136b9f8c66e7e2d6904d2d4 Author: HyukjinKwon AuthorDate: Thu Oct 15 18:15:03 2020 -0500 Replace test-only to testOnly in Developer tools page See also https://github.com/apache/spark/pull/30028. After SBT was upgraded to 1.3, `test-only` should be `testOnly`. Author: HyukjinKwon Closes #295 from HyukjinKwon/test-only-sbt-upgrade. --- developer-tools.md| 2 +- site/developer-tools.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/developer-tools.md b/developer-tools.md index 0078538..9d82a25 100644 --- a/developer-tools.md +++ b/developer-tools.md @@ -267,7 +267,7 @@ it's due to a classpath issue (some classes were probably not compiled). To fix sufficient to run a test from the command line: ``` -build/sbt "test-only org.apache.spark.rdd.SortingSuite" +build/sbt "testOnly org.apache.spark.rdd.SortingSuite" ``` Running Different Test Permutations on Jenkins diff --git a/site/developer-tools.html b/site/developer-tools.html index 86918d8..b9ecb5e 100644 --- a/site/developer-tools.html +++ b/site/developer-tools.html @@ -447,7 +447,7 @@ java.lang.NullPointerException its due to a classpath issue (some classes were probably not compiled). To fix this, it sufficient to run a test from the command line: -build/sbt "test-only org.apache.spark.rdd.SortingSuite" +build/sbt "testOnly org.apache.spark.rdd.SortingSuite" Running Different Test Permutations on Jenkins - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9e37464 -> ba69d68)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9e37464 [SPARK-33078][SQL] Add config for json expression optimization add ba69d68 [SPARK-33080][BUILD] Replace fatal warnings snippet No new revisions were added by this update. Summary of changes: .../shuffle/HostLocalShuffleReadingSuite.scala | 1 + .../apache/spark/storage/BlockManagerSuite.scala | 4 +- project/SparkBuild.scala | 84 -- .../sql/catalyst/optimizer/OptimizerSuite.scala| 2 +- .../spark/sql/catalyst/util/UnsafeArraySuite.scala | 3 +- .../apache/spark/sql/connector/InMemoryTable.scala | 8 +++ .../spark/sql/streaming/StreamingQuerySuite.scala | 2 +- .../spark/sql/hive/thriftserver/CliSuite.scala | 6 +- 8 files changed, 62 insertions(+), 48 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9e37464 -> ba69d68)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9e37464 [SPARK-33078][SQL] Add config for json expression optimization add ba69d68 [SPARK-33080][BUILD] Replace fatal warnings snippet No new revisions were added by this update. Summary of changes: .../shuffle/HostLocalShuffleReadingSuite.scala | 1 + .../apache/spark/storage/BlockManagerSuite.scala | 4 +- project/SparkBuild.scala | 84 -- .../sql/catalyst/optimizer/OptimizerSuite.scala| 2 +- .../spark/sql/catalyst/util/UnsafeArraySuite.scala | 3 +- .../apache/spark/sql/connector/InMemoryTable.scala | 8 +++ .../spark/sql/streaming/StreamingQuerySuite.scala | 2 +- .../spark/sql/hive/thriftserver/CliSuite.scala | 6 +- 8 files changed, 62 insertions(+), 48 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9e37464 -> ba69d68)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9e37464 [SPARK-33078][SQL] Add config for json expression optimization add ba69d68 [SPARK-33080][BUILD] Replace fatal warnings snippet No new revisions were added by this update. Summary of changes: .../shuffle/HostLocalShuffleReadingSuite.scala | 1 + .../apache/spark/storage/BlockManagerSuite.scala | 4 +- project/SparkBuild.scala | 84 -- .../sql/catalyst/optimizer/OptimizerSuite.scala| 2 +- .../spark/sql/catalyst/util/UnsafeArraySuite.scala | 3 +- .../apache/spark/sql/connector/InMemoryTable.scala | 8 +++ .../spark/sql/streaming/StreamingQuerySuite.scala | 2 +- .../spark/sql/hive/thriftserver/CliSuite.scala | 6 +- 8 files changed, 62 insertions(+), 48 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9e37464 -> ba69d68)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9e37464 [SPARK-33078][SQL] Add config for json expression optimization add ba69d68 [SPARK-33080][BUILD] Replace fatal warnings snippet No new revisions were added by this update. Summary of changes: .../shuffle/HostLocalShuffleReadingSuite.scala | 1 + .../apache/spark/storage/BlockManagerSuite.scala | 4 +- project/SparkBuild.scala | 84 -- .../sql/catalyst/optimizer/OptimizerSuite.scala| 2 +- .../spark/sql/catalyst/util/UnsafeArraySuite.scala | 3 +- .../apache/spark/sql/connector/InMemoryTable.scala | 8 +++ .../spark/sql/streaming/StreamingQuerySuite.scala | 2 +- .../spark/sql/hive/thriftserver/CliSuite.scala | 6 +- 8 files changed, 62 insertions(+), 48 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9e37464 -> ba69d68)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9e37464 [SPARK-33078][SQL] Add config for json expression optimization add ba69d68 [SPARK-33080][BUILD] Replace fatal warnings snippet No new revisions were added by this update. Summary of changes: .../shuffle/HostLocalShuffleReadingSuite.scala | 1 + .../apache/spark/storage/BlockManagerSuite.scala | 4 +- project/SparkBuild.scala | 84 -- .../sql/catalyst/optimizer/OptimizerSuite.scala| 2 +- .../spark/sql/catalyst/util/UnsafeArraySuite.scala | 3 +- .../apache/spark/sql/connector/InMemoryTable.scala | 8 +++ .../spark/sql/streaming/StreamingQuerySuite.scala | 2 +- .../spark/sql/hive/thriftserver/CliSuite.scala | 6 +- 8 files changed, 62 insertions(+), 48 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
svn commit: r41940 - /release/spark/KEYS
Author: srowen Date: Thu Oct 15 13:18:43 2020 New Revision: 41940 Log: Add missing key for Ruifeng to Spark KEYS Modified: release/spark/KEYS Modified: release/spark/KEYS == --- release/spark/KEYS (original) +++ release/spark/KEYS Thu Oct 15 13:18:43 2020 @@ -1413,3 +1413,60 @@ Hy4V/RJiJHCHekSXHCNoxgJz8Jc= =+90F -END PGP PUBLIC KEY BLOCK- +pub rsa4096 2020-08-05 [SC] + 5146FBDC4B90744EA948035795E0EE38CF98F9F4 +uid [ultimate] Ruifeng Zheng (CODE SIGNING KEY) +sub rsa4096 2020-08-05 [E] + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQINBF8qcTwBEADNwwXl2aEihlTGLo4uH4CHyF0Et2qJa0widBEj+LkQg1Alsxml +Eqh/yea5QJObPmtfvIH8qgtUhOUUANH6+GY7XTtTrd4SU2jYupns1Z7HuTHx75IX +oi2i2kzffWXPS4LMe9b7QjceHWsAIqKpmG2/tY1Wm9m0emwfa+qDNZaKQFAP+tnp +24CVGUiNQbUyxDDUlpKHszB2Kw+pj/pFsNqAv30x2QweIVfGTYZAhzgzybR3Oid6 +8Bf1BbkWF9UH5at0Y2+Q9dvhMewRxgbW9jonA9OMy4EBfRqRzauYcjz0F7Pzy+Lk +fd1/9SE4eFIGVts2XTT//AK0IUwoAdjmOT+aq9x1qSqxzrHqgIj5pssn7sPheUAB +67a0oiM7r92a/URvskU4csI1LxWJz2oqTeRa1K7cmvw/4nxHqkNCizbXhVWNLiGH +VC3tZZdgHliMCehCKmFFw9/r0F+XM0cJesUhhbfVL0rPLUaA7tZ5zefKaeDUpUDt +JB/XFv5am02yInlT+n4Er6fxW9Pp0bEYgBVZY3Agr11VxcKFGhS3eb4iDl+obFN9 +UnuG7Vkm7l8j5NWPdkuzMzLG1+wdUbz9EcHhzt3NLutyo0nzt3uZiZjQONagIwhV +5SvdTG6eS6QWxKPbgGETmqGaEqKMXbumXTnqgEHm82w2P4J9OU72X+rkPQARAQAB +tDZSdWlmZW5nIFpoZW5nIChDT0RFIFNJR05JTkcgS0VZKSA8cnVpZmVuZ3pAYXBh +Y2hlLm9yZz6JAk4EEwEKADgWIQRRRvvcS5B0TqlIA1eV4O44z5j59AUCXypxPAIb +AwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRCV4O44z5j59P1rD/4mkpvICxd4 +tg7r5zgaVtQIaBwgjK9OnsStAiWkpe/PzG3Q0aDNGBO8vuwhI6LHhgU9fea3Mw0N +tpTFB00qwagKckXTAX9hj2EVcjH6KxUEoDlGyEZHLsUgizzGLy8laF2XaHn/Bs8D +fl41iF+fvl/XYD8y8f5F6eIWaJROx73Bjk22fWhndPJgtO4HeaL5/JOMdUvU12AE +Ipk22YBm416rDYixJucoGLlGfRuxMAImlaPgM18NAb25biU8Rd15+c3HgDtVBrTI +0C3XljKcio1cVAY1MyrcC0mKaTLIhsngD+DsjDItWzp8BYg3kHPFfh/8AMDNA960 +3ACcq436UdoqPzHqA/B6dRgw1M3F+dSlX24DzYZ3qz/sn2d2HmdkMO9+4epnk7lz +gxwz14F0mTPKiH/rx4dXo4A/D/KurFA8Ed1Div4azDwlKkk5au0C8KrjJstEy27u +5x41GtY5XoyI+lGGydMC6yrvoDPLxGLZaOIUgkN6hkz/BrkTZ/oEFybx4XxLkZg6 +gQVQTrtqsXZXEL5IEMD8mCP5TYrrTFRwBQNW6ngR7L7kYGb0ksB5TwIu0ZntRZIY +XgVXMbBCM3ehAWdXR0oj25gtkLzRCZSAkPKK1uMaEbksRrb5uuAnX/F8LxAeunQM +P2jbZ3ydT2pMPi8X1TYWCYa+56TaxjCzAbkCDQRfKnE8ARAAtG+2ME5GIjWPofPR +KZkhlMnjbwYL6bVcy2vUmfzuM/sM2SjP8W3x/yPZA+HHfe7+FRaeBzcOhCBuYTKF +K7F+fw1woljDOU1atVtBJu0MH7r47my/MPtuRg0bltT3AE3qJoAQZeDEefJvCcfZ +TPmZN1jETjjPRe045zkhk9tFt1ZB7d8wk+yo3PWwp0iX2p9LkyiCLvYFBqs0McLW +wQI4fgmeA5fiyMpJZJohZjR170Qbyk+QQ3Jri8EWeZvwJEfAPVxVMt1DOxPBv3PI +2AfYM0V8brEVF/2N/Lorpt3LcN+mAhJfASy4RimvE08gj5nJn3+aA98B3uPCZ6AN +IEOYIZPNWseYCWCqDHbiFFqaRIxnLfxgTygJzw8lvBAoBr15ZG5e6Xe4JRAn3Cvu +frkMs4xlnqhFR1tzNezWLn/j7+dOVHzSiPTiKGAjwEiLvusaxNhkVKqrDu3QoPFu +ogvtfyeSPVYcsP6F5IJ2LQzT5Cq8h+H1/+7/tQrhSWd/KAzRw5+rePuoecbaodfr +VaG9sqSMe/GlCBuhqGG4Y3mFaHnemgZaCj4jm0wvjyPo1ik5V9j4TU6nKPEEOXX3 +x4mHHflEOWslHeT9xX2aG5dnh7bHQnJLbbNbEilJxXtKeeuA/iOyPq6+lHWVDJYf +cDuYdAKr2Gzjffg3pfmN2zlOla8AEQEAAYkCNgQYAQoAIBYhBFFG+9xLkHROqUgD +V5Xg7jjPmPn0BQJfKnE8AhsMAAoJEJXg7jjPmPn0N0UQAIZKhyKBnad4A791bx+4 +iHU/zglxq73nUfRoIy1pxt7Sa7YTSG3029Mj6fsHCr5tCHmcSS8leF28CAz8Qs8S +UHf/i+aDk6wDk20V80jUYa6DkuUaolf2GxGBW3dwJKufq/L2lgPhN0R2MIL2gQM+ +M5EB+tpD+69laGrMVFqztSPcFpJjysnDKDiu5EFVD74zU8F9jn3kDD50DTx3LvrD +JD/X5y6TaxUw1TAjdUgrkG/PARxJu3za4anHMiMfHah6Y6dz7ROtCKFMjWH25y28 +O8TMJnVUZdp6uLu3PzWjit9bwB7UuVVlBUQX9piMr/A5WtucpucLGwn7G0ejuJyE +3Bq502QehItW6Ft0nlI8HGoecHXLQK3HUpLSf3BkBlXNz165iImG/RAgZUucbhHb +u2Bmj4c9bQuZucQ4j3dUsXc3y4M8V14d5V1MXceWZ0sGkUcXEzJQnQcy98yn5b9K +71zAI0i5UmtKXU/Xjss+WAfInBzpyq0bk9f9pur9UP7/2visiHQw70AfrSutXWiU +HzpIypF5A8FUA+gcNsUUPkbm4JeTTxTxb0AEb6iBC5eYmDdehhcMeYnNnE/STejM +5hUDBpGDAkbw0Wgolr/Qpxfxlkzstz8XSy2U6BVxkan1Oji889sTamWhHzLf7Ofo +eGh3VPV1RM3YCRkGY7/1fheg +=/4cF +-END PGP PUBLIC KEY BLOCK- - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (05a62dc -> d8c4a47)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 05a62dc [SPARK-33134][SQL] Return partial results only for root JSON objects add d8c4a47 [SPARK-33061][SQL] Expose inverse hyperbolic trig functions through sql.functions API No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/sql/functions.scala | 50 +- .../org/apache/spark/sql/MathFunctionsSuite.scala | 15 +++ 2 files changed, 64 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (05a62dc -> d8c4a47)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 05a62dc [SPARK-33134][SQL] Return partial results only for root JSON objects add d8c4a47 [SPARK-33061][SQL] Expose inverse hyperbolic trig functions through sql.functions API No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/sql/functions.scala | 50 +- .../org/apache/spark/sql/MathFunctionsSuite.scala | 15 +++ 2 files changed, 64 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (05a62dc -> d8c4a47)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 05a62dc [SPARK-33134][SQL] Return partial results only for root JSON objects add d8c4a47 [SPARK-33061][SQL] Expose inverse hyperbolic trig functions through sql.functions API No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/sql/functions.scala | 50 +- .../org/apache/spark/sql/MathFunctionsSuite.scala | 15 +++ 2 files changed, 64 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (05a62dc -> d8c4a47)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 05a62dc [SPARK-33134][SQL] Return partial results only for root JSON objects add d8c4a47 [SPARK-33061][SQL] Expose inverse hyperbolic trig functions through sql.functions API No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/sql/functions.scala | 50 +- .../org/apache/spark/sql/MathFunctionsSuite.scala | 15 +++ 2 files changed, 64 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (05a62dc -> d8c4a47)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 05a62dc [SPARK-33134][SQL] Return partial results only for root JSON objects add d8c4a47 [SPARK-33061][SQL] Expose inverse hyperbolic trig functions through sql.functions API No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/sql/functions.scala | 50 +- .../org/apache/spark/sql/MathFunctionsSuite.scala | 15 +++ 2 files changed, 64 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a0e3244 -> ed2fe8d)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a0e3244 [SPARK-32704][SQL][FOLLOWUP] Corrects version values of plan logging configs in SQLConf add ed2fe8d [SPARK-33111][ML] aft transform optimization No new revisions were added by this update. Summary of changes: .../apache/spark/ml/regression/AFTSurvivalRegression.scala | 12 ++-- 1 file changed, 10 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a0e3244 -> ed2fe8d)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a0e3244 [SPARK-32704][SQL][FOLLOWUP] Corrects version values of plan logging configs in SQLConf add ed2fe8d [SPARK-33111][ML] aft transform optimization No new revisions were added by this update. Summary of changes: .../apache/spark/ml/regression/AFTSurvivalRegression.scala | 12 ++-- 1 file changed, 10 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a0e3244 -> ed2fe8d)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a0e3244 [SPARK-32704][SQL][FOLLOWUP] Corrects version values of plan logging configs in SQLConf add ed2fe8d [SPARK-33111][ML] aft transform optimization No new revisions were added by this update. Summary of changes: .../apache/spark/ml/regression/AFTSurvivalRegression.scala | 12 ++-- 1 file changed, 10 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Update Spark 3.1 Release Window
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 0cd0bdc Update Spark 3.1 Release Window 0cd0bdc is described below commit 0cd0bdc80503882b4737db7e77cc8f9d17ec12ca Author: Xiao Li AuthorDate: Mon Oct 12 08:59:57 2020 -0500 Update Spark 3.1 Release Window This PR is to update the release window based on the public discussion in the dev list: http://apache-spark-developers-list.1001551.n3.nabble.com/Re-Apache-Spark-3-1-Preparation-Status-Oct-2020-td30255.html Author: Xiao Li Closes #293 from gatorsmile/3.1release. --- site/versioning-policy.html | 6 +++--- versioning-policy.md| 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/site/versioning-policy.html b/site/versioning-policy.html index 3dcf784..acc68f2 100644 --- a/site/versioning-policy.html +++ b/site/versioning-policy.html @@ -339,15 +339,15 @@ in between feature releases. Major releases do not happen according to a fixed s - Early Nov 2020 + Early Dec 2020 Code freeze. Release branch cut. - Mid Nov 2020 + Mid Dec 2020 QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged. - Early Dec 2020 + Early Jan 2020 Release candidates (RC), voting, etc. until final release passes diff --git a/versioning-policy.md b/versioning-policy.md index a081890..3fe2f86 100644 --- a/versioning-policy.md +++ b/versioning-policy.md @@ -107,9 +107,9 @@ in between feature releases. Major releases do not happen according to a fixed s | Date | Event | | - | - | -| Early Nov 2020 | Code freeze. Release branch cut.| -| Mid Nov 2020 | QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.| -| Early Dec 2020 | Release candidates (RC), voting, etc. until final release passes| +| Early Dec 2020 | Code freeze. Release branch cut.| +| Mid Dec 2020 | QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.| +| Early Jan 2020 | Release candidates (RC), voting, etc. until final release passes| Maintenance Releases and EOL - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Update Spark 3.1 Release Window
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 0cd0bdc Update Spark 3.1 Release Window 0cd0bdc is described below commit 0cd0bdc80503882b4737db7e77cc8f9d17ec12ca Author: Xiao Li AuthorDate: Mon Oct 12 08:59:57 2020 -0500 Update Spark 3.1 Release Window This PR is to update the release window based on the public discussion in the dev list: http://apache-spark-developers-list.1001551.n3.nabble.com/Re-Apache-Spark-3-1-Preparation-Status-Oct-2020-td30255.html Author: Xiao Li Closes #293 from gatorsmile/3.1release. --- site/versioning-policy.html | 6 +++--- versioning-policy.md| 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/site/versioning-policy.html b/site/versioning-policy.html index 3dcf784..acc68f2 100644 --- a/site/versioning-policy.html +++ b/site/versioning-policy.html @@ -339,15 +339,15 @@ in between feature releases. Major releases do not happen according to a fixed s - Early Nov 2020 + Early Dec 2020 Code freeze. Release branch cut. - Mid Nov 2020 + Mid Dec 2020 QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged. - Early Dec 2020 + Early Jan 2020 Release candidates (RC), voting, etc. until final release passes diff --git a/versioning-policy.md b/versioning-policy.md index a081890..3fe2f86 100644 --- a/versioning-policy.md +++ b/versioning-policy.md @@ -107,9 +107,9 @@ in between feature releases. Major releases do not happen according to a fixed s | Date | Event | | - | - | -| Early Nov 2020 | Code freeze. Release branch cut.| -| Mid Nov 2020 | QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.| -| Early Dec 2020 | Release candidates (RC), voting, etc. until final release passes| +| Early Dec 2020 | Code freeze. Release branch cut.| +| Mid Dec 2020 | QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.| +| Early Jan 2020 | Release candidates (RC), voting, etc. until final release passes| Maintenance Releases and EOL - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a0e3244 -> ed2fe8d)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a0e3244 [SPARK-32704][SQL][FOLLOWUP] Corrects version values of plan logging configs in SQLConf add ed2fe8d [SPARK-33111][ML] aft transform optimization No new revisions were added by this update. Summary of changes: .../apache/spark/ml/regression/AFTSurvivalRegression.scala | 12 ++-- 1 file changed, 10 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Update Spark 3.1 Release Window
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 0cd0bdc Update Spark 3.1 Release Window 0cd0bdc is described below commit 0cd0bdc80503882b4737db7e77cc8f9d17ec12ca Author: Xiao Li AuthorDate: Mon Oct 12 08:59:57 2020 -0500 Update Spark 3.1 Release Window This PR is to update the release window based on the public discussion in the dev list: http://apache-spark-developers-list.1001551.n3.nabble.com/Re-Apache-Spark-3-1-Preparation-Status-Oct-2020-td30255.html Author: Xiao Li Closes #293 from gatorsmile/3.1release. --- site/versioning-policy.html | 6 +++--- versioning-policy.md| 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/site/versioning-policy.html b/site/versioning-policy.html index 3dcf784..acc68f2 100644 --- a/site/versioning-policy.html +++ b/site/versioning-policy.html @@ -339,15 +339,15 @@ in between feature releases. Major releases do not happen according to a fixed s - Early Nov 2020 + Early Dec 2020 Code freeze. Release branch cut. - Mid Nov 2020 + Mid Dec 2020 QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged. - Early Dec 2020 + Early Jan 2020 Release candidates (RC), voting, etc. until final release passes diff --git a/versioning-policy.md b/versioning-policy.md index a081890..3fe2f86 100644 --- a/versioning-policy.md +++ b/versioning-policy.md @@ -107,9 +107,9 @@ in between feature releases. Major releases do not happen according to a fixed s | Date | Event | | - | - | -| Early Nov 2020 | Code freeze. Release branch cut.| -| Mid Nov 2020 | QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.| -| Early Dec 2020 | Release candidates (RC), voting, etc. until final release passes| +| Early Dec 2020 | Code freeze. Release branch cut.| +| Mid Dec 2020 | QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.| +| Early Jan 2020 | Release candidates (RC), voting, etc. until final release passes| Maintenance Releases and EOL - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a0e3244 -> ed2fe8d)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a0e3244 [SPARK-32704][SQL][FOLLOWUP] Corrects version values of plan logging configs in SQLConf add ed2fe8d [SPARK-33111][ML] aft transform optimization No new revisions were added by this update. Summary of changes: .../apache/spark/ml/regression/AFTSurvivalRegression.scala | 12 ++-- 1 file changed, 10 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
