[spark] branch master updated: [SPARK-31791][CORE][TEST] Improve cache block migration test reliability
This is an automated email from the ASF dual-hosted git repository.
holden pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 721cba5 [SPARK-31791][CORE][TEST] Improve cache block migration test
reliability
721cba5 is described below
commit 721cba540292d8d76102b18922dabe2a7d918dc5
Author: Holden Karau
AuthorDate: Fri May 22 18:19:41 2020 -0700
[SPARK-31791][CORE][TEST] Improve cache block migration test reliability
### What changes were proposed in this pull request?
Increase the timeout and register the listener earlier to avoid any race
condition of the job starting before the listener is registered.
### Why are the changes needed?
The test is currently semi-flaky.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
I'm currently running the following bash script on my dev machine to verify
the flakiness decreases. It has gotten to 356 iterations without any test
failures so I believe issue is fixed.
```
set -ex
./build/sbt clean compile package
((failures=0))
for (( i=0;i<1000;++i )); do
echo "Run $i"
((failed=0))
./build/sbt "core/testOnly
org.apache.spark.scheduler.WorkerDecommissionSuite" || ((failed=1))
echo "Resulted in $failed"
((failures=failures+failed))
echo "Current status is failures: $failures out of $i runs"
done
```
Closes #28614 from
holdenk/SPARK-31791-improve-cache-block-migration-test-reliability.
Authored-by: Holden Karau
Signed-off-by: Holden Karau
---
.../spark/scheduler/WorkerDecommissionSuite.scala | 22 +-
1 file changed, 13 insertions(+), 9 deletions(-)
diff --git
a/core/src/test/scala/org/apache/spark/scheduler/WorkerDecommissionSuite.scala
b/core/src/test/scala/org/apache/spark/scheduler/WorkerDecommissionSuite.scala
index 8c6f86a..148d20e 100644
---
a/core/src/test/scala/org/apache/spark/scheduler/WorkerDecommissionSuite.scala
+++
b/core/src/test/scala/org/apache/spark/scheduler/WorkerDecommissionSuite.scala
@@ -22,7 +22,8 @@ import java.util.concurrent.Semaphore
import scala.concurrent.TimeoutException
import scala.concurrent.duration._
-import org.apache.spark.{LocalSparkContext, SparkConf, SparkContext,
SparkException, SparkFunSuite}
+import org.apache.spark.{LocalSparkContext, SparkConf, SparkContext,
SparkException, SparkFunSuite,
+ TestUtils}
import org.apache.spark.internal.config
import org.apache.spark.scheduler.cluster.StandaloneSchedulerBackend
import org.apache.spark.util.{RpcUtils, SerializableBuffer, ThreadUtils}
@@ -48,12 +49,6 @@ class WorkerDecommissionSuite extends SparkFunSuite with
LocalSparkContext {
test("verify a task with all workers decommissioned succeeds") {
val input = sc.parallelize(1 to 10)
-// Do a count to wait for the executors to be registered.
-input.count()
-val sleepyRdd = input.mapPartitions{ x =>
- Thread.sleep(50)
- x
-}
// Listen for the job
val sem = new Semaphore(0)
sc.addSparkListener(new SparkListener {
@@ -61,22 +56,31 @@ class WorkerDecommissionSuite extends SparkFunSuite with
LocalSparkContext {
sem.release()
}
})
+TestUtils.waitUntilExecutorsUp(sc = sc,
+ numExecutors = 2,
+ timeout = 1) // 10s
+val sleepyRdd = input.mapPartitions{ x =>
+ Thread.sleep(5000) // 5s
+ x
+}
// Start the task.
val asyncCount = sleepyRdd.countAsync()
// Wait for the job to have started
sem.acquire(1)
+// Give it time to make it to the worker otherwise we'll block
+Thread.sleep(2000) // 2s
// Decommission all the executors, this should not halt the current task.
// decom.sh message passing is tested manually.
val sched = sc.schedulerBackend.asInstanceOf[StandaloneSchedulerBackend]
val execs = sched.getExecutorIds()
execs.foreach(execId => sched.decommissionExecutor(execId))
-val asyncCountResult = ThreadUtils.awaitResult(asyncCount, 10.seconds)
+val asyncCountResult = ThreadUtils.awaitResult(asyncCount, 20.seconds)
assert(asyncCountResult === 10)
// Try and launch task after decommissioning, this should fail
val postDecommissioned = input.map(x => x)
val postDecomAsyncCount = postDecommissioned.countAsync()
val thrown = intercept[java.util.concurrent.TimeoutException]{
- val result = ThreadUtils.awaitResult(postDecomAsyncCount, 10.seconds)
+ val result = ThreadUtils.awaitResult(postDecomAsyncCount, 20.seconds)
}
assert(postDecomAsyncCount.isCompleted === false,
"After exec decommission new task could not launch")
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands,
[spark] branch branch-3.0 updated: [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception for invalid length input
This is an automated email from the ASF dual-hosted git repository.
yamamuro pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 2183345 [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw
an exception for invalid length input
2183345 is described below
commit 218334523dacd116a03f2340ad89e33abe93e452
Author: Takeshi Yamamuro
AuthorDate: Sat May 23 08:48:29 2020 +0900
[SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception
for invalid length input
### What changes were proposed in this pull request?
This PR intends to add trivial tests to check
https://github.com/apache/spark/pull/27024 has already been fixed in the master.
Closes #27024
### Why are the changes needed?
For test coverage.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added tests.
Closes #28604 from maropu/SPARK-29854.
Authored-by: Takeshi Yamamuro
Signed-off-by: Takeshi Yamamuro
(cherry picked from commit 7ca73f03fbc6e213c30e725bf480709ed036a376)
Signed-off-by: Takeshi Yamamuro
---
.../sql-tests/inputs/ansi/string-functions.sql | 1 +
.../sql-tests/inputs/string-functions.sql | 6 +++-
.../results/{ => ansi}/string-functions.sql.out| 34 +-
.../sql-tests/results/string-functions.sql.out | 18 +++-
4 files changed, 50 insertions(+), 9 deletions(-)
diff --git
a/sql/core/src/test/resources/sql-tests/inputs/ansi/string-functions.sql
b/sql/core/src/test/resources/sql-tests/inputs/ansi/string-functions.sql
new file mode 100644
index 000..dd28e9b
--- /dev/null
+++ b/sql/core/src/test/resources/sql-tests/inputs/ansi/string-functions.sql
@@ -0,0 +1 @@
+--IMPORT string-functions.sql
diff --git a/sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
b/sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
index 8e33471..f5ed203 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/string-functions.sql
@@ -48,4 +48,8 @@ SELECT trim(LEADING 'xyz' FROM 'zzzytestxyz');
SELECT trim(LEADING 'xy' FROM 'xyxXxyLAST WORD');
SELECT trim(TRAILING 'xyz' FROM 'testxxzx');
SELECT trim(TRAILING 'xyz' FROM 'xyztestxxzx');
-SELECT trim(TRAILING 'xy' FROM 'TURNERyxXxy');
\ No newline at end of file
+SELECT trim(TRAILING 'xy' FROM 'TURNERyxXxy');
+
+-- Check lpad/rpad with invalid length parameter
+SELECT lpad('hi', 'invalid_length');
+SELECT rpad('hi', 'invalid_length');
diff --git
a/sql/core/src/test/resources/sql-tests/results/string-functions.sql.out
b/sql/core/src/test/resources/sql-tests/results/ansi/string-functions.sql.out
similarity index 87%
copy from sql/core/src/test/resources/sql-tests/results/string-functions.sql.out
copy to
sql/core/src/test/resources/sql-tests/results/ansi/string-functions.sql.out
index 43c18f5..b507713 100644
--- a/sql/core/src/test/resources/sql-tests/results/string-functions.sql.out
+++
b/sql/core/src/test/resources/sql-tests/results/ansi/string-functions.sql.out
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
--- Number of queries: 34
+-- Number of queries: 36
-- !query
@@ -63,7 +63,7 @@ struct
+struct
-- !query output
ab abcdab NULL
@@ -71,15 +71,16 @@ ab abcdab NULL
-- !query
select left(null, -2), left("abcd", -2), left("abcd", 0), left("abcd", 'a')
-- !query schema
-struct
+struct<>
-- !query output
-NULL NULL
+java.lang.NumberFormatException
+invalid input syntax for type numeric: a
-- !query
select right("abcd", 2), right("abcd", 5), right("abcd", '2'), right("abcd",
null)
-- !query schema
-struct
+struct
-- !query output
cd abcdcd NULL
@@ -87,9 +88,10 @@ cd abcdcd NULL
-- !query
select right(null, -2), right("abcd", -2), right("abcd", 0), right("abcd", 'a')
-- !query schema
-struct
+struct<>
-- !query output
-NULL NULL
+java.lang.NumberFormatException
+invalid input syntax for type numeric: a
-- !query
@@ -274,3 +276,21 @@ SELECT trim(TRAILING 'xy' FROM 'TURNERyxXxy')
struct
-- !query output
TURNERyxX
+
+
+-- !query
+SELECT lpad('hi', 'invalid_length')
+-- !query schema
+struct<>
+-- !query output
+java.lang.NumberFormatException
+invalid input syntax for type numeric: invalid_length
+
+
+-- !query
+SELECT rpad('hi', 'invalid_length')
+-- !query schema
+struct<>
+-- !query output
+java.lang.NumberFormatException
+invalid input syntax for type numeric: invalid_length
diff --git
a/sql/core/src/test/resources/sql-tests/results/string-functions.sql.out
b/sql/core/src/test/resources/sql-tests/results/string-functions.sql.out
index 43c18f5..cb5501d 100644
---
[spark] branch master updated (5a258b0 -> 7ca73f0)
This is an automated email from the ASF dual-hosted git repository.
yamamuro pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 5a258b0 [SPARK-30915][SS] CompactibleFileStreamLog: Avoid reading the
metadata log file when finding the latest batch ID
add 7ca73f0 [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw
an exception for invalid length input
No new revisions were added by this update.
Summary of changes:
.../sql-tests/inputs/ansi/string-functions.sql | 1 +
.../sql-tests/inputs/string-functions.sql | 6 -
.../results/{ => ansi}/string-functions.sql.out| 30 ++
.../sql-tests/results/string-functions.sql.out | 18 -
4 files changed, 48 insertions(+), 7 deletions(-)
create mode 100644
sql/core/src/test/resources/sql-tests/inputs/ansi/string-functions.sql
copy sql/core/src/test/resources/sql-tests/results/{ =>
ansi}/string-functions.sql.out (90%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31612][SQL][DOCS][FOLLOW-UP] Fix a few issues in SQL ref
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 23019aa [SPARK-31612][SQL][DOCS][FOLLOW-UP] Fix a few issues in SQL ref 23019aa is described below commit 23019aa429d8f0db52b1ed5e9e6dc00ea7b94740 Author: Huaxin Gao AuthorDate: Sat May 23 08:43:16 2020 +0900 [SPARK-31612][SQL][DOCS][FOLLOW-UP] Fix a few issues in SQL ref ### What changes were proposed in this pull request? Fix a few issues in SQL Reference ### Why are the changes needed? To make SQL Reference look better ### Does this PR introduce _any_ user-facing change? Yes. before: https://user-images.githubusercontent.com/13592258/82639052-d0f38a80-9bbc-11ea-81a4-22def4ca5cc0.png;> after: https://user-images.githubusercontent.com/13592258/82639063-d5b83e80-9bbc-11ea-84d1-8361e6bee949.png;> before: https://user-images.githubusercontent.com/13592258/82639252-3e9fb680-9bbd-11ea-863c-e6a6c2f83a06.png;> after: https://user-images.githubusercontent.com/13592258/82639265-42cbd400-9bbd-11ea-8df2-fc5c255b84d3.png;> before: https://user-images.githubusercontent.com/13592258/82639072-db158900-9bbc-11ea-9963-731881cda4fd.png;> after https://user-images.githubusercontent.com/13592258/82639082-dfda3d00-9bbc-11ea-9bd2-f922cc91f175.png;> ### How was this patch tested? Manually build and check Closes #28608 from huaxingao/doc_fix. Authored-by: Huaxin Gao Signed-off-by: Takeshi Yamamuro (cherry picked from commit ad9532a09c70bf6acc8b79b4fdbfcd6afadcbc91) Signed-off-by: Takeshi Yamamuro --- docs/_data/menu-sql.yaml | 42 ++-- docs/sql-ref-syntax-aux-conf-mgmt.md | 2 +- docs/sql-ref-syntax-qry.md | 35 +++--- docs/sql-ref-syntax.md | 28 docs/sql-ref.md | 16 +++--- 5 files changed, 67 insertions(+), 56 deletions(-) diff --git a/docs/_data/menu-sql.yaml b/docs/_data/menu-sql.yaml index 57fc493..289a9d3 100644 --- a/docs/_data/menu-sql.yaml +++ b/docs/_data/menu-sql.yaml @@ -76,14 +76,6 @@ - text: SQL Reference url: sql-ref.html subitems: -- text: Data Types - url: sql-ref-datatypes.html -- text: Identifiers - url: sql-ref-identifier.html -- text: Literals - url: sql-ref-literals.html -- text: Null Semantics - url: sql-ref-null-semantics.html - text: ANSI Compliance url: sql-ref-ansi-compliance.html subitems: @@ -93,6 +85,27 @@ url: sql-ref-ansi-compliance.html#type-conversion - text: SQL Keywords url: sql-ref-ansi-compliance.html#sql-keywords +- text: Data Types + url: sql-ref-datatypes.html +- text: Datetime Pattern + url: sql-ref-datetime-pattern.html +- text: Functions + url: sql-ref-functions.html + subitems: + - text: Built-in Functions +url: sql-ref-functions-builtin.html + - text: Scalar UDFs (User-Defined Functions) +url: sql-ref-functions-udf-scalar.html + - text: UDAFs (User-Defined Aggregate Functions) +url: sql-ref-functions-udf-aggregate.html + - text: Integration with Hive UDFs/UDAFs/UDTFs +url: sql-ref-functions-udf-hive.html +- text: Identifiers + url: sql-ref-identifier.html +- text: Literals + url: sql-ref-literals.html +- text: Null Semantics + url: sql-ref-null-semantics.html - text: SQL Syntax url: sql-ref-syntax.html subitems: @@ -247,16 +260,3 @@ url: sql-ref-syntax-aux-resource-mgmt-list-file.html - text: LIST JAR url: sql-ref-syntax-aux-resource-mgmt-list-jar.html -- text: Functions - url: sql-ref-functions.html - subitems: - - text: Built-in Functions -url: sql-ref-functions-builtin.html - - text: Scalar UDFs (User-Defined Functions) -url: sql-ref-functions-udf-scalar.html - - text: UDAFs (User-Defined Aggregate Functions) -url: sql-ref-functions-udf-aggregate.html - - text: Integration with Hive UDFs/UDAFs/UDTFs -url: sql-ref-functions-udf-hive.html -- text: Datetime Pattern - url: sql-ref-datetime-pattern.html diff --git a/docs/sql-ref-syntax-aux-conf-mgmt.md b/docs/sql-ref-syntax-aux-conf-mgmt.md index f5e48ef2..1900fb7 100644 --- a/docs/sql-ref-syntax-aux-conf-mgmt.md +++ b/docs/sql-ref-syntax-aux-conf-mgmt.md @@ -20,4 +20,4 @@ license: | --- * [SET](sql-ref-syntax-aux-conf-mgmt-set.html) - * [UNSET](sql-ref-syntax-aux-conf-mgmt-reset.html) + * [RESET](sql-ref-syntax-aux-conf-mgmt-reset.html) diff --git a/docs/sql-ref-syntax-qry.md
[spark] branch master updated: [SPARK-30915][SS] CompactibleFileStreamLog: Avoid reading the metadata log file when finding the latest batch ID
This is an automated email from the ASF dual-hosted git repository.
zsxwing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5a258b0 [SPARK-30915][SS] CompactibleFileStreamLog: Avoid reading the
metadata log file when finding the latest batch ID
5a258b0 is described below
commit 5a258b0b67ee7c97a90d8b719c7a2171707c9244
Author: Jungtaek Lim (HeartSaVioR)
AuthorDate: Fri May 22 16:46:17 2020 -0700
[SPARK-30915][SS] CompactibleFileStreamLog: Avoid reading the metadata log
file when finding the latest batch ID
### What changes were proposed in this pull request?
This patch adds the new method `getLatestBatchId()` in
CompactibleFileStreamLog in complement of getLatest() which doesn't read the
content of the latest batch metadata log file, and apply to both
FileStreamSource and FileStreamSink to avoid unnecessary latency on reading log
file.
### Why are the changes needed?
Once compacted metadata log file becomes huge, writing outputs for the
compact + 1 batch is also affected due to unnecessarily reading the compacted
metadata log file. This unnecessary latency can be simply avoided.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
New UT. Also manually tested under query which has huge metadata log on
file stream sink:
> before applying the patch
> after applying the patch
Peaks are compact batches - please compare the next batch after compact
batches, especially the area of "light brown".
Closes #27664 from HeartSaVioR/SPARK-30915.
Authored-by: Jungtaek Lim (HeartSaVioR)
Signed-off-by: Shixiong Zhu
---
.../streaming/CompactibleFileStreamLog.scala | 2 +-
.../sql/execution/streaming/FileStreamSink.scala | 2 +-
.../execution/streaming/FileStreamSourceLog.scala | 2 +-
.../sql/execution/streaming/HDFSMetadataLog.scala | 23 --
.../streaming/FileStreamSinkLogSuite.scala | 83 ++
5 files changed, 102 insertions(+), 10 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLog.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLog.scala
index 10bcfe6..e8ae0ea 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLog.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLog.scala
@@ -213,7 +213,7 @@ abstract class CompactibleFileStreamLog[T <: AnyRef :
ClassTag](
* Returns all files except the deleted ones.
*/
def allFiles(): Array[T] = {
-var latestId = getLatest().map(_._1).getOrElse(-1L)
+var latestId = getLatestBatchId().getOrElse(-1L)
// There is a race condition when `FileStreamSink` is deleting old files
and `StreamFileIndex`
// is calling this method. This loop will retry the reading to deal with
the
// race condition.
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
index b679f16..3224547 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
@@ -142,7 +142,7 @@ class FileStreamSink(
}
override def addBatch(batchId: Long, data: DataFrame): Unit = {
-if (batchId <= fileLog.getLatest().map(_._1).getOrElse(-1L)) {
+if (batchId <= fileLog.getLatestBatchId().getOrElse(-1L)) {
logInfo(s"Skipping already committed batch $batchId")
} else {
val committer = FileCommitProtocol.instantiate(
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSourceLog.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSourceLog.scala
index 7b2ea96..c438877 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSourceLog.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSourceLog.scala
@@ -96,7 +96,7 @@ class FileStreamSourceLog(
val searchKeys = removedBatches.map(_._1)
val retrievedBatches = if (searchKeys.nonEmpty) {
logWarning(s"Get batches from removed files, this is unexpected in the
current code path!!!")
- val latestBatchId = getLatest().map(_._1).getOrElse(-1L)
+ val latestBatchId =
[spark] branch branch-3.0 updated: [SPARK-31612][SQL][DOCS][FOLLOW-UP] Fix a few issues in SQL ref
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 23019aa [SPARK-31612][SQL][DOCS][FOLLOW-UP] Fix a few issues in SQL ref 23019aa is described below commit 23019aa429d8f0db52b1ed5e9e6dc00ea7b94740 Author: Huaxin Gao AuthorDate: Sat May 23 08:43:16 2020 +0900 [SPARK-31612][SQL][DOCS][FOLLOW-UP] Fix a few issues in SQL ref ### What changes were proposed in this pull request? Fix a few issues in SQL Reference ### Why are the changes needed? To make SQL Reference look better ### Does this PR introduce _any_ user-facing change? Yes. before: https://user-images.githubusercontent.com/13592258/82639052-d0f38a80-9bbc-11ea-81a4-22def4ca5cc0.png;> after: https://user-images.githubusercontent.com/13592258/82639063-d5b83e80-9bbc-11ea-84d1-8361e6bee949.png;> before: https://user-images.githubusercontent.com/13592258/82639252-3e9fb680-9bbd-11ea-863c-e6a6c2f83a06.png;> after: https://user-images.githubusercontent.com/13592258/82639265-42cbd400-9bbd-11ea-8df2-fc5c255b84d3.png;> before: https://user-images.githubusercontent.com/13592258/82639072-db158900-9bbc-11ea-9963-731881cda4fd.png;> after https://user-images.githubusercontent.com/13592258/82639082-dfda3d00-9bbc-11ea-9bd2-f922cc91f175.png;> ### How was this patch tested? Manually build and check Closes #28608 from huaxingao/doc_fix. Authored-by: Huaxin Gao Signed-off-by: Takeshi Yamamuro (cherry picked from commit ad9532a09c70bf6acc8b79b4fdbfcd6afadcbc91) Signed-off-by: Takeshi Yamamuro --- docs/_data/menu-sql.yaml | 42 ++-- docs/sql-ref-syntax-aux-conf-mgmt.md | 2 +- docs/sql-ref-syntax-qry.md | 35 +++--- docs/sql-ref-syntax.md | 28 docs/sql-ref.md | 16 +++--- 5 files changed, 67 insertions(+), 56 deletions(-) diff --git a/docs/_data/menu-sql.yaml b/docs/_data/menu-sql.yaml index 57fc493..289a9d3 100644 --- a/docs/_data/menu-sql.yaml +++ b/docs/_data/menu-sql.yaml @@ -76,14 +76,6 @@ - text: SQL Reference url: sql-ref.html subitems: -- text: Data Types - url: sql-ref-datatypes.html -- text: Identifiers - url: sql-ref-identifier.html -- text: Literals - url: sql-ref-literals.html -- text: Null Semantics - url: sql-ref-null-semantics.html - text: ANSI Compliance url: sql-ref-ansi-compliance.html subitems: @@ -93,6 +85,27 @@ url: sql-ref-ansi-compliance.html#type-conversion - text: SQL Keywords url: sql-ref-ansi-compliance.html#sql-keywords +- text: Data Types + url: sql-ref-datatypes.html +- text: Datetime Pattern + url: sql-ref-datetime-pattern.html +- text: Functions + url: sql-ref-functions.html + subitems: + - text: Built-in Functions +url: sql-ref-functions-builtin.html + - text: Scalar UDFs (User-Defined Functions) +url: sql-ref-functions-udf-scalar.html + - text: UDAFs (User-Defined Aggregate Functions) +url: sql-ref-functions-udf-aggregate.html + - text: Integration with Hive UDFs/UDAFs/UDTFs +url: sql-ref-functions-udf-hive.html +- text: Identifiers + url: sql-ref-identifier.html +- text: Literals + url: sql-ref-literals.html +- text: Null Semantics + url: sql-ref-null-semantics.html - text: SQL Syntax url: sql-ref-syntax.html subitems: @@ -247,16 +260,3 @@ url: sql-ref-syntax-aux-resource-mgmt-list-file.html - text: LIST JAR url: sql-ref-syntax-aux-resource-mgmt-list-jar.html -- text: Functions - url: sql-ref-functions.html - subitems: - - text: Built-in Functions -url: sql-ref-functions-builtin.html - - text: Scalar UDFs (User-Defined Functions) -url: sql-ref-functions-udf-scalar.html - - text: UDAFs (User-Defined Aggregate Functions) -url: sql-ref-functions-udf-aggregate.html - - text: Integration with Hive UDFs/UDAFs/UDTFs -url: sql-ref-functions-udf-hive.html -- text: Datetime Pattern - url: sql-ref-datetime-pattern.html diff --git a/docs/sql-ref-syntax-aux-conf-mgmt.md b/docs/sql-ref-syntax-aux-conf-mgmt.md index f5e48ef2..1900fb7 100644 --- a/docs/sql-ref-syntax-aux-conf-mgmt.md +++ b/docs/sql-ref-syntax-aux-conf-mgmt.md @@ -20,4 +20,4 @@ license: | --- * [SET](sql-ref-syntax-aux-conf-mgmt-set.html) - * [UNSET](sql-ref-syntax-aux-conf-mgmt-reset.html) + * [RESET](sql-ref-syntax-aux-conf-mgmt-reset.html) diff --git a/docs/sql-ref-syntax-qry.md
[spark] branch master updated (2115c55 -> ad9532a)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2115c55 [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions add ad9532a [SPARK-31612][SQL][DOCS][FOLLOW-UP] Fix a few issues in SQL ref No new revisions were added by this update. Summary of changes: docs/_data/menu-sql.yaml | 42 ++-- docs/sql-ref-syntax-aux-conf-mgmt.md | 2 +- docs/sql-ref-syntax-qry.md | 35 +++--- docs/sql-ref-syntax.md | 28 docs/sql-ref.md | 16 +++--- 5 files changed, 67 insertions(+), 56 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2115c55 -> ad9532a)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2115c55 [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions add ad9532a [SPARK-31612][SQL][DOCS][FOLLOW-UP] Fix a few issues in SQL ref No new revisions were added by this update. Summary of changes: docs/_data/menu-sql.yaml | 42 ++-- docs/sql-ref-syntax-aux-conf-mgmt.md | 2 +- docs/sql-ref-syntax-qry.md | 35 +++--- docs/sql-ref-syntax.md | 28 docs/sql-ref.md | 16 +++--- 5 files changed, 67 insertions(+), 56 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2115c55 [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS
and TIMESTAMP_MICROS functions
2115c55 is described below
commit 2115c55efe5d4bfe60ff4271f15624bd16a01929
Author: TJX2014
AuthorDate: Fri May 22 14:16:30 2020 +
[SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and
TIMESTAMP_MICROS functions
### What changes were proposed in this pull request?
Add and register three new functions: `TIMESTAMP_SECONDS`,
`TIMESTAMP_MILLIS` and `TIMESTAMP_MICROS`
A test is added.
Reference:
[BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions?hl=en#timestamp_seconds)
### Why are the changes needed?
People will have convenient way to get timestamps from seconds,milliseconds
and microseconds.
### Does this PR introduce _any_ user-facing change?
Yes, people will have the following ways to get timestamp:
```scala
sql("select TIMESTAMP_SECONDS(t.a) as timestamp from
values(1230219000),(-1230219000) as t(a)").show(false)
```
```
+-+
|timestamp |
+-+
|2008-12-25 23:30:00|
|1931-01-07 16:30:00|
+-+
```
```scala
sql("select TIMESTAMP_MILLIS(t.a) as timestamp from
values(1230219000123),(-1230219000123) as t(a)").show(false)
```
```
+---+
|timestamp |
+---+
|2008-12-25 23:30:00.123|
|1931-01-07 16:29:59.877|
+---+
```
```scala
sql("select TIMESTAMP_MICROS(t.a) as timestamp from
values(1230219000123123),(-1230219000123123) as t(a)").show(false)
```
```
++
|timestamp |
++
|2008-12-25 23:30:00.123123|
|1931-01-07 16:29:59.876877|
++
```
### How was this patch tested?
Unit test.
Closes #28534 from TJX2014/master-SPARK-31710.
Authored-by: TJX2014
Signed-off-by: Wenchen Fan
---
.../sql/catalyst/analysis/FunctionRegistry.scala | 3 +
.../catalyst/expressions/datetimeExpressions.scala | 77 ++
.../expressions/DateExpressionsSuite.scala | 22 +++
.../sql-functions/sql-expression-schema.md | 5 +-
.../test/resources/sql-tests/inputs/datetime.sql | 10 +++
.../sql-tests/results/ansi/datetime.sql.out| 62 -
.../resources/sql-tests/results/datetime.sql.out | 60 +
7 files changed, 237 insertions(+), 2 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
index 26d7b3e..5e53927 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
@@ -424,6 +424,9 @@ object FunctionRegistry {
expression[MakeInterval]("make_interval"),
expression[DatePart]("date_part"),
expression[Extract]("extract"),
+expression[SecondsToTimestamp]("timestamp_seconds"),
+expression[MillisToTimestamp]("timestamp_millis"),
+expression[MicrosToTimestamp]("timestamp_micros"),
// collection functions
expression[CreateArray]("array"),
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
index ccedcb4..afc57aa 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
@@ -401,6 +401,83 @@ case class DayOfYear(child: Expression) extends
UnaryExpression with ImplicitCas
}
}
+abstract class NumberToTimestampBase extends UnaryExpression
+ with ExpectsInputTypes {
+
+ protected def upScaleFactor: Long
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(IntegralType)
+
+ override def dataType: DataType = TimestampType
+
+ override def nullSafeEval(input: Any): Any = {
+Math.multiplyExact(input.asInstanceOf[Number].longValue(), upScaleFactor)
+ }
+
+ override protected def doGenCode(ctx: CodegenContext, ev: ExprCode):
ExprCode = {
+if (upScaleFactor == 1) {
+ defineCodeGen(ctx, ev, c => c)
+} else {
+ defineCodeGen(ctx, ev, c =>
[spark] branch master updated (892b600 -> d955708)
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 892b600 [SPARK-31790][DOCS] cast(long as timestamp) show different
result between Hive and Spark
add d955708 [SPARK-31756][WEBUI] Add real headless browser support for UI
test
No new revisions were added by this update.
Summary of changes:
.../tags/{DockerTest.java => ChromeUITest.java}| 3 +-
.../apache/spark/ui/ChromeUISeleniumSuite.scala| 29 +++---
.../spark/ui/RealBrowserUISeleniumSuite.scala | 109 +
.../org/apache/spark/ui/UISeleniumSuite.scala | 27 -
dev/run-tests.py | 5 +
pom.xml| 2 +
6 files changed, 132 insertions(+), 43 deletions(-)
copy common/tags/src/test/java/org/apache/spark/tags/{DockerTest.java =>
ChromeUITest.java} (96%)
copy mllib/src/test/scala/org/apache/spark/ml/util/TempDirectory.scala =>
core/src/test/scala/org/apache/spark/ui/ChromeUISeleniumSuite.scala (62%)
create mode 100644
core/src/test/scala/org/apache/spark/ui/RealBrowserUISeleniumSuite.scala
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (892b600 -> d955708)
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 892b600 [SPARK-31790][DOCS] cast(long as timestamp) show different
result between Hive and Spark
add d955708 [SPARK-31756][WEBUI] Add real headless browser support for UI
test
No new revisions were added by this update.
Summary of changes:
.../tags/{DockerTest.java => ChromeUITest.java}| 3 +-
.../apache/spark/ui/ChromeUISeleniumSuite.scala| 29 +++---
.../spark/ui/RealBrowserUISeleniumSuite.scala | 109 +
.../org/apache/spark/ui/UISeleniumSuite.scala | 27 -
dev/run-tests.py | 5 +
pom.xml| 2 +
6 files changed, 132 insertions(+), 43 deletions(-)
copy common/tags/src/test/java/org/apache/spark/tags/{DockerTest.java =>
ChromeUITest.java} (96%)
copy mllib/src/test/scala/org/apache/spark/ml/util/TempDirectory.scala =>
core/src/test/scala/org/apache/spark/ui/ChromeUISeleniumSuite.scala (62%)
create mode 100644
core/src/test/scala/org/apache/spark/ui/RealBrowserUISeleniumSuite.scala
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (892b600 -> d955708)
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 892b600 [SPARK-31790][DOCS] cast(long as timestamp) show different
result between Hive and Spark
add d955708 [SPARK-31756][WEBUI] Add real headless browser support for UI
test
No new revisions were added by this update.
Summary of changes:
.../tags/{DockerTest.java => ChromeUITest.java}| 3 +-
.../apache/spark/ui/ChromeUISeleniumSuite.scala| 29 +++---
.../spark/ui/RealBrowserUISeleniumSuite.scala | 109 +
.../org/apache/spark/ui/UISeleniumSuite.scala | 27 -
dev/run-tests.py | 5 +
pom.xml| 2 +
6 files changed, 132 insertions(+), 43 deletions(-)
copy common/tags/src/test/java/org/apache/spark/tags/{DockerTest.java =>
ChromeUITest.java} (96%)
copy mllib/src/test/scala/org/apache/spark/ml/util/TempDirectory.scala =>
core/src/test/scala/org/apache/spark/ui/ChromeUISeleniumSuite.scala (62%)
create mode 100644
core/src/test/scala/org/apache/spark/ui/RealBrowserUISeleniumSuite.scala
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (892b600 -> d955708)
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 892b600 [SPARK-31790][DOCS] cast(long as timestamp) show different
result between Hive and Spark
add d955708 [SPARK-31756][WEBUI] Add real headless browser support for UI
test
No new revisions were added by this update.
Summary of changes:
.../tags/{DockerTest.java => ChromeUITest.java}| 3 +-
.../apache/spark/ui/ChromeUISeleniumSuite.scala| 29 +++---
.../spark/ui/RealBrowserUISeleniumSuite.scala | 109 +
.../org/apache/spark/ui/UISeleniumSuite.scala | 27 -
dev/run-tests.py | 5 +
pom.xml| 2 +
6 files changed, 132 insertions(+), 43 deletions(-)
copy common/tags/src/test/java/org/apache/spark/tags/{DockerTest.java =>
ChromeUITest.java} (96%)
copy mllib/src/test/scala/org/apache/spark/ml/util/TempDirectory.scala =>
core/src/test/scala/org/apache/spark/ui/ChromeUISeleniumSuite.scala (62%)
create mode 100644
core/src/test/scala/org/apache/spark/ui/RealBrowserUISeleniumSuite.scala
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (892b600 -> d955708)
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 892b600 [SPARK-31790][DOCS] cast(long as timestamp) show different
result between Hive and Spark
add d955708 [SPARK-31756][WEBUI] Add real headless browser support for UI
test
No new revisions were added by this update.
Summary of changes:
.../tags/{DockerTest.java => ChromeUITest.java}| 3 +-
.../apache/spark/ui/ChromeUISeleniumSuite.scala| 29 +++---
.../spark/ui/RealBrowserUISeleniumSuite.scala | 109 +
.../org/apache/spark/ui/UISeleniumSuite.scala | 27 -
dev/run-tests.py | 5 +
pom.xml| 2 +
6 files changed, 132 insertions(+), 43 deletions(-)
copy common/tags/src/test/java/org/apache/spark/tags/{DockerTest.java =>
ChromeUITest.java} (96%)
copy mllib/src/test/scala/org/apache/spark/ml/util/TempDirectory.scala =>
core/src/test/scala/org/apache/spark/ui/ChromeUISeleniumSuite.scala (62%)
create mode 100644
core/src/test/scala/org/apache/spark/ui/RealBrowserUISeleniumSuite.scala
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (ec80e4b -> c7591ee)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from ec80e4b [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call" add c7591ee [SPARK-31790][DOCS] cast(long as timestamp) show different result between Hive and Spark No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (ec80e4b -> c7591ee)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from ec80e4b [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call" add c7591ee [SPARK-31790][DOCS] cast(long as timestamp) show different result between Hive and Spark No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (ec80e4b -> c7591ee)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from ec80e4b [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call" add c7591ee [SPARK-31790][DOCS] cast(long as timestamp) show different result between Hive and Spark No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ce4da29 -> 892b600)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ce4da29 [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string add 892b600 [SPARK-31790][DOCS] cast(long as timestamp) show different result between Hive and Spark No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (ec80e4b -> c7591ee)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from ec80e4b [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call" add c7591ee [SPARK-31790][DOCS] cast(long as timestamp) show different result between Hive and Spark No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ce4da29 -> 892b600)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ce4da29 [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string add 892b600 [SPARK-31790][DOCS] cast(long as timestamp) show different result between Hive and Spark No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (ec80e4b -> c7591ee)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from ec80e4b [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call" add c7591ee [SPARK-31790][DOCS] cast(long as timestamp) show different result between Hive and Spark No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ce4da29 -> 892b600)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ce4da29 [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string add 892b600 [SPARK-31790][DOCS] cast(long as timestamp) show different result between Hive and Spark No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31790][DOCS] cast(long as timestamp) show different result between Hive and Spark
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 892b600 [SPARK-31790][DOCS] cast(long as timestamp) show different result between Hive and Spark 892b600 is described below commit 892b600ce3a579ea4acabf8ff378c19830a7d89c Author: GuoPhilipse AuthorDate: Fri May 22 22:01:38 2020 +0900 [SPARK-31790][DOCS] cast(long as timestamp) show different result between Hive and Spark ### What changes were proposed in this pull request? add docs for sql migration-guide ### Why are the changes needed? let user know more about the cast scenarios in which Hive and Spark generate different results ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? no need to test Closes #28605 from GuoPhilipse/spark-docs. Lead-authored-by: GuoPhilipse Co-authored-by: GuoPhilipse <46367746+guophili...@users.noreply.github.com> Signed-off-by: HyukjinKwon --- docs/sql-migration-guide.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md index 96f2c5d..2272c90 100644 --- a/docs/sql-migration-guide.md +++ b/docs/sql-migration-guide.md @@ -961,3 +961,4 @@ Below are the scenarios in which Hive and Spark generate different results: * `SQRT(n)` If n < 0, Hive returns null, Spark SQL returns NaN. * `ACOS(n)` If n < -1 or n > 1, Hive returns null, Spark SQL returns NaN. * `ASIN(n)` If n < -1 or n > 1, Hive returns null, Spark SQL returns NaN. +* `CAST(n AS TIMESTAMP)` If n is integral numbers, Hive treats n as milliseconds, Spark SQL treats n as seconds. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (245aee9 -> ce4da29)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 245aee9 [SPARK-31757][CORE] Improve HistoryServerDiskManager.updateAccessTime() add ce4da29 [SPARK-31755][SQL] allow missing year/hour when parsing date/timestamp string No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/util/DateFormatter.scala| 2 +- .../catalyst/util/DateTimeFormatterHelper.scala| 77 .../sql/catalyst/csv/UnivocityParserSuite.scala| 14 +-- .../sql/catalyst/util/DateTimeTestUtils.scala | 8 +- .../sql/catalyst/util/DateTimeUtilsSuite.scala | 4 +- .../apache/spark/sql/util/DateFormatterSuite.scala | 46 --- .../spark/sql/util/TimestampFormatterSuite.scala | 139 - .../test/resources/sql-tests/inputs/datetime.sql | 12 +- .../resources/sql-tests/inputs/json-functions.sql | 15 +++ .../sql-tests/results/ansi/datetime.sql.out| 50 +++- .../resources/sql-tests/results/datetime.sql.out | 50 +++- .../sql-tests/results/json-functions.sql.out | 45 ++- 12 files changed, 370 insertions(+), 92 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (83d0967 -> 245aee9)
This is an automated email from the ASF dual-hosted git repository. jiangxb1987 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 83d0967 [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call" add 245aee9 [SPARK-31757][CORE] Improve HistoryServerDiskManager.updateAccessTime() No new revisions were added by this update. Summary of changes: .../spark/deploy/history/HistoryServerDiskManager.scala | 15 +-- 1 file changed, 9 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (83d0967 -> 245aee9)
This is an automated email from the ASF dual-hosted git repository. jiangxb1987 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 83d0967 [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call" add 245aee9 [SPARK-31757][CORE] Improve HistoryServerDiskManager.updateAccessTime() No new revisions were added by this update. Summary of changes: .../spark/deploy/history/HistoryServerDiskManager.scala | 15 +-- 1 file changed, 9 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"
This is an automated email from the ASF dual-hosted git repository.
jiangxb1987 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new ec80e4b [SPARK-31784][CORE][TEST] Fix test
BarrierTaskContextSuite."share messages with allGather() call"
ec80e4b is described below
commit ec80e4b5f80876378765544e0c4c6af1a704
Author: yi.wu
AuthorDate: Thu May 21 23:34:11 2020 -0700
[SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages
with allGather() call"
### What changes were proposed in this pull request?
Change from `messages.toList.iterator` to
`Iterator.single(messages.toList)`.
### Why are the changes needed?
In this test, the expected result of `rdd2.collect().head` should actually
be `List("0", "1", "2", "3")` but is `"0"` now.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Updated test.
Thanks WeichenXu123 reported this problem.
Closes #28596 from Ngone51/fix_allgather_test.
Authored-by: yi.wu
Signed-off-by: Xingbo Jiang
(cherry picked from commit 83d0967dcc6b205a3fd2003e051f49733f63cb30)
Signed-off-by: Xingbo Jiang
---
.../org/apache/spark/scheduler/BarrierTaskContextSuite.scala | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git
a/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
b/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
index b5614b2..6191e41 100644
---
a/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
+++
b/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
@@ -69,12 +69,12 @@ class BarrierTaskContextSuite extends SparkFunSuite with
LocalSparkContext with
// Pass partitionId message in
val message: String = context.partitionId().toString
val messages: Array[String] = context.allGather(message)
- messages.toList.iterator
+ Iterator.single(messages.toList)
}
-// Take a sorted list of all the partitionId messages
-val messages = rdd2.collect().head
-// All the task partitionIds are shared
-for((x, i) <- messages.view.zipWithIndex) assert(x.toString == i.toString)
+val messages = rdd2.collect()
+// All the task partitionIds are shared across all tasks
+assert(messages.length === 4)
+assert(messages.forall(_ == List("0", "1", "2", "3")))
}
test("throw exception if we attempt to synchronize with different blocking
calls") {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (83d0967 -> 245aee9)
This is an automated email from the ASF dual-hosted git repository. jiangxb1987 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 83d0967 [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call" add 245aee9 [SPARK-31757][CORE] Improve HistoryServerDiskManager.updateAccessTime() No new revisions were added by this update. Summary of changes: .../spark/deploy/history/HistoryServerDiskManager.scala | 15 +-- 1 file changed, 9 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"
This is an automated email from the ASF dual-hosted git repository.
jiangxb1987 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new ec80e4b [SPARK-31784][CORE][TEST] Fix test
BarrierTaskContextSuite."share messages with allGather() call"
ec80e4b is described below
commit ec80e4b5f80876378765544e0c4c6af1a704
Author: yi.wu
AuthorDate: Thu May 21 23:34:11 2020 -0700
[SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages
with allGather() call"
### What changes were proposed in this pull request?
Change from `messages.toList.iterator` to
`Iterator.single(messages.toList)`.
### Why are the changes needed?
In this test, the expected result of `rdd2.collect().head` should actually
be `List("0", "1", "2", "3")` but is `"0"` now.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Updated test.
Thanks WeichenXu123 reported this problem.
Closes #28596 from Ngone51/fix_allgather_test.
Authored-by: yi.wu
Signed-off-by: Xingbo Jiang
(cherry picked from commit 83d0967dcc6b205a3fd2003e051f49733f63cb30)
Signed-off-by: Xingbo Jiang
---
.../org/apache/spark/scheduler/BarrierTaskContextSuite.scala | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git
a/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
b/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
index b5614b2..6191e41 100644
---
a/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
+++
b/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
@@ -69,12 +69,12 @@ class BarrierTaskContextSuite extends SparkFunSuite with
LocalSparkContext with
// Pass partitionId message in
val message: String = context.partitionId().toString
val messages: Array[String] = context.allGather(message)
- messages.toList.iterator
+ Iterator.single(messages.toList)
}
-// Take a sorted list of all the partitionId messages
-val messages = rdd2.collect().head
-// All the task partitionIds are shared
-for((x, i) <- messages.view.zipWithIndex) assert(x.toString == i.toString)
+val messages = rdd2.collect()
+// All the task partitionIds are shared across all tasks
+assert(messages.length === 4)
+assert(messages.forall(_ == List("0", "1", "2", "3")))
}
test("throw exception if we attempt to synchronize with different blocking
calls") {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (60118a2 -> 83d0967)
This is an automated email from the ASF dual-hosted git repository. jiangxb1987 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 60118a2 [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers add 83d0967 [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call" No new revisions were added by this update. Summary of changes: .../org/apache/spark/scheduler/BarrierTaskContextSuite.scala | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"
This is an automated email from the ASF dual-hosted git repository.
jiangxb1987 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new ec80e4b [SPARK-31784][CORE][TEST] Fix test
BarrierTaskContextSuite."share messages with allGather() call"
ec80e4b is described below
commit ec80e4b5f80876378765544e0c4c6af1a704
Author: yi.wu
AuthorDate: Thu May 21 23:34:11 2020 -0700
[SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages
with allGather() call"
### What changes were proposed in this pull request?
Change from `messages.toList.iterator` to
`Iterator.single(messages.toList)`.
### Why are the changes needed?
In this test, the expected result of `rdd2.collect().head` should actually
be `List("0", "1", "2", "3")` but is `"0"` now.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Updated test.
Thanks WeichenXu123 reported this problem.
Closes #28596 from Ngone51/fix_allgather_test.
Authored-by: yi.wu
Signed-off-by: Xingbo Jiang
(cherry picked from commit 83d0967dcc6b205a3fd2003e051f49733f63cb30)
Signed-off-by: Xingbo Jiang
---
.../org/apache/spark/scheduler/BarrierTaskContextSuite.scala | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git
a/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
b/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
index b5614b2..6191e41 100644
---
a/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
+++
b/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
@@ -69,12 +69,12 @@ class BarrierTaskContextSuite extends SparkFunSuite with
LocalSparkContext with
// Pass partitionId message in
val message: String = context.partitionId().toString
val messages: Array[String] = context.allGather(message)
- messages.toList.iterator
+ Iterator.single(messages.toList)
}
-// Take a sorted list of all the partitionId messages
-val messages = rdd2.collect().head
-// All the task partitionIds are shared
-for((x, i) <- messages.view.zipWithIndex) assert(x.toString == i.toString)
+val messages = rdd2.collect()
+// All the task partitionIds are shared across all tasks
+assert(messages.length === 4)
+assert(messages.forall(_ == List("0", "1", "2", "3")))
}
test("throw exception if we attempt to synchronize with different blocking
calls") {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (60118a2 -> 83d0967)
This is an automated email from the ASF dual-hosted git repository. jiangxb1987 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 60118a2 [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers add 83d0967 [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call" No new revisions were added by this update. Summary of changes: .../org/apache/spark/scheduler/BarrierTaskContextSuite.scala | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"
This is an automated email from the ASF dual-hosted git repository.
jiangxb1987 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 83d0967 [SPARK-31784][CORE][TEST] Fix test
BarrierTaskContextSuite."share messages with allGather() call"
83d0967 is described below
commit 83d0967dcc6b205a3fd2003e051f49733f63cb30
Author: yi.wu
AuthorDate: Thu May 21 23:34:11 2020 -0700
[SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages
with allGather() call"
### What changes were proposed in this pull request?
Change from `messages.toList.iterator` to
`Iterator.single(messages.toList)`.
### Why are the changes needed?
In this test, the expected result of `rdd2.collect().head` should actually
be `List("0", "1", "2", "3")` but is `"0"` now.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Updated test.
Thanks WeichenXu123 reported this problem.
Closes #28596 from Ngone51/fix_allgather_test.
Authored-by: yi.wu
Signed-off-by: Xingbo Jiang
---
.../org/apache/spark/scheduler/BarrierTaskContextSuite.scala | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git
a/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
b/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
index b5614b2..6191e41 100644
---
a/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
+++
b/core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala
@@ -69,12 +69,12 @@ class BarrierTaskContextSuite extends SparkFunSuite with
LocalSparkContext with
// Pass partitionId message in
val message: String = context.partitionId().toString
val messages: Array[String] = context.allGather(message)
- messages.toList.iterator
+ Iterator.single(messages.toList)
}
-// Take a sorted list of all the partitionId messages
-val messages = rdd2.collect().head
-// All the task partitionIds are shared
-for((x, i) <- messages.view.zipWithIndex) assert(x.toString == i.toString)
+val messages = rdd2.collect()
+// All the task partitionIds are shared across all tasks
+assert(messages.length === 4)
+assert(messages.forall(_ == List("0", "1", "2", "3")))
}
test("throw exception if we attempt to synchronize with different blocking
calls") {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
