svn commit: r25942 - in /dev/spark/2.3.1-SNAPSHOT-2018_03_24_22_02-523fcaf-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

[pwendell]

Author: pwendell
Date: Sun Mar 25 05:17:00 2018
New Revision: 25942

Log:
Apache Spark 2.3.1-SNAPSHOT-2018_03_24_22_02-523fcaf docs


[This commit notification would consist of 1443 parts, 
which exceeds the limit of 50 ones, so it was shortened to the summary.]

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

svn commit: r25941 - in /dev/spark/2.4.0-SNAPSHOT-2018_03_24_20_01-816a549-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s

[pwendell]

Author: pwendell
Date: Sun Mar 25 03:15:52 2018
New Revision: 25941

Log:
Apache Spark 2.4.0-SNAPSHOT-2018_03_24_20_01-816a549 docs


[This commit notification would consist of 1451 parts, 
which exceeds the limit of 50 ones, so it was shortened to the summary.]

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-23788][SS] Fix race in StreamingQuerySuite

[zsxwing]

Repository: spark
Updated Branches:
  refs/heads/branch-2.2 85ab72b59 -> 6b5f9c374


[SPARK-23788][SS] Fix race in StreamingQuerySuite

## What changes were proposed in this pull request?

The serializability test uses the same MemoryStream instance for 3 different 
queries. If any of those queries ask it to commit before the others have run, 
the rest will see empty dataframes. This can fail the test if q3 is affected.

We should use one instance per query instead.

## How was this patch tested?

Existing unit test. If I move q2.processAllAvailable() before starting q3, the 
test always fails without the fix.

Author: Jose Torres 

Closes #20896 from jose-torres/fixrace.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6b5f9c37
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6b5f9c37
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6b5f9c37

Branch: refs/heads/branch-2.2
Commit: 6b5f9c3745a1005519261fc80825a99377906451
Parents: 85ab72b
Author: Jose Torres 
Authored: Sat Mar 24 18:21:01 2018 -0700
Committer: Shixiong Zhu 
Committed: Sat Mar 24 18:22:15 2018 -0700

--
 .../apache/spark/sql/streaming/StreamingQuerySuite.scala  | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/6b5f9c37/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
--
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
index 01c34b1..9e65aa8 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
@@ -533,22 +533,22 @@ class StreamingQuerySuite extends StreamTest with 
BeforeAndAfter with Logging wi
 .start()
 }
 
-val input = MemoryStream[Int]
-val q1 = startQuery(input.toDS, "stream_serializable_test_1")
-val q2 = startQuery(input.toDS.map { i =>
+val input = MemoryStream[Int] :: MemoryStream[Int] :: MemoryStream[Int] :: 
Nil
+val q1 = startQuery(input(0).toDS, "stream_serializable_test_1")
+val q2 = startQuery(input(1).toDS.map { i =>
   // Emulate that `StreamingQuery` get captured with normal usage 
unintentionally.
   // It should not fail the query.
   q1
   i
 }, "stream_serializable_test_2")
-val q3 = startQuery(input.toDS.map { i =>
+val q3 = startQuery(input(2).toDS.map { i =>
   // Emulate that `StreamingQuery` is used in executors. We should fail 
the query with a clear
   // error message.
   q1.explain()
   i
 }, "stream_serializable_test_3")
 try {
-  input.addData(1)
+  input.foreach(_.addData(1))
 
   // q2 should not fail since it doesn't use `q1` in the closure
   q2.processAllAvailable()


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-23788][SS] Fix race in StreamingQuerySuite

[zsxwing]

Repository: spark
Updated Branches:
  refs/heads/branch-2.3 ea44783ad -> 523fcafc5


[SPARK-23788][SS] Fix race in StreamingQuerySuite

## What changes were proposed in this pull request?

The serializability test uses the same MemoryStream instance for 3 different 
queries. If any of those queries ask it to commit before the others have run, 
the rest will see empty dataframes. This can fail the test if q3 is affected.

We should use one instance per query instead.

## How was this patch tested?

Existing unit test. If I move q2.processAllAvailable() before starting q3, the 
test always fails without the fix.

Author: Jose Torres 

Closes #20896 from jose-torres/fixrace.

(cherry picked from commit 816a5496ba4caac438f70400f72bb10bfcc02418)
Signed-off-by: Shixiong Zhu 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/523fcafc
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/523fcafc
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/523fcafc

Branch: refs/heads/branch-2.3
Commit: 523fcafc5c4a79cf3455f3ceab6d886679399495
Parents: ea44783
Author: Jose Torres 
Authored: Sat Mar 24 18:21:01 2018 -0700
Committer: Shixiong Zhu 
Committed: Sat Mar 24 18:21:14 2018 -0700

--
 .../apache/spark/sql/streaming/StreamingQuerySuite.scala  | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/523fcafc/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
--
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
index 76201c6..2b0ab33 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
@@ -532,22 +532,22 @@ class StreamingQuerySuite extends StreamTest with 
BeforeAndAfter with Logging wi
 .start()
 }
 
-val input = MemoryStream[Int]
-val q1 = startQuery(input.toDS, "stream_serializable_test_1")
-val q2 = startQuery(input.toDS.map { i =>
+val input = MemoryStream[Int] :: MemoryStream[Int] :: MemoryStream[Int] :: 
Nil
+val q1 = startQuery(input(0).toDS, "stream_serializable_test_1")
+val q2 = startQuery(input(1).toDS.map { i =>
   // Emulate that `StreamingQuery` get captured with normal usage 
unintentionally.
   // It should not fail the query.
   q1
   i
 }, "stream_serializable_test_2")
-val q3 = startQuery(input.toDS.map { i =>
+val q3 = startQuery(input(2).toDS.map { i =>
   // Emulate that `StreamingQuery` is used in executors. We should fail 
the query with a clear
   // error message.
   q1.explain()
   i
 }, "stream_serializable_test_3")
 try {
-  input.addData(1)
+  input.foreach(_.addData(1))
 
   // q2 should not fail since it doesn't use `q1` in the closure
   q2.processAllAvailable()


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-23788][SS] Fix race in StreamingQuerySuite

[zsxwing]

Repository: spark
Updated Branches:
  refs/heads/master a33655348 -> 816a5496b


[SPARK-23788][SS] Fix race in StreamingQuerySuite

## What changes were proposed in this pull request?

The serializability test uses the same MemoryStream instance for 3 different 
queries. If any of those queries ask it to commit before the others have run, 
the rest will see empty dataframes. This can fail the test if q3 is affected.

We should use one instance per query instead.

## How was this patch tested?

Existing unit test. If I move q2.processAllAvailable() before starting q3, the 
test always fails without the fix.

Author: Jose Torres 

Closes #20896 from jose-torres/fixrace.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/816a5496
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/816a5496
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/816a5496

Branch: refs/heads/master
Commit: 816a5496ba4caac438f70400f72bb10bfcc02418
Parents: a336553
Author: Jose Torres 
Authored: Sat Mar 24 18:21:01 2018 -0700
Committer: Shixiong Zhu 
Committed: Sat Mar 24 18:21:01 2018 -0700

--
 .../apache/spark/sql/streaming/StreamingQuerySuite.scala  | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/816a5496/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
--
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
index ebc9a87..08749b4 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
@@ -550,22 +550,22 @@ class StreamingQuerySuite extends StreamTest with 
BeforeAndAfter with Logging wi
 .start()
 }
 
-val input = MemoryStream[Int]
-val q1 = startQuery(input.toDS, "stream_serializable_test_1")
-val q2 = startQuery(input.toDS.map { i =>
+val input = MemoryStream[Int] :: MemoryStream[Int] :: MemoryStream[Int] :: 
Nil
+val q1 = startQuery(input(0).toDS, "stream_serializable_test_1")
+val q2 = startQuery(input(1).toDS.map { i =>
   // Emulate that `StreamingQuery` get captured with normal usage 
unintentionally.
   // It should not fail the query.
   q1
   i
 }, "stream_serializable_test_2")
-val q3 = startQuery(input.toDS.map { i =>
+val q3 = startQuery(input(2).toDS.map { i =>
   // Emulate that `StreamingQuery` is used in executors. We should fail 
the query with a clear
   // error message.
   q1.explain()
   i
 }, "stream_serializable_test_3")
 try {
-  input.addData(1)
+  input.foreach(_.addData(1))
 
   // q2 should not fail since it doesn't use `q1` in the closure
   q2.processAllAvailable()


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org