Repository: spark
Updated Branches:
refs/heads/master bc537e40a -> 88a23d3de
[SPARK-20991][SQL] BROADCAST_TIMEOUT conf should be a TimeoutConf
## What changes were proposed in this pull request?
The construction of BROADCAST_TIMEOUT conf should take the TimeUnit argument as
a TimeoutConf.
8168 from zsxwing/SPARK-20940.
(cherry picked from commit 24db35826a81960f08e3eb68556b0f51781144e1)
Signed-off-by: Shixiong Zhu <shixi...@databricks.com>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a607a26b
Tree:
8168 from zsxwing/SPARK-20940.
(cherry picked from commit 24db35826a81960f08e3eb68556b0f51781144e1)
Signed-off-by: Shixiong Zhu <shixi...@databricks.com>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cd870c0c
Tree:
8168 from zsxwing/SPARK-20940.
(cherry picked from commit 24db35826a81960f08e3eb68556b0f51781144e1)
Signed-off-by: Shixiong Zhu <shixi...@databricks.com>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/dade85f7
Tree:
org/jira/browse/SPARK-20666) is an example
of killing SparkContext due to `IllegalAccessError`). I think the correct type
of exception in AccumulatorV2 should be `IllegalStateException`.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #18168 fro
org/apache/spark/sql/execution/datasources/DataSource.scala#L402),
it doesn't make things worse.
## How was this patch tested?
The new added test.
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #18149 from zsxwing/SPARK-20894.
Project: http://git-wip-us.apache.org/repos/asf/s
Repository: spark
Updated Branches:
refs/heads/master 4bb6a53eb -> fa757ee1d
http://git-wip-us.apache.org/repos/asf/spark/blob/fa757ee1/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala
[SPARK-20883][SPARK-20376][SS] Refactored StateStore APIs and added conf to
choose implementation
## What changes were proposed in this pull request?
A bunch of changes to the StateStore APIs and implementation.
Current state store API has a bunch of problems that causes too many transient
Repository: spark
Updated Branches:
refs/heads/branch-2.2 3b79e4cda -> f6730a70c
[SPARK-19968][SS] Use a cached instance of `KafkaProducer` instead of creating
one every batch.
## What changes were proposed in this pull request?
In summary, cost of recreating a KafkaProducer for writing
Repository: spark
Updated Branches:
refs/heads/master 1c7db00c7 -> 96a4d1d08
[SPARK-19968][SS] Use a cached instance of `KafkaProducer` instead of creating
one every batch.
## What changes were proposed in this pull request?
In summary, cost of recreating a KafkaProducer for writing every
Repository: spark
Updated Branches:
refs/heads/branch-2.2 dc51be1e7 -> 26640a269
[SPARK-20907][TEST] Use testQuietly for test suites that generate long log
output
## What changes were proposed in this pull request?
Supress console output by using `testQuietly` in test suites
## How was
Repository: spark
Updated Branches:
refs/heads/master ef9fd920c -> c9749068e
[SPARK-20907][TEST] Use testQuietly for test suites that generate long log
output
## What changes were proposed in this pull request?
Supress console output by using `testQuietly` in test suites
## How was this
How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #18126 from zsxwing/SPARK-20843.
(cherry picked from commit 6c1dbd6fc8d49acf7c1c902d2ebf89ed5e788a4e)
Signed-off-by: Shixiong Zhu <shixi...@databricks.com>
Project: http://git-wip-us.apache.org/
How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #18126 from zsxwing/SPARK-20843.
(cherry picked from commit 6c1dbd6fc8d49acf7c1c902d2ebf89ed5e788a4e)
Signed-off-by: Shixiong Zhu <shixi...@databricks.com>
Project: http://git-wip-us.apache.org/
How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #18126 from zsxwing/SPARK-20843.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6c1dbd6f
Tree: http://git-wip-us.apache.org/repos/
Repository: spark
Updated Branches:
refs/heads/master d935e0a9d -> 473d7552a
[SPARK-20014] Optimize mergeSpillsWithFileStream method
## What changes were proposed in this pull request?
When the individual partition size in a spill is small,
mergeSpillsWithTransferTo method does many small
Repository: spark
Updated Branches:
refs/heads/branch-2.2 92837aeb4 -> 2b59ed4f1
[SPARK-20844] Remove experimental from Structured Streaming APIs
Now that Structured Streaming has been out for several Spark release and has
large production use cases, the `Experimental` label is no longer
Repository: spark
Updated Branches:
refs/heads/master 0fd84b05d -> d935e0a9d
[SPARK-20844] Remove experimental from Structured Streaming APIs
Now that Structured Streaming has been out for several Spark release and has
large production use cases, the `Experimental` label is no longer
Repository: spark
Updated Branches:
refs/heads/branch-2.2 f99456b5f -> 92837aeb4
[SPARK-19372][SQL] Fix throwing a Java exception at df.fliter() due to 64KB
bytecode size limit
## What changes were proposed in this pull request?
When an expression for `df.filter()` has many nodes (e.g.
so that people
can run `bin/run-example StructuredKafkaWordCount ...`.
## How was this patch tested?
manually tested it.
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #18101 from zsxwing/add-missing-example-dep.
(cherry picked from commit 98c3852986a2cb5f2d249d6c8ef602be283bd90e)
S
so that people
can run `bin/run-example StructuredKafkaWordCount ...`.
## How was this patch tested?
manually tested it.
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #18101 from zsxwing/add-missing-example-dep.
(cherry picked from commit 98c3852986a2cb5f2d249d6c8ef602be283bd90e)
S
ple
can run `bin/run-example StructuredKafkaWordCount ...`.
## How was this patch tested?
manually tested it.
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #18101 from zsxwing/add-missing-example-dep.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http:
Repository: spark
Updated Branches:
refs/heads/master bbd8d7def -> 9d6661c82
[SPARK-20792][SS] Support same timeout operations in mapGroupsWithState
function in batch queries as in streaming queries
## What changes were proposed in this pull request?
Currently, in the batch queries, timeout
Repository: spark
Updated Branches:
refs/heads/branch-2.2 3aad5982a -> cfd1bf0be
[SPARK-20792][SS] Support same timeout operations in mapGroupsWithState
function in batch queries as in streaming queries
## What changes were proposed in this pull request?
Currently, in the batch queries,
low
`Await.ready`.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17763 from zsxwing/awaitready.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/324a904d
Tree: http://git-wip-us.a
low
`Await.ready`.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17763 from zsxwing/awaitready.
(cherry picked from commit 324a904d8e80089d8865e4c7edaedb92ab2ec1b2)
Signed-off-by: Shixiong Zhu <shixi...@databricks.com>
Project: http://git-wi
ask is finishing but being killed at the same time.
The fix is pretty easy, just flip the "finished" flag when a task is successful.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #18021 from zsxwing/SPARK-20788.
(cherry
ask is finishing but being killed at the same time.
The fix is pretty easy, just flip the "finished" flag when a task is successful.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #18021 from zsxwing/SPARK-20788.
Project: http://git-wip-
Repository: spark
Updated Branches:
refs/heads/master 9150bca47 -> 6f62e9d9b
[SPARK-19372][SQL] Fix throwing a Java exception at df.fliter() due to 64KB
bytecode size limit
## What changes were proposed in this pull request?
When an expression for `df.filter()` has many nodes (e.g. 400),
How was this patch tested?
The new added unit test.
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17821 from zsxwing/SPARK-20529.
(cherry picked from commit 9150bca47e4b8782e20441386d3d225eb5f2f404)
Signed-off-by: Shixiong Zhu <shixi...@databricks.com>
Project: http://git-wi
How was this patch tested?
The new added unit test.
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17821 from zsxwing/SPARK-20529.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9150bca4
Tree: http://git-wip-us.apache.
Repository: spark
Updated Branches:
refs/heads/master d2416925c -> 499ba2cb4
[SPARK-20717][SS] Minor tweaks to the MapGroupsWithState behavior
## What changes were proposed in this pull request?
Timeout and state data are two independent entities and should be settable
independently.
Repository: spark
Updated Branches:
refs/heads/branch-2.2 82ae1f0ac -> a79a120a8
[SPARK-20717][SS] Minor tweaks to the MapGroupsWithState behavior
## What changes were proposed in this pull request?
Timeout and state data are two independent entities and should be settable
independently.
Repository: spark
Updated Branches:
refs/heads/branch-2.2 0bd918f67 -> 82ae1f0ac
[SPARK-20716][SS] StateStore.abort() should not throw exceptions
## What changes were proposed in this pull request?
StateStore.abort() should do a best effort attempt to clean up temporary
resources. It should
Repository: spark
Updated Branches:
refs/heads/master e1aaab1e2 -> 271175e2b
[SPARK-20716][SS] StateStore.abort() should not throw exceptions
## What changes were proposed in this pull request?
StateStore.abort() should do a best effort attempt to clean up temporary
resources. It should not
Repository: spark
Updated Branches:
refs/heads/branch-2.2 7123ec8e1 -> f14246959
[SPARK-20714][SS] Fix match error when watermark is set with timeout = no
timeout / processing timeout
## What changes were proposed in this pull request?
When watermark is set, and timeout conf is NoTimeout or
Repository: spark
Updated Branches:
refs/heads/master 7d6ff3910 -> 0d3a63193
[SPARK-20714][SS] Fix match error when watermark is set with timeout = no
timeout / processing timeout
## What changes were proposed in this pull request?
When watermark is set, and timeout conf is NoTimeout or
ted` to
propagate the original error.
It also fixes an issue that `TaskCompletionListenerException.getMessage`
doesn't include `previousError`.
## How was this patch tested?
New unit tests.
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17942 from zsxwing/SPARK-20702.
Project: h
ter to `TaskContextImpl.markTaskCompleted` to
propagate the original error.
It also fixes an issue that `TaskCompletionListenerException.getMessage`
doesn't include `previousError`.
## How was this patch tested?
New unit tests.
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17942 from zsxwing/SPARK-20702.
(cher
Repository: spark
Updated Branches:
refs/heads/master 3aa4e464a -> 7144b5180
[SPARK-20600][SS] KafkaRelation should be pretty printed in web UI
## What changes were proposed in this pull request?
User-friendly name of `KafkaRelation` in web UI (under Details for Query).
### Before
Repository: spark
Updated Branches:
refs/heads/branch-2.2 dd9e3b2c9 -> 5844151bc
[SPARK-20600][SS] KafkaRelation should be pretty printed in web UI
## What changes were proposed in this pull request?
User-friendly name of `KafkaRelation` in web UI (under Details for Query).
### Before
Repository: spark
Updated Branches:
refs/heads/branch-2.2 d191b962d -> 7600a7ab6
[SPARK-20373][SQL][SS] Batch queries with 'Dataset/DataFrame.withWatermark()`
does not execute
## What changes were proposed in this pull request?
Any Dataset/DataFrame batch query with the operation
Repository: spark
Updated Branches:
refs/heads/master f79aa285c -> c0189abc7
[SPARK-20373][SQL][SS] Batch queries with 'Dataset/DataFrame.withWatermark()`
does not execute
## What changes were proposed in this pull request?
Any Dataset/DataFrame batch query with the operation
PR changes `offsets.topic.num.partitions` from the default value 50 to 1
to make creating `__consumer_offsets` (50 partitions -> 1 partition) much
faster.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17863 from zsxwing/fix-kafka-flaky-te
PR changes `offsets.topic.num.partitions` from the default value 50 to 1
to make creating `__consumer_offsets` (50 partitions -> 1 partition) much
faster.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17863 from zsxwing/fix-kafka-flaky-te
ges `offsets.topic.num.partitions` from the default value 50 to 1
to make creating `__consumer_offsets` (50 partitions -> 1 partition) much
faster.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17863 from zsxwing/fix-kafka-flaky-test.
P
Repository: spark
Updated Branches:
refs/heads/master 527fc5d0c -> 6b9e49d12
[SPARK-19965][SS] DataFrame batch reader may fail to infer partitions when
reading FileStreamSink's output
## The Problem
Right now DataFrame batch reader may fail to infer partitions when reading
FileStreamSink's
Repository: spark
Updated Branches:
refs/heads/branch-2.2 f0e80aa2d -> 36d807906
[SPARK-19965][SS] DataFrame batch reader may fail to infer partitions when
reading FileStreamSink's output
## The Problem
Right now DataFrame batch reader may fail to infer partitions when reading
Repository: spark
Updated Branches:
refs/heads/branch-2.2 38edb9256 -> 6f0d29672
[SPARK-20464][SS] Add a job group and description for streaming queries and fix
cancellation of running jobs using the job group
## What changes were proposed in this pull request?
Job group: adding a job group
Repository: spark
Updated Branches:
refs/heads/master ab30590f4 -> 6fc6cf88d
[SPARK-20464][SS] Add a job group and description for streaming queries and fix
cancellation of running jobs using the job group
## What changes were proposed in this pull request?
Job group: adding a job group is
ess` to enable/disable it. Credit goes to aramesh117
Closes #17024
## How was this patch tested?
The new unit test.
Author: Shixiong Zhu <shixi...@databricks.com>
Author: Aaditya Ramesh <aram...@conviva.com>
Closes #17789 from zsxwing/pr17024.
Project: http://git-wip-us.apache.org/repos
ess` to enable/disable it. Credit goes to aramesh117
Closes #17024
## How was this patch tested?
The new unit test.
Author: Shixiong Zhu <shixi...@databricks.com>
Author: Aaditya Ramesh <aram...@conviva.com>
Closes #17789 from zsxwing/pr17024.
(cherry pic
ect.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17691 from zsxwing/fix-error-message.
(cherry picked from commit 39e303a8b6db642c26dbc26ba92e87680f50e4da)
Signed-off-by: Shixiong Zhu <shixi...@databricks.com>
Project: http://git-wi
ect.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17691 from zsxwing/fix-error-message.
(cherry picked from commit 39e303a8b6db642c26dbc26ba92e87680f50e4da)
Signed-off-by: Shixiong Zhu <shixi...@databricks.com>
Project: http://git-wi
ect.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17691 from zsxwing/fix-error-message.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/39e303a8
Tree: http://git-wip-us.a
t's
not guaranteed that source has been created. This PR just increases the timeout
of awaitTermination to ensure the parsing error is thrown.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17687 from zsxwing/SPARK-20397.
(cherry picked fr
t's
not guaranteed that source has been created. This PR just increases the timeout
of awaitTermination to ensure the parsing error is thrown.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17687 from zsxwing/SPARK-20397.
Project: http:
Executor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
This PR removes `synchronized` and changes `stopping` to AtomicBoolean to
ensure idempotent to fix the dead-lock.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17610
cutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
This PR removes `synchronized` and changes `stopping` to AtomicBoolean to
ensure idempotent to fix the dead-lock.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #1761
ffsets` is
updated. Then writing the commit log may be interrupted by the following
`StopStream`.
This PR just change the order to write the commit log first.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17594 from zsxwing/SPARK-20282.
Project: ht
eases the timeout to 30 seconds.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17597 from zsxwing/SPARK-20285.
(cherry picked from commit f9a50ba2d1bfa3f55199df031e71154611ba51f6)
Signed-off-by: Shixiong Zhu <shixi...@databricks.com>
Pr
eases the timeout to 30 seconds.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17597 from zsxwing/SPARK-20285.
(cherry picked from commit f9a50ba2d1bfa3f55199df031e71154611ba51f6)
Signed-off-by: Shixiong Zhu <shixi...@databricks.com>
Pr
eases the timeout to 30 seconds.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17597 from zsxwing/SPARK-20285.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f9a50ba2
Tree:
Repository: spark
Updated Branches:
refs/heads/branch-2.1 4b977ff04 -> 710b5554e
[SPARK-19721][SS][BRANCH-2.1] Good error message for version mismatch in log
files
## Problem
There are several places where we write out version identifiers in various logs
for structured streaming (usually
Repository: spark
Updated Branches:
refs/heads/master 8e8f89833 -> 2ea214dd0
[SPARK-19721][SS] Good error message for version mismatch in log files
## Problem
There are several places where we write out version identifiers in various logs
for structured streaming (usually `v1`). However, in
Repository: spark
Updated Branches:
refs/heads/branch-2.1 f9833c66a -> 8c4608046
[SPARK-19853][SS] uppercase kafka topics fail when startingOffsets are
SpecificOffsets
When using the KafkaSource with Structured Streaming, consumer assignments are
not what the user expects if startingOffsets
Repository: spark
Updated Branches:
refs/heads/master 9f8ce4825 -> 0a4d06a7c
[SPARK-19853][SS] uppercase kafka topics fail when startingOffsets are
SpecificOffsets
When using the KafkaSource with Structured Streaming, consumer assignments are
not what the user expects if startingOffsets is
Repository: spark
Updated Branches:
refs/heads/master e29a74d5b -> 2f5187bde
[SPARK-19831][CORE] Reuse the existing cleanupThreadExecutor to clean up the
directories of finished applications to avoid the block
Cleaning the application may cost much time at worker, then it will block that
ion has been thrown.
## How was this patch tested?
Current tests that throw exceptions at runtime will finish faster as a result
of this update.
zsxwing
Please review http://spark.apache.org/contributing.html before opening a pull
request.
Author: Tyson Condie <tcon...@gmail.com>
Clos
hen
an exception has been thrown.
## How was this patch tested?
Current tests that throw exceptions at runtime will finish faster as a result
of this update.
zsxwing
Please review http://spark.apache.org/contributing.html before opening a pull
request.
Author: Tyson Condie <tcon...@gmail.com>
Clos
Repository: spark
Updated Branches:
refs/heads/branch-2.1 ffe65b065 -> a59cc369f
[SPARK-19886] Fix reportDataLoss if statement in SS KafkaSource
## What changes were proposed in this pull request?
Fix the `throw new IllegalStateException` if statement part.
## How is this patch tested
Repository: spark
Updated Branches:
refs/heads/master f79371ad8 -> 82138e09b
[SPARK-19886] Fix reportDataLoss if statement in SS KafkaSource
## What changes were proposed in this pull request?
Fix the `throw new IllegalStateException` if statement part.
## How is this patch tested
Repository: spark
Updated Branches:
refs/heads/branch-2.1 2a76e2420 -> ffe65b065
[SPARK-19861][SS] watermark should not be a negative time.
## What changes were proposed in this pull request?
`watermark` should not be negative. This behavior is invalid, check it before
real run.
## How was
Repository: spark
Updated Branches:
refs/heads/master 40da4d181 -> 30b18e693
[SPARK-19861][SS] watermark should not be a negative time.
## What changes were proposed in this pull request?
`watermark` should not be negative. This behavior is invalid, check it before
real run.
## How was
Repository: spark
Updated Branches:
refs/heads/master 3232e54f2 -> 40da4d181
[SPARK-19715][STRUCTURED STREAMING] Option to Strip Paths in FileSource
## What changes were proposed in this pull request?
Today, we compare the whole path when deciding if a file is new in the
FileSource for
Repository: spark
Updated Branches:
refs/heads/master 029e40b41 -> eeb1d6db8
[SPARK-19859][SS][FOLLOW-UP] The new watermark should override the old one.
## What changes were proposed in this pull request?
A follow up to SPARK-19859:
- extract the calculation of `delayMs` and reuse it.
-
Repository: spark
Updated Branches:
refs/heads/branch-2.1 00859e148 -> 0c140c168
[SPARK-19859][SS][FOLLOW-UP] The new watermark should override the old one.
## What changes were proposed in this pull request?
A follow up to SPARK-19859:
- extract the calculation of `delayMs` and reuse it.
-
kage
because they are internal private APIs.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17217 from zsxwing/SPARK-19874.
(cherry picked from commit 029e40b412e332c9f0fff283d604e203066c78c0)
Signed-off-by: Shixiong Zhu <shixi...@databrick
kage
because they are internal private APIs.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17217 from zsxwing/SPARK-19874.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/029e40
Repository: spark
Updated Branches:
refs/heads/branch-2.1 f6c1ad2eb -> 3457c3229
Revert "[SPARK-19413][SS] MapGroupsWithState for arbitrary stateful operations
for branch-2.1"
This reverts commit 502c927b8c8a99ef2adf4e6e1d7a6d9232d45ef5.
Project:
Repository: spark
Updated Branches:
refs/heads/master 1bf901238 -> 6570cfd7a
[SPARK-19540][SQL] Add ability to clone SparkSession wherein cloned session has
an identical copy of the SessionState
Forking a newSession() from SparkSession currently makes a new SparkSession
that does not retain
all and it makes
ReplSuite unstable.
This PR adds `SparkContext.getActive` to allow `Signaling.cancelOnInterrupt` to
get the active `SparkContext` to avoid the leak.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #16825 from zsxwing/SPARK-19481.
Proj
umn which has a watermark, it may be unexpected.
## How was this patch tested?
The new test.
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17199 from zsxwing/SPARK-19859.
(cherry picked from commit d8830c5039d9c7c5ef03631904c32873ab558e22)
Signed-off-by: Shixiong Zhu <shixi...@da
ich has a watermark, it may be unexpected.
## How was this patch tested?
The new test.
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17199 from zsxwing/SPARK-19859.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spa
est.
Author: Shixiong Zhu <shixi...@databricks.com>
Closes #17183 from zsxwing/SPARK-19841.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ca849ac4
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ca849ac4
Diff: http:
Repository: spark
Updated Branches:
refs/heads/branch-2.0 73801880f -> c7e7b042d
[SPARK-19822][TEST] CheckpointSuite.testCheckpointedOperation: should not
filter checkpointFilesOfLatestTime with the PATH string.
## What changes were proposed in this pull request?
Repository: spark
Updated Branches:
refs/heads/branch-2.1 664c9795c -> ca7a7e8a8
[SPARK-19822][TEST] CheckpointSuite.testCheckpointedOperation: should not
filter checkpointFilesOfLatestTime with the PATH string.
## What changes were proposed in this pull request?
Repository: spark
Updated Branches:
refs/heads/master 224e0e785 -> 207067ead
[SPARK-19822][TEST] CheckpointSuite.testCheckpointedOperation: should not
filter checkpointFilesOfLatestTime with the PATH string.
## What changes were proposed in this pull request?
ks.com>
Closes #17044 from zsxwing/SPARK-19718.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a6a7a95e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a6a7a95e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff
Repository: spark
Updated Branches:
refs/heads/branch-2.1 accbed7c2 -> da04d45c2
[SPARK-19774] StreamExecution should call stop() on sources when a stream fails
## What changes were proposed in this pull request?
We call stop() on a Structured Streaming Source only when the stream is
Repository: spark
Updated Branches:
refs/heads/master 37a1c0e46 -> 9314c0837
[SPARK-19774] StreamExecution should call stop() on sources when a stream fails
## What changes were proposed in this pull request?
We call stop() on a Structured Streaming Source only when the stream is
shutdown
Repository: spark
Updated Branches:
refs/heads/branch-2.1 3a7591ad5 -> 1237aaea2
[SPARK-19779][SS] Delete needless tmp file after restart structured streaming
job
## What changes were proposed in this pull request?
[SPARK-19779](https://issues.apache.org/jira/browse/SPARK-19779)
The PR
Repository: spark
Updated Branches:
refs/heads/branch-2.0 491b47a16 -> 73801880f
[SPARK-19779][SS] Delete needless tmp file after restart structured streaming
job
## What changes were proposed in this pull request?
[SPARK-19779](https://issues.apache.org/jira/browse/SPARK-19779)
The PR
Repository: spark
Updated Branches:
refs/heads/master f37bb1430 -> e24f21b5f
[SPARK-19779][SS] Delete needless tmp file after restart structured streaming
job
## What changes were proposed in this pull request?
[SPARK-19779](https://issues.apache.org/jira/browse/SPARK-19779)
The PR
Repository: spark
Updated Branches:
refs/heads/master 89cd3845b -> 4913c92c2
[SPARK-19633][SS] FileSource read from FileSink
## What changes were proposed in this pull request?
Right now file source always uses `InMemoryFileIndex` to scan files from a
given path.
But when reading the
Repository: spark
Updated Branches:
refs/heads/branch-2.0 a6af60f25 -> dcfb05c86
[SPARK-19677][SS] Committing a delta file atop an existing one should not fail
on HDFS
## What changes were proposed in this pull request?
HDFSBackedStateStoreProvider fails to rename files on HDFS but not on
Repository: spark
Updated Branches:
refs/heads/branch-2.1 4b4c3bf3f -> 947c0cd90
[SPARK-19677][SS] Committing a delta file atop an existing one should not fail
on HDFS
## What changes were proposed in this pull request?
HDFSBackedStateStoreProvider fails to rename files on HDFS but not on
Repository: spark
Updated Branches:
refs/heads/master 7c7fc30b4 -> 9734a928a
[SPARK-19677][SS] Committing a delta file atop an existing one should not fail
on HDFS
## What changes were proposed in this pull request?
HDFSBackedStateStoreProvider fails to rename files on HDFS but not on the
Repository: spark
Updated Branches:
refs/heads/master 16d8472f7 -> 735303835
[SPARK-19749][SS] Name socket source with a meaningful name
## What changes were proposed in this pull request?
Name socket source with a meaningful name
## How was this patch tested?
Jenkins
Author: uncleGen
Repository: spark
Updated Branches:
refs/heads/branch-2.1 20a432951 -> 04fbb9e09
[SPARK-19594][STRUCTURED STREAMING] StreamingQueryListener fails to handle
QueryTerminatedEvent if more then one listeners exists
## What changes were proposed in this pull request?
currently if multiple
201 - 300 of 751 matches
Mail list logo
