Closes #4149 from tdas/SPARK-5147 and squashes the following commits:
730798b [Tathagata Das] Added comments.
c4cf067 [Tathagata Das] Minor fixes
2579b27 [Tathagata Das] Refactored the fix to make sure that the cleanup
respects the remember duration of all the receiver streams
2736fd1 [jerryshao
Repository: spark
Updated Branches:
refs/heads/branch-1.2 cab410c52 -> 5d07488ad
[HOTFIX] Fixed compilation error due to missing SparkContext._ implicit
conversions.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5d07488
Repository: spark
Updated Branches:
refs/heads/master 820ce0359 -> 3c3fa632e
[SPARK-5233][Streaming] Fix error replaying of WAL introduced bug
Because of lacking of `BlockAllocationEvent` in WAL recovery, the dangled event
will mix into the new batch, which will lead to the wrong result. Deta
Repository: spark
Updated Branches:
refs/heads/branch-1.2 5d07488ad -> 5aaf0e0ff
[SPARK-5233][Streaming] Fix error replaying of WAL introduced bug
Because of lacking of `BlockAllocationEvent` in WAL recovery, the dangled event
will mix into the new batch, which will lead to the wrong result.
Repository: spark
Updated Branches:
refs/heads/master 3c3fa632e -> e0f7fb7f9
[SPARK-5315][Streaming] Fix reduceByWindow Java API not work bug
`reduceByWindow` for Java API is actually not Java compatible, change to make
it Java compatible.
Current solution is to deprecate the old one and add
Repository: spark
Updated Branches:
refs/heads/master 683e93824 -> e908322cd
[SPARK-4631][streaming][FIX] Wait for a receiver to start before publishing
test data.
This fixes two sources of non-deterministic failures in this test:
- wait for a receiver to be up before pushing data through MQ
f github.com:apache/spark into kafka
adeeb38 [Davies Liu] Merge pull request #3 from tdas/kafka-python-api
aea8953 [Tathagata Das] Kafka-assembly for Python API
eea16a7 [Davies Liu] refactor
f6ce899 [Davies Liu] add example and fix bugs
98c8d17 [Davies Liu] fix python style
5697a01 [Davies Liu] bypass
rom tdas/SPARK-5153 and squashes the following commits:
dc42762 [Tathagata Das] Increased timeout to deal with delays in overloaded
Jenkins.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/681f9df4
Tree: http://git-wip-us.apache.
rom tdas/SPARK-5153 and squashes the following commits:
dc42762 [Tathagata Das] Increased timeout to deal with delays in overloaded
Jenkins.
(cherry picked from commit 681f9df47ff40f7b0d9175d835e9758d33a13a06)
Signed-off-by: Tathagata Das
Project: http://git-wip-us.apache.org/repos/asf/spark/r
rom tdas/SPARK-5153 and squashes the following commits:
dc42762 [Tathagata Das] Increased timeout to deal with delays in overloaded
Jenkins.
(cherry picked from commit 681f9df47ff40f7b0d9175d835e9758d33a13a06)
Signed-off-by: Tathagata Das
Project: http://git-wip-us.apache.org/repos/asf/spark/r
Repository: spark
Updated Branches:
refs/heads/master 681f9df47 -> 1e8b5394b
[STREAMING] SPARK-4986 Wait for receivers to deregister and receiver job to
terminate
A slow receiver might not have enough time to shutdown cleanly even when
graceful shutdown is used. This PR extends graceful wait
Repository: spark
Updated Branches:
refs/heads/branch-1.3 d644bd96a -> 092d4ba57
[STREAMING] SPARK-4986 Wait for receivers to deregister and receiver job to
terminate
A slow receiver might not have enough time to shutdown cleanly even when
graceful shutdown is used. This PR extends graceful
Repository: spark
Updated Branches:
refs/heads/branch-1.2 36c299430 -> 62c758753
[STREAMING] SPARK-4986 Wait for receivers to deregister and receiver job to
terminate
A slow receiver might not have enough time to shutdown cleanly even when
graceful shutdown is used. This PR extends graceful
ark Streaming
- new unit tests in Scala and Python
This required adding an optional Hadoop configuration param to `fileStream` and
`FileInputStream`, but was otherwise straightforward.
tdas davies
Author: freeman
Closes #3803 from freeman-lab/streaming-binary-records and squashes the
follow
new PySpark Streaming
- new unit tests in Scala and Python
This required adding an optional Hadoop configuration param to `fileStream` and
`FileInputStream`, but was otherwise straightforward.
tdas davies
Author: freeman
Closes #3803 from freeman-lab/streaming-binary-records and squashes
Repository: spark
Updated Branches:
refs/heads/master 6aed719e5 -> 4cf4cba08
[SPARK-5379][Streaming] Add awaitTerminationOrTimeout
Added `awaitTerminationOrTimeout` to return if the waiting time elapsed:
* `true` if it's stopped.
* `false` if the waiting time elapsed before returning from the
Repository: spark
Updated Branches:
refs/heads/branch-1.3 3b7acd22a -> 4d3dbfda3
[SPARK-5379][Streaming] Add awaitTerminationOrTimeout
Added `awaitTerminationOrTimeout` to return if the waiting time elapsed:
* `true` if it's stopped.
* `false` if the waiting time elapsed before returning from
Repository: spark
Updated Branches:
refs/heads/master 4cf4cba08 -> a74cbbf12
[Minor] Fix incorrect warning log
The warning log looks incorrect. Just fix it.
Author: Liang-Chi Hsieh
Closes #4360 from viirya/fixing_typo and squashes the following commits:
48fbe4f [Liang-Chi Hsieh] Fix incorr
Repository: spark
Updated Branches:
refs/heads/branch-1.3 4d3dbfda3 -> 316a4bb54
[Minor] Fix incorrect warning log
The warning log looks incorrect. Just fix it.
Author: Liang-Chi Hsieh
Closes #4360 from viirya/fixing_typo and squashes the following commits:
48fbe4f [Liang-Chi Hsieh] Fix in
Repository: spark
Updated Branches:
refs/heads/branch-1.2 379976320 -> f318af0fd
[Minor] Fix incorrect warning log
The warning log looks incorrect. Just fix it.
Author: Liang-Chi Hsieh
Closes #4360 from viirya/fixing_typo and squashes the following commits:
48fbe4f [Liang-Chi Hsieh] Fix in
a
adf99a6 [cody koeninger] [SPARK-4964] fix serialization issues for checkpointing
1d50749 [cody koeninger] [SPARK-4964] code cleanup per tdas
8bfd6c0 [cody koeninger] [SPARK-4964] configure rate limiting via
spark.streaming.receiver.maxRate
e09045b [cody koeninger] [SPARK-4964] add foreachPa
a
adf99a6 [cody koeninger] [SPARK-4964] fix serialization issues for checkpointing
1d50749 [cody koeninger] [SPARK-4964] code cleanup per tdas
8bfd6c0 [cody koeninger] [SPARK-4964] configure rate limiting via
spark.streaming.receiver.maxRate
e09045b [cody koeninger] [SPARK-4964] add foreachPa
Repository: spark
Updated Branches:
refs/heads/master b0c002195 -> f0500f9fa
[SPARK-4707][STREAMING] Reliable Kafka Receiver can lose data if the blo...
...ck generator fails to store data.
The Reliable Kafka Receiver commits offsets only when events are actually
stored, which ensures that o
Repository: spark
Updated Branches:
refs/heads/branch-1.3 a119cae48 -> 14c9f32d8
[SPARK-4707][STREAMING] Reliable Kafka Receiver can lose data if the blo...
...ck generator fails to store data.
The Reliable Kafka Receiver commits offsets only when events are actually
stored, which ensures th
Repository: spark
Updated Branches:
refs/heads/master b8487713d -> c3f2a8588
SPARK-2932 [STREAMING] Move MasterFailureTest out of "main" source directory
(HT @vanzin) Whatever the reason was for having this test class in `main`, if
there is one, appear to be moot. This may have been a result
Repository: spark
Updated Branches:
refs/heads/master d3a3840e0 -> 8764fe368
SPARK-3744 [STREAMING] FlumeStreamSuite will fail during port contention
Since it looked quite easy, I took the liberty of making a quick PR that just
uses `Utils.startServiceOnPort` to fix this. It works locally for
Repository: spark
Updated Branches:
refs/heads/branch-1.1 b4f690d36 -> 24ee61625
Typo error in KafkaWordCount example
topicpMap to topicMap
Author: Gaspar Munoz
Closes #2614 from gasparms/patch-1 and squashes the following commits:
00aab2c [Gaspar Munoz] Typo error in KafkaWordCount exampl
Repository: spark
Updated Branches:
refs/heads/master 8cc70e7e1 -> b81ee0b46
Typo error in KafkaWordCount example
topicpMap to topicMap
Author: Gaspar Munoz
Closes #2614 from gasparms/patch-1 and squashes the following commits:
00aab2c [Gaspar Munoz] Typo error in KafkaWordCount example
Repository: spark
Updated Branches:
refs/heads/master 7a3f589ef -> 69c67abaa
http://git-wip-us.apache.org/repos/asf/spark/blob/69c67aba/python/pyspark/streaming/tests.py
--
diff --git a/python/pyspark/streaming/tests.py
b/pyth
kagiwa] fied input of socketTextDStream
dd6de81 [Ken Takagiwa] initial commit for socketTextStream
247fd74 [Ken Takagiwa] modified the code base on comment in
https://github.com/tdas/spark/pull/10
4bcb318 [Ken Takagiwa] implementing transform function in Python
38adf95 [Ken Takagiwa] added reduce
out batches.
Author: Tathagata Das
Closes #2773 from tdas/flume-test-fix and squashes the following commits:
93cd7f6 [Tathagata Das] Reimplimented FlumeStreamSuite to be more robust.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/sp
ing HDFS mini cluster.
Author: Hari Shreedharan
Author: Tathagata Das
Closes #2882 from tdas/driver-ha-wal and squashes the following commits:
e4bee20 [Tathagata Das] Removed synchronized, Path.getFileSystem is threadsafe
55514e2 [Tathagata Das] Minor changes based on PR comments.
d29f
s a write ahead log.
Author: Tathagata Das
Closes #2940 from tdas/driver-ha-rbh and squashes the following commits:
78a4aaa [Tathagata Das] Fixed bug causing test failures.
f192f47 [Tathagata Das] Fixed import order.
df5f320 [Tathagata Das] Updated code to use ReceivedBlockStoreResult as the
ret
s a BlockRDD that is backed by HDFS. This BlockRDD can
either read data from the Spark's BlockManager, or read the data from
file-segments in write ahead log in HDFS.
Most of this code has been written by @harishreedharan
Author: Tathagata Das
Author: Hari Shreedharan
Closes #2931 f
hat tests the driver recovery, by
killing and restarting the streaming context, and verifying all the input data
gets processed. This has been implemented but not included in this PR yet. A
sneak peek of that DriverFailureSuite can be found in this PR (on my personal
repo): https://github.com/t
com/tdas/spark/pull/25 I can either include it in this
PR, or submit that as a separate PR after this gets in.
- *WAL cleanup:* Cleaning up the received data write ahead log, by calling
`ReceivedBlockHandler.cleanupOldBlocks`. This is being worked on.
Author: Tathagata Das
Closes #3026 from t
Repository: spark
Updated Branches:
refs/heads/master d6e555244 -> 7c9ec529a
Update JavaCustomReceiver.java
æ°ç»ä¸æ è¶ç
Author: xiao321 <1042460...@qq.com>
Closes #3153 from xiao321/patch-1 and squashes the following commits:
0ed17b5 [xiao321] Update JavaCustomReceiver.java
Project
Repository: spark
Updated Branches:
refs/heads/branch-1.2 47bd8f302 -> 8cefb63c1
Update JavaCustomReceiver.java
æ°ç»ä¸æ è¶ç
Author: xiao321 <1042460...@qq.com>
Closes #3153 from xiao321/patch-1 and squashes the following commits:
0ed17b5 [xiao321] Update JavaCustomReceiver.java
(che
Repository: spark
Updated Branches:
refs/heads/branch-1.1 0a40eac25 -> 4fb26df87
Update JavaCustomReceiver.java
æ°ç»ä¸æ è¶ç
Author: xiao321 <1042460...@qq.com>
Closes #3153 from xiao321/patch-1 and squashes the following commits:
0ed17b5 [xiao321] Update JavaCustomReceiver.java
(che
Repository: spark
Updated Branches:
refs/heads/branch-1.0 76c20cac9 -> 18c8c3833
Update JavaCustomReceiver.java
æ°ç»ä¸æ è¶ç
Author: xiao321 <1042460...@qq.com>
Closes #3153 from xiao321/patch-1 and squashes the following commits:
0ed17b5 [xiao321] Update JavaCustomReceiver.java
(che
Repository: spark
Updated Branches:
refs/heads/master 4af5c7e24 -> 7b41b17f3
[SPARK-4301] StreamingContext should not allow start() to be called after
calling stop()
In Spark 1.0.0+, calling `stop()` on a StreamingContext that has not been
started is a no-op which has no side-effects. This a
Repository: spark
Updated Branches:
refs/heads/branch-1.2 05bffcc02 -> 21b9ac062
[SPARK-4301] StreamingContext should not allow start() to be called after
calling stop()
In Spark 1.0.0+, calling `stop()` on a StreamingContext that has not been
started is a no-op which has no side-effects. Th
Repository: spark
Updated Branches:
refs/heads/branch-1.1 4895f6544 -> 78cd3ab88
[SPARK-4301] StreamingContext should not allow start() to be called after
calling stop()
In Spark 1.0.0+, calling `stop()` on a StreamingContext that has not been
started is a no-op which has no side-effects. Th
Repository: spark
Updated Branches:
refs/heads/branch-1.0 d4aed266d -> 395656c8e
[SPARK-4301] StreamingContext should not allow start() to be called after
calling stop()
In Spark 1.0.0+, calling `stop()` on a StreamingContext that has not been
started is a no-op which has no side-effects. Th
Repository: spark
Updated Branches:
refs/heads/master ed8bf1eac -> 3a02d416c
SPARK-2548 [STREAMING] JavaRecoverableWordCount is missing
Here's my attempt to re-port `RecoverableNetworkWordCount` to Java, following
the example of its Scala and Java siblings. I fixed a few minor doc/formatting
Repository: spark
Updated Branches:
refs/heads/branch-1.2 69dd2997f -> ca3fe8c12
SPARK-2548 [STREAMING] JavaRecoverableWordCount is missing
Here's my attempt to re-port `RecoverableNetworkWordCount` to Java, following
the example of its Scala and Java siblings. I fixed a few minor doc/formatt
Repository: spark
Updated Branches:
refs/heads/branch-1.1 dc38defd2 -> cdcf5467a
SPARK-2548 [STREAMING] JavaRecoverableWordCount is missing
Here's my attempt to re-port `RecoverableNetworkWordCount` to Java, following
the example of its Scala and Java siblings. I fixed a few minor doc/formatt
Repository: spark
Updated Branches:
refs/heads/master 3a02d416c -> 0340c56a9
Update RecoverableNetworkWordCount.scala
Trying this example, I missed the moment when the checkpoint was iniciated
Author: comcmipi
Closes #2735 from comcmipi/patch-1 and squashes the following commits:
b6d8001 [
Repository: spark
Updated Branches:
refs/heads/branch-1.1 cdcf5467a -> 254b13570
Update RecoverableNetworkWordCount.scala
Trying this example, I missed the moment when the checkpoint was iniciated
Author: comcmipi
Closes #2735 from comcmipi/patch-1 and squashes the following commits:
b6d80
Repository: spark
Updated Branches:
refs/heads/branch-1.0 395656c8e -> 66d9070bd
Update RecoverableNetworkWordCount.scala
Trying this example, I missed the moment when the checkpoint was iniciated
Author: comcmipi
Closes #2735 from comcmipi/patch-1 and squashes the following commits:
b6d80
Repository: spark
Updated Branches:
refs/heads/branch-1.1 254b13570 -> 86b1bd031
[SPARK-2548][HOTFIX][Streaming] Removed use of o.a.s.streaming.Durations in
branch 1.1
Author: Tathagata Das
Closes #3188 from tdas/branch-1.1 and squashes the following commits:
f1996d3 [Tathagata
Repository: spark
Updated Branches:
refs/heads/branch-1.2 f0eb0a79c -> 07ba50f7e
[SPARK-3954][Streaming] Optimization to FileInputDStream
about convert files to RDDS there are 3 loops with files sequence in spark
source.
loops files sequence:
1.files.map(...)
2.files.zip(fileRDDs)
3.files-siz
Repository: spark
Updated Branches:
refs/heads/master a1fc059b6 -> ce6ed2abd
[SPARK-3954][Streaming] Optimization to FileInputDStream
about convert files to RDDS there are 3 loops with files sequence in spark
source.
loops files sequence:
1.files.map(...)
2.files.zip(fileRDDs)
3.files-size.fo
Repository: spark
Updated Branches:
refs/heads/branch-1.1 64945f868 -> 3d889dfc1
[SPARK-3954][Streaming] Optimization to FileInputDStream
about convert files to RDDS there are 3 loops with files sequence in spark
source.
loops files sequence:
1.files.map(...)
2.files.zip(fileRDDs)
3.files-siz
of NioBlockTransferService, which required slight modification to unit tests.
Other than that the code is exactly same as in the original PR. Please refer to
discussion in the original PR if you have any thoughts.
Author: Tathagata Das
Closes #3191 from tdas/replication-fix-branch-1.1-backport and squas
Repository: spark
Updated Branches:
refs/heads/master ef29a9a9a -> f8811a569
[SPARK-4295][External]Fix exception in SparkSinkSuite
Handle exception in SparkSinkSuite, please refer to [SPARK-4295]
Author: maji2014
Closes #3177 from maji2014/spark-4295 and squashes the following commits:
312
Repository: spark
Updated Branches:
refs/heads/branch-1.2 e9d009dc3 -> fe8a1cd29
[SPARK-4295][External]Fix exception in SparkSinkSuite
Handle exception in SparkSinkSuite, please refer to [SPARK-4295]
Author: maji2014
Closes #3177 from maji2014/spark-4295 and squashes the following commits:
Repository: spark
Updated Branches:
refs/heads/branch-1.1 b2cb357d7 -> bf867c317
[SPARK-4295][External]Fix exception in SparkSinkSuite
Handle exception in SparkSinkSuite, please refer to [SPARK-4295]
Author: maji2014
Closes #3177 from maji2014/spark-4295 and squashes the following commits:
launched will introduce issue as mentioned in
[SPARK-2383](https://issues.apache.org/jira/browse/SPARK-2383).
So Here we change to offer user to API to explicitly reset offset before create
Kafka stream, while in the meantime keep the same behavior as Kafka 0.8 for
parameter `auto.offset.reset`
hed will introduce issue as mentioned in
[SPARK-2383](https://issues.apache.org/jira/browse/SPARK-2383).
So Here we change to offer user to API to explicitly reset offset before create
Kafka stream, while in the meantime keep the same behavior as Kafka 0.8 for
parameter `auto.offset.reset`.
@tda
Repository: spark
Updated Branches:
refs/heads/master c8850a3d6 -> 6e03de304
[Streaming][Minor]Replace some 'if-else' in Clock
Replace some 'if-else' statement by math.min and math.max in Clock.scala
Author: huangzhaowei
Closes #3088 from SaintBacchus/StreamingClock and squashes the followi
Repository: spark
Updated Branches:
refs/heads/branch-1.2 7710b7156 -> cc1f3a0d6
[Streaming][Minor]Replace some 'if-else' in Clock
Replace some 'if-else' statement by math.min and math.max in Clock.scala
Author: huangzhaowei
Closes #3088 from SaintBacchus/StreamingClock and squashes the fol
Repository: spark
Updated Branches:
refs/heads/master 4b736dbab -> 36ddeb7bf
[SPARK-3660][STREAMING] Initial RDD for updateStateByKey transformation
SPARK-3660 : Initial RDD for updateStateByKey transformation
I have added a sample StatefulNetworkWordCountWithInitial inspired by
StatefulNetw
doc can be seen in
[SPARK-4062](https://issues.apache.org/jira/browse/SPARK-4062).
Author: jerryshao
Author: Tathagata Das
Author: Saisai Shao
Closes #2991 from jerryshao/kafka-refactor and squashes the following commits:
5461f1c [Saisai Shao] Merge pull request #8 from tdas/kafka-refact
can be seen in
[SPARK-4062](https://issues.apache.org/jira/browse/SPARK-4062).
Author: jerryshao
Author: Tathagata Das
Author: Saisai Shao
Closes #2991 from jerryshao/kafka-refactor and squashes the following commits:
5461f1c [Saisai Shao] Merge pull request #8 from tdas/kafka-refactor3
eae4
Repository: spark
Updated Branches:
refs/heads/branch-1.3 db2154d7d -> 1ab423f6e
[SPARK-6766][Streaming] Fix issue about StreamingListenerBatchSubmitted and
StreamingListenerBatchStarted (backport to branch 1.3)
Backport SPARK-6766 #5414 to branch 1.3
Conflicts:
streaming/src/main/
Repository: spark
Updated Branches:
refs/heads/master a76b921a9 -> 6de282e2d
[SPARK-6796][Streaming][WebUI] Add "Active Batches" and "Completed Batches"
lists to StreamingPage
This PR adds two lists, `Active Batches` and `Completed Batches`. Here is the
screenshot:
![batch_list](https://clo
Repository: spark
Updated Branches:
refs/heads/master 03e85b4a1 -> d9e70f331
[HOTFIX][SQL] Fix broken cached test
Added in #5475. Pointed as broken in #5639.
/cc marmbrus
Author: Liang-Chi Hsieh
Closes #5640 from viirya/fix_cached_test and squashes the following commits:
c0cf69a [Liang-Chi
create StreamingContext using the
provided createFunction
TODO: the corresponding Java and Python API has to be added as well.
Author: Tathagata Das
Closes #5428 from tdas/SPARK-6752 and squashes the following commits:
94db63c [Tathagata Das] Fix long line.
524f519 [Tathagata Das] Many changes
Repository: spark
Updated Branches:
refs/heads/master 29576e786 -> 9e4e82b7b
[SPARK-5946] [STREAMING] Add Python API for direct Kafka stream
Currently only added `createDirectStream` API, I'm not sure if `createRDD` is
also needed, since some Java object needs to be wrapped in Python. Please
to add them but want the callback function to be called only
once. This is for internal use only for improvement to Kinesis Receiver that we
are planning to do.
Author: Tathagata Das
Closes #5695 from tdas/SPARK-7138 and squashes the following commits:
a35cf7d [Tathagata Das] Fixed style.
a7a4
/1A2XaOLRFzvIZSi18i_luNw5Rmm9j2j4AigktXxIYxmY/edit?usp=sharing
Things to add.
* Unit tests for WriteAheadLogUtils
Author: Tathagata Das
Closes #5645 from tdas/wal-pluggable and squashes the following commits:
2c431fd [Tathagata Das] Minor fixes.
c2bc7384 [Tathagata Das] More changes based on PR comments
Repository: spark
Updated Branches:
refs/heads/master c0c0ba6d2 -> 1868bd40d
http://git-wip-us.apache.org/repos/asf/spark/blob/1868bd40/streaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockTrackerSuite.scala
--
diff
IRA). This replaces MutableBoolean with AtomicBoolean.
srowen pwendell
Author: Tathagata Das
Closes #5773 from tdas/SPARK-6752 and squashes the following commits:
a0c0ead [Tathagata Das] Fix for hadoop 1.0.4
70ae85b [Tathagata Das] Merge remote-tracking branch 'apache-github/master'
in
Repository: spark
Updated Branches:
refs/heads/master a9c4e2995 -> 3a180c19a
[SPARK-6629] cancelJobGroup() may not work for jobs whose job groups are
inherited from parent threads
When a job is submitted with a job group and that job group is inherited from a
parent thread, there are multipl
Repository: spark
Updated Branches:
refs/heads/master 114bad606 -> 1b7106b86
[SPARK-6862] [STREAMING] [WEBUI] Add BatchPage to display details of a batch
This is an initial commit for SPARK-6862. Once SPARK-6796 is merged, I will add
the links to StreamingPage so that the user can jump to Bat
Repository: spark
Updated Branches:
refs/heads/master beeafcfd6 -> 69a739c7f
[SPARK-7282] [STREAMING] Fix the race conditions in StreamingListenerSuite
Fixed the following flaky test
```Scala
[info] StreamingListenerSuite:
[info] - batch info reporting (782 milliseconds)
[info] - receiver info
Repository: spark
Updated Branches:
refs/heads/master 98e704580 -> ebc25a4dd
[SPARK-7309] [CORE] [STREAMING] Shutdown the thread pools in
ReceivedBlockHandler and DAGScheduler
Shutdown the thread pools in ReceivedBlockHandler and DAGScheduler when
stopping them.
Author: zsxwing
Closes #58
Repository: spark
Updated Branches:
refs/heads/master ebc25a4dd -> b88c275e6
[SPARK-7112][Streaming][WIP] Add a InputInfoTracker to track all the input
streams
Author: jerryshao
Author: Saisai Shao
Closes #5680 from jerryshao/SPARK-7111 and squashes the following commits:
339f854 [Saisai
RK-2808][Streaming][Kafka] add more logging to
python test, see why its timing out in jenkins
2b92d3f [cody koeninger] [SPARK-2808][Streaming][Kafka] wait for leader offsets
in the java test as well
3824ce3 [cody koeninger] [SPARK-2808][Streaming][Kafka] naming / comments per
tdas
61b3464 [cody koen
rom tdas/SPARK-7315 and squashes the following commits:
141afd5 [Tathagata Das] Removed use of FileUtils
b08d4f1 [Tathagata Das] Fix flaky WALBackedBlockRDDSuite
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ecc6eb50
Tree: h
zed WALBackedBlockRDD by skipping block fetch when the block is known
to not exist in Spark
Author: Tathagata Das
Closes #5732 from tdas/SPARK-7139 and squashes the following commits:
575476e [Tathagata Das] Added more tests to get 100% coverage of the
WALBackedBlockRDD
19668ba [Tathagata Das] Merge rem
zed WALBackedBlockRDD by skipping block fetch when the block is known
to not exist in Spark
Author: Tathagata Das
Closes #5732 from tdas/SPARK-7139 and squashes the following commits:
575476e [Tathagata Das] Added more tests to get 100% coverage of the
WALBackedBlockRDD
19668ba [Tathagata Das] Merge rem
edu/jenkins/job/Spark-Master-SBT/2269/
Author: Tathagata Das
Closes #5901 from tdas/ignore-flaky-tests and squashes the following commits:
9cd8667 [Tathagata Das] Ignoring tests.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/com
tps://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/2269/
Author: Tathagata Das
Closes #5901 from tdas/ignore-flaky-tests and squashes the following commits:
9cd8667 [Tathagata Das] Ignoring tests.
(cherry picked from commit 8776fe0b93b6e6d718738bcaf9838a2196e12c8a)
Signed-off-by: Tathagata Das
Project: h
Repository: spark
Updated Branches:
refs/heads/branch-1.4 e8f847a41 -> becdb811d
[SPARK-7113] [STREAMING] Support input information reporting for Direct Kafka
stream
Author: jerryshao
Closes #5879 from jerryshao/SPARK-7113 and squashes the following commits:
b0b506c [jerryshao] Address the
Repository: spark
Updated Branches:
refs/heads/master 8776fe0b9 -> 8436f7e98
[SPARK-7113] [STREAMING] Support input information reporting for Direct Kafka
stream
Author: jerryshao
Closes #5879 from jerryshao/SPARK-7113 and squashes the following commits:
b0b506c [jerryshao] Address the com
Repository: spark
Updated Branches:
refs/heads/master 8436f7e98 -> 4d29867ed
[SPARK-7341] [STREAMING] [TESTS] Fix the flaky test: org.apache.spark.stre...
...aming.InputStreamsSuite.socket input stream
Remove non-deterministic "Thread.sleep" and use deterministic strategies to fix
the flaky
Repository: spark
Updated Branches:
refs/heads/branch-1.4 becdb811d -> 063451068
[SPARK-7341] [STREAMING] [TESTS] Fix the flaky test: org.apache.spark.stre...
...aming.InputStreamsSuite.socket input stream
Remove non-deterministic "Thread.sleep" and use deterministic strategies to fix
the fl
Repository: spark
Updated Branches:
refs/heads/master 47728db7c -> 489700c80
[SPARK-6939] [STREAMING] [WEBUI] Add timeline and histogram graphs for
streaming statistics
This is the initial work of SPARK-6939. Not yet ready for code review. Here are
the screenshots:
![graph1](https://cloud.g
Repository: spark
Updated Branches:
refs/heads/branch-1.4 94ac9eba2 -> 8109c9e10
[SPARK-6939] [STREAMING] [WEBUI] Add timeline and histogram graphs for
streaming statistics
This is the initial work of SPARK-6939. Not yet ready for code review. Here are
the screenshots:
![graph1](https://clo
Repository: spark
Updated Branches:
refs/heads/master 735bc3d04 -> fec7b29f5
[SPARK-7351] [STREAMING] [DOCS] Add spark.streaming.ui.retainedBatches to docs
The default value will be changed to `1000` in #5533. So here I just used
`1000`.
Author: zsxwing
Closes #5899 from zsxwing/SPARK-7351
Repository: spark
Updated Branches:
refs/heads/branch-1.4 c68d0e235 -> 4c95fe5ff
[SPARK-7351] [STREAMING] [DOCS] Add spark.streaming.ui.retainedBatches to docs
The default value will be changed to `1000` in #5533. So here I just used
`1000`.
Author: zsxwing
Closes #5899 from zsxwing/SPARK-
Repository: spark
Updated Branches:
refs/heads/master ce89ff477 -> daa1964b6
[SPARK-8882] [STREAMING] Add a new Receiver scheduling mechanism
The design doc:
https://docs.google.com/document/d/1ZsoRvHjpISPrDmSjsGzuSu8UjwgbtmoCTzmhgTurHJw/edit?usp=sharing
Author: zsxwing
Closes #7276 from z
Repository: spark
Updated Branches:
refs/heads/master 9c5612f4e -> d93ab93d6
[SPARK-9335] [STREAMING] [TESTS] Make sure the test stream is deleted in
KinesisBackedBlockRDDSuite
KinesisBackedBlockRDDSuite should make sure delete the stream.
Author: zsxwing
Closes #7663 from zsxwing/fix-SPAR
Repository: spark
Updated Branches:
refs/heads/master 59b92add7 -> c5ed36953
[STREAMING] [HOTFIX] Ignore ReceiverTrackerSuite flaky test
Author: Tathagata Das
Closes #7738 from tdas/ReceiverTrackerSuite-hotfix and squashes the following
commits:
00f0ee1 [Tathagata Das] ignore flaky t
Repository: spark
Updated Branches:
refs/heads/master 069a4c414 -> 819be46e5
[SPARK-8977] [STREAMING] Defines the RateEstimator interface, and impements the
RateController
Based on #7471.
- [x] add a test that exercises the publish path from driver to receiver
- [ ] remove Serializable from
Repository: spark
Updated Branches:
refs/heads/master 1221849f9 -> 76f2e393a
[SPARK-9335] [TESTS] Enable Kinesis tests only when files in extras/kinesis-asl
are changed
Author: zsxwing
Closes #7711 from zsxwing/SPARK-9335-test and squashes the following commits:
c13ec2f [zsxwing] environs
Repository: spark
Updated Branches:
refs/heads/master 89cda69ec -> 0dbd6963d
[SPARK-9479] [STREAMING] [TESTS] Fix ReceiverTrackerSuite failure for maven
build and other potential test failures in Streaming
See https://issues.apache.org/jira/browse/SPARK-9479 for the failure cause.
The PR inc
at
org.apache.spark.streaming.kinesis.KinesisStreamSuite$$anonfun$3.apply(KinesisStreamSuite.scala:86)
```
This is because attempting to delete a non-existent Kinesis stream throws
uncaught exception. This PR fixes it.
Author: Tathagata Das
Closes #7809 from tdas/kinesis-test-hotfix and squashes
601 - 700 of 1024 matches
Mail list logo