Repository: spark
Updated Branches:
refs/heads/master 0ce6f9b2d -> 1f0f14efe
[SPARK-11462][STREAMING] Add JavaStreamingListener
Currently, StreamingListener is not Java friendly because it exposes some Scala
collections to Java users directly, such as Option, Map.
This PR added a Java
Repository: spark
Updated Branches:
refs/heads/branch-1.6 bdd8a6bd4 -> 9e80db7c7
[SPARK-11359][STREAMING][KINESIS] Checkpoint to DynamoDB even when new data
doesn't come in
Currently, the checkpoints to DynamoDB occur only when new data comes in, as we
update the clock for the
eed to de-aggregate the
records.
tdas Testing is an issue, we need protobuf magic to do the aggregated records.
Maybe we could depend on KPL for tests?
Author: Burak Yavuz <brk...@gmail.com>
Closes #9403 from brkyvz/kinesis-deaggregation.
Project: http://git-wip-us.apache.org/repos/asf/s
ore need to de-aggregate the
records.
tdas Testing is an issue, we need protobuf magic to do the aggregated records.
Maybe we could depend on KPL for tests?
Author: Burak Yavuz <brk...@gmail.com>
Closes #9403 from brkyvz/kinesis-deaggregation.
(cherry picked fr
Repository: spark
Updated Branches:
refs/heads/master 8f888eea1 -> 17f499920
[SPARK-5569][STREAMING] fix ObjectInputStreamWithLoader for supporting load
array classes.
When use Kafka DirectStream API to create checkpoint and restore saved
checkpoint when restart,
ClassNotFound exception
Repository: spark
Updated Branches:
refs/heads/master feb8d6a44 -> 8f888eea1
[SPARK-11270][STREAMING] Add improved equality testing for TopicAndPartition
from the Kafka Streaming API
jerryshao tdas
I know this is kind of minor, and I know you all are busy, but this brings this
cl
Repository: spark
Updated Branches:
refs/heads/branch-1.5 8a6e63c78 -> abb0ca7a9
[SPARK-11270][STREAMING] Add improved equality testing for TopicAndPartition
from the Kafka Streaming API
jerryshao tdas
I know this is kind of minor, and I know you all are busy, but this brings this
cl
ten once the stream
is actually closed.
In case of failure, the data for the last minute (default rolling interval)
will not be properly written. Therefore we need a flag to close the stream
after the write, so that we achieve read after write consistency.
cc tdas zsxwing
Author: Burak Yavuz &
Repository: spark
Updated Branches:
refs/heads/master 80279ac18 -> 63accc796
[SPARK-10891][STREAMING][KINESIS] Add MessageHandler to
KinesisUtils.createStream similar to Direct Kafka
This PR allows users to map a Kinesis `Record` to a generic `T` when creating a
Kinesis stream. This is
sis consumer be able to
read messages generated from Kinesis Producer Library (KPL). The API should be
compatible with old versions.
tdas brkyvz
Author: Xiangrui Meng <m...@databricks.com>
Closes #9153 from mengxr/SPARK-11127.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Com
Repository: spark
Updated Branches:
refs/heads/master 596681794 -> 67582132b
[SPARK-11063] [STREAMING] Change preferredLocations of Receiver's RDD to hosts
rather than hostports
The format of RDD's preferredLocations must be hostname but the format of
Streaming Receiver's scheduling
Repository: spark
Updated Branches:
refs/heads/branch-1.5 648074096 -> 5186ec8ac
[SPARK-11063] [STREAMING] Change preferredLocations of Receiver's RDD to hosts
rather than hostports
The format of RDD's preferredLocations must be hostname but the format of
Streaming Receiver's scheduling
Repository: spark
Updated Branches:
refs/heads/master 3d683a139 -> 369d786f5
[SPARK-10974] [STREAMING] Add progress bar for output operation column and use
red dots for failed batches
Screenshot:
Repository: spark
Updated Branches:
refs/heads/branch-1.5 ba601b1ac -> 3df750029
[SPARK-10955] [STREAMING] Add a warning if dynamic allocation for Streaming
applications
Dynamic allocation can be painful for streaming apps and can lose data. Log a
warning for streaming applications if
Repository: spark
Updated Branches:
refs/heads/master fa3e4d8f5 -> 098412900
[SPARK-10955] [STREAMING] Add a warning if dynamic allocation for Streaming
applications
Dynamic allocation can be painful for streaming apps and can lose data. Log a
warning for streaming applications if dynamic
Repository: spark
Updated Branches:
refs/heads/master 5e035403d -> ffe6831e4
[SPARK-10885] [STREAMING] Display the failed output op in Streaming UI
This PR implements the following features for both `master` and `branch-1.5`.
1. Display the failed output op count in the batch list
2. Display
Repository: spark
Updated Branches:
refs/heads/branch-1.5 6847be6d1 -> 84f510c4f
[SPARK-10885] [STREAMING] Display the failed output op in Streaming UI
This PR implements the following features for both `master` and `branch-1.5`.
1. Display the failed output op count in the batch list
2.
Repository: spark
Updated Branches:
refs/heads/master a609eb20d -> be7c5ff1a
[SPARK-10900] [STREAMING] Add output operation events to StreamingListener
Add output operation events to StreamingListener so as to implement the
following UI features:
1. Progress bar of a batch in the batch
Repository: spark
Updated Branches:
refs/heads/master 5548a2547 -> 44c28abf1
[SPARK-10224] [STREAMING] Fix the issue that blockIntervalTimer won't call
updateCurrentBuffer when stopping
`blockIntervalTimer.stop(interruptTimer = false)` doesn't guarantee calling
`updateCurrentBuffer`. So
Repository: spark
Updated Branches:
refs/heads/branch-1.5 6a616d0d0 -> 4174b94f0
[SPARK-10769] [STREAMING] [TESTS] Fix o.a.s.streaming.CheckpointSuite.maintains
rate controller
Fixed the following failure in
Repository: spark
Updated Branches:
refs/heads/branch-1.5 8a23ef59b -> 6a616d0d0
[SPARK-10224] [STREAMING] Fix the issue that blockIntervalTimer won't call
updateCurrentBuffer when stopping
`blockIntervalTimer.stop(interruptTimer = false)` doesn't guarantee calling
`updateCurrentBuffer`. So
Repository: spark
Updated Branches:
refs/heads/master 44c28abf1 -> 50e463423
[SPARK-10769] [STREAMING] [TESTS] Fix o.a.s.streaming.CheckpointSuite.maintains
rate controller
Fixed the following failure in
Das <tathagata.das1...@gmail.com>
Closes #8791 from tdas/SPARK-10652.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5548a254
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5548a254
Diff: http://git-wip-us.a
ata Das <tathagata.das1...@gmail.com>
Closes #8791 from tdas/SPARK-10652.
(cherry picked from commit 5548a254755bb84edae2768b94ab1816e1b49b91)
Signed-off-by: Tathagata Das <tathagata.das1...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wi
oup and
description for streaming, does not change those properties in the thread that
called streamingContext.start().
Author: Tathagata Das <tathagata.das1...@gmail.com>
Closes #8781 from tdas/SPARK-10649.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.
Repository: spark
Updated Branches:
refs/heads/master e6f8d3686 -> 52b24a602
[SPARK-10492] [STREAMING] [DOCUMENTATION] Update Streaming documentation about
rate limiting and backpressure
Author: Tathagata Das <tathagata.das1...@gmail.com>
Closes #8656 from tdas/SPARK-10492 and
Repository: spark
Updated Branches:
refs/heads/branch-1.5 7fd4674fc -> 63c72b93e
[SPARK-10492] [STREAMING] [DOCUMENTATION] Update Streaming documentation about
rate limiting and backpressure
Author: Tathagata Das <tathagata.das1...@gmail.com>
Closes #8656 from tdas/SPARK-10492 and
Repository: spark
Updated Branches:
refs/heads/master 72f6dbf7b -> 4a5fe0916
[SPARK-10369] [STREAMING] Don't remove ReceiverTrackingInfo when
deregisterReceivering since we may reuse it later
`deregisterReceiver` should not remove `ReceiverTrackingInfo`. Otherwise, it
will throw
Repository: spark
Updated Branches:
refs/heads/branch-1.5 bf5b2f26b -> 33ce274cd
[SPARK-10369] [STREAMING] Don't remove ReceiverTrackingInfo when
deregisterReceivering since we may reuse it later
`deregisterReceiver` should not remove `ReceiverTrackingInfo`. Otherwise, it
will throw
Repository: spark
Updated Branches:
refs/heads/branch-1.5 2239a2036 - 88991dc4f
[SPARK-9786] [STREAMING] [KAFKA] fix backpressure so it works with defaâ¦
â¦ult maxRatePerPartition setting of 0
Author: cody koeninger c...@koeninger.org
Closes #8413 from
Repository: spark
Updated Branches:
refs/heads/branch-1.5 88991dc4f - bb1357f36
[SPARK-10137] [STREAMING] Avoid to restart receivers if scheduleReceivers
returns balanced results
This PR fixes the following cases for `ReceiverSchedulingPolicy`.
1) Assume there are 4 executors: host1, host2,
Repository: spark
Updated Branches:
refs/heads/master 5175ca0c8 - d9c25dec8
[SPARK-9786] [STREAMING] [KAFKA] fix backpressure so it works with defaâ¦
â¦ult maxRatePerPartition setting of 0
Author: cody koeninger c...@koeninger.org
Closes #8413 from koeninger/backpressure-testing-master.
Repository: spark
Updated Branches:
refs/heads/master d9c25dec8 - f023aa2fc
[SPARK-10137] [STREAMING] Avoid to restart receivers if scheduleReceivers
returns balanced results
This PR fixes the following cases for `ReceiverSchedulingPolicy`.
1) Assume there are 4 executors: host1, host2,
...@gmail.com
Closes #8387 from tdas/SPARK-9791 and squashes the following commits:
67f3ee9 [Tathagata Das] Change private class to private[package] class to
prevent them from showing up in the docs
(cherry picked from commit 7478c8b66d6a2b1179f20c38b49e27e37b0caec3)
Signed-off-by: Tathagata Das
Repository: spark
Updated Branches:
refs/heads/branch-1.5 b40059dbd - 36bc50c8d
[SPARK-10168] [STREAMING] Fix the issue that maven publishes wrong artifact jars
This PR removed the `outputFile` configuration from pom.xml and updated
`tests.py` to search jars for both sbt build and maven
...@gmail.com
Closes #8387 from tdas/SPARK-9791 and squashes the following commits:
67f3ee9 [Tathagata Das] Change private class to private[package] class to
prevent them from showing up in the docs
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos
Repository: spark
Updated Branches:
refs/heads/master 053d94fcf - 4e0395ddb
[SPARK-10168] [STREAMING] Fix the issue that maven publishes wrong artifact jars
This PR removed the `outputFile` configuration from pom.xml and updated
`tests.py` to search jars for both sbt build and maven build.
with distributed filesystem
paths in a local unit test. I am going to test it with a distributed file
system manually to verify that this patch works.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #8366 from tdas/SPARK-10142 and squashes the following commits:
3afa666 [Tathagata Das] Added
-10122).
tdas , please help to review.
Author: jerryshao ss...@hortonworks.com
Closes #8347 from jerryshao/SPARK-10122 and squashes the following commits:
4039b16 [jerryshao] Fix getOffsetRanges in transform() bug
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip
Repository: spark
Updated Branches:
refs/heads/master 2f2686a73 - 1f29d502e
[SPARK-9812] [STREAMING] Fix Python 3 compatibility issue in PySpark Streaming
and some docs
This PR includes the following fixes:
1. Use `range` instead of `xrange` in `queue_stream.py` to support Python 3.
2. Fix
Repository: spark
Updated Branches:
refs/heads/master 1f29d502e - affc8a887
[SPARK-10125] [STREAMING] Fix a potential deadlock in JobGenerator.stop
Because `lazy val` uses `this` lock, if JobGenerator.stop and
JobGenerator.doCheckpoint (JobGenerator.shouldCheckpoint has not yet been
Repository: spark
Updated Branches:
refs/heads/branch-1.5 a3ed2c31e - 63922fa4d
[SPARK-10125] [STREAMING] Fix a potential deadlock in JobGenerator.stop
Because `lazy val` uses `this` lock, if JobGenerator.stop and
JobGenerator.doCheckpoint (JobGenerator.shouldCheckpoint has not yet been
Repository: spark
Updated Branches:
refs/heads/branch-1.5 321cb99ca - 16414dae0
[SPARK-9812] [STREAMING] Fix Python 3 compatibility issue in PySpark Streaming
and some docs
This PR includes the following fixes:
1. Use `range` instead of `xrange` in `queue_stream.py` to support Python 3.
2.
}
- Change Java Deprecated annotations to Scala deprecated annotation with more
information.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #8299 from tdas/SPARK-9967.
(cherry picked from commit bc9a0e03235865d2ec33372f6400dec8c770778a)
Signed-off-by: Tathagata Das tathagata.das1
from tdas/SPARK-10128 and squashes the following commits:
f19b1c2 [Tathagata Das] Used correct classloader to deserialize WAL data
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b762f992
Tree: http://git-wip-us.apache.org/repos
#8328 from tdas/SPARK-10128 and squashes the following commits:
f19b1c2 [Tathagata Das] Used correct classloader to deserialize WAL data
(cherry picked from commit b762f9920f7587d3c08493c49dd2fede62110b88)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org
the synchronized to
prevent deadlock.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #8257 from tdas/SPARK-10072.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1aeae05b
Tree: http://git-wip-us.apache.org/repos/asf
the synchronized to
prevent deadlock.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #8257 from tdas/SPARK-10072.
(cherry picked from commit 1aeae05bb20f01ab7ccaa62fe905a63e020074b5)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos
Repository: spark
Updated Branches:
refs/heads/branch-1.5 08c5962a2 - a6f8979c8
[SPARK-10102] [STREAMING] Fix a race condition that startReceiver may happen
before setting trackerState to Started
Test failure:
Repository: spark
Updated Branches:
refs/heads/master 1aeae05bb - 90273eff9
[SPARK-10102] [STREAMING] Fix a race condition that startReceiver may happen
before setting trackerState to Started
Test failure:
Repository: spark
Updated Branches:
refs/heads/master 8bae9015b - bf1d6614d
[SPARK-9574] [STREAMING] Remove unnecessary contents of
spark-streaming-XXX-assembly jars
Removed contents already included in Spark assembly jar from
spark-streaming-XXX-assembly jars.
Author: zsxwing
Das tathagata.das1...@gmail.com
Closes #8289 from tdas/SPARK-10098.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9108eff7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9108eff7
Diff: http://git-wip-us.apache.org
: Tathagata Das tathagata.das1...@gmail.com
Closes #8289 from tdas/SPARK-10098.
(cherry picked from commit 9108eff74a2815986fd067b273c2a344b6315405)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org
Repository: spark
Updated Branches:
refs/heads/branch-1.5 9bd2e6f7c - 2bccd918f
[SPARK-9574] [STREAMING] Remove unnecessary contents of
spark-streaming-XXX-assembly jars
Removed contents already included in Spark assembly jar from
spark-streaming-XXX-assembly jars.
Author: zsxwing
a
parameters that sets a lower bound on the estimated rate, so that the system
always receives some data.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #8199 from tdas/SPARK-9966 and squashes the following commits:
829f793 [Tathagata Das] Fixed unit test and added comments
3a994db [Tathagata
that sets a lower bound on the estimated rate, so that the system
always receives some data.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #8199 from tdas/SPARK-9966 and squashes the following commits:
829f793 [Tathagata Das] Fixed unit test and added comments
3a994db [Tathagata Das
for adding data to the buffer.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #8204 from tdas/SPARK-9968 and squashes the following commits:
8cbcc1b [Tathagata Das] Removed unused val
a73b645 [Tathagata Das] Reduced time spent within synchronized block
(cherry picked from commit
data to the buffer.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #8204 from tdas/SPARK-9968 and squashes the following commits:
8cbcc1b [Tathagata Das] Removed unused val
a73b645 [Tathagata Das] Reduced time spent within synchronized block
Project: http://git-wip-us.apache.org/repos
Repository: spark
Updated Branches:
refs/heads/master 660e6dcff - 8ce60963c
[SPARK-9780] [STREAMING] [KAFKA] prevent NPE if KafkaRDD instantiation â¦
â¦fails
Author: cody koeninger c...@koeninger.org
Closes #8133 from koeninger/SPARK-9780 and squashes the following commits:
406259d [cody
Repository: spark
Updated Branches:
refs/heads/branch-1.5 3298fb69f - 62ab2a4c6
[SPARK-9780] [STREAMING] [KAFKA] prevent NPE if KafkaRDD instantiation â¦
â¦fails
Author: cody koeninger c...@koeninger.org
Closes #8133 from koeninger/SPARK-9780 and squashes the following commits:
406259d
Repository: spark
Updated Branches:
refs/heads/branch-1.5 f9beef998 - c7f009040
[SPARK-9640] [STREAMING] [TEST] Do not run Python Kinesis tests when the
Kinesis assembly JAR has not been generated
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #7961 from tdas/SPARK-9640
Repository: spark
Updated Branches:
refs/heads/master 91e9389f3 - 0f90d6055
[SPARK-9640] [STREAMING] [TEST] Do not run Python Kinesis tests when the
Kinesis assembly JAR has not been generated
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #7961 from tdas/SPARK-9640 and squashes
Repository: spark
Updated Branches:
refs/heads/master 55752d883 - 600031ebe
[SPARK-9727] [STREAMING] [BUILD] Updated streaming kinesis SBT project name to
be more consistent
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #8092 from tdas/SPARK-9727 and squashes the following
Repository: spark
Updated Branches:
refs/heads/branch-1.5 c7f009040 - ebbd3b616
[SPARK-9727] [STREAMING] [BUILD] Updated streaming kinesis SBT project name to
be more consistent
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #8092 from tdas/SPARK-9727 and squashes the following
Repository: spark
Updated Branches:
refs/heads/master dbd778d84 - 5b8bb1b21
[SPARK-9572] [STREAMING] [PYSPARK] Added StreamingContext.getActiveOrCreate()
in Python
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #8080 from tdas/SPARK-9572 and squashes the following commits
Repository: spark
Updated Branches:
refs/heads/branch-1.5 b077f36ea - 71460b889
[SPARK-9572] [STREAMING] [PYSPARK] Added StreamingContext.getActiveOrCreate()
in Python
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #8080 from tdas/SPARK-9572 and squashes the following commits
Repository: spark
Updated Branches:
refs/heads/master c4fd2a242 - 853809e94
[SPARK-5155] [PYSPARK] [STREAMING] Mqtt streaming support in Python
This PR is based on #4229, thanks prabeesh.
Closes #4229
Author: Prabeesh K prabsma...@gmail.com
Author: zsxwing zsxw...@gmail.com
Author: prabs
Repository: spark
Updated Branches:
refs/heads/branch-1.5 51406becc - 8f4014fda
[SPARK-5155] [PYSPARK] [STREAMING] Mqtt streaming support in Python
This PR is based on #4229, thanks prabeesh.
Closes #4229
Author: Prabeesh K prabsma...@gmail.com
Author: zsxwing zsxw...@gmail.com
Author:
Repository: spark
Updated Branches:
refs/heads/branch-1.4 4b5bbc589 - 6dde38026
[SPARK-9801] [STREAMING] Check if file exists before deleting temporary files.
Spark streaming deletes the temp file and backup files without checking if they
exist or not
Author: Hao Zhu viadea...@gmail.com
Repository: spark
Updated Branches:
refs/heads/branch-1.5 8f4014fda - 94692bb14
[SPARK-9801] [STREAMING] Check if file exists before deleting temporary files.
Spark streaming deletes the temp file and backup files without checking if they
exist or not
Author: Hao Zhu viadea...@gmail.com
Repository: spark
Updated Branches:
refs/heads/master 853809e94 - 3c9802d94
[SPARK-9801] [STREAMING] Check if file exists before deleting temporary files.
Spark streaming deletes the temp file and backup files without checking if they
exist or not
Author: Hao Zhu viadea...@gmail.com
Closes
Repository: spark
Updated Branches:
refs/heads/branch-1.3 b104501d3 - a98603f8c
[SPARK-9801] [STREAMING] Check if file exists before deleting temporary files.
Spark streaming deletes the temp file and backup files without checking if they
exist or not
Author: Hao Zhu viadea...@gmail.com
Repository: spark
Updated Branches:
refs/heads/branch-1.5 8a7956283 - 8b00c0690
[SPARK-8978] [STREAMING] Implements the DirectKafkaRateController
Author: Dean Wampler d...@concurrentthought.com
Author: Nilanjan Raychaudhuri nraychaudh...@gmail.com
Author: François Garillot
Repository: spark
Updated Branches:
refs/heads/master 0d7aac99d - a1bbf1bc5
[SPARK-8978] [STREAMING] Implements the DirectKafkaRateController
Author: Dean Wampler d...@concurrentthought.com
Author: Nilanjan Raychaudhuri nraychaudh...@gmail.com
Author: François Garillot franc...@garillot.net
Repository: spark
Updated Branches:
refs/heads/branch-1.5 3997dd3fd - 8ecfb05e3
[DOCS] [STREAMING] make the existing parameter docs for OffsetRange acâ¦
â¦tually visible
Author: cody koeninger c...@koeninger.org
Closes #7995 from koeninger/doc-fixes and squashes the following commits:
Repository: spark
Updated Branches:
refs/heads/master 0a078303d - 1723e3489
[DOCS] [STREAMING] make the existing parameter docs for OffsetRange acâ¦
â¦tually visible
Author: cody koeninger c...@koeninger.org
Closes #7995 from koeninger/doc-fixes and squashes the following commits:
internal API to return the current rate of block generators as
Long instead of Option\[Long\] (was inconsistent at places).
- Updated existing `ReceiverTrackerSuite` to test that custom block generators
get rate updates as well.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #7913 from tdas
API to return the current rate of block generators as
Long instead of Option\[Long\] (was inconsistent at places).
- Updated existing `ReceiverTrackerSuite` to test that custom block generators
get rate updates as well.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #7913 from tdas
Repository: spark
Updated Branches:
refs/heads/branch-1.5 8ecfb05e3 - 980687206
[SPARK-9639] [STREAMING] Fix a potential NPE in Streaming JobScheduler
Because `JobScheduler.stop(false)` may set `eventLoop` to null when
`JobHandler` is running, then it's possible that when `post` is called,
tathagata.das1...@gmail.com
Closes #7825 from tdas/kinesis-receiver and squashes the following commits:
2159be9 [Tathagata Das] Fixed bug
569be83 [Tathagata Das] Fix scala style issue
bf31e22 [Tathagata Das] Added more documentation to make the kinesis test
endpoint more configurable
3ad8361 [Tathagata Das
...@gmail.com
Closes #7825 from tdas/kinesis-receiver and squashes the following commits:
2159be9 [Tathagata Das] Fixed bug
569be83 [Tathagata Das] Fix scala style issue
bf31e22 [Tathagata Das] Added more documentation to make the kinesis test
endpoint more configurable
3ad8361 [Tathagata Das
Repository: spark
Updated Branches:
refs/heads/master 6d8a6e416 - 1bf608b5e
[SPARK-9601] [DOCS] Fix JavaPairDStream signature for stream-stream and
windowed join in streaming guide doc
Author: Namit Katariya katariya.na...@gmail.com
Closes #7935 from namitk/SPARK-9601 and squashes the
Repository: spark
Updated Branches:
refs/heads/branch-1.5 7fa419535 - 6306019ff
[SPARK-9601] [DOCS] Fix JavaPairDStream signature for stream-stream and
windowed join in streaming guide doc
Author: Namit Katariya katariya.na...@gmail.com
Closes #7935 from namitk/SPARK-9601 and squashes the
Repository: spark
Updated Branches:
refs/heads/branch-1.5 d196d3607 - 6e72d24e2
[SPARK-9504] [STREAMING] [TESTS] Fix o.a.s.streaming.StreamingContextSuite.stop
gracefully again
The test failure is here:
Repository: spark
Updated Branches:
refs/heads/master 2b67fdb60 - d34bac0e1
[SPARK-9504] [STREAMING] [TESTS] Fix o.a.s.streaming.StreamingContextSuite.stop
gracefully again
The test failure is here:
Repository: spark
Updated Branches:
refs/heads/master 69f5a7c93 - b41a32718
[SPARK-1855] Local checkpointing
Certain use cases of Spark involve RDDs with long lineages that must be
truncated periodically (e.g. GraphX). The existing way of doing it is through
`rdd.checkpoint()`, which is
Repository: spark
Updated Branches:
refs/heads/master 3c0d2e552 - 060c79aab
[SPARK-9056] [STREAMING] Rename configuration
`spark.streaming.minRememberDuration` to
`spark.streaming.fileStream.minRememberDuration`
Rename configuration `spark.streaming.minRememberDuration` to
] at
org.apache.spark.streaming.kinesis.KinesisStreamSuite$$anonfun$3.apply(KinesisStreamSuite.scala:86)
```
This is because attempting to delete a non-existent Kinesis stream throws
uncaught exception. This PR fixes it.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #7809 from tdas/kinesis-test
Repository: spark
Updated Branches:
refs/heads/master 89cda69ec - 0dbd6963d
[SPARK-9479] [STREAMING] [TESTS] Fix ReceiverTrackerSuite failure for maven
build and other potential test failures in Streaming
See https://issues.apache.org/jira/browse/SPARK-9479 for the failure cause.
The PR
Repository: spark
Updated Branches:
refs/heads/master 3c66ff727 - 9307f5653
[SPARK-9472] [STREAMING] consistent hadoop configuration, streaming only
Author: cody koeninger c...@koeninger.org
Closes #7772 from koeninger/streaming-hadoop-config and squashes the following
commits:
5267284
Repository: spark
Updated Branches:
refs/heads/master 1221849f9 - 76f2e393a
[SPARK-9335] [TESTS] Enable Kinesis tests only when files in extras/kinesis-asl
are changed
Author: zsxwing zsxw...@gmail.com
Closes #7711 from zsxwing/SPARK-9335-test and squashes the following commits:
c13ec2f
Repository: spark
Updated Branches:
refs/heads/master 069a4c414 - 819be46e5
[SPARK-8977] [STREAMING] Defines the RateEstimator interface, and impements the
RateController
Based on #7471.
- [x] add a test that exercises the publish path from driver to receiver
- [ ] remove Serializable from
Repository: spark
Updated Branches:
refs/heads/master 59b92add7 - c5ed36953
[STREAMING] [HOTFIX] Ignore ReceiverTrackerSuite flaky test
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #7738 from tdas/ReceiverTrackerSuite-hotfix and squashes the following
commits:
00f0ee1
Repository: spark
Updated Branches:
refs/heads/master 9c5612f4e - d93ab93d6
[SPARK-9335] [STREAMING] [TESTS] Make sure the test stream is deleted in
KinesisBackedBlockRDDSuite
KinesisBackedBlockRDDSuite should make sure delete the stream.
Author: zsxwing zsxw...@gmail.com
Closes #7663 from
Repository: spark
Updated Branches:
refs/heads/master ce89ff477 - daa1964b6
[SPARK-8882] [STREAMING] Add a new Receiver scheduling mechanism
The design doc:
https://docs.google.com/document/d/1ZsoRvHjpISPrDmSjsGzuSu8UjwgbtmoCTzmhgTurHJw/edit?usp=sharing
Author: zsxwing zsxw...@gmail.com
Repository: spark
Updated Branches:
refs/heads/master fe26584a1 - 798dff7b4
[SPARK-8975] [STREAMING] Adds a mechanism to send a new rate from the driver to
the block generator
First step for [SPARK-7398](https://issues.apache.org/jira/browse/SPARK-7398).
tdas huitseeker
Author: Iulian
Repository: spark
Updated Branches:
refs/heads/branch-1.3 016332535 - 596a4cb8c
Disable flaky test: ReceiverSuite block generator throttling.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/596a4cb8
Tree:
will not run by default. It will
only run when the relevant environment variables are set.
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #7413 from tdas/kinesis-tests and squashes the following commits:
0e16db5 [Tathagata Das] Added more comments regarding testOrIgnore
1ea5ce0 [Tathagata Das
Repository: spark
Updated Branches:
refs/heads/master 074085d67 - ad0954f6d
[SPARK-5681] [STREAMING] Move 'stopReceivers' to the event loop to resolve the
race condition
This is an alternative way to fix `SPARK-5681`. It minimizes the changes.
Closes #4467
Author: zsxwing zsxw...@gmail.com
Repository: spark
Updated Branches:
refs/heads/master fec10f0c6 - 031d7d414
[SPARK-6304] [STREAMING] Fix checkpointing doesn't retain driver port issue.
Author: jerryshao saisai.s...@intel.com
Author: Saisai Shao saisai.s...@intel.com
Closes #5060 from jerryshao/SPARK-6304 and squashes the
401 - 500 of 847 matches
Mail list logo
