t/d/1k0dl270EnK7uExrsCE7jYw7PYx0YC935uBcxn3p0f58/edit
Author: Tathagata Das
Closes #7578 from tdas/kinesis-rdd and squashes the following commits:
543d208 [Tathagata Das] Fixed scala style
5082a30 [Tathagata Das] Fixed scala style
3f40c2d [Tathagata Das] Addressed comments
c4f25d2 [Tathagata Das] Addressed comment
d3d6
Repository: spark
Updated Branches:
refs/heads/master ce89ff477 -> daa1964b6
[SPARK-8882] [STREAMING] Add a new Receiver scheduling mechanism
The design doc:
https://docs.google.com/document/d/1ZsoRvHjpISPrDmSjsGzuSu8UjwgbtmoCTzmhgTurHJw/edit?usp=sharing
Author: zsxwing
Closes #7276 from z
Repository: spark
Updated Branches:
refs/heads/master 9c5612f4e -> d93ab93d6
[SPARK-9335] [STREAMING] [TESTS] Make sure the test stream is deleted in
KinesisBackedBlockRDDSuite
KinesisBackedBlockRDDSuite should make sure delete the stream.
Author: zsxwing
Closes #7663 from zsxwing/fix-SPAR
Repository: spark
Updated Branches:
refs/heads/master 59b92add7 -> c5ed36953
[STREAMING] [HOTFIX] Ignore ReceiverTrackerSuite flaky test
Author: Tathagata Das
Closes #7738 from tdas/ReceiverTrackerSuite-hotfix and squashes the following
commits:
00f0ee1 [Tathagata Das] ignore flaky t
Repository: spark
Updated Branches:
refs/heads/master 069a4c414 -> 819be46e5
[SPARK-8977] [STREAMING] Defines the RateEstimator interface, and impements the
RateController
Based on #7471.
- [x] add a test that exercises the publish path from driver to receiver
- [ ] remove Serializable from
Repository: spark
Updated Branches:
refs/heads/master 1221849f9 -> 76f2e393a
[SPARK-9335] [TESTS] Enable Kinesis tests only when files in extras/kinesis-asl
are changed
Author: zsxwing
Closes #7711 from zsxwing/SPARK-9335-test and squashes the following commits:
c13ec2f [zsxwing] environs
Repository: spark
Updated Branches:
refs/heads/master 89cda69ec -> 0dbd6963d
[SPARK-9479] [STREAMING] [TESTS] Fix ReceiverTrackerSuite failure for maven
build and other potential test failures in Streaming
See https://issues.apache.org/jira/browse/SPARK-9479 for the failure cause.
The PR inc
at
org.apache.spark.streaming.kinesis.KinesisStreamSuite$$anonfun$3.apply(KinesisStreamSuite.scala:86)
```
This is because attempting to delete a non-existent Kinesis stream throws
uncaught exception. This PR fixes it.
Author: Tathagata Das
Closes #7809 from tdas/kinesis-test-hotfix and squashes
Repository: spark
Updated Branches:
refs/heads/master 3c66ff727 -> 9307f5653
[SPARK-9472] [STREAMING] consistent hadoop configuration, streaming only
Author: cody koeninger
Closes #7772 from koeninger/streaming-hadoop-config and squashes the following
commits:
5267284 [cody koeninger] [SPA
Repository: spark
Updated Branches:
refs/heads/master e8bdcdeab -> 0a1d2ca42
[SPARK-8979] Add a PID based rate estimator
Based on #7600
/cc tdas
Author: Iulian Dragos
Author: François Garillot
Closes #7648 from dragos/topic/streaming-bp/pid and squashes the following
commits:
aa5b
Repository: spark
Updated Branches:
refs/heads/master 39ab199a3 -> 3afc1de89
[SPARK-8564] [STREAMING] Add the Python API for Kinesis
This PR adds the Python API for Kinesis, including a Python example and a
simple unit test.
Author: zsxwing
Closes #6955 from zsxwing/kinesis-python and squa
Repository: spark
Updated Branches:
refs/heads/master 3afc1de89 -> d04634701
[SPARK-9504] [STREAMING] [TESTS] Use eventually to fix the flaky test
The previous code uses `ssc.awaitTerminationOrTimeout(500)`. Since nobody will
stop it during `awaitTerminationOrTimeout`, it's just like `sleep(5
Repository: spark
Updated Branches:
refs/heads/master 3c0d2e552 -> 060c79aab
[SPARK-9056] [STREAMING] Rename configuration
`spark.streaming.minRememberDuration` to
`spark.streaming.fileStream.minRememberDuration`
Rename configuration `spark.streaming.minRememberDuration` to
`spark.streaming
Repository: spark
Updated Branches:
refs/heads/master 69f5a7c93 -> b41a32718
[SPARK-1855] Local checkpointing
Certain use cases of Spark involve RDDs with long lineages that must be
truncated periodically (e.g. GraphX). The existing way of doing it is through
`rdd.checkpoint()`, which is exp
Repository: spark
Updated Branches:
refs/heads/master 2b67fdb60 -> d34bac0e1
[SPARK-9504] [STREAMING] [TESTS] Fix o.a.s.streaming.StreamingContextSuite.stop
gracefully again
The test failure is here:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/3150/AMPLAB_JENKINS_BUILD_PROFIL
Repository: spark
Updated Branches:
refs/heads/branch-1.5 d196d3607 -> 6e72d24e2
[SPARK-9504] [STREAMING] [TESTS] Fix o.a.s.streaming.StreamingContextSuite.stop
gracefully again
The test failure is here:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/3150/AMPLAB_JENKINS_BUILD_PR
ses #7825 from tdas/kinesis-receiver and squashes the following commits:
2159be9 [Tathagata Das] Fixed bug
569be83 [Tathagata Das] Fix scala style issue
bf31e22 [Tathagata Das] Added more documentation to make the kinesis test
endpoint more configurable
3ad8361 [Tathagata Das] Merge remote-track
825 from tdas/kinesis-receiver and squashes the following commits:
2159be9 [Tathagata Das] Fixed bug
569be83 [Tathagata Das] Fix scala style issue
bf31e22 [Tathagata Das] Added more documentation to make the kinesis test
endpoint more configurable
3ad8361 [Tathagata Das] Merge remote-tracking bra
Repository: spark
Updated Branches:
refs/heads/master 6d8a6e416 -> 1bf608b5e
[SPARK-9601] [DOCS] Fix JavaPairDStream signature for stream-stream and
windowed join in streaming guide doc
Author: Namit Katariya
Closes #7935 from namitk/SPARK-9601 and squashes the following commits:
03b5784 [
Repository: spark
Updated Branches:
refs/heads/branch-1.5 7fa419535 -> 6306019ff
[SPARK-9601] [DOCS] Fix JavaPairDStream signature for stream-stream and
windowed join in streaming guide doc
Author: Namit Katariya
Closes #7935 from namitk/SPARK-9601 and squashes the following commits:
03b57
Repository: spark
Updated Branches:
refs/heads/master 0d7aac99d -> a1bbf1bc5
[SPARK-8978] [STREAMING] Implements the DirectKafkaRateController
Author: Dean Wampler
Author: Nilanjan Raychaudhuri
Author: François Garillot
Closes #7796 from dragos/topic/streaming-bp/kafka-direct and squashe
Repository: spark
Updated Branches:
refs/heads/branch-1.5 8a7956283 -> 8b00c0690
[SPARK-8978] [STREAMING] Implements the DirectKafkaRateController
Author: Dean Wampler
Author: Nilanjan Raychaudhuri
Author: François Garillot
Closes #7796 from dragos/topic/streaming-bp/kafka-direct and squ
nal API to return the current rate of block generators as
Long instead of Option\[Long\] (was inconsistent at places).
- Updated existing `ReceiverTrackerSuite` to test that custom block generators
get rate updates as well.
Author: Tathagata Das
Closes #7913 from tdas/SPARK-9556 and squashes
few internal API to return the current rate of block generators as
Long instead of Option\[Long\] (was inconsistent at places).
- Updated existing `ReceiverTrackerSuite` to test that custom block generators
get rate updates as well.
Author: Tathagata Das
Closes #7913 from tdas/SPARK-9556 and squas
Repository: spark
Updated Branches:
refs/heads/branch-1.5 3997dd3fd -> 8ecfb05e3
[DOCS] [STREAMING] make the existing parameter docs for OffsetRange acâ¦
â¦tually visible
Author: cody koeninger
Closes #7995 from koeninger/doc-fixes and squashes the following commits:
87af9ea [cody koenin
Repository: spark
Updated Branches:
refs/heads/master 0a078303d -> 1723e3489
[DOCS] [STREAMING] make the existing parameter docs for OffsetRange acâ¦
â¦tually visible
Author: cody koeninger
Closes #7995 from koeninger/doc-fixes and squashes the following commits:
87af9ea [cody koeninger]
Repository: spark
Updated Branches:
refs/heads/master 1723e3489 -> 346209097
[SPARK-9639] [STREAMING] Fix a potential NPE in Streaming JobScheduler
Because `JobScheduler.stop(false)` may set `eventLoop` to null when
`JobHandler` is running, then it's possible that when `post` is called,
`eve
Repository: spark
Updated Branches:
refs/heads/branch-1.5 8ecfb05e3 -> 980687206
[SPARK-9639] [STREAMING] Fix a potential NPE in Streaming JobScheduler
Because `JobScheduler.stop(false)` may set `eventLoop` to null when
`JobHandler` is running, then it's possible that when `post` is called,
Repository: spark
Updated Branches:
refs/heads/master c4fd2a242 -> 853809e94
[SPARK-5155] [PYSPARK] [STREAMING] Mqtt streaming support in Python
This PR is based on #4229, thanks prabeesh.
Closes #4229
Author: Prabeesh K
Author: zsxwing
Author: prabs
Author: Prabeesh K
Closes #7833 from
Repository: spark
Updated Branches:
refs/heads/branch-1.5 51406becc -> 8f4014fda
[SPARK-5155] [PYSPARK] [STREAMING] Mqtt streaming support in Python
This PR is based on #4229, thanks prabeesh.
Closes #4229
Author: Prabeesh K
Author: zsxwing
Author: prabs
Author: Prabeesh K
Closes #7833
Repository: spark
Updated Branches:
refs/heads/branch-1.5 8f4014fda -> 94692bb14
[SPARK-9801] [STREAMING] Check if file exists before deleting temporary files.
Spark streaming deletes the temp file and backup files without checking if they
exist or not
Author: Hao Zhu
Closes #8082 from via
Repository: spark
Updated Branches:
refs/heads/master 853809e94 -> 3c9802d94
[SPARK-9801] [STREAMING] Check if file exists before deleting temporary files.
Spark streaming deletes the temp file and backup files without checking if they
exist or not
Author: Hao Zhu
Closes #8082 from viadea/
Repository: spark
Updated Branches:
refs/heads/branch-1.4 4b5bbc589 -> 6dde38026
[SPARK-9801] [STREAMING] Check if file exists before deleting temporary files.
Spark streaming deletes the temp file and backup files without checking if they
exist or not
Author: Hao Zhu
Closes #8082 from via
Repository: spark
Updated Branches:
refs/heads/branch-1.3 b104501d3 -> a98603f8c
[SPARK-9801] [STREAMING] Check if file exists before deleting temporary files.
Spark streaming deletes the temp file and backup files without checking if they
exist or not
Author: Hao Zhu
Closes #8082 from via
Repository: spark
Updated Branches:
refs/heads/master 91e9389f3 -> 0f90d6055
[SPARK-9640] [STREAMING] [TEST] Do not run Python Kinesis tests when the
Kinesis assembly JAR has not been generated
Author: Tathagata Das
Closes #7961 from tdas/SPARK-9640 and squashes the following comm
Repository: spark
Updated Branches:
refs/heads/branch-1.5 f9beef998 -> c7f009040
[SPARK-9640] [STREAMING] [TEST] Do not run Python Kinesis tests when the
Kinesis assembly JAR has not been generated
Author: Tathagata Das
Closes #7961 from tdas/SPARK-9640 and squashes the following comm
Repository: spark
Updated Branches:
refs/heads/master 55752d883 -> 600031ebe
[SPARK-9727] [STREAMING] [BUILD] Updated streaming kinesis SBT project name to
be more consistent
Author: Tathagata Das
Closes #8092 from tdas/SPARK-9727 and squashes the following commits:
b1b01fd [Tathagata
Repository: spark
Updated Branches:
refs/heads/branch-1.5 c7f009040 -> ebbd3b616
[SPARK-9727] [STREAMING] [BUILD] Updated streaming kinesis SBT project name to
be more consistent
Author: Tathagata Das
Closes #8092 from tdas/SPARK-9727 and squashes the following commits:
b1b01fd [Tathag
Repository: spark
Updated Branches:
refs/heads/master dbd778d84 -> 5b8bb1b21
[SPARK-9572] [STREAMING] [PYSPARK] Added StreamingContext.getActiveOrCreate()
in Python
Author: Tathagata Das
Closes #8080 from tdas/SPARK-9572 and squashes the following commits:
64a231d [Tathagata Das]
Repository: spark
Updated Branches:
refs/heads/branch-1.5 b077f36ea -> 71460b889
[SPARK-9572] [STREAMING] [PYSPARK] Added StreamingContext.getActiveOrCreate()
in Python
Author: Tathagata Das
Closes #8080 from tdas/SPARK-9572 and squashes the following commits:
64a231d [Tathagata Das]
Repository: spark
Updated Branches:
refs/heads/branch-1.5 3298fb69f -> 62ab2a4c6
[SPARK-9780] [STREAMING] [KAFKA] prevent NPE if KafkaRDD instantiation â¦
â¦fails
Author: cody koeninger
Closes #8133 from koeninger/SPARK-9780 and squashes the following commits:
406259d [cody koeninger] [S
Repository: spark
Updated Branches:
refs/heads/master 660e6dcff -> 8ce60963c
[SPARK-9780] [STREAMING] [KAFKA] prevent NPE if KafkaRDD instantiation â¦
â¦fails
Author: cody koeninger
Closes #8133 from koeninger/SPARK-9780 and squashes the following commits:
406259d [cody koeninger] [SPARK
ers that sets a lower bound on the estimated rate, so that the system
always receives some data.
Author: Tathagata Das
Closes #8199 from tdas/SPARK-9966 and squashes the following commits:
829f793 [Tathagata Das] Fixed unit test and added comments
3a994db [Tathagata Das] Added min rate and upda
e a
parameters that sets a lower bound on the estimated rate, so that the system
always receives some data.
Author: Tathagata Das
Closes #8199 from tdas/SPARK-9966 and squashes the following commits:
829f793 [Tathagata Das] Fixed unit test and added comments
3a994db [Tathagata Das] Added min rate
ing data to the buffer.
Author: Tathagata Das
Closes #8204 from tdas/SPARK-9968 and squashes the following commits:
8cbcc1b [Tathagata Das] Removed unused val
a73b645 [Tathagata Das] Reduced time spent within synchronized block
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: h
for adding data to the buffer.
Author: Tathagata Das
Closes #8204 from tdas/SPARK-9968 and squashes the following commits:
8cbcc1b [Tathagata Das] Removed unused val
a73b645 [Tathagata Das] Reduced time spent within synchronized block
(cherry picked from commit 18a761ef7a01a4dfa1dd91abe78cd68f2f8fd
Repository: spark
Updated Branches:
refs/heads/master 8bae9015b -> bf1d6614d
[SPARK-9574] [STREAMING] Remove unnecessary contents of
spark-streaming-XXX-assembly jars
Removed contents already included in Spark assembly jar from
spark-streaming-XXX-assembly jars.
Author: zsxwing
Closes #80
Repository: spark
Updated Branches:
refs/heads/branch-1.5 9bd2e6f7c -> 2bccd918f
[SPARK-9574] [STREAMING] Remove unnecessary contents of
spark-streaming-XXX-assembly jars
Removed contents already included in Spark assembly jar from
spark-streaming-XXX-assembly jars.
Author: zsxwing
Closes
hor: Tathagata Das
Closes #8289 from tdas/SPARK-10098.
(cherry picked from commit 9108eff74a2815986fd067b273c2a344b6315405)
Signed-off-by: Tathagata Das
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e1b50c7d
Tree: http://git-
ata Das
Closes #8289 from tdas/SPARK-10098.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9108eff7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9108eff7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9108e
eue outside the synchronized to
prevent deadlock.
Author: Tathagata Das
Closes #8257 from tdas/SPARK-10072.
(cherry picked from commit 1aeae05bb20f01ab7ccaa62fe905a63e020074b5)
Signed-off-by: Tathagata Das
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.
ide the synchronized to
prevent deadlock.
Author: Tathagata Das
Closes #8257 from tdas/SPARK-10072.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1aeae05b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1aeae05b
D
Repository: spark
Updated Branches:
refs/heads/master 1aeae05bb -> 90273eff9
[SPARK-10102] [STREAMING] Fix a race condition that startReceiver may happen
before setting trackerState to Started
Test failure:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFIL
Repository: spark
Updated Branches:
refs/heads/branch-1.5 08c5962a2 -> a6f8979c8
[SPARK-10102] [STREAMING] Fix a race condition that startReceiver may happen
before setting trackerState to Started
Test failure:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PR
; enabled}
- Change Java Deprecated annotations to Scala deprecated annotation with more
information.
Author: Tathagata Das
Closes #8299 from tdas/SPARK-9967.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bc9a0e03
Tree: http:
; enabled}
- Change Java Deprecated annotations to Scala deprecated annotation with more
information.
Author: Tathagata Das
Closes #8299 from tdas/SPARK-9967.
(cherry picked from commit bc9a0e03235865d2ec33372f6400dec8c770778a)
Signed-off-by: Tathagata Das
Project: http://git-wip-us.apache.
Repository: spark
Updated Branches:
refs/heads/master 2f2686a73 -> 1f29d502e
[SPARK-9812] [STREAMING] Fix Python 3 compatibility issue in PySpark Streaming
and some docs
This PR includes the following fixes:
1. Use `range` instead of `xrange` in `queue_stream.py` to support Python 3.
2. Fix t
Repository: spark
Updated Branches:
refs/heads/branch-1.5 321cb99ca -> 16414dae0
[SPARK-9812] [STREAMING] Fix Python 3 compatibility issue in PySpark Streaming
and some docs
This PR includes the following fixes:
1. Use `range` instead of `xrange` in `queue_stream.py` to support Python 3.
2. F
Repository: spark
Updated Branches:
refs/heads/master 1f29d502e -> affc8a887
[SPARK-10125] [STREAMING] Fix a potential deadlock in JobGenerator.stop
Because `lazy val` uses `this` lock, if JobGenerator.stop and
JobGenerator.doCheckpoint (JobGenerator.shouldCheckpoint has not yet been
initial
Repository: spark
Updated Branches:
refs/heads/branch-1.5 a3ed2c31e -> 63922fa4d
[SPARK-10125] [STREAMING] Fix a potential deadlock in JobGenerator.stop
Because `lazy val` uses `this` lock, if JobGenerator.stop and
JobGenerator.doCheckpoint (JobGenerator.shouldCheckpoint has not yet been
ini
ect classloader and the
SequenceNumberRanges class (in streaming-kinesis-asl package) cannot be found
(added through spark-submit) while deserializing. The solution is to use
`Thread.currentThread().getContextClassLoader` while deserializing.
Author: Tathagata Das
Closes #8328 from tdas/SPARK-10128
ect classloader and the
SequenceNumberRanges class (in streaming-kinesis-asl package) cannot be found
(added through spark-submit) while deserializing. The solution is to use
`Thread.currentThread().getContextClassLoader` while deserializing.
Author: Tathagata Das
Closes #8328 from tdas/SPARK-10
122).
tdas , please help to review.
Author: jerryshao
Closes #8347 from jerryshao/SPARK-10122 and squashes the following commits:
4039b16 [jerryshao] Fix getOffsetRanges in transform() bug
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/
ARK-10122).
tdas , please help to review.
Author: jerryshao
Closes #8347 from jerryshao/SPARK-10122 and squashes the following commits:
4039b16 [jerryshao] Fix getOffsetRanges in transform() bug
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/re
ary support for "delta" to implement correctly. So
disabling the support for multiple streaming aggregations.
## How was this patch tested?
Additional unit tests
Author: Tathagata Das
Closes #13210 from tdas/SPARK-15428.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
the necessary support for "delta" to implement correctly. So
disabling the support for multiple streaming aggregations.
## How was this patch tested?
Additional unit tests
Author: Tathagata Das
Closes #13210 from tdas/SPARK-15428.
(cherry picked from commit 1ffa608ba5a849739a56047bda8
onf that determines whether schema inference for
file streams is allowed or not. It is disabled by default.
## How was this patch tested?
Updated unit tests that test error behavior with and without schema inference
enabled.
Author: Tathagata Das
Closes #13238 from tdas/SPARK-15458.
Proj
e a SQLConf that determines whether schema inference for
file streams is allowed or not. It is disabled by default.
## How was this patch tested?
Updated unit tests that test error behavior with and without schema inference
enabled.
Author: Tathagata Das
Closes #13238 from tdas/SPARK-15458.
(che
Repository: spark
Updated Branches:
refs/heads/master 695dbc816 -> 0cfd6192f
[SPARK-15580][SQL] Add ContinuousQueryInfo to make ContinuousQueryListener
events serializable
## What changes were proposed in this pull request?
This PR adds ContinuousQueryInfo to make ContinuousQueryListener eve
Repository: spark
Updated Branches:
refs/heads/branch-2.0 ec556fec0 -> 003c44792
[SPARK-15580][SQL] Add ContinuousQueryInfo to make ContinuousQueryListener
events serializable
## What changes were proposed in this pull request?
This PR adds ContinuousQueryInfo to make ContinuousQueryListener
Repository: spark
Updated Branches:
refs/heads/master b914e1930 -> 4d9d9cc58
[SPARK-15853][SQL] HDFSMetadataLog.get should close the input stream
## What changes were proposed in this pull request?
This PR closes the input stream created in `HDFSMetadataLog.get`
## How was this patch tested?
Repository: spark
Updated Branches:
refs/heads/branch-2.0 00bbf7873 -> ca0801120
[SPARK-15853][SQL] HDFSMetadataLog.get should close the input stream
## What changes were proposed in this pull request?
This PR closes the input stream created in `HDFSMetadataLog.get`
## How was this patch tes
Repository: spark
Updated Branches:
refs/heads/master 5a3533e77 -> 00c310133
[SPARK-15593][SQL] Add DataFrameWriter.foreach to allow the user consuming data
in ContinuousQuery
## What changes were proposed in this pull request?
* Add DataFrameWriter.foreach to allow the user consuming data i
Repository: spark
Updated Branches:
refs/heads/branch-2.0 1371d5ece -> 02ed7b536
[SPARK-15593][SQL] Add DataFrameWriter.foreach to allow the user consuming data
in ContinuousQuery
## What changes were proposed in this pull request?
* Add DataFrameWriter.foreach to allow the user consuming da
mon useful functionality.
Support for other operations will come later.
## How was this patch tested?
Additional unit tests.
Author: Tathagata Das
Closes #13549 from tdas/SPARK-15812.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/sp
mon useful functionality.
Support for other operations will come later.
## How was this patch tested?
Additional unit tests.
Author: Tathagata Das
Closes #13549 from tdas/SPARK-15812.
(cherry picked from commit abdb5d42c5802c8f60876aa1285c803d02881258)
Signed-off-by: Tathagata Das
Proj
Repository: spark
Updated Branches:
refs/heads/master 4134653e5 -> d32e22778
[MINOR][SQL] Standardize 'continuous queries' to 'streaming Datasets/DataFrames'
## What changes were proposed in this pull request?
This patch does some replacing (as `streaming Datasets/DataFrames` is the term
we'
Repository: spark
Updated Branches:
refs/heads/branch-2.0 413826d40 -> bd27dc489
[MINOR][SQL] Standardize 'continuous queries' to 'streaming Datasets/DataFrames'
## What changes were proposed in this pull request?
This patch does some replacing (as `streaming Datasets/DataFrames` is the term
jenkins/view/Spark%20QA%20Compile/job/spark-branch-2.0-compile-maven-hadoop-2.3/326/console
Author: Tathagata Das
Closes #13645 from tdas/build-break.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a6a18a45
Tree: http://git-
.edu/jenkins/view/Spark%20QA%20Compile/job/spark-branch-2.0-compile-maven-hadoop-2.3/326/console
Author: Tathagata Das
Closes #13645 from tdas/build-break.
(cherry picked from commit a6a18a4573515e76d78534f1a19fcc2c3819f6c5)
Signed-off-by: Tathagata Das
Project: http://git-wip-us.apache.org/repo
Repository: spark
Updated Branches:
refs/heads/master a87a56f5c -> 96c3500c6
[SPARK-15935][PYSPARK] Enable test for sql/streaming.py and fix these tests
## What changes were proposed in this pull request?
This PR just enables tests for sql/streaming.py and also fixes the failures.
## How was
Repository: spark
Updated Branches:
refs/heads/branch-2.0 cf52375b9 -> d59859d38
[SPARK-15935][PYSPARK] Enable test for sql/streaming.py and fix these tests
## What changes were proposed in this pull request?
This PR just enables tests for sql/streaming.py and also fixes the failures.
## How
for DataFrameReader/Writer and
DataStreamReader/Writer.
Author: Tathagata Das
Closes #13653 from tdas/SPARK-15933.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/214adb14
Tree: http://git-wip-us.apache.org/repos/asf/spark
Repository: spark
Updated Branches:
refs/heads/master 5d50d4f0f -> 214adb14b
http://git-wip-us.apache.org/repos/asf/spark/blob/214adb14/sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
--
diff --git
Repository: spark
Updated Branches:
refs/heads/branch-2.0 9adba414c -> 96274d73e
http://git-wip-us.apache.org/repos/asf/spark/blob/96274d73/sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
--
diff -
for DataFrameReader/Writer and
DataStreamReader/Writer.
Author: Tathagata Das
Closes #13653 from tdas/SPARK-15933.
(cherry picked from commit 214adb14b8d1f1c4dce0c97dd6dc09efedbaa643)
Signed-off-by: Tathagata Das
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip
Repository: spark
Updated Branches:
refs/heads/master 8a977b065 -> 3554713a1
[SPARK-16114][SQL] structured streaming network word count examples
## What changes were proposed in this pull request?
Network word count example for structured streaming
## How was this patch tested?
Run locally
Repository: spark
Updated Branches:
refs/heads/branch-2.0 d73c38ed0 -> 5fb7804e5
[SPARK-16114][SQL] structured streaming network word count examples
## What changes were proposed in this pull request?
Network word count example for structured streaming
## How was this patch tested?
Run loca
Repository: spark
Updated Branches:
refs/heads/master 823518c2b -> 5bf8881b3
[SPARK-16268][PYSPARK] SQLContext should import DataStreamReader
## What changes were proposed in this pull request?
Fixed the following error:
```
>>> sqlContext.readStream
Traceback (most recent call last):
File
Repository: spark
Updated Branches:
refs/heads/branch-2.0 d7a59f1f4 -> 835c5a3bd
[SPARK-16268][PYSPARK] SQLContext should import DataStreamReader
## What changes were proposed in this pull request?
Fixed the following error:
```
>>> sqlContext.readStream
Traceback (most recent call last):
F
Repository: spark
Updated Branches:
refs/heads/branch-2.0 6650c0533 -> 904122335
[TRIVIAL][DOCS][STREAMING][SQL] The return type mentioned in the Javadoc is
incorrect for toJavaRDD, â¦
## What changes were proposed in this pull request?
Change the return type mentioned in the JavaDoc for `t
Repository: spark
Updated Branches:
refs/heads/master f454a7f9f -> 757dc2c09
[TRIVIAL][DOCS][STREAMING][SQL] The return type mentioned in the Javadoc is
incorrect for toJavaRDD, â¦
## What changes were proposed in this pull request?
Change the return type mentioned in the JavaDoc for `toJav
Repository: spark
Updated Branches:
refs/heads/master cb1b9d34f -> 64132a14f
[SPARK-16256][SQL][STREAMING] Added Structured Streaming Programming Guide
Title defines all.
Author: Tathagata Das
Closes #13945 from tdas/SPARK-16256.
Project: http://git-wip-us.apache.org/repos/asf/spark/r
Repository: spark
Updated Branches:
refs/heads/branch-2.0 edd1905c0 -> 3cc258efb
[SPARK-16256][SQL][STREAMING] Added Structured Streaming Programming Guide
Title defines all.
Author: Tathagata Das
Closes #13945 from tdas/SPARK-16256.
(cherry picked from com
[SPARK-12177][STREAMING][KAFKA] Update KafkaDStreams to new Kafka 0.10 Consumer
API
## What changes were proposed in this pull request?
New Kafka consumer api for the released 0.10 version of Kafka
## How was this patch tested?
Unit tests, manual tests
Author: cody koeninger
Closes #11863 f
Repository: spark
Updated Branches:
refs/heads/master bde1d6a61 -> dedbceec1
http://git-wip-us.apache.org/repos/asf/spark/blob/dedbceec/external/kafka-0-10/src/test/java/org/apache/spark/streaming/kafka010/JavaConsumerStrategySuite.java
--
[SPARK-12177][STREAMING][KAFKA] Update KafkaDStreams to new Kafka 0.10 Consumer
API
## What changes were proposed in this pull request?
New Kafka consumer api for the released 0.10 version of Kafka
## How was this patch tested?
Unit tests, manual tests
Author: cody koeninger
Closes #11863 f
Repository: spark
Updated Branches:
refs/heads/branch-2.0 a54852350 -> 3134f116a
http://git-wip-us.apache.org/repos/asf/spark/blob/3134f116/external/kafka-0-10/src/test/java/org/apache/spark/streaming/kafka010/JavaConsumerStrategySuite.java
--
Repository: spark
Updated Branches:
refs/heads/master dedbceec1 -> 2c3d96134
[SPARK-16256][DOCS] Minor fixes on the Structured Streaming Programming Guide
Author: Tathagata Das
Closes #13978 from tdas/SPARK-16256-1.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: h
Repository: spark
Updated Branches:
refs/heads/branch-2.0 3134f116a -> c8a7c2305
[SPARK-16256][DOCS] Minor fixes on the Structured Streaming Programming Guide
Author: Tathagata Das
Closes #13978 from tdas/SPARK-16256-1.
(cherry picked from commit 2c3d96134dcc0428983eea087db7e91072215
1 - 100 of 1024 matches
Mail list logo
