thanks, thats helpful.
On Wed, May 30, 2018 at 5:05 PM, Lalwani, Jayesh <
jayesh.lalw...@capitalone.com> wrote:
> Few things
>
>
>
>1. Append mode is going to output data that falls out of the watermark
>2. Structured streaming isn’t time based. It reacts only when it sees
>input
Few things
1. Append mode is going to output data that falls out of the watermark
2. Structured streaming isn’t time based. It reacts only when it sees input
data. If no data appears in the input it will not move the aggregation window
3. Clock time is irrelevant to structured
let me ask this another way: if i run this program and then feed it a
single value (on nc), it returns a single result, which is an empty batch.
it will not return anything else after that, no matter how long i wait.
this only happens with watermarking and append output mode.
what do i do to
hello all,
just playing with structured streaming aggregations for the first time.
this is my little program i run inside sbt:
import org.apache.spark.sql.functions._
val lines = spark.readStream
.format("socket")
.option("host", "localhost")
.option("port", )
