Please refer the structured streaming guide doc which is very clear of
representing when the query will have unbounded state.
http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#inner-joins-with-optional-watermarking
Quoting the doc:
In other words, you will
You mean to say that Spark will store all the data in memory forever :)
> On 10-Dec-2018, at 6:16 PM, Sandeep Katta
> wrote:
>
> Hi Abhijeet,
>
> You are using inner join with unbounded state which means every data in
> stream ll match with other stream infinitely,
> If you want the
Hello,
I’m using watermark to join two streams as you can see below:
val order_wm = order_details.withWatermark("tstamp_trans", "20 seconds")
val invoice_wm = invoice_details.withWatermark("tstamp_trans", "20 seconds")
val join_df = order_wm
.join(invoice_wm, order_wm.col("s_order_id") ===
