Looks like you're trying to add two stateful operations in a chain -
actually this would trigger the limitation of global watermark and lead the
output "possibly" to be incorrect.
We've documented the limitations in the SS guide doc starting from Spark
3.0, so please take time to read the doc to
I am trying to write a Spark Structured Streaming application consisting
of GroupState construct followed by aggregation.
Events arriving from event sources are bucketized by deviceId and quantized
timestamp, composed together into group state key idTime.
Logical plan consists of stages (in the
Cool, thanks!
On Sat, Feb 20, 2021 at 3:07 PM Sean Owen wrote:
> Another RC is starting imminently, which looks pretty good. If it
> succeeds, probably next week.
> It will support Scala 2.12, but I believe a Scala 2.13 build is only
> coming in 3.2.0.
>
> On Sat, Feb 20, 2021 at 1:54 PM
Another RC is starting imminently, which looks pretty good. If it succeeds,
probably next week.
It will support Scala 2.12, but I believe a Scala 2.13 build is only coming
in 3.2.0.
On Sat, Feb 20, 2021 at 1:54 PM Bulldog20630405
wrote:
>
> what is the expected ballpark release date of spark
what is the expected ballpark release date of spark 3.1.1 ?
will it be built with scala 2.13 ?
