Re: What's the root cause of not supporting multiple aggregations in structured streaming?

Nice blog! Thanks for sharing, Etienne! Let's try to raise this discussion again after the 3.1 release. I do think more committers/contributors had realized the issue of global watermark per SPARK-24634 and SPARK-33259

Re: Spark 3.1 branch cut 4th Dec?

Just as a gentle reminder, we have one week left before the branch-cut. Please focus on finalizing the works to land in Spark 3.1.0. After the branch-cut, I will look through the JIRAs and update the fields correctly, for example, retargeting to 3.2.0 in the JIRAs targeted 3.1.0. As Dongjoon point

How to convert InternalRow to Row.

Hi dev, i'm working on generating custom pipeline on the fly, which means I generate SparkPlan along with each node in my pipeline. So, my pipeline end up with PipeLineRelation extending BaseRelation like: case class PipeLineRelation(schema: StructType, pipeLinePlan: LogicalPlan)(@transient over

Re: What's the root cause of not supporting multiple aggregations in structured streaming?

Hi, Regarding this subject I wrote a blog article that gives details about the watermark architecture proposal that was discussed in the design doc and in the PR: https://echauchot.blogspot.com/2020/11/watermark-architecture-proposal-for.html Best Etienne On 29/09/2020 03:24, Yuanjian Li w