Re: Spark StructuredStreaming - watermark not working as expected

Hi Mich, I'm currently testing this on my mac .. are you able to reproduce this issue ? Note - the code is similar .. except outputMode is set to update. wrt outputMode - when using aggregation + watermark, the outputMode should be either append Or update, in your code - you have used 'complete'

Re: Spark StructuredStreaming - watermark not working as expected

Hi Karan, The version tested was 3.1.1. Are you running on Dataproc serverless 3.1.3? Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Palantir Technologies Limited view my Linkedin profile