Re: [Structured Streaming] Avoid one microbatch delay with multiple stateful operations

2024-01-10 Thread Ant Kutschera
Hi *Do we have any option to make streaming queries with multiple stateful operations output data without waiting this extra iteration? One of my ideas was to force an empty microbatch to run and propagate late events watermark without any new data. While this conceptually works, I didn't find a

Re: Structured Streaming Process Each Records Individually

2024-01-10 Thread Ant Kutschera
quot;city": "Bangalore",* >>> *"state": "KA",* >>> *"pin": "577401"* >>> *},* >>> *{* >>> *"city": "Mumbai",* >>> *"state": "MH",* >>> *"pin": "576003"* >>> *}* >>> *],* >>> *"@table": "product"**}* >> >> >> so for the above json , there are 3 tables created >> 1. Product (@type) THis is a parent table >> 2. poduct_zones and product_devices , child table >> > -- ___ Dr Ant Kutschera