Re: Filtering JSON records when there isn't an exact schema match in Spark

2023-07-04 Thread Hill Liu
I think you can define schema with column z and filter out records with z is null. On Tue, Jul 4, 2023 at 3:24 PM Shashank Rao wrote: > Yes, drop malformed does filter out record4. However, record 5 is not. > > On Tue, 4 Jul 2023 at 07:41, Vikas Kumar wrote: > >> Have you tried dropmalformed

Structured streaming append mode picture question

2023-05-31 Thread Hill Liu
Hi, I have a question related to this picture https://spark.apache.org/docs/latest/img/structured-streaming-watermark-append-mode.png in structured streaming programming guide web page. At 12:20, the wm is 12:11, so the window 12:00 ~ 12:10 will be flushed out. But in this pic, it seems the window