Hi: I am using spark structured streaming (v 2.2.0) to read data from files. I have configured checkpoint location. On stopping and restarting the application, it looks like it is reading the previously ingested files. Is that expected behavior ? Is there anyway to prevent reading files that have already been ingested ? If a file is partially ingested, on restart - can we start reading the file from previously checkpointed offset ? Thanks
- Apache Spark - Structured Streaming from file - checkp... M Singh
- Re: Apache Spark - Structured Streaming from file... Diogo Munaro Vieira