Hi, to my best knowledge, the existing FileStreamSource reads all the files in a directory (hive table). However, I need to be able to specify an initial partition it should start from (i.e. like a Kafka offset/initial warmed-up state) and then only read data which is semantically (i.e. using a file path lexicographically) greater than the minimum committed initial state?
After playing around with the internals of the file format I have come to the conclusion that manually modifying it and setting some values (i.e. the last processed & committed partition) is not feasible as spark regardless will pick up all the files (even the older partitions) https://stackoverflow.com/questions/58004832/spark-structured-streaming-file-source-read-from-a-certain-partition-onwards Is this correct? This leads me to the conclusion I need a custom StatefulFileStreamSource. I tried to create one ( https://gist.github.com/geoHeil/6c0c51e43469ace71550b426cfcce1c1), but so far fail to instantiate it (even though it is just a copy of the original one as the constructor without any arguments does not seem to be defined: NoSuchMethodException: org.apache.spark.sql.execution.streaming.StatefulFileStreamSource.<init>() Why is the default constructor not found? Even for a simple copy of an existing (and presumably working class? Note, I am currently working on spark 2.2.3 Best, Georg