Hi, You can monitor a filesystem directory as streaming source as long as the files placed there are atomically copied/moved into the directory. Updating the files is not supported.
kr, Gerard. On Mon, Jan 15, 2018 at 11:41 PM, kant kodali <kanth...@gmail.com> wrote: > Hi All, > > I am wondering if HDFS can be a streaming source like Kafka in Spark > 2.2.0? For example can I have stream1 reading from Kafka and writing to > HDFS and stream2 to read from HDFS and write it back to Kakfa ? such that > stream2 will be pulling the latest updates written by stream1. > > Thanks! >
