The context is different here. The file path are coming as messages in kafka topic. Spark streaming (structured) consumes form this topic. Now it have to get the value from the message , thus the path to file. read the json stored at the file location into another df.
Thanks Deepak On Sun, Jun 9, 2019 at 11:03 PM vaquar khan <vaquar.k...@gmail.com> wrote: > Hi Deepak, > > You can use textFileStream. > > https://spark.apache.org/docs/2.2.0/streaming-programming-guide.html > > Plz start using stackoverflow to ask question to other ppl so get benefits > of answer > > > Regards, > Vaquar khan > > On Sun, Jun 9, 2019, 8:08 AM Deepak Sharma <deepakmc...@gmail.com> wrote: > >> I am using spark streaming application to read from kafka. >> The value coming from kafka message is path to hdfs file. >> I am using spark 2.x , spark.read.stream. >> What is the best way to read this path in spark streaming and then read >> the json stored at the hdfs path , may be using spark.read.json , into a df >> inside the spark streaming app. >> Thanks a lot in advance >> >> -- >> Thanks >> Deepak >> > -- Thanks Deepak www.bigdatabig.com www.keosha.net
