Hi Deepak, Why are you getting paths from kafka topic? any specific reason to do so ?
Regards, Shyam On Mon, Jun 10, 2019 at 10:44 AM Deepak Sharma <[email protected]> wrote: > The context is different here. > The file path are coming as messages in kafka topic. > Spark streaming (structured) consumes form this topic. > Now it have to get the value from the message , thus the path to file. > read the json stored at the file location into another df. > > Thanks > Deepak > > On Sun, Jun 9, 2019 at 11:03 PM vaquar khan <[email protected]> wrote: > >> Hi Deepak, >> >> You can use textFileStream. >> >> https://spark.apache.org/docs/2.2.0/streaming-programming-guide.html >> >> Plz start using stackoverflow to ask question to other ppl so get >> benefits of answer >> >> >> Regards, >> Vaquar khan >> >> On Sun, Jun 9, 2019, 8:08 AM Deepak Sharma <[email protected]> wrote: >> >>> I am using spark streaming application to read from kafka. >>> The value coming from kafka message is path to hdfs file. >>> I am using spark 2.x , spark.read.stream. >>> What is the best way to read this path in spark streaming and then read >>> the json stored at the hdfs path , may be using spark.read.json , into a df >>> inside the spark streaming app. >>> Thanks a lot in advance >>> >>> -- >>> Thanks >>> Deepak >>> >> > > -- > Thanks > Deepak > www.bigdatabig.com > www.keosha.net >
