Hi Deepak,
Please let us know - how you managed it ?
Thanks,
NJ
On Mon, Jun 10, 2019 at 4:42 PM Deepak Sharma wrote:
> Thanks All.
> I managed to get this working.
> Marking this thread as closed.
>
> On Mon, Jun 10, 2019 at 4:14 PM Deepak Sharma
> wrote:
>
>> This is the project requirement
Thanks All.
I managed to get this working.
Marking this thread as closed.
On Mon, Jun 10, 2019 at 4:14 PM Deepak Sharma wrote:
> This is the project requirement , where paths are being streamed in kafka
> topic.
> Seems it's not possible using spark structured streaming.
>
>
> On Mon, Jun 10,
Hi Deepak,
Why are you getting paths from kafka topic? any specific reason to do so ?
Regards,
Shyam
On Mon, Jun 10, 2019 at 10:44 AM Deepak Sharma
wrote:
> The context is different here.
> The file path are coming as messages in kafka topic.
> Spark streaming (structured) consumes form this
The context is different here.
The file path are coming as messages in kafka topic.
Spark streaming (structured) consumes form this topic.
Now it have to get the value from the message , thus the path to file.
read the json stored at the file location into another df.
Thanks
Deepak
On Sun, Jun
Hi Deepak,
You can use textFileStream.
https://spark.apache.org/docs/2.2.0/streaming-programming-guide.html
Plz start using stackoverflow to ask question to other ppl so get benefits
of answer
Regards,
Vaquar khan
On Sun, Jun 9, 2019, 8:08 AM Deepak Sharma wrote:
> I am using spark
I am using spark streaming application to read from kafka.
The value coming from kafka message is path to hdfs file.
I am using spark 2.x , spark.read.stream.
What is the best way to read this path in spark streaming and then read the
json stored at the hdfs path , may be using spark.read.json ,
