Thanks Akhil, I will have a look.
I have a dude regarding to spark streaming and filestream. If spark
streaming crashs and while spark was down new files are created in input
folder, when spark streaming is launched again, how can I process these
files?
Thanks.
Regards.
Miguel.
On Thu, Aug
Have a look at the spark streaming. You can make use of the ssc.fileStream.
Eg:
val avroStream = ssc.fileStream[AvroKey[GenericRecord], NullWritable,
AvroKeyInputFormat[GenericRecord]](input)
You can also specify a filter function
If you have enabled checkpointing the spark will handle that for you.
Thanks
Best Regards
On Thu, Aug 27, 2015 at 4:21 PM, Masf masfwo...@gmail.com wrote:
Thanks Akhil, I will have a look.
I have a dude regarding to spark streaming and filestream. If spark
streaming crashs and while spark
Hi.
I'd like to read Avro files using this library
https://github.com/databricks/spark-avro
I need to load several files from a folder, not all files. Is there some
functionality to filter the files to load?
And... Is is possible to know the name of the files loaded from a folder?
My problem