Hi Ayan,
If you have multiple files (example 12 files )and you are using following
code then you will get 12 partition.
r = sc.textFile("file://my/file/*")
Not sure what you want to know about file system ,please check API doc.
Regards,
Vaquar khan
On Jun 8, 2017 10:44 AM, "ayan guha" <[email protected]> wrote:
Any one?
On Thu, 8 Jun 2017 at 3:26 pm, ayan guha <[email protected]> wrote:
> Hi Guys
>
> Quick one: How spark deals (ie create partitions) with large files sitting
> on NFS, assuming the all executors can see the file exactly same way.
>
> ie, when I run
>
> r = sc.textFile("file://my/file")
>
> what happens if the file is on NFS?
>
> is there any difference from
>
> r = sc.textFile("hdfs://my/file")
>
> Are the input formats used same in both cases?
>
>
> --
> Best Regards,
> Ayan Guha
>
--
Best Regards,
Ayan Guha
