Hi My question is what happens if I have 1 file of say 100gb. Then how many partitions will be there?
Best Ayan On Sun, 11 Jun 2017 at 9:36 am, vaquar khan <[email protected]> wrote: > Hi Ayan, > > If you have multiple files (example 12 files )and you are using following > code then you will get 12 partition. > > r = sc.textFile("file://my/file/*") > > Not sure what you want to know about file system ,please check API doc. > > > Regards, > Vaquar khan > > > On Jun 8, 2017 10:44 AM, "ayan guha" <[email protected]> wrote: > > Any one? > > On Thu, 8 Jun 2017 at 3:26 pm, ayan guha <[email protected]> wrote: > >> Hi Guys >> >> Quick one: How spark deals (ie create partitions) with large files >> sitting on NFS, assuming the all executors can see the file exactly same >> way. >> >> ie, when I run >> >> r = sc.textFile("file://my/file") >> >> what happens if the file is on NFS? >> >> is there any difference from >> >> r = sc.textFile("hdfs://my/file") >> >> Are the input formats used same in both cases? >> >> >> -- >> Best Regards, >> Ayan Guha >> > -- > Best Regards, > Ayan Guha > > > -- Best Regards, Ayan Guha
