Hi, Thanks for your answers. Much appreciated
I know that we can cache the data frame in memory or disk but I want to understand when the data frame is loaded initially and where does it reside by default? Thanks, Sid On Wed, Jun 22, 2022 at 6:10 AM Yong Walt <yongw...@gmail.com> wrote: > These are the basic concepts in spark :) > You may take a bit time to read this small book: > https://cloudcache.net/resume/PDDWS2-V2.pdf > > regards > > > On Wed, Jun 22, 2022 at 3:17 AM Sid <flinkbyhe...@gmail.com> wrote: > >> Hi Team, >> >> I have a few doubts about the below questions: >> >> 1) data frame will reside where? memory? disk? memory allocation about >> data frame? >> 2) How do you configure each partition? >> 3) Is there any way to calculate the exact partitions needed to load a >> specific file? >> >> Thanks, >> Sid >> >
