> > 1) Is there any difference in terms performance when we use datasets over > dataframes? Is it significant to choose 1 over other. I do realise there > would be some overhead due case classes but how significant is that? Are > there any other implications.
As long as you use the DataFrame functions the performance is going to be the same since they operate directly with Tungsten rows, but as soon as you try to do any typed-operations like `.map` performance is going to be hit because Spark would have to create Java objects from Tungsten memory. 2) Is the Tungsten code generation done only for datasets or is there any > internal process to generate bytecode for dataframes as well? Since its > related to jvm , I think its just for datasets but I couldn’t find anything > that tells it specifically. If its just for datasets , does that mean we > miss out on the project tungsten optimisation for dataframes? Code generation is done for both On Mon, Feb 18, 2019 at 9:09 PM Akhilanand <akhilanand...@gmail.com> wrote: > > Hello, > > I have been recently exploring about dataset and dataframes. I would > really appreciate if someone could answer these questions: > > 1) Is there any difference in terms performance when we use datasets over > dataframes? Is it significant to choose 1 over other. I do realise there > would be some overhead due case classes but how significant is that? Are > there any other implications. > > 2) Is the Tungsten code generation done only for datasets or is there any > internal process to generate bytecode for dataframes as well? Since its > related to jvm , I think its just for datasets but I couldn’t find anything > that tells it specifically. If its just for datasets , does that mean we > miss out on the project tungsten optimisation for dataframes? > > > > Regards, > Akhilanand BV > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > -- Sent from my iPhone
