Yes it is operating on the sorted column 2016-07-07 11:43 GMT+02:00 Ted Yu <yuzhih...@gmail.com>:
> Does the filter under consideration operate on sorted column(s) ? > > Cheers > > > On Jul 7, 2016, at 2:25 AM, tan shai <tan.shai...@gmail.com> wrote: > > > > Hi, > > > > I have a sorted dataframe, I need to optimize the filter operations. > > How does Spark performs filter operations on sorted dataframe? > > > > It is scanning all the data? > > > > Many thanks. >
