Removing the user facing config seems like a good idea from the standpoint of reducing cognitive load, and documentation
On Fri, Jan 4, 2019 at 7:03 AM Sean Owen <sro...@apache.org> wrote: > OK, maybe leave in tungsten for 3.0. > I did a quick check, and removing StaticMemoryManager saves a few hundred > lines. It's used in MemoryStore tests internally though, and not a trivial > change to remove it. It's also used directly in HashedRelation. It could > still be worth removing it as a user-facing option to reduce confusion > about memory tuning, but it wouldn't take out much code. What do you all > think? > > On Thu, Jan 3, 2019 at 9:41 PM Reynold Xin <r...@databricks.com> wrote: > >> The issue with the offheap mode is it is a pretty big behavior change and >> does require additional setup (also for users that run with UDFs that >> allocate a lot of heap memory, it might not be as good). >> >> I can see us removing the legacy mode since it's been legacy for a long >> time and perhaps very few users need it. How much code does it remove >> though? >> >> >> On Thu, Jan 03, 2019 at 2:55 PM, Sean Owen <sro...@apache.org> wrote: >> >>> Just wondering if there is a good reason to keep around the pre-tungsten >>> on-heap memory mode for Spark 3, and make spark.memory.offHeap.enabled >>> always true? It would simplify the code somewhat, but I don't feel I'm so >>> aware of the tradeoffs. >>> >>> I know we didn't deprecate it, but it's been off by default for a long >>> time. It could be deprecated, too. >>> >>> Same question for spark.memory.useLegacyMode and all its various >>> associated settings? Seems like these should go away at some point, and >>> Spark 3 is a good point. Same issue about deprecation though. >>> >>> --------------------------------------------------------------------- To >>> unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >> >>
