You probably don't want swapping in any environment. Some tasks will grind to a halt under mem pressure rather than just fail quickly. You would want to simply provision more memory.
On Tue, Feb 16, 2021, 7:57 AM Jahar Tyagi <jaharty...@gmail.com> wrote: > Hi, > > We have recently migrated from Spark 2.4.4 to Spark 3.0.1 and using Spark > in virtual machine/bare metal as standalone deployment and as kubernetes > deployment as well. > > There is a kernel parameter named as 'vm.swappiness' and we keep its value > as '1' in standard deployment. Now since we are moving to kubernetes and on > kubernetes worker nodes the value of this parameter is '60'. > > Now my question is if it is OK to keep such a high value of > 'vm.swappiness'=60 in kubernetes environment for Spark workloads. > > Will such high value of this kernel parameter have performance impact on > Spark PODs? > As per below link from cloudera, they suggest not to set such a high > value. > > > https://docs.cloudera.com/cloudera-manager/7.2.6/managing-clusters/topics/cm-setting-vmswappiness-linux-kernel-parameter.html > > Any thoughts/suggestions on this are highly appreciated. > > Regards > Jahar Tyagi > >
