We experienced this issue on CDH6 and HDP3, so roughly Hadoop 3.0.x and 3.1.x. Hermanth experienced the same issue on Hadoop 3.1.1 as well (HDFS-15569 <https://issues.apache.org/jira/browse/HDFS-15569>)
On Mon, Oct 5, 2020 at 11:03 AM Igor Dvorzhak <i...@google.com> wrote: > What Hadoop 3 version do you use? > > On Mon, Oct 5, 2020 at 10:03 AM Wei-Chiu Chuang <weic...@apache.org> > wrote: > >> I have anecdotally learned of multiple data points where during the >> upgrading from Hadoop 2 to Hadoop 3, DN heap usage increases to the point >> where it goes OOM. >> >> Don't have much logs for this issue, but I suspect it's caused by the >> layout change added in Hadoop 2.8.0. >> >> Does anyone else observe the same issue and how do you mitigate this? For >> now we suggested increasing DN heap size prior to upgrade as part of >> pre-upgrade checklist. >> >> Thanks, >> Wei-Chiu >> >
