Which layout change are you referring to? The only layout change I know of was done in 2.7, IIRC. We backported that to 2.6 and did not see any adverse effects at that time.
Is datanode using more heap all the time? Or is it running into trouble when generating full block reports? Kihwal On Mon, Oct 5, 2020 at 1:40 PM Wei-Chiu Chuang <weic...@cloudera.com.invalid> wrote: > We experienced this issue on CDH6 and HDP3, so roughly Hadoop 3.0.x and > 3.1.x. > Hermanth experienced the same issue on Hadoop 3.1.1 as well (HDFS-15569 > < > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_HDFS-2D15569&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=b6gUZYewojO-9YMJdyeI_g&m=itpohwgKPN5qoauYyyMxhGSnasaP3LLbbMVezETEenA&s=kgWYVv2utuAyPWBhv0KVH8ZZGJqQBMvUM7dZ8J0jaa8&e= > >) > > On Mon, Oct 5, 2020 at 11:03 AM Igor Dvorzhak <i...@google.com> wrote: > > > What Hadoop 3 version do you use? > > > > On Mon, Oct 5, 2020 at 10:03 AM Wei-Chiu Chuang <weic...@apache.org> > > wrote: > > > >> I have anecdotally learned of multiple data points where during the > >> upgrading from Hadoop 2 to Hadoop 3, DN heap usage increases to the > point > >> where it goes OOM. > >> > >> Don't have much logs for this issue, but I suspect it's caused by the > >> layout change added in Hadoop 2.8.0. > >> > >> Does anyone else observe the same issue and how do you mitigate this? > For > >> now we suggested increasing DN heap size prior to upgrade as part of > >> pre-upgrade checklist. > >> > >> Thanks, > >> Wei-Chiu > >> > > >
