To be clear, we are running 2.8 and 2.10. Although we don't see any issues, I am curious whether the change in heap usage is amplified on dense datanodes.
Kihwal On Tue, Oct 6, 2020 at 5:00 PM Kihwal Lee <kih...@verizonmedia.com> wrote: > Which layout change are you referring to? The only layout change I know of > was done in 2.7, IIRC. We backported that to 2.6 and did not see any > adverse effects at that time. > > Is datanode using more heap all the time? Or is it running into trouble > when generating full block reports? > > Kihwal > > On Mon, Oct 5, 2020 at 1:40 PM Wei-Chiu Chuang > <weic...@cloudera.com.invalid> wrote: > >> We experienced this issue on CDH6 and HDP3, so roughly Hadoop 3.0.x and >> 3.1.x. >> Hermanth experienced the same issue on Hadoop 3.1.1 as well (HDFS-15569 >> < >> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_HDFS-2D15569&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=b6gUZYewojO-9YMJdyeI_g&m=itpohwgKPN5qoauYyyMxhGSnasaP3LLbbMVezETEenA&s=kgWYVv2utuAyPWBhv0KVH8ZZGJqQBMvUM7dZ8J0jaa8&e= >> >) >> >> On Mon, Oct 5, 2020 at 11:03 AM Igor Dvorzhak <i...@google.com> wrote: >> >> > What Hadoop 3 version do you use? >> > >> > On Mon, Oct 5, 2020 at 10:03 AM Wei-Chiu Chuang <weic...@apache.org> >> > wrote: >> > >> >> I have anecdotally learned of multiple data points where during the >> >> upgrading from Hadoop 2 to Hadoop 3, DN heap usage increases to the >> point >> >> where it goes OOM. >> >> >> >> Don't have much logs for this issue, but I suspect it's caused by the >> >> layout change added in Hadoop 2.8.0. >> >> >> >> Does anyone else observe the same issue and how do you mitigate this? >> For >> >> now we suggested increasing DN heap size prior to upgrade as part of >> >> pre-upgrade checklist. >> >> >> >> Thanks, >> >> Wei-Chiu >> >> >> > >> >
