Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

Thank you Arpit Agarwal for notifying me on this user mail. Yes, heap pressure is introduced due to block-layout migration. Right, high heap usage is only during the upgrade, and once the upgrade is done then heap usage back to normal. Have experienced this issue from many clusters(20+) but only

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

We haven't experienced anything like that up to 2.8. We are still in the process of stabilizing 2.10 as we upgrade some of the bigger clusters. We will know soon how 2.10 datanodes behave under heavy load and storage utilization. If you are seeing a significant change, it might be something post-2

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

Sorry for not being specific. I was referring to HDFS-8791 (block ID-based DN storage layout can be very slow for datanode on ext4) where it is in 2.8 and above. As I understand it, the increased heap usage only occurs during upgrade. No issue afte

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

To be clear, we are running 2.8 and 2.10. Although we don't see any issues, I am curious whether the change in heap usage is amplified on dense datanodes. Kihwal On Tue, Oct 6, 2020 at 5:00 PM Kihwal Lee wrote: > Which layout change are you referring to? The only layout change I know of > was d

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

Which layout change are you referring to? The only layout change I know of was done in 2.7, IIRC. We backported that to 2.6 and did not see any adverse effects at that time. Is datanode using more heap all the time? Or is it running into trouble when generating full block reports? Kihwal On Mon,

Re: Increased DN heap usage during Hadoop 3 upgrade

We experienced this issue on CDH6 and HDP3, so roughly Hadoop 3.0.x and 3.1.x. Hermanth experienced the same issue on Hadoop 3.1.1 as well (HDFS-15569 ) On Mon, Oct 5, 2020 at 11:03 AM Igor Dvorzhak wrote: > What Hadoop 3 version do you use? > >

Re: Increased DN heap usage during Hadoop 3 upgrade

What Hadoop 3 version do you use? On Mon, Oct 5, 2020 at 10:03 AM Wei-Chiu Chuang wrote: > I have anecdotally learned of multiple data points where during the > upgrading from Hadoop 2 to Hadoop 3, DN heap usage increases to the point > where it goes OOM. > > Don't have much logs for this issue,

Increased DN heap usage during Hadoop 3 upgrade

I have anecdotally learned of multiple data points where during the upgrading from Hadoop 2 to Hadoop 3, DN heap usage increases to the point where it goes OOM. Don't have much logs for this issue, but I suspect it's caused by the layout change added in Hadoop 2.8.0. Does anyone else observe the