Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

Sorry for not being specific. I was referring to HDFS-8791 (block ID-based DN storage layout can be very slow for datanode on ext4) where it is in 2.8 and above. As I understand it, the increased heap usage only occurs during upgrade. No issue afte

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

To be clear, we are running 2.8 and 2.10. Although we don't see any issues, I am curious whether the change in heap usage is amplified on dense datanodes. Kihwal On Tue, Oct 6, 2020 at 5:00 PM Kihwal Lee wrote: > Which layout change are you referring to? The only layout change I know of > was d

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

Which layout change are you referring to? The only layout change I know of was done in 2.7, IIRC. We backported that to 2.6 and did not see any adverse effects at that time. Is datanode using more heap all the time? Or is it running into trouble when generating full block reports? Kihwal On Mon,