There are more details in this jira: https://issues.apache.org/jira/browse/HADOOP-16452
Denser DataNodes are common. It is not uncommon to find a DataNode with > 7 > million blocks these days. > With such a high number of blocks, the block report message can exceed the > 64mb limit (defined by ipc.maximum.data.length). The block reports are > rejected, causing missing blocks in HDFS. We had to double this > configuration value in order to work around the issue. On Fri, Nov 8, 2019 at 1:48 AM Carey, Paul <paul.ca...@sc.com.invalid> wrote: > Hi > > > > The NameNode logs in my HDFS instance recently started logging warnings of > the form `Requested data length 145530837 is longer than maximum configured > RPC length 144217728`. > > > > This ultimately manifested itself as the NameNode declaring thousands of > blocks to be missing and 19 files to be corrupt. > > > > The situation was resolved by updating `ipc.maximum.data.length` to a > value greater than the requested data length listed above. This is not a > satisfying resolution though. I'd like to understand how this issue > occurred. > > > > I've run `hdfs fsck -files -blocks -locations` and the largest block is of > length `1342177728`. > > > > - Is there some overhead for RPC calls? Could a block of length > `1342177728` be resulting in the original warning log at the top of this > post? > > - My understanding is that the only way a client writing to HDFS can > specify a block size is via either `-Ddfs.blocksize` or setting the > corresponding property on the `Configuration` object when initialising the > HDFS connection. Is this correct, or are there any other routes to creating > excessively large blocks? > > - Other than overly large blocks, are there any other issues that could > trigger the warning above? > > > > Many thanks > > > > Paul > > This email and any attachments are confidential and may also be > privileged. If you are not the intended recipient, please delete all copies > and notify the sender immediately. You may wish to refer to the > incorporation details of Standard Chartered PLC, Standard Chartered Bank > and their subsidiaries at https://www.sc.com/en/our-locations. Please > refer to https://www.sc.com/en/privacy-policy/ for Standard Chartered > Bank’s Privacy Policy. >
