Hey everyone,
We are noticing a file descriptor leak that is only affecting nodes in our
cluster running 5.7.0, not those still running 5.3.8. I ran an lsof against
an affected regionserver, and noticed that there were 10k+ unix sockets
that are just called "socket", as well as another 10k+ of the form
"/dev/shm/HadoopShortCircuitShm_DFSClient_NONMAPREDUCE_-<int>_1_<int>". The
2 seem related based on how closely the counts match.
We are in the middle of a rolling upgrade from CDH5.3.8 to CDH5.7.0 (we
handled the namenode upgrade separately). The 5.3.8 nodes *do not*
experience this issue. The 5.7.0 nodes *do. *We are holding off upgrading
more regionservers until we can figure this out. I'm not sure if any
intermediate versions between the 2 have the issue.
We traced the root cause to a hadoop job running against a basic table:
'my-table-1', {TABLE_ATTRIBUTES => {MAX_FILESIZE => '107374182400',
MEMSTORE_FLUSHSIZE => '67108864'}, {NAME => '0', VERSIONS => '50',
BLOOMFILTER => 'NONE', COMPRESSION => 'LZO', METADATA =>
{'COMPRESSION_COMPACT' => 'LZO', 'ENCODE_ON_DISK' => 'true'}}
This is very similar to all of our other tables (we have many). However,
it's regions are getting up there in size, 40+gb per region, compressed.
This has not been an issue for us previously.
The hadoop job is a simple TableMapper job with no special parameters,
though we haven't updated our client yet to the latest (will do that once
we finish the server side). The hadoop job runs on a separate hadoop
cluster, remotely accessing the HBase cluster. It does not do any other
reads or writes, outside of the TableMapper scans.
Moving the regions off of an affected server, or killing the hadoop job,
causes the file descriptors to gradually go back down to normal.
Any ideas?
Thanks,
Bryan
