On Mon, May 23, 2016 at 9:55 AM, Bryan Beaudreault <bbeaudrea...@hubspot.com > wrote:
> Hey everyone, > > We are noticing a file descriptor leak that is only affecting nodes in our > cluster running 5.7.0, not those still running 5.3.8. Translation: roughly hbase-1.2.0+hadoop-2.6.0 vs hbase-0.98.6+hadoop-2.5.0. > I ran an lsof against > an affected regionserver, and noticed that there were 10k+ unix sockets > that are just called "socket", as well as another 10k+ of the form > "/dev/shm/HadoopShortCircuitShm_DFSClient_NONMAPREDUCE_-<int>_1_<int>". The > 2 seem related based on how closely the counts match. > > We are in the middle of a rolling upgrade from CDH5.3.8 to CDH5.7.0 (we > handled the namenode upgrade separately). The 5.3.8 nodes *do not* > experience this issue. The 5.7.0 nodes *do. *We are holding off upgrading > more regionservers until we can figure this out. I'm not sure if any > intermediate versions between the 2 have the issue. > > We traced the root cause to a hadoop job running against a basic table: > > 'my-table-1', {TABLE_ATTRIBUTES => {MAX_FILESIZE => '107374182400', > MEMSTORE_FLUSHSIZE => '67108864'}, {NAME => '0', VERSIONS => '50', > BLOOMFILTER => 'NONE', COMPRESSION => 'LZO', METADATA => > {'COMPRESSION_COMPACT' => 'LZO', 'ENCODE_ON_DISK' => 'true'}} > > This is very similar to all of our other tables (we have many). You are doing MR against some of these also? They have different schemas? No leaks here? > However, > it's regions are getting up there in size, 40+gb per region, compressed. > This has not been an issue for us previously. > > The hadoop job is a simple TableMapper job with no special parameters, > though we haven't updated our client yet to the latest (will do that once > we finish the server side). The hadoop job runs on a separate hadoop > cluster, remotely accessing the HBase cluster. It does not do any other > reads or writes, outside of the TableMapper scans. > > Moving the regions off of an affected server, or killing the hadoop job, > causes the file descriptors to gradually go back down to normal. > > Any ideas? > > Is it just the FD cache running 'normally'? 10k seems like a lot though. 256 seems to be the default in hdfs but maybe it is different in CM or in hbase? What is your dfs.client.read.shortcircuit.streams.cache.size set to? St.Ack > Thanks, > > Bryan >