Re: Disk space leak when using HBase and HDFS ShortCircuit

2014-06-27 Thread Andrew Purtell
Giuseppe mentioned a contributing factor is HDP ups the cache size from 256 to 4096. On Fri, Jun 27, 2014 at 2:31 PM, lars hofhansl wrote: > Sounds scary! > HDP 2.1 ships with Hadoop 2.4.0, right? > > > Sounds like a serious HDFS bug with SSR. > Maybe we need to at least document this behavior

Re: Disk space leak when using HBase and HDFS ShortCircuit

2014-06-27 Thread Ted Yu
FYI: HDFS-6604 has been opened for this bug. On Fri, Jun 27, 2014 at 2:31 PM, lars hofhansl wrote: > Sounds scary! > HDP 2.1 ships with Hadoop 2.4.0, right? > > > Sounds like a serious HDFS bug with SSR. > Maybe we need to at least document this behavior and recommend settings in > the HBase bo

Re: Disk space leak when using HBase and HDFS ShortCircuit

2014-06-27 Thread lars hofhansl
Sounds scary! HDP 2.1 ships with Hadoop 2.4.0, right? Sounds like a serious HDFS bug with SSR. Maybe we need to at least document this behavior and recommend settings in the HBase book. -- Lars - Original Message - From: Giuseppe Reina To: user@hbase.apache.org Cc: Sent: Wednesday,

Re: Disk space leak when using HBase and HDFS ShortCircuit

2014-06-26 Thread Giuseppe Reina
r be accessed, and if it will this could potentially lead to inconsistent states. Giuseppe > > Enis > > > On Wed, Jun 25, 2014 at 3:29 PM, Andrew Purtell > wrote: > > > Forwarded > > > > -- Forwarded message ------ > > From: Vladimir Ro

Re: Disk space leak when using HBase and HDFS ShortCircuit

2014-06-25 Thread Enis Söztutar
block is deleted. What was your dfs.client.read.shortcircuit.streams.cache.size ? Enis On Wed, Jun 25, 2014 at 3:29 PM, Andrew Purtell wrote: > Forwarded > > -- Forwarded message -- > From: Vladimir Rodionov > Date: Wed, Jun 25, 2014 at 12:03 PM > Subject: R

RE: Disk space leak when using HBase and HDFS ShortCircuit

2014-06-25 Thread Vladimir Rodionov
>> Apparently those file descriptors were stored by the HDFS >> ShortCircuit cache. As far as I understand this is issue of HDFS shorty-circuit-reads implementation not HBase. HBase uses HDFS API to access files. Did you ask this question on hdfs dev list? This looks like a very serious bug. Be