Re: [DISCUSS] FileSystem Quotas in HBase

Thanks, Josh. Looking forward to the patches. On Thu, Nov 3, 2016 at 12:58 PM, Josh Elser wrote: > Done. > > Ted Yu wrote: > >> Josh: >> Please capture the following in design doc. >> >> Thanks >> >> On Wed, Nov 2, 2016 at 3:28 PM, Enis Söztutar wrote:

Re: [DISCUSS] FileSystem Quotas in HBase

Done. Ted Yu wrote: Josh: Please capture the following in design doc. Thanks On Wed, Nov 2, 2016 at 3:28 PM, Enis Söztutar wrote: Thanks Andrew, I forgot to mention that we have considered using the HDFS quota enforcement directly as well, but decided against it for a

Re: [DISCUSS] FileSystem Quotas in HBase

Great points. Thanks fellas! Andrew Purtell wrote: Right, these drawbacks are what I was getting at with "​impose limits on how HBase structures storage on the filesystem". It does imply major changes to filesystem structure and multiple WALs. I didn't think of snapshots, you are right that

Re: [DISCUSS] FileSystem Quotas in HBase

Josh: Please capture the following in design doc. Thanks On Wed, Nov 2, 2016 at 3:28 PM, Enis Söztutar wrote: > Thanks Andrew, > > I forgot to mention that we have considered using the HDFS quota > enforcement directly as well, but decided against it for a couple of >

Re: [DISCUSS] FileSystem Quotas in HBase

Thanks Andrew, I forgot to mention that we have considered using the HDFS quota enforcement directly as well, but decided against it for a couple of reasons. - Our current layout has files in the data directory, as well as archive directory and WALs, etc. Since there is no option for HDFS quotas

Re: [DISCUSS] FileSystem Quotas in HBase

Another approach to hard limits could be pushing the quota down to the HDFS level, because HDFS would have a very accurate assessment of quota utilization at all times, but this would only work with HDFS and impose limits on how HBase structures storage on the filesystem (e.g. all files for a

Re: [DISCUSS] FileSystem Quotas in HBase

Thanks Josh for the doc and pursuing this. I was involved with some of the design choices so consider me a +1 on the general approach. One topic which is not covered here is that the other design decision that we could have pursued is a more strict control on the quota usage so that we would

Re: [DISCUSS] FileSystem Quotas in HBase

Thanks for the reviews so far, Ted and Stack. The comments were great and much appreciated. Interpreting consensus from lack of objection, I'm going to move ahead in earnest starting to work on what was described in the doc. Expect to see some work break-out happening under HBASE-16961 and

Re: [DISCUSS] FileSystem Quotas in HBase

Sure thing, Ted. https://docs.google.com/document/d/1VtLWDkB2tpwc_zgCNPE1ulZOeecF-YA2FYSK3TSs_bw/edit?usp=sharing Let me open an umbrella issue for now. I can break up the work later. https://issues.apache.org/jira/browse/HBASE-16961 Ted Yu wrote: Josh: Can you put the doc in google doc so

Re: [DISCUSS] FileSystem Quotas in HBase

Josh: Can you put the doc in google doc so that people can comment on it ? Is there a JIRA opened for this work ? Please open one if there is none. Thanks On Fri, Oct 28, 2016 at 9:00 AM, Josh Elser wrote: > Hi folks, > > I'd like to propose the introduction of FileSystem

[DISCUSS] FileSystem Quotas in HBase

Hi folks, I'd like to propose the introduction of FileSystem quotas to HBase. Here's a design doc[1] available which (hopefully) covers all of the salient points of what I think an initial version of such a feature would include. tl;dr We can define quotas on tables and namespaces. Region