Re: [DISCUSS] HBASE-24749

On Wed, Jul 22, 2020 at 17:28 Tak-Lon (Stephen) Wu wrote: > just a dummy question on how we contribute, should we have a feature > branch? or can we stream into the master if we don't break anything ? > I would prefer a feature branch that is regularly rebased onto master. The feature branch help

Re: [DISCUSS] HBASE-24749

Having high-level milestones nail down on the umbrella JIRA HBASE-24749 should be good and should be the next step after we gather all the feedback. just a dummy question on how we contribute, should we have a feature branch? or can we stream into the master if we don't break anything ? -Stephen

Re: [DISCUSS] HBASE-24749

Yeah, that's the struggle with the multiple branches -- we want to see our changes in a version of HBase we're using, but that may not be the right place to land the changes :) Since this is an "opt-in" and you obviously have _something_ working (given the benchmarks), I'd suggest breaking dow

Re: [DISCUSS] HBASE-24749

Thanks Josh, and yeah object store is a bit different lol. the major reason we didn't try to fold that into meta table were that we don't know how well meta table can be scale, e.g. as Stack mentioned about a previous design in HBASE-14090, it matches our initial estimate that these piece of new d

Re: [DISCUSS] HBASE-24749

Oh, and don't forget, you have to update bulk load to work with this approach. Never knew that we had a utility to pick up files that folks wrote directly into the hbase.rootdir (RefreshHFilesClient). I am 110% behind ripping that out. We have bulk loading as the supported path for a reason :)

Re: [DISCUSS] HBASE-24749

Great idea -- big problem on slow storage :) (but I'm sure I'm not telling you anything new). A quick quesiton: with the split-able meta work going on, any reason to not put these files in meta itself (rather than yet-another-system-table)? FWIW, listing the files for each region in the meta

[DISCUSS] HBASE-24749

Hi guys, I'm sending this email to get more comments and thoughts from the dev@list for an open discussion item on HBASE-24749 . mainly we're proposing a feature with a new store engine to skip the use of .tmp directory in the HFile commit stage