[jira] [Commented] (HBASE-8701) distributedLogReplay need to apply wal edits in the receiving order of those edits
[
https://issues.apache.org/jira/browse/HBASE-8701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13690072#comment-13690072
]
Jeffrey Zhong commented on HBASE-8701:
--
{quote}
But some can be -ve? So they will be out of order?
{quote}
the negative mvcc is only used in KV comparison to resolve conflicts when row +
timestamp is same. The hfile sequence number used in flush is from current
region server log sequence number not the mvcc value. KVs of the same
key+timestamp can be out of order in multiple store files while the right KV is
selected with the help of these negative mvcc values(origin log sequence
number). (You can refer to KVScannerComparator#compare)
{quote}
Is that safe presumption to make in replay?
Is this the least sequenceid of the batch?
Do we have to add it to WALEdit at all?
{quote}
I checked the wal serialization code, kv isn't PBed and we don't write mvcc
values into wal. Therefore, I still need to add the "original log sequence
number" into receiving RS hlogkey of a wal entry to maintain the original order
of changes when re-replaying these edits for a RS chain failure scenario.
During replay, we can use the last sequence id of a batch because a batch
request is from a single wal and the relative order of changes is maintained
when constructing the batch request. So the last KV wins if there are multiple
KVs of the same key + timestamp.
I don't have to put the "original seq number" into WALEdit while need to change
several method signatures to pass the info down.
> distributedLogReplay need to apply wal edits in the receiving order of those
> edits
> --
>
> Key: HBASE-8701
> URL: https://issues.apache.org/jira/browse/HBASE-8701
> Project: HBase
> Issue Type: Bug
> Components: MTTR
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Fix For: 0.98.0, 0.95.2
>
> Attachments: 8701-v3.txt, hbase-8701-v4.patch, hbase-8701-v5.patch,
> hbase-8701-v6.patch, hbase-8701-v7.patch
>
>
> This issue happens in distributedLogReplay mode when recovering multiple puts
> of the same key + version(timestamp). After replay, the value is
> nondeterministic of the key
> h5. The original concern situation raised from [~eclark]:
> For all edits the rowkey is the same.
> There's a log with: [ A (ts = 0), B (ts = 0) ]
> Replay the first half of the log.
> A user puts in C (ts = 0)
> Memstore has to flush
> A new Hfile will be created with [ C, A ] and MaxSequenceId = C's seqid.
> Replay the rest of the Log.
> Flush
> The issue will happen in similar situation like Put(key, t=T) in WAL1 and
> Put(key,t=T) in WAL2
> h5. Below is the option(proposed by Ted) I'd like to use:
> a) During replay, we pass original wal sequence number of each edit to the
> receiving RS
> b) In receiving RS, we store negative original sequence number of wal edits
> into mvcc field of KVs of wal edits
> c) Add handling of negative MVCC in KVScannerComparator and KVComparator
> d) In receiving RS, write original sequence number into an optional field of
> wal file for chained RS failure situation
> e) When opening a region, we add a safety bumper(a large number) in order for
> the new sequence number of a newly opened region not to collide with old
> sequence numbers.
> In the future, when we stores sequence number along with KVs, we can adjust
> the above solution a little bit by avoiding to overload MVCC field.
> h5. The other alternative options are listed below for references:
> Option one
> a) disallow writes during recovery
> b) during replay, we pass original wal sequence ids
> c) hold flush till all wals of a recovering region are replayed. Memstore
> should hold because we only recover unflushed wal edits. For edits with same
> key + version, whichever with larger sequence Id wins.
> Option two
> a) During replay, we pass original wal sequence ids
> b) for each wal edit, we store each edit's original sequence id along with
> its key.
> c) during scanning, we use the original sequence id if it's present otherwise
> its store file sequence Id
> d) compaction can just leave put with max sequence id
> Please let me know if you have better ideas.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8779) add mutateRow method support to thrift2
[ https://issues.apache.org/jira/browse/HBASE-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13690048#comment-13690048 ] duanhongyi commented on HBASE-8779: --- patch hbase.thrift thrift2-HBASE-8779.patch patch ThriftHBaseServiceHandler.java thrift-ThriftHBaseServiceHandler.java-HBASE-8779.patch patch ThriftUtilities.java thrift-ThriftUtilities.java-HBASE-8779.patch > add mutateRow method support to thrift2 > --- > > Key: HBASE-8779 > URL: https://issues.apache.org/jira/browse/HBASE-8779 > Project: HBase > Issue Type: New Feature > Components: Thrift >Reporter: duanhongyi > Attachments: thrift2-HBASE-8779.patch, > thrift-ThriftHBaseServiceHandler.java-HBASE-8779.patch, > thrift-ThriftUtilities.java-HBASE-8779.patch > > > examples py: > TPut put = TPut() > m1 = TMutation(put=put) > TDelete single_delete = TDelete() > m2 = TMutation(singleDelete=single_delete) > row_mu = TRowMutations(row,[m1,m2]) > thbase_service.mutateRow(table,row_mu) # atomically on a single row -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7967) implement compactor for stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13690035#comment-13690035 ] stack commented on HBASE-7967: -- [~sershe] So this patch makes i/o worse, 2x worse? Any circumstance where it makes for less i/o? If we skip the L0, this will lessen write amplification? > implement compactor for stripe compactions > -- > > Key: HBASE-7967 > URL: https://issues.apache.org/jira/browse/HBASE-7967 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7967-latest-with-dependencies.patch, > HBASE-7967-latest-with-dependencies.patch, > HBASE-7967-latest-with-dependencies.patch, > HBASE-7967-latest-with-dependencies.patch, HBASE-7967-v0.patch, > HBASE-7967-v1.patch, HBASE-7967-v2.patch, HBASE-7967-v3.patch, > HBASE-7967-v4.patch, HBASE-7967-v5.patch, HBASE-7967-v6.patch, > HBASE-7967-v7.patch, HBASE-7967-v7.patch, HBASE-7967-v7.patch, > HBASE-7967-v8.patch > > > Compactor needs to be implemented. See details in parent and blocking jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13690025#comment-13690025 ] stack commented on HBASE-8015: -- bq. One concern that I see internally is that all our users will have to update their code to use the new apis so we can migrate them to new NS. You mean Y! They are using '.' for separator currently? bq. As opposed to having just a table name then it's just a simple config change and restart. No override of APIs? We have a special KV comparator for .META. table. It leverages the ',' position as delimiter between table name and startcode in region name. I suppose we could change the KV comparator so it looks for ','. I thought you had trick to make '.' work? > Support for Namespaces > -- > > Key: HBASE-8015 > URL: https://issues.apache.org/jira/browse/HBASE-8015 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8721) Deletes can mask puts that happen after the delete
[ https://issues.apache.org/jira/browse/HBASE-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13690018#comment-13690018 ] stack commented on HBASE-8721: -- [~andrew.purt...@gmail.com] Mind if I reopen? The lads are putting up a spirited fight and the args are quality? > Deletes can mask puts that happen after the delete > -- > > Key: HBASE-8721 > URL: https://issues.apache.org/jira/browse/HBASE-8721 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Feng Honghua > Attachments: HBASE-8721-0.94-V0.patch > > > this fix aims for bug mentioned in http://hbase.apache.org/book.html 5.8.2.1: > "Deletes mask puts, even puts that happened after the delete was entered. > Remember that a delete writes a tombstone, which only disappears after then > next major compaction has run. Suppose you do a delete of everything <= T. > After this you do a new put with a timestamp <= T. This put, even if it > happened after the delete, will be masked by the delete tombstone. Performing > the put will not fail, but when you do a get you will notice the put did have > no effect. It will start working again after the major compaction has run. > These issues should not be a problem if you use always-increasing versions > for new puts to a row. But they can occur even if you do not care about time: > just do delete and put immediately after each other, and there is some chance > they happen within the same millisecond." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7667) Support stripe compaction
[ https://issues.apache.org/jira/browse/HBASE-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13690015#comment-13690015 ] stack commented on HBASE-7667: -- bq. splitting uses more I/O than not splitting Sorry. You mean stripes uses more i/o because we L0 first then rewrite into stripes? > Support stripe compaction > - > > Key: HBASE-7667 > URL: https://issues.apache.org/jira/browse/HBASE-7667 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: stripe-cdf.pdf, Stripe compaction perf evaluation.pdf, > Stripe compaction perf evaluation.pdf, Stripe compaction perf evaluation.pdf, > Stripe compactions.pdf, Stripe compactions.pdf, Stripe compactions.pdf, > Stripe compactions.pdf, Using stripe compactions.pdf, Using stripe > compactions.pdf, Using stripe compactions.pdf > > > So I was thinking about having many regions as the way to make compactions > more manageable, and writing the level db doc about how level db range > overlap and data mixing breaks seqNum sorting, and discussing it with Jimmy, > Matteo and Ted, and thinking about how to avoid Level DB I/O multiplication > factor. > And I suggest the following idea, let's call it stripe compactions. It's a > mix between level db ideas and having many small regions. > It allows us to have a subset of benefits of many regions (wrt reads and > compactions) without many of the drawbacks (managing and current > memstore/etc. limitation). > It also doesn't break seqNum-based file sorting for any one key. > It works like this. > The region key space is separated into configurable number of fixed-boundary > stripes (determined the first time we stripe the data, see below). > All the data from memstores is written to normal files with all keys present > (not striped), similar to L0 in LevelDb, or current files. > Compaction policy does 3 types of compactions. > First is L0 compaction, which takes all L0 files and breaks them down by > stripe. It may be optimized by adding more small files from different > stripes, but the main logical outcome is that there are no more L0 files and > all data is striped. > Second is exactly similar to current compaction, but compacting one single > stripe. In future, nothing prevents us from applying compaction rules and > compacting part of the stripe (e.g. similar to current policy with rations > and stuff, tiers, whatever), but for the first cut I'd argue let it "major > compact" the entire stripe. Or just have the ratio and no more complexity. > Finally, the third addresses the concern of the fixed boundaries causing > stripes to be very unbalanced. > It's exactly like the 2nd, except it takes 2+ adjacent stripes and writes the > results out with different boundaries. > There's a tradeoff here - if we always take 2 adjacent stripes, compactions > will be smaller but rebalancing will take ridiculous amount of I/O. > If we take many stripes we are essentially getting into the > epic-major-compaction problem again. Some heuristics will have to be in place. > In general, if, before stripes are determined, we initially let L0 grow > before determining the stripes, we will get better boundaries. > Also, unless unbalancing is really large we don't need to rebalance really. > Obviously this scheme (as well as level) is not applicable for all scenarios, > e.g. if timestamp is your key it completely falls apart. > The end result: > - many small compactions that can be spread out in time. > - reads still read from a small number of files (one stripe + L0). > - region splits become marvelously simple (if we could move files between > regions, no references would be needed). > Main advantage over Level (for HBase) is that default store can still open > the files and get correct results - there are no range overlap shenanigans. > It also needs no metadata, although we may record some for convenience. > It also would appear to not cause as much I/O. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7857) Sub-region management
[ https://issues.apache.org/jira/browse/HBASE-7857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-7857. -- Resolution: Incomplete Resolving speculative issue pulled from comments made in other issues w/ no fit criteria for judging when this issue is done. > Sub-region management > - > > Key: HBASE-7857 > URL: https://issues.apache.org/jira/browse/HBASE-7857 > Project: HBase > Issue Type: New Feature >Reporter: Ted Yu > > Sub-region provides a light weight management below region level. > Matt Corgan has a nice summary of the relationship between region size and > number of regions on region server: > https://issues.apache.org/jira/browse/HBASE-7667?focusedCommentId=13575024&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13575024 > HBASE-7667 proposed stripe compaction. However, to fully achieve fine-grained > management, more components should participate: > * memstore flush should ideally have knowledge about what makes stripe > compaction work efficiently > * we need to figure out where to store sub-region boundary information so > that components have easy access. Boundary information should sustain after > region moves from one server to another. > Since HBASE-7667 focuses on compaction aspect, this JIRA discusses sub-region > management in other components so that we better understand the benefits and > complexities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2678) version the REST interface
[
https://issues.apache.org/jira/browse/HBASE-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13690002#comment-13690002
]
Andrew Purtell commented on HBASE-2678:
---
Well, /v1/... or /v2/... Those are valid table names, so there's a backwards
compatibility complication. Changing the MIME types as described is
unambiguous.
Anyway, if you have a patch please consider putting it up so we can take a look
at it.
> version the REST interface
> --
>
> Key: HBASE-2678
> URL: https://issues.apache.org/jira/browse/HBASE-2678
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>
> As mentioned in HBASE-2563, we should deprecate all uses of _cell_ in the API
> and replace with new APIs that use _value_ instead. So we need a way to
> version the REST interface, to provide an updated API while maintaining
> access to the deprecated one until the next major revision. However,
> something like this I consider wrong:
> {{/path/to/v1/resource}}
> and also:
> {{/v2/path/to/resource}}
> because the resource is the same regardless of the representation change.
> REST makes a distinction between the representation and the resource using
> media types.
> Currently Stargate supports the following encodings:
> * {{text/plain}} (in some cases)
> * binary: {{application/octet-stream}} (freeform, no schema change needed)
> * XML: {{text/xml}}
> * JSON: {{application/json}}
> * protobufs: {{application/x-protobuf}}
> We can add Avro encoding support in HBASE-2557 with the new representation as
> {{application/x-avro-binary}} immediately. For XML, JSON, and protobuf
> encoding, we can support new representations using the following new media
> types in the current version:
> * XML: {{application/vnd.hbase+xml}}
> * JSON: {{application/vnd.hbase+json}}
> * protobufs: {{application/vnd.hbase+protobuf}}
> * and for sake of consistency: {{application/vnd.hbase+avro}}
> and then in the next major version recognize both MIME types but return the
> same (newer) representation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8779) add mutateRow method support to thrift2
[ https://issues.apache.org/jira/browse/HBASE-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] duanhongyi updated HBASE-8779: -- Fix Version/s: (was: 0.94.7) > add mutateRow method support to thrift2 > --- > > Key: HBASE-8779 > URL: https://issues.apache.org/jira/browse/HBASE-8779 > Project: HBase > Issue Type: New Feature > Components: Thrift >Reporter: duanhongyi > Attachments: thrift2-HBASE-8779.patch, > thrift-ThriftHBaseServiceHandler.java-HBASE-8779.patch, > thrift-ThriftUtilities.java-HBASE-8779.patch > > > examples py: > TPut put = TPut() > m1 = TMutation(put=put) > TDelete single_delete = TDelete() > m2 = TMutation(singleDelete=single_delete) > row_mu = TRowMutations(row,[m1,m2]) > thbase_service.mutateRow(table,row_mu) # atomically on a single row -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8669) change exploring compaction policy to prefer smaller compactions on blocked stores
[ https://issues.apache.org/jira/browse/HBASE-8669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689996#comment-13689996 ] Hudson commented on HBASE-8669: --- Integrated in HBase-TRUNK #4193 (See [https://builds.apache.org/job/HBase-TRUNK/4193/]) HBASE-8669 change exploring compaction policy to prefer smaller compactions on blocked stores (Revision 1495261) Result = FAILURE sershe : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java > change exploring compaction policy to prefer smaller compactions on blocked > stores > -- > > Key: HBASE-8669 > URL: https://issues.apache.org/jira/browse/HBASE-8669 > Project: HBase > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.95.2 > > Attachments: HBASE-8669-v0.patch > > > Side-note from HBASE-8665 discussion. When we compact a blocked store, we > might want to use a different heuristic to choose between the options. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689993#comment-13689993 ] Francis Liu commented on HBASE-8015: ZK restrictions: http://zookeeper.apache.org/doc/r3.4.3/zookeeperProgrammers.html#ch_zkDataModel If only they printed the chars instead of code points. > Support for Namespaces > -- > > Key: HBASE-8015 > URL: https://issues.apache.org/jira/browse/HBASE-8015 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689989#comment-13689989 ] Francis Liu commented on HBASE-8015: We can broaden the range if we encode the delimiter when stored on fs and zk. We are mainly limited by region and table names. > Support for Namespaces > -- > > Key: HBASE-8015 > URL: https://issues.apache.org/jira/browse/HBASE-8015 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8669) change exploring compaction policy to prefer smaller compactions on blocked stores
[ https://issues.apache.org/jira/browse/HBASE-8669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689987#comment-13689987 ] Hudson commented on HBASE-8669: --- Integrated in hbase-0.95 #256 (See [https://builds.apache.org/job/hbase-0.95/256/]) HBASE-8669 change exploring compaction policy to prefer smaller compactions on blocked stores (Revision 1495262) Result = SUCCESS sershe : Files : * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java > change exploring compaction policy to prefer smaller compactions on blocked > stores > -- > > Key: HBASE-8669 > URL: https://issues.apache.org/jira/browse/HBASE-8669 > Project: HBase > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.95.2 > > Attachments: HBASE-8669-v0.patch > > > Side-note from HBASE-8665 discussion. When we compact a blocked store, we > might want to use a different heuristic to choose between the options. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689986#comment-13689986 ] Enis Soztutar commented on HBASE-8015: -- +thrift as well. That is why I am in favor of option 2 now. bq. ',' looks nice but you mentioned it's already used any way we can update that cleanly to take this into account? I think we can use this. We are relying on it for partitioning region names, and in META. So it should be compatible with that framework. > Support for Namespaces > -- > > Key: HBASE-8015 > URL: https://issues.apache.org/jira/browse/HBASE-8015 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2678) version the REST interface
[
https://issues.apache.org/jira/browse/HBASE-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689981#comment-13689981
]
Francis Liu commented on HBASE-2678:
Reviving this jira. Can we just add version to the path? ie v1/ v2/? This is
the standard we use, oozie uses the same scheme as well.
> version the REST interface
> --
>
> Key: HBASE-2678
> URL: https://issues.apache.org/jira/browse/HBASE-2678
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>
> As mentioned in HBASE-2563, we should deprecate all uses of _cell_ in the API
> and replace with new APIs that use _value_ instead. So we need a way to
> version the REST interface, to provide an updated API while maintaining
> access to the deprecated one until the next major revision. However,
> something like this I consider wrong:
> {{/path/to/v1/resource}}
> and also:
> {{/v2/path/to/resource}}
> because the resource is the same regardless of the representation change.
> REST makes a distinction between the representation and the resource using
> media types.
> Currently Stargate supports the following encodings:
> * {{text/plain}} (in some cases)
> * binary: {{application/octet-stream}} (freeform, no schema change needed)
> * XML: {{text/xml}}
> * JSON: {{application/json}}
> * protobufs: {{application/x-protobuf}}
> We can add Avro encoding support in HBASE-2557 with the new representation as
> {{application/x-avro-binary}} immediately. For XML, JSON, and protobuf
> encoding, we can support new representations using the following new media
> types in the current version:
> * XML: {{application/vnd.hbase+xml}}
> * JSON: {{application/vnd.hbase+json}}
> * protobufs: {{application/vnd.hbase+protobuf}}
> * and for sake of consistency: {{application/vnd.hbase+avro}}
> and then in the next major version recognize both MIME types but return the
> same (newer) representation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8329) Limit compaction speed
[ https://issues.apache.org/jira/browse/HBASE-8329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-8329: Attachment: HBASE-8329-4-trunk.patch > Limit compaction speed > -- > > Key: HBASE-8329 > URL: https://issues.apache.org/jira/browse/HBASE-8329 > Project: HBase > Issue Type: Improvement > Components: Compaction >Reporter: binlijin > Attachments: HBASE-8329-2-trunk.patch, HBASE-8329-3-trunk.patch, > HBASE-8329-4-trunk.patch, HBASE-8329-trunk.patch > > > There is no speed or resource limit for compaction，I think we should add this > feature especially when request burst. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689979#comment-13689979 ] Francis Liu commented on HBASE-8015: Also I'm wondering how this will affect the rest service. Will we need a v2 set of apis to support this? > Support for Namespaces > -- > > Key: HBASE-8015 > URL: https://issues.apache.org/jira/browse/HBASE-8015 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[
https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689977#comment-13689977
]
Francis Liu commented on HBASE-8015:
{quote}
The FQTN (Fully Qualified Table Name) would be an internal or something that
could be passed to external tools (command-line, shell).
{quote}
I see, so what delimiter are we planning on using? '^' and '@' isn't appealing.
',' looks nice but you mentioned it's already used any way we can update that
cleanly to take this into account?
> Support for Namespaces
> --
>
> Key: HBASE-8015
> URL: https://issues.apache.org/jira/browse/HBASE-8015
> Project: HBase
> Issue Type: New Feature
>Reporter: Francis Liu
>Assignee: Francis Liu
> Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8721) Deletes can mask puts that happen after the delete
[ https://issues.apache.org/jira/browse/HBASE-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689975#comment-13689975 ] Feng Honghua commented on HBASE-8721: - [~sershe] : It still need 10+ms, at best, for checkAndPut / increment when the KV read isn't in RS's block-cache and need access to HDFS, though this read in RS can save 2 rpcs. Actually two months ago one of our Xiaomi's internal customers gave up using checkAndPut since they can't afford the poor performance, though they did admit they love the atomicity checkAndPut provides. > Deletes can mask puts that happen after the delete > -- > > Key: HBASE-8721 > URL: https://issues.apache.org/jira/browse/HBASE-8721 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Feng Honghua > Attachments: HBASE-8721-0.94-V0.patch > > > this fix aims for bug mentioned in http://hbase.apache.org/book.html 5.8.2.1: > "Deletes mask puts, even puts that happened after the delete was entered. > Remember that a delete writes a tombstone, which only disappears after then > next major compaction has run. Suppose you do a delete of everything <= T. > After this you do a new put with a timestamp <= T. This put, even if it > happened after the delete, will be masked by the delete tombstone. Performing > the put will not fail, but when you do a get you will notice the put did have > no effect. It will start working again after the major compaction has run. > These issues should not be a problem if you use always-increasing versions > for new puts to a row. But they can occur even if you do not care about time: > just do delete and put immediately after each other, and there is some chance > they happen within the same millisecond." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8779) add mutateRow method support to thrift2
[ https://issues.apache.org/jira/browse/HBASE-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] duanhongyi updated HBASE-8779: -- Fix Version/s: 0.94.7 > add mutateRow method support to thrift2 > --- > > Key: HBASE-8779 > URL: https://issues.apache.org/jira/browse/HBASE-8779 > Project: HBase > Issue Type: New Feature > Components: Thrift >Reporter: duanhongyi > Fix For: 0.94.7 > > Attachments: thrift2-HBASE-8779.patch, > thrift-ThriftHBaseServiceHandler.java-HBASE-8779.patch, > thrift-ThriftUtilities.java-HBASE-8779.patch > > > examples py: > TPut put = TPut() > m1 = TMutation(put=put) > TDelete single_delete = TDelete() > m2 = TMutation(singleDelete=single_delete) > row_mu = TRowMutations(row,[m1,m2]) > thbase_service.mutateRow(table,row_mu) # atomically on a single row -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8779) add mutateRow method support to thrift2
[ https://issues.apache.org/jira/browse/HBASE-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] duanhongyi updated HBASE-8779: -- Attachment: thrift-ThriftUtilities.java-HBASE-8779.patch thrift-ThriftHBaseServiceHandler.java-HBASE-8779.patch thrift2-HBASE-8779.patch ThriftHBaseServiceHandler.java ThriftUtilities.java thrift2/hbase.thrift > add mutateRow method support to thrift2 > --- > > Key: HBASE-8779 > URL: https://issues.apache.org/jira/browse/HBASE-8779 > Project: HBase > Issue Type: New Feature > Components: Thrift >Reporter: duanhongyi > Attachments: thrift2-HBASE-8779.patch, > thrift-ThriftHBaseServiceHandler.java-HBASE-8779.patch, > thrift-ThriftUtilities.java-HBASE-8779.patch > > > examples py: > TPut put = TPut() > m1 = TMutation(put=put) > TDelete single_delete = TDelete() > m2 = TMutation(singleDelete=single_delete) > row_mu = TRowMutations(row,[m1,m2]) > thbase_service.mutateRow(table,row_mu) # atomically on a single row -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8780) a column Family can have VERSIONS less than zero
[
https://issues.apache.org/jira/browse/HBASE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Demai Ni updated HBASE-8780:
Attachment: HBASE-8780-test.txt
HBASE-8780-0.94.8-v0.patch
the fix moves the error checking from constructor to the setters with minor
twist.
Fix is built and tested against 94.8. Thanks for reviewing this.
> a column Family can have VERSIONS less than zero
> -
>
> Key: HBASE-8780
> URL: https://issues.apache.org/jira/browse/HBASE-8780
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.94.8
>Reporter: Demai Ni
>Priority: Trivial
> Attachments: HBASE-8780-0.94.8-v0.patch, HBASE-8780-test.txt
>
>
> User can create/alter a columnfam and set its VERSION(aka maxVERSIONS) to a
> negative or zero value. Although there is a checking in
> HColumnDesciptor#construtor, hbase shell command will invoke the
> setter(setMaxVersions and setMinVersions) directly, hence by pass the
> checking. For example:
> {code:title=set VERSIONS = -1}
> hbase(main):016:0> create 't5_dn',{NAME=>'cf1',VERSIONS=>-1}
> 0 row(s) in 1.0420 seconds
> hbase(main):017:0> put 't5_dn','row1','cf1:q1','row1cf1_v1'
> 0 row(s) in 0.0700 seconds
> hbase(main):018:0> scan 't5_dn'
> ROW COLUMN+CELL
>
> 0 row(s) in 0.0090 seconds
> hbase(main):019:0> describe 't5_dn'
> DESCRIPTION ENABLED
>
> 't5_dn', {NAME => 'cf1', REPLICATION_SCOPE => '0', true
> KEEP_DELETED_CELLS => 'false', COMPRESSION => 'NONE
>
> ', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true',
> MIN_VERSIONS => '0', DATA_BLOCK_ENCODING => 'NONE',
>
> IN_MEMORY => 'false', BLOOMFILTER => 'NONE', TTL =
>
> > '2147483647', VERSIONS => '-1', BLOCKSIZE => '655
> >
> 36'}
> 1 row(s) in 0.0410 seconds
> {code}
> above example shows VERSIONS => '-1', and put/scan doesn't keep the data
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8772) Separate Replication from HBase RegionServer process
[ https://issues.apache.org/jira/browse/HBASE-8772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689967#comment-13689967 ] Andrew Purtell commented on HBASE-8772: --- bq. yeah, a main() would be almost all we need +1 > Separate Replication from HBase RegionServer process > > > Key: HBASE-8772 > URL: https://issues.apache.org/jira/browse/HBASE-8772 > Project: HBase > Issue Type: New Feature > Components: regionserver, Replication >Reporter: Sameer Vaishampayan > Labels: performance > > Replication is a separate functionality than managing regions and should be > able to be managed separately as a service rather than rolled into > RegionServer. Load on RegionServer, gc etc shouldn't affect the replication > service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[
https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689968#comment-13689968
]
Francis Liu commented on HBASE-8015:
{quote}
I think the old API will be against default NS.
The FQTN (Fully Qualified Table Name) would be an internal or something that
could be passed to external tools (command-line, shell).
{quote}
Is it possible to support FQTN as well? Or is going this route to avoid
surprise completely? One concern that I see internally is that all our users
will have to update their code to use the new apis so we can migrate them to
new NS. As opposed to having just a table name then it's just a simple config
change and restart.
> Support for Namespaces
> --
>
> Key: HBASE-8015
> URL: https://issues.apache.org/jira/browse/HBASE-8015
> Project: HBase
> Issue Type: New Feature
>Reporter: Francis Liu
>Assignee: Francis Liu
> Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8772) Separate Replication from HBase RegionServer process
[ https://issues.apache.org/jira/browse/HBASE-8772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689964#comment-13689964 ] Jesse Yates commented on HBASE-8772: yeah, a main() would be almost all we need... definitely an "advanced usage" feature. > Separate Replication from HBase RegionServer process > > > Key: HBASE-8772 > URL: https://issues.apache.org/jira/browse/HBASE-8772 > Project: HBase > Issue Type: New Feature > Components: regionserver, Replication >Reporter: Sameer Vaishampayan > Labels: performance > > Replication is a separate functionality than managing regions and should be > able to be managed separately as a service rather than rolled into > RegionServer. Load on RegionServer, gc etc shouldn't affect the replication > service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8772) Separate Replication from HBase RegionServer process
[ https://issues.apache.org/jira/browse/HBASE-8772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689963#comment-13689963 ] Jesse Yates commented on HBASE-8772: this came up from some discussions we've been having recently. The replication stuff is already logically separate from everything else (even has minimum cross-package dependencies). IMO, i don't think we should remove the ability to make it a built-in part of HBase, but the ability to run it separately could make it better (less chance of conflicts with other elements in a RS) and more flexible (maybe support a pull based mechanism?). > Separate Replication from HBase RegionServer process > > > Key: HBASE-8772 > URL: https://issues.apache.org/jira/browse/HBASE-8772 > Project: HBase > Issue Type: New Feature > Components: regionserver, Replication >Reporter: Sameer Vaishampayan > Labels: performance > > Replication is a separate functionality than managing regions and should be > able to be managed separately as a service rather than rolled into > RegionServer. Load on RegionServer, gc etc shouldn't affect the replication > service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-8772) Separate Replication from HBase RegionServer process
[ https://issues.apache.org/jira/browse/HBASE-8772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689951#comment-13689951 ] Andrew Purtell edited comment on HBASE-8772 at 6/21/13 1:58 AM: So we should set up and manage separate replicator processes? HBase is already considered to have too many moving parts. How do we deal with that? On the other hand, maybe a small refactoring would give us a Main somewhere where it could be an option if you want to take it on. But this would not be a default. Would that work? was (Author: apurtell): So we should set up and manage separate replicator processes? HBase is already considered to have too many moving parts? How do we deal with that? > Separate Replication from HBase RegionServer process > > > Key: HBASE-8772 > URL: https://issues.apache.org/jira/browse/HBASE-8772 > Project: HBase > Issue Type: New Feature > Components: regionserver, Replication >Reporter: Sameer Vaishampayan > Labels: performance > > Replication is a separate functionality than managing regions and should be > able to be managed separately as a service rather than rolled into > RegionServer. Load on RegionServer, gc etc shouldn't affect the replication > service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8780) a column Family can have VERSIONS less than zero
Demai Ni created HBASE-8780:
---
Summary: a column Family can have VERSIONS less than zero
Key: HBASE-8780
URL: https://issues.apache.org/jira/browse/HBASE-8780
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.94.8
Reporter: Demai Ni
Priority: Trivial
User can create/alter a columnfam and set its VERSION(aka maxVERSIONS) to a
negative or zero value. Although there is a checking in
HColumnDesciptor#construtor, hbase shell command will invoke the
setter(setMaxVersions and setMinVersions) directly, hence by pass the checking.
For example:
{code:title=set VERSIONS = -1}
hbase(main):016:0> create 't5_dn',{NAME=>'cf1',VERSIONS=>-1}
0 row(s) in 1.0420 seconds
hbase(main):017:0> put 't5_dn','row1','cf1:q1','row1cf1_v1'
0 row(s) in 0.0700 seconds
hbase(main):018:0> scan 't5_dn'
ROW COLUMN+CELL
0 row(s) in 0.0090 seconds
hbase(main):019:0> describe 't5_dn'
DESCRIPTION ENABLED
't5_dn', {NAME => 'cf1', REPLICATION_SCOPE => '0', true
KEEP_DELETED_CELLS => 'false', COMPRESSION => 'NONE
', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true',
MIN_VERSIONS => '0', DATA_BLOCK_ENCODING => 'NONE',
IN_MEMORY => 'false', BLOOMFILTER => 'NONE', TTL =
> '2147483647', VERSIONS => '-1', BLOCKSIZE => '655
36'}
1 row(s) in 0.0410 seconds
{code}
above example shows VERSIONS => '-1', and put/scan doesn't keep the data
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8779) add mutateRow method support to thrift2
duanhongyi created HBASE-8779: - Summary: add mutateRow method support to thrift2 Key: HBASE-8779 URL: https://issues.apache.org/jira/browse/HBASE-8779 Project: HBase Issue Type: New Feature Components: Thrift Reporter: duanhongyi examples py: TPut put = TPut() m1 = TMutation(put=put) TDelete single_delete = TDelete() m2 = TMutation(singleDelete=single_delete) row_mu = TRowMutations(row,[m1,m2]) thbase_service.mutateRow(table,row_mu) # atomically on a single row -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8721) Deletes can mask puts that happen after the delete
[ https://issues.apache.org/jira/browse/HBASE-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689962#comment-13689962 ] Sergey Shelukhin commented on HBASE-8721: - I don't think such a significant semantic change is worth gaining some perf in certain narrow scenarios. If you do it (delete latest/all versions) inside the regionserver, you don't pay cost of 2 rpcs; see checkAndPut for example, or increment - increment does get and put inside the server and is quite a viable API. But as I said I'm -0 not -1, esp. if it can be configured. So if some other people support it I am alright with implementation being there, with reasonable care for all the externalities (see Lars' comment above), it can be done. > Deletes can mask puts that happen after the delete > -- > > Key: HBASE-8721 > URL: https://issues.apache.org/jira/browse/HBASE-8721 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Feng Honghua > Attachments: HBASE-8721-0.94-V0.patch > > > this fix aims for bug mentioned in http://hbase.apache.org/book.html 5.8.2.1: > "Deletes mask puts, even puts that happened after the delete was entered. > Remember that a delete writes a tombstone, which only disappears after then > next major compaction has run. Suppose you do a delete of everything <= T. > After this you do a new put with a timestamp <= T. This put, even if it > happened after the delete, will be masked by the delete tombstone. Performing > the put will not fail, but when you do a get you will notice the put did have > no effect. It will start working again after the major compaction has run. > These issues should not be a problem if you use always-increasing versions > for new puts to a row. But they can occur even if you do not care about time: > just do delete and put immediately after each other, and there is some chance > they happen within the same millisecond." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7667) Support stripe compaction
[ https://issues.apache.org/jira/browse/HBASE-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689954#comment-13689954 ] Sergey Shelukhin commented on HBASE-7667: - I'm looking at how to merge policies. Unfortunately splitting uses more I/O than not splitting (who would have though... :)), resulting in worse perf. Also, the system cannot really predict future data patterns, no more than region splitting can do it (at least not with a lot of complexity added), so hint flag for how to split would need to be provided. W.r.t. producing files to contain metadata, that is unfortunately necessary. These files shouldn't have effect. Stripes with only expired files can be merged. I've taken a stab at auto-detecting stripes from file metadata, in general case it's very complex, in simplified realistic case it's just complex. Merge will drop everything into L0, yes. This could be improved, but has to be done now anyway due to references, same with split, so no need to do it now. On-by-default would require smart default settings. Let me comment tomorrow on HBASE-7680, if I can make a size-count hybrid quickly I will post final patch without a lot of logic changes there, and hopefully we can commit 3 initial patches and build on top of that. > Support stripe compaction > - > > Key: HBASE-7667 > URL: https://issues.apache.org/jira/browse/HBASE-7667 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: stripe-cdf.pdf, Stripe compaction perf evaluation.pdf, > Stripe compaction perf evaluation.pdf, Stripe compaction perf evaluation.pdf, > Stripe compactions.pdf, Stripe compactions.pdf, Stripe compactions.pdf, > Stripe compactions.pdf, Using stripe compactions.pdf, Using stripe > compactions.pdf, Using stripe compactions.pdf > > > So I was thinking about having many regions as the way to make compactions > more manageable, and writing the level db doc about how level db range > overlap and data mixing breaks seqNum sorting, and discussing it with Jimmy, > Matteo and Ted, and thinking about how to avoid Level DB I/O multiplication > factor. > And I suggest the following idea, let's call it stripe compactions. It's a > mix between level db ideas and having many small regions. > It allows us to have a subset of benefits of many regions (wrt reads and > compactions) without many of the drawbacks (managing and current > memstore/etc. limitation). > It also doesn't break seqNum-based file sorting for any one key. > It works like this. > The region key space is separated into configurable number of fixed-boundary > stripes (determined the first time we stripe the data, see below). > All the data from memstores is written to normal files with all keys present > (not striped), similar to L0 in LevelDb, or current files. > Compaction policy does 3 types of compactions. > First is L0 compaction, which takes all L0 files and breaks them down by > stripe. It may be optimized by adding more small files from different > stripes, but the main logical outcome is that there are no more L0 files and > all data is striped. > Second is exactly similar to current compaction, but compacting one single > stripe. In future, nothing prevents us from applying compaction rules and > compacting part of the stripe (e.g. similar to current policy with rations > and stuff, tiers, whatever), but for the first cut I'd argue let it "major > compact" the entire stripe. Or just have the ratio and no more complexity. > Finally, the third addresses the concern of the fixed boundaries causing > stripes to be very unbalanced. > It's exactly like the 2nd, except it takes 2+ adjacent stripes and writes the > results out with different boundaries. > There's a tradeoff here - if we always take 2 adjacent stripes, compactions > will be smaller but rebalancing will take ridiculous amount of I/O. > If we take many stripes we are essentially getting into the > epic-major-compaction problem again. Some heuristics will have to be in place. > In general, if, before stripes are determined, we initially let L0 grow > before determining the stripes, we will get better boundaries. > Also, unless unbalancing is really large we don't need to rebalance really. > Obviously this scheme (as well as level) is not applicable for all scenarios, > e.g. if timestamp is your key it completely falls apart. > The end result: > - many small compactions that can be spread out in time. > - reads still read from a small number of files (one stripe + L0). > - region splits become marvelously simple (if we could move files between > regions, no references would be needed). > Main advantage over Level (for HBase) is that default store can still ope
[jira] [Commented] (HBASE-8753) Provide new delete flag which can delete all cells under a column-family which have a same designated timestamp
[
https://issues.apache.org/jira/browse/HBASE-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689953#comment-13689953
]
Feng Honghua commented on HBASE-8753:
-
By design, DeleteFamilyVersion KV's qualifier is null, hence its comparison to
any qualifier must be "== 0" or "< 0", so execution can never enter the branch
of "throw new IllegalStateException("isDelete failed: deleteBuffer=" when an
old RS handles a DeleteFamilyVersion. (DeleteFamilyVersion will be
mis-explained as Delete type with qualifier = null)
And I'll do the backwards-compatibility test as you suggest.
> Provide new delete flag which can delete all cells under a column-family
> which have a same designated timestamp
> ---
>
> Key: HBASE-8753
> URL: https://issues.apache.org/jira/browse/HBASE-8753
> Project: HBase
> Issue Type: New Feature
> Components: Deletes
>Reporter: Feng Honghua
> Attachments: HBASE-8753-0.94-V0.patch, HBASE-8753-trunk-V0.patch
>
>
> In one of our production scenario (Xiaomi message search), multiple cells
> will be put in batch using a same timestamp with different column names under
> a specific column-family.
> And after some time these cells also need to be deleted in batch by given a
> specific timestamp. But the column names are parsed tokens which can be
> arbitrary words , so such batch delete is impossible without first retrieving
> all KVs from that CF and get the column name list which has KV with that
> given timestamp, and then issuing individual deleteColumn for each column in
> that column-list.
> Though it's possible to do such batch delete, its performance is poor, and
> customers also find their code is quite clumsy by first retrieving and
> populating the column list and then issuing a deleteColumn for each column in
> that column-list.
> This feature resolves this problem by introducing a new delete flag:
> DeleteFamilyVersion.
> 1). When you need to delete all KVs under a column-family with a given
> timestamp, just call Delete.deleteFamilyVersion(cfName, timestamp); only a
> DeleteFamilyVersion type KV is put to HBase (like DeleteFamily / DeleteColumn
> / Delete) without read operation;
> 2). Like other delete types, DeleteFamilyVersion takes effect in
> get/scan/flush/compact operations, the ScanDeleteTracker now parses out and
> uses DeleteFamilyVersion to prevent all KVs under the specific CF which has
> the same timestamp as the DeleteFamilyVersion KV to pop-up as part of a
> get/scan result (also in flush/compact).
> Our customers find this feature efficient, clean and easy-to-use since it
> does its work without knowing the exact column names list that needs to be
> deleted.
> This feature has been running smoothly for a couple of months in our
> production clusters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8772) Separate Replication from HBase RegionServer process
[ https://issues.apache.org/jira/browse/HBASE-8772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689951#comment-13689951 ] Andrew Purtell commented on HBASE-8772: --- So we should set up and manage separate replicator processes? HBase is already considered to have too many moving parts? How do we deal with that? > Separate Replication from HBase RegionServer process > > > Key: HBASE-8772 > URL: https://issues.apache.org/jira/browse/HBASE-8772 > Project: HBase > Issue Type: New Feature > Components: regionserver, Replication >Reporter: Sameer Vaishampayan > Labels: performance > > Replication is a separate functionality than managing regions and should be > able to be managed separately as a service rather than rolled into > RegionServer. Load on RegionServer, gc etc shouldn't affect the replication > service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8721) Deletes can mask puts that happen after the delete
[ https://issues.apache.org/jira/browse/HBASE-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689947#comment-13689947 ] Feng Honghua commented on HBASE-8721: - [~sershe] btw, HBase does support point version deletes as far as I see. So specific version can be deleted if desired. Should we add APIs to "delete latest version"? We can even add API to "delete all existing versions", won't be very efficient with many versions (scan or get+bunch of deletes on server side), but it will work without changing internals ===> Yes, I know what you mean. But what I mean is that the deleteColumn (without providing timestamp, AKA the 'delete latest version') is not efficient since it incurs a 'read' in RS to get the timestamp of the latest version (and set it to the Delete type KV). This operation can be tuned by removing the 'read' in RS. You can find the implementation detail in one of my above comments. > Deletes can mask puts that happen after the delete > -- > > Key: HBASE-8721 > URL: https://issues.apache.org/jira/browse/HBASE-8721 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Feng Honghua > Attachments: HBASE-8721-0.94-V0.patch > > > this fix aims for bug mentioned in http://hbase.apache.org/book.html 5.8.2.1: > "Deletes mask puts, even puts that happened after the delete was entered. > Remember that a delete writes a tombstone, which only disappears after then > next major compaction has run. Suppose you do a delete of everything <= T. > After this you do a new put with a timestamp <= T. This put, even if it > happened after the delete, will be masked by the delete tombstone. Performing > the put will not fail, but when you do a get you will notice the put did have > no effect. It will start working again after the major compaction has run. > These issues should not be a problem if you use always-increasing versions > for new puts to a row. But they can occur even if you do not care about time: > just do delete and put immediately after each other, and there is some chance > they happen within the same millisecond." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8721) Deletes can mask puts that happen after the delete
[ https://issues.apache.org/jira/browse/HBASE-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689938#comment-13689938 ] Feng Honghua commented on HBASE-8721: - [~apurtell] / [~lhofhansl]: Maybe I miss something here, is there -1 for providing config to the adjustment? > Deletes can mask puts that happen after the delete > -- > > Key: HBASE-8721 > URL: https://issues.apache.org/jira/browse/HBASE-8721 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Feng Honghua > Attachments: HBASE-8721-0.94-V0.patch > > > this fix aims for bug mentioned in http://hbase.apache.org/book.html 5.8.2.1: > "Deletes mask puts, even puts that happened after the delete was entered. > Remember that a delete writes a tombstone, which only disappears after then > next major compaction has run. Suppose you do a delete of everything <= T. > After this you do a new put with a timestamp <= T. This put, even if it > happened after the delete, will be masked by the delete tombstone. Performing > the put will not fail, but when you do a get you will notice the put did have > no effect. It will start working again after the major compaction has run. > These issues should not be a problem if you use always-increasing versions > for new puts to a row. But they can occur even if you do not care about time: > just do delete and put immediately after each other, and there is some chance > they happen within the same millisecond." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8721) Deletes can mask puts that happen after the delete
[ https://issues.apache.org/jira/browse/HBASE-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689935#comment-13689935 ] Feng Honghua commented on HBASE-8721: - [~lhofhansl] Now we're extending this to Puts as well, and are saying that a Put that hits the RegionServer later should be considered newer even if its TS is old, this opens another can of worms ===> Maybe you misunderstand me here, I never proposed 'a Put that hits the RegionServer later should be considered newer even if its TS is old'. The sequence 'put T3, put T2, put T1' (where T3>T2>T1) to a CF with max-version = 2 will result in (T3,T2) and T3 is the first version, though T1 is the last one hits RS, this is what I mean by 'timestamp is the only dimension which determines version ordering/survival by rule 'the bigger wins'' ===> What I proposed is this (can via a config to provide customers the option if they want this behavior) : the delete masks (existing) puts with timestamps less than or equal to its (not changed); and customers can choose whether the delete can mask puts still not written to HBase (future puts) according their individual real-world application logic / requirement. KEEP_DELETED_CELLS would still work fine, but their main goal is to allow correct point-in-time-queries, which among others is important for consistent backups ===> KEEP_DELETED_CELLS indeed can prevent the inconsistency in the example scenario 'put - delete - (major-compact) - put - get', and it provides a consistent result of 'get nothing'. But this result is also unacceptable for our customers since they expect the later 'put' not masked by the earlier delete. > Deletes can mask puts that happen after the delete > -- > > Key: HBASE-8721 > URL: https://issues.apache.org/jira/browse/HBASE-8721 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Feng Honghua > Attachments: HBASE-8721-0.94-V0.patch > > > this fix aims for bug mentioned in http://hbase.apache.org/book.html 5.8.2.1: > "Deletes mask puts, even puts that happened after the delete was entered. > Remember that a delete writes a tombstone, which only disappears after then > next major compaction has run. Suppose you do a delete of everything <= T. > After this you do a new put with a timestamp <= T. This put, even if it > happened after the delete, will be masked by the delete tombstone. Performing > the put will not fail, but when you do a get you will notice the put did have > no effect. It will start working again after the major compaction has run. > These issues should not be a problem if you use always-increasing versions > for new puts to a row. But they can occur even if you do not care about time: > just do delete and put immediately after each other, and there is some chance > they happen within the same millisecond." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8669) change exploring compaction policy to prefer smaller compactions on blocked stores
[ https://issues.apache.org/jira/browse/HBASE-8669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-8669: Fix Version/s: 0.95.2 > change exploring compaction policy to prefer smaller compactions on blocked > stores > -- > > Key: HBASE-8669 > URL: https://issues.apache.org/jira/browse/HBASE-8669 > Project: HBase > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.95.2 > > Attachments: HBASE-8669-v0.patch > > > Side-note from HBASE-8665 discussion. When we compact a blocked store, we > might want to use a different heuristic to choose between the options. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8669) change exploring compaction policy to prefer smaller compactions on blocked stores
[ https://issues.apache.org/jira/browse/HBASE-8669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-8669: Resolution: Fixed Status: Resolved (was: Patch Available) committed to 95 and trunk > change exploring compaction policy to prefer smaller compactions on blocked > stores > -- > > Key: HBASE-8669 > URL: https://issues.apache.org/jira/browse/HBASE-8669 > Project: HBase > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HBASE-8669-v0.patch > > > Side-note from HBASE-8665 discussion. When we compact a blocked store, we > might want to use a different heuristic to choose between the options. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-8750) MetaServerShutdownHandler stucks if .META. assignment fails in previous attempt
[ https://issues.apache.org/jira/browse/HBASE-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha reassigned HBASE-8750: -- Assignee: Himanshu Vashishtha > MetaServerShutdownHandler stucks if .META. assignment fails in previous > attempt > --- > > Key: HBASE-8750 > URL: https://issues.apache.org/jira/browse/HBASE-8750 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Attachments: HBASE-8750-v0.patch, HBASE-8750-v1.patch > > > While running log replay on a one node setup, I killed meta regionserver. The > MetaSSH tries to assign the meta table, but it failed as there was no other > regionservers to assign to. But the meta server znode was already updated to > null. When the assignment fails, the metaSSH is retried. But from the next > iteration, it will not try to assign the meta region, but keeps on waiting > for meta region. This keeps on going even after regionserver is brought up > again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8750) MetaServerShutdownHandler stucks if .META. assignment fails in previous attempt
[ https://issues.apache.org/jira/browse/HBASE-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8750: --- Attachment: HBASE-8750-v1.patch Thanks for the review. This patch uses isLocationAvailable as suggested by Jimmy. Ran a patched version locally, and it works good as the last one. > MetaServerShutdownHandler stucks if .META. assignment fails in previous > attempt > --- > > Key: HBASE-8750 > URL: https://issues.apache.org/jira/browse/HBASE-8750 > Project: HBase > Issue Type: Bug > Components: MTTR >Affects Versions: 0.95.1 >Reporter: Himanshu Vashishtha > Attachments: HBASE-8750-v0.patch, HBASE-8750-v1.patch > > > While running log replay on a one node setup, I killed meta regionserver. The > MetaSSH tries to assign the meta table, but it failed as there was no other > regionservers to assign to. But the meta server znode was already updated to > null. When the assignment fails, the metaSSH is retried. But from the next > iteration, it will not try to assign the meta region, but keeps on waiting > for meta region. This keeps on going even after regionserver is brought up > again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8409) Security support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689910#comment-13689910 ] Vandana Ayyalasomayajula commented on HBASE-8409: - Review board link: https://reviews.apache.org/r/12020/ > Security support for namespaces > --- > > Key: HBASE-8409 > URL: https://issues.apache.org/jira/browse/HBASE-8409 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Vandana Ayyalasomayajula > Attachments: HBASE-8049_trunk.patch > > > This task adds the security piece to the namespace feature. The work related > to migration of the existing acl table to the new namespace is remaining and > will be completed in the follow up patch. Permissions can be granted to a > namespace by the hbase admin, by appending '@' to the namespace name. A user > with write or admin permissions on a given namespace can create tables in > that namespace. The other privileges (R, X, C ) do not have any special > meaning w.r.t namespaces. Any users of hbase can list tables in a namespace. > > The following commands can only be executed by HBase admins. > 1. Grant privileges for user on Namespace. > 2. Revoke privileges for user on Namespace > Grant Command: > hbase> grant 'tenant-A' 'W' '@N1' > In the above example, the command will grant the user 'tenant-A' write > privileges for a namespace named "N1". > Revoke Command: > hbase> revoke 'tenant-A''@N1' > In the above example, the command will revoke all privileges from user > 'tenant-A' for namespace named "N1". > Lets see an example on how privileges work with namespaces. > > User "Mike" request for a namespace named "hbase_perf" with the hbase admin. > whoami: hbase > hbase shell >> namespace_create 'hbase_perf' > hbase shell >> grant 'mike', 'W', '@hbase_perf' > Mike creates two tables "table20" and "table50" in the above workspace. > whoami: mike > hbase shell >> create 'hbase_perf.table20', 'family1' > hbase shell >> create 'hbase_perf.table50', 'family1' > Note: As Mike was able to create tables 'hbase_perf.table20', > 'hbase_perf.table50', he becomes the owner of those tables. > This means he has "RWXCA" perms on those tables. > Another team member of Mike, Alice wants also to share the same workspace > "hbase_perf". HBase admin grants Alice also permission to create tables in > "hbase_perf" namespace. > whoami: hbase > hbase shell >> grant 'alice', 'W', '@hbase_perf' > Now Alice can create new tables under "hbase_perf" namespace, but cannot > read,write,alter,delete existing tables in the namespace. > > whoami: alice > hbase shell >> namespace_list_tables 'hbase_perf' > hbase_perf.table20 > hbase_perf.table50 > hbase shell >> scan 'hbase_perf.table20' > AccessDeniedException > > If Alice wants to read or write to existing tables in the "hbase_perf" > namespace, hbase admins need to explicitly grant permission. > > whoami: hbase > hbase shell >> grant 'alice', 'RW', 'hbase_perf.table20' > hbase shell >> grant 'alice', 'RW', 'hbase_perf.table50' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8409) Security support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vandana Ayyalasomayajula updated HBASE-8409: Attachment: HBASE-8049_trunk.patch First draft of the patch. > Security support for namespaces > --- > > Key: HBASE-8409 > URL: https://issues.apache.org/jira/browse/HBASE-8409 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Vandana Ayyalasomayajula > Attachments: HBASE-8049_trunk.patch > > > This task adds the security piece to the namespace feature. The work related > to migration of the existing acl table to the new namespace is remaining and > will be completed in the follow up patch. Permissions can be granted to a > namespace by the hbase admin, by appending '@' to the namespace name. A user > with write or admin permissions on a given namespace can create tables in > that namespace. The other privileges (R, X, C ) do not have any special > meaning w.r.t namespaces. Any users of hbase can list tables in a namespace. > > The following commands can only be executed by HBase admins. > 1. Grant privileges for user on Namespace. > 2. Revoke privileges for user on Namespace > Grant Command: > hbase> grant 'tenant-A' 'W' '@N1' > In the above example, the command will grant the user 'tenant-A' write > privileges for a namespace named "N1". > Revoke Command: > hbase> revoke 'tenant-A''@N1' > In the above example, the command will revoke all privileges from user > 'tenant-A' for namespace named "N1". > Lets see an example on how privileges work with namespaces. > > User "Mike" request for a namespace named "hbase_perf" with the hbase admin. > whoami: hbase > hbase shell >> namespace_create 'hbase_perf' > hbase shell >> grant 'mike', 'W', '@hbase_perf' > Mike creates two tables "table20" and "table50" in the above workspace. > whoami: mike > hbase shell >> create 'hbase_perf.table20', 'family1' > hbase shell >> create 'hbase_perf.table50', 'family1' > Note: As Mike was able to create tables 'hbase_perf.table20', > 'hbase_perf.table50', he becomes the owner of those tables. > This means he has "RWXCA" perms on those tables. > Another team member of Mike, Alice wants also to share the same workspace > "hbase_perf". HBase admin grants Alice also permission to create tables in > "hbase_perf" namespace. > whoami: hbase > hbase shell >> grant 'alice', 'W', '@hbase_perf' > Now Alice can create new tables under "hbase_perf" namespace, but cannot > read,write,alter,delete existing tables in the namespace. > > whoami: alice > hbase shell >> namespace_list_tables 'hbase_perf' > hbase_perf.table20 > hbase_perf.table50 > hbase shell >> scan 'hbase_perf.table20' > AccessDeniedException > > If Alice wants to read or write to existing tables in the "hbase_perf" > namespace, hbase admins need to explicitly grant permission. > > whoami: hbase > hbase shell >> grant 'alice', 'RW', 'hbase_perf.table20' > hbase shell >> grant 'alice', 'RW', 'hbase_perf.table50' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8409) Security support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vandana Ayyalasomayajula updated HBASE-8409: Description: This task adds the security piece to the namespace feature. The work related to migration of the existing acl table to the new namespace is remaining and will be completed in the follow up patch. Permissions can be granted to a namespace by the hbase admin, by appending '@' to the namespace name. A user with write or admin permissions on a given namespace can create tables in that namespace. The other privileges (R, X, C ) do not have any special meaning w.r.t namespaces. Any users of hbase can list tables in a namespace. The following commands can only be executed by HBase admins. 1. Grant privileges for user on Namespace. 2. Revoke privileges for user on Namespace Grant Command: hbase> grant 'tenant-A' 'W' '@N1' In the above example, the command will grant the user 'tenant-A' write privileges for a namespace named "N1". Revoke Command: hbase> revoke 'tenant-A''@N1' In the above example, the command will revoke all privileges from user 'tenant-A' for namespace named "N1". Lets see an example on how privileges work with namespaces. User "Mike" request for a namespace named "hbase_perf" with the hbase admin. whoami: hbase hbase shell >> namespace_create 'hbase_perf' hbase shell >> grant 'mike', 'W', '@hbase_perf' Mike creates two tables "table20" and "table50" in the above workspace. whoami: mike hbase shell >> create 'hbase_perf.table20', 'family1' hbase shell >> create 'hbase_perf.table50', 'family1' Note: As Mike was able to create tables 'hbase_perf.table20', 'hbase_perf.table50', he becomes the owner of those tables. This means he has "RWXCA" perms on those tables. Another team member of Mike, Alice wants also to share the same workspace "hbase_perf". HBase admin grants Alice also permission to create tables in "hbase_perf" namespace. whoami: hbase hbase shell >> grant 'alice', 'W', '@hbase_perf' Now Alice can create new tables under "hbase_perf" namespace, but cannot read,write,alter,delete existing tables in the namespace. whoami: alice hbase shell >> namespace_list_tables 'hbase_perf' hbase_perf.table20 hbase_perf.table50 hbase shell >> scan 'hbase_perf.table20' AccessDeniedException If Alice wants to read or write to existing tables in the "hbase_perf" namespace, hbase admins need to explicitly grant permission. whoami: hbase hbase shell >> grant 'alice', 'RW', 'hbase_perf.table20' hbase shell >> grant 'alice', 'RW', 'hbase_perf.table50' > Security support for namespaces > --- > > Key: HBASE-8409 > URL: https://issues.apache.org/jira/browse/HBASE-8409 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Vandana Ayyalasomayajula > > This task adds the security piece to the namespace feature. The work related > to migration of the existing acl table to the new namespace is remaining and > will be completed in the follow up patch. Permissions can be granted to a > namespace by the hbase admin, by appending '@' to the namespace name. A user > with write or admin permissions on a given namespace can create tables in > that namespace. The other privileges (R, X, C ) do not have any special > meaning w.r.t namespaces. Any users of hbase can list tables in a namespace. > > The following commands can only be executed by HBase admins. > 1. Grant privileges for user on Namespace. > 2. Revoke privileges for user on Namespace > Grant Command: > hbase> grant 'tenant-A' 'W' '@N1' > In the above example, the command will grant the user 'tenant-A' write > privileges for a namespace named "N1". > Revoke Command: > hbase> revoke 'tenant-A''@N1' > In the above example, the command will revoke all privileges from user > 'tenant-A' for namespace named "N1". > Lets see an example on how privileges work with namespaces. > > User "Mike" request for a namespace named "hbase_perf" with the hbase admin. > whoami: hbase > hbase shell >> namespace_create 'hbase_perf' > hbase shell >> grant 'mike', 'W', '@hbase_perf' > Mike creates two tables "table20" and "table50" in the above workspace. > whoami: mike > hbase shell >> create 'hbase_perf.table20', 'family1' > hbase shell >> create 'hbase_perf.table50', 'family1' > Note: As Mike was able to create tables 'hbase_perf.table20', > 'hbase_perf.table50', he becomes the owner of those tables. > This means he has "RWXCA" perms on those tables. > Another team member of Mike, Alice wants also to share the same workspace > "hbase_perf". HBase admin grants Alice also
[jira] [Commented] (HBASE-7963) HBase VerifyReplication not working when security enabled
[
https://issues.apache.org/jira/browse/HBASE-7963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689891#comment-13689891
]
Enis Soztutar commented on HBASE-7963:
--
That is strange. Are you saying that on znode creation, if we do not have
permission, but the node is there, we are not getting NodeExistsException in
createWithParents()?
> HBase VerifyReplication not working when security enabled
> -
>
> Key: HBASE-7963
> URL: https://issues.apache.org/jira/browse/HBASE-7963
> Project: HBase
> Issue Type: Bug
> Components: Replication, security
>Affects Versions: 0.94.5
> Environment: Security is enabled on both clusters and all principals
> are in the same realm.
>Reporter: Water Chow
> Attachments: HBASE-7963-0.94.patch, HBASE-7963-0.94-v1.patch
>
>
> When security is enabled, HBase VerifyReplication fails for two reasons:
> 1.MapReduce do not have the auth to read the replication paths
> "/hbase/replication/*" on ZK;
> 2.VerifyReplication does not get the token for slave cluster, it's different
> from HBASE-7442, this VerifyReplication does not have the output.
> {noformat}
> WARN [main]
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> RemoteException connecting to RS
> javax.security.sasl.SaslException: DIGEST-MD5: digest response format
> violation. Mismatched response.
>at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:112)
>at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:174)
>at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:177)
>at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:85)
>at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:284)
>at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:281)
>at java.security.AccessController.doPrivileged(Native Method)
>at javax.security.auth.Subject.doAs(Subject.java:396)
>at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>at java.lang.reflect.Method.invoke(Method.java:597)
>at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
>at org.apache.hadoop.hbase.security.User.call(User.java:586)
>at
> org.apache.hadoop.hbase.security.User.access$700(User.java:50)
>at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440)
>at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:280)
>at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1150)
>at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1000)
>at
> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:164)
>at $Proxy13.getProtocolVersion(Unknown Source)
>at
> org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:208)
>at
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:335)
>at
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:312)
>at
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:364)
>at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
>at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1313)
>at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1269)
>at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1256)
>at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:965)
>at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:860)
>at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:962)
>at
> org.apache.had
[jira] [Commented] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background
[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689886#comment-13689886
]
Nicolas Liochon commented on HBASE-6295:
[~sershe] Thanks for the update. Ok, will do.
[~jmspaggi] As Sergey :-). Your tests results have been pretty stable, and I
will try not to break everything. I've also seen that some other jira need you
as well ;-). And I'm quite interested by the results on 0.94 too.
> Possible performance improvement in client batch operations: presplit and
> send in background
>
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
>Affects Versions: 0.95.2
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Labels: noob
> Fix For: 0.98.0
>
> Attachments: 6295.v11.patch, 6295.v12.patch, 6295.v14.patch,
> 6295.v1.patch, 6295.v2.patch, 6295.v3.patch, 6295.v4.patch, 6295.v5.patch,
> 6295.v6.patch, 6295.v8.patch, 6295.v9.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8777) HBase client should determine the destination server after retry time
[ https://issues.apache.org/jira/browse/HBASE-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689882#comment-13689882 ] Sergey Shelukhin commented on HBASE-8777: - No, we use delayed callable... I just saw it when some integration test failed > HBase client should determine the destination server after retry time > - > > Key: HBASE-8777 > URL: https://issues.apache.org/jira/browse/HBASE-8777 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Sergey Shelukhin > > HBase currently determines which server to go to, then creates delayed > callable with pre-determined server and goes there. For later 16-32-... > second retries this approach is suboptimal, the cluster could have seen > massive changes in the meantime, so retry might be completely useless. > We should re-locate regions after the delay, at least for longer retries. > Given how grouping is currently done it would be a bit of a refactoring. > The effect of this is alleviated (to a degree) on trunk by server-based > retries (if we fail going to the pre-delay server after delay and then > determine the server has changed, we will go to the new server immediately, > so we only lose the failed round-trip time); on 94, if the region is opened > on some other server during the delay, we'd go to the old one, fail, then > find out it's on different server, wait a bunch more time because it's a > late-stage retry and THEN go to the new one, as far as I see. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background
[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689881#comment-13689881
]
Sergey Shelukhin commented on HBASE-6295:
-
it's the feature that keeps tabs on wait time based on server to which we are
sending the request, so for ex. in simple case, if we just did 16-second retry
and now learn that region is on different server we don't wait for 32sec. to go
there, but do so immediately.
> Possible performance improvement in client batch operations: presplit and
> send in background
>
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
>Affects Versions: 0.95.2
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Labels: noob
> Fix For: 0.98.0
>
> Attachments: 6295.v11.patch, 6295.v12.patch, 6295.v14.patch,
> 6295.v1.patch, 6295.v2.patch, 6295.v3.patch, 6295.v4.patch, 6295.v5.patch,
> 6295.v6.patch, 6295.v8.patch, 6295.v9.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background
[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689880#comment-13689880
]
Sergey Shelukhin commented on HBASE-6295:
-
Probably not, it is MTTR feature rather than perf
> Possible performance improvement in client batch operations: presplit and
> send in background
>
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
>Affects Versions: 0.95.2
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Labels: noob
> Fix For: 0.98.0
>
> Attachments: 6295.v11.patch, 6295.v12.patch, 6295.v14.patch,
> 6295.v1.patch, 6295.v2.patch, 6295.v3.patch, 6295.v4.patch, 6295.v5.patch,
> 6295.v6.patch, 6295.v8.patch, 6295.v9.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background
[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689877#comment-13689877
]
Jean-Marc Spaggiari commented on HBASE-6295:
Will I need to re-run the tests?
> Possible performance improvement in client batch operations: presplit and
> send in background
>
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
>Affects Versions: 0.95.2
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Labels: noob
> Fix For: 0.98.0
>
> Attachments: 6295.v11.patch, 6295.v12.patch, 6295.v14.patch,
> 6295.v1.patch, 6295.v2.patch, 6295.v3.patch, 6295.v4.patch, 6295.v5.patch,
> 6295.v6.patch, 6295.v8.patch, 6295.v9.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background
[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689876#comment-13689876
]
Nicolas Liochon commented on HBASE-6295:
I don't fully remember what it is and I didn't remove it on purpose but
hopefully I can put it back. Will do this tomorrow.
> Possible performance improvement in client batch operations: presplit and
> send in background
>
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
>Affects Versions: 0.95.2
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Labels: noob
> Fix For: 0.98.0
>
> Attachments: 6295.v11.patch, 6295.v12.patch, 6295.v14.patch,
> 6295.v1.patch, 6295.v2.patch, 6295.v3.patch, 6295.v4.patch, 6295.v5.patch,
> 6295.v6.patch, 6295.v8.patch, 6295.v9.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8777) HBase client should determine the destination server after retry time
[ https://issues.apache.org/jira/browse/HBASE-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689875#comment-13689875 ] Nicolas Liochon commented on HBASE-8777: I won't dare backporting 6295 to 0.95 :-) But iirc in 0.94 we were doing the split after the sleep (it may have changed, I haven't looked for a while) > HBase client should determine the destination server after retry time > - > > Key: HBASE-8777 > URL: https://issues.apache.org/jira/browse/HBASE-8777 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Sergey Shelukhin > > HBase currently determines which server to go to, then creates delayed > callable with pre-determined server and goes there. For later 16-32-... > second retries this approach is suboptimal, the cluster could have seen > massive changes in the meantime, so retry might be completely useless. > We should re-locate regions after the delay, at least for longer retries. > Given how grouping is currently done it would be a bit of a refactoring. > The effect of this is alleviated (to a degree) on trunk by server-based > retries (if we fail going to the pre-delay server after delay and then > determine the server has changed, we will go to the new server immediately, > so we only lose the failed round-trip time); on 94, if the region is opened > on some other server during the delay, we'd go to the old one, fail, then > find out it's on different server, wait a bunch more time because it's a > late-stage retry and THEN go to the new one, as far as I see. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8777) HBase client should determine the destination server after retry time
[ https://issues.apache.org/jira/browse/HBASE-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689869#comment-13689869 ] Sergey Shelukhin commented on HBASE-8777: - not in 94 though? > HBase client should determine the destination server after retry time > - > > Key: HBASE-8777 > URL: https://issues.apache.org/jira/browse/HBASE-8777 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Sergey Shelukhin > > HBase currently determines which server to go to, then creates delayed > callable with pre-determined server and goes there. For later 16-32-... > second retries this approach is suboptimal, the cluster could have seen > massive changes in the meantime, so retry might be completely useless. > We should re-locate regions after the delay, at least for longer retries. > Given how grouping is currently done it would be a bit of a refactoring. > The effect of this is alleviated (to a degree) on trunk by server-based > retries (if we fail going to the pre-delay server after delay and then > determine the server has changed, we will go to the new server immediately, > so we only lose the failed round-trip time); on 94, if the region is opened > on some other server during the delay, we'd go to the old one, fail, then > find out it's on different server, wait a bunch more time because it's a > late-stage retry and THEN go to the new one, as far as I see. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background
[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689867#comment-13689867
]
Sergey Shelukhin commented on HBASE-6295:
-
Hmm, I just noticed this test removed usage of
errorsByServer.calculateBackoffTime.
Can it please be put back? I have to withdraw my +1... :(
> Possible performance improvement in client batch operations: presplit and
> send in background
>
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
>Affects Versions: 0.95.2
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Labels: noob
> Fix For: 0.98.0
>
> Attachments: 6295.v11.patch, 6295.v12.patch, 6295.v14.patch,
> 6295.v1.patch, 6295.v2.patch, 6295.v3.patch, 6295.v4.patch, 6295.v5.patch,
> 6295.v6.patch, 6295.v8.patch, 6295.v9.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8776) port HBASE-8723 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689861#comment-13689861 ] Sergey Shelukhin commented on HBASE-8776: - there's only one 8, and 32. The problem is that we determine the server before delay, so recovery has to happen before the delay for last retry (I filed a JIRA for that). 1+1+1+2+2+4+4+8+16 = 39. Recovery after zk timeout is also not instant. > port HBASE-8723 to 0.94 > --- > > Key: HBASE-8776 > URL: https://issues.apache.org/jira/browse/HBASE-8776 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.8 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.94.9 > > Attachments: HBASE-8776-v0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689856#comment-13689856 ] stack commented on HBASE-8015: -- bq. If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names? I think the old API will be against default NS. The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell). > Support for Namespaces > -- > > Key: HBASE-8015 > URL: https://issues.apache.org/jira/browse/HBASE-8015 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8778) Region assigments scan table directory making them slow for huge tables
[
https://issues.apache.org/jira/browse/HBASE-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-8778:
---
Attachment: HBASE-8778-0.94.5.patch
One solution is to instead keep the table descriptor files in a subdirectory of
the table directory so that only that subdirectory needs a scan. The attached
patch is one from 0.94.5 that implements this scheme. In order to be
applicable in a rolling restart scenario, the new descriptor is written to both
the table directory and the subdirectory. Readers first read the subdirectory,
then fall back to the table directory. In order to be robust against failures
or races, a lock file is used in the subdirectory during writes. The patch
also refactors the FSTableDescriptors class to require a Configuration (to
determine lock wait duration) as well as updates so that it more uniformly
enforces the fsreadonly flag (RegionServers never do writes) and stick with
using instance methods rather than static methods. We are proceeding with this
and hope to roll it out to our cluster. To update to this patch once the
writers (HBase Master, tools like hbck, merge, compact) are upgraded then old
writers should not be used.
I would love to hear the opinion of the HBase community regarding this issue.
Some questions:
- Is it worth fixing? (I strongly believe so as it has a big impact on MTTR
for large clusters)
- What's the best approach to fixing?
- Some other possibilities:
- Using a lock file and well known table descriptor file rather than
sequence ids
- Relying on more descriptor caching rather than hitting hdfs on every
region assignment (as bulk assignment already does).
- Move table descriptors to a different location in hdfs (single location
for all tables?)
- Move table descriptors out of hdfs to ZK
- How and when can we migrate to that approach?
- For the patch above once the cluster has been upgraded and updated the
location of the descriptor files to have a copy in the subdirectory it would be
easy to have the next version use only those files.
- Alternatively, for the singularity there could be a one-time piece of
migration code that just moved the files there.
> Region assigments scan table directory making them slow for huge tables
> ---
>
> Key: HBASE-8778
> URL: https://issues.apache.org/jira/browse/HBASE-8778
> Project: HBase
> Issue Type: Improvement
>Reporter: Dave Latham
> Attachments: HBASE-8778-0.94.5.patch
>
>
> On a table with 130k regions it takes about 3 seconds for a region server to
> open a region once it has been assigned.
> Watching the threads for a region server running 0.94.5 that is opening many
> such regions shows the thread opening the reigon in code like this:
> {noformat}
> "PRI IPC Server handler 4 on 60020" daemon prio=10 tid=0x2aaac07e9000
> nid=0x6566 runnable [0x4c46d000]
>java.lang.Thread.State: RUNNABLE
> at java.lang.String.indexOf(String.java:1521)
> at java.net.URI$Parser.scan(URI.java:2912)
> at java.net.URI$Parser.parse(URI.java:3004)
> at java.net.URI.(URI.java:736)
> at org.apache.hadoop.fs.Path.initialize(Path.java:145)
> at org.apache.hadoop.fs.Path.(Path.java:126)
> at org.apache.hadoop.fs.Path.(Path.java:50)
> at
> org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:215)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:311)
> at
> org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:159)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:842)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:867)
> at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1168)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:269)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:255)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoModtime(FSTableDescriptors.java:368)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:155)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:126)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2834)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2807)
> at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
> at
> sun.r
[jira] [Created] (HBASE-8778) Region assigments scan table directory making them slow for huge tables
Dave Latham created HBASE-8778:
--
Summary: Region assigments scan table directory making them slow
for huge tables
Key: HBASE-8778
URL: https://issues.apache.org/jira/browse/HBASE-8778
Project: HBase
Issue Type: Improvement
Reporter: Dave Latham
Attachments: HBASE-8778-0.94.5.patch
On a table with 130k regions it takes about 3 seconds for a region server to
open a region once it has been assigned.
Watching the threads for a region server running 0.94.5 that is opening many
such regions shows the thread opening the reigon in code like this:
{noformat}
"PRI IPC Server handler 4 on 60020" daemon prio=10 tid=0x2aaac07e9000
nid=0x6566 runnable [0x4c46d000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.indexOf(String.java:1521)
at java.net.URI$Parser.scan(URI.java:2912)
at java.net.URI$Parser.parse(URI.java:3004)
at java.net.URI.(URI.java:736)
at org.apache.hadoop.fs.Path.initialize(Path.java:145)
at org.apache.hadoop.fs.Path.(Path.java:126)
at org.apache.hadoop.fs.Path.(Path.java:50)
at
org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:215)
at
org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:252)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:311)
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:159)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:842)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:867)
at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1168)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:269)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:255)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoModtime(FSTableDescriptors.java:368)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:155)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:126)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2834)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2807)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
{noformat}
To open the region, the region server first loads the latest HTableDescriptor.
Since HBASE-4553 HTableDescriptor's are stored in the file system at
"/hbase//.tableinfo.". The file with the largest
sequenceNum is the current descriptor. This is done so that the current
descirptor is updated atomically. However, since the filename is not known in
advance FSTableDescriptors it has to do a FileSystem.listStatus operation which
has to list all files in the directory to find it. The directory also contains
all the region directories, so in our case it has to load 130k FileStatus
objects. Even using a globStatus matching function still transfers all the
objects to the client before performing the pattern matching. Furthermore HDFS
uses a default of transferring 1000 directory entries in each RPC call, so it
requires 130 roundtrips to the namenode to fetch all the directory entries.
Consequently, to reassign all the regions of a table (or a constant fraction
thereof) requires time proportional to the square of the number of regions.
In our case, if a region server fails with 200 such regions, it takes 10+
minutes for them all to be reassigned, after the zk expiration and log
splitting.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8777) HBase client should determine the destination server after retry time
[ https://issues.apache.org/jira/browse/HBASE-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689819#comment-13689819 ] Nicolas Liochon commented on HBASE-8777: It's actually implemented this way in 6295. > HBase client should determine the destination server after retry time > - > > Key: HBASE-8777 > URL: https://issues.apache.org/jira/browse/HBASE-8777 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Sergey Shelukhin > > HBase currently determines which server to go to, then creates delayed > callable with pre-determined server and goes there. For later 16-32-... > second retries this approach is suboptimal, the cluster could have seen > massive changes in the meantime, so retry might be completely useless. > We should re-locate regions after the delay, at least for longer retries. > Given how grouping is currently done it would be a bit of a refactoring. > The effect of this is alleviated (to a degree) on trunk by server-based > retries (if we fail going to the pre-delay server after delay and then > determine the server has changed, we will go to the new server immediately, > so we only lose the failed round-trip time); on 94, if the region is opened > on some other server during the delay, we'd go to the old one, fail, then > find out it's on different server, wait a bunch more time because it's a > late-stage retry and THEN go to the new one, as far as I see. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6620) Test org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor.testPoolBehavior flaps in autobuilds.
[ https://issues.apache.org/jira/browse/HBASE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-6620. -- Resolution: Cannot Reproduce We have not seen this in a long time. Closing. Can open a new one if we see it again. > Test > org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor.testPoolBehavior > flaps in autobuilds. > --- > > Key: HBASE-6620 > URL: https://issues.apache.org/jira/browse/HBASE-6620 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Sameer Vaishampayan > > Test flaps in autobuilds with assertion failure. > org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor.testPoolBehavior > Failing for the past 1 build (Since #2602 ) > Took 3 ms. > Error Message > expected:<3> but was:<4> > Stacktrace > java.lang.AssertionError: expected:<3> but was:<4> > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.failNotEquals(Assert.java:647) > at org.junit.Assert.assertEquals(Assert.java:128) > at org.junit.Assert.assertEquals(Assert.java:472) > at org.junit.Assert.assertEquals(Assert.java:456) > at > org.apache.hadoop.hbase.client.TestFromClientSide.testPoolBehavior(TestFromClientSide.java:4334) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47) > at org.junit.rules.RunRules.evaluate(RunRules.java:18) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) > at org.junit.runners.ParentRunner.run(ParentRunner.java:300) > at org.junit.runners.Suite.runChild(Suite.java:128) > at org.junit.runners.Suite.runChild(Suite.java:24) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689812#comment-13689812 ] Enis Soztutar commented on HBASE-8015: -- One problem with option 4 is that we want to pay the price of migration only one between 0.94->0.96. If we do that, then it means we have to carry the exception tables code for all the releases going forward. Option 1 better than this I think? Note that surprise #1 also applies here as well. bq. Sounds like we are going with overloading all the existing apis to take a namespace parameter. If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names? It should use the default ns. I think the idea is that there will not be a public facing thing called "fully qualified table name" in Elliot's approach. Although internally, we will need one, hence my tendency to go with option 2 over 3 (see my above comment): "namespace,table" seems good enough for me. > Support for Namespaces > -- > > Key: HBASE-8015 > URL: https://issues.apache.org/jira/browse/HBASE-8015 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8776) port HBASE-8723 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689801#comment-13689801 ] Lars Hofhansl commented on HBASE-8776: -- Don't the current defaults already add up to 47? 1+1+1+2+2+4+4+8+8+16 = 47 10 seems good enough, unless I am missing something. Will check the original jira tomorrow. > port HBASE-8723 to 0.94 > --- > > Key: HBASE-8776 > URL: https://issues.apache.org/jira/browse/HBASE-8776 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.8 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.94.9 > > Attachments: HBASE-8776-v0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8777) HBase client should determine the destination server after retry time
Sergey Shelukhin created HBASE-8777: --- Summary: HBase client should determine the destination server after retry time Key: HBASE-8777 URL: https://issues.apache.org/jira/browse/HBASE-8777 Project: HBase Issue Type: Improvement Components: Client Reporter: Sergey Shelukhin HBase currently determines which server to go to, then creates delayed callable with pre-determined server and goes there. For later 16-32-... second retries this approach is suboptimal, the cluster could have seen massive changes in the meantime, so retry might be completely useless. We should re-locate regions after the delay, at least for longer retries. Given how grouping is currently done it would be a bit of a refactoring. The effect of this is alleviated (to a degree) on trunk by server-based retries (if we fail going to the pre-delay server after delay and then determine the server has changed, we will go to the new server immediately, so we only lose the failed round-trip time); on 94, if the region is opened on some other server during the delay, we'd go to the old one, fail, then find out it's on different server, wait a bunch more time because it's a late-stage retry and THEN go to the new one, as far as I see. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8776) port HBASE-8723 to 0.94
[
https://issues.apache.org/jira/browse/HBASE-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689783#comment-13689783
]
Hadoop QA commented on HBASE-8776:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588947/HBASE-8776-v0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6086//console
This message is automatically generated.
> port HBASE-8723 to 0.94
> ---
>
> Key: HBASE-8776
> URL: https://issues.apache.org/jira/browse/HBASE-8776
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.8
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.94.9
>
> Attachments: HBASE-8776-v0.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8776) port HBASE-8723 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689781#comment-13689781 ] Sergey Shelukhin commented on HBASE-8776: - [~lhofhansl] are you ok with this change to client retries? > port HBASE-8723 to 0.94 > --- > > Key: HBASE-8776 > URL: https://issues.apache.org/jira/browse/HBASE-8776 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.8 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.94.9 > > Attachments: HBASE-8776-v0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8776) port HBASE-8723 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-8776: Affects Version/s: 0.94.8 > port HBASE-8723 to 0.94 > --- > > Key: HBASE-8776 > URL: https://issues.apache.org/jira/browse/HBASE-8776 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.8 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-8776-v0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8776) port HBASE-8723 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-8776: Status: Patch Available (was: Open) > port HBASE-8723 to 0.94 > --- > > Key: HBASE-8776 > URL: https://issues.apache.org/jira/browse/HBASE-8776 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-8776-v0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8776) port HBASE-8723 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-8776: Fix Version/s: 0.94.9 > port HBASE-8723 to 0.94 > --- > > Key: HBASE-8776 > URL: https://issues.apache.org/jira/browse/HBASE-8776 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.8 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.94.9 > > Attachments: HBASE-8776-v0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8776) port HBASE-8723 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-8776: Attachment: HBASE-8776-v0.patch I am increasing retry count less aggressively than original; this should be more than enough to ride over server failure given the default negotiated ZK timeout of 40s. > port HBASE-8723 to 0.94 > --- > > Key: HBASE-8776 > URL: https://issues.apache.org/jira/browse/HBASE-8776 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-8776-v0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8776) port HBASE-8723 to 0.94
Sergey Shelukhin created HBASE-8776: --- Summary: port HBASE-8723 to 0.94 Key: HBASE-8776 URL: https://issues.apache.org/jira/browse/HBASE-8776 Project: HBase Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4755) HBase based block placement in DFS
[ https://issues.apache.org/jira/browse/HBASE-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689754#comment-13689754 ] Devaraj Das commented on HBASE-4755: [~jiangbinglover], yes HBase would need to periodically refresh the mappings, and also when compactions happen, the data would be rewritten in the three current nodes. I need to implement the balancer in FavoredNodeLoadBalancer (balanceCluster method). I should have something shortly. > HBase based block placement in DFS > -- > > Key: HBASE-4755 > URL: https://issues.apache.org/jira/browse/HBASE-4755 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.0 >Reporter: Karthik Ranganathan >Assignee: Christopher Gist >Priority: Critical > Attachments: 4755-wip-1.patch, hbase-4755-notes.txt > > > The feature as is only useful for HBase clusters that care about data > locality on regionservers, but this feature can also enable a lot of nice > features down the road. > The basic idea is as follows: instead of letting HDFS determine where to > replicate data (r=3) by place blocks on various regions, it is better to let > HBase do so by providing hints to HDFS through the DFS client. That way > instead of replicating data at a blocks level, we can replicate data at a > per-region level (each region owned by a promary, a secondary and a tertiary > regionserver). This is better for 2 things: > - Can make region failover faster on clusters which benefit from data affinity > - On large clusters with random block placement policy, this helps reduce the > probability of data loss > The algo is as follows: > - Each region in META will have 3 columns which are the preferred > regionservers for that region (primary, secondary and tertiary) > - Preferred assignment can be controlled by a config knob > - Upon cluster start, HMaster will enter a mapping from each region to 3 > regionservers (random hash, could use current locality, etc) > - The load balancer would assign out regions preferring region assignments to > primary over secondary over tertiary over any other node > - Periodically (say weekly, configurable) the HMaster would run a locality > checked and make sure the map it has for region to regionservers is optimal. > Down the road, this can be enhanced to control region placement in the > following cases: > - Mixed hardware SKU where some regionservers can hold fewer regions > - Load balancing across tables where we dont want multiple regions of a table > to get assigned to the same regionservers > - Multi-tenancy, where we can restrict the assignment of the regions of some > table to a subset of regionservers, so an abusive app cannot take down the > whole HBase cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-1177) Delay when client is located on the same node as the regionserver
[ https://issues.apache.org/jira/browse/HBASE-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-1177. -- Resolution: Invalid Resolving as no longer valid. Looks like Nagles' anyways. > Delay when client is located on the same node as the regionserver > - > > Key: HBASE-1177 > URL: https://issues.apache.org/jira/browse/HBASE-1177 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 0.19.0 > Environment: Linux 2.6.25 x86_64 >Reporter: Jonathan Gray > Labels: noob > Attachments: Contribution of getClosest to getRow time.jpg, > Contribution of next to getRow time.jpg, Contribution of seekTo to getClosest > time.jpg, Elapsed time of RowResults.readFields.jpg, getRow + round-trip vs # > columns.jpg, getRow times.jpg, ReadDelayTest.java, RowResults.readFields > zoomed.jpg, screenshot-1.jpg, screenshot-2.jpg, screenshot-3.jpg, > screenshot-4.jpg, zoom of columns vs round-trip blowup.jpg > > > During testing of HBASE-80, we uncovered a strange 40ms delay for random > reads. We ran a series of tests and found that it only happens when the > client is on the same node as the RS and for a certain range of payloads (not > specifically related to number of columns or size of them, only total > payload). It appears to be precisely 40ms every time. > Unsure if this is particular to our architecture, but it does happen on all > nodes we've tried. Issue completely goes away with very large payloads or > moving the client. > Will post a test program tomorrow if anyone can test on a different > architecture. > Making a blocker for 0.20. Since this happens when you have an MR task > running local to the RS, and this is what we try to do, might also consider > making this a blocker for 0.19.1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8721) Deletes can mask puts that happen after the delete
[ https://issues.apache.org/jira/browse/HBASE-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689717#comment-13689717 ] Sergey Shelukhin commented on HBASE-8721: - btw, HBase does support point version deletes as far as I see. So specific version can be deleted if desired. Should we add APIs to "delete latest version"? We can even add API to "delete all existing versions", won't be very efficient with many versions (scan or get+bunch of deletes on server side), but it will work without changing internals > Deletes can mask puts that happen after the delete > -- > > Key: HBASE-8721 > URL: https://issues.apache.org/jira/browse/HBASE-8721 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Feng Honghua > Attachments: HBASE-8721-0.94-V0.patch > > > this fix aims for bug mentioned in http://hbase.apache.org/book.html 5.8.2.1: > "Deletes mask puts, even puts that happened after the delete was entered. > Remember that a delete writes a tombstone, which only disappears after then > next major compaction has run. Suppose you do a delete of everything <= T. > After this you do a new put with a timestamp <= T. This put, even if it > happened after the delete, will be masked by the delete tombstone. Performing > the put will not fail, but when you do a get you will notice the put did have > no effect. It will start working again after the major compaction has run. > These issues should not be a problem if you use always-increasing versions > for new puts to a row. But they can occur even if you do not care about time: > just do delete and put immediately after each other, and there is some chance > they happen within the same millisecond." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[
https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689631#comment-13689631
]
Francis Liu commented on HBASE-8015:
{quote}
Francis, we should still do the Elliott suggestion even if dot, right? The dot
would be for 'external' tools or a useful facility in shell but we want
namespaces to be first class in API too.
{quote}
The approach I proposed earlier would avoid having to do all the api stuff as
part of the first namespace checkin as well as make use of '.' as a delimeter.
The suprises are as I mentioned. We can incrementally add the apis.
Sounds like we are going with overloading all the existing apis to take a
namespace parameter. If so what would be the behavior when using the old api?
Will it always reference default namespace or will we support fully qualified
table names?
For some reason I'm not getting any jira or RB emails. Will take a look.
> Support for Namespaces
> --
>
> Key: HBASE-8015
> URL: https://issues.apache.org/jira/browse/HBASE-8015
> Project: HBase
> Issue Type: New Feature
>Reporter: Francis Liu
>Assignee: Francis Liu
> Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8662) [rest] support impersonation
[ https://issues.apache.org/jira/browse/HBASE-8662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689623#comment-13689623 ] Francis Liu commented on HBASE-8662: Yep. OptionServlet, what for? > [rest] support impersonation > > > Key: HBASE-8662 > URL: https://issues.apache.org/jira/browse/HBASE-8662 > Project: HBase > Issue Type: Sub-task > Components: REST, security >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.98.0 > > Attachments: method_doas.patch, secure_rest.patch, trunk-8662.patch, > trunk-8662_v2.patch, trunk-8662_v3.patch > > > Currently, our client API uses a fixed user: the current user. It should > accept a user passed in, if authenticated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background
[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689578#comment-13689578
]
Jean-Marc Spaggiari commented on HBASE-6295:
It's time for x lines, depending of the tests it's not the same number of lines.
For RandomReadTest you need to divide by 1048576
For RandomScanWithRange100Test you need to divide by 4096
For RandomSeekScanTest you need to divide by 40960.
For RandomWriteTest you need to divide by 1048576
For SequentialWriteTest you need to divide by 1048576
This is the number of lines per ms. So multiply by 1000 to have the same
result. Some are rows/minutes, so just adjust that.
So if you want to compare, here are the numbers in the same format as te PDF
that I usually produce:
||Test||Trunk||Nic||0.95||
|org,apache,hadoop,hbase,PerformanceEvaluation$RandomReadTest|1377.08|1420.14|1390.50|
|org,apache,hadoop,hbase,PerformanceEvaluation$RandomScanWithRange100Test|11243.12|10992.68|10971.09|
|org,apache,hadoop,hbase,PerformanceEvaluation$RandomSeekScanTest|304.66|296.43|305.25|
|org,apache,hadoop,hbase,PerformanceEvaluation$RandomWriteTest|9176.07|13619.59|9134.09|
|org,apache,hadoop,hbase,PerformanceEvaluation$SequentialWriteTest|13592.40|42655.52|13255.12|
I already noticed the RandomWriteTest impact compared to 0.94 branch and 0.95...
I will re-run the 0.94 tests to make sure, but overall, I really think 0.95 is
not doing as good as 0.95 for the RandomWriteTest.
> Possible performance improvement in client batch operations: presplit and
> send in background
>
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
>Affects Versions: 0.95.2
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Labels: noob
> Fix For: 0.98.0
>
> Attachments: 6295.v11.patch, 6295.v12.patch, 6295.v14.patch,
> 6295.v1.patch, 6295.v2.patch, 6295.v3.patch, 6295.v4.patch, 6295.v5.patch,
> 6295.v6.patch, 6295.v8.patch, 6295.v9.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-8662) [rest] support impersonation
[ https://issues.apache.org/jira/browse/HBASE-8662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689521#comment-13689521 ] Jimmy Xiang edited comment on HBASE-8662 at 6/20/13 6:59 PM: - The AuthFilter in your patch looks very familiar. Got from Ooize? Do we need the optionsServlet? was (Author: jxiang): The AuthFilter in your patch looks very familiar. Got from Ooize? > [rest] support impersonation > > > Key: HBASE-8662 > URL: https://issues.apache.org/jira/browse/HBASE-8662 > Project: HBase > Issue Type: Sub-task > Components: REST, security >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.98.0 > > Attachments: method_doas.patch, secure_rest.patch, trunk-8662.patch, > trunk-8662_v2.patch, trunk-8662_v3.patch > > > Currently, our client API uses a fixed user: the current user. It should > accept a user passed in, if authenticated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8662) [rest] support impersonation
[ https://issues.apache.org/jira/browse/HBASE-8662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689521#comment-13689521 ] Jimmy Xiang commented on HBASE-8662: The AuthFilter in your patch looks very familiar. Got from Ooize? > [rest] support impersonation > > > Key: HBASE-8662 > URL: https://issues.apache.org/jira/browse/HBASE-8662 > Project: HBase > Issue Type: Sub-task > Components: REST, security >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.98.0 > > Attachments: method_doas.patch, secure_rest.patch, trunk-8662.patch, > trunk-8662_v2.patch, trunk-8662_v3.patch > > > Currently, our client API uses a fixed user: the current user. It should > accept a user passed in, if authenticated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background
[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689519#comment-13689519
]
Nicolas Liochon commented on HBASE-6295:
Can I do 2097152 / 79 = 26500 to compare with the performances tests previously
described in
http://www.spaggiari.org/media/blogs/hbase/pictures/performances_20130321.pdf?
Because the performances were better previously (~35k / rows second).
Same for 2097152 / 114 = 18396 vs. ~30k
Or is it calculated differently?
Anyway, thanks a lot for all these great tests. I will commit tomorrow morning
my time if there is no objection.
> Possible performance improvement in client batch operations: presplit and
> send in background
>
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
>Affects Versions: 0.95.2
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Labels: noob
> Fix For: 0.98.0
>
> Attachments: 6295.v11.patch, 6295.v12.patch, 6295.v14.patch,
> 6295.v1.patch, 6295.v2.patch, 6295.v3.patch, 6295.v4.patch, 6295.v5.patch,
> 6295.v6.patch, 6295.v8.patch, 6295.v9.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3149) Make flush decisions per column family
[ https://issues.apache.org/jira/browse/HBASE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3149: - Fix Version/s: (was: 0.92.3) > Make flush decisions per column family > -- > > Key: HBASE-3149 > URL: https://issues.apache.org/jira/browse/HBASE-3149 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Karthik Ranganathan >Priority: Critical > > Today, the flush decision is made using the aggregate size of all column > families. When large and small column families co-exist, this causes many > small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8759) Family Delete Markers not getting purged after major compaction
[ https://issues.apache.org/jira/browse/HBASE-8759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689513#comment-13689513 ] James Taylor commented on HBASE-8759: - Thanks, @larsh. Now aren't you supposed to be on vacation! Go drink a liter or two of beer! :-) > Family Delete Markers not getting purged after major compaction > --- > > Key: HBASE-8759 > URL: https://issues.apache.org/jira/browse/HBASE-8759 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 0.94.7 >Reporter: Mujtaba Chohan >Priority: Minor > > On table with VERSIONS => '1', KEEP_DELETED_CELLS => 'true'. Family Delete > Markers does not get purged after put > delete > major compaction (they keep > on incrementing after every put > delete > major compaction) > Following is the raw scan output after 10 iterations of put > delete > major > compaction. > ROW COLUMN+CELL > > > Acolumn=CF:, timestamp=1371512706683, > type=DeleteFamily > > Acolumn=CF:, timestamp=1371512706394, > type=DeleteFamily > > Acolumn=CF:, timestamp=1371512706054, > type=DeleteFamily > > Acolumn=CF:, timestamp=1371512705763, > type=DeleteFamily > > Acolumn=CF:, timestamp=1371512705457, > type=DeleteFamily > > Acolumn=CF:, timestamp=1371512705149, > type=DeleteFamily > > Acolumn=CF:, timestamp=1371512704836, > type=DeleteFamily > > Acolumn=CF:, timestamp=1371512704518, > type=DeleteFamily > > Acolumn=CF:, timestamp=1371512704162, > type=DeleteFamily > > Acolumn=CF:, timestamp=1371512703779, > type=DeleteFamily > > Acolumn=CF:COL, > timestamp=1371512706682, value=X > [~lhofhansl] > Code to repro this issue: > http://phoenix-bin.github.io/client/code/delete.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689512#comment-13689512 ] stack commented on HBASE-8015: -- [~toffer] There will be a migration evaluation script that will looks for presence of stuff like hfile v1s -- they must be compacted away before you can upgrade -- and this same step could check table names and if any w/ dot found, list options. This script would be run against 0.94 install before shutting down for upgrade (Yes two scripts, a checker, and then a doer). Francis, we should still do the Elliott suggestion even if dot, right? The dot would be for 'external' tools or a useful facility in shell but we want namespaces to be first class in API too. Did you get my review comments up on rb francis? On dots in namespace, no, if it simplifies. > Support for Namespaces > -- > > Key: HBASE-8015 > URL: https://issues.apache.org/jira/browse/HBASE-8015 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3149) Make flush decisions per column family
[ https://issues.apache.org/jira/browse/HBASE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-3149: --- Assignee: (was: Himanshu Vashishtha) I figured that it increases mttr time. I will probably look into it after we fixed mttr issues of late. Un-assigning it for the meanwhile. > Make flush decisions per column family > -- > > Key: HBASE-3149 > URL: https://issues.apache.org/jira/browse/HBASE-3149 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Karthik Ranganathan >Priority: Critical > Fix For: 0.92.3 > > > Today, the flush decision is made using the aggregate size of all column > families. When large and small column families co-exist, this causes many > small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-8721) Deletes can mask puts that happen after the delete
[ https://issues.apache.org/jira/browse/HBASE-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-8721. --- Resolution: Won't Fix bq. It is unlikely that this will be changed as you have to a find committers to +1 this. All we got up to this points are a -1 unless it is configurable and a couple of -0s. Agreed, resolved as WONTFIX. Interested parties are encouraged to go to the followups HBASE-8763 and HBASE-8770 > Deletes can mask puts that happen after the delete > -- > > Key: HBASE-8721 > URL: https://issues.apache.org/jira/browse/HBASE-8721 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Feng Honghua > Attachments: HBASE-8721-0.94-V0.patch > > > this fix aims for bug mentioned in http://hbase.apache.org/book.html 5.8.2.1: > "Deletes mask puts, even puts that happened after the delete was entered. > Remember that a delete writes a tombstone, which only disappears after then > next major compaction has run. Suppose you do a delete of everything <= T. > After this you do a new put with a timestamp <= T. This put, even if it > happened after the delete, will be masked by the delete tombstone. Performing > the put will not fail, but when you do a get you will notice the put did have > no effect. It will start working again after the major compaction has run. > These issues should not be a problem if you use always-increasing versions > for new puts to a row. But they can occur even if you do not care about time: > just do delete and put immediately after each other, and there is some chance > they happen within the same millisecond." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8662) [rest] support impersonation
[ https://issues.apache.org/jira/browse/HBASE-8662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8662: --- Status: Open (was: Patch Available) > [rest] support impersonation > > > Key: HBASE-8662 > URL: https://issues.apache.org/jira/browse/HBASE-8662 > Project: HBase > Issue Type: Sub-task > Components: REST, security >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.98.0 > > Attachments: method_doas.patch, secure_rest.patch, trunk-8662.patch, > trunk-8662_v2.patch, trunk-8662_v3.patch > > > Currently, our client API uses a fixed user: the current user. It should > accept a user passed in, if authenticated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[
https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689501#comment-13689501
]
Sergey Shelukhin commented on HBASE-8015:
-
{code}exceptionNS.add(tableName.getNamespaceAsString()); {code}
What is the current thinking on dots in namespaces and names? Presumably one
table could prevent the creation of multiple namespaces if dots are allowed in
namespace name, which I thought they are
> Support for Namespaces
> --
>
> Key: HBASE-8015
> URL: https://issues.apache.org/jira/browse/HBASE-8015
> Project: HBase
> Issue Type: New Feature
>Reporter: Francis Liu
>Assignee: Francis Liu
> Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8627) HBCK can not fix meta not assigned issue
[ https://issues.apache.org/jira/browse/HBASE-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689495#comment-13689495 ] Sergey Shelukhin commented on HBASE-8627: - lgtm > HBCK can not fix meta not assigned issue > > > Key: HBASE-8627 > URL: https://issues.apache.org/jira/browse/HBASE-8627 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.95.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Attachments: HBASE-8627_Trunk.patch, HBASE-8627_Trunk-V2.patch > > > When meta table region is not assigned to any RS, HBCK run will get > exception. I can see code added in checkMetaRegion() to solve this issue but > it wont work. It still refers to ROOT region! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[ https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689492#comment-13689492 ] Jean-Marc Spaggiari commented on HBASE-8755: Sure! Let me prepare that. I will read this JIRA from the beginning and try to start the tests today. > A new write thread model for HLog to improve the overall HBase write > throughput > --- > > Key: HBASE-8755 > URL: https://issues.apache.org/jira/browse/HBASE-8755 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Feng Honghua > Attachments: HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, > HBASE-8755-trunk-V0.patch > > > In current write model, each write handler thread (executing put()) will > individually go through a full 'append (hlog local buffer) => HLog writer > append (write to hdfs) => HLog writer sync (sync hdfs)' cycle for each write, > which incurs heavy race condition on updateLock and flushLock. > The only optimization where checking if current syncTillHere > txid in > expectation for other thread help write/sync its own txid to hdfs and > omitting the write/sync actually help much less than expectation. > Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi > proposed a new write thread model for writing hdfs sequence file and the > prototype implementation shows a 4X improvement for throughput (from 17000 to > 7+). > I apply this new write thread model in HLog and the performance test in our > test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1 > RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size) > even beats the one of BigTable (Precolator published in 2011 says Bigtable's > write throughput then is 31002). I can provide the detailed performance test > results if anyone is interested. > The change for new write thread model is as below: > 1> All put handler threads append the edits to HLog's local pending buffer; > (it notifies AsyncWriter thread that there is new edits in local buffer) > 2> All put handler threads wait in HLog.syncer() function for underlying > threads to finish the sync that contains its txid; > 3> An single AsyncWriter thread is responsible for retrieve all the buffered > edits in HLog's local pending buffer and write to the hdfs > (hlog.writer.append); (it notifies AsyncFlusher thread that there is new > writes to hdfs that needs a sync) > 4> An single AsyncFlusher thread is responsible for issuing a sync to hdfs > to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread > that sync watermark increases) > 5> An single AsyncNotifier thread is responsible for notifying all pending > put handler threads which are waiting in the HLog.syncer() function > 6> No LogSyncer thread any more (since there is always > AsyncWriter/AsyncFlusher threads do the same job it does) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8771) ensure replication_scope's value is either local(0) or global(1)
[
https://issues.apache.org/jira/browse/HBASE-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689489#comment-13689489
]
Demai Ni commented on HBASE-8771:
-
[~ctrezzo], another interesting finding while I was playing different
approaches (shell vs constructor). Using maxVersion as example, which has code
like below to check value:
{code}
if (maxVersions <= 0) {
// TODO: Allow maxVersion of 0 to be the way you say "Keep all versions".
// Until there is support, consider 0 or < 0 -- a configuration error.
throw new IllegalArgumentException("Maximum versions must be positive");
}
{code}
Above code can catch the illegal arg only when user call the HColumnDescriptor
constructor, but won't work in hbase shell or call setMaxVersion() direclty.
{code:title=set Max Version = -1 in Shell. No Error thrown because shell called
setMaxVersions directly}
hbase(main):016:0> create 't5_dn',{NAME=>'cf1',VERSIONS=>-1}
0 row(s) in 1.0420 seconds
hbase(main):017:0> put 't5_dn','row1','cf1:q1','row1cf1_v1'
0 row(s) in 0.0700 seconds
hbase(main):018:0> scan 't5_dn'
ROW COLUMN+CELL
0 row(s) in 0.0090 seconds
hbase(main):019:0> describe 't5_dn'
DESCRIPTION ENABLED
't5_dn', {NAME => 'cf1', VERSIONS => '-1',...}
{code}
{code:title=set Max Version = -999 through constructor. Error caught inside }
HTableDescriptor ht = new HTableDescriptor( "t3_dn" );
HColumnDescriptor cfd = new HColumnDescriptor(Bytes.toBytes(
"cf1"),-999,"NONE",false,false,100,"NONE");
...
Exception in thread "main" java.lang.IllegalArgumentException: Maximum versions
must be positive
at
org.apache.hadoop.hbase.HColumnDescriptor.(HColumnDescriptor.java:386)
at
org.apache.hadoop.hbase.HColumnDescriptor.(HColumnDescriptor.java:334)
at
org.apache.hadoop.hbase.HColumnDescriptor.(HColumnDescriptor.java:302)
at CreateTable_version.main(CreateTable_version.java:23)
{code}
> ensure replication_scope's value is either local(0) or global(1)
>
>
> Key: HBASE-8771
> URL: https://issues.apache.org/jira/browse/HBASE-8771
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 0.94.8
>Reporter: Demai Ni
>Priority: Minor
> Fix For: 0.94.9
>
> Attachments: HBASE-8771-0.94.8-v0.patch
>
>
> For replication_scope, only two values are meaningful:
> {code}
> public static final int REPLICATION_SCOPE_LOCAL = 0;
> public static final int REPLICATION_SCOPE_GLOBAL = 1;
> {code}
> However, there is no checking for that, so currently user can set it to any
> integer value. And all non-zero value will be treated as 1(GLOBAL).
> This jira is to add a checking in HColumnDescriptor#setScope() so that only 0
> and 1 will be accept during create_table or alter_table.
> In the future, we can leverage replication_scope to store for info. For
> example:
> -1: A columnfam is replicated from another cluster in MASTER_SLAVE setup (i.e
> readonly)
> 2 : A columnfam is set MASTER_MASTER
> Probably a major improve JIRA is needed for the future usage. It will be good
> to ensure the scope value at this moment.
> {code:title=Testing|borderStyle=solid}
> hbase(main):002:0> create 't1_dn',{NAME=>'cf1',REPLICATION_SCOPE=>2}
> ERROR: java.lang.IllegalArgumentException: Replication Scope must be either
> 0(local) or 1(global)
> ...
> hbase(main):004:0> alter 't1_dn',{NAME=>'cf1',REPLICATION_SCOPE=>-1}
> ERROR: java.lang.IllegalArgumentException: Replication Scope must be either
> 0(local) or 1(global)
> ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8775) Throttle online schema changes.
Shane Hogan created HBASE-8775: -- Summary: Throttle online schema changes. Key: HBASE-8775 URL: https://issues.apache.org/jira/browse/HBASE-8775 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.89-fb Reporter: Shane Hogan Priority: Minor Fix For: 0.89-fb Throttle the open and close of the regions after an online schema change -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6295) Possible performance improvement in client batch operations: presplit and send in background
[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689480#comment-13689480
]
Jean-Marc Spaggiari commented on HBASE-6295:
||Test||Trunk||Nic||0.95||
|org.apache.hadoop.hbase.PerformanceEvaluation$RandomReadTest|761449.8|738362.4|754100|
|org.apache.hadoop.hbase.PerformanceEvaluation$RandomScanWithRange100Test|21858.7|22356.7|22400.7|
|org.apache.hadoop.hbase.PerformanceEvaluation$RandomSeekScanTest|13.6|138179.3|134186.7|
|org.apache.hadoop.hbase.PerformanceEvaluation$RandomWriteTest|114272.9|76990.3|114798.1|
|org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest|77144.275|24582.425|79107.25|
so Trunk and 0.95 are consistent, while Nic's version show a nice improvement
on the write operations (both Random and Sequentials), and a very small
degradation on SeekScan. Also a small improvement on RandomRead.
Do you need the IntegrationTestBigLinkedList for the 3 releases too?
> Possible performance improvement in client batch operations: presplit and
> send in background
>
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
>Affects Versions: 0.95.2
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Labels: noob
> Fix For: 0.98.0
>
> Attachments: 6295.v11.patch, 6295.v12.patch, 6295.v14.patch,
> 6295.v1.patch, 6295.v2.patch, 6295.v3.patch, 6295.v4.patch, 6295.v5.patch,
> 6295.v6.patch, 6295.v8.patch, 6295.v9.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689468#comment-13689468 ] Francis Liu commented on HBASE-8015: Oops sorry I guess I'm talking about two scripts. One to check if some surprising migration needs to be done and provide links/options. And another that does the actual migration. > Support for Namespaces > -- > > Key: HBASE-8015 > URL: https://issues.apache.org/jira/browse/HBASE-8015 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8774) Add BatchSize and Filter to Thrift2
[
https://issues.apache.org/jira/browse/HBASE-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689454#comment-13689454
]
Hamed Madani commented on HBASE-8774:
-
'true' is there because the rest of the thrift and thrift2 had that format. For
example
{code}
boolean this_present_columns = true && this.isSetColumns();
boolean that_present_columns = true && that.isSetColumns();
{code}
HBASE-6073 patch is missing the modification to thrift2.generated.TScan.java
and thrift2.generated.TGet.java.
> Add BatchSize and Filter to Thrift2
> ---
>
> Key: HBASE-8774
> URL: https://issues.apache.org/jira/browse/HBASE-8774
> Project: HBase
> Issue Type: New Feature
> Components: Thrift
>Affects Versions: 0.95.1
>Reporter: Hamed Madani
> Attachments: HBASE_8774.patch
>
>
> Attached Patch will add BatchSize and Filter support to Thrift2
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8771) ensure replication_scope's value is either local(0) or global(1)
[
https://issues.apache.org/jira/browse/HBASE-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689449#comment-13689449
]
Demai Ni commented on HBASE-8771:
-
[~anoop.hbase], thanks for you comments.
since we plan to use values other than 0 or 1 in the future, it may be better
to block other values now to avoid conflict in the future.
For example, today a userA can use value -1 and 2 for scope, and hbase code
will treat it as '1' (global replication). Then, a future JIRA gives value '2'
another feature. userA will face a difficult replication behavior.
With that, it is better to block values such as -1 and 2 earlier to reduce such
potential issues.
> ensure replication_scope's value is either local(0) or global(1)
>
>
> Key: HBASE-8771
> URL: https://issues.apache.org/jira/browse/HBASE-8771
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 0.94.8
>Reporter: Demai Ni
>Priority: Minor
> Fix For: 0.94.9
>
> Attachments: HBASE-8771-0.94.8-v0.patch
>
>
> For replication_scope, only two values are meaningful:
> {code}
> public static final int REPLICATION_SCOPE_LOCAL = 0;
> public static final int REPLICATION_SCOPE_GLOBAL = 1;
> {code}
> However, there is no checking for that, so currently user can set it to any
> integer value. And all non-zero value will be treated as 1(GLOBAL).
> This jira is to add a checking in HColumnDescriptor#setScope() so that only 0
> and 1 will be accept during create_table or alter_table.
> In the future, we can leverage replication_scope to store for info. For
> example:
> -1: A columnfam is replicated from another cluster in MASTER_SLAVE setup (i.e
> readonly)
> 2 : A columnfam is set MASTER_MASTER
> Probably a major improve JIRA is needed for the future usage. It will be good
> to ensure the scope value at this moment.
> {code:title=Testing|borderStyle=solid}
> hbase(main):002:0> create 't1_dn',{NAME=>'cf1',REPLICATION_SCOPE=>2}
> ERROR: java.lang.IllegalArgumentException: Replication Scope must be either
> 0(local) or 1(global)
> ...
> hbase(main):004:0> alter 't1_dn',{NAME=>'cf1',REPLICATION_SCOPE=>-1}
> ERROR: java.lang.IllegalArgumentException: Replication Scope must be either
> 0(local) or 1(global)
> ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8060) "Num compacting KVs" diverges from "num compacted KVs" over time
[ https://issues.apache.org/jira/browse/HBASE-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689442#comment-13689442 ] Lars Hofhansl commented on HBASE-8060: -- I see. So it does not really change the meaning - it's still the number of KV compacted in last compaction - it just corrects the value. > "Num compacting KVs" diverges from "num compacted KVs" over time > > > Key: HBASE-8060 > URL: https://issues.apache.org/jira/browse/HBASE-8060 > Project: HBase > Issue Type: Bug > Components: Compaction, UI >Affects Versions: 0.94.6, 0.95.0, 0.95.2 >Reporter: Andrew Purtell >Assignee: Sergey Shelukhin > Attachments: HBASE-8060-v0.patch, screenshot.png > > > I have been running what amounts to an ingestion test for a day or so. This > is an all-in-one cluster launched with './bin/hbase master start' from > sources. In the RS stats on the master UI, the "num compacting KVs" has > diverged from "num compacted KVs" even though compaction has been completed > from perspective of selection, no compaction tasks are running on the RS. I > think this could be confusing -- is compaction happening or not? > Or maybe I'm misunderstanding what this is supposed to show? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8753) Provide new delete flag which can delete all cells under a column-family which have a same designated timestamp
[
https://issues.apache.org/jira/browse/HBASE-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689441#comment-13689441
]
Lars Hofhansl commented on HBASE-8753:
--
May you not run into this case in isDeleted:
{code}
int ret = Bytes.compareTo(deleteBuffer, deleteOffset, deleteLength,
buffer, qualifierOffset, qualifierLength);
if (ret == 0) {
...
} else if(ret < 0){
...
} else {
throw new IllegalStateException("isDelete failed: deleteBuffer="
...
{code}
In any case, we should just test it: Write an HFile with a new RS, start an old
RS, scan that file, check that it works fine.
> Provide new delete flag which can delete all cells under a column-family
> which have a same designated timestamp
> ---
>
> Key: HBASE-8753
> URL: https://issues.apache.org/jira/browse/HBASE-8753
> Project: HBase
> Issue Type: New Feature
> Components: Deletes
>Reporter: Feng Honghua
> Attachments: HBASE-8753-0.94-V0.patch, HBASE-8753-trunk-V0.patch
>
>
> In one of our production scenario (Xiaomi message search), multiple cells
> will be put in batch using a same timestamp with different column names under
> a specific column-family.
> And after some time these cells also need to be deleted in batch by given a
> specific timestamp. But the column names are parsed tokens which can be
> arbitrary words , so such batch delete is impossible without first retrieving
> all KVs from that CF and get the column name list which has KV with that
> given timestamp, and then issuing individual deleteColumn for each column in
> that column-list.
> Though it's possible to do such batch delete, its performance is poor, and
> customers also find their code is quite clumsy by first retrieving and
> populating the column list and then issuing a deleteColumn for each column in
> that column-list.
> This feature resolves this problem by introducing a new delete flag:
> DeleteFamilyVersion.
> 1). When you need to delete all KVs under a column-family with a given
> timestamp, just call Delete.deleteFamilyVersion(cfName, timestamp); only a
> DeleteFamilyVersion type KV is put to HBase (like DeleteFamily / DeleteColumn
> / Delete) without read operation;
> 2). Like other delete types, DeleteFamilyVersion takes effect in
> get/scan/flush/compact operations, the ScanDeleteTracker now parses out and
> uses DeleteFamilyVersion to prevent all KVs under the specific CF which has
> the same timestamp as the DeleteFamilyVersion KV to pop-up as part of a
> get/scan result (also in flush/compact).
> Our customers find this feature efficient, clean and easy-to-use since it
> does its work without knowing the exact column names list that needs to be
> deleted.
> This feature has been running smoothly for a couple of months in our
> production clusters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8771) ensure replication_scope's value is either local(0) or global(1)
[
https://issues.apache.org/jira/browse/HBASE-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689439#comment-13689439
]
Demai Ni commented on HBASE-8771:
-
[~ctrezzo] , sorry that I didn't explain clear the first time. Although the
setScope is currently only used in such the columndescriptor constructor, the
setScope() is a public method. so user can do this
{code}
...
HTableDescriptor ht = new HTableDescriptor( "t3_dn" );
HColumnDescriptor cfd = new HColumnDescriptor( "cf1" );
cfd.setScope(-1000);
ht.addFamily( cfd );
...
{code}
so if the checking is put inside the constructor (similar as the logic for
minVersions and maxVersions), the above code will not be caught.
> ensure replication_scope's value is either local(0) or global(1)
>
>
> Key: HBASE-8771
> URL: https://issues.apache.org/jira/browse/HBASE-8771
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 0.94.8
>Reporter: Demai Ni
>Priority: Minor
> Fix For: 0.94.9
>
> Attachments: HBASE-8771-0.94.8-v0.patch
>
>
> For replication_scope, only two values are meaningful:
> {code}
> public static final int REPLICATION_SCOPE_LOCAL = 0;
> public static final int REPLICATION_SCOPE_GLOBAL = 1;
> {code}
> However, there is no checking for that, so currently user can set it to any
> integer value. And all non-zero value will be treated as 1(GLOBAL).
> This jira is to add a checking in HColumnDescriptor#setScope() so that only 0
> and 1 will be accept during create_table or alter_table.
> In the future, we can leverage replication_scope to store for info. For
> example:
> -1: A columnfam is replicated from another cluster in MASTER_SLAVE setup (i.e
> readonly)
> 2 : A columnfam is set MASTER_MASTER
> Probably a major improve JIRA is needed for the future usage. It will be good
> to ensure the scope value at this moment.
> {code:title=Testing|borderStyle=solid}
> hbase(main):002:0> create 't1_dn',{NAME=>'cf1',REPLICATION_SCOPE=>2}
> ERROR: java.lang.IllegalArgumentException: Replication Scope must be either
> 0(local) or 1(global)
> ...
> hbase(main):004:0> alter 't1_dn',{NAME=>'cf1',REPLICATION_SCOPE=>-1}
> ERROR: java.lang.IllegalArgumentException: Replication Scope must be either
> 0(local) or 1(global)
> ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8721) Deletes can mask puts that happen after the delete
[ https://issues.apache.org/jira/browse/HBASE-8721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689435#comment-13689435 ] Lars Hofhansl commented on HBASE-8721: -- KEEP_DELETED_CELLS would still work fine, but their main goal is to allow correct point-in-time-queries, which among others is important for consistent backups. Regarding all the points above. Let's please not go overboard. Now we're extending this to Puts as well, and are saying that a Put that hits the RegionServer later should be considered newer even if its TS is old, this opens another can of worms. It is unlikely that this will be changed as you have to a find committers to +1 this. All we got up to this points are a -1 unless it is configurable and a couple of -0s. > Deletes can mask puts that happen after the delete > -- > > Key: HBASE-8721 > URL: https://issues.apache.org/jira/browse/HBASE-8721 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Feng Honghua > Attachments: HBASE-8721-0.94-V0.patch > > > this fix aims for bug mentioned in http://hbase.apache.org/book.html 5.8.2.1: > "Deletes mask puts, even puts that happened after the delete was entered. > Remember that a delete writes a tombstone, which only disappears after then > next major compaction has run. Suppose you do a delete of everything <= T. > After this you do a new put with a timestamp <= T. This put, even if it > happened after the delete, will be masked by the delete tombstone. Performing > the put will not fail, but when you do a get you will notice the put did have > no effect. It will start working again after the major compaction has run. > These issues should not be a problem if you use always-increasing versions > for new puts to a row. But they can occur even if you do not care about time: > just do delete and put immediately after each other, and there is some chance > they happen within the same millisecond." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8060) "Num compacting KVs" diverges from "num compacted KVs" over time
[ https://issues.apache.org/jira/browse/HBASE-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689424#comment-13689424 ] Sergey Shelukhin commented on HBASE-8060: - The meaning of total compacting KVs is the number of KVs in the input for last-started compaction (this historically assumes one compaction at a time per store). I.e. either current compaction or the last finished one. It is an estimate of the number of KVs compaction will see, used to track progress. Patch does change meaning of it to reconcile the (often incorrect) estimate with reality.. > "Num compacting KVs" diverges from "num compacted KVs" over time > > > Key: HBASE-8060 > URL: https://issues.apache.org/jira/browse/HBASE-8060 > Project: HBase > Issue Type: Bug > Components: Compaction, UI >Affects Versions: 0.94.6, 0.95.0, 0.95.2 >Reporter: Andrew Purtell >Assignee: Sergey Shelukhin > Attachments: HBASE-8060-v0.patch, screenshot.png > > > I have been running what amounts to an ingestion test for a day or so. This > is an all-in-one cluster launched with './bin/hbase master start' from > sources. In the RS stats on the master UI, the "num compacting KVs" has > diverged from "num compacted KVs" even though compaction has been completed > from perspective of selection, no compaction tasks are running on the RS. I > think this could be confusing -- is compaction happening or not? > Or maybe I'm misunderstanding what this is supposed to show? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689419#comment-13689419 ] Francis Liu commented on HBASE-8015: I thought of a way to implement [~sershe] idea. It was simple enough so I thought I'd give it a try. Essentially keep an in-memory list of tables which make use of the delimiter (ie '.') and consider these tables as exceptions to the namespace rule and handle them properly to make sure they are part of the default namespace. Have an added constraint that prevent creation of namespaces and tables that would conflict with any of the exception tables (ie ns1 and ns1.foo). Surprise here is: - you can't create tables with the delimiter no longer unless you create the appropriate namespace. - you can't create tables/namespace which conflict the exception tables/namespaces - the exception list is derived by scanning the default namespace directories in .tmp, .data and .archive Here's a sample of how it works. I've updated the TestNamespaceUpgrade test to verify that it works: https://github.com/francisliu/hbase_namespace/tree/core_8408_exception_list > Support for Namespaces > -- > > Key: HBASE-8015 > URL: https://issues.apache.org/jira/browse/HBASE-8015 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689417#comment-13689417 ] Francis Liu commented on HBASE-8015: [~saint@gmail.com], I'm leaning towards having the migration operation done manually by calling a script as well. Which options do we provide the user? Also it might be better if the script is portable enough that they can run on an existing 0.94 cluster so they don't have to find out during the actual upgrade process. > Support for Namespaces > -- > > Key: HBASE-8015 > URL: https://issues.apache.org/jira/browse/HBASE-8015 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
