[jira] [Updated] (HBASE-17432) Use less threads to read WAL in replication
[ https://issues.apache.org/jira/browse/HBASE-17432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17432: -- Fix Version/s: (was: 2.0.0) > Use less threads to read WAL in replication > --- > > Key: HBASE-17432 > URL: https://issues.apache.org/jira/browse/HBASE-17432 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Phil Yang >Assignee: Phil Yang > > Now for each peer and each failover source we use different threads to read > WAL, which means one WAL file will be read many times if we have many peers. > We can read logs in one thread and push them to peers concurrently. > Some cases should be considered. For example, if one of peers is stuck, the > reading thread should kick it out of concurrent peers and use an independent > thread to read logs for this peer just like old style. If finally this peer's > progress catch up other peers, they can be merged back. > Of course, failover threads should still be independent because they read > different WAL files. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17432) Use less threads to read WAL in replication
[ https://issues.apache.org/jira/browse/HBASE-17432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177586#comment-16177586 ] Phil Yang commented on HBASE-17432: --- Remove fix version as we may not get it in before 2.0 release, thanks. > Use less threads to read WAL in replication > --- > > Key: HBASE-17432 > URL: https://issues.apache.org/jira/browse/HBASE-17432 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Phil Yang >Assignee: Phil Yang > > Now for each peer and each failover source we use different threads to read > WAL, which means one WAL file will be read many times if we have many peers. > We can read logs in one thread and push them to peers concurrently. > Some cases should be considered. For example, if one of peers is stuck, the > reading thread should kick it out of concurrent peers and use an independent > thread to read logs for this peer just like old style. If finally this peer's > progress catch up other peers, they can be merged back. > Of course, failover threads should still be independent because they read > different WAL files. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156995#comment-16156995
]
Phil Yang commented on HBASE-15968:
---
No jira currently. It will be a new HFile format version?
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18546) Append/Increment a cell with custom timestamp
[
https://issues.apache.org/jira/browse/HBASE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126758#comment-16126758
]
Phil Yang commented on HBASE-18546:
---
IIRC, for append/increment we will set a ts which must be larger than the
previous one in WAL. Does we really need setting timestamp for append/increment
at client?
> Append/Increment a cell with custom timestamp
> -
>
> Key: HBASE-18546
> URL: https://issues.apache.org/jira/browse/HBASE-18546
> Project: HBase
> Issue Type: New Feature
> Components: API, Client
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Labels: incompatibleChange
> Fix For: 2.0.0
>
> Attachments: HBASE-18546.v0.patch, HBASE-18546.v1.patch
>
>
> We don't pass the custom timestamp for Increment, and the increment's
> timestamp always be rewrite. Hence, user can't increment a cell with custom
> timestamp.
> {code:title=ProtobufUtil.java}
> if (values != null && values.size() > 0) {
> for (Cell cell: values) {
> valueBuilder.clear();
> valueBuilder.setQualifier(UnsafeByteOperations.unsafeWrap(
> cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength()));
> valueBuilder.setValue(UnsafeByteOperations.unsafeWrap(
> cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength()));
> if (cell.getTagsLength() > 0) {
>
> valueBuilder.setTags(UnsafeByteOperations.unsafeWrap(cell.getTagsArray(),
> cell.getTagsOffset(), cell.getTagsLength()));
> }
> columnBuilder.addQualifierValue(valueBuilder.build());
> }
> }
> {code}
> In contrast to Increment, user can append the cell with custom timestamp. It
> would be better that make their behavior consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099622#comment-16099622
]
Phil Yang edited comment on HBASE-15968 at 7/25/17 8:07 AM:
Pushed to master and branch-2. Thanks for reviews.
Future works:
Enable it by default since 3.0?
If enable it by default, we may not use vlong in mvcc in HFile because we will
always keep them rather than set to 0.
Performance issue.
More tests, maybe IT-tests?
Documents.
Refactor? We merge two trackers into one object.
was (Author: yangzhe1991):
Pushed to master and branch-2. Thanks for reviews.
Future works:
Enable it by default since 3.0?
If enable it by default, we may not use vlong in mvcc in HFile because we will
always keep them rather than set to 0.
Performance issue.
More tests, maybe IT-tests?
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabl
[jira] [Updated] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to master and branch-2. Thanks for reviews.
Future works:
Enable it by default since 3.0?
If enable it by default, we may not use vlong in mvcc in HFile because we will
always keep them rather than set to 0.
Performance issue.
More tests, maybe IT-tests?
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099611#comment-16099611
]
Phil Yang commented on HBASE-15968:
---
They are not related and I'll remove them when commit
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Fix Version/s: 2.0.0-alpha-2
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Release Note:
This issue resolved two long-term issues in HBase:
Puts may be masked by a delete before them.
Major compactions change query results.
This issue offer a new behavior to fix this issue with a little performance
reduction. Set NEW_VERSION_BEHAVIOR to true to enable this feature in CF level.
See HBASE-15968 for details.
Note if you enable this feature, the order of Mutations matters. But
replication will disorder the entries by default. So you have to enable serial
replication if you have slave clusters. See HBASE-9465 for details.
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099595#comment-16099595
]
Phil Yang commented on HBASE-15968:
---
Checked two ruby tests, not fully understand the meaning.. But TestShell passed
and it only add a new attribute so I think it is OK. Will push to master and
branch-2 shortly.
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16097923#comment-16097923
]
Phil Yang commented on HBASE-15968:
---
Uploaded latest patch on https://reviews.apache.org/r/61070/
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Attachment: HBASE-15968.v07.patch
retry
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096069#comment-16096069
]
Phil Yang commented on HBASE-15968:
---
Checked two failing tests, in current head of master TestShell also failed.
TestBlockCacheReporting always passed locally...
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968-v1.patch, HBASE-15968-v2.patch,
> HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18334) Remove sync client implementation and wrap async client under sync client interface
[ https://issues.apache.org/jira/browse/HBASE-18334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096016#comment-16096016 ] Phil Yang edited comment on HBASE-18334 at 7/21/17 9:27 AM: In [HBaseCon 2017 (in page 39) |https://www.slideshare.net/HBaseCon/hbase-at-xiaomi?qid=96198dc1-bac8-4d13-a7df-64dbad476c34&v=&b=&from_search=18] we showed some basic PE results. was (Author: yangzhe1991): In [HBaseCon 2017 |https://www.slideshare.net/HBaseCon/hbase-at-xiaomi?qid=96198dc1-bac8-4d13-a7df-64dbad476c34&v=&b=&from_search=18] we showed some basic PE results. > Remove sync client implementation and wrap async client under sync client > interface > --- > > Key: HBASE-18334 > URL: https://issues.apache.org/jira/browse/HBASE-18334 > Project: HBase > Issue Type: Task >Reporter: Phil Yang > Fix For: 3.0.0 > > > Since 2.0 we have async client, now we have two client implementations. We > can implement an sync client (Table interface) by using async client, getting > a CompletableFuture and then waiting it done directly. This can reduce the > maintenance work at client side in the future. > Async client is almost done, we tested the performance and it showed it has > same performance with sync client. In branch-2 we can keep old sync client > implementations and remove it in master branch (since 3.0). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18334) Remove sync client implementation and wrap async client under sync client interface
[ https://issues.apache.org/jira/browse/HBASE-18334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18334: -- Description: Since 2.0 we have async client, now we have two client implementations. We can implement an sync client (Table interface) by using async client, getting a CompletableFuture and then waiting it done directly. This can reduce the maintenance work at client side in the future. Async client is done, we tested the performance and it showed it has same performance with sync client. In branch-2 we can keep old sync client implementations and remove it in master branch (since 3.0). was: Since 2.0 we have async client, now we have two client implementations. We can implement an sync client (Table interface) by using async client, getting a CompletableFuture and then waiting it done directly. This can reduce the maintenance work at client side in the future. Async client is almost done, we tested the performance and it showed it has same performance with sync client. In branch-2 we can keep old sync client implementations and remove it in master branch (since 3.0). > Remove sync client implementation and wrap async client under sync client > interface > --- > > Key: HBASE-18334 > URL: https://issues.apache.org/jira/browse/HBASE-18334 > Project: HBase > Issue Type: Task >Reporter: Phil Yang > Fix For: 3.0.0 > > > Since 2.0 we have async client, now we have two client implementations. We > can implement an sync client (Table interface) by using async client, getting > a CompletableFuture and then waiting it done directly. This can reduce the > maintenance work at client side in the future. > Async client is done, we tested the performance and it showed it has same > performance with sync client. In branch-2 we can keep old sync client > implementations and remove it in master branch (since 3.0). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18334) Remove sync client implementation and wrap async client under sync client interface
[ https://issues.apache.org/jira/browse/HBASE-18334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096016#comment-16096016 ] Phil Yang commented on HBASE-18334: --- In [HBaseCon 2017 |https://www.slideshare.net/HBaseCon/hbase-at-xiaomi?qid=96198dc1-bac8-4d13-a7df-64dbad476c34&v=&b=&from_search=18] we showed some basic PE results. > Remove sync client implementation and wrap async client under sync client > interface > --- > > Key: HBASE-18334 > URL: https://issues.apache.org/jira/browse/HBASE-18334 > Project: HBase > Issue Type: Task >Reporter: Phil Yang > Fix For: 3.0.0 > > > Since 2.0 we have async client, now we have two client implementations. We > can implement an sync client (Table interface) by using async client, getting > a CompletableFuture and then waiting it done directly. This can reduce the > maintenance work at client side in the future. > Async client is almost done, we tested the performance and it showed it has > same performance with sync client. In branch-2 we can keep old sync client > implementations and remove it in master branch (since 3.0). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Attachment: HBASE-15968.v07.patch
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968-v1.patch, HBASE-15968-v2.patch,
> HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Attachment: HBASE-15968.v06.patch
Fix failed test and findbugs
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968-v1.patch, HBASE-15968-v2.patch, HBASE-15968-v3.patch,
> HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094433#comment-16094433
]
Phil Yang commented on HBASE-15968:
---
This patch is for master, not sure when we can push to branch-2 and versions of
branch-2 can not be determined now (alpha-2 or alpha-3 or beta-xx..)
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Fix Version/s: 3.0.0
Status: Patch Available (was: Open)
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Attachment: HBASE-15968.v05.patch
Back to this issue. Rebase to current master. And change new config to
NEW_VERSION_BEHAVIOR, default is false which is same as current logic.
See QA results first. If no failing tests I'll post the patch to rb. Thanks.
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968.v05.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Status: Open (was: Patch Available)
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch, HBASE-15968-v2.patch,
> HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Summary: New behavior of versions considering mvcc and ts rather than ts
only (was: MVCC-sensitive semantics of versions)
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch, HBASE-15968-v2.patch,
> HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092870#comment-16092870 ] Phil Yang commented on HBASE-18390: --- I reset to the previous commit af359d03b5e2cc798cee8ba52d2a9fcbb1022104 it is also failed. So it is not broken by this issue. > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch, HBASE-18390.v02.patch, > HBASE-18390.v03.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18390: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to all branches (we have 7 branches now...), thanks all for review. > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch, HBASE-18390.v02.patch, > HBASE-18390.v03.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18390: -- Fix Version/s: 1.5.0 > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch, HBASE-18390.v02.patch, > HBASE-18390.v03.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18366) Fix flaky test hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta
[ https://issues.apache.org/jira/browse/HBASE-18366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092516#comment-16092516 ] Phil Yang commented on HBASE-18366: --- bq. why this logic is not included in AssignmentManager.processAssignQueue(). There is another scenario that master is not restarted and there is no RS crashed, but there is a RS with highest version coming in. We should move all system tables to this new RS. Does processAssignQueue can handle this? Thanks. > Fix flaky test > hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta > - > > Key: HBASE-18366 > URL: https://issues.apache.org/jira/browse/HBASE-18366 > Project: HBase > Issue Type: Bug >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Blocker > Fix For: 2.0.0 > > > It worked for a few days after enabling it with HBASE-18278. But started > failing after commits: > 6786b2b > 68436c9 > 75d2eca > 50bb045 > df93c13 > It works with one commit before: c5abb6c. Need to see what changed with those > commits. > Currently it fails with TableNotFoundException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092509#comment-16092509 ] Phil Yang commented on HBASE-18390: --- Yes, the main problem for this logic is we will always sleep 10 seconds, even the server has been in the dead list for several seconds. It will break our backoff sleeping retrying logic and the availability will be damaged. > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch, HBASE-18390.v02.patch, > HBASE-18390.v03.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091415#comment-16091415 ] Phil Yang commented on HBASE-18390: --- Any other concerns? Thanks > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch, HBASE-18390.v02.patch, > HBASE-18390.v03.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091208#comment-16091208 ] Phil Yang commented on HBASE-18390: --- I think findbugs warning is unrelated. And whitespace can be fixed while committing. bq. ConnectionUtils.addJitter is useless. Can we git rid of it? Yes, can remove it and its tests while committing. > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch, HBASE-18390.v02.patch, > HBASE-18390.v03.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18390: -- Attachment: HBASE-18390.v03.patch Remove same logic in RegionAdminServiceCallable > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch, HBASE-18390.v02.patch, > HBASE-18390.v03.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089615#comment-16089615 ] Phil Yang commented on HBASE-18390: --- I think it is HBASE-7590 > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch, HBASE-18390.v02.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18390: -- Attachment: HBASE-18390.v02.patch Trigger test in hbase-server. Have no idea how to simulate scanning meta failure. If we can simulate we can add a test here. > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch, HBASE-18390.v02.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18390: -- Fix Version/s: 2.0.0-alpha-2 3.0.0 > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18390: -- Affects Version/s: (was: 1.4.0) > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18390: -- Fix Version/s: 1.1.12 1.2.7 1.3.2 1.4.0 > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-2, 1.1.12 > > Attachments: HBASE-18390.v01.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18390: -- Affects Version/s: 1.4.0 1.3.1 1.2.6 1.1.11 2.0.0-alpha-1 > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0, 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-18390.v01.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18390: -- Description: If RegionServerCallable#prepare failed when getRegionLocation, the location in this callable object is null. And before we retry we will sleep. However, when location is null we will sleep at least 10 seconds. And the request will be failed directly if operation timeout is less than 10 seconds. I think it is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is ok for most cases. (was: If RegionServerCallable#prepare failed when getRegionLocation, the location in this callable object is null. And before we retry we will sleep. However, when location is null we will sleep at least 10 seconds. I think it is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is ok for most cases.) > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0, 1.3.1, 1.2.6, 1.1.11, 2.0.0-alpha-1 >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-18390.v01.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. And the request will > be failed directly if operation timeout is less than 10 seconds. I think it > is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is > ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18390: -- Status: Patch Available (was: Open) > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-18390.v01.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. I think it is no > need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is ok for > most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18390) Sleep too long when finding region location failed
[ https://issues.apache.org/jira/browse/HBASE-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18390: -- Attachment: HBASE-18390.v01.patch > Sleep too long when finding region location failed > -- > > Key: HBASE-18390 > URL: https://issues.apache.org/jira/browse/HBASE-18390 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-18390.v01.patch > > > If RegionServerCallable#prepare failed when getRegionLocation, the location > in this callable object is null. And before we retry we will sleep. However, > when location is null we will sleep at least 10 seconds. I think it is no > need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is ok for > most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18390) Sleep too long when finding region location failed
Phil Yang created HBASE-18390: - Summary: Sleep too long when finding region location failed Key: HBASE-18390 URL: https://issues.apache.org/jira/browse/HBASE-18390 Project: HBase Issue Type: Bug Reporter: Phil Yang Assignee: Phil Yang If RegionServerCallable#prepare failed when getRegionLocation, the location in this callable object is null. And before we retry we will sleep. However, when location is null we will sleep at least 10 seconds. I think it is no need to keep MIN_WAIT_DEAD_SERVER logic. Use backoff sleeping logic is ok for most cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18366) Fix flaky test hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta
[ https://issues.apache.org/jira/browse/HBASE-18366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088480#comment-16088480 ] Phil Yang commented on HBASE-18366: --- Thanks [~uagashe]. Reconsidered HBASE-17931, checkIfShouldMoveSystemRegionAsync may only needed when there is a region server startup. When RS stop or HMaster startup we can use a suitable Procedure to make sure we keep system tables in highest version RSs , right? > Fix flaky test > hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta > - > > Key: HBASE-18366 > URL: https://issues.apache.org/jira/browse/HBASE-18366 > Project: HBase > Issue Type: Bug >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Blocker > Fix For: 2.0.0 > > > It worked for a few days after enabling it with HBASE-18278. But started > failing after commits: > 6786b2b > 68436c9 > 75d2eca > 50bb045 > df93c13 > It works with one commit before: c5abb6c. Need to see what changed with those > commits. > Currently it fails with TableNotFoundException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18366) Fix flaky test hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta
[ https://issues.apache.org/jira/browse/HBASE-18366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083369#comment-16083369 ] Phil Yang commented on HBASE-18366: --- HBASE-17931 add a logic to check if there are region servers whose version is not same with others, if so we will move system tables to RS with highest version. But in all UTs our RSs are in same version so we may not have any difference with or without this commit. I run TestServerCrashProcedure in master branch locally several times but all passed. > Fix flaky test > hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta > - > > Key: HBASE-18366 > URL: https://issues.apache.org/jira/browse/HBASE-18366 > Project: HBase > Issue Type: Bug >Reporter: Umesh Agashe >Assignee: Umesh Agashe > > It worked for a few days after enabling it with HBASE-18278. But started > failing after commits: > 6786b2b > 68436c9 > 75d2eca > 50bb045 > df93c13 > It works with one commit before: c5abb6c. Need to see what changed with those > commits. > Currently it fails with TableNotFoundException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18334) Remove sync client implementation and wrap async client under sync client interface
[ https://issues.apache.org/jira/browse/HBASE-18334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18334: -- Description: Since 2.0 we have async client, now we have two client implementations. We can implement an sync client (Table interface) by using async client, getting a CompletableFuture and then waiting it done directly. This can reduce the maintenance work at client side in the future. Async client is almost done, we have test the performance and it showed it has same performance with sync client. In branch-2 we can keep old sync client implementations and remove it in master branch (since 3.0). was: Since 2.0 we have async client, now we have two client implementations. We can implement an sync client (Table interface) by using async client, getting a CompletableFuture and then waiting it done directly. This can reduce the maintenance work at client side in the furture. Async client is almost done, we have test the performance and it showed it has same performance with sync client. In branch-2 we can keep old sync client implementations and remove it in master branch (since 3.0). > Remove sync client implementation and wrap async client under sync client > interface > --- > > Key: HBASE-18334 > URL: https://issues.apache.org/jira/browse/HBASE-18334 > Project: HBase > Issue Type: Task >Reporter: Phil Yang > Fix For: 3.0.0 > > > Since 2.0 we have async client, now we have two client implementations. We > can implement an sync client (Table interface) by using async client, getting > a CompletableFuture and then waiting it done directly. This can reduce the > maintenance work at client side in the future. > Async client is almost done, we have test the performance and it showed it > has same performance with sync client. In branch-2 we can keep old sync > client implementations and remove it in master branch (since 3.0). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18334) Remove sync client implementation and wrap async client under sync client interface
[ https://issues.apache.org/jira/browse/HBASE-18334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18334: -- Description: Since 2.0 we have async client, now we have two client implementations. We can implement an sync client (Table interface) by using async client, getting a CompletableFuture and then waiting it done directly. This can reduce the maintenance work at client side in the furture. Async client is almost done, we have test the performance and it showed it has same performance with sync client. In branch-2 we can keep old sync client implementations and remove it in master branch (since 3.0). was: Since 2.0 we have async client, now we have two client implementations. We can implement an sync client (Table interface) by using async client, getting a CompletableFuture and then waiting it done directly. This can reduce the maintenance work at client side. Async client is almost done, we have test the performance and it showed it has same performance with sync client. In branch-2 we can keep old sync client implementations and remove it in master branch (since 3.0). > Remove sync client implementation and wrap async client under sync client > interface > --- > > Key: HBASE-18334 > URL: https://issues.apache.org/jira/browse/HBASE-18334 > Project: HBase > Issue Type: Task >Reporter: Phil Yang > Fix For: 3.0.0 > > > Since 2.0 we have async client, now we have two client implementations. We > can implement an sync client (Table interface) by using async client, getting > a CompletableFuture and then waiting it done directly. This can reduce the > maintenance work at client side in the furture. > Async client is almost done, we have test the performance and it showed it > has same performance with sync client. In branch-2 we can keep old sync > client implementations and remove it in master branch (since 3.0). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18334) Remove sync client implementation and wrap async client under sync client interface
[ https://issues.apache.org/jira/browse/HBASE-18334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18334: -- Description: Since 2.0 we have async client, now we have two client implementations. We can implement an sync client (Table interface) by using async client, getting a CompletableFuture and then waiting it done directly. This can reduce the maintenance work at client side in the future. Async client is almost done, we tested the performance and it showed it has same performance with sync client. In branch-2 we can keep old sync client implementations and remove it in master branch (since 3.0). was: Since 2.0 we have async client, now we have two client implementations. We can implement an sync client (Table interface) by using async client, getting a CompletableFuture and then waiting it done directly. This can reduce the maintenance work at client side in the future. Async client is almost done, we have test the performance and it showed it has same performance with sync client. In branch-2 we can keep old sync client implementations and remove it in master branch (since 3.0). > Remove sync client implementation and wrap async client under sync client > interface > --- > > Key: HBASE-18334 > URL: https://issues.apache.org/jira/browse/HBASE-18334 > Project: HBase > Issue Type: Task >Reporter: Phil Yang > Fix For: 3.0.0 > > > Since 2.0 we have async client, now we have two client implementations. We > can implement an sync client (Table interface) by using async client, getting > a CompletableFuture and then waiting it done directly. This can reduce the > maintenance work at client side in the future. > Async client is almost done, we tested the performance and it showed it has > same performance with sync client. In branch-2 we can keep old sync client > implementations and remove it in master branch (since 3.0). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18334) Remove sync client implementation and wrap async client under sync client interface
[ https://issues.apache.org/jira/browse/HBASE-18334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-18334: -- Description: Since 2.0 we have async client, now we have two client implementations. We can implement an sync client (Table interface) by using async client, getting a CompletableFuture and then waiting it done directly. This can reduce the maintenance work at client side. Async client is almost done, we have test the performance and it showed it has same performance with sync client. In branch-2 we can keep old sync client implementations and remove it in master branch (since 3.0). was: Since 2.0 we have async client, now we have two client implementations. We can implement an sync client by using async client, getting a CompletableFuture and then waiting it done directly. This can reduce the maintenance work at client side. Async client is almost done, we have test the performance and it showed it has same performance with sync client. In branch-2 we can keep old sync client implementations and remove it in master branch (since 3.0). > Remove sync client implementation and wrap async client under sync client > interface > --- > > Key: HBASE-18334 > URL: https://issues.apache.org/jira/browse/HBASE-18334 > Project: HBase > Issue Type: Task >Reporter: Phil Yang > Fix For: 3.0.0 > > > Since 2.0 we have async client, now we have two client implementations. We > can implement an sync client (Table interface) by using async client, getting > a CompletableFuture and then waiting it done directly. This can reduce the > maintenance work at client side. > Async client is almost done, we have test the performance and it showed it > has same performance with sync client. In branch-2 we can keep old sync > client implementations and remove it in master branch (since 3.0). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18334) Remove sync client implementation and wrap async client under sync client interface
Phil Yang created HBASE-18334: - Summary: Remove sync client implementation and wrap async client under sync client interface Key: HBASE-18334 URL: https://issues.apache.org/jira/browse/HBASE-18334 Project: HBase Issue Type: Task Reporter: Phil Yang Fix For: 3.0.0 Since 2.0 we have async client, now we have two client implementations. We can implement an sync client by using async client, getting a CompletableFuture and then waiting it done directly. This can reduce the maintenance work at client side. Async client is almost done, we have test the performance and it showed it has same performance with sync client. In branch-2 we can keep old sync client implementations and remove it in master branch (since 3.0). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077537#comment-16077537 ] Phil Yang commented on HBASE-17931: --- This test in the version before this commit also failed > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 3.0.0, 1.4.0, 2.0.0-alpha-2 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch, HBASE-17931.v04.patch, HBASE-17931.v04.patch, > HBASE-17931.v05.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076304#comment-16076304 ] Phil Yang commented on HBASE-17931: --- It passed in pre-commit build, how to determine this test is failed since #798? We only keep latest 5 builds and the last success build which is #791 > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 3.0.0, 1.4.0, 2.0.0-alpha-2 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch, HBASE-17931.v04.patch, HBASE-17931.v04.patch, > HBASE-17931.v05.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-1/branch-2/master. Thanks all for reviewing. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 3.0.0, 1.4.0, 2.0.0-alpha-2 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch, HBASE-17931.v04.patch, HBASE-17931.v04.patch, > HBASE-17931.v05.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Fix Version/s: (was: 2.0.0) 2.0.0-alpha-2 3.0.0 > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 3.0.0, 1.4.0, 2.0.0-alpha-2 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch, HBASE-17931.v04.patch, HBASE-17931.v04.patch, > HBASE-17931.v05.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076176#comment-16076176 ] Phil Yang commented on HBASE-17931: --- All tests passed in build 7508, the logic is almost same for master. If no objection I'll push to branch-2/master later. Thanks. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch, HBASE-17931.v04.patch, HBASE-17931.v04.patch, > HBASE-17931.v05.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Release Note: We usually keep compatibility between old client and new server so we can do rolling upgrade, HBase cluster first, then HBase client. But we don't guarantee new client can access old server. In an HBase cluster, we have system tables and region servers will access these tables so for servers they are also an HBase client. So if the system tables are in region servers with lower version we may get trouble because region servers with higher version may can not access them. After this patch, we will move all system regions to region servers with highest version. So when we do a rolling upgrade across two major or minor versions, we should ALWAYS UPGRADE MASTER FIRST and then upgrade region servers. The new master will handle system tables correctly. was: We usually keep compatibility between old client and new server so we can do rolling upgrade, HBase cluster first, then HBase client. But we don't guarantee new client can access old server. In an HBase cluster, we have system tables and region servers will access these tables so for servers they are also an HBase client. So if the system tables are in region servers with lower version we may get trouble because region servers with higher version may can not access them. After this patch, we will move all system regions to region servers with highest version. So when we do a rolling upgrade across two major or minor versions, we should upgrade master first and then upgrade region servers. The new master will handle system tables correctly. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch, HBASE-17931.v04.patch, HBASE-17931.v04.patch, > HBASE-17931.v05.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Release Note: We usually keep compatibility between old client and new server so we can do rolling upgrade, HBase cluster first, then HBase client. But we don't guarantee new client can access old server. In an HBase cluster, we have system tables and region servers will access these tables so for servers they are also an HBase client. So if the system tables are in region servers with lower version we may get trouble because region servers with higher version may can not access them. After this patch, we will move all system regions to region servers with highest version. So when we do a rolling upgrade across two major or minor versions, we should upgrade master first and then upgrade region servers. The new master will handle system tables correctly. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch, HBASE-17931.v04.patch, HBASE-17931.v04.patch, > HBASE-17931.v05.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.v05.patch Add a log info. The difference of the patches from branch-1 to master is we using AMv2 so the changing destination logic is in createMoveRegionProcedure. The other logics are same with branch-1. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch, HBASE-17931.v04.patch, HBASE-17931.v04.patch, > HBASE-17931.v05.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.v04.patch > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch, HBASE-17931.v04.patch, HBASE-17931.v04.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.v04.patch Patch for master/branch-2 > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch, HBASE-17931.v04.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071887#comment-16071887 ] Phil Yang commented on HBASE-17931: --- Yes. I'll push the patch to branch-1 and prepare a new patch for master/branch-2. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069895#comment-16069895 ] Phil Yang commented on HBASE-17931: --- Any comments for the latest branch-1 patch? Thanks. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18299) TestCompactionInDeadRegionServer is flaky
[ https://issues.apache.org/jira/browse/HBASE-18299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069616#comment-16069616 ] Phil Yang commented on HBASE-18299: --- [~Apache9] FYI, thanks > TestCompactionInDeadRegionServer is flaky > - > > Key: HBASE-18299 > URL: https://issues.apache.org/jira/browse/HBASE-18299 > Project: HBase > Issue Type: Test >Affects Versions: 2.0.0, 1.4.0 >Reporter: Phil Yang > > In this test, when master marks a RS dead and rename its WAL directory, we > expect the compaction throwing an exception. However, if RS finds the WAL is > not writable it will close itself and mark HRegion#areWritesEnabled false and > in compaction it will just skip and doesn't throw any exception. > See > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java#L1644 > and > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1964 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18299) TestCompactionInDeadRegionServer is flaky
Phil Yang created HBASE-18299: - Summary: TestCompactionInDeadRegionServer is flaky Key: HBASE-18299 URL: https://issues.apache.org/jira/browse/HBASE-18299 Project: HBase Issue Type: Test Affects Versions: 2.0.0, 1.4.0 Reporter: Phil Yang In this test, when master marks a RS dead and rename its WAL directory, we expect the compaction throwing an exception. However, if RS finds the WAL is not writable it will close itself and mark HRegion#areWritesEnabled false and in compaction it will just skip and doesn't throw any exception. See https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java#L1644 and https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L1964 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069608#comment-16069608 ] Phil Yang commented on HBASE-17931: --- Failing tests are not related and I think TestCompactionInDeadRegionServer has some bugs. File a new issue to handle it. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-14190) Assign system tables ahead of user region assignment
[
https://issues.apache.org/jira/browse/HBASE-14190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067981#comment-16067981
]
Phil Yang commented on HBASE-14190:
---
If I am not wrong, for system tables expect meta we still share logs with user
regions, so even if we have some logic to assign system tables first, we still
need wait log splitting done and we can not guarantee logs of system tables are
split first. So there may be a blocking task that we should make multi-log
feature having priority and write entries of system table in the highest
priority logs. Thanks.
> Assign system tables ahead of user region assignment
>
>
> Key: HBASE-14190
> URL: https://issues.apache.org/jira/browse/HBASE-14190
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 14190-system-wal-v1.txt, 14190-v12.4.txt, 14190-v12.txt
>
>
> Currently the namespace table region is assigned like user regions.
> I spent several hours working with a customer where master couldn't finish
> initialization.
> Even though master was restarted quite a few times, it went down with the
> following:
> {code}
> 2015-08-05 17:16:57,530 FATAL [hdpmaster1:6.activeMasterManager]
> master.HMaster: Master server abort: loaded coprocessors are: []
> 2015-08-05 17:16:57,530 FATAL [hdpmaster1:6.activeMasterManager]
> master.HMaster: Unhandled exception. Starting shutdown.
> java.io.IOException: Timedout 30ms waiting for namespace table to be
> assigned
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
> at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:985)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:779)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
> at java.lang.Thread.run(Thread.java:744)
> {code}
> During previous run(s), namespace table was created, hence leaving an entry
> in hbase:meta.
> The following if block in TableNamespaceManager#start() was skipped:
> {code}
> if (!MetaTableAccessor.tableExists(masterServices.getConnection(),
> TableName.NAMESPACE_TABLE_NAME)) {
> {code}
> TableNamespaceManager#start() spins, waiting for namespace region to be
> assigned.
> There was issue in master assigning user regions.
> We tried issuing 'assign' command from hbase shell which didn't work because
> of the following check in MasterRpcServices#assignRegion():
> {code}
> master.checkInitialized();
> {code}
> This scenario can be avoided if we assign hbase:namespace table after
> hbase:meta is assigned but before user table region assignment.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.branch-1.v06.patch > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: (was: HBASE-17931.branch-1.v06.patch) > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.v01.patch, HBASE-17931.v02.patch, HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.branch-1.v06.patch > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.branch-1.v06.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.branch-1.v05.patch > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.branch-1.v05.patch, > HBASE-17931.v01.patch, HBASE-17931.v02.patch, HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.branch-1.v05.patch Fix failed test and address reviews > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.branch-1.v05.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.branch-1.v04.patch Retry > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.branch-1.v04.patch, > HBASE-17931.v01.patch, HBASE-17931.v02.patch, HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060471#comment-16060471 ] Phil Yang commented on HBASE-17931: --- RB for branch-1, https://reviews.apache.org/r/60386 After AMv2, the patches for branch-1 and master will not be similar, so let's do them one by one. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.branch-1.v04.patch Unblock mater init and some trivial improvements. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060424#comment-16060424 ] Phil Yang edited comment on HBASE-17931 at 6/23/17 6:08 AM: Fix javadoc warning and failed test. was (Author: yangzhe1991): Fix javadoc warning and failed test. TestCompactionInDeadRegionServer failed because it runs a compaction after the region server is regarded as dead in master. But close log file may be delay so we should wait seconds. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.v01.patch, HBASE-17931.v02.patch, HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.branch-1.v03.patch > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.v01.patch, HBASE-17931.v02.patch, HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: (was: HBASE-17931.branch-1.v03.patch) > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.branch-1.v03.patch Fix javadoc warning and failed test. TestCompactionInDeadRegionServer failed because it runs a compaction after the region server is regarded as dead in master. But close log file may be delay so we should wait seconds. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.v01.patch, HBASE-17931.v02.patch, HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058792#comment-16058792
]
Phil Yang commented on HBASE-17125:
---
I think setVersions may still confused us and users. Call it
setMaxVersionsAfterFilters? And I think we should add some comments here to
tell users how we deal with cf's VERSIONS, filters and Scan#setMaxVersions
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: example.diff, HBASE-17125.master.001.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.003.patch, HBASE-17125.master.004.patch,
> HBASE-17125.master.005.patch, HBASE-17125.master.006.patch,
> HBASE-17125.master.007.patch, HBASE-17125.master.008.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.010.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18171) Scanning cursor for async client
[ https://issues.apache.org/jira/browse/HBASE-18171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16055457#comment-16055457 ] Phil Yang commented on HBASE-18171: --- No, I'm working on HBASE-17931 > Scanning cursor for async client > > > Key: HBASE-18171 > URL: https://issues.apache.org/jira/browse/HBASE-18171 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.branch-1.v02.patch > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-14614) Procedure v2: Core Assignment Manager
[
https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053604#comment-16053604
]
Phil Yang commented on HBASE-14614:
---
I see there is a comment:
{code:java|AssignmentManager.java}
// TODO: AMv2
// - handle region migration from hbase1 to hbase2.
{code}
Does it mean we can not do a rolling upgrade from 1.x to 2.0 right now?
> Procedure v2: Core Assignment Manager
> -
>
> Key: HBASE-14614
> URL: https://issues.apache.org/jira/browse/HBASE-14614
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Stephen Yuan Jiang
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: HBASE-14614.master.003.patch,
> HBASE-14614.master.004.patch, HBASE-14614.master.005.patch,
> HBASE-14614.master.006.patch, HBASE-14614.master.007.patch,
> HBASE-14614.master.008.patch, HBASE-14614.master.009.patch,
> HBASE-14614.master.010.patch, HBASE-14614.master.014.patch,
> HBASE-14614.master.015.patch, HBASE-14614.master.017.patch,
> HBASE-14614.master.018.patch, HBASE-14614.master.019.patch,
> HBASE-14614.master.020.patch, HBASE-14614.master.022.patch,
> HBASE-14614.master.023.patch, HBASE-14614.master.024.patch,
> HBASE-14614.master.025.patch, HBASE-14614.master.026.patch,
> HBASE-14614.master.027.patch, HBASE-14614.master.028.patch,
> HBASE-14614.master.029.patch, HBASE-14614.master.030.patch,
> HBASE-14614.master.033.patch, HBASE-14614.master.038.patch,
> HBASE-14614.master.039.patch, HBASE-14614.master.040.patch,
> HBASE-14614.master.041.patch, HBASE-14614.master.042.patch,
> HBASE-14614.master.043.patch, HBASE-14614.master.044.patch,
> HBASE-14614.master.045.patch, HBASE-14614.master.045.patch,
> HBASE-14614.master.046.patch, HBASE-14614.master.047.patch,
> HBASE-14614.master.048.patch, HBASE-14614.master.049.patch,
> HBASE-14614.master.050.patch, HBASE-14614.master.051.patch
>
>
> New AssignmentManager implemented using proc-v2.
> - AssignProcedure handle assignment operation
> - UnassignProcedure handle unassign operation
> - MoveRegionProcedure handle move/balance operation
> Concurrent Assign operations are batched together and sent to the balancer
> Concurrent Assign and Unassign operation ready to be sent to the RS are
> batched together
> This patch is an intermediate state where we add the new AM as
> AssignmentManager2() to the master, to be reached by tests. but the new AM
> will not be integrated with the rest of the system. Only new am unit-tests
> will exercise the new assigment manager. The integration with the master code
> is part of HBASE-14616
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.branch-1.v01.patch Rebased v03 patch for branch-1, branch-1 uses old AM > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, HBASE-17931.v01.patch, > HBASE-17931.v02.patch, HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18217) Correct comment errors in TestScannerHeartbeatMessages
[
https://issues.apache.org/jira/browse/HBASE-18217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16049971#comment-16049971
]
Phil Yang commented on HBASE-18217:
---
Time limit is half of timeout, see RSRpcServices
> Correct comment errors in TestScannerHeartbeatMessages
> --
>
> Key: HBASE-18217
> URL: https://issues.apache.org/jira/browse/HBASE-18217
> Project: HBase
> Issue Type: Bug
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
>
> 1. For CLIENT_TIMEOUT
> {code}
> // Time, in milliseconds, that the client will wait for a response from the
> server before timing
> // out. This value is used server side to determine when it is necessary to
> send a heartbeat
> // message to the client. Time limit will be 500 ms.
> private static int CLIENT_TIMEOUT = 1000;
> {code}
> I think “Time limit will be 500 ms.” should be removed
> 2. For DEFAULT_ROW_SLEEP_TIME
> Typo, "same" -> "some", in
> // In this test, we sleep after reading each row. So we should make sure
> after we get some number
> // of rows and sleep {color:red}same{color} times we must reach time limit,
> and do not timeout after next sleeping.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042489#comment-16042489 ] Phil Yang edited comment on HBASE-17931 at 6/8/17 10:02 AM: 2.0.0-alpha will be released soon so back to this issue. org/apache/hadoop/hbase/master/AssignmentManager.java is gone so I'm going to prepare a new patch. was (Author: yangzhe1991): 2.0.0-alpha will be released soon so back to this issue. AssignmentManager is gone so I'm going to prepare a new patch. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042489#comment-16042489 ] Phil Yang commented on HBASE-17931: --- 2.0.0-alpha will be released soon so back to this issue. AssignmentManager is gone so I'm going to prepare a new patch. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15302) Reenable the other tests disabled by HBASE-14678
[ https://issues.apache.org/jira/browse/HBASE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15302: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Close this issue because we have a commit to reenable some tests. File a new issue if we still miss some tests. > Reenable the other tests disabled by HBASE-14678 > > > Key: HBASE-15302 > URL: https://issues.apache.org/jira/browse/HBASE-15302 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7 > > Attachments: HBASE-15302-branch-1.3-append-v1.patch, > HBASE-15302-branch-1.3-append-v1.patch, HBASE-15302-branch-1-append-v1.patch, > HBASE-15302-branch-1-v1.patch, HBASE-15302-v1.txt, HBASE-15302-v1.txt > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15576) Scanning cursor to prevent blocking long time on ResultScanner.next()
[
https://issues.apache.org/jira/browse/HBASE-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15576:
--
Release Note: If you don't like scanning being blocked too long because of
heartbeat and partial result, you can use Scan#setNeedCursorResult(true) to get
a special result within scanning timeout setting time which will tell you where
row the server is scanning. See its javadoc for more details. (was: If you
don't like scanning being blocked too long because of heartbeat and partial
result, you can use Scan#setNeedCursorResult(true) to get a special result
within scanning timeout setting time. See its javadoc for more details.)
> Scanning cursor to prevent blocking long time on ResultScanner.next()
> -
>
> Key: HBASE-15576
> URL: https://issues.apache.org/jira/browse/HBASE-15576
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15576.branch-1.v01.patch,
> HBASE-15576.branch-1.v01.patch, HBASE-15576.branch-1.v01.patch,
> HBASE-15576.branch-1.v02.patch, HBASE-15576.v01.patch, HBASE-15576.v02.patch,
> HBASE-15576.v03.patch, HBASE-15576.v03.patch, HBASE-15576.v04.patch,
> HBASE-15576.v04.patch, HBASE-15576.v05.patch, HBASE-15576.v06.patch,
> HBASE-15576.v07.patch
>
>
> After 1.1.0 released, we have partial and heartbeat protocol in scanning to
> prevent responding large data or timeout. Now for ResultScanner.next(), we
> may block for longer time larger than timeout settings to get a Result if the
> row is very large, or filter is sparse, or there are too many delete markers
> in files.
> However, in some scenes, we don't want it to be blocked for too long. For
> example, a web service which handles requests from mobile devices whose
> network is not stable and we can not set timeout too long(eg. only 5 seconds)
> between mobile and web service. This service will scan rows from HBase and
> return it to mobile devices. In this scene, the simplest way is to make the
> web service stateless. Apps in mobile devices will send several requests one
> by one to get the data until enough just like paging a list. In each request
> it will carry a start position which depends on the last result from web
> service. Different requests can be sent to different web service server
> because it is stateless.
> Therefore, the stateless web service need a cursor from HBase telling where
> we have scanned in RegionScanner when HBase client receives an empty
> heartbeat. And the service will return the cursor to mobile device although
> the response has no data. In next request we can start at the position of
> cursor, without the cursor we have to scan from last returned result and we
> may timeout forever. And of course even if the heartbeat message is not empty
> we can still use cursor to prevent re-scan the same rows/cells which has beed
> skipped.
> Obviously, we will give up consistency for scanning because even HBase client
> is also stateless, but it is acceptable in this scene. And maybe we can keep
> mvcc in cursor so we can get a consistent view?
> HBASE-13099 had some discussion, but it has no further progress by now.
> API:
> In Scan we need a new method setNeedCursorResult(true) to get the cursor row
> key when there is a RPC response but client can not return any Result. In
> this mode we will not block ResultScanner.next() longer than this timeout
> setting.
> {code}
> while (r = scanner.next() && r != null) {
> if(r.isCursor()){
> // scanning is not end, it is a cursor, save its row key and close scanner
> if you want, or
> // just continue the loop to call next().
> } else {
> // just like before
> }
> }
> // scanning is end
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15576) Scanning cursor to prevent blocking long time on ResultScanner.next()
[
https://issues.apache.org/jira/browse/HBASE-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15576:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Release Note: If you don't like scanning being blocked too long because of
heartbeat and partial result, you can use Scan#setNeedCursorResult(true) to get
a special result within scanning timeout setting time. See its javadoc for more
details.
Status: Resolved (was: Patch Available)
Pushed to master and branch-1, thanks [~Apache9] for review.
> Scanning cursor to prevent blocking long time on ResultScanner.next()
> -
>
> Key: HBASE-15576
> URL: https://issues.apache.org/jira/browse/HBASE-15576
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15576.branch-1.v01.patch,
> HBASE-15576.branch-1.v01.patch, HBASE-15576.branch-1.v01.patch,
> HBASE-15576.branch-1.v02.patch, HBASE-15576.v01.patch, HBASE-15576.v02.patch,
> HBASE-15576.v03.patch, HBASE-15576.v03.patch, HBASE-15576.v04.patch,
> HBASE-15576.v04.patch, HBASE-15576.v05.patch, HBASE-15576.v06.patch,
> HBASE-15576.v07.patch
>
>
> After 1.1.0 released, we have partial and heartbeat protocol in scanning to
> prevent responding large data or timeout. Now for ResultScanner.next(), we
> may block for longer time larger than timeout settings to get a Result if the
> row is very large, or filter is sparse, or there are too many delete markers
> in files.
> However, in some scenes, we don't want it to be blocked for too long. For
> example, a web service which handles requests from mobile devices whose
> network is not stable and we can not set timeout too long(eg. only 5 seconds)
> between mobile and web service. This service will scan rows from HBase and
> return it to mobile devices. In this scene, the simplest way is to make the
> web service stateless. Apps in mobile devices will send several requests one
> by one to get the data until enough just like paging a list. In each request
> it will carry a start position which depends on the last result from web
> service. Different requests can be sent to different web service server
> because it is stateless.
> Therefore, the stateless web service need a cursor from HBase telling where
> we have scanned in RegionScanner when HBase client receives an empty
> heartbeat. And the service will return the cursor to mobile device although
> the response has no data. In next request we can start at the position of
> cursor, without the cursor we have to scan from last returned result and we
> may timeout forever. And of course even if the heartbeat message is not empty
> we can still use cursor to prevent re-scan the same rows/cells which has beed
> skipped.
> Obviously, we will give up consistency for scanning because even HBase client
> is also stateless, but it is acceptable in this scene. And maybe we can keep
> mvcc in cursor so we can get a consistent view?
> HBASE-13099 had some discussion, but it has no further progress by now.
> API:
> In Scan we need a new method setNeedCursorResult(true) to get the cursor row
> key when there is a RPC response but client can not return any Result. In
> this mode we will not block ResultScanner.next() longer than this timeout
> setting.
> {code}
> while (r = scanner.next() && r != null) {
> if(r.isCursor()){
> // scanning is not end, it is a cursor, save its row key and close scanner
> if you want, or
> // just continue the loop to call next().
> } else {
> // just like before
> }
> }
> // scanning is end
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-18171) Scanning cursor for async client
Phil Yang created HBASE-18171: - Summary: Scanning cursor for async client Key: HBASE-18171 URL: https://issues.apache.org/jira/browse/HBASE-18171 Project: HBase Issue Type: New Feature Reporter: Phil Yang Assignee: Phil Yang -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15576) Scanning cursor to prevent blocking long time on ResultScanner.next()
[
https://issues.apache.org/jira/browse/HBASE-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15576:
--
Attachment: HBASE-15576.v07.patch
> Scanning cursor to prevent blocking long time on ResultScanner.next()
> -
>
> Key: HBASE-15576
> URL: https://issues.apache.org/jira/browse/HBASE-15576
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15576.branch-1.v01.patch,
> HBASE-15576.branch-1.v01.patch, HBASE-15576.branch-1.v01.patch,
> HBASE-15576.branch-1.v02.patch, HBASE-15576.v01.patch, HBASE-15576.v02.patch,
> HBASE-15576.v03.patch, HBASE-15576.v03.patch, HBASE-15576.v04.patch,
> HBASE-15576.v04.patch, HBASE-15576.v05.patch, HBASE-15576.v06.patch,
> HBASE-15576.v07.patch
>
>
> After 1.1.0 released, we have partial and heartbeat protocol in scanning to
> prevent responding large data or timeout. Now for ResultScanner.next(), we
> may block for longer time larger than timeout settings to get a Result if the
> row is very large, or filter is sparse, or there are too many delete markers
> in files.
> However, in some scenes, we don't want it to be blocked for too long. For
> example, a web service which handles requests from mobile devices whose
> network is not stable and we can not set timeout too long(eg. only 5 seconds)
> between mobile and web service. This service will scan rows from HBase and
> return it to mobile devices. In this scene, the simplest way is to make the
> web service stateless. Apps in mobile devices will send several requests one
> by one to get the data until enough just like paging a list. In each request
> it will carry a start position which depends on the last result from web
> service. Different requests can be sent to different web service server
> because it is stateless.
> Therefore, the stateless web service need a cursor from HBase telling where
> we have scanned in RegionScanner when HBase client receives an empty
> heartbeat. And the service will return the cursor to mobile device although
> the response has no data. In next request we can start at the position of
> cursor, without the cursor we have to scan from last returned result and we
> may timeout forever. And of course even if the heartbeat message is not empty
> we can still use cursor to prevent re-scan the same rows/cells which has beed
> skipped.
> Obviously, we will give up consistency for scanning because even HBase client
> is also stateless, but it is acceptable in this scene. And maybe we can keep
> mvcc in cursor so we can get a consistent view?
> HBASE-13099 had some discussion, but it has no further progress by now.
> API:
> In Scan we need a new method setNeedCursorResult(true) to get the cursor row
> key when there is a RPC response but client can not return any Result. In
> this mode we will not block ResultScanner.next() longer than this timeout
> setting.
> {code}
> while (r = scanner.next() && r != null) {
> if(r.isCursor()){
> // scanning is not end, it is a cursor, save its row key and close scanner
> if you want, or
> // just continue the loop to call next().
> } else {
> // just like before
> }
> }
> // scanning is end
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15576) Scanning cursor to prevent blocking long time on ResultScanner.next()
[
https://issues.apache.org/jira/browse/HBASE-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15576:
--
Attachment: HBASE-15576.branch-1.v02.patch
> Scanning cursor to prevent blocking long time on ResultScanner.next()
> -
>
> Key: HBASE-15576
> URL: https://issues.apache.org/jira/browse/HBASE-15576
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15576.branch-1.v01.patch,
> HBASE-15576.branch-1.v01.patch, HBASE-15576.branch-1.v01.patch,
> HBASE-15576.branch-1.v02.patch, HBASE-15576.v01.patch, HBASE-15576.v02.patch,
> HBASE-15576.v03.patch, HBASE-15576.v03.patch, HBASE-15576.v04.patch,
> HBASE-15576.v04.patch, HBASE-15576.v05.patch, HBASE-15576.v06.patch
>
>
> After 1.1.0 released, we have partial and heartbeat protocol in scanning to
> prevent responding large data or timeout. Now for ResultScanner.next(), we
> may block for longer time larger than timeout settings to get a Result if the
> row is very large, or filter is sparse, or there are too many delete markers
> in files.
> However, in some scenes, we don't want it to be blocked for too long. For
> example, a web service which handles requests from mobile devices whose
> network is not stable and we can not set timeout too long(eg. only 5 seconds)
> between mobile and web service. This service will scan rows from HBase and
> return it to mobile devices. In this scene, the simplest way is to make the
> web service stateless. Apps in mobile devices will send several requests one
> by one to get the data until enough just like paging a list. In each request
> it will carry a start position which depends on the last result from web
> service. Different requests can be sent to different web service server
> because it is stateless.
> Therefore, the stateless web service need a cursor from HBase telling where
> we have scanned in RegionScanner when HBase client receives an empty
> heartbeat. And the service will return the cursor to mobile device although
> the response has no data. In next request we can start at the position of
> cursor, without the cursor we have to scan from last returned result and we
> may timeout forever. And of course even if the heartbeat message is not empty
> we can still use cursor to prevent re-scan the same rows/cells which has beed
> skipped.
> Obviously, we will give up consistency for scanning because even HBase client
> is also stateless, but it is acceptable in this scene. And maybe we can keep
> mvcc in cursor so we can get a consistent view?
> HBASE-13099 had some discussion, but it has no further progress by now.
> API:
> In Scan we need a new method setNeedCursorResult(true) to get the cursor row
> key when there is a RPC response but client can not return any Result. In
> this mode we will not block ResultScanner.next() longer than this timeout
> setting.
> {code}
> while (r = scanner.next() && r != null) {
> if(r.isCursor()){
> // scanning is not end, it is a cursor, save its row key and close scanner
> if you want, or
> // just continue the loop to call next().
> } else {
> // just like before
> }
> }
> // scanning is end
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15576) Scanning cursor to prevent blocking long time on ResultScanner.next()
[
https://issues.apache.org/jira/browse/HBASE-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15576:
--
Attachment: HBASE-15576.branch-1.v01.patch
> Scanning cursor to prevent blocking long time on ResultScanner.next()
> -
>
> Key: HBASE-15576
> URL: https://issues.apache.org/jira/browse/HBASE-15576
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15576.branch-1.v01.patch,
> HBASE-15576.branch-1.v01.patch, HBASE-15576.branch-1.v01.patch,
> HBASE-15576.v01.patch, HBASE-15576.v02.patch, HBASE-15576.v03.patch,
> HBASE-15576.v03.patch, HBASE-15576.v04.patch, HBASE-15576.v04.patch,
> HBASE-15576.v05.patch, HBASE-15576.v06.patch
>
>
> After 1.1.0 released, we have partial and heartbeat protocol in scanning to
> prevent responding large data or timeout. Now for ResultScanner.next(), we
> may block for longer time larger than timeout settings to get a Result if the
> row is very large, or filter is sparse, or there are too many delete markers
> in files.
> However, in some scenes, we don't want it to be blocked for too long. For
> example, a web service which handles requests from mobile devices whose
> network is not stable and we can not set timeout too long(eg. only 5 seconds)
> between mobile and web service. This service will scan rows from HBase and
> return it to mobile devices. In this scene, the simplest way is to make the
> web service stateless. Apps in mobile devices will send several requests one
> by one to get the data until enough just like paging a list. In each request
> it will carry a start position which depends on the last result from web
> service. Different requests can be sent to different web service server
> because it is stateless.
> Therefore, the stateless web service need a cursor from HBase telling where
> we have scanned in RegionScanner when HBase client receives an empty
> heartbeat. And the service will return the cursor to mobile device although
> the response has no data. In next request we can start at the position of
> cursor, without the cursor we have to scan from last returned result and we
> may timeout forever. And of course even if the heartbeat message is not empty
> we can still use cursor to prevent re-scan the same rows/cells which has beed
> skipped.
> Obviously, we will give up consistency for scanning because even HBase client
> is also stateless, but it is acceptable in this scene. And maybe we can keep
> mvcc in cursor so we can get a consistent view?
> HBASE-13099 had some discussion, but it has no further progress by now.
> API:
> In Scan we need a new method setNeedCursorResult(true) to get the cursor row
> key when there is a RPC response but client can not return any Result. In
> this mode we will not block ResultScanner.next() longer than this timeout
> setting.
> {code}
> while (r = scanner.next() && r != null) {
> if(r.isCursor()){
> // scanning is not end, it is a cursor, save its row key and close scanner
> if you want, or
> // just continue the loop to call next().
> } else {
> // just like before
> }
> }
> // scanning is end
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-8770) deletes and puts with the same ts should be resolved according to mvcc/seqNum
[ https://issues.apache.org/jira/browse/HBASE-8770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038097#comment-16038097 ] Phil Yang edited comment on HBASE-8770 at 6/6/17 4:23 AM: -- bq. If not, would simply changing the sort order to order by seq id before cell type for this issue be a possibility for 2.0? We will remove mvcc in HFile in minor compaction (to save capacity) and delete/put will have same mvcc if they are in one same file. If I am not wrong we can not just change the comparator :( At least we should always keep mvcc in HFiles just like HBASE-15968 does. bq. Is there hope for HBASE-15968 to make it into 2.0? The patch is available and I think we can get it in after rebasing and reviewing. But we need more tests (I am testing in our own cluster) and I am not sure if it can be a default behavior because the performance will be a little worse and a new feature may have some bugs... And I am not sure if the patch is still right if we compare mvcc first before type. Need more checking. was (Author: yangzhe1991): bq. If not, would simply changing the sort order to order by seq id before cell type for this issue be a possibility for 2.0? We will remove mvcc in HFile in minor compaction (to save capacity) and delete/put will have same mvcc if they are in one same file. If I am not wrong we can not just change the comparator :( At least we should always keep mvcc in HFiles just like HBASE-15968 does. bq. Is there hope for HBASE-15968 to make it into 2.0? The patch is available and I think we can get it in after rebasing and reviewing. But we need more tests (I am testing in our own cluster) and I am not sure if it can be a default behavior because the performance will be a little worse and a new feature may have some bugs... So I prefer to always keep mvcc in HFile by default, compare mvcc before type by default, but the other behavior is configurable. > deletes and puts with the same ts should be resolved according to mvcc/seqNum > - > > Key: HBASE-8770 > URL: https://issues.apache.org/jira/browse/HBASE-8770 > Project: HBase > Issue Type: Brainstorming >Reporter: Sergey Shelukhin >Priority: Critical > > This came up during HBASE-8721. Puts with the same ts are resolved by seqNum. > It's not clear why deletes with the same ts as a put should always mask the > put, rather than also being resolve by seqNum. > What do you think? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-8770) deletes and puts with the same ts should be resolved according to mvcc/seqNum
[ https://issues.apache.org/jira/browse/HBASE-8770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038097#comment-16038097 ] Phil Yang commented on HBASE-8770: -- bq. If not, would simply changing the sort order to order by seq id before cell type for this issue be a possibility for 2.0? We will remove mvcc in HFile in minor compaction (to save capacity) and delete/put will have same mvcc if they are in one same file. If I am not wrong we can not just change the comparator :( At least we should always keep mvcc in HFiles just like HBASE-15968 does. bq. Is there hope for HBASE-15968 to make it into 2.0? The patch is available and I think we can get it in after rebasing and reviewing. But we need more tests (I am testing in our own cluster) and I am not sure if it can be a default behavior because the performance will be a little worse and a new feature may have some bugs... So I prefer to always keep mvcc in HFile by default, compare mvcc before type by default, but the other behavior is configurable. > deletes and puts with the same ts should be resolved according to mvcc/seqNum > - > > Key: HBASE-8770 > URL: https://issues.apache.org/jira/browse/HBASE-8770 > Project: HBase > Issue Type: Brainstorming >Reporter: Sergey Shelukhin >Priority: Critical > > This came up during HBASE-8721. Puts with the same ts are resolved by seqNum. > It's not clear why deletes with the same ts as a put should always mask the > put, rather than also being resolve by seqNum. > What do you think? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15576) Scanning cursor to prevent blocking long time on ResultScanner.next()
[
https://issues.apache.org/jira/browse/HBASE-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15576:
--
Attachment: HBASE-15576.branch-1.v01.patch
> Scanning cursor to prevent blocking long time on ResultScanner.next()
> -
>
> Key: HBASE-15576
> URL: https://issues.apache.org/jira/browse/HBASE-15576
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15576.branch-1.v01.patch,
> HBASE-15576.branch-1.v01.patch, HBASE-15576.v01.patch, HBASE-15576.v02.patch,
> HBASE-15576.v03.patch, HBASE-15576.v03.patch, HBASE-15576.v04.patch,
> HBASE-15576.v04.patch, HBASE-15576.v05.patch, HBASE-15576.v06.patch
>
>
> After 1.1.0 released, we have partial and heartbeat protocol in scanning to
> prevent responding large data or timeout. Now for ResultScanner.next(), we
> may block for longer time larger than timeout settings to get a Result if the
> row is very large, or filter is sparse, or there are too many delete markers
> in files.
> However, in some scenes, we don't want it to be blocked for too long. For
> example, a web service which handles requests from mobile devices whose
> network is not stable and we can not set timeout too long(eg. only 5 seconds)
> between mobile and web service. This service will scan rows from HBase and
> return it to mobile devices. In this scene, the simplest way is to make the
> web service stateless. Apps in mobile devices will send several requests one
> by one to get the data until enough just like paging a list. In each request
> it will carry a start position which depends on the last result from web
> service. Different requests can be sent to different web service server
> because it is stateless.
> Therefore, the stateless web service need a cursor from HBase telling where
> we have scanned in RegionScanner when HBase client receives an empty
> heartbeat. And the service will return the cursor to mobile device although
> the response has no data. In next request we can start at the position of
> cursor, without the cursor we have to scan from last returned result and we
> may timeout forever. And of course even if the heartbeat message is not empty
> we can still use cursor to prevent re-scan the same rows/cells which has beed
> skipped.
> Obviously, we will give up consistency for scanning because even HBase client
> is also stateless, but it is acceptable in this scene. And maybe we can keep
> mvcc in cursor so we can get a consistent view?
> HBASE-13099 had some discussion, but it has no further progress by now.
> API:
> In Scan we need a new method setNeedCursorResult(true) to get the cursor row
> key when there is a RPC response but client can not return any Result. In
> this mode we will not block ResultScanner.next() longer than this timeout
> setting.
> {code}
> while (r = scanner.next() && r != null) {
> if(r.isCursor()){
> // scanning is not end, it is a cursor, save its row key and close scanner
> if you want, or
> // just continue the loop to call next().
> } else {
> // just like before
> }
> }
> // scanning is end
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15576) Scanning cursor to prevent blocking long time on ResultScanner.next()
[
https://issues.apache.org/jira/browse/HBASE-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15576:
--
Attachment: HBASE-15576.branch-1.v01.patch
> Scanning cursor to prevent blocking long time on ResultScanner.next()
> -
>
> Key: HBASE-15576
> URL: https://issues.apache.org/jira/browse/HBASE-15576
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15576.branch-1.v01.patch, HBASE-15576.v01.patch,
> HBASE-15576.v02.patch, HBASE-15576.v03.patch, HBASE-15576.v03.patch,
> HBASE-15576.v04.patch, HBASE-15576.v04.patch, HBASE-15576.v05.patch,
> HBASE-15576.v06.patch
>
>
> After 1.1.0 released, we have partial and heartbeat protocol in scanning to
> prevent responding large data or timeout. Now for ResultScanner.next(), we
> may block for longer time larger than timeout settings to get a Result if the
> row is very large, or filter is sparse, or there are too many delete markers
> in files.
> However, in some scenes, we don't want it to be blocked for too long. For
> example, a web service which handles requests from mobile devices whose
> network is not stable and we can not set timeout too long(eg. only 5 seconds)
> between mobile and web service. This service will scan rows from HBase and
> return it to mobile devices. In this scene, the simplest way is to make the
> web service stateless. Apps in mobile devices will send several requests one
> by one to get the data until enough just like paging a list. In each request
> it will carry a start position which depends on the last result from web
> service. Different requests can be sent to different web service server
> because it is stateless.
> Therefore, the stateless web service need a cursor from HBase telling where
> we have scanned in RegionScanner when HBase client receives an empty
> heartbeat. And the service will return the cursor to mobile device although
> the response has no data. In next request we can start at the position of
> cursor, without the cursor we have to scan from last returned result and we
> may timeout forever. And of course even if the heartbeat message is not empty
> we can still use cursor to prevent re-scan the same rows/cells which has beed
> skipped.
> Obviously, we will give up consistency for scanning because even HBase client
> is also stateless, but it is acceptable in this scene. And maybe we can keep
> mvcc in cursor so we can get a consistent view?
> HBASE-13099 had some discussion, but it has no further progress by now.
> API:
> In Scan we need a new method setNeedCursorResult(true) to get the cursor row
> key when there is a RPC response but client can not return any Result. In
> this mode we will not block ResultScanner.next() longer than this timeout
> setting.
> {code}
> while (r = scanner.next() && r != null) {
> if(r.isCursor()){
> // scanning is not end, it is a cursor, save its row key and close scanner
> if you want, or
> // just continue the loop to call next().
> } else {
> // just like before
> }
> }
> // scanning is end
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15576) Scanning cursor to prevent blocking long time on ResultScanner.next()
[
https://issues.apache.org/jira/browse/HBASE-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16036480#comment-16036480
]
Phil Yang commented on HBASE-15576:
---
All UTs pass. Any comments? Thanks.
> Scanning cursor to prevent blocking long time on ResultScanner.next()
> -
>
> Key: HBASE-15576
> URL: https://issues.apache.org/jira/browse/HBASE-15576
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15576.v01.patch, HBASE-15576.v02.patch,
> HBASE-15576.v03.patch, HBASE-15576.v03.patch, HBASE-15576.v04.patch,
> HBASE-15576.v04.patch, HBASE-15576.v05.patch, HBASE-15576.v06.patch
>
>
> After 1.1.0 released, we have partial and heartbeat protocol in scanning to
> prevent responding large data or timeout. Now for ResultScanner.next(), we
> may block for longer time larger than timeout settings to get a Result if the
> row is very large, or filter is sparse, or there are too many delete markers
> in files.
> However, in some scenes, we don't want it to be blocked for too long. For
> example, a web service which handles requests from mobile devices whose
> network is not stable and we can not set timeout too long(eg. only 5 seconds)
> between mobile and web service. This service will scan rows from HBase and
> return it to mobile devices. In this scene, the simplest way is to make the
> web service stateless. Apps in mobile devices will send several requests one
> by one to get the data until enough just like paging a list. In each request
> it will carry a start position which depends on the last result from web
> service. Different requests can be sent to different web service server
> because it is stateless.
> Therefore, the stateless web service need a cursor from HBase telling where
> we have scanned in RegionScanner when HBase client receives an empty
> heartbeat. And the service will return the cursor to mobile device although
> the response has no data. In next request we can start at the position of
> cursor, without the cursor we have to scan from last returned result and we
> may timeout forever. And of course even if the heartbeat message is not empty
> we can still use cursor to prevent re-scan the same rows/cells which has beed
> skipped.
> Obviously, we will give up consistency for scanning because even HBase client
> is also stateless, but it is acceptable in this scene. And maybe we can keep
> mvcc in cursor so we can get a consistent view?
> HBASE-13099 had some discussion, but it has no further progress by now.
> API:
> In Scan we need a new method setNeedCursorResult(true) to get the cursor row
> key when there is a RPC response but client can not return any Result. In
> this mode we will not block ResultScanner.next() longer than this timeout
> setting.
> {code}
> while (r = scanner.next() && r != null) {
> if(r.isCursor()){
> // scanning is not end, it is a cursor, save its row key and close scanner
> if you want, or
> // just continue the loop to call next().
> } else {
> // just like before
> }
> }
> // scanning is end
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-8770) deletes and puts with the same ts should be resolved according to mvcc/seqNum
[ https://issues.apache.org/jira/browse/HBASE-8770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16036302#comment-16036302 ] Phil Yang commented on HBASE-8770: -- We are fixing HBASE-15968 which is more complicated because we have delete versions which will delete all Puts whose ts is not larger than it. So the delete marker and put may not be same. But after HBASE-15968 being fixed this issue will be also fixed. > deletes and puts with the same ts should be resolved according to mvcc/seqNum > - > > Key: HBASE-8770 > URL: https://issues.apache.org/jira/browse/HBASE-8770 > Project: HBase > Issue Type: Brainstorming >Reporter: Sergey Shelukhin >Priority: Critical > > This came up during HBASE-8721. Puts with the same ts are resolved by seqNum. > It's not clear why deletes with the same ts as a put should always mask the > put, rather than also being resolve by seqNum. > What do you think? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15160) Put back HFile's HDFS op latency sampling code and add metrics for monitoring
[ https://issues.apache.org/jira/browse/HBASE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034517#comment-16034517 ] Phil Yang commented on HBASE-15160: --- File a new issue HBASE-18151 to fix the Histogram problems. I think we can use ms-level here before we fix the issue. > Put back HFile's HDFS op latency sampling code and add metrics for monitoring > - > > Key: HBASE-15160 > URL: https://issues.apache.org/jira/browse/HBASE-15160 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0, 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Critical > Attachments: HBASE-15160.patch, HBASE-15160_v2.patch, > HBASE-15160_v3.patch, hbase-15160_v4.patch, hbase-15160_v5.patch, > hbase-15160_v6.patch, hbase-15160_v7.patch > > > In HBASE-11586 all HDFS op latency sampling code, including fsReadLatency, > fsPreadLatency and fsWriteLatency, have been removed. There was some > discussion about putting them back in a new JIRA but never happened. > According to our experience, these metrics are useful to judge whether issue > lies on HDFS when slow request occurs, so we propose to put them back in this > JIRA, and add the metrics for monitoring as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18151) Improvement on Histogram
Phil Yang created HBASE-18151: - Summary: Improvement on Histogram Key: HBASE-18151 URL: https://issues.apache.org/jira/browse/HBASE-18151 Project: HBase Issue Type: Improvement Reporter: Phil Yang There are some bugs and some places we can improvement in the new native Histogram. The default maxExpected value in HistogramImpl is Integer.MAX_VALUE << 2 (https://github.com/apache/hbase/blob/master/hbase-metrics/src/main/java/org/apache/hadoop/hbase/metrics/impl/HistogramImpl.java#L41 ) which is overflow... and will be regarded as -4... And binsMax pf FastLongHistogram will be 255(by the reason of this bug) which is too low (In FastLongHistogram#L75 it says the expected binsMax is 1000? I think it may be also too low. Even if there is no overflow, Integer.MAX_VALUE*4 is too large? We may need pass the right expected min/max value when we use it because the value may be different in scenarios. And the default value should not be 1000 or Integer.MAX_VALUE*4. There is an assumption that the value is in uniform distribution. But if the value is latency which is the most use case, the distribution is not uniform. The lower the value, the higher the probability. Because of these reasons, the 75/90/99 or other percentages may have big error than we expected. We need find a way to reduce the error. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15160) Put back HFile's HDFS op latency sampling code and add metrics for monitoring
[ https://issues.apache.org/jira/browse/HBASE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034497#comment-16034497 ] Phil Yang commented on HBASE-15160: --- [~carp84] I think the performance issue is not because the performance of nanoTime (although it is a little slower than currentTimeMillis), the issue is Histogram use several buckets to estimate percentile and it assums a uniform distribution within a range and the max value is only 1000 or maybe less (see https://github.com/apache/hbase/blob/master/hbase-metrics/src/main/java/org/apache/hadoop/hbase/metrics/impl/FastLongHistogram.java#L69-L93 ) So if you use nanoTime, all values is much longer than 1000 and they are all in the last bucket. Then the bottleneck is single one LongAdder. > Put back HFile's HDFS op latency sampling code and add metrics for monitoring > - > > Key: HBASE-15160 > URL: https://issues.apache.org/jira/browse/HBASE-15160 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0, 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Critical > Attachments: HBASE-15160.patch, HBASE-15160_v2.patch, > HBASE-15160_v3.patch, hbase-15160_v4.patch, hbase-15160_v5.patch, > hbase-15160_v6.patch, hbase-15160_v7.patch > > > In HBASE-11586 all HDFS op latency sampling code, including fsReadLatency, > fsPreadLatency and fsWriteLatency, have been removed. There was some > discussion about putting them back in a new JIRA but never happened. > According to our experience, these metrics are useful to judge whether issue > lies on HDFS when slow request occurs, so we propose to put them back in this > JIRA, and add the metrics for monitoring as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
