[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/13#discussion_r130809257
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/TTable.java ---
@@ -396,20 +443,48 @@ public ResultScanner getScanner(Transaction tx, Scan 
scan) throws IOException {
 return commitCache;
 }
 
-private boolean isCellInSnapshot(Cell kv, HBaseTransaction 
transaction, Map commitCache)
-throws IOException {
+private void buildFamilyDeletionCache(List rawCells, Map familyDeletionCache) {
 
-long startTimestamp = transaction.getStartTimestamp();
+for (Cell cell : rawCells) {
+if (CellUtil.matchingQualifier(cell, 
CellUtils.FAMILY_DELETE_QUALIFIER) &&
+CellUtil.matchingValue(cell, 
HConstants.EMPTY_BYTE_ARRAY)) {
+
+String row = Bytes.toString(cell.getRow());
+List cells = familyDeletionCache.get(row);
--- End diff --

This is Map's return value.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/13#discussion_r130805885
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/TTable.java ---
@@ -228,21 +261,12 @@ public void delete(Transaction tx, Delete delete) 
throws IOException {
 }
 }
 }
-if (issueGet) {
-// It's better to perform a transactional get to avoid 
deleting more
-// than necessary
-Result result = this.get(transaction, deleteG);
-if (!result.isEmpty()) {
-for (Entry>> entryF : result.getMap()
-.entrySet()) {
-byte[] family = entryF.getKey();
-for (Entry> entryQ 
: entryF.getValue().entrySet()) {
-byte[] qualifier = entryQ.getKey();
-deleteP.add(family, qualifier, 
CellUtils.DELETE_TOMBSTONE);
-transaction.addWriteSetElement(new 
HBaseCellId(table, delete.getRow(), family, qualifier,
-   
transaction.getStartTimestamp()));
-}
-}
+
+if (deleteFamily) {
--- End diff --

1. This is a deletion path without read requirement. It is doable only in 
row level conflict detection mode where we only need to mark the row as a 
player for conflict analysis. 
2. TTable works only for HBase. You can see that each one of the method 
starts with a check that the transaction is hbase transaction. IMHO TTable 
should inherit from HTableInterface and override its functions, but this is for 
 a different patch :).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...

[ebortnik]

Github user ebortnik commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/13#discussion_r130566612
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/TTable.java ---
@@ -396,20 +443,48 @@ public ResultScanner getScanner(Transaction tx, Scan 
scan) throws IOException {
 return commitCache;
 }
 
-private boolean isCellInSnapshot(Cell kv, HBaseTransaction 
transaction, Map commitCache)
-throws IOException {
+private void buildFamilyDeletionCache(List rawCells, Map familyDeletionCache) {
 
-long startTimestamp = transaction.getStartTimestamp();
+for (Cell cell : rawCells) {
+if (CellUtil.matchingQualifier(cell, 
CellUtils.FAMILY_DELETE_QUALIFIER) &&
+CellUtil.matchingValue(cell, 
HConstants.EMPTY_BYTE_ARRAY)) {
+
+String row = Bytes.toString(cell.getRow());
+List cells = familyDeletionCache.get(row);
--- End diff --

Please decide if you return null or Optional from methods, the code looks 
non-uniform


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...

[ebortnik]

Github user ebortnik commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/13#discussion_r130567034
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/TTable.java ---
@@ -794,13 +871,24 @@ private HBaseTransaction 
enforceHBaseTransactionAsParam(Transaction tx) {
 }
 }
 
-static ImmutableList 
groupCellsByColumnFilteringShadowCells(List rawCells) {
+private HBaseTransactionManager 
enforceHBaseTransactionManagerAsParam(TransactionManager tm) {
+if (tm instanceof HBaseTransactionManager) {
+return (HBaseTransactionManager) tm;
+} else {
+throw new IllegalArgumentException(
+String.format("The transaction manager object passed %s is 
not an instance of HBaseTransactionManager ",
+  tm.getClass().getName()));
+}
+}
+
+static ImmutableList 
groupCellsByColumnFilteringShadowCellsAndFamilyDeletion(List rawCells) {
 
-Predicate shadowCellFilter = new Predicate() {
+Predicate shadowCellAndFamilyDeletionFilter = new 
Predicate() {
 
 @Override
 public boolean apply(Cell cell) {
-return cell != null && !CellUtils.isShadowCell(cell);
+return cell != null && !CellUtils.isShadowCell(cell) && 
+!(CellUtil.matchingQualifier(cell, 
CellUtils.FAMILY_DELETE_QUALIFIER) && CellUtil.matchingValue(cell, 
HConstants.EMPTY_BYTE_ARRAY));
--- End diff --

Maybe makes sense to store the last predicate in a variable, for 
readability. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...

[ebortnik]

Github user ebortnik commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/13#discussion_r130566417
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/TTable.java ---
@@ -333,7 +360,7 @@ public ResultScanner getScanner(Transaction tx, Scan 
scan) throws IOException {
  * @return Filtered KVs belonging to the transaction snapshot
  */
 List filterCellsForSnapshot(List rawCells, 
HBaseTransaction transaction,
-  int versionsToRequest) throws 
IOException {
+  int versionsToRequest, Map familyDeletionCache) throws IOException {
--- End diff --

Please explain the last param


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...

[ebortnik]

Github user ebortnik commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/13#discussion_r130549676
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/TTable.java ---
@@ -346,11 +373,31 @@ public ResultScanner getScanner(Transaction tx, Scan 
scan) throws IOException {
 }
 
 Map commitCache = buildCommitCache(rawCells);
+buildFamilyDeletionCache(rawCells, familyDeletionCache);
 
-for (Collection columnCells : 
groupCellsByColumnFilteringShadowCells(rawCells)) {
+for (Collection columnCells : 
groupCellsByColumnFilteringShadowCellsAndFamilyDeletion(rawCells)) {
 boolean snapshotValueFound = false;
 Cell oldestCell = null;
 for (Cell cell : columnCells) {
+List familyDeletionCells = 
familyDeletionCache.get(Bytes.toString((cell.getRow(;
+if (familyDeletionCells != null) {
--- End diff --

Maybe use Optional, just like a few lines below


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...

[ebortnik]

Github user ebortnik commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/13#discussion_r130563983
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/TTable.java ---
@@ -158,12 +162,40 @@ public Result get(Transaction tx, final Get get) 
throws IOException {
 Result result = table.get(tsget);
 List filteredKeyValues = Collections.emptyList();
 if (!result.isEmpty()) {
-filteredKeyValues = filterCellsForSnapshot(result.listCells(), 
transaction, tsget.getMaxVersions());
+filteredKeyValues = filterCellsForSnapshot(result.listCells(), 
transaction, tsget.getMaxVersions(), new HashMap());
 }
 
 return Result.create(filteredKeyValues);
 }
 
+private void familyQualifierBasedDeletion(HBaseTransaction tx, Put 
deleteP, Get deleteG) throws IOException {
+Result result = this.get(tx, deleteG);
+if (!result.isEmpty()) {
--- End diff --

Better to return here if the result is empty


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/13

[OMID-74] Efficient column family deletion in Row level conflict analysis

The idea is to use a qualifier to denote that all the columns of a specific 
family were deleted.
Current implementation reads from HBase the entire family and then writes a 
tombstone to each one of its cells.
The new implementation does not need to perform the read and only writes 
the qualifier to denote that the family was deleted.
This is true only for Row level conflict detection since in Cell level we 
need to read the cells and add these to the write set.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid 
FamilyDeletionTombstone-squash

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/13.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #13


commit 35ba4c79b4d3f713c6c38f791d5898f778153e2a
Author: Ohad Shacham 
Date:   2017-07-27T13:12:31Z

[OMID-74] Efficient column family deletion in Row level conflict analysis
The idea is to use a qualifier to denote that all the columns of a specific 
family were deleted.
Current implementation reads from HBase the entire family and then writes a 
tombstone to each one of its cells.
The new implementation does not need to perform the read and only writes 
the qualifier to denote that the family was deleted.
This is true only for Row level conflict detection since in Cell level we 
need to read the cells and add these to the write set.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---