[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/46#discussion_r223974301 --- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java --- @@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e // 2) Then check the commit table // If the data was written at a previous epoch, check whether the transaction was invalidated -Optional commitTimeStamp = commitTableClient.getCommitTimestamp(cellStartTimestamp).get(); -if (commitTimeStamp.isPresent()) { -return commitTimeStamp.get(); +boolean invalidatedByOther = false; +Optional commitTimestampFromCT = commitTableClient.getCommitTimestamp(cellStartTimestamp).get(); +if (commitTimestampFromCT.isPresent()) { +if (isLowLatency && !commitTimestampFromCT.get().isValid()) +invalidatedByOther = true; +else +return commitTimestampFromCT.get(); } // 3) Read from shadow cell -commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator); +Optional commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator); if (commitTimeStamp.isPresent()) { return commitTimeStamp.get(); } +// In case of LL, if found invalid ct cell, still must check sc in stage 3 then return +if (invalidatedByOther) { +assert(!commitTimestampFromCT.get().isValid()); +return commitTimestampFromCT.get(); +} + // 4) Check the epoch and invalidate the entry --- End diff -- or invalidate in a non latency mode ---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/46#discussion_r223993765 --- Diff: tso-server/src/main/java/org/apache/omid/tso/RequestProcessorSkipCT.java --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.omid.tso; + +import com.google.inject.Inject; +import org.apache.omid.metrics.MetricsRegistry; +import org.jboss.netty.channel.Channel; + +import java.io.IOException; + +public class RequestProcessorSkipCT extends AbstractRequestProcessor { + + +private final ReplyProcessor replyProcessor; + +private final LeaseManagement leaseManager; +private final Panicker panicker; +private final String tsoHostAndPort; + +@Inject +RequestProcessorSkipCT(MetricsRegistry metrics, + TimestampOracle timestampOracle, + ReplyProcessor replyProcessor, + Panicker panicker, + LeaseManagement leaseManager, + TSOServerConfig config, + LowWatermarkWriter lowWatermarkWriter, + String tsoHostAndPort) throws IOException { +super(metrics, timestampOracle, panicker, config, lowWatermarkWriter); +this.replyProcessor = replyProcessor; +this.tsoHostAndPort = tsoHostAndPort; +requestRing = disruptor.start(); +this.leaseManager = leaseManager; +this.panicker = panicker; +} + +private void commitSuicideIfNotMaster() { +if (!leaseManager.stillInLeasePeriod()) { +panicker.panic("Replica " + tsoHostAndPort + " lost mastership whilst flushing data. Committing suicide"); +} +} + +@Override +public void forwardCommit(long startTimestamp, long commitTimestamp, Channel c, MonitoringContext monCtx) { +commitSuicideIfNotMaster(); --- End diff -- Add a comment that his is required since returning commit results when the TSO is not the leader might violate snapshot isolation. This is because we have to guarantee that committing transaction from previous tso has to persist all its data because a new transaction started by a new tso. ---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/46#discussion_r223989230 --- Diff: transaction-client/src/main/java/org/apache/omid/transaction/AbstractTransactionManager.java --- @@ -350,6 +356,43 @@ private void markReadOnlyTransaction(AbstractTransaction readO } +private void commitLowLatencyTransaction(AbstractTransaction tx) +throws RollbackException, TransactionException { +try { + +long commitTs = tsoClient.commit(tx.getStartTimestamp(), tx.getWriteSet(), tx.getConflictFreeWriteSet()).get(); +boolean committed = commitTableWriter.atomicAddCommittedTransaction(tx.getStartTimestamp(),commitTs); +if (!committed) { +// Transaction has been invalidated by other client +rollback(tx); + commitTableClient.completeTransaction(tx.getStartTimestamp()); +rolledbackTxsCounter.inc(); +throw new RollbackException("Transaction " + tx.getTransactionId() + " got invalidated"); +} +certifyCommitForTx(tx, commitTs); +updateShadowCellsAndRemoveCommitTableEntry(tx, postCommitter); + +} catch (ExecutionException e) { +if (e.getCause() instanceof AbortException) { // TSO reports Tx conflicts as AbortExceptions in the future +rollback(tx); +rolledbackTxsCounter.inc(); +throw new RollbackException("Conflicts detected in tx writeset", e.getCause()); +} + +if (e.getCause() instanceof ServiceUnavailableException || e.getCause() instanceof ConnectionException) { +errorTxsCounter.inc(); +rollback(tx); // Rollback proactively cause it's likely that a new TSOServer is now master --- End diff -- the leader ---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/46#discussion_r223975170 --- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java --- @@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e // 2) Then check the commit table // If the data was written at a previous epoch, check whether the transaction was invalidated -Optional commitTimeStamp = commitTableClient.getCommitTimestamp(cellStartTimestamp).get(); -if (commitTimeStamp.isPresent()) { -return commitTimeStamp.get(); +boolean invalidatedByOther = false; +Optional commitTimestampFromCT = commitTableClient.getCommitTimestamp(cellStartTimestamp).get(); +if (commitTimestampFromCT.isPresent()) { +if (isLowLatency && !commitTimestampFromCT.get().isValid()) +invalidatedByOther = true; +else +return commitTimestampFromCT.get(); } // 3) Read from shadow cell -commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator); +Optional commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator); if (commitTimeStamp.isPresent()) { return commitTimeStamp.get(); } +// In case of LL, if found invalid ct cell, still must check sc in stage 3 then return +if (invalidatedByOther) { +assert(!commitTimestampFromCT.get().isValid()); +return commitTimestampFromCT.get(); +} + // 4) Check the epoch and invalidate the entry // if the data was written by a transaction from a previous epoch (previous TSO) -if (cellStartTimestamp < epoch) { +if (cellStartTimestamp < epoch || isLowLatency) { boolean invalidated = commitTableClient.tryInvalidateTransaction(cellStartTimestamp).get(); if (invalidated) { // Invalid commit timestamp + +// If we are running lowLatency Omid, we could have manged to invalidate a ct entry, +// but the committing client already wrote to shadow cells: --- End diff -- This can happen only in low latency mode, since in the regular mode the client keeps the commit table entry if persisting the commit was done after the tso lost its lease. ---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/46#discussion_r223974621 --- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java --- @@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e // 2) Then check the commit table // If the data was written at a previous epoch, check whether the transaction was invalidated -Optional commitTimeStamp = commitTableClient.getCommitTimestamp(cellStartTimestamp).get(); -if (commitTimeStamp.isPresent()) { -return commitTimeStamp.get(); +boolean invalidatedByOther = false; +Optional commitTimestampFromCT = commitTableClient.getCommitTimestamp(cellStartTimestamp).get(); +if (commitTimestampFromCT.isPresent()) { +if (isLowLatency && !commitTimestampFromCT.get().isValid()) +invalidatedByOther = true; +else +return commitTimestampFromCT.get(); } // 3) Read from shadow cell -commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator); +Optional commitTimeStamp = readCommitTimestampFromShadowCell(cellStartTimestamp, locator); if (commitTimeStamp.isPresent()) { return commitTimeStamp.get(); } +// In case of LL, if found invalid ct cell, still must check sc in stage 3 then return +if (invalidatedByOther) { +assert(!commitTimestampFromCT.get().isValid()); +return commitTimestampFromCT.get(); +} + // 4) Check the epoch and invalidate the entry // if the data was written by a transaction from a previous epoch (previous TSO) -if (cellStartTimestamp < epoch) { +if (cellStartTimestamp < epoch || isLowLatency) { boolean invalidated = commitTableClient.tryInvalidateTransaction(cellStartTimestamp).get(); if (invalidated) { // Invalid commit timestamp + +// If we are running lowLatency Omid, we could have manged to invalidate a ct entry, --- End diff -- managed ---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/46#discussion_r223971768 --- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java --- @@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e // 2) Then check the commit table // If the data was written at a previous epoch, check whether the transaction was invalidated --- End diff -- This "If the data was written at a previous epoch," can be removed. ---
[GitHub] incubator-omid issue #43: [OMID-110]
Github user ohadshacham commented on the issue: https://github.com/apache/incubator-omid/pull/43 +1 ---
[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/41#discussion_r206860096 --- Diff: hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java --- @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.omid.transaction; + +import com.google.common.base.Optional; +import com.sun.istack.Nullable; +import org.apache.hadoop.hbase.Cell; +import org.apache.hadoop.hbase.CellUtil; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.filter.Filter; +import org.apache.hadoop.hbase.filter.FilterBase; +import org.apache.hadoop.hbase.util.Bytes; + +import java.io.IOException; +import java.util.*; + +public class TransactionVisibilityFilter extends FilterBase { + +// optional sub-filter to apply to visible cells +private final Filter userFilter; +private final SnapshotFilterImpl snapshotFilter; +private final Map shadowCellCache; +private final HBaseTransaction hbaseTransaction; +private final Map familyDeletionCache; --- End diff -- I would add a comment that the row info is redundant in here since reset is called between rows and we clear this map in reset. ---
[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/41#discussion_r206861622 --- Diff: hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java --- @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.omid.transaction; + +import com.google.common.base.Optional; +import com.sun.istack.Nullable; +import org.apache.hadoop.hbase.Cell; +import org.apache.hadoop.hbase.CellUtil; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.filter.Filter; +import org.apache.hadoop.hbase.filter.FilterBase; +import org.apache.hadoop.hbase.util.Bytes; + +import java.io.IOException; +import java.util.*; + +public class TransactionVisibilityFilter extends FilterBase { + +// optional sub-filter to apply to visible cells +private final Filter userFilter; +private final SnapshotFilterImpl snapshotFilter; +private final Map shadowCellCache; +private final HBaseTransaction hbaseTransaction; +private final Map familyDeletionCache; + +public SnapshotFilter getSnapshotFilter() { +return snapshotFilter; +} + +public TransactionVisibilityFilter(@Nullable Filter cellFilter, + SnapshotFilterImpl snapshotFilter, + HBaseTransaction hbaseTransaction) { +this.userFilter = cellFilter; +this.snapshotFilter = snapshotFilter; +shadowCellCache = new HashMap<>(); +this.hbaseTransaction = hbaseTransaction; +familyDeletionCache = new HashMap(); +} + +@Override +public ReturnCode filterKeyValue(Cell v) throws IOException { +if (CellUtils.isShadowCell(v)) { +Long commitTs = Bytes.toLong(CellUtil.cloneValue(v)); +shadowCellCache.put(v.getTimestamp(), commitTs); +// Continue getting shadow cells until one of them fits this transaction +if (hbaseTransaction.getStartTimestamp() >= commitTs) { +return ReturnCode.NEXT_COL; +} else { +return ReturnCode.SKIP; +} +} else if (CellUtils.isFamilyDeleteCell(v)) { +//Delete is part of this transaction +if (snapshotFilter.getTSIfInTransaction(v, hbaseTransaction).isPresent()) { + familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), v.getTimestamp()); +return ReturnCode.NEXT_COL; +} + +if (shadowCellCache.containsKey(v.getTimestamp()) && +hbaseTransaction.getStartTimestamp() >= shadowCellCache.get(v.getTimestamp())) { + familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), shadowCellCache.get(v.getTimestamp())); +return ReturnCode.NEXT_COL; +} + +// Try to get shadow cell from region +final Get get = new Get(CellUtil.cloneRow(v)); +get.setTimeStamp(v.getTimestamp()).setMaxVersions(1); +get.addColumn(CellUtil.cloneFamily(v), CellUtils.addShadowCellSuffix(CellUtils.FAMILY_DELETE_QUALIFIER)); +Result deleteFamilySC = snapshotFilter.getTableAccessWrapper().get(get); + +if (!deleteFamilySC.isEmpty() && + Bytes.toLong(CellUtil.cloneValue(deleteFamilySC.rawCells()[0] )) < hbaseTransaction.getStartTimestamp()){ + familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), Bytes.toLong(CellUtil.cloneValue(deleteFamilySC.rawCells()[0]))); +return ReturnCode.NEXT_COL; +} + +//At last go to com
[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/41#discussion_r206848753 --- Diff: hbase-coprocessor/src/main/java/org/apache/omid/transaction/OmidSnapshotFilter.java --- @@ -83,92 +76,97 @@ public void start(CoprocessorEnvironment env) throws IOException { if (commitTableName != null) { commitTableConf.setTableName(commitTableName); } -if (commitTableClient == null) { -commitTableClient = initAndGetCommitTableClient(); -} -snapshotFilter = new SnapshotFilterImpl(commitTableClient); - LOG.info("Snapshot filter started"); } @Override public void stop(CoprocessorEnvironment e) throws IOException { LOG.info("Stopping snapshot filter coprocessor"); -commitTableClient.close(); +if (snapshotFilterQueue != null) { +for (SnapshotFilter snapshotFilter: snapshotFilterQueue) { + ((SnapshotFilterImpl)snapshotFilter).getCommitTableClient().close(); +} +} LOG.info("Snapshot filter stopped"); } -public void setCommitTableClient(CommitTable.Client commitTableClient) { -this.commitTableClient = commitTableClient; -this.snapshotFilter.setCommitTableClient(commitTableClient); -} @Override -public void preGetOp(ObserverContext c, Get get, List result) throws IOException { - -if (get.getAttribute(CellUtils.CLIENT_GET_ATTRIBUTE) == null) return; - -try { -get.setAttribute(CellUtils.CLIENT_GET_ATTRIBUTE, null); -RegionAccessWrapper regionAccessWrapper = new RegionAccessWrapper(HBaseShims.getRegionCoprocessorRegion(c.getEnvironment())); -Result res = regionAccessWrapper.get(get); // get parameters were set at the client side - -snapshotFilter.setTableAccessWrapper(regionAccessWrapper); +public void postGetOp(ObserverContext e, Get get, List results) +throws IOException { +if (get.getFilter() != null) { +//This get had a filter and used a commit table client that must put back +assert (get.getFilter() instanceof TransactionVisibilityFilter); +TransactionVisibilityFilter filter = (TransactionVisibilityFilter)get.getFilter(); + snapshotFilterQueue.add((SnapshotFilterImpl)filter.getSnapshotFilter()); +} +} -List filteredKeyValues = Collections.emptyList(); -if (!res.isEmpty()) { -TSOProto.Transaction transaction = TSOProto.Transaction.parseFrom(get.getAttribute(CellUtils.TRANSACTION_ATTRIBUTE)); -long id = transaction.getTimestamp(); -long readTs = transaction.getReadTimestamp(); -long epoch = transaction.getEpoch(); -VisibilityLevel visibilityLevel = VisibilityLevel.fromInteger(transaction.getVisibilityLevel()); +@Override +public void preGetOp(ObserverContext e, Get get, List results) +throws IOException { +if (get.getAttribute(CellUtils.CLIENT_GET_ATTRIBUTE) == null) return; -HBaseTransaction hbaseTransaction = new HBaseTransaction(id, readTs, visibilityLevel, epoch, new HashSet(), new HashSet(), null); -filteredKeyValues = snapshotFilter.filterCellsForSnapshot(res.listCells(), hbaseTransaction, get.getMaxVersions(), new HashMap(), get.getAttributesMap()); -} +HBaseTransaction hbaseTransaction = getHBaseTransaction(get.getAttribute(CellUtils.TRANSACTION_ATTRIBUTE)); +SnapshotFilterImpl snapshotFilter = getSnapshotFilter(e); +get.setMaxVersions(); --- End diff -- I would add a comment that we set to max versions since we are doing Omid filtering in the VisibilityFilter ---
[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/41#discussion_r206867892 --- Diff: hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java --- @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.omid.transaction; + +import com.google.common.base.Optional; +import com.sun.istack.Nullable; +import org.apache.hadoop.hbase.Cell; +import org.apache.hadoop.hbase.CellUtil; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.filter.Filter; +import org.apache.hadoop.hbase.filter.FilterBase; +import org.apache.hadoop.hbase.util.Bytes; + +import java.io.IOException; +import java.util.*; + +public class TransactionVisibilityFilter extends FilterBase { + +// optional sub-filter to apply to visible cells +private final Filter userFilter; +private final SnapshotFilterImpl snapshotFilter; +private final Map shadowCellCache; +private final HBaseTransaction hbaseTransaction; +private final Map familyDeletionCache; + +public SnapshotFilter getSnapshotFilter() { +return snapshotFilter; +} + +public TransactionVisibilityFilter(@Nullable Filter cellFilter, + SnapshotFilterImpl snapshotFilter, + HBaseTransaction hbaseTransaction) { +this.userFilter = cellFilter; +this.snapshotFilter = snapshotFilter; +shadowCellCache = new HashMap<>(); +this.hbaseTransaction = hbaseTransaction; +familyDeletionCache = new HashMap(); +} + +@Override +public ReturnCode filterKeyValue(Cell v) throws IOException { +if (CellUtils.isShadowCell(v)) { +Long commitTs = Bytes.toLong(CellUtil.cloneValue(v)); +shadowCellCache.put(v.getTimestamp(), commitTs); +// Continue getting shadow cells until one of them fits this transaction +if (hbaseTransaction.getStartTimestamp() >= commitTs) { +return ReturnCode.NEXT_COL; +} else { +return ReturnCode.SKIP; +} +} else if (CellUtils.isFamilyDeleteCell(v)) { +//Delete is part of this transaction +if (snapshotFilter.getTSIfInTransaction(v, hbaseTransaction).isPresent()) { + familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), v.getTimestamp()); +return ReturnCode.NEXT_COL; +} + +if (shadowCellCache.containsKey(v.getTimestamp()) && +hbaseTransaction.getStartTimestamp() >= shadowCellCache.get(v.getTimestamp())) { + familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), shadowCellCache.get(v.getTimestamp())); +return ReturnCode.NEXT_COL; +} + +// Try to get shadow cell from region +final Get get = new Get(CellUtil.cloneRow(v)); +get.setTimeStamp(v.getTimestamp()).setMaxVersions(1); +get.addColumn(CellUtil.cloneFamily(v), CellUtils.addShadowCellSuffix(CellUtils.FAMILY_DELETE_QUALIFIER)); +Result deleteFamilySC = snapshotFilter.getTableAccessWrapper().get(get); + +if (!deleteFamilySC.isEmpty() && + Bytes.toLong(CellUtil.cloneValue(deleteFamilySC.rawCells()[0] )) < hbaseTransaction.getStartTimestamp()){ + familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), Bytes.toLong(CellUtil.cloneValue(deleteFamilySC.rawCells()[0]))); +return ReturnCode.NEXT_COL; +} + +//At last go to com
[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/41#discussion_r206856182 --- Diff: hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java --- @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.omid.transaction; + +import com.google.common.base.Optional; +import com.sun.istack.Nullable; +import org.apache.hadoop.hbase.Cell; +import org.apache.hadoop.hbase.CellUtil; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.filter.Filter; +import org.apache.hadoop.hbase.filter.FilterBase; +import org.apache.hadoop.hbase.util.Bytes; + +import java.io.IOException; +import java.util.*; + +public class TransactionVisibilityFilter extends FilterBase { + +// optional sub-filter to apply to visible cells +private final Filter userFilter; +private final SnapshotFilterImpl snapshotFilter; +private final Map shadowCellCache; +private final HBaseTransaction hbaseTransaction; +private final Map familyDeletionCache; + +public SnapshotFilter getSnapshotFilter() { +return snapshotFilter; +} + +public TransactionVisibilityFilter(@Nullable Filter cellFilter, + SnapshotFilterImpl snapshotFilter, + HBaseTransaction hbaseTransaction) { +this.userFilter = cellFilter; +this.snapshotFilter = snapshotFilter; +shadowCellCache = new HashMap<>(); +this.hbaseTransaction = hbaseTransaction; +familyDeletionCache = new HashMap(); +} + +@Override +public ReturnCode filterKeyValue(Cell v) throws IOException { +if (CellUtils.isShadowCell(v)) { +Long commitTs = Bytes.toLong(CellUtil.cloneValue(v)); +shadowCellCache.put(v.getTimestamp(), commitTs); +// Continue getting shadow cells until one of them fits this transaction +if (hbaseTransaction.getStartTimestamp() >= commitTs) { +return ReturnCode.NEXT_COL; +} else { +return ReturnCode.SKIP; +} +} else if (CellUtils.isFamilyDeleteCell(v)) { +//Delete is part of this transaction +if (snapshotFilter.getTSIfInTransaction(v, hbaseTransaction).isPresent()) { + familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), v.getTimestamp()); +return ReturnCode.NEXT_COL; +} + +if (shadowCellCache.containsKey(v.getTimestamp()) && --- End diff -- You can do a get and check the existence using the return value. It saves a collection access. ---
[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/41#discussion_r206857467 --- Diff: hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java --- @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.omid.transaction; + +import com.google.common.base.Optional; +import com.sun.istack.Nullable; +import org.apache.hadoop.hbase.Cell; +import org.apache.hadoop.hbase.CellUtil; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.filter.Filter; +import org.apache.hadoop.hbase.filter.FilterBase; +import org.apache.hadoop.hbase.util.Bytes; + +import java.io.IOException; +import java.util.*; + +public class TransactionVisibilityFilter extends FilterBase { + +// optional sub-filter to apply to visible cells +private final Filter userFilter; +private final SnapshotFilterImpl snapshotFilter; +private final Map shadowCellCache; +private final HBaseTransaction hbaseTransaction; +private final Map familyDeletionCache; + +public SnapshotFilter getSnapshotFilter() { +return snapshotFilter; +} + +public TransactionVisibilityFilter(@Nullable Filter cellFilter, + SnapshotFilterImpl snapshotFilter, + HBaseTransaction hbaseTransaction) { +this.userFilter = cellFilter; +this.snapshotFilter = snapshotFilter; +shadowCellCache = new HashMap<>(); +this.hbaseTransaction = hbaseTransaction; +familyDeletionCache = new HashMap(); +} + +@Override +public ReturnCode filterKeyValue(Cell v) throws IOException { +if (CellUtils.isShadowCell(v)) { +Long commitTs = Bytes.toLong(CellUtil.cloneValue(v)); +shadowCellCache.put(v.getTimestamp(), commitTs); +// Continue getting shadow cells until one of them fits this transaction +if (hbaseTransaction.getStartTimestamp() >= commitTs) { +return ReturnCode.NEXT_COL; +} else { +return ReturnCode.SKIP; +} +} else if (CellUtils.isFamilyDeleteCell(v)) { +//Delete is part of this transaction +if (snapshotFilter.getTSIfInTransaction(v, hbaseTransaction).isPresent()) { + familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), v.getTimestamp()); +return ReturnCode.NEXT_COL; +} + +if (shadowCellCache.containsKey(v.getTimestamp()) && +hbaseTransaction.getStartTimestamp() >= shadowCellCache.get(v.getTimestamp())) { + familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), shadowCellCache.get(v.getTimestamp())); +return ReturnCode.NEXT_COL; +} + +// Try to get shadow cell from region +final Get get = new Get(CellUtil.cloneRow(v)); +get.setTimeStamp(v.getTimestamp()).setMaxVersions(1); +get.addColumn(CellUtil.cloneFamily(v), CellUtils.addShadowCellSuffix(CellUtils.FAMILY_DELETE_QUALIFIER)); --- End diff -- Change the name of addShadowCellSuffix to addShadowCellMetadata? ---
[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/41#discussion_r206862264 --- Diff: hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java --- @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.omid.transaction; + +import com.google.common.base.Optional; +import com.sun.istack.Nullable; +import org.apache.hadoop.hbase.Cell; +import org.apache.hadoop.hbase.CellUtil; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.filter.Filter; +import org.apache.hadoop.hbase.filter.FilterBase; +import org.apache.hadoop.hbase.util.Bytes; + +import java.io.IOException; +import java.util.*; + +public class TransactionVisibilityFilter extends FilterBase { + +// optional sub-filter to apply to visible cells +private final Filter userFilter; +private final SnapshotFilterImpl snapshotFilter; +private final Map shadowCellCache; +private final HBaseTransaction hbaseTransaction; +private final Map familyDeletionCache; + +public SnapshotFilter getSnapshotFilter() { +return snapshotFilter; +} + +public TransactionVisibilityFilter(@Nullable Filter cellFilter, + SnapshotFilterImpl snapshotFilter, + HBaseTransaction hbaseTransaction) { +this.userFilter = cellFilter; +this.snapshotFilter = snapshotFilter; +shadowCellCache = new HashMap<>(); +this.hbaseTransaction = hbaseTransaction; +familyDeletionCache = new HashMap(); +} + +@Override +public ReturnCode filterKeyValue(Cell v) throws IOException { +if (CellUtils.isShadowCell(v)) { +Long commitTs = Bytes.toLong(CellUtil.cloneValue(v)); +shadowCellCache.put(v.getTimestamp(), commitTs); +// Continue getting shadow cells until one of them fits this transaction +if (hbaseTransaction.getStartTimestamp() >= commitTs) { +return ReturnCode.NEXT_COL; +} else { +return ReturnCode.SKIP; +} +} else if (CellUtils.isFamilyDeleteCell(v)) { +//Delete is part of this transaction +if (snapshotFilter.getTSIfInTransaction(v, hbaseTransaction).isPresent()) { + familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), v.getTimestamp()); +return ReturnCode.NEXT_COL; +} + +if (shadowCellCache.containsKey(v.getTimestamp()) && +hbaseTransaction.getStartTimestamp() >= shadowCellCache.get(v.getTimestamp())) { + familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), shadowCellCache.get(v.getTimestamp())); +return ReturnCode.NEXT_COL; +} + +// Try to get shadow cell from region +final Get get = new Get(CellUtil.cloneRow(v)); +get.setTimeStamp(v.getTimestamp()).setMaxVersions(1); +get.addColumn(CellUtil.cloneFamily(v), CellUtils.addShadowCellSuffix(CellUtils.FAMILY_DELETE_QUALIFIER)); +Result deleteFamilySC = snapshotFilter.getTableAccessWrapper().get(get); + +if (!deleteFamilySC.isEmpty() && + Bytes.toLong(CellUtil.cloneValue(deleteFamilySC.rawCells()[0] )) < hbaseTransaction.getStartTimestamp()){ + familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), Bytes.toLong(CellUtil.cloneValue(deleteFamilySC.rawCells()[0]))); +return ReturnCode.NEXT_COL; +} + +//At last go to com
[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/41#discussion_r206843754 --- Diff: hbase-common/src/main/java/org/apache/omid/transaction/CellUtils.java --- @@ -382,13 +385,16 @@ public int hashCode() { hasher.putBytes(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength()); hasher.putBytes(cell.getFamilyArray(), cell.getFamilyOffset(), cell.getFamilyLength()); int qualifierLength = cell.getQualifierLength(); +int qualifierOffset = cell.getQualifierOffset(); if (isShadowCell()) { // Update qualifier length when qualifier is shadow cell qualifierLength = qualifierLengthFromShadowCellQualifier(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength()); +qualifierOffset = qualifierOffset + 1; --- End diff -- Will it work when the shadow cell prefix is absent? legacy data. ---
[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/41#discussion_r206852241 --- Diff: hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java --- @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.omid.transaction; + +import com.google.common.base.Optional; +import com.sun.istack.Nullable; +import org.apache.hadoop.hbase.Cell; +import org.apache.hadoop.hbase.CellUtil; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.filter.Filter; +import org.apache.hadoop.hbase.filter.FilterBase; +import org.apache.hadoop.hbase.util.Bytes; + +import java.io.IOException; +import java.util.*; + +public class TransactionVisibilityFilter extends FilterBase { + +// optional sub-filter to apply to visible cells +private final Filter userFilter; +private final SnapshotFilterImpl snapshotFilter; +private final Map shadowCellCache; +private final HBaseTransaction hbaseTransaction; +private final Map familyDeletionCache; + +public SnapshotFilter getSnapshotFilter() { +return snapshotFilter; +} + +public TransactionVisibilityFilter(@Nullable Filter cellFilter, + SnapshotFilterImpl snapshotFilter, + HBaseTransaction hbaseTransaction) { +this.userFilter = cellFilter; +this.snapshotFilter = snapshotFilter; +shadowCellCache = new HashMap<>(); +this.hbaseTransaction = hbaseTransaction; +familyDeletionCache = new HashMap(); +} + +@Override +public ReturnCode filterKeyValue(Cell v) throws IOException { +if (CellUtils.isShadowCell(v)) { +Long commitTs = Bytes.toLong(CellUtil.cloneValue(v)); +shadowCellCache.put(v.getTimestamp(), commitTs); +// Continue getting shadow cells until one of them fits this transaction +if (hbaseTransaction.getStartTimestamp() >= commitTs) { --- End diff -- Why do we keep ones that committed after the transaction timestamp? ---
[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/41#discussion_r206839768 --- Diff: hbase-client/src/test/java/org/apache/omid/transaction/TestCellUtils.java --- @@ -99,11 +99,11 @@ public void testShadowCellQualifiers(byte[] shadowCellSuffixToTest) throws IOExc public void testCorrectMapingOfCellsToShadowCells() throws IOException { // Create the required data final byte[] validShadowCellQualifier = -com.google.common.primitives.Bytes.concat(qualifier, SHADOW_CELL_SUFFIX); +com.google.common.primitives.Bytes.concat(new byte[1], qualifier, SHADOW_CELL_SUFFIX); --- End diff -- Define SHADOW_CELL_PREFIX to "new byte[1]" and replace all the new byte[1]. ---
[GitHub] incubator-omid pull request #40: [OMID-106] Delete should use write timestam...
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/40 ---
[GitHub] incubator-omid pull request #39: [OMID-105] When a tentative family deletion...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/39#discussion_r206422618 --- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java --- @@ -395,8 +427,8 @@ private Get createPendingGet(Cell cell, int versionCount) throws IOException { } } -if (isCellInTransaction(cell, transaction, commitCache) || -isCellInSnapshot(cell, transaction, commitCache)) { +if (isCellInTransaction(cell, transaction, commitCache).isPresent() || +isCellInSnapshot(cell, transaction, commitCache).isPresent()) { --- End diff -- Same as before ---
[GitHub] incubator-omid pull request #39: [OMID-105] When a tentative family deletion...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/39#discussion_r206422025 --- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java --- @@ -298,26 +291,62 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e return commitCache; } -private void buildFamilyDeletionCache(List rawCells, Map> familyDeletionCache) { - +private void buildFamilyDeletionCache(HBaseTransaction transaction, List rawCells, Map familyDeletionCache, Map commitCache, Map attributeMap) throws IOException { for (Cell cell : rawCells) { if (CellUtil.matchingQualifier(cell, CellUtils.FAMILY_DELETE_QUALIFIER) && CellUtil.matchingValue(cell, HConstants.EMPTY_BYTE_ARRAY)) { - String row = Bytes.toString(cell.getRow()); -List cells = familyDeletionCache.get(row); -if (cells == null) { -cells = new ArrayList<>(); -familyDeletionCache.put(row, cells); +String family = Bytes.toString(cell.getFamily()); +String key = row + ":" + family; + +if (familyDeletionCache.containsKey(key)) +return; + +Optional commitTimeStamp = isCellInSnapshot(cell, transaction, commitCache); + +if (!commitTimeStamp.isPresent()) { +commitTimeStamp = isCellInTransaction(cell, transaction, commitCache); } -cells.add(cell); +if (commitTimeStamp.isPresent()) { +familyDeletionCache.put(key, commitTimeStamp.get()); +} else { +Cell lastCell = cell; +Map cmtCache; +boolean foundCommitttedFamilyDeletion = false; +while (!foundCommitttedFamilyDeletion) { + +Get g = createPendingGet(lastCell, 3); +for (Map.Entry entry : attributeMap.entrySet()) { --- End diff -- I don't know which attribute the user wrote and whether he/she wrote a coprocessor that looks on these attributes and do something. ---
[GitHub] incubator-omid pull request #36: [OMID-72] bug fix, accessed tables should b...
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/36 ---
[GitHub] incubator-omid pull request #36: [OMID-72] bug fix, accessed tables should b...
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/36 [OMID-72] bug fix, accessed tables should be sent to transaction mana⦠â¦ger also for conflict free writes. This is because fences should also force conflict free transactions to abort. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid OMID-72-bug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/36.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #36 commit d96ff96e4fa264472ae684bdb1834e791a6d25ef Author: Ohad Shacham Date: 2018-06-03T13:41:45Z [OMID-72] bug fix, accessed tables should be sent to transaction manager also for conflict free writes. This is because fences should also force conflict free transactions to abort. ---
[GitHub] incubator-omid pull request #35: [OMID-100] James Taylor's patch to: https:/...
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/35 ---
[GitHub] incubator-omid pull request #34: [OMID-99] change TestNG version to 6.10.
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/34 [OMID-99] change TestNG version to 6.10. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid OMID-99 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/34.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #34 ---
[GitHub] incubator-omid pull request #33: [OMID-98] Adding license headers to default...
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/33 ---
[GitHub] incubator-omid pull request #29: [OMID-44] Since YCSB files are includes in ...
GitHub user ohadshacham reopened a pull request: https://github.com/apache/incubator-omid/pull/29 [OMID-44] Since YCSB files are includes in OMID a reference in Yahoo ⦠â¦is needed in the NOTICE file. Subject raised at: https://lists.apache.org/thread.html/10fd534d9ea4b0a4574637ab49e5d823a47a864ec27f317f90b14196@%3Cgeneral.incubator.apache.org%3E You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid OMID-44 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/29.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #29 commit af7acfaa60fc67c1523069be3e6acd4d7c018daf Author: Ohad Shacham <ohads@...> Date: 2018-03-26T08:31:37Z [OMID-44] Since YCSB files are includes in OMID a reference in Yahoo is needed in the NOTICE file. Subject raised at: https://lists.apache.org/thread.html/10fd534d9ea4b0a4574637ab49e5d823a47a864ec27f317f90b14196@%3Cgeneral.incubator.apache.org%3E ---
[GitHub] incubator-omid pull request #32: [OMID-97] New MacBooks include a TouchBar. ...
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/32 ---
[GitHub] incubator-omid pull request #33: [OMID-98] Adding license headers to default...
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/33 [OMID-98] Adding license headers to default-omid-server-configuration⦠â¦.yml and default-tso-server-benchmark-config.yml. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid ymlLicenseHeader Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/33.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #33 ---
[GitHub] incubator-omid pull request #32: [OMID-97] New MacBooks include a TouchBar. ...
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/32 [OMID-97] New MacBooks include a TouchBar. This touch bar has a usb l⦠â¦ink that called iBridge. When choosing a network interface Omid accidentally chooses this iBridge instead of choosing the Ethernet or the Wifi. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid fixNetworkInterface Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/32.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #32 commit 4ae4bbd52749908fabda0e445577b253bcb4bda2 Author: Ohad Shacham <ohads@...> Date: 2018-04-24T12:39:59Z [OMID-97] New MacBooks include a TouchBar. This touch bar has a usb link that called iBridge. When choosing a network interface Omid accidentally chooses this iBridge instead of choosing the Ethernet or the Wifi. ---
[GitHub] incubator-omid pull request #31: [OMID-96] Enable compactor on all column fa...
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/31 ---
[GitHub] incubator-omid pull request #30: [OMID-95] Set hbase-1 as the default compil...
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/30 ---
[GitHub] incubator-omid pull request #29: [OMID-44] Since YCSB files are includes in ...
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/29 ---
[GitHub] incubator-omid pull request #29: [OMID-44] Since YCSB files are includes in ...
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/29 [OMID-44] Since YCSB files are includes in OMID a reference in Yahoo ⦠â¦is needed in the NOTICE file. Subject raised at: https://lists.apache.org/thread.html/10fd534d9ea4b0a4574637ab49e5d823a47a864ec27f317f90b14196@%3Cgeneral.incubator.apache.org%3E You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid OMID-44 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/29.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #29 commit af7acfaa60fc67c1523069be3e6acd4d7c018daf Author: Ohad Shacham <ohads@...> Date: 2018-03-26T08:31:37Z [OMID-44] Since YCSB files are includes in OMID a reference in Yahoo is needed in the NOTICE file. Subject raised at: https://lists.apache.org/thread.html/10fd534d9ea4b0a4574637ab49e5d823a47a864ec27f317f90b14196@%3Cgeneral.incubator.apache.org%3E ---
[GitHub] incubator-omid pull request #30: [OMID-95] Set hbase-1 as the default compil...
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/30 [OMID-95] Set hbase-1 as the default compilation profile You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid OMID-95 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/30.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #30 commit 505bf4c4a1e16e9b8ebdc62aa45f1028cf832c4c Author: Ohad Shacham <ohads@...> Date: 2018-03-26T10:59:40Z [OMID-95] Set hbase-1 as the default compilation profile ---
[GitHub] incubator-omid pull request #31: [OMID-96] Enable compactor on all column fa...
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/31 [OMID-96] Enable compactor on all column families during initializati⦠â¦on. This comes in addition to the option of marking each column family using HBase metadata. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid OMID-96 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/31.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #31 commit 1c264e3f409d418057f9ab1cc768648357804493 Author: Ohad Shacham <ohads@...> Date: 2018-03-26T13:07:49Z [OMID-96] Enable compactor on all column families during initialization. This comes in addition to the option of marking each column family using HBase metadata. ---
[GitHub] incubator-omid pull request #25: [OMID-93] mark mutation as committed
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/25 ---
[GitHub] incubator-omid pull request #26: A bug fix in [OMID-74].
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/26 ---
[GitHub] incubator-omid pull request #28: [OMID-94] Tune Omid for Phoenix testing env...
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/28 [OMID-94] Tune Omid for Phoenix testing environment. This commit changes visibility of several function in order to run Omid in testing mode from Phoenix testing environment. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid OMID-94 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/28.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #28 commit d1b9a5b338429c6135201cff7266dcb670486880 Author: Ohad Shacham <ohads@...> Date: 2018-03-19T07:48:58Z [OMID-94] Tune Omid for Phoenix testing environment. This commit changes visibility of several function in order to run Omid in testing mode from Phoenix testing environment. ---
[GitHub] incubator-omid pull request #26: A bug fix in [OMID-74].
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/26 A bug fix in [OMID-74]. The update of the write set is incorrect since the family deletion qualifier needs to be added instead of a row marker. Therefore, this commit fixes this case. This is crucial since the write set information is needed for adding shadow cells, when transaction successfully commits, and for garbage collection when transaction aborts. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid OMID-74-reopen Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/26.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #26 commit d6ed73fc4cf34c84a4f283011521a746f257d66e Author: Ohad Shacham <ohads@...> Date: 2018-03-15T12:33:57Z A bug fix in [OMID-74]. The update of the write set is incorrect since the family deletion qualifier needs to be added instead of a row marker. Therefore, this commit fixes this case. This is crucial since the write set information is needed for adding shadow cells, when transaction successfully commits, and for garbage collection when transaction aborts. ---
[GitHub] incubator-omid pull request #27: reopen [OMID-70] in order to bind WorldCloc...
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/27 reopen [OMID-70] in order to bind WorldClockOracleImpl in TSOMockModule. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid OMID-70-reopen Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/27.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #27 commit 0eca99f9911a0e23fa099531c59a231098b97fa5 Author: Ohad Shacham <ohads@...> Date: 2018-03-15T14:22:35Z reopen [OMID-70] in order to bind WorldClockOracleImpl in TSOMockModule. ---
[GitHub] incubator-omid pull request #25: [OMID-93] mark mutation as committed
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/25 [OMID-93] mark mutation as committed This pull request adds an option to add commit metadata (shadow cells) to an existing mutation. This feature is required by Apache Phoenix both for local index population and update. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid OMID-93 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/25.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #25 commit 93b2c2a1e7f2e3937feb6a3a507899af13c7c38c Author: Ohad Shacham <ohads@...> Date: 2018-03-13T12:22:39Z [OMID-93] This commit adds an option to add commit metadata (shadow cells) to an existing mutation. This feature is required by Apache Phoenix both for local index population and update. commit 361a41373b81c9efa9e77e70c1c58dcdc3b6deb4 Author: Ohad Shacham <ohads@...> Date: 2018-03-13T12:33:36Z cosmetics ---
[GitHub] incubator-omid pull request #24: [OMID-92] Add Apache Rat to the build proce...
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/24 ---
[GitHub] incubator-omid pull request #24: [OMID-92] Add Apache Rat to the build proce...
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/24 [OMID-92] Add Apache Rat to the build process and align code with Rat⦠â¦'s requirement. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid OMID-92 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/24.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #24 commit aea8b149366264da01a883c616b34675dc74d4bf Author: Ohad Shacham <ohads@...> Date: 2018-03-12T13:04:27Z [OMID-92] Add Apache Rat to the build process and align code with Rat's requirement. ---
[GitHub] incubator-omid pull request #23: [OMID-89] Fix metrics in Persistence proces...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/23#discussion_r167857492 --- Diff: tso-server/src/test/java/org/apache/omid/tso/TestPersistenceProcessorHandler.java --- @@ -126,6 +126,36 @@ void afterMethod() { Mockito.reset(mockWriter); } +@Test(timeOut = 1_000) +public void testPersistentProcessorHandlerIdsAreCreatedConsecutive() throws Exception { + +TSOServerConfig tsoConfig = new TSOServerConfig(); +tsoConfig.setNumConcurrentCTWriters(32); + +PersistenceProcessorHandler[] handlers = new PersistenceProcessorHandler[tsoConfig.getNumConcurrentCTWriters()]; +for (int i = 0; i < tsoConfig.getNumConcurrentCTWriters(); i++) { +handlers[i] = new PersistenceProcessorHandler(metrics, + "localhost:1234", + mock(LeaseManager.class), + commitTable, + mock(ReplyProcessor.class), + retryProcessor, + panicker); +} + +for (int i = 0; i < tsoConfig.getNumConcurrentCTWriters(); i++) { +// Required to generalize the cases when other tests have increased the static variable assigning the ids +if (i + 1 < tsoConfig.getNumConcurrentCTWriters()) { +int followingHandlerIdAsInt = Integer.valueOf(handlers[i + 1].getId()); +assertEquals(handlers[i].getId(), String.valueOf(followingHandlerIdAsInt - 1)); --- End diff -- We kind of testing the atomic integer :) ---
[GitHub] incubator-omid pull request #17: [OMID-83] Attributes added to Put, Get, and...
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/17 ---
[GitHub] incubator-omid pull request #20: [OMID-85] Writing directly to HBase using s...
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/20 ---
[GitHub] incubator-omid pull request #20: [OMID-85] Writing directly to HBase using s...
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/20 [OMID-85] Writing directly to HBase using specific version marks the ⦠â¦write as a write that was done by a specific transaction. However, due to lack of shadow cells, getting the commit timestamp of the transaction can be done only by access the commit table. The motivation of this feature is to add the shadow cells during the write and save the commit table access. This feature is required by Apache Phoenix that during index creation adds the data table's entries, appeared before creation, to the index. In this case, the version and the commit timestamp should be the fence id and therefore, a direct write to HBase with the addition of shadow cells is required. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid AutoCommit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/20.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20 commit 462984833997c6192ae30788f5f7a3d866d41dce Author: Ohad Shacham <ohads@...> Date: 2018-02-01T15:13:08Z [OMID-85] Writing directly to HBase using specific version marks the write as a write that was done by a specific transaction. However, due to lack of shadow cells, getting the commit timestamp of the transaction can be done only by access the commit table. The motivation of this feature is to add the shadow cells during the write and save the commit table access. This feature is required by Apache Phoenix that during index creation adds the data table's entries, appeared before creation, to the index. In this case, the version and the commit timestamp should be the fence id and therefore, a direct write to HBase with the addition of shadow cells is required. ---
[GitHub] incubator-omid pull request #19: [OMID-84] Today, all the writes done by a t...
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/19 [OMID-84] Today, all the writes done by a transaction are taking part⦠⦠in conflict analysis. The purpose of this feature is to let the user decide for each write, whether it should take part in the conflict analysis. The motivation infers from Apache Phoenix that utilizes this feature when writing to the secondary index and also when writing to the data table for immutable tables (each key is added once and is not modified). You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid OMID-84-from-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/19.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19 commit 8fad7b57e3bb6ce6b69e753252c43c33af5a8eeb Author: Ohad Shacham <ohads@...> Date: 2018-01-31T14:52:43Z [OMID-84] Today, all the writes done by a transaction are taking part in conflict analysis. The purpose of this feature is to let the user decide for each write, whether it should take part in the conflict analysis. The motivation infers from Apache Phoenix that utilizes this feature when writing to the secondary index and also when writing to the data table for immutable tables (each key is added once and is not modified). ---
[GitHub] incubator-omid pull request #17: [OMID-83] Attributes added to Put, Get, and...
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/17 [OMID-83] Attributes added to Put, Get, and Scan are not propagated t⦠â¦o HBase. In many cases, as in the Phoenix case, these attributes are required and should be propagated to the server side. In Phoenix for example, attributes are used to mark data as one that should propagate to the secondary index. This commit propagates the attributes to HBase side. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid AttributesSupport Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/17.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17 commit 2bdf776bbb9bbf0e70dd49bec41239f6fcaf654c Author: Ohad Shacham <oh...@yahoo-inc.com> Date: 2017-12-19T08:24:26Z [OMID-83] Attributes added to Put, Get, and Scan are not propagated to HBase. In many cases, as in the Phoenix case, these attributes are required and should be propagated to the server side. In Phoenix for example, attributes are used to mark data as one that should propagate to the secondary index. This commit propagates the attributes to HBase side. ---
[GitHub] incubator-omid pull request #16: [OMID-78] Identify transaction snapshot at ...
Github user ohadshacham closed the pull request at: https://github.com/apache/incubator-omid/pull/16 ---
[GitHub] incubator-omid pull request #16: [OMID-78] Identify transaction snapshot at ...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/16#discussion_r150795077 --- Diff: hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/regionserver/RegionAccessWrapper.java --- @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; + +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.HTableInterface; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; +import org.apache.omid.transaction.TableAccessWrapper; + +public class RegionAccessWrapper implements TableAccessWrapper { --- End diff -- done ---
[GitHub] incubator-omid pull request #16: [OMID-78] Identify transaction snapshot at ...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/16#discussion_r150794291 --- Diff: hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/regionserver/OmidRegionScanner.java --- @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.apache.hadoop.hbase.Cell; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.regionserver.RegionScanner; +import org.apache.hadoop.hbase.regionserver.ScannerContext; +import org.apache.omid.transaction.HBaseTransaction; +import org.apache.omid.transaction.SnapshotFilter; + +public class OmidRegionScanner implements RegionScanner { + +RegionScanner scanner; +SnapshotFilter snapshotFilter; +HBaseTransaction transaction; +int maxVersions; +Map<String, List> familyDeletionCache; + +public OmidRegionScanner(SnapshotFilter snapshotFilter, + RegionScanner s, + HBaseTransaction transaction, + int maxVersions) { +this.snapshotFilter = snapshotFilter; +this.scanner = s; +this.transaction = transaction; +this.maxVersions = maxVersions; +this.familyDeletionCache = new HashMap<String, List>(); +} + +@Override +public boolean next(List results) throws IOException { + return next(results, Integer.MAX_VALUE); +} + +public boolean next(List result, int limit) throws IOException { --- End diff -- The function is defined in RegionScanner interface, we are just override it. ---
[GitHub] incubator-omid pull request #16: [OMID-78] Identify transaction snapshot at ...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/16#discussion_r150794375 --- Diff: hbase-client/src/main/java/org/apache/omid/transaction/HTableAccessWrapper.java --- @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.omid.transaction; + +import java.io.IOException; +import java.util.List; + +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.HTableInterface; +import org.apache.hadoop.hbase.client.Put; +import org.apache.hadoop.hbase.client.Result; + +public class HTableAccessWrapper implements TableAccessWrapper { --- End diff -- RegionAccessWrapper ---
[GitHub] incubator-omid pull request #16: [OMID-78] Identify transaction snapshot at ...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/16#discussion_r150794314 --- Diff: hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/regionserver/OmidRegionScanner.java --- @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.apache.hadoop.hbase.Cell; +import org.apache.hadoop.hbase.HRegionInfo; +import org.apache.hadoop.hbase.regionserver.RegionScanner; +import org.apache.hadoop.hbase.regionserver.ScannerContext; +import org.apache.omid.transaction.HBaseTransaction; +import org.apache.omid.transaction.SnapshotFilter; + +public class OmidRegionScanner implements RegionScanner { + +RegionScanner scanner; +SnapshotFilter snapshotFilter; +HBaseTransaction transaction; +int maxVersions; +Map<String, List> familyDeletionCache; + +public OmidRegionScanner(SnapshotFilter snapshotFilter, + RegionScanner s, + HBaseTransaction transaction, + int maxVersions) { +this.snapshotFilter = snapshotFilter; +this.scanner = s; +this.transaction = transaction; +this.maxVersions = maxVersions; +this.familyDeletionCache = new HashMap<String, List>(); +} + +@Override +public boolean next(List results) throws IOException { + return next(results, Integer.MAX_VALUE); +} + +public boolean next(List result, int limit) throws IOException { +return nextRaw(result, limit); +} + +@Override +public void close() throws IOException { +scanner.close(); +} + +@Override +public HRegionInfo getRegionInfo() { +return scanner.getRegionInfo(); +} + +@Override +public boolean isFilterDone() throws IOException { +return scanner.isFilterDone(); +} + +@Override +public boolean reseek(byte[] row) throws IOException { +throw new RuntimeException("Not implemented"); +} + +@Override +public long getMaxResultSize() { +return scanner.getMaxResultSize(); +} + +@Override +public long getMvccReadPoint() { +return scanner.getMvccReadPoint(); +} + +@Override +public boolean nextRaw(List result) throws IOException { --- End diff -- The function is defined in RegionScanner interface, we are just override it. ---
[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/13#discussion_r130809257 --- Diff: hbase-client/src/main/java/org/apache/omid/transaction/TTable.java --- @@ -396,20 +443,48 @@ public ResultScanner getScanner(Transaction tx, Scan scan) throws IOException { return commitCache; } -private boolean isCellInSnapshot(Cell kv, HBaseTransaction transaction, Map<Long, Long> commitCache) -throws IOException { +private void buildFamilyDeletionCache(List rawCells, Map<String, List> familyDeletionCache) { -long startTimestamp = transaction.getStartTimestamp(); +for (Cell cell : rawCells) { +if (CellUtil.matchingQualifier(cell, CellUtils.FAMILY_DELETE_QUALIFIER) && +CellUtil.matchingValue(cell, HConstants.EMPTY_BYTE_ARRAY)) { + +String row = Bytes.toString(cell.getRow()); +List cells = familyDeletionCache.get(row); --- End diff -- This is Map's return value. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/13#discussion_r130805885 --- Diff: hbase-client/src/main/java/org/apache/omid/transaction/TTable.java --- @@ -228,21 +261,12 @@ public void delete(Transaction tx, Delete delete) throws IOException { } } } -if (issueGet) { -// It's better to perform a transactional get to avoid deleting more -// than necessary -Result result = this.get(transaction, deleteG); -if (!result.isEmpty()) { -for (Entry<byte[], NavigableMap<byte[], NavigableMap<Long, byte[]>>> entryF : result.getMap() -.entrySet()) { -byte[] family = entryF.getKey(); -for (Entry<byte[], NavigableMap<Long, byte[]>> entryQ : entryF.getValue().entrySet()) { -byte[] qualifier = entryQ.getKey(); -deleteP.add(family, qualifier, CellUtils.DELETE_TOMBSTONE); -transaction.addWriteSetElement(new HBaseCellId(table, delete.getRow(), family, qualifier, - transaction.getStartTimestamp())); -} -} + +if (deleteFamily) { --- End diff -- 1. This is a deletion path without read requirement. It is doable only in row level conflict detection mode where we only need to mark the row as a player for conflict analysis. 2. TTable works only for HBase. You can see that each one of the method starts with a check that the transaction is hbase transaction. IMHO TTable should inherit from HTableInterface and override its functions, but this is for a different patch :). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...
GitHub user ohadshacham opened a pull request: https://github.com/apache/incubator-omid/pull/13 [OMID-74] Efficient column family deletion in Row level conflict analysis The idea is to use a qualifier to denote that all the columns of a specific family were deleted. Current implementation reads from HBase the entire family and then writes a tombstone to each one of its cells. The new implementation does not need to perform the read and only writes the qualifier to denote that the family was deleted. This is true only for Row level conflict detection since in Cell level we need to read the cells and add these to the write set. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ohadshacham/incubator-omid FamilyDeletionTombstone-squash Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/13.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13 commit 35ba4c79b4d3f713c6c38f791d5898f778153e2a Author: Ohad Shacham <oh...@yahoo-inc.com> Date: 2017-07-27T13:12:31Z [OMID-74] Efficient column family deletion in Row level conflict analysis The idea is to use a qualifier to denote that all the columns of a specific family were deleted. Current implementation reads from HBase the entire family and then writes a tombstone to each one of its cells. The new implementation does not need to perform the read and only writes the qualifier to denote that the family was deleted. This is true only for Row level conflict detection since in Cell level we need to read the cells and add these to the write set. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---