[
https://issues.apache.org/jira/browse/PHOENIX-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16975135#comment-16975135
]
chenglei edited comment on PHOENIX-5494 at 11/15/19 2:42 PM:
-------------------------------------------------------------
[~kozdemir] , thank you for the reply and explain, but I can not catch why
this optimization can not be aplplied to the reply writes ?
bq. The reply writes can have cells with different timestamps and can have
multiple mutations for the same row key within a batch
yes, these cells may have different timestamps and can have multiple
mutations, but we can limit the timestamp to the max timestamp of the all the
mutations when we pre-scan, and the pre-scan can scan all the versions. when we
get the cells from the pre-scan results, we can filter the cells which
timestamp are newer than the dataTable mutation:
such as following code when we pre-scan for replay writes in
{{IndexBuildManager.preScanAllRequiredRows}} of my patch:
{code:java}
if(indexMetaData.getReplayWrite() != null) {
long timestamp =
getMaxTimestamp(dataTableMutationsWithSameRowKeyAndTimestamp);
scan.setTimeRange(0, timestamp);
}
{code}
When we get the cells from the pre-scan results for replay writes in
{{CachedLocalTable.getCurrentRowState}} of my patch:
{code:java}
long timestamp =
mutation.getFamilyCellMap().values().iterator().next().get(0).getTimestamp();
List<Cell> newCells = new ArrayList<Cell>();
for(Cell cell : cells) {
if(cell.getTimestamp() < timestamp ) {
newCells.add(cell);
}
}
return newCells;
{code}
The logical in above code is same as the old {{LocalTable.getCurrentRowState}}
method.
bq.Regarding the concurrency issues for regular writes, they are taken care by
the existing row locks and using ConcurrentHashSet.
When preparing for index mutation, regular writes and replay writes are both
under row locks protection of {{IndexRegionObserver}}, so why there are
concurrency issues?
The optimization is applied to replay write in my uploaded patch and there is
no test failed.
was (Author: comnetwork):
[~kozdemir] , thank you for the reply and explain, but I can not catch why
this optimization can not be aplplied to the reply writes ?
bq. The reply writes can have cells with different timestamps and can have
multiple mutations for the same row key within a batch
yes, these cells may have different timestamps and can have multiple
mutations, but we can limit the timestamp to the max timestamp of the all the
mutations when we pre-scan, and the pre-scan can scan all the versions. when we
get the cells from the pre-scan results, we can filter the cells which
timestamp are newer than the dataTable mutation:
such as following code when we pre-scan for replay writes in
{{IndexBuildManager.preScanAllRequiredRows}} of my patch:
{code:java}
if(indexMetaData.getReplayWrite() != null) {
long timestamp =
getMaxTimestamp(dataTableMutationsWithSameRowKeyAndTimestamp);
scan.setTimeRange(0, timestamp);
}
{code}
When we get the cells from the pre-scan results for replay writes in
{{CachedLocalTable.getCurrentRowState}} of my patch:
{code:java}
long timestamp =
mutation.getFamilyCellMap().values().iterator().next().get(0).getTimestamp();
List<Cell> newCells = new ArrayList<Cell>();
for(Cell cell : cells) {
if(cell.getTimestamp() < timestamp ) {
newCells.add(cell);
}
}
return newCells;
{code}
bq.Regarding the concurrency issues for regular writes, they are taken care by
the existing row locks and using ConcurrentHashSet.
Regular writes and replay writes are both under row locks protection of
{{IndexRegionObserver}}, so why there are concurrency issues?
> Batched, mutable Index updates are unnecessarily run one-by-one
> ---------------------------------------------------------------
>
> Key: PHOENIX-5494
> URL: https://issues.apache.org/jira/browse/PHOENIX-5494
> Project: Phoenix
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Kadir OZDEMIR
> Priority: Major
> Labels: performance
> Attachments: 5494-4.x-HBase-1.5.txt,
> PHOENIX-5494-4.x-HBase-1.4.patch, PHOENIX-5494.master.001.patch,
> PHOENIX-5494.master.002.patch, PHOENIX-5494.master.003.patch,
> Screenshot_20191110_160243.png, Screenshot_20191110_160351.png,
> Screenshot_20191110_161453.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> I just noticed that index updates on mutable tables retrieve their deletes
> (to invalidate the old index entry) one-by-one.
> For batches, this can be *the* major time spent during an index update. The
> cost is mostly incured by the repeated setup (and seeking) of the new region
> scanner (for each row).
> We can instead do a skip scan and get all updates in a single scan per region.
> (Logically that is simple, but it will require some refactoring)
> I won't be getting to this, but recording it here in case someone feels
> inclined.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
