[
https://issues.apache.org/jira/browse/PHOENIX-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16975135#comment-16975135
]
chenglei edited comment on PHOENIX-5494 at 11/15/19 3:03 PM:
-------------------------------------------------------------
[~kozdemir] , thank you for the reply and explain, but I can not catch why
this optimization can not be aplplied to the reply writes ?
bq. The reply writes can have cells with different timestamps and can have
multiple mutations for the same row key within a batch
yes, these cells may have different timestamps and can have multiple
mutations, but we can limit the timestamp to the max timestamp of the all the
mutations when we pre-scan. When we get the cells from the pre-scan results, we
can filter the cells which timestamp are newer than the dataTable mutation,
such as following code:
when we pre-scan for replay writes in
{{IndexBuildManager.preScanAllRequiredRows}} of my patch:
{code:java}
if(indexMetaData.getReplayWrite() != null) {
long timestamp =
getMaxTimestamp(dataTableMutationsWithSameRowKeyAndTimestamp);
scan.setTimeRange(0, timestamp);
}
{code}
When we get the cells from the pre-scan results for replay writes in
{{CachedLocalTable.getCurrentRowState}} of my patch:
{code:java}
long timestamp =
mutation.getFamilyCellMap().values().iterator().next().get(0).getTimestamp();
List<Cell> newCells = new ArrayList<Cell>();
for(Cell cell : cells) {
if(cell.getTimestamp() < timestamp ) {
newCells.add(cell);
}
}
return newCells;
{code}
The logical in above code is same as the old {{LocalTable.getCurrentRowState}}
method.
bq.Regarding the concurrency issues for regular writes, they are taken care by
the existing row locks and using ConcurrentHashSet.
What the existing row locks mean and the {{ConcurrentHashSet}} is used for
what ? There is no row locks when prepare index mutations in
{{IndexRegionObserver}} for replay writes, but if we take the pre-scan results
as a local variable as my suggestion, there is no concurrent issues.
The optimization is applied to replay writes in my uploaded patch, there is no
failed tests.
was (Author: comnetwork):
[~kozdemir] , thank you for the reply and explain, but I can not catch why
this optimization can not be aplplied to the reply writes ?
bq. The reply writes can have cells with different timestamps and can have
multiple mutations for the same row key within a batch
yes, these cells may have different timestamps and can have multiple
mutations, but we can limit the timestamp to the max timestamp of the all the
mutations when we pre-scan. When we get the cells from the pre-scan results, we
can filter the cells which timestamp are newer than the dataTable mutation,
such as following code:
when we pre-scan for replay writes in
{{IndexBuildManager.preScanAllRequiredRows}} of my patch:
{code:java}
if(indexMetaData.getReplayWrite() != null) {
long timestamp =
getMaxTimestamp(dataTableMutationsWithSameRowKeyAndTimestamp);
scan.setTimeRange(0, timestamp);
}
{code}
When we get the cells from the pre-scan results for replay writes in
{{CachedLocalTable.getCurrentRowState}} of my patch:
{code:java}
long timestamp =
mutation.getFamilyCellMap().values().iterator().next().get(0).getTimestamp();
List<Cell> newCells = new ArrayList<Cell>();
for(Cell cell : cells) {
if(cell.getTimestamp() < timestamp ) {
newCells.add(cell);
}
}
return newCells;
{code}
The logical in above code is same as the old {{LocalTable.getCurrentRowState}}
method.
> Batched, mutable Index updates are unnecessarily run one-by-one
> ---------------------------------------------------------------
>
> Key: PHOENIX-5494
> URL: https://issues.apache.org/jira/browse/PHOENIX-5494
> Project: Phoenix
> Issue Type: Improvement
> Reporter: Lars Hofhansl
> Assignee: Kadir OZDEMIR
> Priority: Major
> Labels: performance
> Attachments: 5494-4.x-HBase-1.5.txt,
> PHOENIX-5494-4.x-HBase-1.4.patch, PHOENIX-5494.master.001.patch,
> PHOENIX-5494.master.002.patch, PHOENIX-5494.master.003.patch,
> Screenshot_20191110_160243.png, Screenshot_20191110_160351.png,
> Screenshot_20191110_161453.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> I just noticed that index updates on mutable tables retrieve their deletes
> (to invalidate the old index entry) one-by-one.
> For batches, this can be *the* major time spent during an index update. The
> cost is mostly incured by the repeated setup (and seeking) of the new region
> scanner (for each row).
> We can instead do a skip scan and get all updates in a single scan per region.
> (Logically that is simple, but it will require some refactoring)
> I won't be getting to this, but recording it here in case someone feels
> inclined.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
