[
https://issues.apache.org/jira/browse/PHOENIX-3853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021748#comment-16021748
]
Rajeshbabu Chintaguntla commented on PHOENIX-3853:
--------------------------------------------------
One big thing I have observed is that calling postPut,postDelete coprocessor
hooks, scan related hooks all at a time on the region causing the slowness. The
reason for slowness is that we are doing scans for index maintenance and
writing data altogether to same region and and may be intermediate compactions
and splits adds little overhead with huge batch size. Do we really need
postPut, postDelete coprocessor implementations now after PHOENIX-3789 because
we are doing the same functionality in postBatchMutateIndispensably already?
The hooks will be called for every mutation and adding lot of overhead.
Observed 5x performance improvement without the hooks implementation.
With hooks:
{noformat}
csv columns from database.
CSV Upsert complete. 900000 rows upserted
Time: 1156.882 sec(s)
{noformat}
Without hooks:
{noformat}
csv columns from database.
CSV Upsert complete. 900000 rows upserted
Time: 194.737 sec(s)
{noformat}
> Local Index - Writes to local index are twice as slow as global and get
> exponentially slower with PHOENIX-3827_v2 patch
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-3853
> URL: https://issues.apache.org/jira/browse/PHOENIX-3853
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.10.0
> Environment: HBase 1.3.1 4GB heap in local mode
> Reporter: Mujtaba Chohan
> Fix For: 4.11.0
>
> Attachments: after_3858.png, batch_mutable.png, batch_time.png,
> PHOENIX-3853_test_only.patch
>
>
> HBase 1.3.1 with head of Phoenix 4.x with/without PHOENIX-3827 v2 patch
> applied. This is with immutable non-covered local/global index on a single
> column with varying batch size when writing data to base table plus index.
> !batch_time.png!
> | Batch Size | Local Index with PHOENIX-3827_v2 patch (sec)| Local Index
> without PHOENIX-3827_v2.patch (sec)| Global (sec)|
> | 100 | 0.02 | 0.03 | 0.013 |
> | 1000 | 0.3 | 0.3 | 0.13 |
> | 10000 | 4.3 | 2.6 | 1.3 |
> | 12500 | 8.1 | 3 | 1.6 |
> | 15000 | 13.3 | 3.1 | 1.9 |
> Schema and index
> {noformat}
> CREATE TABLE IF NOT EXISTS T (OID CHAR(15) NOT NULL, PKP CHAR(3) NOT NULL,
> PIH CHAR(15) NOT NULL, FD DATE NOT NULL, SB CHAR(15) NOT NULL, BJ CHAR(15),
> JR VARCHAR, FIELD VARCHAR, YM VARCHAR, WN VARCHAR, LG VARCHAR, XHJ VARCHAR,
> HF VARCHAR, GA VARCHAR, MX VARCHAR, NZ DECIMAL, JV DECIMAL, AG DATE, KV DATE,
> JK VARCHAR, DK VARCHAR, EU DATE, OE VARCHAR, DV INTEGER, IK VARCHAR
> CONSTRAINT PK PRIMARY KEY ( OID, PKP, PIH, FD DESC, SB ))
> VERSIONS=1,IMMUTABLE_ROWS=true
> CREATE INDEX IF NOT EXISTS IDXT ON T (JV)
> {noformat}
> Data CSV
> https://expirebox.com/download/1cea73af1831b5193f0539d6e3442292.html
> [~rajeshbabu], [~lhofhansl], [~jamestaylor]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
