Hi,
Is there a way to call of MajorDeltaCompactionOp for a table/tablet/rowset?
We've faced with an issue:
0. Kudu table is created
1. Data is inserted there
2. Run select query - it goes fast (matter of a few seconds)
3. Delete all data from the table (but not dropping the table)
4. Run select query - it goes slow (4-6 minutes)
Investigating and reading documentation of Kudu has leaded to a thought that
delete operations are done logically, but physically the table contains written
data and deletes are applied each time on top of it.
I had a look at kudu tablet and there are quite large "redo" blocks (see one of
rowset examples below).
There was a thought that compression and encoding play their role (reducing the
chances to run compaction), but removing them (keeping column defaults) hasn't
helped as well.
We run tservers
- maintenance_manager_num_threads=10 (increased comparing to default)
- tablet_delta_store_major_compact_min_ratio=0.10000000149011612
(default value)
- kudu 1.7.0-cdh5.15.0
>From documentation and comments in code I saw the description of
>tablet_delta_store_major_compact_min_ratio: "Minimum ratio of sizeof(deltas)
>to sizeof(base data) before a major compaction."
And "Major compactions: the score will be the result of
sizeof(deltas)/sizeof(base data), unless it is smaller than
tablet_delta_store_major_compact_min_ratio or if the delta files are only
composed of deletes, in which case the score is brought down to zero."
So basically the table stays in such state for more than a day.
While majority of tables will have mostly scans, there will be a couple of
large tables with large number of deletions (but not of all data).
Could you advise how to improve scans after large deletions?
block-id | block-kind | column| cfile-size | cfile-data-type |
cfile-delta-stats
| cfile-encoding | cfile-compression
----------+-------------+-------+-----------
+-----------------+-----------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------
24693586 | column | var1 | 2.80M | int64 |
| BIT_SHUFFLE | NO_COMPRESSION
24693587 | column | var2 | 100.7K | int64 |
| BIT_SHUFFLE | NO_COMPRESSION
24693588 | column | var3 | 4.95M | int64 |
| BIT_SHUFFLE | NO_COMPRESSION
24693589 | column | var4 | 1.58M | string |
| DICT_ENCODING | LZ4
24693590 | column | var5 | 8.82M | string |
| PLAIN_ENCODING | LZ4
24693591 | column | var6 | 2.7K | string |
| DICT_ENCODING | LZ4
24700691 | redo | | 14.04M | binary | ts
range=[6319363930065100800, 6319364908129189926], delete_count=[2190649],
reinsert_count=[0], update_counts_by_col_id=[] | PLAIN_ENCODING | LZ4
24693592 | bloom | | 5.04M | binary |
| PLAIN_ENCODING | NO_COMPRESSION
24693593 | adhoc-index | | 8.94M | binary |
| PREFIX_ENCODING | LZ4
Kind Regards,
Sergejs Andrejevs
