[
https://issues.apache.org/jira/browse/KYLIN-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786662#comment-17786662
]
Hongrong Cao commented on KYLIN-5718:
-------------------------------------
h2. Root Cause
The global dictionary V3 is stored as a Delta Table, and incremental
dictionaries are Merged into the existing dictionary for each build. As the
number of build tasks grows, the dictionary file grows with it.
h2. Dev Design
Add two parameters
{quote}*kylin.build.v3dict-file-num-limit=10*
Controls the number of files in the V3 dictionary, beyond which a file merge is
performed on the next build of the V3 dictionary. The default value is 10 files.
*kylin.build.v3dict-file-retention=3d*
Controls the length of time that garbage files of a V3 dictionary are retained,
after which they are cleaned up and deleted in the next build of the V3
dictionary. The default value is 3 days.
{quote}
[https://docs.delta.io/2.0.2/delta-utility.html#id2]
> Support automatic merging of V3 dictionary files
> ------------------------------------------------
>
> Key: KYLIN-5718
> URL: https://issues.apache.org/jira/browse/KYLIN-5718
> Project: Kylin
> Issue Type: Improvement
> Affects Versions: 5.0-beta
> Reporter: Hongrong Cao
> Assignee: Yaguang Jia
> Priority: Blocker
> Fix For: 5.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
