[
https://issues.apache.org/jira/browse/KYLIN-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15970363#comment-15970363
]
liyang commented on KYLIN-2506:
-------------------------------
[~kangkaisen], I briefly reviewed at the module and class level. Made some
changes, rebased on master, and pushed to {{master-KYLIN-2506}}. We can
continue work together on that branch.
- It is not a good design to let dictionary to use a job lock. Refactored to
split the JobLock from a general purpose DistributedLock.
- Renamed many methods on the interfaces so they look better on one class
(ZookeeperDistributedJobLock).
- Removed the curator dependency from core-common.
Also some open questions / suggestions.
- Seems the number of classes in {{org.apache.kylin.dict}} are too many, maybe
move the global dictionary classes to a new sub-package?
- DictNode / DictSlice / DictSliceKey are vague names, better rename them to
show their are specific to global dictionary
- Are the global dict metadata still backward compatibility? If user upgrade
from an older version, anything to watch out? Move/rename classes may cause
problem here.
- How necessary or urgent is this for 2.0.0 release? Personally I feel it is
too big for 2.0.0.
> Refactor Global Dictionary
> --------------------------
>
> Key: KYLIN-2506
> URL: https://issues.apache.org/jira/browse/KYLIN-2506
> Project: Kylin
> Issue Type: Improvement
> Components: General
> Affects Versions: v2.0.0
> Reporter: kangkaisen
> Assignee: kangkaisen
> Fix For: v2.0.0
>
>
> The main points of this refactor:
> 1 Fix the bug that the RemoveListener of LoadingCache swallowed any
> exceptions when building the GlobalDict.
> 2 Fix the bug that the HDFS filename of DictSliceKey had Illegal characters.
> 3 Fix the bug that the HDFS filename of DictSliceKey maybe longer than 255.
> 4 Fix the bug that DictNode split failed if value length greater than 255
> bytes.
> 5 Decouple the build and query of GlobalDict:
> Abstract the builder of AppendTrieDictionary to AppendTrieDictionaryBuilder;
> Add LoadingCache to AppendTrieDictionary and make AppendTrieDictionary is
> only readable.
> 6 Remove dependence of LoadingCache when building the GlobalDict.
> 7 Abstract the HDFS operations to GlobalDictStore.
> 8 Abstract the metadata of GlobalDict to GlobalDictMetadata.
> 9 Delete CachedTreeMap.
> 10 Add distributed lock for GlobalDict.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
