[
https://issues.apache.org/jira/browse/KYLIN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786627#comment-17786627
]
Hongrong Cao commented on KYLIN-5714:
-------------------------------------
h2. Root Cause
The global configuration is used when typing task diagnostic packages, so if
the event log dir is adjusted for a project or model separately, the event log
for the corresponding task will not be found in the global configuration, and
the diagnostic package will not be typed.
h2. Fix Design
h2. Proposal1 - Current Adoption
Support project or model level configuration of
kylin.engine.spark-conf.spark.eventLog.dir when typing task diagnostic
packages, as described in the current one product configuration
kylin.engine.spark-conf.* also supports model level configuration:In practice,
you need to differentiate the available configuration levels by task type.
*Supported up to the project level*
Tasks that do not involve models, such as sampling, snapshot building, snapshot
refreshing, etc.
*Support to model level*
Tasks that involve the model, e.g. index building, data loading, data
refreshing, etc.
Configuration prioritization remains the mainstream practice: model > project >
global.
{*}Known limitation{*}: If the path to this configuration changes, event logs
for old tasks cannot be downloaded to the
h2. Proposal2
Parses the spark.eventLog.dir parameter in the task metadata at the time of
task submission, and gets the path to the event log file from that parameter.
*Advantages:* No need to worry about configuration path changes, get the
configuration path at the moment of task execution
*Disadvantages:* Increases complexity, and changes to the configuration are a
low-frequency operation.
> When the project-level build job event log directory is inconsistent with the
> system level, the event log file is missing in the job_history directory of
> the job diagnostic package
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KYLIN-5714
> URL: https://issues.apache.org/jira/browse/KYLIN-5714
> Project: Kylin
> Issue Type: Bug
> Affects Versions: 5.0-beta
> Reporter: Hongrong Cao
> Assignee: Yinghao Lin
> Priority: Minor
> Fix For: 5.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
