[
https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393231#comment-15393231
]
Hitesh Shah commented on TEZ-3359:
----------------------------------
Comments:
{code}
} else {
1063 // Validate and set value from tezConf.
1064 logLevel =
tezConf.get(TezConfiguration.TEZ_HISTORY_LOGGING_LOGLEVEL);
1065 if (logLevel != null) {
1066 if (!HistoryLogLevel.validateLogLevel(logLevel)) {
1067 throw new IllegalArgumentException(
1068 "Config: " +
TezConfiguration.TEZ_HISTORY_LOGGING_LOGLEVEL +
1069 " is set to invalid value: " + logLevel);
1070 }
1071 PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
1072 kvp.setKey(TezConfiguration.TEZ_HISTORY_LOGGING_LOGLEVEL);
1073 kvp.setValue(logLevel);
1074 confProtoBuilder.addConfKeyValues(kvp);
1075 }
1076 }
{code}
- Shouldnt only validation be needed here given that the DAG will fall back
to the log level from the AM conf if there is no config set at the dag level (
i.e. no need to copy the am config value into the dag conf)?
{code}
// Set value in dag, should override tez conf value.
404 dag.setHistoryLogLevel(HistoryLogLevel.VERTEX);
405 dagPlan = dag.createDag(tezConf, null, null, null, false);
406 Assert.assertEquals("VERTEX",
findKVP(DAGPlan.newBuilder(dagPlan).getDagConf(),
407 TezConfiguration.TEZ_HISTORY_LOGGING_LOGLEVEL));
{code}
- could be augmented with
dag.setConf(TezConfiguration.TEZ_HISTORY_LOGGING_LOGLEVEL, "invalid") to test
all code paths
Patch mostly looks good.
Also, noticed that grouping of dags does not account for dag ids starting from
1 instead of 0. Maybe another sub task for that?
> Add granular log levels for HistoryLoggingService.
> --------------------------------------------------
>
> Key: TEZ-3359
> URL: https://issues.apache.org/jira/browse/TEZ-3359
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Harish Jaiprakash
> Assignee: Harish Jaiprakash
> Attachments: TEZ-3359.01.patch, TEZ-3359.02.patch, TEZ-3359.03.patch,
> TEZ-3359.04.patch, TEZ-3359.05.patch, TEZ-3359.06.patch, TEZ-3359.07.patch
>
>
> We publish too many events to ATS, this increases the file size for ATS.
> Reduce data size logged into ATS by having more granular control over the
> loglevel. Ex: disable task level logs, or log only until VERTEX and so on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
