[
https://issues.apache.org/jira/browse/OOZIE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343997#comment-14343997
]
Rohini Palaniswamy commented on OOZIE-2156:
-------------------------------------------
Tez has other files than stdout, stderr and syslog in the log directory in
YARN. So I was wondering if it possible to add the hive query logs to the log
collection framework and asked [~jlowe] from our hadoop team. Below is his
answer.
YARN log aggregation will collect any file readable by the NM user (mapred)
that's at the top of the container's log directory. All the logs you see for
Tez jobs (e.g.: DAG logs, post logs, etc.) are just random files YARN sees left
in the logs directory, so it aggregates them. That's also why you see weird
files aggregated sometimes, like the .crc file for a local history file. As
another example, we've had some users direct GC logging or task profiling
output to files under <LOG_DIR>/ and YARN picks them up as part of the
container logs automatically. So other than making sure the file is readable
by the mapred user you should be all set just making another random file in the
container log directory.
[~egashira],
Can you check what is the permission of the querylog file created by hive
(dfs umask set to 077) and if it is readable by all (444), then set the
location to yarn log dir location if HadoopShims.isYARN() ? Else we can leave
it at this.
> override hive.querylog.location in hive-site.xml
> -------------------------------------------------
>
> Key: OOZIE-2156
> URL: https://issues.apache.org/jira/browse/OOZIE-2156
> Project: Oozie
> Issue Type: Bug
> Affects Versions: trunk
> Reporter: Ryota Egashira
> Assignee: Ryota Egashira
> Fix For: trunk
>
> Attachments: OOZIE-2156.patch
>
>
> OOZIE-2068 allow users to fetch hive-site.xml from sharelib.
> but user still have to override hive.querylog.location in their workflow
> definition (to point to current working directory on compute node), otherwise
> it fails. Oozie should override for user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
