If log file aggregation is enabled, aggregated logs for terminated logs
should be available on HDFS and retrievable with:
*yarn logs -applicationId **<application ID>*
Log aggregation is discussed further at:
https://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
and is controlled by the following configuration parameters:
*yarn.log-aggregation-enable*
*yarn.log-aggregation.retain-check-interval-seconds*
*yarn.log-aggregation.retain-seconds*
*yarn.nodemanager.log-aggregation.compression-type*
*yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds*
For running applications, logs are not on HDFS but rather on the local
filesystems of the individual nodes; the location
depends on the following configuration parameters:
*yarn.nodemanager.remote-app-log-dir*
*yarn.log.dir*
*yarn.nodemanager.log-dirs*
Some example locations are discussed in the Beginner's Guide at:
http://docs.datatorrent.com/beginner/
in the section entitled: *Checking logs*
Ram
On Thu, Mar 23, 2017 at 6:43 AM, Mukkamula, Suryavamshivardhan (CWM-NR) <
[email protected]> wrote:
> Hello Team,
>
> One of our data torrent application , we cannot view the logs from web
> console due to firewall issues(Datatorrent installed by service user ‘A’
> and application is being launched by service user ‘B’).
>
> Can you please let us know where can I find the path of the application
> logs (running and history jobs) either on the HDFS or individual nodes ? ,
> we have yarn logs from the Hadoop web console but due to the same fire wall
> issues we can only view the logs through window server of same domain and
> it is very painful and time consuming.
>
> While the platform team trying to fix the issue , can you please let us
> know where can I find the log files on the HDFS or individual nodes.
>
> Regards,
> Surya Vamshi
>
>
>
>
> _______________________________________________________________________
>
> If you received this email in error, please advise the sender (by return
> email or otherwise) immediately. You have consented to receive the attached
> electronically at the above-noted email address; please retain a copy of
> this confirmation for future reference.
>
> Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur
> immédiatement, par retour de courriel ou par un autre moyen. Vous avez
> accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à
> l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de
> cette confirmation pour les fins de reference future.
>
>
--
_______________________________________________________
Munagala V. Ramanath
Software Engineer
E: [email protected] | M: (408) 331-5034 | Twitter: @UnknownRam
www.datatorrent.com | apex.apache.org
