Hi Richard, thanks for forwarding my answer to the list! I fear that Flink does not have a built-in solution for serving the logs of a finished TaskManager while a YARN session is still running.
I agree with Yangze that you probably have to rely on an external logging service, such as ElasticSearch or Splunk to index your log events. Maybe there's also some tooling from Cloudera specifically made for YARN? On Mon, Jul 27, 2020 at 3:46 AM Yangze Guo <karma...@gmail.com> wrote: > Hi, Richard > > Before the session has been terminated, you could not fetch the > terminated TM logs. One possible solution could be leveraging the > log4j2 appenders[1]. Flink uses log4j2 as default in the latest > release 1.11. > > [1] https://logging.apache.org/log4j/2.x/manual/appenders.html > > Best, > Yangze Guo > > On Sat, Jul 25, 2020 at 2:37 AM Richard Moorhead > <richard.moorh...@gmail.com> wrote: > > > > > > > > ---------- Forwarded message --------- > > From: Robert Metzger <rmetz...@apache.org> > > Date: Fri, Jul 24, 2020 at 1:09 PM > > Subject: Re: Flink Session TM Logs > > To: Richard Moorhead <richard.moorh...@gmail.com> > > > > > > I accidentally replied to you directly, not to the user@ mailing list. > Is it okay for you to publish the thread on the list again? > > > > > > > > On Fri, Jul 24, 2020 at 8:01 PM Richard Moorhead < > richard.moorh...@gmail.com> wrote: > >> > >> It is enabled. The issue is that for a long running flink session > -which may execute many jobs- the task managers, after a job is completed, > are gone, and their logs arent available. > >> > >> What I have noticed is that when the session is terminated I am able to > find the logs in the job history server under the associated yarn > application id. > >> > >> On Fri, Jul 24, 2020 at 12:51 PM Robert Metzger <rmetz...@apache.org> > wrote: > >>> > >>> Hi Richard, > >>> > >>> you need to enable YARN log aggregation to access logs of finished > YARN applications. > >>> > >>> On Fri, Jul 24, 2020 at 5:58 PM Richard Moorhead < > richard.moorh...@gmail.com> wrote: > >>>> > >>>> When running a flink session on YARN, task manager logs for a job are > not available after completion. How do we locate these logs? > >>>> >
