Hey Diwakar, how are you deploying Flink on EMR? Are you using YARN? If so, you could also use log aggregation to see all the logs at once (from both JobManager and TaskManagers). (yarn logs -applicationId <Application ID>)
Could you post (or upload somewhere) all logs you have of one run? It is much easier for us to debug something if we have the full logs (the logs show for example the classpath that you are using, we would see how you are deploying Flink, etc.) >From the information available, my guess is that you have modified your deployment in some way (use of a custom logging version, custom deployment method, version mixup with jars from both Flink 1.8 and 1.11, ...). Best, Robert On Tue, Oct 27, 2020 at 12:41 AM Diwakar Jha <diwakar.n...@gmail.com> wrote: > This is what I see on the WebUI. > > 23:19:24.263 [flink-akka.actor.default-dispatcher-1865] ERROR > org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler > - Failed to transfer file from TaskExecutor > container_1603649952937_0002_01_000004. > java.util.concurrent.CompletionException: > org.apache.flink.util.FlinkException: The file LOG does not exist on the > TaskExecutor. at > org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$25( > TaskExecutor.java:1742 <http://taskexecutor.java:1742/>) > ~[flink-dist_2.12-1.11.0.jar:1.11.0] at > java.util.concurrent.CompletableFuture$AsyncSupply.run > <http://java.util.concurrent.completablefuture$asyncsupply.run/>( > CompletableFuture.java:1604 <http://completablefuture.java:1604/>) > ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1149 <http://threadpoolexecutor.java:1149/>) > ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor$Worker.run > <http://java.util.concurrent.threadpoolexecutor$worker.run/>( > ThreadPoolExecutor.java:624 <http://threadpoolexecutor.java:624/>) > ~[?:1.8.0_252] at java.lang.Thread.run <http://java.lang.thread.run/>( > Thread.java:748 <http://thread.java:748/>) ~[?:1.8.0_252] Caused by: > org.apache.flink.util.FlinkException: The file LOG does not exist on the > TaskExecutor. ... 5 more 23:19:24.275 > [flink-akka.actor.default-dispatcher-1865] ERROR > org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler > - Unhandled exception. org.apache.flink.util.FlinkException: The file LOG > does not exist on the TaskExecutor. at > org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$25( > TaskExecutor.java:1742 <http://taskexecutor.java:1742/>) > ~[flink-dist_2.12-1.11.0.jar:1.11.0] at > java.util.concurrent.CompletableFuture$AsyncSupply.run > <http://java.util.concurrent.completablefuture$asyncsupply.run/>( > CompletableFuture.java:1604 <http://completablefuture.java:1604/>) > ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1149 <http://threadpoolexecutor.java:1149/>) > ~[?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor$Worker.run > <http://java.util.concurrent.threadpoolexecutor$worker.run/>( > ThreadPoolExecutor.java:624 <http://threadpoolexecutor.java:624/>) > ~[?:1.8.0_252] at java.lang.Thread.run <http://java.lang.thread.run/>( > Thread.java:748 <http://thread.java:748/>) ~[?:1.8.0_252] > > Appreciate if anyone has any pointer for this. > > On Mon, Oct 26, 2020 at 10:45 AM Chesnay Schepler <ches...@apache.org> > wrote: > >> Flink 1.11 uses slf4j 1.7.15; the easiest way to check the log files is >> usually via the WebUI. >> >> On 10/26/2020 5:30 PM, Diwakar Jha wrote: >> >> I think my problem is with Sl4j library. I'm using sl4j 1.7 with Flink >> 1.11. If that's correct then i appreciate if someone can point me to the >> exact Slf4j library that i should use with Flink 1.11 >> >> Flink = 1.11.x; >> Slf4j = 1.7; >> >> >> On Sun, Oct 25, 2020 at 8:00 PM Diwakar Jha <diwakar.n...@gmail.com> >> wrote: >> >>> Thanks for checking my configurations. Could you also point me where I >>> can see the log files? Just to give more details. I'm trying to access >>> these logs in AWS cloudwatch. >>> >>> Best, >>> Diwakar >>> >>> On Sun, Oct 25, 2020 at 2:16 PM Chesnay Schepler <ches...@apache.org> >>> wrote: >>> >>>> With Flink 1.11 reporters were refactored to plugins, and are now >>>> accessible by default (so you no longer have to bother with copying jars >>>> around). >>>> >>>> Your configuration appears to be correct, so I suggest to take a look >>>> at the log files. >>>> >>>> On 10/25/2020 9:52 PM, Diwakar Jha wrote: >>>> >>>> Hello Everyone, >>>> >>>> I'm new to flink and i'm trying to upgrade from flink 1.8 to flink 1.11 >>>> on an emr cluster. after upgrading to flink1.11 One of the differences that >>>> i see is i don't get any metrics. I found out that flink 1.11 does not have >>>> *org.apache.flink.metrics.statsd.StatsDReporterFactory* jar in >>>> /usr/lib/flink/opt which was the case for flink 1.8. Could anyone have any >>>> pointer to locate >>>> *org.apache.flink.metrics.statsd.StatsDReporterFactory* jar or how to >>>> use metrics in flink.1.11? >>>> >>>> Things i tried : >>>> a) the below setup >>>> >>>> metrics.reporters: stsdmetrics.reporter.stsd.factory.class: >>>> org.apache.flink.metrics.statsd.StatsDReporterFactorymetrics.reporter.stsd.host: >>>> localhostmetrics.reporter.stsd.port: 8125 >>>> >>>> b) I tried downloading the statsd jar from >>>> https://mvnrepository.com/artifact/org.apache.flink/flink-metrics-statsd >>>> putting it inside plugins/statsd directory. >>>> >>>> >>>> -- >>>> Best, >>>> Diwakar Jha. >>>> >>>> >>>> >>> >>> -- >>> Best, >>> Diwakar Jha. >>> >> >> >> -- >> Best, >> Diwakar Jha. >> >> >> > > -- > Best, > Diwakar Jha. >
