Re: Flink 'Job Cluster' mode Ui Access
Hi, Recently i upgraded flink version to 1.8.3 For Session cluster it shows the version correctly. But for job cluster. I get this in the logs *Starting StandaloneJobClusterEntryPoint (Version: , Rev:6322618, Date:04.09.2019 @ 22:07:41 CST)* And my Classpath has these jars: *Classpath: /opt/flink/lib/flink-metrics-prometheus-1.8.3.jar:/opt/flink/lib/flink-shaded-hadoop-2-uber-2.6.5-7.0.jar:/opt/flink/lib/job.jar:/opt/flink/lib/log4j-over-slf4j-1.7.28.jar:/opt/flink/lib/logback-classic-1.2.3.jar:/opt/flink/lib/logback-core-1.2.3.jar:/opt/flink/lib/flink-dist_2.11-1.8.3.jar::/opt/hdfs:* Do you have any idea what could have caused this? Best Regards, Jatin On Fri, Dec 13, 2019 at 6:18 PM Chesnay Schepler wrote: > Thank you for the logs. > > Flink can indeed find the WebUI files in the distribution, which is a bit > odd. > Since there are no static files serve in this case, the > StaticFileServerHandler is never set up in the first place (hence why we > didn't find any log statements). > > What I also found in the logs (and, looking back, in one of your earlier > replies) was this: Version: , Rev:ceba8af, Date:11.02.2019 @ > 22:17:09 CST > > This tells us a few things. > a) You are not using 1.8.1, but 1.7.2 (based on the revision) > b) You are not using an official release, since the build-date differs > from the official releases > > I tried one of the official 1.7.2 releases, and the WebUI is shown both > when using: > (after copying the wordcount example into /lib) > ./bin/standalone-job.sh start-foreground -j > org.apache.flink.examples.java.wordcount.WordCount > ./bin/standalone-job.sh start -j > org.apache.flink.examples.java.wordcount.WordCount > > Right now I don't know what else to look for; there are some discrepancies > as to what your environment is vs what you described, and as such I can > only recommend to carefully evaluate what you have actually running and > possibly try again with an official release. > > Regards, > Chesnay > > On 13/12/2019 09:58, Jatin Banger wrote: > > Sure, here it is. > Job Manager Logs with logging level as DEBUG > > On Wed, Dec 11, 2019 at 3:14 PM Chesnay Schepler > wrote: > >> Would it be possible for you to provide us with full debug log file? >> >> On 10/12/2019 18:07, Jatin Banger wrote: >> >> Yes, I did. >> >> On Tue, Dec 10, 2019 at 3:47 PM Arvid Heise wrote: >> >>> Hi Jatin, >>> >>> just to be sure. Did you increase the log level to debug [1] before >>> checking for *StaticFileServerHandler*? >>> >>> Best, >>> >>> Arvid >>> >>> [1] >>> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/logging.html#configuring-log4j >>> >>> On Mon, Dec 9, 2019 at 7:54 AM Jatin Banger >>> wrote: >>> >>>> Hi, >>>> >>>> I have checked the logs with this keyword *StaticFileServerHandler *in >>>> it, But there were no logs coming for "Flink Job Cluster". >>>> Then i checked for Flink Session Cluster, i was able to find the logs >>>> for the *StaticFileServerHandler *keyword. >>>> >>>> Can i raise this as bug ? >>>> >>>> Best Regards, >>>> Jatin >>>> >>>> >>>> On Thu, Dec 5, 2019 at 8:59 PM Chesnay Schepler >>>> wrote: >>>> >>>>> Ok, it's good to know that the WebUI files are there. >>>>> >>>>> Please enable DEBUG logging and try again, searching for messages from >>>>> the StaticFileServerHandler. >>>>> >>>>> This handler logs every file that is requested (which effectively >>>>> happens when the WebUI is being served); let's see what is actually being >>>>> requested. >>>>> >>>>> On 05/12/2019 05:57, Jatin Banger wrote: >>>>> >>>>> I have tried that already using >>>>> '$FLINK_HOME/bin/jobmanager.sh" start-foreground >>>>> Ui comes fine with this one. >>>>> Which means web/index.html is present. >>>>> >>>>> >>>>> On Wed, Dec 4, 2019 at 9:01 PM Chesnay Schepler >>>>> wrote: >>>>> >>>>>> hmm...this is quite odd. >>>>>> >>>>>> Let's try to narrow things down a bit. >>>>>> >>>>>> Could you try starting a local cluster (using the same distribution) >>>>>> and checking whether the UI is accessible? >>>>
Flink 'Job Cluster' mode Ui Access
Hi, It seems there is Web Ui for Flink Session cluster, But for Flink Job Cluster it is Showing {"errors":["Not found."]} Is it the expected behavior for Flink Job Cluster Mode ? Best Regards, Jatin
Re: Flink 'Job Cluster' mode Ui Access
Hi, I checked the log file there is no error. And I checked the pods internal ports by using rest api. # curl : 4081 {"errors":["Not found."]} 4081 is the Ui port # curl :4081/config {"refresh-interval":3000,"timezone-name":"Coordinated Universal Time","timezone-offset":0,"flink-version":"","flink-revision":"ceba8af @ 11.02.2019 @ 22:17:09 CST"} # curl :4081/jobs {"jobs":[{"id":"___job_Id_","status":"RUNNING"}]} Which shows the state of the job as running. What else can we do ? Best regards, Jatin On Thu, Nov 28, 2019 at 1:28 PM vino yang wrote: > Hi Jatin, > > Flink web UI does not depend on any deployment mode. > > You should check if there are error logs in the log file and the job > status is running state. > > Best, > Vino > > Jatin Banger 于2019年11月28日周四 下午3:43写道: > >> Hi, >> >> It seems there is Web Ui for Flink Session cluster, But for Flink Job >> Cluster it is Showing >> >> {"errors":["Not found."]} >> >> Is it the expected behavior for Flink Job Cluster Mode ? >> >> Best Regards, >> Jatin >> >
Re: Flink 'Job Cluster' mode Ui Access
Hi, @vino yang I am using flink 1.8.1 I am using the following procedure for the deployment: https://github.com/apache/flink/blob/master/flink-container/docker/README.md And i tried accessing the path you mentioned: # curl :4081/#/overview {"errors":["Not found."]} Best Regards, Jatin On Thu, Nov 28, 2019 at 10:21 PM Chesnay Schepler wrote: > Could you try accessing :/#/overview ? > > The REST API is obviously accessible, and hence the WebUI should be too. > > How did you setup the session cluster? Are you using some custom Flink > build or something, which potentially excluded flink-runtime-web from the > classpath? > > On 28/11/2019 10:02, Jatin Banger wrote: > > Hi, > > I checked the log file there is no error. > And I checked the pods internal ports by using rest api. > > # curl : 4081 > {"errors":["Not found."]} > 4081 is the Ui port > > # curl :4081/config > {"refresh-interval":3000,"timezone-name":"Coordinated Universal > Time","timezone-offset":0,"flink-version":"","flink-revision":"ceba8af > @ 11.02.2019 @ 22:17:09 CST"} > > # curl :4081/jobs > {"jobs":[{"id":"___job_Id_","status":"RUNNING"}]} > > Which shows the state of the job as running. > > What else can we do ? > > Best regards, > Jatin > > On Thu, Nov 28, 2019 at 1:28 PM vino yang wrote: > >> Hi Jatin, >> >> Flink web UI does not depend on any deployment mode. >> >> You should check if there are error logs in the log file and the job >> status is running state. >> >> Best, >> Vino >> >> Jatin Banger 于2019年11月28日周四 下午3:43写道: >> >>> Hi, >>> >>> It seems there is Web Ui for Flink Session cluster, But for Flink Job >>> Cluster it is Showing >>> >>> {"errors":["Not found."]} >>> >>> Is it the expected behavior for Flink Job Cluster Mode ? >>> >>> Best Regards, >>> Jatin >>> >> >
Re: Flink 'Job Cluster' mode Ui Access
Hi, I am using flink binary directly. I am using this command to deploy the script. "$FLINK_HOME/bin/standalone-job.sh" start-foreground --job-classname ${ARGS_FOR_JOB} where ARGS_FOR_JOB contain job class name and all other necessary details needed by the job. Best regards, Jatin On Fri, Nov 29, 2019 at 4:18 PM Chesnay Schepler wrote: > To clarify, you ran "mvn package -pl flink-dist -am" to build Fink? > > If so, could you run that again and provide us with the maven output? > > On 29/11/2019 11:23, Jatin Banger wrote: > > Hi, > > @vino yang I am using flink 1.8.1 > > I am using the following procedure for the deployment: > > https://github.com/apache/flink/blob/master/flink-container/docker/README.md > > And i tried accessing the path you mentioned: > > # curl :4081/#/overview > {"errors":["Not found."]} > > Best Regards, > Jatin > > On Thu, Nov 28, 2019 at 10:21 PM Chesnay Schepler > wrote: > >> Could you try accessing :/#/overview ? >> >> The REST API is obviously accessible, and hence the WebUI should be too. >> >> How did you setup the session cluster? Are you using some custom Flink >> build or something, which potentially excluded flink-runtime-web from the >> classpath? >> >> On 28/11/2019 10:02, Jatin Banger wrote: >> >> Hi, >> >> I checked the log file there is no error. >> And I checked the pods internal ports by using rest api. >> >> # curl : 4081 >> {"errors":["Not found."]} >> 4081 is the Ui port >> >> # curl :4081/config >> {"refresh-interval":3000,"timezone-name":"Coordinated Universal >> Time","timezone-offset":0,"flink-version":"","flink-revision":"ceba8af >> @ 11.02.2019 @ 22:17:09 CST"} >> >> # curl :4081/jobs >> {"jobs":[{"id":"___job_Id_","status":"RUNNING"}]} >> >> Which shows the state of the job as running. >> >> What else can we do ? >> >> Best regards, >> Jatin >> >> On Thu, Nov 28, 2019 at 1:28 PM vino yang wrote: >> >>> Hi Jatin, >>> >>> Flink web UI does not depend on any deployment mode. >>> >>> You should check if there are error logs in the log file and the job >>> status is running state. >>> >>> Best, >>> Vino >>> >>> Jatin Banger 于2019年11月28日周四 下午3:43写道: >>> >>>> Hi, >>>> >>>> It seems there is Web Ui for Flink Session cluster, But for Flink Job >>>> Cluster it is Showing >>>> >>>> {"errors":["Not found."]} >>>> >>>> Is it the expected behavior for Flink Job Cluster Mode ? >>>> >>>> Best Regards, >>>> Jatin >>>> >>> >> >
Re: Flink 'Job Cluster' mode Ui Access
I have tried that already using '$FLINK_HOME/bin/jobmanager.sh" start-foreground Ui comes fine with this one. Which means web/index.html is present. On Wed, Dec 4, 2019 at 9:01 PM Chesnay Schepler wrote: > hmm...this is quite odd. > > Let's try to narrow things down a bit. > > Could you try starting a local cluster (using the same distribution) and > checking whether the UI is accessible? > > Could you also check whether the flink-dist.jar in /lib contains > web/index.html? > On 04/12/2019 06:02, Jatin Banger wrote: > > Hi, > > I am using flink binary directly. > > I am using this command to deploy the script. > > "$FLINK_HOME/bin/standalone-job.sh" > start-foreground --job-classname ${ARGS_FOR_JOB} > where ARGS_FOR_JOB contain job class name and all other necessary details > needed by the job. > > Best regards, > Jatin > > > On Fri, Nov 29, 2019 at 4:18 PM Chesnay Schepler > wrote: > >> To clarify, you ran "mvn package -pl flink-dist -am" to build Fink? >> >> If so, could you run that again and provide us with the maven output? >> >> On 29/11/2019 11:23, Jatin Banger wrote: >> >> Hi, >> >> @vino yang I am using flink 1.8.1 >> >> I am using the following procedure for the deployment: >> >> https://github.com/apache/flink/blob/master/flink-container/docker/README.md >> >> And i tried accessing the path you mentioned: >> >> # curl :4081/#/overview >> {"errors":["Not found."]} >> >> Best Regards, >> Jatin >> >> On Thu, Nov 28, 2019 at 10:21 PM Chesnay Schepler >> wrote: >> >>> Could you try accessing :/#/overview ? >>> >>> The REST API is obviously accessible, and hence the WebUI should be too. >>> >>> How did you setup the session cluster? Are you using some custom Flink >>> build or something, which potentially excluded flink-runtime-web from the >>> classpath? >>> >>> On 28/11/2019 10:02, Jatin Banger wrote: >>> >>> Hi, >>> >>> I checked the log file there is no error. >>> And I checked the pods internal ports by using rest api. >>> >>> # curl : 4081 >>> {"errors":["Not found."]} >>> 4081 is the Ui port >>> >>> # curl :4081/config >>> {"refresh-interval":3000,"timezone-name":"Coordinated Universal >>> Time","timezone-offset":0,"flink-version":"","flink-revision":"ceba8af >>> @ 11.02.2019 @ 22:17:09 CST"} >>> >>> # curl :4081/jobs >>> {"jobs":[{"id":"___job_Id_","status":"RUNNING"}]} >>> >>> Which shows the state of the job as running. >>> >>> What else can we do ? >>> >>> Best regards, >>> Jatin >>> >>> On Thu, Nov 28, 2019 at 1:28 PM vino yang wrote: >>> >>>> Hi Jatin, >>>> >>>> Flink web UI does not depend on any deployment mode. >>>> >>>> You should check if there are error logs in the log file and the job >>>> status is running state. >>>> >>>> Best, >>>> Vino >>>> >>>> Jatin Banger 于2019年11月28日周四 下午3:43写道: >>>> >>>>> Hi, >>>>> >>>>> It seems there is Web Ui for Flink Session cluster, But for Flink Job >>>>> Cluster it is Showing >>>>> >>>>> {"errors":["Not found."]} >>>>> >>>>> Is it the expected behavior for Flink Job Cluster Mode ? >>>>> >>>>> Best Regards, >>>>> Jatin >>>>> >>>> >>> >> >
Re: Flink 'Job Cluster' mode Ui Access
Hi, I have checked the logs with this keyword *StaticFileServerHandler *in it, But there were no logs coming for "Flink Job Cluster". Then i checked for Flink Session Cluster, i was able to find the logs for the *StaticFileServerHandler *keyword. Can i raise this as bug ? Best Regards, Jatin On Thu, Dec 5, 2019 at 8:59 PM Chesnay Schepler wrote: > Ok, it's good to know that the WebUI files are there. > > Please enable DEBUG logging and try again, searching for messages from the > StaticFileServerHandler. > > This handler logs every file that is requested (which effectively happens > when the WebUI is being served); let's see what is actually being requested. > > On 05/12/2019 05:57, Jatin Banger wrote: > > I have tried that already using > '$FLINK_HOME/bin/jobmanager.sh" start-foreground > Ui comes fine with this one. > Which means web/index.html is present. > > > On Wed, Dec 4, 2019 at 9:01 PM Chesnay Schepler > wrote: > >> hmm...this is quite odd. >> >> Let's try to narrow things down a bit. >> >> Could you try starting a local cluster (using the same distribution) and >> checking whether the UI is accessible? >> >> Could you also check whether the flink-dist.jar in /lib contains >> web/index.html? >> On 04/12/2019 06:02, Jatin Banger wrote: >> >> Hi, >> >> I am using flink binary directly. >> >> I am using this command to deploy the script. >> >> "$FLINK_HOME/bin/standalone-job.sh" >> start-foreground --job-classname ${ARGS_FOR_JOB} >> where ARGS_FOR_JOB contain job class name and all other necessary details >> needed by the job. >> >> Best regards, >> Jatin >> >> >> On Fri, Nov 29, 2019 at 4:18 PM Chesnay Schepler >> wrote: >> >>> To clarify, you ran "mvn package -pl flink-dist -am" to build Fink? >>> >>> If so, could you run that again and provide us with the maven output? >>> >>> On 29/11/2019 11:23, Jatin Banger wrote: >>> >>> Hi, >>> >>> @vino yang I am using flink 1.8.1 >>> >>> I am using the following procedure for the deployment: >>> >>> https://github.com/apache/flink/blob/master/flink-container/docker/README.md >>> >>> And i tried accessing the path you mentioned: >>> >>> # curl :4081/#/overview >>> {"errors":["Not found."]} >>> >>> Best Regards, >>> Jatin >>> >>> On Thu, Nov 28, 2019 at 10:21 PM Chesnay Schepler >>> wrote: >>> >>>> Could you try accessing :/#/overview ? >>>> >>>> The REST API is obviously accessible, and hence the WebUI should be too. >>>> >>>> How did you setup the session cluster? Are you using some custom Flink >>>> build or something, which potentially excluded flink-runtime-web from the >>>> classpath? >>>> >>>> On 28/11/2019 10:02, Jatin Banger wrote: >>>> >>>> Hi, >>>> >>>> I checked the log file there is no error. >>>> And I checked the pods internal ports by using rest api. >>>> >>>> # curl : 4081 >>>> {"errors":["Not found."]} >>>> 4081 is the Ui port >>>> >>>> # curl :4081/config >>>> {"refresh-interval":3000,"timezone-name":"Coordinated Universal >>>> Time","timezone-offset":0,"flink-version":"","flink-revision":"ceba8af >>>> @ 11.02.2019 @ 22:17:09 CST"} >>>> >>>> # curl :4081/jobs >>>> {"jobs":[{"id":"___job_Id_","status":"RUNNING"}]} >>>> >>>> Which shows the state of the job as running. >>>> >>>> What else can we do ? >>>> >>>> Best regards, >>>> Jatin >>>> >>>> On Thu, Nov 28, 2019 at 1:28 PM vino yang >>>> wrote: >>>> >>>>> Hi Jatin, >>>>> >>>>> Flink web UI does not depend on any deployment mode. >>>>> >>>>> You should check if there are error logs in the log file and the job >>>>> status is running state. >>>>> >>>>> Best, >>>>> Vino >>>>> >>>>> Jatin Banger 于2019年11月28日周四 下午3:43写道: >>>>> >>>>>> Hi, >>>>>> >>>>>> It seems there is Web Ui for Flink Session cluster, But for Flink Job >>>>>> Cluster it is Showing >>>>>> >>>>>> {"errors":["Not found."]} >>>>>> >>>>>> Is it the expected behavior for Flink Job Cluster Mode ? >>>>>> >>>>>> Best Regards, >>>>>> Jatin >>>>>> >>>>> >>>> >>> >> >
Re: Flink 'Job Cluster' mode Ui Access
Yes, I did. On Tue, Dec 10, 2019 at 3:47 PM Arvid Heise wrote: > Hi Jatin, > > just to be sure. Did you increase the log level to debug [1] before > checking for *StaticFileServerHandler*? > > Best, > > Arvid > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/logging.html#configuring-log4j > > On Mon, Dec 9, 2019 at 7:54 AM Jatin Banger > wrote: > >> Hi, >> >> I have checked the logs with this keyword *StaticFileServerHandler *in >> it, But there were no logs coming for "Flink Job Cluster". >> Then i checked for Flink Session Cluster, i was able to find the logs for >> the *StaticFileServerHandler *keyword. >> >> Can i raise this as bug ? >> >> Best Regards, >> Jatin >> >> >> On Thu, Dec 5, 2019 at 8:59 PM Chesnay Schepler >> wrote: >> >>> Ok, it's good to know that the WebUI files are there. >>> >>> Please enable DEBUG logging and try again, searching for messages from >>> the StaticFileServerHandler. >>> >>> This handler logs every file that is requested (which effectively >>> happens when the WebUI is being served); let's see what is actually being >>> requested. >>> >>> On 05/12/2019 05:57, Jatin Banger wrote: >>> >>> I have tried that already using >>> '$FLINK_HOME/bin/jobmanager.sh" start-foreground >>> Ui comes fine with this one. >>> Which means web/index.html is present. >>> >>> >>> On Wed, Dec 4, 2019 at 9:01 PM Chesnay Schepler >>> wrote: >>> >>>> hmm...this is quite odd. >>>> >>>> Let's try to narrow things down a bit. >>>> >>>> Could you try starting a local cluster (using the same distribution) >>>> and checking whether the UI is accessible? >>>> >>>> Could you also check whether the flink-dist.jar in /lib contains >>>> web/index.html? >>>> On 04/12/2019 06:02, Jatin Banger wrote: >>>> >>>> Hi, >>>> >>>> I am using flink binary directly. >>>> >>>> I am using this command to deploy the script. >>>> >>>> "$FLINK_HOME/bin/standalone-job.sh" >>>> start-foreground --job-classname ${ARGS_FOR_JOB} >>>> where ARGS_FOR_JOB contain job class name and all other necessary >>>> details needed by the job. >>>> >>>> Best regards, >>>> Jatin >>>> >>>> >>>> On Fri, Nov 29, 2019 at 4:18 PM Chesnay Schepler >>>> wrote: >>>> >>>>> To clarify, you ran "mvn package -pl flink-dist -am" to build Fink? >>>>> >>>>> If so, could you run that again and provide us with the maven output? >>>>> >>>>> On 29/11/2019 11:23, Jatin Banger wrote: >>>>> >>>>> Hi, >>>>> >>>>> @vino yang I am using flink 1.8.1 >>>>> >>>>> I am using the following procedure for the deployment: >>>>> >>>>> https://github.com/apache/flink/blob/master/flink-container/docker/README.md >>>>> >>>>> And i tried accessing the path you mentioned: >>>>> >>>>> # curl :4081/#/overview >>>>> {"errors":["Not found."]} >>>>> >>>>> Best Regards, >>>>> Jatin >>>>> >>>>> On Thu, Nov 28, 2019 at 10:21 PM Chesnay Schepler >>>>> wrote: >>>>> >>>>>> Could you try accessing :/#/overview ? >>>>>> >>>>>> The REST API is obviously accessible, and hence the WebUI should be >>>>>> too. >>>>>> >>>>>> How did you setup the session cluster? Are you using some custom >>>>>> Flink build or something, which potentially excluded flink-runtime-web >>>>>> from >>>>>> the classpath? >>>>>> >>>>>> On 28/11/2019 10:02, Jatin Banger wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I checked the log file there is no error. >>>>>> And I checked the pods internal ports by using rest api. >>>>>> >>>>>> # curl : 4081 >>>>>> {"errors":["Not found."]} >>>>>> 4081 is the Ui port >>>>>> >>>>>> # curl :4081/config >>>>>> {"refresh-interval":3000,"timezone-name":"Coordinated Universal >>>>>> Time","timezone-offset":0,"flink-version":"","flink-revision":"ceba8af >>>>>> @ 11.02.2019 @ 22:17:09 CST"} >>>>>> >>>>>> # curl :4081/jobs >>>>>> {"jobs":[{"id":"___job_Id_","status":"RUNNING"}]} >>>>>> >>>>>> Which shows the state of the job as running. >>>>>> >>>>>> What else can we do ? >>>>>> >>>>>> Best regards, >>>>>> Jatin >>>>>> >>>>>> On Thu, Nov 28, 2019 at 1:28 PM vino yang >>>>>> wrote: >>>>>> >>>>>>> Hi Jatin, >>>>>>> >>>>>>> Flink web UI does not depend on any deployment mode. >>>>>>> >>>>>>> You should check if there are error logs in the log file and the job >>>>>>> status is running state. >>>>>>> >>>>>>> Best, >>>>>>> Vino >>>>>>> >>>>>>> Jatin Banger 于2019年11月28日周四 下午3:43写道: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> It seems there is Web Ui for Flink Session cluster, But for Flink >>>>>>>> Job Cluster it is Showing >>>>>>>> >>>>>>>> {"errors":["Not found."]} >>>>>>>> >>>>>>>> Is it the expected behavior for Flink Job Cluster Mode ? >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> Jatin >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>