Jungtaek, What is shown at the new URL and how would this improve usability?
On Thu, Feb 7, 2019 at 12:45 AM Jungtaek Lim <kabh...@gmail.com> wrote: > Hi devs, > > Based on the suggestion Tom Graves gave me in SPARK-26792 > <https://issues.apache.org/jira/browse/SPARK-26792>, I'd like to hear > voices on changing default executor log URLs for YARN, specifically > removing "stdout" and "stderr" and provide link which shows log file"s". > For example, instead of referring two links below: > > http:// > <NM_HOST>:<NM_PORT>/node/containerlogs/<CONTAINER_ID>/<USER>/<stdout|stderr>?start=-4096 > > we just refer only one link below: > > http://<NM_HOST>:<NM_PORT>/node/containerlogs/<CONTAINER_ID>/<USER> > > I've checked new URL works with redirection on NM to jobhistory, so it > won't break what we currently supported. Going through the actual log file > would require two clicks instead of one click though. > > Given it introduces the change on UX I'd like to hear voices on this > before submitting a patch. If we'd rather keep this as it is, I would just > open the chance to apply custom log URL for Spark UI as well. > > Thanks in advance! > > FYI, below is the rationalization on discussion: > > While I worked regarding SPARK-23155 > <https://issues.apache.org/jira/browse/SPARK-23155>, I've got some inputs > around linking "log directory" instead of log urls for each "stdout" and > "stderr", because in real case end users would put more files then only > stdout and stderr (like gc logs). > > SPARK-23155 provides the way to modify log URL but it's only applied to > SHS, and in Spark UI in running apps it still only shows "stdout" and > "stderr". SPARK-26792 is for applying this to Spark UI as well, but I've > got suggestion to just change the default log URL. > > Thanks again, > Jungtaek Lim (HeartSaVioR) > -- Ryan Blue Software Engineer Netflix
