Thanks Manu for your response. I already checked the logs and didn't see anything that can help me understanding the issue.
The more weird thing, i have a small CI cluster which run on single NameNode and i see the Spark2 job in the UI, i'm still not sure if it may related to the NameNode HA, i tried to replace the logdir from NameNode HA to the activeNameNode like this http://server:8020/user/spark/spark2historyapplication in the spark2 default conf but the UI still showing the the path with the HA NameNode event after a restart of Spark2. The issue become more intersting :) On Fri, Aug 17, 2018 at 2:01 AM Manu Zhang <owenzhang1...@gmail.com> wrote: > Hi Fawze, > > Sorry but I'm not familiar with CM. Maybe you can look into the logs (or > turn on DEBUG log). > > On Thu, Aug 16, 2018 at 3:05 PM Fawze Abujaber <fawz...@gmail.com> wrote: > >> Hi Manu, >> >> I'm using cloudera manager with single user mode and every process is >> running with cloudera-scm user, the cloudera-scm is a super user and this >> is why i was confused how it worked in spark 1.6 and not in spark 2.3 >> >> >> On Thu, Aug 16, 2018 at 5:34 AM Manu Zhang <owenzhang1...@gmail.com> >> wrote: >> >>> If you are able to log onto the node where UI has been launched, then >>> try `ps -aux | grep HistoryServer` and the first column of output should be >>> the user. >>> >>> On Wed, Aug 15, 2018 at 10:26 PM Fawze Abujaber <fawz...@gmail.com> >>> wrote: >>> >>>> Thanks Manu, Do you know how i can see which user the UI is running, >>>> because i'm using cloudera manager and i created a user for cloudera >>>> manager and called it spark but this didn't solve me issue and here i'm >>>> trying to find out the user for the spark hisotry UI. >>>> >>>> On Wed, Aug 15, 2018 at 5:11 PM Manu Zhang <owenzhang1...@gmail.com> >>>> wrote: >>>> >>>>> Hi Fawze, >>>>> >>>>> A) The file permission is currently hard coded to 770 ( >>>>> https://github.com/apache/spark/blob/branch-2.3/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L287 >>>>> ). >>>>> B) I think add all users (including UI) to the group like Spark will >>>>> do. >>>>> >>>>> >>>>> On Wed, Aug 15, 2018 at 6:38 PM Fawze Abujaber <fawz...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Manu, >>>>>> >>>>>> Thanks for your response. >>>>>> >>>>>> Yes, i see but still interesting to know how i can see these >>>>>> applications from the spark history UI. >>>>>> >>>>>> How i can know with which user i'm logged in when i'm navigating the >>>>>> spark history UI. >>>>>> >>>>>> The Spark process is running with cloudera-scm and the events written >>>>>> in the spark2history folder at the HDFS written with the user name who is >>>>>> running the application and group spark (770 permissions). >>>>>> >>>>>> I'm interesting to see if i can force these logs to be written with >>>>>> 774 or 775 permission or finding another solutions that enable Rnd or >>>>>> anyone to be able to investigate his application logs using the UI. >>>>>> >>>>>> for example : can i use such spark conf : >>>>>> spark.eventLog.permissions=755 >>>>>> >>>>>> The 2 options i see here: >>>>>> >>>>>> A) find a way to enforce these logs to be written with other >>>>>> permissions. >>>>>> >>>>>> B) Find the user that the UI running with as creating LDAP groups and >>>>>> user that can handle this. >>>>>> >>>>>> for example creating group called Spark and create the user that the >>>>>> UI running with and add this user to the spark group. >>>>>> not sure if this option will work as i don't know if these steps >>>>>> authenticate against the LDAP. >>>>>> >>>>> >>>> >>>> -- >>>> Take Care >>>> Fawze Abujaber >>>> >>> >> >> -- >> Take Care >> Fawze Abujaber >> > -- Take Care Fawze Abujaber
