Hi Fawze, In Spark 2.3, HistoryServer will check for file permissions when reading event logs written by your applications. (Please check https://issues.apache.org/jira/browse/SPARK-20172). With file permissions of 770, HistoryServer is not permitted to read the event log. That's why you were able to see applications once changing file permissions to 777.
Regards, Manu Zhang On Mon, Aug 13, 2018 at 4:53 PM Fawze Abujaber <fawz...@gmail.com> wrote: > Hi Guys, > > Any help here? > > On Wed, Aug 8, 2018 at 7:56 AM Fawze Abujaber <fawz...@gmail.com> wrote: > >> Hello Community, >> >> I'm using Spark 2.3 and Spark 1.6.0 in my cluster with Cloudera >> distribution 5.13.0. >> >> Both are configured to run on Yarn, but i'm unable to see completed >> application in Spark2 history server, while in Spark 1.6.0 i did. >> >> 1) I checked the HDFS permissions for both folders and both have the same >> permissions. >> >> drwxrwxrwt - cloudera-scm spark 0 2018-08-08 00:46 >> /user/spark/applicationHistory >> drwxrwxrwt - cloudera-scm spark 0 2018-08-08 00:46 >> /user/spark/spark2ApplicationHistory >> >> The applications file itself running with permissions 770 in both. >> >> -rwxrwx--- 3 fawzea spark 4743751 2018-08-07 23:32 >> /user/spark/spark2ApplicationHistory/application_1527404701551_672816_1 >> -rwxrwx--- 3 fawzea spark 134315 2018-08-08 00:41 >> /user/spark/applicationHistory/application_1527404701551_673359_1 >> >> 2) No error in the Spark2 history server log. >> >> 3) Compared the configurations between Spark 1.6 and Spark 2.3 like >> system user, enable log, etc ... all looks the same. >> >> 4) Once i changed the permissions for the above Spark2 applications to >> 777, i was able to see the application in the spark2 history server UI. >> >> Tried to figure out if the 2 Sparks UIs running with different users but >> was unable to find it. >> >> Anyone who ran into this issue and solved it? >> >> Thanks in advance. >> >> >> -- >> Take Care >> Fawze Abujaber >> > > > -- > Take Care > Fawze Abujaber >
