Hello Community, I'm using Spark 2.3 and Spark 1.6.0 in my cluster with Cloudera distribution 5.13.0.
Both are configured to run on Yarn, but i'm unable to see completed application in Spark2 history server, while in Spark 1.6.0 i did. 1) I checked the HDFS permissions for both folders and both have the same permissions. drwxrwxrwt - cloudera-scm spark 0 2018-08-08 00:46 /user/spark/applicationHistory drwxrwxrwt - cloudera-scm spark 0 2018-08-08 00:46 /user/spark/spark2ApplicationHistory The applications file itself running with permissions 770 in both. -rwxrwx--- 3 fawzea spark 4743751 2018-08-07 23:32 /user/spark/spark2ApplicationHistory/application_1527404701551_672816_1 -rwxrwx--- 3 fawzea spark 134315 2018-08-08 00:41 /user/spark/applicationHistory/application_1527404701551_673359_1 2) No error in the Spark2 history server log. 3) Compared the configurations between Spark 1.6 and Spark 2.3 like system user, enable log, etc ... all looks the same. 4) Once i changed the permissions for the above Spark2 applications to 777, i was able to see the application in the spark2 history server UI. Tried to figure out if the 2 Sparks UIs running with different users but was unable to find it. Anyone who ran into this issue and solved it? Thanks in advance. -- Take Care Fawze Abujaber
