For a pyspark job with 54 executors all of the task outputs have a single line in both the stderr and stdout similar to:
Error: invalid log directory /shared/sparkmaven/work/app-20161119222540-0000/0/ Note: the directory /shared/sparkmaven/work exists and is owned by the same user running the job. There are plenty of other app-*** subdirectories that do have contents in the stdout/stderr files. $ls -lrta /shared/sparkmaven/work total 0 drwxr-xr-x 59 steve staff 2006 Nov 23 05:01 .. drwxr-xr-x 41 steve staff 1394 Nov 23 18:22 app-20161123050122-0002 drwxr-xr-x 6 steve staff 204 Nov 23 18:22 app-20161123182031-0005 drwxr-xr-x 6 steve staff 204 Nov 23 18:44 app-20161123184349-0006 drwxr-xr-x 6 steve staff 204 Nov 23 18:46 app-20161123184613-0007 drwxr-xr-x 3 steve staff 102 Nov 23 19:20 app-20161123192048-0008 Here is a sample of the contents /shared/sparkmaven/work/app-20161123184613-0007/2: total 16 -rw-r--r-- 1 steve staff 0 Nov 23 18:46 stdout drwxr-xr-x 4 steve staff 136 Nov 23 18:46 . -rw-r--r-- 1 steve staff 4830 Nov 23 18:46 stderr drwxr-xr-x 6 steve staff 204 Nov 23 18:46 .. /shared/sparkmaven/work/app-20161123184613-0007/3: total 16 -rw-r--r-- 1 steve staff 0 Nov 23 18:46 stdout drwxr-xr-x 6 steve staff 204 Nov 23 18:46 .. drwxr-xr-x 4 steve staff 136 Nov 23 18:46 . -rw-r--r-- 1 steve staff 4830 Nov 23 18:46 stderr Note also: the *SparkPI* program does run succesfully - which validates the basic spark installation/functionality.
