[ https://issues.apache.org/jira/browse/SPARK-25222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591386#comment-16591386 ]
Rob Vesse commented on SPARK-25222: ----------------------------------- There is also a similar issue with task failure: {noformat} 2018-08-24 09:11:57 WARN TaskSetManager:66 - Lost task 2.3 in stage 0.0 (TID 13, 10.244.3.199, executor 8): ExecutorLostFailure (executor 8 exited caused by one of the running tasks) Reason: The executor with id 8 exited with exit code 52. The API gave the following brief reason: null The API gave the following message: null The API gave the following container statuses: ContainerStatus(containerID=docker://353f78fd634d312ec8115032c32da56748fb5d8da2c5ae54b1d0a9f112fb4d1d, image=rvesse/spark:latest, imageID=docker-pullable://rvesse/spark@sha256:92abf0b718743d0f5a26068fc94ec42233db0493c55a8570dc8c851c62a4bc0a, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=executor, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://353f78fd634d312ec8115032c32da56748fb5d8da2c5ae54b1d0a9f112fb4d1d, exitCode=52, finishedAt=Time(time=2018-08-24T09:11:56Z, additionalProperties={}), message=null, reason=Error, signal=null, startedAt=Time(time=2018-08-24T09:11:48Z, additionalProperties={}), additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={}) {noformat} > Spark on Kubernetes Pod Watcher dumps raw container status > ---------------------------------------------------------- > > Key: SPARK-25222 > URL: https://issues.apache.org/jira/browse/SPARK-25222 > Project: Spark > Issue Type: Improvement > Components: Kubernetes > Affects Versions: 2.3.0, 2.3.1 > Reporter: Rob Vesse > Priority: Minor > > Spark on Kubernetes provides logging of the pod/container status as a monitor > of the job progress. However the logger just dumps the raw container status > object leading to fairly unreadable output like so: > {noformat} > 18/08/24 09:03:27 INFO LoggingPodStatusWatcherImpl: State changed, new state: > pod name: spark-groupby-1535101393784-driver > namespace: default > labels: spark-app-selector -> spark-47f7248122b9444b8d5fd3701028a1e8, > spark-role -> driver > pod uid: 88de6467-a77c-11e8-b9da-a4bf0128b75b > creation time: 2018-08-24T09:03:14Z > service account name: spark > volumes: spark-local-dir-1, spark-conf-volume, spark-token-kjxkv > node name: tab-cmp4 > start time: 2018-08-24T09:03:14Z > container images: rvesse/spark:latest > phase: Running > status: > [ContainerStatus(containerID=docker://23ae58571f59505e837dca40455d0347fb90e9b88e2a2b145a38e2919fceb447, > image=rvesse/spark:latest, > imageID=docker-pullable://rvesse/spark@sha256:92abf0b718743d0f5a26068fc94ec42233db0493c55a8570dc8c851c62a4bc0a, > lastState=ContainerState(running=null, terminated=null, waiting=null, > additionalProperties={}), name=spark-kubernetes-driver, ready=true, > restartCount=0, > state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-08-24T09:03:26Z, > additionalProperties={}), additionalProperties={}), terminated=null, > waiting=null, additionalProperties={}), additionalProperties={})] > {noformat} > The {{LoggingPodStatusWatcher}} actually already includes code to nicely > format this information but only invokes it at the end of the job: > {noformat} > 18/08/24 09:04:07 INFO LoggingPodStatusWatcherImpl: Container final statuses: > Container name: spark-kubernetes-driver > Container image: rvesse/spark:latest > Container state: Terminated > Exit code: 0 > {noformat} > It would be nice if we continually used the nice formatting throughout the > logging. > We already have patched this on our internal fork and will upstream a fix > shortly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org