Hi, I'm runing a shark sql, but I don't know How spark build these stages and what work it does in each stage . I waited the job 1 hour , it seems that the task 504 has some problem.
there are some logs in the machine which runs task 504 : 14/01/09 20:22:55 INFO executor.Executor: Finished task ID 287 14/01/09 20:23:00 INFO executor.StandaloneExecutorBackend: Got assigned task 504 14/01/09 20:23:00 INFO executor.Executor: Running task ID 504 14/01/09 20:23:00 INFO executor.Executor: Its epoch is 2 14/01/09 20:23:00 INFO spark.MapOutputTracker: Updating epoch to 2 and clearing cache 14/01/09 20:23:00 INFO spark.MapOutputTracker: Don't have map outputs for shuffle 2, fetching them 14/01/09 20:23:00 INFO spark.MapOutputTracker: Doing the fetch; tracker actor = Actor[akka://spark@OCDC-DD-002:53976/user/MapOutputTracker] 14/01/09 20:23:00 INFO spark.MapOutputTracker: Got the output locations 14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, minRequest: 10066329 14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, minRequest: 10066329 14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, minRequest: 10066329 14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight: 50331648, minRequest: 10066329 14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 24 non-zero-bytes blocks out of 264 blocks 14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 1 remote gets in 74 ms 14/01/09 20:23:00 INFO network.SendingConnection: Initiating connection to [OCDC-DATA-013/172.17.254.33:54930] 14/01/09 20:23:00 INFO network.SendingConnection: Connected to [OCDC-DATA-013/172.17.254.33:54930], 1 messages pending 14/01/09 20:23:00 INFO network.ConnectionManager: Accepted connection from [OCDC-DATA-013/172.17.254.33] 14/01/09 20:23:18 INFO network.SendingConnection: Initiating connection to [OCDC-DATA-012/172.17.254.32:45718] 14/01/09 20:23:18 INFO network.SendingConnection: Connected to [OCDC-DATA-012/172.17.254.32:45718], 1 messages pending 14/01/09 20:23:18 INFO network.ConnectionManager: Accepted connection from [OCDC-DATA-012/172.17.254.32] 14/01/09 20:23:46 INFO network.SendingConnection: Initiating connection to [OCDC-DATA-011/172.17.254.31:51067] 14/01/09 20:23:46 INFO network.SendingConnection: Connected to [OCDC-DATA-011/172.17.254.31:51067], 1 messages pending 14/01/09 20:23:46 INFO network.ConnectionManager: Accepted connection from [OCDC-DATA-011/172.17.254.31] 14/01/09 20:56:48 INFO storage.BlockManager: BlockManager reregistering with master 14/01/09 20:56:48 INFO storage.BlockManagerMaster: Trying to register BlockManager 14/01/09 20:56:48 INFO storage.BlockManagerMaster: Registered BlockManager 14/01/09 20:56:48 INFO storage.BlockManager: Reporting 117 blocks to the master. 14/01/09 21:04:43 INFO storage.BlockManager: BlockManager reregistering with master 14/01/09 21:04:43 INFO storage.BlockManagerMaster: Trying to register BlockManager 14/01/09 21:04:43 INFO storage.BlockManagerMaster: Registered BlockManager 14/01/09 21:04:43 INFO storage.BlockManager: Reporting 117 blocks to the master. 14/01/09 21:06:43 INFO storage.BlockManager: BlockManager reregistering with master 14/01/09 21:06:43 INFO storage.BlockManagerMaster: Trying to register BlockManager 14/01/09 21:06:43 INFO storage.BlockManagerMaster: Registered BlockManager 14/01/09 21:06:43 INFO storage.BlockManager: Reporting 117 blocks to the master. 14/01/09 21:11:48 INFO storage.BlockManager: BlockManager reregistering with master 14/01/09 21:11:48 INFO storage.BlockManagerMaster: Trying to register BlockManager 14/01/09 21:11:48 INFO storage.BlockManagerMaster: Registered BlockManager 14/01/09 21:11:48 INFO storage.BlockManager: Reporting 117 blocks to the master. 14/01/09 21:12:48 INFO storage.BlockManager: BlockManager reregistering with master 14/01/09 21:12:48 INFO storage.BlockManagerMaster: Trying to register BlockManager 14/01/09 21:12:48 INFO storage.BlockManagerMaster: Registered BlockManager 14/01/09 21:12:48 INFO storage.BlockManager: Reporting 117 blocks to the master. 14/01/09 21:13:51 INFO storage.BlockManager: BlockManager reregistering with master 14/01/09 21:13:51 INFO storage.BlockManagerMaster: Trying to register BlockManager can anyone give me the hint thank you ! leosand...@gmail.com