It could be cause you are out of memory on the worker nodes & blocks are not getting registered.. A older issue with 0.6.0 was with dead nodes causing loss of task & then resubmission of data in an infinite loop... It was fixed in 0.7.0 though. Are you seeing a crash log in this log.. or in the worker log @ 192.168.222.164 or any of the machines where the crash log is displayed.
Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi <https://twitter.com/mayur_rustagi> On Wed, Jul 2, 2014 at 7:51 AM, Yana Kadiyska <yana.kadiy...@gmail.com> wrote: > A lot of things can get funny when you run distributed as opposed to > local -- e.g. some jar not making it over. Do you see anything of > interest in the log on the executor machines -- I'm guessing > 192.168.222.152/192.168.222.164. From here > > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala > seems like the warning message is logged after the task fails -- but I > wonder if you might see something more useful as to why it failed to > begin with. As an example we've had cases in Hdfs where a small > example would work, but on a larger example we'd hit a bad file. But > the executor log is usually pretty explicit as to what happened... > > On Tue, Jul 1, 2014 at 8:57 PM, Mohammed Guller <moham...@glassbeam.com> > wrote: > > I am running Spark 1.0 on a 4-node standalone spark cluster (1 master + 3 > > worker). Our app is fetching data from Cassandra and doing a basic > filter, > > map, and countByKey on that data. I have run into a strange problem. > Even if > > the number of rows in Cassandra is just 1M, the Spark job goes seems to > go > > into an infinite loop and runs for hours. With a small amount of data > (less > > than 100 rows), the job does finish, but takes almost 30-40 seconds and > we > > frequently see the messages shown below. If we run the same application > on a > > single node Spark (--master local[4]), then we don’t see these warnings > and > > the task finishes in less than 6-7 seconds. Any idea what could be the > cause > > for these problems when we run our application on a standalone 4-node > spark > > cluster? > > > > > > > > 14/06/30 19:30:16 WARN TaskSetManager: Lost TID 25036 (task 6.0:90) > > > > 14/06/30 19:30:16 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:18 WARN TaskSetManager: Lost TID 25310 (task 6.1:0) > > > > 14/06/30 19:30:18 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:19 WARN TaskSetManager: Lost TID 25582 (task 6.2:0) > > > > 14/06/30 19:30:19 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:21 WARN TaskSetManager: Lost TID 25882 (task 6.3:34) > > > > 14/06/30 19:30:21 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(0, 192.168.222.142, 39342, 0) > > > > 14/06/30 19:30:22 WARN TaskSetManager: Lost TID 26152 (task 6.4:0) > > > > 14/06/30 19:30:22 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(0, 192.168.222.142, 39342, 0) > > > > 14/06/30 19:30:23 WARN TaskSetManager: Lost TID 26427 (task 6.5:4) > > > > 14/06/30 19:30:23 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:25 WARN TaskSetManager: Lost TID 26690 (task 6.6:0) > > > > 14/06/30 19:30:25 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:26 WARN TaskSetManager: Lost TID 26959 (task 6.7:0) > > > > 14/06/30 19:30:26 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:28 WARN TaskSetManager: Lost TID 27449 (task 6.8:218) > > > > 14/06/30 19:30:28 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:30 WARN TaskSetManager: Lost TID 27718 (task 6.9:0) > > > > 14/06/30 19:30:30 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:30 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:31 WARN TaskSetManager: Lost TID 27991 (task 6.10:1) > > > > 14/06/30 19:30:31 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:33 WARN TaskSetManager: Lost TID 28265 (task 6.11:0) > > > > 14/06/30 19:30:33 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:34 WARN TaskSetManager: Lost TID 28550 (task 6.12:0) > > > > 14/06/30 19:30:34 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:36 WARN TaskSetManager: Lost TID 28822 (task 6.13:0) > > > > 14/06/30 19:30:36 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:37 WARN TaskSetManager: Lost TID 29093 (task 6.14:0) > > > > 14/06/30 19:30:37 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:39 WARN TaskSetManager: Lost TID 29366 (task 6.15:0) > > > > 14/06/30 19:30:39 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:40 WARN TaskSetManager: Lost TID 29648 (task 6.16:9) > > > > 14/06/30 19:30:40 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:42 WARN TaskSetManager: Lost TID 29924 (task 6.17:0) > > > > 14/06/30 19:30:42 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:43 WARN TaskSetManager: Lost TID 30193 (task 6.18:0) > > > > 14/06/30 19:30:43 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(2, 192.168.222.164, 57185, 0) > > > > 14/06/30 19:30:45 WARN TaskSetManager: Lost TID 30559 (task 6.19:98) > > > > 14/06/30 19:30:45 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(1, 192.168.222.152, 45896, 0) > > > > 14/06/30 19:30:46 WARN TaskSetManager: Lost TID 30826 (task 6.20:0) > > > > 14/06/30 19:30:46 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(1, 192.168.222.152, 45896, 0) > > > > 14/06/30 19:30:48 WARN TaskSetManager: Lost TID 31098 (task 6.21:0) > > > > 14/06/30 19:30:48 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(1, 192.168.222.152, 45896, 0) > > > > 14/06/30 19:30:50 WARN TaskSetManager: Lost TID 31370 (task 6.22:0) > > > > 14/06/30 19:30:50 WARN TaskSetManager: Loss was due to fetch failure from > > BlockManagerId(1, 192.168.222.152, 45896, 0) > > > > > > > > Thanks. > > > > > > > > Mohammed > > > > >