Hmm, interesting. I created https://issues.apache.org/jira/browse/SPARK-1499to track the issue of Workers continuously spewing bad executors, but the real issue seems to be a combination of that and some other bug in Shark or Spark which fails to handle the situation properly.
Please let us know if you can reproduce it (especially if deterministic!), or if you can provide any more details about exceptions thrown. A preliminary search didn't bring up much about the error code 101... On Mon, Apr 14, 2014 at 10:03 PM, Praveen R <prav...@sigmoidanalytics.com>wrote: > Unfortunately queries kept failing with SparkTask101 errors and had them > working after removing the troublesome node. > > FAILED: Execution Error, return code -101 from shark.execution.SparkTask > > I wish it would have been easy to re-produce it. I shall give a try to > hard remove write permissions on one node to see if the same error happens. > > > > On Tue, Apr 15, 2014 at 9:17 AM, Aaron Davidson <ilike...@gmail.com>wrote: > >> Cool! It's pretty rare to actually get logs from a wild hardware failure. >> The problem is as you said, that the executor keeps failing, but the worker >> doesn't get the hint, so it keeps creating new, bad executors. >> >> However, this issue should not have caused your cluster to fail to start >> up. In the linked logs, for instance, the shark shell started up just fine >> (though the "shark>" was lost in some of the log messages). Queries should >> have been able to execute just fine. Was this not the case? >> >> >> On Mon, Apr 14, 2014 at 7:38 AM, Praveen R >> <prav...@sigmoidanalytics.com>wrote: >> >>> Configuration comes from spark-ec2 setup script, sets spark.local.dir to >>> use /mnt/spark, /mnt2/spark. >>> Setup actually worked for quite sometime and then on one of the node >>> there were some disk errors as >>> >>> mv: cannot remove >>> `/mnt2/spark/spark-local-20140409182103-c775/09/shuffle_1_248_0': Read-only >>> file system >>> mv: cannot remove >>> `/mnt2/spark/spark-local-20140409182103-c775/24/shuffle_1_260_0': Read-only >>> file system >>> mv: cannot remove >>> `/mnt2/spark/spark-local-20140409182103-c775/24/shuffle_2_658_0': Read-only >>> file system >>> >>> I understand the issue is hardware level but thought it would be great >>> if spark could handle it and avoid cluster going down. >>> >>> >>> On Mon, Apr 14, 2014 at 7:58 PM, giive chen <thegi...@gmail.com> wrote: >>> >>>> Hi Praveen >>>> >>>> What is your config about "* spark.local.dir" ? * >>>> Is all your worker has this dir and all worker has right permission on >>>> this dir? >>>> >>>> I think this is the reason of your error >>>> >>>> Wisely Chen >>>> >>>> >>>> On Mon, Apr 14, 2014 at 9:29 PM, Praveen R < >>>> prav...@sigmoidanalytics.com> wrote: >>>> >>>>> Had below error while running shark queries on 30 node cluster and was >>>>> not able to start shark server or run any jobs. >>>>> >>>>> *14/04/11 19:06:52 ERROR scheduler.TaskSchedulerImpl: Lost an executor >>>>> 4 (already removed): Failed to create local directory (bad >>>>> spark.local.dir?)* >>>>> *Full log: *https://gist.github.com/praveenr019/10647049 >>>>> >>>>> After spending quite some time, found it was due to disk read errors >>>>> on one node and had the cluster working after removing the node. >>>>> >>>>> Wanted to know if there is any configuration (like akkaTimeout) which >>>>> can handle this or does mesos help ? >>>>> >>>>> Shouldn't the worker be marked dead in such scenario, instead of >>>>> making the cluster non-usable so the debugging can be done at leisure. >>>>> >>>>> Thanks, >>>>> Praveen R >>>>> >>>>> >>>>> >>>> >>> >> >
