You mean YARN cluster, right? Also, my jobs runs thru all their stages just fine. But the entire code crashes when I do a "saveAsTextFile".
On 8 August 2014 13:24, Sandy Ryza <[email protected]> wrote: > Hi Avishek, > > As of Spark 1.0, PySpark does in fact run on YARN. > > -Sandy > > > On Fri, Aug 8, 2014 at 12:47 PM, Avishek Saha <[email protected]> > wrote: >> >> So I think I have a better idea of the problem now. >> >> The environment is YARN client and IIRC PySpark doesn't run on YARN >> cluster. >> >> So my client is heavily loaded which causes iy loose a lot of e executors >> which might be part of the problem. >> >> Btw any plans in supporting PySpark in YARN clusters mode? >> >> On Aug 7, 2014 3:04 PM, "Davies Liu" <[email protected]> wrote: >>> >>> What is the environment ? YARN or Mesos or Standalone? >>> >>> It will be more helpful if you could show more loggings. >>> >>> On Wed, Aug 6, 2014 at 7:25 PM, Avishek Saha <[email protected]> >>> wrote: >>> > Hi, >>> > >>> > I get a lot of executor lost error for "saveAsTextFile" with PySpark >>> > and Hadoop 2.4. >>> > >>> > For small datasets this error occurs but since the dataset is small it >>> > gets eventually written to the file. >>> > For large datasets, it takes forever to write the final output. >>> > >>> > Any help is appreciated. >>> > Avishek >>> > >>> > --------------------------------------------------------------------- >>> > To unsubscribe, e-mail: [email protected] >>> > For additional commands, e-mail: [email protected] >>> > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
