I'm using Spark 1.0.0
On Mon, Aug 11, 2014 at 4:14 PM, Grzegorz Białek < grzegorz.bia...@codilime.com> wrote: > Hi, > > I ran Spark application in local mode with command: > $SPARK_HOME/bin/spark-submit --driver-memory 1g <class> <jar> > with set master=local. > > After around 10 minutes of computing it started to slow down > significantly that next stage took around 50 minutes and next after 5 > hours in 80% > done and CPU usage decreased from 160% to almost 0% (according to system > monitor) where 200% is max for one core. (This stage that took more than 5h > was > saveAsTextFile on 50MB RDD). > During that computing my system was significantly slower and less > responsible. > > Moreover when I wanted to interrupt this application I tried ctrl-c and > current stage was interrupted but program didn't exit. When I shut down my > computer system > didn't want to shutdown because killing Spark app failed. (It shut down > after ctrl-alt-del). After reboot system didn't want to start and finally I > had to reinstall it. > > The same thing was when I killed next app using kill -9, system was > corrupted and I had to reinstall it. > > > When I ran this app on a bit smaller data everything was ok. I have Linux > Mint 17, 8GB RAM, Intel Core i7-3630QM (4x2,4GHz). > > Do you have any idea why Spark slow down or how to properly kill spark app > run through spark-submit? > > Thanks, > Grzegorz > -- Grzegorz Białek Software Engineer ---------------------------------- M: +48 664 377 399 E: grzegorz.bialek@ <bart...@codilime.com>codilime.com ---------------------------------- CodiLime Sp. z o.o. - Ltd. company with its registered office in Poland, 01-167 Warsaw, ul. Zawiszy 14/97. Registered by The District Court for the Capital City of Warsaw, XII Commercial Department of the National Court Register. Entered into National Court Register under No. KRS 0000388871. Tax identification number (NIP) 5272657478. Statistical number (REGON) 142974628. ----------------------------------------- The information in this email is confidential and may be legally privileged, it may contain information that is confidential in CodiLime Sp. z o.o. It is intended solely for the addressee. Any access to this email by third parties is unauthorized. If you are not the intended recipient of this message, any disclosure, copying, distribution or any action undertaken or neglected in reliance thereon is prohibited and may result in your liability for damages.
