Hi Ted, Thanks for the information. The post seems little different with my requirement: suppose we defined different functions to do different streaming work (e.g. 50 functions), i want to test these 50 functions in the spark shell, and the shell will always throw OOM at the middle of test (yes, it could be solved by increasing the jvm memory size, but if we have more functions, the issue still will happen). The main issue is that the shell keeps track all the information (class, objects...) from started, so the java memory will increase time to time when define/invoke the functions.
Thanks! - Terry Ted Yu <yuzhih...@gmail.com>于2015年7月17日周五 下午12:02写道: > See this recent thread: > > > http://search-hadoop.com/m/q3RTtFW7iMDkrj61/Spark+shell+oom+&subj=java+lang+OutOfMemoryError+PermGen+space > > > > On Jul 16, 2015, at 8:51 PM, Terry Hole <hujie.ea...@gmail.com> wrote: > > Hi, > > Background: The spark shell will get out of memory error after dealing > lots of spark work. > > Is there any method which can reset the spark shell to the startup status? > I tried "*:reset*", but it seems not working: i can not create spark > context anymore (some compile error as below) after the "*:reset*". (I > have to restart the shell after OOM to workaround) > > == Expanded type of tree == > TypeRef(TypeSymbol(class $read extends Serializable)) > uncaught exception during compilation: java.lang.AssertionError > java.lang.AssertionError: assertion failed: Tried to find '$line16' in > 'C:\Users\jhu\AppData\Local\Temp\spark-2ad09490-c0c6-41e2-addb-63087ce0ae63' > but it is not a directory > That entry seems to have slain the compiler. Shall I replayyour session? > I can re-run each line except the last one.[y/n] > Abandoning crashed session. > > Thanks! > -Terry > >