Hi Moon, I am using Spark/Scala code as below
import java.util.Calendar import org.joda.time._ val option = z.input("option","2").toString.toInt val today = new DateTime() val minutes = -15 val minutesago = today.plusMinutes(minutes).toString.toString.substring(11,19) val date = java.time.LocalDate.now.toString val hour = java.time.LocalTime.now.toString val location="hdfs://rhes564:9000/data/prices/"+date+"/prices.*" case class columns(SECURITY: String, TIMECREATED: String, PRICE: String) val rs = option match { case 1 => { println("option = 1") val df1 = spark.read.option("header", false).csv(location) val df2 = df1.map(p => columns(p(0).toString,p(1).toString, p(2).toString)) val rs = df2.filter('security > "").filter('price > "15" && substring('TIMECREATED,12,19)> minutesago).select(substring('TIMECREATED,12,19).as("TIMECREATED"), 'SECURITY, substring('PRICE,1,7).as("PRICE")).orderBy('TIMECREATED) rs } case 2 => { println("option = 2") val df2 = spark.table("test.marketData").select('TIMECREATED,'SECURITY,'PRICE, 'DateStamp) val rs = df2.filter('security > "").filter('DateStamp >= "${today}" && 'price > "15" && 'TIMECREATED > date && substring('TIMECREATED,12,19)> minutesago).select(substring('TIMECREATED,12,19).as("TIMECREATED"), 'SECURITY, substring('PRICE,1,7).as("PRICE")).orderBy('TIMECREATED) rs } case _ => { sys.error("No valid option provided") } } z.show(rs) Anything I can do to reduce the effect? The UI is kept open and I just run the code again and again if I need to Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 5 October 2016 at 15:33, moon soo Lee <m...@apache.org> wrote: > Hi, > > Thanks for sharing problem. > > If you're using Scala in SparkInterpreter, Scala REPL may have some > problem [1] when there is a lot of calls. > > If you're using only SparkSQL or other interpreter, but still experience > the same, then it is Zeppelin's own problem. > > Could you share little more information to reproduce the problem? > > Thanks, > moon > > [1] https://issues.scala-lang.org/browse/SI-4331 > > > On Mon, Oct 3, 2016 at 6:21 PM Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> Hi, >> >> I have been testing Zeppelin in conjunction with Spark sql and Spark >> functional programming. I notice that as time gets by and many calls are >> made with the same session open with different codes and plots, Zeppelin >> becomes very slow and eventually one gets a message "pipe broken" in the >> run. >> >> Then in order to make this work I recycle (stop/start) Zeppelin daemon >> and things start working. >> >> Is this behavior expected or is this due to excessive memory usage/GC >> issue. >> >> FYI I am using zeppelin-0.6.1 >> >> Thanks >> >> >> >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >