To get these metrics out, you need to open the driver ui running on port 4040. And in there you will see Stages information and for each stage you can see how much time it is spending on GC etc. In your case, the parallelism seems 4, the more # of parallelism the more # of tasks you will see.
Thanks Best Regards On Thu, Mar 19, 2015 at 1:15 PM, Su She <suhsheka...@gmail.com> wrote: > Hi Akhil, > > 1) How could I see how much time it is spending on stage 1? Or what if, > like above, it doesn't get past stage 1? > > 2) How could I check if its a GC time? and where would I increase the > parallelism for the model? I have a Spark Master and 2 Workers running on > CDH 5.3...what would the default spark-shell level of parallelism be...I > thought it would be 3? > > Thank you for the help! > > -Su > > > On Thu, Mar 19, 2015 at 12:32 AM, Akhil Das <ak...@sigmoidanalytics.com> > wrote: > >> Can you see where exactly it is spending time? Like you said it goes to >> Stage 2, then you will be able to see how much time it spend on Stage 1. >> See if its a GC time, then try increasing the level of parallelism or >> repartition it like sc.getDefaultParallelism*3. >> >> Thanks >> Best Regards >> >> On Thu, Mar 19, 2015 at 12:15 PM, Su She <suhsheka...@gmail.com> wrote: >> >>> Hello Everyone, >>> >>> I am trying to run this MLlib example from Learning Spark: >>> >>> https://github.com/databricks/learning-spark/blob/master/src/main/scala/com/oreilly/learningsparkexamples/scala/MLlib.scala#L48 >>> >>> Things I'm doing differently: >>> >>> 1) Using spark shell instead of an application >>> >>> 2) instead of their spam.txt and normal.txt I have text files with 3700 >>> and 2700 words...nothing huge at all and just plain text >>> >>> 3) I've used numFeatures = 100, 1000 and 10,000 >>> >>> *Error: *I keep getting stuck when I try to run the model: >>> >>> val model = new LogisticRegressionWithSGD().run(trainingData) >>> >>> It will freeze on something like this: >>> >>> [Stage 1:==============> (1 + >>> 0) / 4] >>> >>> Sometimes its Stage 1, 2 or 3. >>> >>> I am not sure what I am doing wrong...any help is much appreciated, >>> thank you! >>> >>> -Su >>> >>> >>> >> >
