Re: SparkR execution hang on when handle a RDD which is converted from DataFrame

40GB 2016-10-14 14:20 GMT+08:00 Felix Cheung : > How big is the metrics_moveing_detection_cube table? > > > > > > On Thu, Oct 13, 2016 at 8:51 PM -0700, "Lantao Jin" > wrote: > > sqlContext <- sparkRHive.init(sc) > sqlString<- > "SELECT > key_id,

SparkR execution hang on when handle a RDD which is converted from DataFrame

sqlContext <- sparkRHive.init(sc) sqlString<- "SELECT key_id, rtl_week_beg_dt rawdate, gmv_plan_rate_amt value FROM metrics_moveing_detection_cube " df <- sql(sqlString) rdd<-SparkR:::toRDD(df) #hang on case one: take from rdd #take(rdd,3) #hang on case two: convert back to dataframe