Re: Review Request 28064: HIVE-8844 Choose a persisent policy for RDD caching [Spark Branch]

2014-11-15 Thread Xuefu Zhang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28064/#review61642 ---

Re: Review Request 28064: HIVE-8844 Choose a persisent policy for RDD caching [Spark Branch]

2014-11-15 Thread Jimmy Xiang
On Nov. 15, 2014, 10:28 a.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java, line 154 https://reviews.apache.org/r/28064/diff/2/?file=764710#file764710line154 I don't think we need to make this configurable. I think changing

Review Request 28064: HIVE-8844 Choose a persisent policy for RDD caching [Spark Branch]

2014-11-14 Thread Jimmy Xiang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28064/ --- Review request for hive and Xuefu Zhang. Bugs: HIVE-8844

Re: Review Request 28064: HIVE-8844 Choose a persisent policy for RDD caching [Spark Branch]

2014-11-14 Thread Szehon Ho
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28064/#review61605 --- Looks good, some nits:

Re: Review Request 28064: HIVE-8844 Choose a persisent policy for RDD caching [Spark Branch]

2014-11-14 Thread Szehon Ho
On Nov. 15, 2014, 2:08 a.m., Szehon Ho wrote: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java, line 155 https://reviews.apache.org/r/28064/diff/1/?file=764643#file764643line155 Instead of hardcoding, can we just use the StorageHandler.MEMORY_AND_DISK

Re: Review Request 28064: HIVE-8844 Choose a persisent policy for RDD caching [Spark Branch]

2014-11-14 Thread Jimmy Xiang
On Nov. 15, 2014, 2:08 a.m., Szehon Ho wrote: Looks good, some nits: Thanks a lot for the review. On Nov. 15, 2014, 2:08 a.m., Szehon Ho wrote: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java, line 156

Re: Review Request 28064: HIVE-8844 Choose a persisent policy for RDD caching [Spark Branch]

2014-11-14 Thread Szehon Ho
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28064/#review61616 --- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ShuffleTran.java

Re: Review Request 28064: HIVE-8844 Choose a persisent policy for RDD caching [Spark Branch]

2014-11-14 Thread Jimmy Xiang
On Nov. 15, 2014, 2:34 a.m., Szehon Ho wrote: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ShuffleTran.java, line 39 https://reviews.apache.org/r/28064/diff/1/?file=764642#file764642line39 OK, does spark handle that if we pass NONE in by doing no-op? If that's the case,

Re: Review Request 28064: HIVE-8844 Choose a persisent policy for RDD caching [Spark Branch]

2014-11-14 Thread Chao Sun
On Nov. 15, 2014, 2:34 a.m., Szehon Ho wrote: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ShuffleTran.java, line 39 https://reviews.apache.org/r/28064/diff/1/?file=764642#file764642line39 OK, does spark handle that if we pass NONE in by doing no-op? If that's the case,

Re: Review Request 28064: HIVE-8844 Choose a persisent policy for RDD caching [Spark Branch]

2014-11-14 Thread Jimmy Xiang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28064/ --- (Updated Nov. 15, 2014, 3:14 a.m.) Review request for hive and Xuefu Zhang.

Re: Review Request 28064: HIVE-8844 Choose a persisent policy for RDD caching [Spark Branch]

2014-11-14 Thread Jimmy Xiang
On Nov. 15, 2014, 2:34 a.m., Szehon Ho wrote: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ShuffleTran.java, line 39 https://reviews.apache.org/r/28064/diff/1/?file=764642#file764642line39 OK, does spark handle that if we pass NONE in by doing no-op? If that's the case,