> On Dec. 17, 2014, 3:06 a.m., Xuefu Zhang wrote: > >
Fixed and attached v3 to the jira. Could not update the diff on this RB because RB has problem to find the file in the repo suddenly. > On Dec. 17, 2014, 3:06 a.m., Xuefu Zhang wrote: > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java, line 65 > > <https://reviews.apache.org/r/29063/diff/2/?file=793364#file793364line65> > > > > To be safe, let's do this instead: > > result = hadoopRDD.mapToPair(new CopyFunction()); > > sparkPlan.addCachedRDDId(result.id()); > > result = result.persist(StorageLevel.MEMORY_AND_DISK()); > > > > Maybe rdd.persist() returns the same reference, but that's the > > implementation. This is just to be on the safe side. Sure. - Jimmy ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29063/#review65280 ----------------------------------------------------------- On Dec. 16, 2014, 11:29 p.m., Jimmy Xiang wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/29063/ > ----------------------------------------------------------- > > (Updated Dec. 16, 2014, 11:29 p.m.) > > > Review request for hive, chengxiang li and Xuefu Zhang. > > > Bugs: HIVE-8843 > https://issues.apache.org/jira/browse/HIVE-8843 > > > Repository: hive-git > > > Description > ------- > > Release cached RDD after a job is done, if any. > > > Diffs > ----- > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java > 3102ab2 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java f748ee0 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java > e1946d5 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ShuffleTran.java 94c6c98 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java 53dd4a0 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java > 438efab > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/LocalSparkJobStatus.java > b8cc6e6 > spark-client/src/main/java/org/apache/hive/spark/client/JobContext.java > 8565bd8 > spark-client/src/main/java/org/apache/hive/spark/client/JobContextImpl.java > 137539b > > spark-client/src/main/java/org/apache/hive/spark/client/MonitorCallback.java > 3693f8f > spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java > 8c07ee1 > > spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java > 7f5ecd0 > > Diff: https://reviews.apache.org/r/29063/diff/ > > > Testing > ------- > > > Thanks, > > Jimmy Xiang > >
