----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30055/#review68678 -----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java <https://reviews.apache.org/r/30055/#comment113024> I think we'll be doing this lookup potentially hundreds of times. Let's make HIVE_SPARK_RSC_CONFIGS an ImmutableSet to improve lookup speed? - Brock Noland On Jan. 19, 2015, 11:16 p.m., Szehon Ho wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/30055/ > ----------------------------------------------------------- > > (Updated Jan. 19, 2015, 11:16 p.m.) > > > Review request for hive and chengxiang li. > > > Bugs: HIVE-9337 > https://issues.apache.org/jira/browse/HIVE-9337 > > > Repository: hive-git > > > Description > ------- > > This change allows the Remote Spark Driver's properties to be set dynamically > via Hive configuration (ie, set commands). > > Went through the Remote Spark Driver's properties and added them to HiveConf, > fixing the descriptions so that they're more clear in a global context with > other Hive properties. Also fixed a bug in description that stated default > value of max message size is 10MB, should read 50MB. One open question is > that I did not move 'hive.spark.log.dir' as I could not find where it was > read, and did not know if its still being used somewhere? > > The passing of these properties between client (Hive) and RemoteSparkDriver > is done via the properties file. One note is that these properties have to > be appended with 'spark', as SparkConf only accepts those. I tried a long > time to pass them via 'conf' but found that it won't work (see > SparkSubmitArguments.scala). It may be possible to pass them each as another > argument (like --hive.spark.XXX=YYY), but I think its more scalable to do it > via properties file. > > On the Remote Spark Driver side, I kept the defensive logic to provide a > default value in case the conf object doesn't contain the property. This may > occur if a prop is unset. For this, I had to instantiate a HiveConf on that > process to get the default value, as some of the timeout props need a > hiveConf instance to do calculation on. > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 068c962 > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java > 334c191 > > spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java > 865e03e > spark-client/src/main/java/org/apache/hive/spark/client/rpc/Rpc.java > ac71ae9 > > spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcConfiguration.java > 5a826ba > spark-client/src/test/java/org/apache/hive/spark/client/rpc/TestRpc.java > a2dd3e6 > > Diff: https://reviews.apache.org/r/30055/diff/ > > > Testing > ------- > > > Thanks, > > Szehon Ho > >
