abstractdog commented on a change in pull request #1941:
URL: https://github.com/apache/hive/pull/1941#discussion_r570030445
##########
File path:
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -338,6 +338,24 @@ public void setupConfiguration(Configuration conf) {
conf.setInt(MRJobConfig.REDUCE_MEMORY_MB, 512);
conf.setInt(MRJobConfig.MR_AM_VMEM_MB, 128);
}
+
+ // we want this shim to override these values only if the default value is
set, otherwise there
+ // is a good chance that the user of the minicluster set another value
intentionally
+ protected void overrideIntIfDefaultIsSet(Configuration conf, String key,
int defaultVal,
+ int newVal) {
+ if (conf.getInt(key, defaultVal) == defaultVal) {
+ LOG.info("Hadoop23Shims overrides '{}' from {} to {}", key,
defaultVal, newVal);
+ conf.setInt(key, newVal);
+ }
Review comment:
right, I'm adding logs for untouched configs too
##########
File path:
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -338,6 +338,24 @@ public void setupConfiguration(Configuration conf) {
conf.setInt(MRJobConfig.REDUCE_MEMORY_MB, 512);
conf.setInt(MRJobConfig.MR_AM_VMEM_MB, 128);
}
+
+ // we want this shim to override these values only if the default value is
set, otherwise there
+ // is a good chance that the user of the minicluster set another value
intentionally
+ protected void overrideIntIfDefaultIsSet(Configuration conf, String key,
int defaultVal,
+ int newVal) {
+ if (conf.getInt(key, defaultVal) == defaultVal) {
+ LOG.info("Hadoop23Shims overrides '{}' from {} to {}", key,
defaultVal, newVal);
+ conf.setInt(key, newVal);
+ }
Review comment:
added in second commit:
https://github.com/apache/hive/pull/1941/commits/697dc4a9e37c12b4d7d34589cc1014d6e08a6700
edited PR description with new log contents and some instructions
##########
File path:
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -446,16 +478,23 @@ public void shutdown() throws IOException {
@Override
public void setupConfiguration(Configuration conf) {
Configuration config = mr.getConfig();
- for (Map.Entry<String, String> pair: config) {
+ for (Map.Entry<String, String> pair : config) {
Review comment:
not, but it's autoformatted by eclipse, actually I liked the change, I
always put spaces before and after colon
##########
File path:
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -446,16 +478,23 @@ public void shutdown() throws IOException {
@Override
public void setupConfiguration(Configuration conf) {
Configuration config = mr.getConfig();
- for (Map.Entry<String, String> pair: config) {
+ for (Map.Entry<String, String> pair : config) {
Review comment:
not, but it's autoformatted by eclipse, actually I liked the change, I
always put spaces before and after colon in iterations
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
