[GitHub] [hudi] xushiyan commented on a diff in pull request #9221: [HUDI-6550] Add Hadoop conf to HiveConf for HiveSyncConfig
xushiyan commented on code in PR #9221:
URL: https://github.com/apache/hudi/pull/9221#discussion_r1299711286
##
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfig.java:
##
@@ -98,8 +98,9 @@ public HiveSyncConfig(Properties props) {
public HiveSyncConfig(Properties props, Configuration hadoopConf) {
super(props, hadoopConf);
-HiveConf hiveConf = hadoopConf instanceof HiveConf
-? (HiveConf) hadoopConf : new HiveConf(hadoopConf, HiveConf.class);
+HiveConf hiveConf = new HiveConf();
+// HiveConf needs to load Hadoop conf to allow instantiation via
AWSGlueClientFactory
+hiveConf.addResource(hadoopConf);
Review Comment:
@CTTY always creating a new HiveConf looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #9221: [HUDI-6550] Add Hadoop conf to HiveConf for HiveSyncConfig
xushiyan commented on code in PR #9221:
URL: https://github.com/apache/hudi/pull/9221#discussion_r1273870474
##
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfig.java:
##
@@ -98,8 +98,9 @@ public HiveSyncConfig(Properties props) {
public HiveSyncConfig(Properties props, Configuration hadoopConf) {
super(props, hadoopConf);
-HiveConf hiveConf = hadoopConf instanceof HiveConf
-? (HiveConf) hadoopConf : new HiveConf(hadoopConf, HiveConf.class);
+HiveConf hiveConf = new HiveConf();
+// HiveConf needs to load Hadoop conf to allow instantiation via
AWSGlueClientFactory
+hiveConf.addResource(hadoopConf);
Review Comment:
i think the ideal approach is to make the passed-in `hiveConf` load hadoop
conf properly to use `AWSGlueClientFactory` at the very beginning (when
creating hive sync config) so that nothing needs to load at this point. cc
@yihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #9221: [HUDI-6550] Add Hadoop conf to HiveConf for HiveSyncConfig
xushiyan commented on code in PR #9221:
URL: https://github.com/apache/hudi/pull/9221#discussion_r1273223195
##
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfig.java:
##
@@ -98,8 +98,9 @@ public HiveSyncConfig(Properties props) {
public HiveSyncConfig(Properties props, Configuration hadoopConf) {
super(props, hadoopConf);
-HiveConf hiveConf = hadoopConf instanceof HiveConf
-? (HiveConf) hadoopConf : new HiveConf(hadoopConf, HiveConf.class);
+HiveConf hiveConf = new HiveConf();
+// HiveConf needs to load Hadoop conf to allow instantiation via
AWSGlueClientFactory
+hiveConf.addResource(hadoopConf);
Review Comment:
not so sure if this is equivalent to holding the original `hadoopConf` as
this changes the order of addResources() during constructing. We should be good
only if we can verify the equivalence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
