[GitHub] [hudi] test-wangxiaoyu commented on a change in pull request #3771: [HUDI-2402] Add Kerberos configuration options to Hive Sync
test-wangxiaoyu commented on a change in pull request #3771:
URL: https://github.com/apache/hudi/pull/3771#discussion_r818224037
##
File path:
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java
##
@@ -77,6 +77,10 @@ public HiveSyncTool(HiveSyncConfig cfg, HiveConf
configuration, FileSystem fs) {
super(configuration.getAllProperties(), fs);
try {
+ if (cfg.useKerberos) {
+configuration.set("hive.metastore.sasl.enabled", "true");
+configuration.set("hive.metastore.kerberos.principal",
cfg.kerberosPrincipal);
Review comment:
I designed to configure the Fkink permission authentication to the
cluster level. Our Flink cluster has already configured Kerberos, so we only
need to configure the Meta Client Principal of Hive to connect.If Kerberos
permission authentication is not configured for the Flink cluster,
authentication must be performed at the Task level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] test-wangxiaoyu commented on a change in pull request #3771: [HUDI-2402] Add Kerberos configuration options to Hive Sync
test-wangxiaoyu commented on a change in pull request #3771:
URL: https://github.com/apache/hudi/pull/3771#discussion_r754856973
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala
##
@@ -84,6 +85,46 @@ object HoodieWriterUtils {
HIVE_SYNC_USE_KERBEROS.key -> HIVE_SYNC_USE_KERBEROS.defaultValue,
HIVE_SYNC_KERBEROS_PRINCIPAL.key ->
HIVE_SYNC_KERBEROS_PRINCIPAL.defaultValue
) ++ DataSourceOptionsHelper.translateConfigurations(parameters)
+
+val globalProps = DFSPropertiesConfiguration.getGlobalProps.asScala
Review comment:
This is the information that I handled on the github interface, and the
information that was written by the code conflict, github, I just resubmitted
the code of the master branch and resubmitted it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] test-wangxiaoyu commented on a change in pull request #3771: [HUDI-2402] Add Kerberos configuration options to Hive Sync
test-wangxiaoyu commented on a change in pull request #3771:
URL: https://github.com/apache/hudi/pull/3771#discussion_r730753592
##
File path:
hudi-sync/hudi-hive-sync/src/test/java/org/apache/hudi/hive/TestHiveSyncTool.java
##
@@ -1017,4 +1019,34 @@ public void testTypeConverter(String syncMode) throws
Exception {
.containsValue("BIGINT"), errorMsg);
ddlExecutor.runSQL(dropTableSql);
}
+
+ @Test
+ public void testHiveSyncOfKerberosEnvironment() throws Exception {
+// Path to the krb5 file
+String KER5_PATH = "/krb5.conf";
+// Path to the user keytab file
+String PRINCIPAL = "/test.keytab";
+// User Principal
+String KEYTAB_PATH = "test@HADOOP";
+// Kerberos authentication of the client
+System.setProperty("java.security.krb5.conf", KER5_PATH);
+Configuration conf = new Configuration();
+conf.set("hadoop.security.authentication", "kerberos");
+UserGroupInformation.setConfiguration(conf);
+UserGroupInformation.loginUserFromKeytab(PRINCIPAL, KEYTAB_PATH);
+// If flink or Spark is used, you need to use the Kerberos authentication
method recommended by Flink or Spark
+
+// Setting Kerberos Parameters
+// Hive Principal
+String HIVE_PRINCIPAL = "hive@HADOOP";
+HiveTestUtil.hiveSyncConfig.useKerberos = true;
+HiveTestUtil.hiveSyncConfig.kerberosPrincipal = HIVE_PRINCIPAL;
+HiveSyncTool hiveSyncTool = new HiveSyncTool(hiveSyncConfig,
HiveTestUtil.getHiveConf(), HiveTestUtil.fileSystem);
+HoodieHiveClient hoodieHiveClient = hiveSyncTool.hoodieHiveClient;
+
+// Create a database in Hive. If the database is created successfully
+// the client synchronizes the Kerberos metadata for managing Hive
+hoodieHiveClient.createDatabase(hiveSyncConfig.databaseName);
+
Review comment:
Ok, I added Assert to determine if the database was created
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] test-wangxiaoyu commented on a change in pull request #3771: [HUDI-2402] Add Kerberos configuration options to Hive Sync
test-wangxiaoyu commented on a change in pull request #3771: URL: https://github.com/apache/hudi/pull/3771#discussion_r728802039 ## File path: hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java ## @@ -307,6 +307,10 @@ public static HiveSyncConfig buildHiveSyncConfig(TypedProperties props, String b DataSourceWriteOptions.HIVE_SKIP_RO_SUFFIX_FOR_READ_OPTIMIZED_TABLE().defaultValue())); hiveSyncConfig.supportTimestamp = Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_SUPPORT_TIMESTAMP_TYPE().key(), DataSourceWriteOptions.HIVE_SUPPORT_TIMESTAMP_TYPE().defaultValue())); +hiveSyncConfig.useKerberos = + Boolean.valueOf(props.getString(DataSourceWriteOptions.HIVE_SYNC_USE_KERBEROS().key(),DataSourceWriteOptions.HIVE_SYNC_USE_KERBEROS().defaultValue())); Review comment: I added in the TestHiveSyncTool testHiveSyncOfKerberosEnvironment method is used to test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] test-wangxiaoyu commented on a change in pull request #3771: [HUDI-2402] Add Kerberos configuration options to Hive Sync
test-wangxiaoyu commented on a change in pull request #3771:
URL: https://github.com/apache/hudi/pull/3771#discussion_r728800351
##
File path:
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java
##
@@ -77,6 +77,10 @@ public HiveSyncTool(HiveSyncConfig cfg, HiveConf
configuration, FileSystem fs) {
super(configuration.getAllProperties(), fs);
try {
+ if (cfg.useKerberos) {
+configuration.set("hive.metastore.sasl.enabled", "true");
+configuration.set("hive.metastore.kerberos.principal",
cfg.kerberosPrincipal);
Review comment:
Thanks to review
I set it to its initial value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
