lixiuxiu opened a new issue, #6027:
URL: https://github.com/apache/iceberg/issues/6027
### Query engine
java -> 1.8
iceberg-> 0.13.1
hive-metastore -> 2.1.1-cdh6.3.0
hadoop -> 3.0.0-cdh6.3.0
### Question
We have two hadoop clusters named A and B with High availability， config
that dfs.nameservices ：nameservice1，when I use my program to create iceberg
table on A，it works，and than create table on B，the metadata.json will appear on
A cluster , so the table create on B can not be read . with can not find
metadata error. If I restart my program , the first table create successfully,
and the table created on second cluster failed , I seems that the hadoop config
have a cache. please help me , thank you very much !
` configuration.set("dfs.nameservices", "nameservice1");
configuration.set("dfs.ha.namenodes.nameservice1",
catalogConf.getNameNodes());
configuration.set("dfs.namenode.rpc-address.nameservice1."+catalogConf.getNameNode1(),
catalogConf.getNameNode1Value());
configuration.set("dfs.namenode.rpc-address.nameservice1."+catalogConf.getNameNode2(),
catalogConf.getNameNode2Value());
configuration.set("dfs.client.failover.proxy.provider.nameservice1",
"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");
configuration.set("fs.hdfs.impl",
"org.apache.hadoop.hdfs.DistributedFileSystem");
configuration.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY,
catalogConf.getDefaultFS());
HiveCatalog hivecatalog = new HiveCatalog();
hivecatalog.setConf(configuration); // Configure using Spark's
Hadoop configuration
Map <String, String> properties = new HashMap<String, String>();
properties.put(ICEBERG_CATALOG_TYPE, catalogType);
properties.put(CatalogProperties.WAREHOUSE_LOCATION,
catalogConf.getWarehouse());
properties.put(CatalogProperties.URI, catalogConf.getMetastore());
properties.put(CatalogProperties.USER,
StringUtils.isEmpty(owner)?"user":owner);
hivecatalog.initialize("hive", properties);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
