rdblue commented on a change in pull request #2038:
URL: https://github.com/apache/iceberg/pull/2038#discussion_r559048014
##########
File path:
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java
##########
@@ -206,33 +207,42 @@ public static PartitionSpec spec(Configuration config) {
* @param config The target configuration to store to
* @param table The table which we want to store to the configuration
*/
+
@VisibleForTesting
- static void put(Configuration config, Table table) {
- // The Table contains a FileIO and the FileIO serializes the configuration
so we might end up recursively
- // serializing the objects. To avoid this unset the values for now before
serializing.
- config.unset(InputFormatConfig.SERIALIZED_TABLE);
- config.unset(InputFormatConfig.FILE_IO);
- config.unset(InputFormatConfig.LOCATION_PROVIDER);
- config.unset(InputFormatConfig.ENCRYPTION_MANAGER);
- config.unset(InputFormatConfig.TABLE_LOCATION);
- config.unset(InputFormatConfig.TABLE_SCHEMA);
- config.unset(InputFormatConfig.PARTITION_SPEC);
-
- String base64Table = table instanceof Serializable ?
SerializationUtil.serializeToBase64(table) : null;
- String base64Io = SerializationUtil.serializeToBase64(table.io());
- String base64LocationProvider =
SerializationUtil.serializeToBase64(table.locationProvider());
- String base64EncryptionManager =
SerializationUtil.serializeToBase64(table.encryption());
-
- if (base64Table != null) {
- config.set(InputFormatConfig.SERIALIZED_TABLE, base64Table);
+ static void overlayTableProperties(Configuration configuration, TableDesc
tableDesc, Map<String, String> map) {
+ Properties props = tableDesc.getProperties();
+ Table table = Catalogs.loadTable(configuration, props);
+ String schemaJson = SchemaParser.toJson(table.schema());
+
+ Map<String, String> original = new HashMap<>(map);
+ map.clear();
+
+ map.putAll(Maps.fromProperties(props));
+ map.putAll(original);
Review comment:
I would prefer not to use replacement for the order here because it
requires copying the map, adding properties, and then adding the map back,
which is convoluted. Technically, you don't even need `map.clear()` because the
properties would overwrite and then the original would overwrite.
I think it would be better to filter properties instead:
```java
Maps.fromProperties(props).entrySet().stream()
.filter(entry -> !map.containsKey(entry.getKey())) // map overrides
tableDesc properties
.forEach(entry -> map.put(entry.getKey(), entry.getValue()));
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
