marton-bod commented on a change in pull request #2475:
URL: https://github.com/apache/hive/pull/2475#discussion_r681561292
##########
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java
##########
@@ -507,35 +506,38 @@ private void
handleAddColumns(org.apache.hadoop.hive.metastore.api.Table hmsTabl
}
private void handleReplaceColumns(org.apache.hadoop.hive.metastore.api.Table
hmsTable) throws MetaException {
- HiveSchemaUtil.SchemaDifference schemaDifference =
HiveSchemaUtil.getSchemaDiff(hmsTable.getSd().getCols(),
- HiveSchemaUtil.convert(icebergTable.schema()), true);
- if (!schemaDifference.isEmpty()) {
- updateSchema = icebergTable.updateSchema();
- } else {
- // we should get here if the user restated the exactly the existing
columns in the REPLACE COLUMNS command
- LOG.info("Found no difference between new and old schema for ALTER TABLE
REPLACE COLUMNS for" +
- " table: {}. There will be no Iceberg commit.",
hmsTable.getTableName());
- return;
- }
+ List<FieldSchema> hmsCols = hmsTable.getSd().getCols();
+ List<FieldSchema> icebergCols =
HiveSchemaUtil.convert(icebergTable.schema());
+ HiveSchemaUtil.SchemaDifference schemaDifference =
HiveSchemaUtil.getSchemaDiff(hmsCols, icebergCols, true);
- for (FieldSchema droppedCol : schemaDifference.getMissingFromFirst()) {
- updateSchema.deleteColumn(droppedCol.getName());
+ // if there are columns dropped, let's remove them from the iceberg schema
as well so we can compare the order
+ if (!schemaDifference.getMissingFromFirst().isEmpty()) {
+ schemaDifference.getMissingFromFirst().forEach(icebergCols::remove);
}
- for (FieldSchema addedCol : schemaDifference.getMissingFromSecond()) {
- updateSchema.addColumn(
- addedCol.getName(),
-
HiveSchemaUtil.convert(TypeInfoUtils.getTypeInfoFromTypeString(addedCol.getType())),
- addedCol.getComment()
- );
- }
+ Pair<String, Optional<String>> outOfOrder =
HiveSchemaUtil.getFirstOutOfOrderColumn(
+ hmsCols, icebergCols, ImmutableMap.of());
- for (FieldSchema updatedCol : schemaDifference.getTypeChanged()) {
- updateSchema.updateColumn(updatedCol.getName(),
getPrimitiveTypeOrThrow(updatedCol), updatedCol.getComment());
+ // limit the scope of this operation to only dropping columns
+ if (!schemaDifference.getMissingFromSecond().isEmpty() ||
!schemaDifference.getTypeChanged().isEmpty() ||
+ !schemaDifference.getCommentChanged().isEmpty() || outOfOrder != null)
{
+ throw new MetaException("Unsupported operation to use REPLACE COLUMNS
for adding a column, changing a " +
+ "column type, column comment or reordering columns. Only use REPLACE
COLUMNS for dropping columns. " +
+ "For the other operations, consider using the ADD COLUMNS or CHANGE
COLUMN commands.");
}
- for (FieldSchema updatedCol : schemaDifference.getCommentChanged()) {
- updateSchema.updateColumnDoc(updatedCol.getName(),
updatedCol.getComment());
+ // check if there were any column drops
+ if (!schemaDifference.getMissingFromFirst().isEmpty()) {
Review comment:
As discussed offline, let's throw an exception for no-ops so that users
are aware they did not change anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
