[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=613890=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-613890
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 23/Jun/21 08:33
Start Date: 23/Jun/21 08:33
Worklog Time Spent: 10m
Work Description: lcspinter merged pull request #2382:
URL: https://github.com/apache/hive/pull/2382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 613890)
Time Spent: 2h 40m (was: 2.5h)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=613865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-613865
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 23/Jun/21 07:47
Start Date: 23/Jun/21 07:47
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2382:
URL: https://github.com/apache/hive/pull/2382#discussion_r656844765
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java
##
@@ -106,4 +111,61 @@ public static PartitionSpec spec(Configuration
configuration, Schema schema) {
});
return builder.build();
}
+
+ public static void updateSpec(Configuration configuration, Table table) {
+// get the new partition transform spec
+PartitionSpec newPartitionSpec = spec(configuration, table.schema());
+if (newPartitionSpec == null) {
+ LOG.debug("Iceberg Partition spec is not updated due to empty partition
spec definition.");
+ return;
+}
+
+List newPartitionNames =
+
newPartitionSpec.fields().stream().map(PartitionField::name).collect(Collectors.toList());
+List currentPartitionNames =
table.spec().fields().stream().map(PartitionField::name)
+.collect(Collectors.toList());
+List intersectingPartitionNames =
+
currentPartitionNames.stream().filter(newPartitionNames::contains).collect(Collectors.toList());
+
+// delete those partitions which are not present among the new partion spec
+UpdatePartitionSpec updatePartitionSpec = table.updateSpec();
+currentPartitionNames.stream().filter(p ->
!intersectingPartitionNames.contains(p))
+.forEach(updatePartitionSpec::removeField);
+updatePartitionSpec.apply();
+
+// add new partitions which are not yet present
+List partitionTransformSpecList
= SessionStateUtil
+.getResource(configuration,
hive_metastoreConstants.PARTITION_TRANSFORM_SPEC)
+.map(o -> (List)
o).orElseGet(() -> null);
+IntStream.range(0, partitionTransformSpecList.size())
+.filter(i ->
!intersectingPartitionNames.contains(newPartitionSpec.fields().get(i).name()))
+.forEach(i -> {
+ PartitionTransform.PartitionTransformSpec spec =
partitionTransformSpecList.get(i);
+ switch (spec.transformType) {
Review comment:
PartitionSpecVisitor would be the class which we should implement in
this case (to be honest in both cases)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 613865)
Time Spent: 2.5h (was: 2h 20m)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=613484=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-613484
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 22/Jun/21 13:34
Start Date: 22/Jun/21 13:34
Worklog Time Spent: 10m
Work Description: lcspinter commented on a change in pull request #2382:
URL: https://github.com/apache/hive/pull/2382#discussion_r656226909
##
File path:
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java
##
@@ -155,6 +155,93 @@ public void after() throws Exception {
HiveIcebergStorageHandlerTestUtils.close(shell);
}
+ @Test
+ public void testPartitionEvolution() {
Review comment:
A Semantic exception will be raised. I created a q test to cover this
scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 613484)
Time Spent: 2h 20m (was: 2h 10m)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=613455=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-613455
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 22/Jun/21 12:29
Start Date: 22/Jun/21 12:29
Worklog Time Spent: 10m
Work Description: lcspinter commented on a change in pull request #2382:
URL: https://github.com/apache/hive/pull/2382#discussion_r656170476
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java
##
@@ -106,4 +111,61 @@ public static PartitionSpec spec(Configuration
configuration, Schema schema) {
});
return builder.build();
}
+
+ public static void updateSpec(Configuration configuration, Table table) {
+// get the new partition transform spec
+PartitionSpec newPartitionSpec = spec(configuration, table.schema());
+if (newPartitionSpec == null) {
+ LOG.debug("Iceberg Partition spec is not updated due to empty partition
spec definition.");
+ return;
+}
+
+List newPartitionNames =
+
newPartitionSpec.fields().stream().map(PartitionField::name).collect(Collectors.toList());
+List currentPartitionNames =
table.spec().fields().stream().map(PartitionField::name)
+.collect(Collectors.toList());
+List intersectingPartitionNames =
+
currentPartitionNames.stream().filter(newPartitionNames::contains).collect(Collectors.toList());
+
+// delete those partitions which are not present among the new partion spec
+UpdatePartitionSpec updatePartitionSpec = table.updateSpec();
+currentPartitionNames.stream().filter(p ->
!intersectingPartitionNames.contains(p))
+.forEach(updatePartitionSpec::removeField);
+updatePartitionSpec.apply();
+
+// add new partitions which are not yet present
+List partitionTransformSpecList
= SessionStateUtil
+.getResource(configuration,
hive_metastoreConstants.PARTITION_TRANSFORM_SPEC)
+.map(o -> (List)
o).orElseGet(() -> null);
+IntStream.range(0, partitionTransformSpecList.size())
+.filter(i ->
!intersectingPartitionNames.contains(newPartitionSpec.fields().get(i).name()))
+.forEach(i -> {
+ PartitionTransform.PartitionTransformSpec spec =
partitionTransformSpecList.get(i);
+ switch (spec.transformType) {
Review comment:
I'm not sure we can combine the two switch statements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 613455)
Time Spent: 2h 10m (was: 2h)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=613453=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-613453
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 22/Jun/21 12:26
Start Date: 22/Jun/21 12:26
Worklog Time Spent: 10m
Work Description: lcspinter commented on a change in pull request #2382:
URL: https://github.com/apache/hive/pull/2382#discussion_r656168619
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java
##
@@ -214,7 +219,7 @@ public void
commitDropTable(org.apache.hadoop.hive.metastore.api.Table hmsTable,
@Override
public void preAlterTable(org.apache.hadoop.hive.metastore.api.Table
hmsTable, EnvironmentContext context)
throws MetaException {
-super.preAlterTable(hmsTable, context);
+setupAlterOperationType(hmsTable, context);
Review comment:
Yes, this was intentional. The validation of supported alter operation
types was moved from the superclass to the IcebergMetaHook
(`setupAlterOperationType`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 613453)
Time Spent: 2h (was: 1h 50m)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=613161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-613161
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 22/Jun/21 07:54
Start Date: 22/Jun/21 07:54
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2382:
URL: https://github.com/apache/hive/pull/2382#discussion_r655272146
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java
##
@@ -214,7 +219,7 @@ public void
commitDropTable(org.apache.hadoop.hive.metastore.api.Table hmsTable,
@Override
public void preAlterTable(org.apache.hadoop.hive.metastore.api.Table
hmsTable, EnvironmentContext context)
throws MetaException {
-super.preAlterTable(hmsTable, context);
+setupAlterOperationType(hmsTable, context);
Review comment:
We removed the `super.preAlterTable`. Is this intentional?
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java
##
@@ -106,4 +111,61 @@ public static PartitionSpec spec(Configuration
configuration, Schema schema) {
});
return builder.build();
}
+
+ public static void updateSpec(Configuration configuration, Table table) {
+// get the new partition transform spec
+PartitionSpec newPartitionSpec = spec(configuration, table.schema());
+if (newPartitionSpec == null) {
+ LOG.debug("Iceberg Partition spec is not updated due to empty partition
spec definition.");
+ return;
+}
+
+List newPartitionNames =
+
newPartitionSpec.fields().stream().map(PartitionField::name).collect(Collectors.toList());
+List currentPartitionNames =
table.spec().fields().stream().map(PartitionField::name)
+.collect(Collectors.toList());
+List intersectingPartitionNames =
+
currentPartitionNames.stream().filter(newPartitionNames::contains).collect(Collectors.toList());
+
+// delete those partitions which are not present among the new partion spec
+UpdatePartitionSpec updatePartitionSpec = table.updateSpec();
+currentPartitionNames.stream().filter(p ->
!intersectingPartitionNames.contains(p))
+.forEach(updatePartitionSpec::removeField);
+updatePartitionSpec.apply();
+
+// add new partitions which are not yet present
+List partitionTransformSpecList
= SessionStateUtil
+.getResource(configuration,
hive_metastoreConstants.PARTITION_TRANSFORM_SPEC)
+.map(o -> (List)
o).orElseGet(() -> null);
+IntStream.range(0, partitionTransformSpecList.size())
+.filter(i ->
!intersectingPartitionNames.contains(newPartitionSpec.fields().get(i).name()))
+.forEach(i -> {
+ PartitionTransform.PartitionTransformSpec spec =
partitionTransformSpecList.get(i);
+ switch (spec.transformType) {
Review comment:
This `switch` is very similar to the one where we are converting the
json to the spec. Do we have a way to reuse code?
##
File path:
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java
##
@@ -155,6 +155,93 @@ public void after() throws Exception {
HiveIcebergStorageHandlerTestUtils.close(shell);
}
+ @Test
+ public void testPartitionEvolution() {
Review comment:
What happens when I try to alter an HBase table with partition spec, or
a normal Hive (non-Iceberg) table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 613161)
Time Spent: 1h 50m (was: 1h 40m)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=612545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612545
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 21/Jun/21 10:59
Start Date: 21/Jun/21 10:59
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2382:
URL: https://github.com/apache/hive/pull/2382#discussion_r655278621
##
File path:
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java
##
@@ -155,6 +155,93 @@ public void after() throws Exception {
HiveIcebergStorageHandlerTestUtils.close(shell);
}
+ @Test
+ public void testPartitionEvolution() {
Review comment:
What happens when I try to alter an HBase table with partition spec, or
a normal Hive (non-Iceberg) table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 612545)
Time Spent: 1h 40m (was: 1.5h)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=612544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612544
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 21/Jun/21 10:59
Start Date: 21/Jun/21 10:59
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2382:
URL: https://github.com/apache/hive/pull/2382#discussion_r655278190
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java
##
@@ -106,4 +111,61 @@ public static PartitionSpec spec(Configuration
configuration, Schema schema) {
});
return builder.build();
}
+
+ public static void updateSpec(Configuration configuration, Table table) {
+// get the new partition transform spec
+PartitionSpec newPartitionSpec = spec(configuration, table.schema());
+if (newPartitionSpec == null) {
+ LOG.debug("Iceberg Partition spec is not updated due to empty partition
spec definition.");
+ return;
+}
+
+List newPartitionNames =
+
newPartitionSpec.fields().stream().map(PartitionField::name).collect(Collectors.toList());
+List currentPartitionNames =
table.spec().fields().stream().map(PartitionField::name)
+.collect(Collectors.toList());
+List intersectingPartitionNames =
+
currentPartitionNames.stream().filter(newPartitionNames::contains).collect(Collectors.toList());
+
+// delete those partitions which are not present among the new partion spec
+UpdatePartitionSpec updatePartitionSpec = table.updateSpec();
+currentPartitionNames.stream().filter(p ->
!intersectingPartitionNames.contains(p))
+.forEach(updatePartitionSpec::removeField);
+updatePartitionSpec.apply();
+
+// add new partitions which are not yet present
+List partitionTransformSpecList
= SessionStateUtil
+.getResource(configuration,
hive_metastoreConstants.PARTITION_TRANSFORM_SPEC)
+.map(o -> (List)
o).orElseGet(() -> null);
+IntStream.range(0, partitionTransformSpecList.size())
+.filter(i ->
!intersectingPartitionNames.contains(newPartitionSpec.fields().get(i).name()))
+.forEach(i -> {
+ PartitionTransform.PartitionTransformSpec spec =
partitionTransformSpecList.get(i);
+ switch (spec.transformType) {
Review comment:
This `switch` is very similar to the one where we are converting the
json to the spec. Do we have a way to reuse code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 612544)
Time Spent: 1.5h (was: 1h 20m)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=612542=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612542
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 21/Jun/21 10:48
Start Date: 21/Jun/21 10:48
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2382:
URL: https://github.com/apache/hive/pull/2382#discussion_r655272146
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java
##
@@ -214,7 +219,7 @@ public void
commitDropTable(org.apache.hadoop.hive.metastore.api.Table hmsTable,
@Override
public void preAlterTable(org.apache.hadoop.hive.metastore.api.Table
hmsTable, EnvironmentContext context)
throws MetaException {
-super.preAlterTable(hmsTable, context);
+setupAlterOperationType(hmsTable, context);
Review comment:
We removed the `super.preAlterTable`. Is this intentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 612542)
Time Spent: 1h 20m (was: 1h 10m)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=611283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-611283
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 15/Jun/21 12:18
Start Date: 15/Jun/21 12:18
Worklog Time Spent: 10m
Work Description: lcspinter commented on a change in pull request #2382:
URL: https://github.com/apache/hive/pull/2382#discussion_r651732394
##
File path:
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java
##
@@ -155,6 +155,58 @@ public void after() throws Exception {
HiveIcebergStorageHandlerTestUtils.close(shell);
}
+ @Test
+ public void testSetPartitionTransform() {
+Schema schema = new Schema(
+optional(1, "id", Types.LongType.get()),
+optional(2, "year_field", Types.DateType.get()),
+optional(3, "month_field", Types.TimestampType.withZone()),
+optional(4, "day_field", Types.TimestampType.withoutZone()),
+optional(5, "hour_field", Types.TimestampType.withoutZone()),
+optional(6, "truncate_field", Types.StringType.get()),
+optional(7, "bucket_field", Types.StringType.get()),
+optional(8, "identity_field", Types.StringType.get())
+);
+
+TableIdentifier identifier = TableIdentifier.of("default", "part_test");
+shell.executeStatement("CREATE EXTERNAL TABLE " + identifier +
+" PARTITIONED BY SPEC (year(year_field), hour(hour_field), " +
+"truncate(2, truncate_field), bucket(2, bucket_field),
identity_field)" +
+" STORED BY ICEBERG " +
+testTables.locationForCreateTableSQL(identifier) +
+"TBLPROPERTIES ('" + InputFormatConfig.TABLE_SCHEMA + "'='" +
+SchemaParser.toJson(schema) + "', " +
+"'" + InputFormatConfig.CATALOG_NAME + "'='" +
Catalogs.ICEBERG_DEFAULT_CATALOG_NAME + "')");
+
+PartitionSpec spec = PartitionSpec.builderFor(schema)
+.year("year_field")
+.hour("hour_field")
+.truncate("truncate_field", 2)
+.bucket("bucket_field", 2)
+.identity("identity_field")
+.build();
+
+Table table = testTables.loadTable(identifier);
+Assert.assertEquals(spec, table.spec());
+
+shell.executeStatement("ALTER TABLE default.part_test SET PARTITION
SPEC(year(year_field), month(month_field), " +
+"day(day_field))");
+
+spec = PartitionSpec.builderFor(schema)
+.withSpecId(1)
+.year("year_field")
+.alwaysNull("hour_field", "hour_field_hour")
+.alwaysNull("truncate_field", "truncate_field_trunc")
+.alwaysNull("bucket_field", "bucket_field_bucket")
+.alwaysNull("identity_field", "identity_field")
+.month("month_field")
+.day("day_field")
+.build();
+
+table.refresh();
+Assert.assertEquals(spec, table.spec());
+ }
+
Review comment:
I added an additional test case to cover the partition evolution. Thanks
for the idea!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 611283)
Time Spent: 1h 10m (was: 1h)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=610645=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-610645
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 14/Jun/21 12:05
Start Date: 14/Jun/21 12:05
Worklog Time Spent: 10m
Work Description: lcspinter commented on a change in pull request #2382:
URL: https://github.com/apache/hive/pull/2382#discussion_r650886501
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java
##
@@ -106,4 +111,61 @@ public static PartitionSpec spec(Configuration
configuration, Schema schema) {
});
return builder.build();
}
+
+ public static void updateSpec(Configuration configuration, Table table) {
+// get the new partition transform spec
+PartitionSpec newPartitionSpec = spec(configuration, table.schema());
+if (newPartitionSpec == null) {
+ LOG.debug("Iceberg Partition spec is not updated due to empty partition
spec definition.");
+ return;
+}
+
+List partitionTransformSpecList
= SessionStateUtil
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 610645)
Time Spent: 1h (was: 50m)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=610644=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-610644
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 14/Jun/21 12:04
Start Date: 14/Jun/21 12:04
Worklog Time Spent: 10m
Work Description: lcspinter commented on a change in pull request #2382:
URL: https://github.com/apache/hive/pull/2382#discussion_r650885975
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java
##
@@ -311,6 +312,38 @@ public void
rollbackAlterTable(org.apache.hadoop.hive.metastore.api.Table hmsTab
}
}
+ private void alterTableProperties(org.apache.hadoop.hive.metastore.api.Table
hmsTable,
+ Map contextProperties) {
+Map hmsTableParameters = hmsTable.getParameters();
+Splitter splitter = Splitter.on(PROPERTIES_SEPARATOR);
+UpdateProperties icebergUpdateProperties = icebergTable.updateProperties();
+if (contextProperties.containsKey(SET_PROPERTIES)) {
+ splitter.splitToList(contextProperties.get(SET_PROPERTIES))
+ .forEach(k -> icebergUpdateProperties.set(k,
hmsTableParameters.get(k)));
Review comment:
You are right, we should do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 610644)
Time Spent: 50m (was: 40m)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=610643=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-610643
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 14/Jun/21 12:03
Start Date: 14/Jun/21 12:03
Worklog Time Spent: 10m
Work Description: lcspinter commented on a change in pull request #2382:
URL: https://github.com/apache/hive/pull/2382#discussion_r650885011
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java
##
@@ -106,4 +111,61 @@ public static PartitionSpec spec(Configuration
configuration, Schema schema) {
});
return builder.build();
}
+
+ public static void updateSpec(Configuration configuration, Table table) {
+// get the new partition transform spec
+PartitionSpec newPartitionSpec = spec(configuration, table.schema());
+if (newPartitionSpec == null) {
+ LOG.debug("Iceberg Partition spec is not updated due to empty partition
spec definition.");
Review comment:
We can get null values if the SessionState is somehow lost. Or the
method is called from a different process (like Tez). I just wanted to make
sure we log this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 610643)
Time Spent: 40m (was: 0.5h)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=610121=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-610121
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 14/Jun/21 07:19
Start Date: 14/Jun/21 07:19
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #2382:
URL: https://github.com/apache/hive/pull/2382#discussion_r649853361
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java
##
@@ -311,6 +312,38 @@ public void
rollbackAlterTable(org.apache.hadoop.hive.metastore.api.Table hmsTab
}
}
+ private void alterTableProperties(org.apache.hadoop.hive.metastore.api.Table
hmsTable,
+ Map contextProperties) {
+Map hmsTableParameters = hmsTable.getParameters();
+Splitter splitter = Splitter.on(PROPERTIES_SEPARATOR);
+UpdateProperties icebergUpdateProperties = icebergTable.updateProperties();
+if (contextProperties.containsKey(SET_PROPERTIES)) {
+ splitter.splitToList(contextProperties.get(SET_PROPERTIES))
+ .forEach(k -> icebergUpdateProperties.set(k,
hmsTableParameters.get(k)));
Review comment:
I know it's not part of this PR's topic, but I just noticed: shouldn't
we do the hms-iceberg property key translation here? instead of just `k` ->
`HiveTableOperations.translateToIcebergProp(k)`
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java
##
@@ -106,4 +111,61 @@ public static PartitionSpec spec(Configuration
configuration, Schema schema) {
});
return builder.build();
}
+
+ public static void updateSpec(Configuration configuration, Table table) {
+// get the new partition transform spec
+PartitionSpec newPartitionSpec = spec(configuration, table.schema());
+if (newPartitionSpec == null) {
+ LOG.debug("Iceberg Partition spec is not updated due to empty partition
spec definition.");
Review comment:
Is this scenario possible? Would this happen if you ran `ALTER TABLE tbl
SET PARTITION SPEC()`? The parser would not complain about this?
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java
##
@@ -106,4 +111,61 @@ public static PartitionSpec spec(Configuration
configuration, Schema schema) {
});
return builder.build();
}
+
+ public static void updateSpec(Configuration configuration, Table table) {
+// get the new partition transform spec
+PartitionSpec newPartitionSpec = spec(configuration, table.schema());
+if (newPartitionSpec == null) {
+ LOG.debug("Iceberg Partition spec is not updated due to empty partition
spec definition.");
+ return;
+}
+
+List partitionTransformSpecList
= SessionStateUtil
+.getResource(configuration,
hive_metastoreConstants.PARTITION_TRANSFORM_SPEC)
+.map(o -> (List)
o).orElseGet(() -> null);
+List newPartitionNames =
+
newPartitionSpec.fields().stream().map(PartitionField::name).collect(Collectors.toList());
+List currentPartitionNames =
table.spec().fields().stream().map(PartitionField::name)
+.collect(Collectors.toList());
+List intersectingPartitionNames =
+
currentPartitionNames.stream().filter(newPartitionNames::contains).collect(Collectors.toList());
+
+// delete those partitions which are not present among the new partion spec
+UpdatePartitionSpec updatePartitionSpec = table.updateSpec();
+currentPartitionNames.stream().filter(p ->
!intersectingPartitionNames.contains(p))
+.forEach(updatePartitionSpec::removeField);
+updatePartitionSpec.apply();
+
+// add new partitions which are not yet present
+IntStream.range(0, partitionTransformSpecList.size())
Review comment:
Could we loop through partitionTransformSpecList directly without the
IntStream?
Maybe this is possible?
```
partitionTransformSpecList.stream()
.filter(spec -> !intersectingPartitionNames.contains(spec.name)
.forEach(...)
```
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java
##
@@ -106,4 +111,61 @@ public static PartitionSpec spec(Configuration
configuration, Schema schema) {
});
return builder.build();
}
+
+ public static void updateSpec(Configuration configuration, Table table) {
+// get the new partition transform spec
+PartitionSpec newPartitionSpec = spec(configuration, table.schema());
+if (newPartitionSpec == null) {
+ LOG.debug("Iceberg Partition spec is not updated due to empty partition
spec definition.");
+ return;
+}
+
+List partitionTransformSpecList
= SessionStateUtil
Review comment:
nit: can you move this closer to
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=609882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609882
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 10/Jun/21 18:54
Start Date: 10/Jun/21 18:54
Worklog Time Spent: 10m
Work Description: lcspinter commented on pull request #2382:
URL: https://github.com/apache/hive/pull/2382#issuecomment-858911481
@marton-bod @pvary @szlta Could you please review this change? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 609882)
Time Spent: 20m (was: 10m)
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25234) Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-25234?focusedWorklogId=609733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609733
]
ASF GitHub Bot logged work on HIVE-25234:
-
Author: ASF GitHub Bot
Created on: 10/Jun/21 14:38
Start Date: 10/Jun/21 14:38
Worklog Time Spent: 10m
Work Description: lcspinter opened a new pull request #2382:
URL: https://github.com/apache/hive/pull/2382
### What changes were proposed in this pull request?
Create new syntax to support partition spec change on Iceberg tables.
`ALTER TABLE tbl SET PARTITION SPEC(years(ts), id)`
During the alter operation, the new partition spec will overwrite the
existing partition spec.
### Why are the changes needed?
HIVE-25179 introduced support for partition transforms from DDL. We need a
way to alter existing partition transforms
### Does this PR introduce _any_ user-facing change?
New syntax.
### How was this patch tested?
Manual test, unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 609733)
Remaining Estimate: 0h
Time Spent: 10m
> Implement ALTER TABLE ... SET PARTITION SPEC to change partitioning on
> Iceberg tables
> -
>
> Key: HIVE-25234
> URL: https://issues.apache.org/jira/browse/HIVE-25234
> Project: Hive
> Issue Type: Improvement
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Provide a way to change the schema and the Iceberg partitioning specification
> using Hive syntax.
> {code:sql}
> ALTER TABLE tbl SET PARTITION SPEC(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
