[jira] [Work logged] (HIVE-26158) TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after rename table
[
https://issues.apache.org/jira/browse/HIVE-26158?focusedWorklogId=769129&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-769129
]
ASF GitHub Bot logged work on HIVE-26158:
-
Author: ASF GitHub Bot
Created on: 11/May/22 15:10
Start Date: 11/May/22 15:10
Worklog Time Spent: 10m
Work Description: kgyrtkirk merged PR #3255:
URL: https://github.com/apache/hive/pull/3255
Issue Time Tracking
---
Worklog Id: (was: 769129)
Time Spent: 1h (was: 50m)
> TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after
> rename table
> --
>
> Key: HIVE-26158
> URL: https://issues.apache.org/jira/browse/HIVE-26158
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: tanghui
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: metastore_translator, pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> After the patch is updated, the partition table location and hdfs data
> directory are displayed normally, but the partition location of the table in
> the SDS in the Hive metabase is still displayed as the location of the old
> table, resulting in no data in the query partition.
>
> in beeline:
>
> set hive.create.as.external.legacy=true;
> CREATE TABLE part_test(
> c1 string
> ,c2 string
> )PARTITIONED BY (dat string)
> insert into part_test values ("11","th","20220101")
> insert into part_test values ("22","th","20220102")
> alter table part_test rename to part_test11;
> --this result is null.
> select * from part_test11 where dat="20220101";
> ||part_test.c1||part_test.c2||part_test.dat||
> | | | |
> -
> SDS in the Hive metabase:
> select SDS.LOCATION from TBLS,SDS where TBLS.TBL_NAME="part_test11" AND
> TBLS.TBL_ID=SDS.CD_ID;
> ---
> |*LOCATION*|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test11|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220101|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220102|
> ---
>
> We need to modify the partition location of the table in SDS to ensure that
> the query results are normal
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Work logged] (HIVE-26158) TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after rename table
[
https://issues.apache.org/jira/browse/HIVE-26158?focusedWorklogId=768492&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-768492
]
ASF GitHub Bot logged work on HIVE-26158:
-
Author: ASF GitHub Bot
Created on: 10/May/22 13:13
Start Date: 10/May/22 13:13
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on code in PR #3255:
URL: https://github.com/apache/hive/pull/3255#discussion_r869217593
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##
@@ -244,6 +244,13 @@ public static boolean isExternalTable(Table table) {
return isExternal(params);
}
+ public static boolean isTranslatedToExternalTable(Table table) {
+Map p = table.getParameters();
Review Comment:
renamed it to `params` as its being used in this file
Issue Time Tracking
---
Worklog Id: (was: 768492)
Time Spent: 50m (was: 40m)
> TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after
> rename table
> --
>
> Key: HIVE-26158
> URL: https://issues.apache.org/jira/browse/HIVE-26158
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: tanghui
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: metastore_translator, pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> After the patch is updated, the partition table location and hdfs data
> directory are displayed normally, but the partition location of the table in
> the SDS in the Hive metabase is still displayed as the location of the old
> table, resulting in no data in the query partition.
>
> in beeline:
>
> set hive.create.as.external.legacy=true;
> CREATE TABLE part_test(
> c1 string
> ,c2 string
> )PARTITIONED BY (dat string)
> insert into part_test values ("11","th","20220101")
> insert into part_test values ("22","th","20220102")
> alter table part_test rename to part_test11;
> --this result is null.
> select * from part_test11 where dat="20220101";
> ||part_test.c1||part_test.c2||part_test.dat||
> | | | |
> -
> SDS in the Hive metabase:
> select SDS.LOCATION from TBLS,SDS where TBLS.TBL_NAME="part_test11" AND
> TBLS.TBL_ID=SDS.CD_ID;
> ---
> |*LOCATION*|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test11|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220101|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220102|
> ---
>
> We need to modify the partition location of the table in SDS to ensure that
> the query results are normal
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Work logged] (HIVE-26158) TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after rename table
[
https://issues.apache.org/jira/browse/HIVE-26158?focusedWorklogId=768490&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-768490
]
ASF GitHub Bot logged work on HIVE-26158:
-
Author: ASF GitHub Bot
Created on: 10/May/22 13:10
Start Date: 10/May/22 13:10
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on code in PR #3255:
URL: https://github.com/apache/hive/pull/3255#discussion_r869215853
##
ql/src/test/queries/clientpositive/translated_external_rename3.q:
##
@@ -1,26 +1,25 @@
set
metastore.metadata.transformer.class=org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer;
-set metastore.metadata.transformer.location.mode=force;
Review Comment:
this was a mistake - I've clobbered this existing test too much; I've moved
the new testcase into a new qfile.
the location mode shouldn't matter for this test at all.
Issue Time Tracking
---
Worklog Id: (was: 768490)
Time Spent: 40m (was: 0.5h)
> TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after
> rename table
> --
>
> Key: HIVE-26158
> URL: https://issues.apache.org/jira/browse/HIVE-26158
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: tanghui
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: metastore_translator, pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> After the patch is updated, the partition table location and hdfs data
> directory are displayed normally, but the partition location of the table in
> the SDS in the Hive metabase is still displayed as the location of the old
> table, resulting in no data in the query partition.
>
> in beeline:
>
> set hive.create.as.external.legacy=true;
> CREATE TABLE part_test(
> c1 string
> ,c2 string
> )PARTITIONED BY (dat string)
> insert into part_test values ("11","th","20220101")
> insert into part_test values ("22","th","20220102")
> alter table part_test rename to part_test11;
> --this result is null.
> select * from part_test11 where dat="20220101";
> ||part_test.c1||part_test.c2||part_test.dat||
> | | | |
> -
> SDS in the Hive metabase:
> select SDS.LOCATION from TBLS,SDS where TBLS.TBL_NAME="part_test11" AND
> TBLS.TBL_ID=SDS.CD_ID;
> ---
> |*LOCATION*|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test11|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220101|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220102|
> ---
>
> We need to modify the partition location of the table in SDS to ensure that
> the query results are normal
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Work logged] (HIVE-26158) TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after rename table
[
https://issues.apache.org/jira/browse/HIVE-26158?focusedWorklogId=768488&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-768488
]
ASF GitHub Bot logged work on HIVE-26158:
-
Author: ASF GitHub Bot
Created on: 10/May/22 13:09
Start Date: 10/May/22 13:09
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on code in PR #3255:
URL: https://github.com/apache/hive/pull/3255#discussion_r869212555
##
ql/src/test/results/clientpositive/llap/translated_external_rename3.q.out:
##
@@ -95,15 +64,17 @@ Retention: 0
A masked pattern was here
Table Type:EXTERNAL_TABLE
Table Parameters:
- COLUMN_STATS_ACCURATE
{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\"}}
+ COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
Review Comment:
this is a partitioned table; and we (by mistake) have the
`COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}` on the table object.
stat values for for partitioned tables really matter at partition level
Issue Time Tracking
---
Worklog Id: (was: 768488)
Time Spent: 0.5h (was: 20m)
> TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after
> rename table
> --
>
> Key: HIVE-26158
> URL: https://issues.apache.org/jira/browse/HIVE-26158
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: tanghui
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: metastore_translator, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> After the patch is updated, the partition table location and hdfs data
> directory are displayed normally, but the partition location of the table in
> the SDS in the Hive metabase is still displayed as the location of the old
> table, resulting in no data in the query partition.
>
> in beeline:
>
> set hive.create.as.external.legacy=true;
> CREATE TABLE part_test(
> c1 string
> ,c2 string
> )PARTITIONED BY (dat string)
> insert into part_test values ("11","th","20220101")
> insert into part_test values ("22","th","20220102")
> alter table part_test rename to part_test11;
> --this result is null.
> select * from part_test11 where dat="20220101";
> ||part_test.c1||part_test.c2||part_test.dat||
> | | | |
> -
> SDS in the Hive metabase:
> select SDS.LOCATION from TBLS,SDS where TBLS.TBL_NAME="part_test11" AND
> TBLS.TBL_ID=SDS.CD_ID;
> ---
> |*LOCATION*|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test11|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220101|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220102|
> ---
>
> We need to modify the partition location of the table in SDS to ensure that
> the query results are normal
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Work logged] (HIVE-26158) TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after rename table
[
https://issues.apache.org/jira/browse/HIVE-26158?focusedWorklogId=768222&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-768222
]
ASF GitHub Bot logged work on HIVE-26158:
-
Author: ASF GitHub Bot
Created on: 09/May/22 23:32
Start Date: 09/May/22 23:32
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera commented on code in PR #3255:
URL: https://github.com/apache/hive/pull/3255#discussion_r867627190
##
ql/src/test/results/clientpositive/llap/translated_external_rename3.q.out:
##
@@ -95,15 +64,17 @@ Retention: 0
A masked pattern was here
Table Type:EXTERNAL_TABLE
Table Parameters:
- COLUMN_STATS_ACCURATE
{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\"}}
+ COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
Review Comment:
I'm wondering why the column_stats are missing here when we do a describe on
the table.
##
ql/src/test/queries/clientpositive/translated_external_rename3.q:
##
@@ -1,26 +1,25 @@
set
metastore.metadata.transformer.class=org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer;
-set metastore.metadata.transformer.location.mode=force;
Review Comment:
this is also working in force mode also. Can you please give some info about
why you had to change it to seqprefix?
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##
@@ -244,6 +244,13 @@ public static boolean isExternalTable(Table table) {
return isExternal(params);
}
+ public static boolean isTranslatedToExternalTable(Table table) {
+Map p = table.getParameters();
Review Comment:
Can you change the name of this variable 'p' to something more meaningful
like tblProperties?
Issue Time Tracking
---
Worklog Id: (was: 768222)
Time Spent: 20m (was: 10m)
> TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after
> rename table
> --
>
> Key: HIVE-26158
> URL: https://issues.apache.org/jira/browse/HIVE-26158
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: tanghui
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: metastore_translator, pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> After the patch is updated, the partition table location and hdfs data
> directory are displayed normally, but the partition location of the table in
> the SDS in the Hive metabase is still displayed as the location of the old
> table, resulting in no data in the query partition.
>
> in beeline:
>
> set hive.create.as.external.legacy=true;
> CREATE TABLE part_test(
> c1 string
> ,c2 string
> )PARTITIONED BY (dat string)
> insert into part_test values ("11","th","20220101")
> insert into part_test values ("22","th","20220102")
> alter table part_test rename to part_test11;
> --this result is null.
> select * from part_test11 where dat="20220101";
> ||part_test.c1||part_test.c2||part_test.dat||
> | | | |
> -
> SDS in the Hive metabase:
> select SDS.LOCATION from TBLS,SDS where TBLS.TBL_NAME="part_test11" AND
> TBLS.TBL_ID=SDS.CD_ID;
> ---
> |*LOCATION*|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test11|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220101|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220102|
> ---
>
> We need to modify the partition location of the table in SDS to ensure that
> the query results are normal
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Work logged] (HIVE-26158) TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after rename table
[
https://issues.apache.org/jira/browse/HIVE-26158?focusedWorklogId=763543&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-763543
]
ASF GitHub Bot logged work on HIVE-26158:
-
Author: ASF GitHub Bot
Created on: 28/Apr/22 14:27
Start Date: 28/Apr/22 14:27
Worklog Time Spent: 10m
Work Description: kgyrtkirk opened a new pull request, #3255:
URL: https://github.com/apache/hive/pull/3255
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
Issue Time Tracking
---
Worklog Id: (was: 763543)
Remaining Estimate: 0h
Time Spent: 10m
> TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after
> rename table
> --
>
> Key: HIVE-26158
> URL: https://issues.apache.org/jira/browse/HIVE-26158
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: tanghui
>Assignee: Zoltan Haindrich
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After the patch is updated, the partition table location and hdfs data
> directory are displayed normally, but the partition location of the table in
> the SDS in the Hive metabase is still displayed as the location of the old
> table, resulting in no data in the query partition.
>
> in beeline:
>
> set hive.create.as.external.legacy=true;
> CREATE TABLE part_test(
> c1 string
> ,c2 string
> )PARTITIONED BY (dat string)
> insert into part_test values ("11","th","20220101")
> insert into part_test values ("22","th","20220102")
> alter table part_test rename to part_test11;
> --this result is null.
> select * from part_test11 where dat="20220101";
> ||part_test.c1||part_test.c2||part_test.dat||
> | | | |
> -
> SDS in the Hive metabase:
> select SDS.LOCATION from TBLS,SDS where TBLS.TBL_NAME="part_test11" AND
> TBLS.TBL_ID=SDS.CD_ID;
> ---
> |*LOCATION*|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test11|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220101|
> |hdfs://nameservice1/warehouse/tablespace/external/hive/part_test/dat=20220102|
> ---
>
> We need to modify the partition location of the table in SDS to ensure that
> the query results are normal
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
