[jira] [Updated] (HIVE-17570) Fix view deletion related test failures (create_view.q etc)
[ https://issues.apache.org/jira/browse/HIVE-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17570: -- Status: Open (was: Patch Available) > Fix view deletion related test failures (create_view.q etc) > --- > > Key: HIVE-17570 > URL: https://issues.apache.org/jira/browse/HIVE-17570 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17570.1.patch, HIVE-17570.2.patch > > > Fixing the bug introduced by HIVE-17459. Sorry that did not capture that in a > timely fashion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Status: Patch Available (was: Open) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch, HIVE-17558.2.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Attachment: HIVE-17558.2.patch > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch, HIVE-17558.2.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Attachment: (was: HIVE-17558.2.patch) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Attachment: (was: HIVE-17558.2.patch) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch, HIVE-17558.2.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Attachment: HIVE-17558.2.patch > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch, HIVE-17558.2.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Status: Open (was: Patch Available) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch, HIVE-17558.2.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Status: Open (was: Patch Available) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch, HIVE-17558.2.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Attachment: HIVE-17558.2.patch > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch, HIVE-17558.2.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Status: Patch Available (was: Open) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch, HIVE-17558.2.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17556) The test udf_mask_hash.q is failing
[
https://issues.apache.org/jira/browse/HIVE-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173636#comment-16173636
]
Tao Li commented on HIVE-17556:
---
Thanks [~kuczoram] for the fix and [~kgyrtkirk] for the reivew.
> The test udf_mask_hash.q is failing
> ---
>
> Key: HIVE-17556
> URL: https://issues.apache.org/jira/browse/HIVE-17556
> Project: Hive
> Issue Type: Bug
> Components: Test
>Affects Versions: 3.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
> Attachments: HIVE-17556.1.patch
>
>
> After committing
> [HIVE-17515|https://issues.apache.org/jira/browse/HIVE-17515], the
> udf_mask_hash.q test is failing with the following error message:
> {noformat}
> Error Message
> Client Execution succeeded but contained differences (error code = 1) after
> executing udf_mask_hash.q
> 34c34
> < expressions:
> '8b44d559dc5d60e4453c9b4edf2a455fbce054bb8504cd3eb9b5f391bd239c90' (type:
> string)
> ---
> > expressions: 'dd78d68ad1b23bde126812482dd70ac6' (type: string)
> 36c36
> < Statistics: Num rows: 1 Data size: 148 Basic stats: COMPLETE
> Column stats: COMPLETE
> ---
> > Statistics: Num rows: 1 Data size: 116 Basic stats: COMPLETE
> > Column stats: COMPLETE
> 61c61
> < 8b44d559dc5d60e4453c9b4edf2a455fbce054bb8504cd3eb9b5f391bd239c90
> 8b44d559dc5d60e4453c9b4edf2a455fbce054bb8504cd3eb9b5f391bd239c90
> 30a88603135d3a6f7a66b4f9193da1ab4423aed45fb8fe736c2f2a08977f2bdd
>
>
> NULL NULL NULL NULL NULL
> ---
> > dd78d68ad1b23bde126812482dd70ac6 dd78d68ad1b23bde126812482dd70ac6
> > 835735ba20f1297683efca69fabd0fba
> >
> >
> > NULL NULL NULL NULL NULL
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17459) View deletion operation failed to replicate on target cluster
[ https://issues.apache.org/jira/browse/HIVE-17459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173628#comment-16173628 ] Tao Li commented on HIVE-17459: --- Working on HIVE-17570 to fix the bug. > View deletion operation failed to replicate on target cluster > - > > Key: HIVE-17459 > URL: https://issues.apache.org/jira/browse/HIVE-17459 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Fix For: 3.0.0 > > Attachments: HIVE-17459.1.patch, HIVE-17459.2.patch > > > View dropping is not replicated during incremental repl. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17570) Fix view deletion related test failures (create_view.q etc)
[ https://issues.apache.org/jira/browse/HIVE-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17570: -- Attachment: HIVE-17570.1.patch > Fix view deletion related test failures (create_view.q etc) > --- > > Key: HIVE-17570 > URL: https://issues.apache.org/jira/browse/HIVE-17570 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17570.1.patch > > > Fixing the bug introduced by HIVE-17459. Sorry that did not capture that in a > timely fashion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17570) Fix view deletion related test failures (create_view.q etc)
[ https://issues.apache.org/jira/browse/HIVE-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17570: -- Status: Patch Available (was: Open) > Fix view deletion related test failures (create_view.q etc) > --- > > Key: HIVE-17570 > URL: https://issues.apache.org/jira/browse/HIVE-17570 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17570.1.patch > > > Fixing the bug introduced by HIVE-17459. Sorry that did not capture that in a > timely fashion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17570) Fix view deletion related test failures (create_view.q etc)
[ https://issues.apache.org/jira/browse/HIVE-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li reassigned HIVE-17570: - > Fix view deletion related test failures (create_view.q etc) > --- > > Key: HIVE-17570 > URL: https://issues.apache.org/jira/browse/HIVE-17570 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > > Fixing the bug introduced by HIVE-17459. Sorry that did not capture that in a > timely fashion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17459) View deletion operation failed to replicate on target cluster
[ https://issues.apache.org/jira/browse/HIVE-17459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173341#comment-16173341 ] Tao Li commented on HIVE-17459: --- Taking a look. > View deletion operation failed to replicate on target cluster > - > > Key: HIVE-17459 > URL: https://issues.apache.org/jira/browse/HIVE-17459 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Fix For: 3.0.0 > > Attachments: HIVE-17459.1.patch, HIVE-17459.2.patch > > > View dropping is not replicated during incremental repl. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Attachment: HIVE-17558.1.patch > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Status: Patch Available (was: Open) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Attachment: (was: HIVE-17558.1.patch) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Status: Open (was: Patch Available) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Attachment: HIVE-17558.1.patch > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17558: -- Status: Patch Available (was: Open) > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17558.1.patch > > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171970#comment-16171970 ] Tao Li commented on HIVE-17558: --- cc [~daijy] > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17558) Skip non-native/temporary tables for constraint related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li reassigned HIVE-17558: - > Skip non-native/temporary tables for constraint related scenarios > - > > Key: HIVE-17558 > URL: https://issues.apache.org/jira/browse/HIVE-17558 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > > The change would be similar to HIVE-17422. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17515) Use SHA-256 for GenericUDFMaskHash to improve security
[ https://issues.apache.org/jira/browse/HIVE-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171896#comment-16171896 ] Tao Li commented on HIVE-17515: --- [~kgyrtkirk] Thanks for that fix! > Use SHA-256 for GenericUDFMaskHash to improve security > -- > > Key: HIVE-17515 > URL: https://issues.apache.org/jira/browse/HIVE-17515 > Project: Hive > Issue Type: Sub-task > Components: UDF >Reporter: Tao Li >Assignee: Tao Li > Fix For: 3.0.0 > > Attachments: HIVE-17515.1.patch > > > See HIVE-17226 for detailed description. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171135#comment-16171135 ] Tao Li commented on HIVE-17496: --- Test results look good now. > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch, > HIVE-17496.6.patch, HIVE-17496.7.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: HIVE-17496.7.patch > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch, > HIVE-17496.6.patch, HIVE-17496.7.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Patch Available (was: Open) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch, > HIVE-17496.6.patch, HIVE-17496.7.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[
https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170718#comment-16170718
]
Tao Li commented on HIVE-17496:
---
The cause to the deletion failure is because the table dir was deleted on
target db for some reason. We did not try to drop the table so not sure why the
table dir was deleted. See log below. This issue cannot be reproduced locally.
{noformat}
2017-09-18T12:49:20,642 ERROR [main] parse.TestReplicationScenarios: Error
verifying the staging dir deletion
java.io.FileNotFoundException: File
file:/home/hiveptest/130.211.162.127-hiveptest-0/apache-github-source-source/itests/hive-unit/target/warehouse/deleteStagingDir_org_apache_hadoop_hive_ql_parse_testreplicationscenarios_1505763609715_dupe.db/unptned
does not exist
{noformat}
So for now the fix is to add a existence check.
> Bootstrap repl is not cleaning up staging dirs
> --
>
> Key: HIVE-17496
> URL: https://issues.apache.org/jira/browse/HIVE-17496
> Project: Hive
> Issue Type: Bug
> Components: repl
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch,
> HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch, HIVE-17496.6.patch
>
>
> This will put more pressure on the HDFS file limit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: (was: HIVE-17496.7.patch) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch, HIVE-17496.6.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Open (was: Patch Available) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch, HIVE-17496.6.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Patch Available (was: Open) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch, > HIVE-17496.6.patch, HIVE-17496.7.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: HIVE-17496.7.patch > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch, > HIVE-17496.6.patch, HIVE-17496.7.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Open (was: Patch Available) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch, HIVE-17496.6.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[
https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170144#comment-16170144
]
Tao Li edited comment on HIVE-17496 at 9/18/17 3:28 PM:
According to the logs below, I did not see any error from HDFS deletion for the
testDeleteStagingDir test, according to change in HIVE-17496.5.patch. Looks
like the file should have been deleted successfully. So I don't know why the
test fails.
{noformat}
2017-09-18T02:25:52,399 INFO [main] ql.Context: Deleting scratch dir:
pfile:/home/hiveptest/104.198.168.233-hiveptest-0/apache-github-source-source/itests/hive-unit/target/warehouse/deletestagingdir_org_apache_hadoop_hive_ql_parse_testreplicationscenarios_1505726246743_dupe.db/unptned/.hive-staging_hive_2017-09-18_02-25-52_081_7805993483018006783-1
{noformat}
But in some other repl test logs, I do see some deletion that failed. It
indicates some possibility there.
{noformat}
2017-09-18T02:25:49,746 INFO [main] ql.Context: Deleting scratch dir:
file:/home/hiveptest/104.198.168.233-hiveptest-0/apache-github-source-source/itests/hive-unit/target/tmp/localscratchdir/6b15a44f-aec0-43f8-851d-81db07e4e329/hive_2017-09-18_02-25-49_687_551878613091072616-1/-mr-10001/.hive-staging_hive_2017-09-18_02-25-49_687_551878613091072616-1
2017-09-18T02:25:49,746 ERROR [main] ql.Context: File failes to be deleted when
removing Scratch (HIVE-17496)
{noformat}
was (Author: taoli-hwx):
According to the logs below, I did not see any error from HDFS deletion for the
testDeleteStagingDir test, according to change in HIVE-17496.5.patch. Looks
like the file should have been deleted successfully. So I don't know why the
test fails.
{noformat}
2017-09-18T02:25:52,399 INFO [main] ql.Context: Deleting scratch dir:
pfile:/home/hiveptest/104.198.168.233-hiveptest-0/apache-github-source-source/itests/hive-unit/target/warehouse/deletestagingdir_org_apache_hadoop_hive_ql_parse_testreplicationscenarios_1505726246743_dupe.db/unptned/.hive-staging_hive_2017-09-18_02-25-52_081_7805993483018006783-1
{noformat}
But in some other repl test logs, I do see some deletion that failed:
{noformat}
2017-09-18T02:25:49,746 INFO [main] ql.Context: Deleting scratch dir:
file:/home/hiveptest/104.198.168.233-hiveptest-0/apache-github-source-source/itests/hive-unit/target/tmp/localscratchdir/6b15a44f-aec0-43f8-851d-81db07e4e329/hive_2017-09-18_02-25-49_687_551878613091072616-1/-mr-10001/.hive-staging_hive_2017-09-18_02-25-49_687_551878613091072616-1
2017-09-18T02:25:49,746 ERROR [main] ql.Context: File failes to be deleted when
removing Scratch (HIVE-17496)
{noformat}
> Bootstrap repl is not cleaning up staging dirs
> --
>
> Key: HIVE-17496
> URL: https://issues.apache.org/jira/browse/HIVE-17496
> Project: Hive
> Issue Type: Bug
> Components: repl
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch,
> HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch, HIVE-17496.6.patch
>
>
> This will put more pressure on the HDFS file limit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: HIVE-17496.6.patch > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch, HIVE-17496.6.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Patch Available (was: Open) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch, HIVE-17496.6.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Open (was: Patch Available) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[
https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170144#comment-16170144
]
Tao Li commented on HIVE-17496:
---
According to the logs below, I did not see any error from HDFS deletion for the
testDeleteStagingDir test, according to change in HIVE-17496.5.patch. Looks
like the file should have been deleted successfully. So I don't know why the
test fails.
{noformat}
2017-09-18T02:25:52,399 INFO [main] ql.Context: Deleting scratch dir:
pfile:/home/hiveptest/104.198.168.233-hiveptest-0/apache-github-source-source/itests/hive-unit/target/warehouse/deletestagingdir_org_apache_hadoop_hive_ql_parse_testreplicationscenarios_1505726246743_dupe.db/unptned/.hive-staging_hive_2017-09-18_02-25-52_081_7805993483018006783-1
{noformat}
But in some other repl test logs, I do see some deletion that failed:
{noformat}
2017-09-18T02:25:49,746 INFO [main] ql.Context: Deleting scratch dir:
file:/home/hiveptest/104.198.168.233-hiveptest-0/apache-github-source-source/itests/hive-unit/target/tmp/localscratchdir/6b15a44f-aec0-43f8-851d-81db07e4e329/hive_2017-09-18_02-25-49_687_551878613091072616-1/-mr-10001/.hive-staging_hive_2017-09-18_02-25-49_687_551878613091072616-1
2017-09-18T02:25:49,746 ERROR [main] ql.Context: File failes to be deleted when
removing Scratch (HIVE-17496)
{noformat}
> Bootstrap repl is not cleaning up staging dirs
> --
>
> Key: HIVE-17496
> URL: https://issues.apache.org/jira/browse/HIVE-17496
> Project: Hive
> Issue Type: Bug
> Components: repl
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch,
> HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch
>
>
> This will put more pressure on the HDFS file limit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Open (was: Patch Available) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: (was: HIVE-17496.5.patch) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: HIVE-17496.5.patch > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Patch Available (was: Open) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Patch Available (was: Open) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Open (was: Patch Available) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: HIVE-17496.5.patch > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch, HIVE-17496.5.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169445#comment-16169445 ] Tao Li commented on HIVE-17422: --- Test resut looks good. > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Attachment: HIVE-17422.4.patch > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Status: Patch Available (was: Open) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Attachment: (was: HIVE-17422.4.patch) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Status: Open (was: Patch Available) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Status: Patch Available (was: Open) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Attachment: HIVE-17422.4.patch > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Status: Open (was: Patch Available) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch, HIVE-17422.4.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Status: Patch Available (was: Open) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Attachment: HIVE-17422.3.patch > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Status: Open (was: Patch Available) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch, > HIVE-17422.3.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168298#comment-16168298 ] Tao Li commented on HIVE-17422: --- Fixing the test failure "testNoopReplEximCommands". > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Patch Available (was: Open) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: HIVE-17496.4.patch > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: (was: HIVE-17496.4.patch) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Open (was: Patch Available) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Patch Available (was: Open) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: HIVE-17496.4.patch > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch, HIVE-17496.4.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Open (was: Patch Available) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: HIVE-17496.4.patch > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: (was: HIVE-17496.4.patch) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Open (was: Patch Available) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Patch Available (was: Open) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Status: Patch Available (was: Open) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Status: Open (was: Patch Available) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17522) cleanup old 'repl dump' dirs
[ https://issues.apache.org/jira/browse/HIVE-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165617#comment-16165617 ] Tao Li commented on HIVE-17522: --- Test looks good (failures are unrelated). [~daijy] Can you please take a look at this change? > cleanup old 'repl dump' dirs > > > Key: HIVE-17522 > URL: https://issues.apache.org/jira/browse/HIVE-17522 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17522.1.patch > > > We want to clean up the old dump dirs to save space and reduce scan time when > needed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17514) Use SHA-256 for cookie signer to improve security
[ https://issues.apache.org/jira/browse/HIVE-17514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165573#comment-16165573 ] Tao Li commented on HIVE-17514: --- Tests look good. [~thejas] Can you please take a look at this change? > Use SHA-256 for cookie signer to improve security > - > > Key: HIVE-17514 > URL: https://issues.apache.org/jira/browse/HIVE-17514 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17514.1.patch > > > See HIVE-17226 for detailed description. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17515) Use SHA-256 for GenericUDFMaskHash to improve security
[ https://issues.apache.org/jira/browse/HIVE-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165567#comment-16165567 ] Tao Li commented on HIVE-17515: --- Tests look good. [~thejas] Can you please take a look? > Use SHA-256 for GenericUDFMaskHash to improve security > -- > > Key: HIVE-17515 > URL: https://issues.apache.org/jira/browse/HIVE-17515 > Project: Hive > Issue Type: Sub-task > Components: UDF >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17515.1.patch > > > See HIVE-17226 for detailed description. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17515) Use SHA-256 for GenericUDFMaskHash to improve security
[ https://issues.apache.org/jira/browse/HIVE-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165567#comment-16165567 ] Tao Li edited comment on HIVE-17515 at 9/14/17 1:05 AM: Tests look good. [~thejas] Can you please take a look at this change? was (Author: taoli-hwx): Tests look good. [~thejas] Can you please take a look? > Use SHA-256 for GenericUDFMaskHash to improve security > -- > > Key: HIVE-17515 > URL: https://issues.apache.org/jira/browse/HIVE-17515 > Project: Hive > Issue Type: Sub-task > Components: UDF >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17515.1.patch > > > See HIVE-17226 for detailed description. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Open (was: Patch Available) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Patch Available (was: Open) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: HIVE-17496.3.patch > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Attachment: (was: HIVE-17496.3.patch) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Status: Patch Available (was: Open) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17513) Refactor PathUtils to not contain instance fields
[ https://issues.apache.org/jira/browse/HIVE-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165157#comment-16165157 ] Tao Li commented on HIVE-17513: --- Test result looks good. > Refactor PathUtils to not contain instance fields > - > > Key: HIVE-17513 > URL: https://issues.apache.org/jira/browse/HIVE-17513 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li >Priority: Minor > Attachments: HIVE-17513.1.patch > > > This util class should just provide the static helper methods. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164809#comment-16164809 ] Tao Li commented on HIVE-17496: --- Investigating the "testDeleteStagingDir" failure which is the added test for the staging dir cleanup. > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17522) cleanup old 'repl dump' dirs
[ https://issues.apache.org/jira/browse/HIVE-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17522: -- Status: Patch Available (was: Open) > cleanup old 'repl dump' dirs > > > Key: HIVE-17522 > URL: https://issues.apache.org/jira/browse/HIVE-17522 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17522.1.patch > > > We want to clean up the old dump dirs to save space and reduce scan time when > needed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17522) cleanup old 'repl dump' dirs
[ https://issues.apache.org/jira/browse/HIVE-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17522: -- Attachment: HIVE-17522.1.patch > cleanup old 'repl dump' dirs > > > Key: HIVE-17522 > URL: https://issues.apache.org/jira/browse/HIVE-17522 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17522.1.patch > > > We want to clean up the old dump dirs to save space and reduce scan time when > needed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17522) cleanup old 'repl dump' dirs
[ https://issues.apache.org/jira/browse/HIVE-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li reassigned HIVE-17522: - > cleanup old 'repl dump' dirs > > > Key: HIVE-17522 > URL: https://issues.apache.org/jira/browse/HIVE-17522 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > > We want to clean up the old dump dirs to save space and reduce scan time when > needed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Attachment: HIVE-17422.2.patch > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch, HIVE-17422.2.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Attachment: HIVE-17422.1.patch > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17422.1.patch > > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Summary: Skip non-native/temporary tables for all major table related scenarios (was: Don't dump non-native/temporary tables during incremental dump) > Skip non-native/temporary tables for all major table related scenarios > -- > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Skip non-native/temporary tables for all major table/partition related scenarios
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Summary: Skip non-native/temporary tables for all major table/partition related scenarios (was: Skip non-native/temporary tables for all major table related scenarios) > Skip non-native/temporary tables for all major table/partition related > scenarios > > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17422) Don't dump non-native/temporary tables during incremental dump
[ https://issues.apache.org/jira/browse/HIVE-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17422: -- Description: Currently during incremental dump, the non-native/temporary table info is partially dumped in metadata file and will be ignored later by the repl load. We can optimize it by moving the check (whether the table should be exported or not) earlier so that we don't save any info to dump file for such types of tables. CreateTableHandler already has this optimization, so we just need to apply similar logic to other scenarios. The change is to apply the EximUtil.shouldExportTable check to all scenarios (e.g. alter table) that calls into the common dump method. was:Currently during incremental dump, the non-native/temporary table info is partially dumped in metadata file and will be ignored later by the repl load. We can optimize it by moving the check (whether the table should be exported or not) earlier so that we don't save any info to dump file for such types of tables. CreateTableHandler already has this optimization, so we just need to apply similar logic to other scenarios. > Don't dump non-native/temporary tables during incremental dump > -- > > Key: HIVE-17422 > URL: https://issues.apache.org/jira/browse/HIVE-17422 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li > > Currently during incremental dump, the non-native/temporary table info is > partially dumped in metadata file and will be ignored later by the repl load. > We can optimize it by moving the check (whether the table should be exported > or not) earlier so that we don't save any info to dump file for such types of > tables. CreateTableHandler already has this optimization, so we just need to > apply similar logic to other scenarios. > The change is to apply the EximUtil.shouldExportTable check to all scenarios > (e.g. alter table) that calls into the common dump method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17513) Refactor PathUtils to not contain instance fields
[ https://issues.apache.org/jira/browse/HIVE-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17513: -- Status: Patch Available (was: Open) > Refactor PathUtils to not contain instance fields > - > > Key: HIVE-17513 > URL: https://issues.apache.org/jira/browse/HIVE-17513 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li >Priority: Minor > Attachments: HIVE-17513.1.patch > > > This util class should just provide the static helper methods. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17513) Refactor PathUtils to not contain instance fields
[ https://issues.apache.org/jira/browse/HIVE-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17513: -- Attachment: HIVE-17513.1.patch > Refactor PathUtils to not contain instance fields > - > > Key: HIVE-17513 > URL: https://issues.apache.org/jira/browse/HIVE-17513 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li >Priority: Minor > Attachments: HIVE-17513.1.patch > > > This util class should just provide the static helper methods. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17513) Refactor PathUtils to not contain instance fields
[ https://issues.apache.org/jira/browse/HIVE-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17513: -- Summary: Refactor PathUtils to not contain instance fields (was: Refactor PathUtils to not contain state (instance fields)) > Refactor PathUtils to not contain instance fields > - > > Key: HIVE-17513 > URL: https://issues.apache.org/jira/browse/HIVE-17513 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Tao Li >Assignee: Tao Li >Priority: Minor > > This util class should just provide the static helper methods. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Patch Available (was: Open) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17496) Bootstrap repl is not cleaning up staging dirs
[ https://issues.apache.org/jira/browse/HIVE-17496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17496: -- Status: Open (was: Patch Available) > Bootstrap repl is not cleaning up staging dirs > -- > > Key: HIVE-17496 > URL: https://issues.apache.org/jira/browse/HIVE-17496 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17496.1.patch, HIVE-17496.2.patch, > HIVE-17496.3.patch > > > This will put more pressure on the HDFS file limit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17515) Use SHA-256 for GenericUDFMaskHash to improve security
[ https://issues.apache.org/jira/browse/HIVE-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17515: -- Status: Patch Available (was: Open) > Use SHA-256 for GenericUDFMaskHash to improve security > -- > > Key: HIVE-17515 > URL: https://issues.apache.org/jira/browse/HIVE-17515 > Project: Hive > Issue Type: Sub-task > Components: UDF >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17515.1.patch > > > See HIVE-17226 for detailed description. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17514) Use SHA-256 for cookie signer to improve security
[ https://issues.apache.org/jira/browse/HIVE-17514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17514: -- Attachment: HIVE-17514.1.patch > Use SHA-256 for cookie signer to improve security > - > > Key: HIVE-17514 > URL: https://issues.apache.org/jira/browse/HIVE-17514 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17514.1.patch > > > See HIVE-17226 for detailed description. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17514) Use SHA-256 for cookie signer to improve security
[ https://issues.apache.org/jira/browse/HIVE-17514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17514: -- Status: Patch Available (was: Open) > Use SHA-256 for cookie signer to improve security > - > > Key: HIVE-17514 > URL: https://issues.apache.org/jira/browse/HIVE-17514 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17514.1.patch > > > See HIVE-17226 for detailed description. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17515) Use SHA-256 for GenericUDFMaskHash to improve security
[ https://issues.apache.org/jira/browse/HIVE-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17515: -- Attachment: HIVE-17515.1.patch > Use SHA-256 for GenericUDFMaskHash to improve security > -- > > Key: HIVE-17515 > URL: https://issues.apache.org/jira/browse/HIVE-17515 > Project: Hive > Issue Type: Sub-task > Components: UDF >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-17515.1.patch > > > See HIVE-17226 for detailed description. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17515) Use SHA-256 for GenericUDFMaskHash to improve security
[ https://issues.apache.org/jira/browse/HIVE-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-17515: -- Description: See HIVE-17226 for detailed description. > Use SHA-256 for GenericUDFMaskHash to improve security > -- > > Key: HIVE-17515 > URL: https://issues.apache.org/jira/browse/HIVE-17515 > Project: Hive > Issue Type: Sub-task > Components: UDF >Reporter: Tao Li >Assignee: Tao Li > > See HIVE-17226 for detailed description. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
