[jira] [Commented] (HIVE-17367) IMPORT table doesn't load from data dump if a metadata-only dump was already imported.
[ https://issues.apache.org/jira/browse/HIVE-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149952#comment-16149952 ] Sankar Hariappan commented on HIVE-17367: - Thanks [~anishek] for the review and [~thejas] for the commit! > IMPORT table doesn't load from data dump if a metadata-only dump was already > imported. > -- > > Key: HIVE-17367 > URL: https://issues.apache.org/jira/browse/HIVE-17367 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Import/Export, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17367.01.patch, HIVE-17367.02.patch, > HIVE-17367.03.patch > > > Repl v1 creates a set of EXPORT/IMPORT commands to replicate modified data > (as per events) across clusters. > For instance, let's say, insert generates 2 events such as > ALTER_TABLE (ID: 10) > INSERT (ID: 11) > Each event generates a set of EXPORT and IMPORT commands. > ALTER_TABLE event generates metadata only export/import > INSERT generates metadata+data export/import. > As Hive always dump the latest copy of table during export, it sets the > latest notification event ID as current state of it. So, in this example, > import of metadata by ALTER_TABLE event sets the current state of the table > as 11. > Now, when we try to import the data dumped by INSERT event, it is noop as the > table's current state(11) is equal to the dump state (11) which in-turn leads > to the data never gets replicated to target cluster. > So, it is necessary to allow overwrite of table/partition if their current > state equals the dump state. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17367) IMPORT table doesn't load from data dump if a metadata-only dump was already imported.
[ https://issues.apache.org/jira/browse/HIVE-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149729#comment-16149729 ] Thejas M Nair commented on HIVE-17367: -- +1 > IMPORT table doesn't load from data dump if a metadata-only dump was already > imported. > -- > > Key: HIVE-17367 > URL: https://issues.apache.org/jira/browse/HIVE-17367 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Import/Export, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17367.01.patch, HIVE-17367.02.patch, > HIVE-17367.03.patch > > > Repl v1 creates a set of EXPORT/IMPORT commands to replicate modified data > (as per events) across clusters. > For instance, let's say, insert generates 2 events such as > ALTER_TABLE (ID: 10) > INSERT (ID: 11) > Each event generates a set of EXPORT and IMPORT commands. > ALTER_TABLE event generates metadata only export/import > INSERT generates metadata+data export/import. > As Hive always dump the latest copy of table during export, it sets the > latest notification event ID as current state of it. So, in this example, > import of metadata by ALTER_TABLE event sets the current state of the table > as 11. > Now, when we try to import the data dumped by INSERT event, it is noop as the > table's current state(11) is equal to the dump state (11) which in-turn leads > to the data never gets replicated to target cluster. > So, it is necessary to allow overwrite of table/partition if their current > state equals the dump state. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17367) IMPORT table doesn't load from data dump if a metadata-only dump was already imported.
[
https://issues.apache.org/jira/browse/HIVE-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149684#comment-16149684
]
Hive QA commented on HIVE-17367:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12884766/HIVE-17367.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11021 tests
executed
*Failed tests:*
{noformat}
TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out)
(batchId=280)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_ppd]
(batchId=84)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6624/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6624/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6624/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12884766 - PreCommit-HIVE-Build
> IMPORT table doesn't load from data dump if a metadata-only dump was already
> imported.
> --
>
> Key: HIVE-17367
> URL: https://issues.apache.org/jira/browse/HIVE-17367
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Import/Export, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17367.01.patch, HIVE-17367.02.patch,
> HIVE-17367.03.patch
>
>
> Repl v1 creates a set of EXPORT/IMPORT commands to replicate modified data
> (as per events) across clusters.
> For instance, let's say, insert generates 2 events such as
> ALTER_TABLE (ID: 10)
> INSERT (ID: 11)
> Each event generates a set of EXPORT and IMPORT commands.
> ALTER_TABLE event generates metadata only export/import
> INSERT generates metadata+data export/import.
> As Hive always dump the latest copy of table during export, it sets the
> latest notification event ID as current state of it. So, in this example,
> import of metadata by ALTER_TABLE event sets the current state of the table
> as 11.
> Now, when we try to import the data dumped by INSERT event, it is noop as the
> table's current state(11) is equal to the dump state (11) which in-turn leads
> to the data never gets replicated to target cluster.
> So, it is necessary to allow overwrite of table/partition if their current
> state equals the dump state.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17367) IMPORT table doesn't load from data dump if a metadata-only dump was already imported.
[ https://issues.apache.org/jira/browse/HIVE-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149516#comment-16149516 ] anishek commented on HIVE-17367: +1 cc [~thejas]/[~daijy] > IMPORT table doesn't load from data dump if a metadata-only dump was already > imported. > -- > > Key: HIVE-17367 > URL: https://issues.apache.org/jira/browse/HIVE-17367 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Import/Export, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17367.01.patch, HIVE-17367.02.patch, > HIVE-17367.03.patch > > > Repl v1 creates a set of EXPORT/IMPORT commands to replicate modified data > (as per events) across clusters. > For instance, let's say, insert generates 2 events such as > ALTER_TABLE (ID: 10) > INSERT (ID: 11) > Each event generates a set of EXPORT and IMPORT commands. > ALTER_TABLE event generates metadata only export/import > INSERT generates metadata+data export/import. > As Hive always dump the latest copy of table during export, it sets the > latest notification event ID as current state of it. So, in this example, > import of metadata by ALTER_TABLE event sets the current state of the table > as 11. > Now, when we try to import the data dumped by INSERT event, it is noop as the > table's current state(11) is equal to the dump state (11) which in-turn leads > to the data never gets replicated to target cluster. > So, it is necessary to allow overwrite of table/partition if their current > state equals the dump state. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17367) IMPORT table doesn't load from data dump if a metadata-only dump was already imported.
[ https://issues.apache.org/jira/browse/HIVE-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149237#comment-16149237 ] anishek commented on HIVE-17367: [~sankarh]can you rebase it, does not apply cleanly currently. Also can you provide a pull request for the same please. one initial comment, The replicationSpec should not be updating the currentEventId in TableExport, all of that should be done before we call table export, for replication that is done already in RepldumpTask for export this should be done in ExportSemanticAnalyzer and not in TableExport. > IMPORT table doesn't load from data dump if a metadata-only dump was already > imported. > -- > > Key: HIVE-17367 > URL: https://issues.apache.org/jira/browse/HIVE-17367 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Import/Export, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17367.01.patch, HIVE-17367.02.patch > > > Repl v1 creates a set of EXPORT/IMPORT commands to replicate modified data > (as per events) across clusters. > For instance, let's say, insert generates 2 events such as > ALTER_TABLE (ID: 10) > INSERT (ID: 11) > Each event generates a set of EXPORT and IMPORT commands. > ALTER_TABLE event generates metadata only export/import > INSERT generates metadata+data export/import. > As Hive always dump the latest copy of table during export, it sets the > latest notification event ID as current state of it. So, in this example, > import of metadata by ALTER_TABLE event sets the current state of the table > as 11. > Now, when we try to import the data dumped by INSERT event, it is noop as the > table's current state(11) is equal to the dump state (11) which in-turn leads > to the data never gets replicated to target cluster. > So, it is necessary to allow overwrite of table/partition if their current > state equals the dump state. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17367) IMPORT table doesn't load from data dump if a metadata-only dump was already imported.
[
https://issues.apache.org/jira/browse/HIVE-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142152#comment-16142152
]
Hive QA commented on HIVE-17367:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12883786/HIVE-17367.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11005 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=231)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreWithEnvironmentContext.testEnvironmentContext
(batchId=209)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout
(batchId=228)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6543/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6543/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6543/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12883786 - PreCommit-HIVE-Build
> IMPORT table doesn't load from data dump if a metadata-only dump was already
> imported.
> --
>
> Key: HIVE-17367
> URL: https://issues.apache.org/jira/browse/HIVE-17367
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Import/Export, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17367.01.patch, HIVE-17367.02.patch
>
>
> Repl v1 creates a set of EXPORT/IMPORT commands to replicate modified data
> (as per events) across clusters.
> For instance, let's say, insert generates 2 events such as
> ALTER_TABLE (ID: 10)
> INSERT (ID: 11)
> Each event generates a set of EXPORT and IMPORT commands.
> ALTER_TABLE event generates metadata only export/import
> INSERT generates metadata+data export/import.
> As Hive always dump the latest copy of table during export, it sets the
> latest notification event ID as current state of it. So, in this example,
> import of metadata by ALTER_TABLE event sets the current state of the table
> as 11.
> Now, when we try to import the data dumped by INSERT event, it is noop as the
> table's current state(11) is equal to the dump state (11) which in-turn leads
> to the data never gets replicated to target cluster.
> So, it is necessary to allow overwrite of table/partition if their current
> state equals the dump state.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17367) IMPORT table doesn't load from data dump if a metadata-only dump was already imported.
[
https://issues.apache.org/jira/browse/HIVE-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141376#comment-16141376
]
Hive QA commented on HIVE-17367:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12883632/HIVE-17367.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 11001 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6534/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6534/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6534/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12883632 - PreCommit-HIVE-Build
> IMPORT table doesn't load from data dump if a metadata-only dump was already
> imported.
> --
>
> Key: HIVE-17367
> URL: https://issues.apache.org/jira/browse/HIVE-17367
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Import/Export, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17367.01.patch
>
>
> Repl v1 creates a set of EXPORT/IMPORT commands to replicate modified data
> (as per events) across clusters.
> For instance, let's say, insert generates 2 events such as
> ALTER_TABLE (ID: 10)
> INSERT (ID: 11)
> Each event generates a set of EXPORT and IMPORT commands.
> ALTER_TABLE event generates metadata only export/import
> INSERT generates metadata+data export/import.
> As Hive always dump the latest copy of table during export, it sets the
> latest notification event ID as current state of it. So, in this example,
> import of metadata by ALTER_TABLE event sets the current state of the table
> as 11.
> Now, when we try to import the data dumped by INSERT event, it is noop as the
> table's current state(11) is equal to the dump state (11) which in-turn leads
> to the data never gets replicated to target cluster.
> So, it is necessary to allow overwrite of table/partition if their current
> state equals the dump state.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
