[jira] [Commented] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[ https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129106#comment-16129106 ] Mithun Radhakrishnan commented on HIVE-17181: - As established in [HIVE-8472|https://issues.apache.org/jira/browse/HIVE-8472?focusedCommentId=16128283=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16128283], these tests seem busted on {{branch-2}}. The failures are in tests that do not use {{HCatOutputFormat}}. > HCatOutputFormat should expose complete output-schema (including > partition-keys) for dynamic-partitioning MR jobs > - > > Key: HIVE-17181 > URL: https://issues.apache.org/jira/browse/HIVE-17181 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 2.2.0, 3.0.0 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Attachments: HIVE-17181.1-branch-2.patch, HIVE-17181.1.patch, > HIVE-17181.2.patch, HIVE-17181.3.patch > > > Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic > partitioning are expected to call the following API methods: > # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to > write to. This call populates the {{OutputJobInfo}} with details fetched from > the Metastore. > # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data > being written. > It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows: > {code:java} > HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf)); > {code} > Unfortunately, {{getTableSchema()}} returns only the record-schema, not the > entire table's schema. We'll need a better API for use in M/R jobs to get the > complete table-schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[ https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128256#comment-16128256 ] Hive QA commented on HIVE-17181: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12882046/HIVE-17181.1-branch-2.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10584 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explaindenpendencydiffengs] (batchId=38) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=142) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] (batchId=139) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=144) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=102) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[explaindenpendencydiffengs] (batchId=115) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf] (batchId=125) org.apache.hadoop.hive.ql.security.TestExtendedAcls.testPartition (batchId=228) org.apache.hadoop.hive.ql.security.TestFolderPermissions.testPartition (batchId=217) org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=176) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6411/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6411/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6411/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 11 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12882046 - PreCommit-HIVE-Build > HCatOutputFormat should expose complete output-schema (including > partition-keys) for dynamic-partitioning MR jobs > - > > Key: HIVE-17181 > URL: https://issues.apache.org/jira/browse/HIVE-17181 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 2.2.0, 3.0.0 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Attachments: HIVE-17181.1-branch-2.patch, HIVE-17181.1.patch, > HIVE-17181.2.patch, HIVE-17181.3.patch > > > Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic > partitioning are expected to call the following API methods: > # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to > write to. This call populates the {{OutputJobInfo}} with details fetched from > the Metastore. > # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data > being written. > It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows: > {code:java} > HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf)); > {code} > Unfortunately, {{getTableSchema()}} returns only the record-schema, not the > entire table's schema. We'll need a better API for use in M/R jobs to get the > complete table-schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[ https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127642#comment-16127642 ] Mithun Radhakrishnan commented on HIVE-17181: - Yes, sir. I'm lining the commits up right now. I'd like to repeat the {{branch-2}} tests before I commit there. > HCatOutputFormat should expose complete output-schema (including > partition-keys) for dynamic-partitioning MR jobs > - > > Key: HIVE-17181 > URL: https://issues.apache.org/jira/browse/HIVE-17181 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 2.2.0, 3.0.0 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Attachments: HIVE-17181.1.patch, HIVE-17181.2.patch, > HIVE-17181.3.patch, HIVE-17181.branch-2.patch > > > Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic > partitioning are expected to call the following API methods: > # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to > write to. This call populates the {{OutputJobInfo}} with details fetched from > the Metastore. > # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data > being written. > It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows: > {code:java} > HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf)); > {code} > Unfortunately, {{getTableSchema()}} returns only the record-schema, not the > entire table's schema. We'll need a better API for use in M/R jobs to get the > complete table-schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[ https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126836#comment-16126836 ] Thejas M Nair commented on HIVE-17181: -- +1 to latest patch Please go ahead and commit > HCatOutputFormat should expose complete output-schema (including > partition-keys) for dynamic-partitioning MR jobs > - > > Key: HIVE-17181 > URL: https://issues.apache.org/jira/browse/HIVE-17181 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 2.2.0, 3.0.0 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Attachments: HIVE-17181.1.patch, HIVE-17181.2.patch, > HIVE-17181.3.patch, HIVE-17181.branch-2.patch > > > Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic > partitioning are expected to call the following API methods: > # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to > write to. This call populates the {{OutputJobInfo}} with details fetched from > the Metastore. > # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data > being written. > It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows: > {code:java} > HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf)); > {code} > Unfortunately, {{getTableSchema()}} returns only the record-schema, not the > entire table's schema. We'll need a better API for use in M/R jobs to get the > complete table-schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[ https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126779#comment-16126779 ] Mithun Radhakrishnan commented on HIVE-17181: - Hey, [~thejas]. Does the latest version of this patch look better? The test failures seem once again to be those being handled in HIVE-16908 and HIVE-15058. > HCatOutputFormat should expose complete output-schema (including > partition-keys) for dynamic-partitioning MR jobs > - > > Key: HIVE-17181 > URL: https://issues.apache.org/jira/browse/HIVE-17181 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 2.2.0, 3.0.0 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Attachments: HIVE-17181.1.patch, HIVE-17181.2.patch, > HIVE-17181.3.patch, HIVE-17181.branch-2.patch > > > Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic > partitioning are expected to call the following API methods: > # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to > write to. This call populates the {{OutputJobInfo}} with details fetched from > the Metastore. > # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data > being written. > It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows: > {code:java} > HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf)); > {code} > Unfortunately, {{getTableSchema()}} returns only the record-schema, not the > entire table's schema. We'll need a better API for use in M/R jobs to get the > complete table-schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[ https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117120#comment-16117120 ] Hive QA commented on HIVE-17181: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12880670/HIVE-17181.3.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10994 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed] (batchId=56) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=99) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=180) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout (batchId=228) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6281/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6281/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6281/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12880670 - PreCommit-HIVE-Build > HCatOutputFormat should expose complete output-schema (including > partition-keys) for dynamic-partitioning MR jobs > - > > Key: HIVE-17181 > URL: https://issues.apache.org/jira/browse/HIVE-17181 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 2.2.0, 3.0.0 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Attachments: HIVE-17181.1.patch, HIVE-17181.2.patch, > HIVE-17181.3.patch, HIVE-17181.branch-2.patch > > > Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic > partitioning are expected to call the following API methods: > # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to > write to. This call populates the {{OutputJobInfo}} with details fetched from > the Metastore. > # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data > being written. > It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows: > {code:java} > HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf)); > {code} > Unfortunately, {{getTableSchema()}} returns only the record-schema, not the > entire table's schema. We'll need a better API for use in M/R jobs to get the > complete table-schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[ https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115307#comment-16115307 ] Hive QA commented on HIVE-17181: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12880482/HIVE-17181.2.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10990 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed] (batchId=56) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=168) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=99) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=179) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=179) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=179) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6269/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6269/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6269/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12880482 - PreCommit-HIVE-Build > HCatOutputFormat should expose complete output-schema (including > partition-keys) for dynamic-partitioning MR jobs > - > > Key: HIVE-17181 > URL: https://issues.apache.org/jira/browse/HIVE-17181 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 2.2.0 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Attachments: HIVE-17181.1.patch, HIVE-17181.2.patch, > HIVE-17181.branch-2.patch > > > Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic > partitioning are expected to call the following API methods: > # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to > write to. This call populates the {{OutputJobInfo}} with details fetched from > the Metastore. > # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data > being written. > It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows: > {code:java} > HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf)); > {code} > Unfortunately, {{getTableSchema()}} returns only the record-schema, not the > entire table's schema. We'll need a better API for use in M/R jobs to get the > complete table-schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[ https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115151#comment-16115151 ] Thejas M Nair commented on HIVE-17181: -- Thanks for the test Mithun. Won't junit fail method stop further query execution ? Will line 214 below get executed ? {code} 212 catch (Exception unexpected) { 213 fail("Unexpected failure! " + unexpected.getMessage()); 214 unexpected.printStackTrace(); {code} One option seems to be just letting that exception happen - https://stackoverflow.com/questions/16596418/how-to-handle-exceptions-in-junit > HCatOutputFormat should expose complete output-schema (including > partition-keys) for dynamic-partitioning MR jobs > - > > Key: HIVE-17181 > URL: https://issues.apache.org/jira/browse/HIVE-17181 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 2.2.0 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Attachments: HIVE-17181.1.patch, HIVE-17181.2.patch, > HIVE-17181.branch-2.patch > > > Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic > partitioning are expected to call the following API methods: > # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to > write to. This call populates the {{OutputJobInfo}} with details fetched from > the Metastore. > # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data > being written. > It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows: > {code:java} > HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf)); > {code} > Unfortunately, {{getTableSchema()}} returns only the record-schema, not the > entire table's schema. We'll need a better API for use in M/R jobs to get the > complete table-schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[ https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114791#comment-16114791 ] Thejas M Nair commented on HIVE-17181: -- The change looks good to me. Can you also please add a unit test ? > HCatOutputFormat should expose complete output-schema (including > partition-keys) for dynamic-partitioning MR jobs > - > > Key: HIVE-17181 > URL: https://issues.apache.org/jira/browse/HIVE-17181 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 2.2.0 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Attachments: HIVE-17181.1.patch, HIVE-17181.branch-2.patch > > > Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic > partitioning are expected to call the following API methods: > # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to > write to. This call populates the {{OutputJobInfo}} with details fetched from > the Metastore. > # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data > being written. > It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows: > {code:java} > HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf)); > {code} > Unfortunately, {{getTableSchema()}} returns only the record-schema, not the > entire table's schema. We'll need a better API for use in M/R jobs to get the > complete table-schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[ https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103062#comment-16103062 ] Hive QA commented on HIVE-17181: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12879104/HIVE-17181.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11006 tests executed *Failed tests:* {noformat} TestPerfCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite] (batchId=240) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=144) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning] (batchId=168) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=168) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=99) org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver (batchId=242) org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge (batchId=206) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=179) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=179) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=179) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6147/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6147/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6147/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12879104 - PreCommit-HIVE-Build > HCatOutputFormat should expose complete output-schema (including > partition-keys) for dynamic-partitioning MR jobs > - > > Key: HIVE-17181 > URL: https://issues.apache.org/jira/browse/HIVE-17181 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 2.2.0 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Attachments: HIVE-17181.1.patch, HIVE-17181.branch-2.patch > > > Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic > partitioning are expected to call the following API methods: > # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to > write to. This call populates the {{OutputJobInfo}} with details fetched from > the Metastore. > # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data > being written. > It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows: > {code:java} > HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf)); > {code} > Unfortunately, {{getTableSchema()}} returns only the record-schema, not the > entire table's schema. We'll need a better API for use in M/R jobs to get the > complete table-schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)