[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143309#comment-16143309 ] Mithun Radhakrishnan commented on HIVE-16908: - Thanks for checking this patch in in my absence, [~pvary]. We are agreed that this solution is not ideal. I look forward to reviewing HIVE-17384, or contributing to it, if time permits. > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Mithun Radhakrishnan > Fix For: 3.0.0 > > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch, > HIVE-16908.3.patch, HIVE-16908.4.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138358#comment-16138358 ] Peter Vary commented on HIVE-16908: --- The failing tests are not related: - TestBeeLineDriver.testCliDriver create_merge_compressed is caused by HIVE-17370: Some tests are failing with java.io.FileNotFoundException: File file:/tmp/hadoop/mapred/... - TestMiniSparkOnYarnCliDriver.testCliDriver spark_vectorized_dynamic_partition_pruning - HIVE-17122 spark_vectorized_dynamic_partition_pruning.q is continuously failing - TestFileUtils are caused by HIVE-17327 LLAP IO: restrict native file ID usage to default FS to avoid hypothetical collisions when HDFS federation is used - The other two are known flaky > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Mithun Radhakrishnan > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch, > HIVE-16908.3.patch, HIVE-16908.4.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138225#comment-16138225 ] Hive QA commented on HIVE-16908: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12883297/HIVE-16908.4.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10994 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed] (batchId=240) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.common.TestFileUtils.testCopyWithDistCpAs (batchId=250) org.apache.hadoop.hive.common.TestFileUtils.testCopyWithDistcp (batchId=250) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6500/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6500/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6500/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12883297 - PreCommit-HIVE-Build > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Mithun Radhakrishnan > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch, > HIVE-16908.3.patch, HIVE-16908.4.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137390#comment-16137390 ] Peter Vary commented on HIVE-16908: --- I think then the "proper" way to do this to assign this jira to [~mithun], resubmit his patch, and create a followup jira to tackle the remaining problems later. Do we agree here? Thanks, Peter > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch, > HIVE-16908.3.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137364#comment-16137364 ] Sunitha Beeram commented on HIVE-16908: --- [~pvary] sorry about the delay on this. Its going to take a bit more time from my end to get these fixed correctly, but in the meantime we can commit [~mithun]'s patch - he is the original author of the tests and I am comfortable taking in his fixes for it. On a more fundamental level though, we need to fix the tests that launch multiple metastores in the same JVM (and there are more tests than these that do that) - so we can tackle that sepearately. > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch, > HIVE-16908.3.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136855#comment-16136855 ] Peter Vary commented on HIVE-16908: --- I propose to disable the tests until this is solved. See: HIVE-17369. What do you think? Thanks, Peter > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch, > HIVE-16908.3.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111334#comment-16111334 ] Mithun Radhakrishnan commented on HIVE-16908: - bq. I am looking into the option of launching metastore in a different process. Thank you for holding out for the proper solution to this, [~sbeeram]. A couple of challenges come to mind: # The test will need to communicate which port the remote metastore was started at. We might consider pre-setting the port, but I wonder if this might make the test's start-up brittle. # Cleanup for the remote metastore's state, after the test. bq. sorry about being so anal about this... I fear that label fits my stand more than yours. :/ In the spirit of detente, I'd be happy to revisit your first proposal, if the multi-process solution proves cumbersome. > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch, > HIVE-16908.3.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110809#comment-16110809 ] Sunitha Beeram commented on HIVE-16908: --- [~mithun] sorry about being so anal about this, but with 2 metastores running in the same JVM we are always vulnerable to tests passing or failing due to weird reasons. I don't think fixing static-state is as much a priority (as we don't really have a use case to support running multiple metastores in the same JVM) - so if you think the tests have to use 2 metastores, then lets launch them in a way they would be launched? I am looking into the option of launching metastore in a different process. > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch, > HIVE-16908.3.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107778#comment-16107778 ] Sunitha Beeram commented on HIVE-16908: --- [~mithun] thanks - will take a look at the earliest. > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch, > HIVE-16908.3.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107621#comment-16107621 ] Mithun Radhakrishnan commented on HIVE-16908: - After co-opting this JIRA so that I could provide an alternative fix, I've reset ownership back to [~sbeeram]. Rather than rewrite the test to launch the target metastore in a separate process, I have reworded the failing tests without changing their intention. In the places where calls to the source and target metastores are interleaved, I fetch a new HMS client. Fixing the static-state in the metastore is a larger problem that should be addressed separately. [~sbeeram], I wonder if you might find this satisfactory. > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch, > HIVE-16908.3.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106123#comment-16106123 ] Hive QA commented on HIVE-16908: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12879456/HIVE-16908.3.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11017 tests executed *Failed tests:* {noformat} TestPerfCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype] (batchId=158) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=168) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=100) org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge (batchId=206) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6187/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6187/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6187/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12879456 - PreCommit-HIVE-Build > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Mithun Radhakrishnan > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch, > HIVE-16908.3.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105880#comment-16105880 ] Mithun Radhakrishnan commented on HIVE-16908: - I'm taking a crack at this failure. Assigning this to myself, for the moment, for the sake of uploading a patch. > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105373#comment-16105373 ] Mithun Radhakrishnan commented on HIVE-16908: - [~sbeeram]: I have raised HIVE-17201 to temporarily disable these three tests, to clean up the tests for others. This is only temporary, until we fix the tests properly. bq. If the target metastore instance were accessed through a different classloader... I made an initial pass at doing this. I don't have a proper solution yet. Will update. > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078973#comment-16078973 ] Sunitha Beeram commented on HIVE-16908: --- [~mithun] thanks for the review! Since we still have unresolved questions in HIVE-16844, I will focus on that first and return to failing tests if/when the solution in HIVE-16844 gets updated. > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075593#comment-16075593 ] Mithun Radhakrishnan commented on HIVE-16908: - Yes, this is what I was afraid of. The intention of {{testTableSchemaPropagation()}} was to simulate table-propagation across different clusters/HCat instances, as Apache Falcon (or similar projects) do. I wonder if this change dilutes that intention. :/ I do recognize that the static state in {{ObjectStore}} makes this problematic. I'm trying to figure out an alternative. Question: If the target metastore instance were accessed through a different classloader, their states would be isolated, right? Would that be an acceptable solution? > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075241#comment-16075241 ] Mithun Radhakrishnan commented on HIVE-16908: - I'll need a little time to review. On the face of it, this change is disconcerting, since it looks like changes the intention of the tests added in HIVE-7341. :/ Let me take a closer look. > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067525#comment-16067525 ] Thejas M Nair commented on HIVE-16908: -- [~mithun] Can you please review this change ? cc [~sushanth] > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059651#comment-16059651 ] Ratandeep Ratti commented on HIVE-16908: [~sbeeram] . The tests look OK to me. > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058797#comment-16058797 ] Hive QA commented on HIVE-16908: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12873983/HIVE-16908.2.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10827 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main] (batchId=149) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=145) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=232) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query16] (batchId=232) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query94] (batchId=232) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=103) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootstrapFunctionReplication (batchId=216) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionIncrementalReplication (batchId=216) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionWithFunctionBinaryJarsOnHDFS (batchId=216) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5727/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5727/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5727/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12873983 - PreCommit-HIVE-Build > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058561#comment-16058561 ] Hive QA commented on HIVE-16908: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12873958/HIVE-16908.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10841 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main] (batchId=149) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=232) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query16] (batchId=232) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=232) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query94] (batchId=232) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootstrapFunctionReplication (batchId=216) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionIncrementalReplication (batchId=216) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionWithFunctionBinaryJarsOnHDFS (batchId=216) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5722/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5722/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5722/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12873958 - PreCommit-HIVE-Build > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058336#comment-16058336 ] Sunitha Beeram commented on HIVE-16908: --- [~sankarh] would be great if you reviewed the patch or recommended someone else more familiar with the code/original changes to also review the changes. Hopefully I haven't oversimplified or relied on other/incorrect assumptions that make the tests pass currently. I am including the description from the RB here for quick reference: "HIVE-16844 fixed a connection leak issue which subsequently exposed failures in TestHCatClient. The connection leak gets triggered if a metastore instance is updated with a different JDO configuration. TestHCatClient uses 2 metastore instances to test replication related methods. Unfortunately, it does so by providing a different derby db name for the second instance. Since the 2 metastores run in the same JVM, the path fixed in HIVE-16844 gets triggered, resulting in "sourceMetastore"'s connection being closed and thus resulting in failures. It appears to me that running 2 metastore instances within the same JVM is error prone as there could be unintentional side-effects due to statics in the code (as was exposed by fixing HIVE-16844). This patch provides a way to test the replication related methods without involving a second instance. The changes mainly validate the serialize/deserialize methods. One of the tests, testPartitionRegistrationWithCustomSchema, uses addPartitions method to verify propogation of changes and it appeared that addPartitions wasn't covered by other tests in TestHCatClient and there wasn't a better way to verify the intended path, so I used an approach where the original database and table are dropped and recreated using the serialized-string and captured partition spec." > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16908.1.patch > > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[ https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050440#comment-16050440 ] Sunitha Beeram commented on HIVE-16908: --- +Watchers > Failures in TestHcatClient due to HIVE-16844 > > > Key: HIVE-16908 > URL: https://issues.apache.org/jira/browse/HIVE-16908 > Project: Hive > Issue Type: Bug >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > > Some of the tests in TestHCatClient.java, for ex: > {noformat} > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema > (batchId=177) > org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation > (batchId=177) > {noformat} > are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new > configuration object is set on the ObjectStore. TestHCatClient fires up a > second instance of metastore thread with a different conf object that results > in the PersistenceMangaerFactory closure and hence tests fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)