[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16848670#comment-16848670
]
Peter Vary commented on HIVE-21550:
---
[~abstractdog]: These are still failing often. When running MetaStore tests we
had problems with parallel runs where the jdbc url was not changed from the
default and other tests were removing / reinitializing the Derby database.
Could this be the problem here?
Do you think adding something like this could help?
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/MetaStoreTestUtils.java#L156
{code}
jdbcUrl = "jdbc:derby:;databaseName=" + TMP_DIR + File.separator
+ "junit_metastore_db_" + metaStorePort + ";create=true";
MetastoreConf.setVar(conf, ConfVars.CONNECT_URL_KEY, jdbcUrl);
{code}
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:61)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1882)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.g
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16849544#comment-16849544
]
Laszlo Bodor commented on HIVE-21550:
-
[~pvary]: thanks, I think it can help, I've introduced a random string after
the in-memory derby instance URL (haven't seen the original issue since)
moreover, the test used to start with cleaning up the database, which seems a
test smell to me, as it implies that the same database is shared between
multiple tests (I don't think it's necessary unless it involves a huge memory
footprint)
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21550.01.patch, HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:61)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1882)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:759)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850032#comment-16850032
]
Hive QA commented on HIVE-21550:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
27s{color} | {color:blue} standalone-metastore/metastore-server in master has
183 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17330/dev-support/hive-personality.sh
|
| git revision | master / cbd8c7b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17330/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21550.01.patch, HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.T
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850083#comment-16850083
]
Hive QA commented on HIVE-21550:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970023/HIVE-21550.01.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 16062 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheInSession
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=232)
org.apache.hive.hcatalog.mapreduce.TestHCatMutableNonPartitioned.testHCatNonPartitionedTable[6]
(batchId=216)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17330/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17330/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17330/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970023 - PreCommit-HIVE-Build
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21550.01.patch, HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TB
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850598#comment-16850598
]
Laszlo Bodor commented on HIVE-21550:
-
[~pvary]: could you please review? unfortunately, I cannot be 100% that this
solved the problem, maybe we can reopen later if needed
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21550.01.patch, HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:61)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1882)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:759)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2677)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3606)
> [classes/:?] at
> org.
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850599#comment-16850599
]
Laszlo Bodor commented on HIVE-21550:
-
TestObjectStore failures are related, I'm checking them
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21550.01.patch, HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:61)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1882)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:759)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2677)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3606)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.java:
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850618#comment-16850618
]
Peter Vary commented on HIVE-21550:
---
[~abstractdog]: We have TestObjectStore in the last test run. Are those related?
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21550.01.patch, HIVE-21550.02.patch,
> HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:61)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1882)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:759)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2677)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3606)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.Object
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850623#comment-16850623
]
Laszlo Bodor commented on HIVE-21550:
-
[~pvary]: fixed the issue in 02.patch, waiting for tests, could you please
review?
cannot validate by unit test that the fix solves the problem, however, I
couldn't have reproduced the issue since (maybe we can reopen this if needed)
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21550.01.patch, HIVE-21550.02.patch,
> HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:61)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1882)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:759)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2677)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2673)
> [cl
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850641#comment-16850641
]
Peter Vary commented on HIVE-21550:
---
+1 pending tests
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21550.01.patch, HIVE-21550.02.patch,
> HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:61)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1882)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:759)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2677)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3606)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.java:2689)
> [classes/:
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851204#comment-16851204
]
Hive QA commented on HIVE-21550:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
23s{color} | {color:blue} standalone-metastore/metastore-server in master has
183 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17342/dev-support/hive-personality.sh
|
| git revision | master / f0c29ac |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17342/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21550.01.patch, HIVE-21550.02.patch,
> HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851251#comment-16851251
]
Hive QA commented on HIVE-21550:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970133/HIVE-21550.02.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16063 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17342/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17342/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17342/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970133 - PreCommit-HIVE-Build
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21550.01.patch, HIVE-21550.02.patch,
> HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.Metas
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806093#comment-16806093
]
Laszlo Bodor commented on HIVE-21550:
-
reproduced with [^HIVE-21550.repro.patch], result:
[^org.apache.hadoop.hive.metastore.TestObjectStore-output.txt]
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:61)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1882)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:759)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2677)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3606)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.j
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806127#comment-16806127
]
Laszlo Bodor commented on HIVE-21550:
-
after upgrading to 10.14.1.0 I cannot reproduce this lock issue
(https://issues.apache.org/jira/secure/attachment/12964318/HIVE-21553.01.patch)
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:61)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1882)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:759)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2677)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:2673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3606)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore.getPa
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902476#comment-16902476
]
Vineet Garg commented on HIVE-21550:
These tests are still failing albeit with NullPointerException now
https://builds.apache.org/job/PreCommit-HIVE-Build/18277/testReport/
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hive.metastore.TestObjectStore.setUp(TestObjectStore.java:147)
{code}
Not sure if it is related to this change at all.
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21550.01.patch, HIVE-21550.02.patch,
> HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:61)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1882)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:759)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907246#comment-16907246
]
Laszlo Bodor commented on HIVE-21550:
-
thanks [~vgarg], looking at current state of TestObjectStore:
https://github.com/apache/hive/blob/4510efd15f44cc4c217bbc65ad2147c14261bccc/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java#L147
NPE is thrown while calling a method of a recently created object, not really
sure how this is possible
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21550.01.patch, HIVE-21550.02.patch,
> HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:61)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1882)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:759)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:673)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.ObjectStore
