[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699559#comment-16699559 ]
Vihang Karajgaonkar edited comment on HIVE-20740 at 11/26/18 8:41 PM: ---------------------------------------------------------------------- The test failures occur only on the precommit job. The logs do not have enough information to debug these failures. I will try to observer on the ptest server itself while the batch containing {{TestObjectStore}} test is running. Batch 230 has {{TestObjectStore}} 2018-11-26 20:25:19,796 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch: UnitTestBatch [name=230_UTBatch_standalone-metastore__metastore-server_20_tests, id=230, moduleName=standalone-metastore/metastore-server, batchSize=20, isParallel=true, testList=[TestMetaStoreConnectionUrlHook, TestSchemaToolForMetastore, TestMetastoreSchemaTool, TestMetaStoreSchemaFactory, TestRetryingHMSHandler, TestAdminUser, TestJSONMessageDeserializer, TestCatalogNonDefaultSvr, TestObjectStoreInitRetry, TestHdfsUtils, TestMetaStoreServerUtils, TestHiveMetaStoreSchemaMethods, TestOldSchema, TestCachedStore, TestCatalogCaching, TestDeadline, TestMetaStoreListenersError, TestMetaStoreEventListenerOnlyOnCommit, TestMetaStoreSchemaInfo, TestMarkPartition]] was (Author: vihangk1): The test failures occur only on the precommit job. The logs do not have enough information to debug these failures. I will try to observer on the ptest server itself while the batch containing {{TestObjectStore}} test is running. Batch 230 has {{TestObjectStore}} {noformat} 2018-11-26 20:25:19,796 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch: UnitTestBatch [name=230_UTBatch_standalone-metastore__metastore-server_20_tests, id=230, moduleName=standalone-metastore/metastore-server, batchSize=20, isParallel=true, testList=[TestMetaStoreConnectionUrlHook, TestSchemaToolForMetastore, TestMetastoreSchemaTool, TestMetaStoreSchemaFactory, TestRetryingHMSHandler, TestAdminUser, TestJSONMessageDeserializer, TestCatalogNonDefaultSvr, TestObjectStoreInitRetry, TestHdfsUtils, TestMetaStoreServerUtils, TestHiveMetaStoreSchemaMethods, TestOldSchema, TestCachedStore, TestCatalogCaching, TestDeadline, TestMetaStoreListenersError, TestMetaStoreEventListenerOnlyOnCommit, TestMetaStoreSchemaInfo, TestMarkPartition]] {noformat} > Remove global lock in ObjectStore.setConf method > ------------------------------------------------ > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement > Reporter: Vihang Karajgaonkar > Assignee: Vihang Karajgaonkar > Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch, > HIVE-20740.11.patch, HIVE-20740.12.patch, HIVE-20740.13.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)