[jira] [Created] (HIVE-20741) Disable or fix random failed tests
Yongzhi Chen created HIVE-20741:
---
Summary: Disable or fix random failed tests
Key: HIVE-20741
URL: https://issues.apache.org/jira/browse/HIVE-20741
Project: Hive
Issue Type: Bug
Reporter: Yongzhi Chen
Two qfile tests for TestCliDriver, they may all relate to number precision
issues:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_context_ngrams]
(batchId=79)
Error:
Client Execution succeeded but contained differences (error code = 1) after
executing udaf_context_ngrams.q
43c43
< [{"ngram":["travelling"],"estfrequency":1.0}]
---
> [{"ngram":["travelling"],"estfrequency":3.0}]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_corr] (batchId=84)
Client Execution succeeded but contained differences (error code = 1) after
executing udaf_corr.q
100c100
< 0.6633880657639324
---
> 0.6633880657639326
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-20740) Remove global lock in ObjectStore.setConf method
Vihang Karajgaonkar created HIVE-20740:
--
Summary: Remove global lock in ObjectStore.setConf method
Key: HIVE-20740
URL: https://issues.apache.org/jira/browse/HIVE-20740
Project: Hive
Issue Type: Improvement
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar
The ObjectStore#setConf method has a global lock which can block other clients
in concurrent workloads.
{code}
@Override
@SuppressWarnings("nls")
public void setConf(Configuration conf) {
// Although an instance of ObjectStore is accessed by one thread, there may
// be many threads with ObjectStore instances. So the static variables
// pmf and prop need to be protected with locks.
pmfPropLock.lock();
try {
isInitialized = false;
this.conf = conf;
this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
ConfVars.HIVE_TXN_STATS_ENABLED);
configureSSL(conf);
Properties propsFromConf = getDataSourceProps(conf);
boolean propsChanged = !propsFromConf.equals(prop);
if (propsChanged) {
if (pmf != null){
clearOutPmfClassLoaderCache(pmf);
if (!forTwoMetastoreTesting) {
// close the underlying connection pool to avoid leaks
pmf.close();
}
}
pmf = null;
prop = null;
}
assert(!isActiveTransaction());
shutdown();
// Always want to re-create pm as we don't know if it were created by the
// most recent instance of the pmf
pm = null;
directSql = null;
expressionProxy = null;
openTrasactionCalls = 0;
currentTransaction = null;
transactionStatus = TXN_STATUS.NO_STATE;
initialize(propsFromConf);
String partitionValidationRegex =
MetastoreConf.getVar(this.conf,
ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
if (partitionValidationRegex != null &&
!partitionValidationRegex.isEmpty()) {
partitionValidationPattern = Pattern.compile(partitionValidationRegex);
} else {
partitionValidationPattern = null;
}
// Note, if metrics have not been initialized this will return null,
which means we aren't
// using metrics. Thus we should always check whether this is non-null
before using.
MetricRegistry registry = Metrics.getRegistry();
if (registry != null) {
directSqlErrors =
Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
}
this.batchSize = MetastoreConf.getIntVar(conf,
ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
if (!isInitialized) {
throw new RuntimeException(
"Unable to create persistence manager. Check dss.log for details");
} else {
LOG.debug("Initialized ObjectStore");
}
} finally {
pmfPropLock.unlock();
}
}
{code}
The {{pmfPropLock}} is a static object and it disallows any other new
connection to HMS which is trying to instantiate ObjectStore. We should either
remove the lock or reduce the scope of the lock so that it is held for a very
small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Review Request 69011: HIVE-20713: Use percentage for join conversion size thresholds
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69011/ --- Review request for hive and Ashutosh Chauhan. Bugs: HIVE-20713 https://issues.apache.org/jira/browse/HIVE-20713 Repository: hive-git Description --- HIVE-20713: Use percentage for join conversion size thresholds Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cc6239c3135714fb65aa1afc2882852460b68b37 data/conf/hive-site.xml 0daf9adc717bc1c4413d2e34691c26a3e2585c77 data/conf/llap/hive-site.xml 44ca6c9daf092a35f1c58c26dfa3575c303464ce data/conf/tez/hive-site.xml 236adc7087b43f4e9ab95b2fa57436cf75c679aa itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 40dd992455f2fa6bae85d9d02338bc820a370ebe ql/src/java/org/apache/hadoop/hive/ql/exec/MemoryInfo.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java a6b0dbc0dc956d81d027f08a55fbdf0ca452638f ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java 54377428eafdb79e1bbdc8a182eafb46f8febd23 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 151d1b39eb17bba336bb02ab06f216361e4f6ff2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 5217208611a6ab63b218a585e820516221022ec6 ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationOptimizer.java 7ff92edd91018f55dd90200f184869a74868dd46 ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java 5269eb6f6a77ac031d144a96a2687331e3444f4b ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinResolver.java a84534a9495b44ac319ad52a1f841428861d62e9 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java a52e5e60d97d24996c813b463caffb946ce5b61a ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java 0d1990a8cb3882c2c8b271c90e2675e88c79a7ea ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MemoryDecider.java 64f1e7b830f85e898252f964bdbfb73267224ef6 ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java ad6db21fe6041e18adc646ca20759ca3cde58713 ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 22f3266c87f1d42c254893b424b68e757fb2953b ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java fdc963506cabd96c125ec15b6a1eacb71fd33d42 ql/src/test/org/apache/hadoop/hive/ql/exec/TestOperators.java c7cd4ad3f6deba320b343cda071640cd1c78743e ql/src/test/queries/clientpositive/acid_mapjoin.q e3d14cc97d1e193a6bbb2a0e25e0741a192e2638 ql/src/test/queries/clientpositive/autoColumnStats_1.q 9a62127dabee743b8880f0effdd5184fb24975d0 ql/src/test/queries/clientpositive/autoColumnStats_2.q 51f252b303fb579d315aabcabb75b4429c7b5d4d ql/src/test/queries/clientpositive/autoColumnStats_3.q 7a419118902ffab767780057d8e83fbde3fbdba9 ql/src/test/queries/clientpositive/auto_join_reordering_values.q 39b2e1d9a78f77c50d60a749085b05ba9acd6b20 ql/src/test/queries/clientpositive/auto_join_stats.q 8b377bf4ca70ccaf43b625afc51b1439916f ql/src/test/queries/clientpositive/auto_join_without_localtask.q 15697f2f11ac4669c7bdf6e4144f02fd3cb465b2 ql/src/test/queries/clientpositive/auto_sortmerge_join_1.q 1fbe8f79cff97ef7baf8eb8afce36665b425b23d ql/src/test/queries/clientpositive/auto_sortmerge_join_10.q 1b15a740120862a7d56e7b36e07a0e3c4d7c835d ql/src/test/queries/clientpositive/auto_sortmerge_join_11.q 76e615fbde1d8d52eb2a8de5d08d69d26e304a99 ql/src/test/queries/clientpositive/auto_sortmerge_join_12.q d68efea393af588527a698dda2a578433195472e ql/src/test/queries/clientpositive/auto_sortmerge_join_13.q bbd7afab07b62d1e6b63b408a5cce71db5ef55f1 ql/src/test/queries/clientpositive/auto_sortmerge_join_14.q 1e21c92e59f56688d05d3fbc21184a70c9779996 ql/src/test/queries/clientpositive/auto_sortmerge_join_15.q b3dd8e50d7cd4e3f95f8b595c10acbfb15732943 ql/src/test/queries/clientpositive/auto_sortmerge_join_16.q 0a72ddfc6b859912962b30719637645648d625ad ql/src/test/queries/clientpositive/auto_sortmerge_join_2.q c6d5318f43daa419b8ad0fedcf6782ea3f333f5d ql/src/test/queries/clientpositive/auto_sortmerge_join_3.q f41b0970c9ff98f5d72bfa2b3ad07b4b7581e6ab ql/src/test/queries/clientpositive/auto_sortmerge_join_4.q e28e0ab4a09cf0214ce946ec79bb34063705e9ad ql/src/test/queries/clientpositive/auto_sortmerge_join_5.q 9550e21382237f26f9b7187bc024053221fe365f ql/src/test/queries/clientpositive/auto_sortmerge_join_6.q 0aeec0e50fa3dcff0e20f77cf9541d67c141d4db ql/src/test/queries/clientpositive/auto_sortmerge_join_7.q 8c9dbacbbad8efe42986556a7d2f6cb62fd3b906 ql/src/test/queries/clientpositive/auto_sortmerge_join_8.q 51647e86b2bc1d680b59c30aeeaef6f949f21427 ql/src/test/queries/cli
[jira] [Created] (HIVE-20739) CBO: Rewrite INTERSECTS queries as EXISTS semi-joins
Gopal V created HIVE-20739:
--
Summary: CBO: Rewrite INTERSECTS queries as EXISTS semi-joins
Key: HIVE-20739
URL: https://issues.apache.org/jira/browse/HIVE-20739
Project: Hive
Issue Type: Improvement
Reporter: Gopal V
INTERSECTS clause currently materializes both sides before removing entries
from each other.
This means that if the intersects has filters on one side which apply to the
other side, it is not transitively pushed to the other side.
Here's the snippet from the TPC-DS Query8 where that is visible.
{code}
SELECT substr(ca_zip,1,5) ca_zip
FROM customer_address
WHERE substr(ca_zip,1,5) IN (
'89436', /* ... */ ','15492'))
intersect
(select ca_zip
from (SELECT substr(ca_zip,1,5) ca_zip,count(*) cnt
FROM customer_address, customer
WHERE ca_address_sk = c_current_addr_sk and
c_preferred_cust_flag='Y'
group by ca_zip
having count(*) > 10)A1))A2
{code}
https://github.com/hortonworks/hive-testbench/blob/hdp3/sample-queries-tpcds/query8.sql#L92
where the entire join output of (customer_address x customer) is produced where
the transitive filter inference could be applied to push the filter to the
customer_address in the 2nd set in intersection.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-20738) Enable Delete Event filtering in VectorizedOrcAcidRowBatchReader
Eugene Koifman created HIVE-20738: - Summary: Enable Delete Event filtering in VectorizedOrcAcidRowBatchReader Key: HIVE-20738 URL: https://issues.apache.org/jira/browse/HIVE-20738 Project: Hive Issue Type: Improvement Components: Transactions Affects Versions: 3.0.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Currently DeleteEventRegistry loads all delete events which can take time and use a lot of memory. Should minimize the number of deletes loaded based on the insert events included in the Split. This is an umbrella jira for several tasks that make up the work. See individual tasks for details. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20737) LocalHiveSparkClient and SparkSession race condition fix
Denys Kuzmenko created HIVE-20737: - Summary: LocalHiveSparkClient and SparkSession race condition fix Key: HIVE-20737 URL: https://issues.apache.org/jira/browse/HIVE-20737 Project: Hive Issue Type: Bug Components: Hive Reporter: Denys Kuzmenko -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20736) Eagerly allocate Containers from Yarn
slim bouguerra created HIVE-20736: - Summary: Eagerly allocate Containers from Yarn Key: HIVE-20736 URL: https://issues.apache.org/jira/browse/HIVE-20736 Project: Hive Issue Type: Bug Components: llap, Tez Affects Versions: 4.0.0 Reporter: slim bouguerra Assignee: Sergey Shelukhin According to [~sershe] HS2 interactive at startup-time tries to eagerly allocated what ever needed to execute queries. But currently this process is kind of broken. AS of now HS2I starts and tries to allocate the resources but in cases there is not enough Yarn resources in the desired queue, HS2I will keep trying in the background forever and will not bubble this up as an issue. trying forever to allocate without signaling error defeats the idea of eagerly allocate in my opinion. I think HS2I has to fail the start if after XX minutes can not eagerly allocate the minimum space needed to run the max concurrent query. CC [~hagleitn]/[~t3rmin4t0r] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20735) Address some of the review comments.
slim bouguerra created HIVE-20735: - Summary: Address some of the review comments. Key: HIVE-20735 URL: https://issues.apache.org/jira/browse/HIVE-20735 Project: Hive Issue Type: Sub-task Components: kafka integration Reporter: slim bouguerra Assignee: slim bouguerra As part of the review comments we agreed to: # remove start and end offsets columns # remove the best effort mode # make the 2pc as default protocol for EOS Also this patch will include an additional enhancement to add kerberos support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20734) Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it should use the system username/dummy password instead of prompting for one
Vaibhav Gumashta created HIVE-20734: --- Summary: Beeline: When beeline-site.xml is and hive CLI redirects to beeline, it should use the system username/dummy password instead of prompting for one Key: HIVE-20734 URL: https://issues.apache.org/jira/browse/HIVE-20734 Project: Hive Issue Type: Bug Components: Beeline Affects Versions: 3.1.0 Reporter: Vaibhav Gumashta -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20733) GenericUDFOPEqualNS may not use = in plan descriptions
Zoltan Haindrich created HIVE-20733: --- Summary: GenericUDFOPEqualNS may not use = in plan descriptions Key: HIVE-20733 URL: https://issues.apache.org/jira/browse/HIVE-20733 Project: Hive Issue Type: Bug Reporter: Zoltan Haindrich right now GenericUDFOPEqualNS is displayed a "=" in explains; however it should be "<=>" this may cause some confusion... related qtest: is_distinct_from.q same: GenericUDFOPNotEqualNS -- This message was sent by Atlassian JIRA (v7.6.3#76005)
