[jira] [Commented] (HIVE-20966) Support bootstrap and incremental replication to a target with hive.strict.managed.tables enabled.
[
https://issues.apache.org/jira/browse/HIVE-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713572#comment-16713572
]
Hive QA commented on HIVE-20966:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951087/HIVE-20966.05.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 49 failed/errored test(s), 15647 tests
executed
*Failed tests:*
{noformat}
TestAlterTableMetadata - did not produce a TEST-*.xml file (likely timed out)
(batchId=248)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=248)
TestReplAcidTablesWithJsonMessage - did not produce a TEST-*.xml file (likely
timed out) (batchId=248)
TestSemanticAnalyzerHookLoading - did not produce a TEST-*.xml file (likely
timed out) (batchId=248)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTableCascade
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterViewParititon
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testColumnStatistics
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTypeApi
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testConcurrentMetastores
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateAndGetTableWithDriver
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateTableSettingId
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBLocationChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwner
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwnerChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabase
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocation
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocationWithPermissionProblems
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropDatabaseCascadeMVMultiDB
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterLastPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterSinglePartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFunctionWithResources
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetConfigValue
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetMetastoreUuid
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetPartitionsWithSpec
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetSchemaWithNoClassDefFoundError
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetTableObjects
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetUUIDInParallel
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testJDOPersistanceManagerCleanup
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionNames
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitions
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionsWihtLimitEnabled
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testNameMethods
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartitionFilter
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRenamePartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRetriableClientWithConnLifetime
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleFunction
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindH
[jira] [Commented] (HIVE-20966) Support bootstrap and incremental replication to a target with hive.strict.managed.tables enabled.
[
https://issues.apache.org/jira/browse/HIVE-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713570#comment-16713570
]
Hive QA commented on HIVE-20966:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
11s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
184 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} hcatalog/server-extensions in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} common: The patch generated 1 new + 427 unchanged - 0
fixed = 428 total (was 427) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 55 new + 854 unchanged - 11
fixed = 909 total (was 865) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} itests/hive-unit: The patch generated 146 new + 658
unchanged - 58 fixed = 804 total (was 716) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 16 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
8s{color} | {color:red} standalone-metastore/metastore-server generated 4 new +
184 unchanged - 0 fixed = 188 total (was 184) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
54s{color} | {color:red} ql generated 1 new + 2310 unchanged - 1 fixed = 2311
total (was 2311) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} hcatalog/server-extensions generated 2 new + 1
unchanged - 0 fixed = 3 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Boxed value is unboxed and then immediately reboxed in
org.apache.hadoop.hive.metastore.messaging.json.JSONAlte
[jira] [Issue Comment Deleted] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-20915: Comment: was deleted (was: I believe beyond column stats autogather there might be other features which have be tampered with - and because the q.out updates seem to be careless. I think in that case this feature should be disabled globally - because its buggyand after all known issues were addressed; enable it - but take special care for every q.out update Please understand that we just can't let to break existing features...if the development spans to month or years; those broken features will eventually be released...) > Make dynamic sort partition optimization available to HoS and MR > > > Key: HIVE-20915 > URL: https://issues.apache.org/jira/browse/HIVE-20915 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch, > HIVE-20915.3.patch, HIVE-20915.4.patch, HIVE-20915.5.patch > > > HIVE-20703 put dynamic sort partition optimization under cost based decision, > but it also makes the optimizer only available to tez. > hive.optimize.sort.dynamic.partition works with other execution engines for a > long time, we should keep the optimizer available to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713566#comment-16713566 ] Morio Ramdenbourg commented on HIVE-20992: -- Also, adding Shims back is not necessary. The method [MetastoreConf.getPassword()|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java#L1695-L1711] takes care of this for us. It already handles secure password access, and automatically checks from the CredentialsProvider. > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Attachments: HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.use.SSL_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * *_javax.net.ssl.trustStore_* > ** Truststore location > ** Default: None > ** E.g. _/tmp/truststore_ > * *_javax.net.ssl.trustStorePassword_* > ** Truststore password > ** Default: None > ** E.g. _password_ > * *_javax.net.ssl.trustStoreType_* > ** Truststore type > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713562#comment-16713562 ] Morio Ramdenbourg edited comment on HIVE-20992 at 12/8/18 6:33 AM: --- ReviewBoard: [https://reviews.apache.org/r/69534/] This is my biggest Hive patch so far, so any feedback, even on small things, would be greatly appreciated. :) was (Author: mramdenbourg): ReviewBoard: [https://reviews.apache.org/r/69534/] This is my biggest Hive review so far, so any feedback, even on small things, would be greatly appreciated. :) > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Attachments: HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.use.SSL_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * *_javax.net.ssl.trustStore_* > ** Truststore location > ** Default: None > ** E.g. _/tmp/truststore_ > * *_javax.net.ssl.trustStorePassword_* > ** Truststore password > ** Default: None > ** E.g. _password_ > * *_javax.net.ssl.trustStoreType_* > ** Truststore type > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21021) Scalar subquery with only aggregate in subquery (no group by) has unnecessary sq_count_check branch
[
https://issues.apache.org/jira/browse/HIVE-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713559#comment-16713559
]
Hive QA commented on HIVE-21021:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951076/HIVE-21021.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 61 failed/errored test(s), 15633 tests
executed
*Failed tests:*
{noformat}
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=24)
[authorization_cli_stdconfigauth.q,timestamp_comparison.q,parquet_vectorization_10.q,materialized_view_rewrite_2.q,stats10.q,vector_join_part_col_char.q,nonblock_op_deduplicate.q,udf_testlength2.q,udf4.q,column_pruner_multiple_children.q,udf_stddev.q,parquet_analyze.q,select_dummy_source.q,tez_multi_union.q,orc_diff_part_cols.q,partition_boolean.q,default_file_format.q,udf_to_utc_timestamp.q,windowing_ntile.q,autoColumnStats_1.q,alter_rename_partition.q,authorization_create_func1.q,smb_mapjoin_3.q,timestamp_formats.q,orc_ppd_basic.q,udf_notop.q,input_testxpath3.q,pointlookup5.q,vector_null_map.q,array_map_access_nonconstant.q]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=169)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=128)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query14]
(batchId=270)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=270)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query14]
(batchId=270)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query14]
(batchId=270)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTableCascade
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterViewParititon
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testColumnStatistics
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTypeApi
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testConcurrentMetastores
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateAndGetTableWithDriver
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateTableSettingId
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBLocationChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwner
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwnerChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabase
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocation
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocationWithPermissionProblems
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropDatabaseCascadeMVMultiDB
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterLastPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterSinglePartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFunctionWithResources
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetConfigValue
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetMetastoreUuid
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetPartitionsWithSpec
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetSchemaWithNoClassDefFoundError
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetTableObjects
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetUUIDInParallel
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testJDOPersistanceManagerCleanup
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionNames
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitions
(batch
[jira] [Updated] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Morio Ramdenbourg updated HIVE-20992: - Attachment: HIVE-20992.patch Status: Patch Available (was: In Progress) ReviewBoard: [https://reviews.apache.org/r/69534/] This is my biggest Hive review so far, so any feedback, even on small things, would be greatly appreciated. :) > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Attachments: HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.use.SSL_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * *_javax.net.ssl.trustStore_* > ** Truststore location > ** Default: None > ** E.g. _/tmp/truststore_ > * *_javax.net.ssl.trustStorePassword_* > ** Truststore password > ** Default: None > ** E.g. _password_ > * *_javax.net.ssl.trustStoreType_* > ** Truststore type > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713558#comment-16713558 ] Zoltan Haindrich commented on HIVE-20915: - I believe beyond column stats autogather there might be other features which have be tampered with - and because the q.out updates seem to be careless. I think in that case this feature should be disabled globally - because its buggyand after all known issues were addressed; enable it - but take special care for every q.out update Please understand that we just can't let to break existing features...if the development spans to month or years; those broken features will eventually be released... > Make dynamic sort partition optimization available to HoS and MR > > > Key: HIVE-20915 > URL: https://issues.apache.org/jira/browse/HIVE-20915 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch, > HIVE-20915.3.patch, HIVE-20915.4.patch, HIVE-20915.5.patch > > > HIVE-20703 put dynamic sort partition optimization under cost based decision, > but it also makes the optimizer only available to tez. > hive.optimize.sort.dynamic.partition works with other execution engines for a > long time, we should keep the optimizer available to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20966) Support bootstrap and incremental replication to a target with hive.strict.managed.tables enabled.
[ https://issues.apache.org/jira/browse/HIVE-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20966: --- Attachment: HIVE-20966.05.patch > Support bootstrap and incremental replication to a target with > hive.strict.managed.tables enabled. > -- > > Key: HIVE-20966 > URL: https://issues.apache.org/jira/browse/HIVE-20966 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR > Attachments: HIVE-20966.01.patch, HIVE-20966.02.patch, > HIVE-20966.03.patch, HIVE-20966.04.patch, HIVE-20966.04.patch, > HIVE-20966.05.patch > > > *Requirements:* > Hive2 supports replication of managed tables. But in Hive3 with > hive.strict.managed.tables=true, some of these managed tables are converted > to ACID or MM tables. Also, some of them are converted to external tables > based on below rules. > - Avro format with external schema, Storage handlers, List bucketed tabled > are converted to external tables. > - Location not owned by "hive" user are converted to external table. > - Hive owned ORC format are converted to full ACID transactional table. > - Hive owned Non-ORC format are converted to MM transactional table. > REPL LOAD should apply these rules during bootstrap and incremental phases > and convert the tables accordingly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20966) Support bootstrap and incremental replication to a target with hive.strict.managed.tables enabled.
[ https://issues.apache.org/jira/browse/HIVE-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20966: --- Status: Open (was: Patch Available) > Support bootstrap and incremental replication to a target with > hive.strict.managed.tables enabled. > -- > > Key: HIVE-20966 > URL: https://issues.apache.org/jira/browse/HIVE-20966 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR > Attachments: HIVE-20966.01.patch, HIVE-20966.02.patch, > HIVE-20966.03.patch, HIVE-20966.04.patch, HIVE-20966.04.patch, > HIVE-20966.05.patch > > > *Requirements:* > Hive2 supports replication of managed tables. But in Hive3 with > hive.strict.managed.tables=true, some of these managed tables are converted > to ACID or MM tables. Also, some of them are converted to external tables > based on below rules. > - Avro format with external schema, Storage handlers, List bucketed tabled > are converted to external tables. > - Location not owned by "hive" user are converted to external table. > - Hive owned ORC format are converted to full ACID transactional table. > - Hive owned Non-ORC format are converted to MM transactional table. > REPL LOAD should apply these rules during bootstrap and incremental phases > and convert the tables accordingly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20966) Support bootstrap and incremental replication to a target with hive.strict.managed.tables enabled.
[ https://issues.apache.org/jira/browse/HIVE-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20966: --- Status: Patch Available (was: Open) > Support bootstrap and incremental replication to a target with > hive.strict.managed.tables enabled. > -- > > Key: HIVE-20966 > URL: https://issues.apache.org/jira/browse/HIVE-20966 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR > Attachments: HIVE-20966.01.patch, HIVE-20966.02.patch, > HIVE-20966.03.patch, HIVE-20966.04.patch, HIVE-20966.04.patch, > HIVE-20966.05.patch > > > *Requirements:* > Hive2 supports replication of managed tables. But in Hive3 with > hive.strict.managed.tables=true, some of these managed tables are converted > to ACID or MM tables. Also, some of them are converted to external tables > based on below rules. > - Avro format with external schema, Storage handlers, List bucketed tabled > are converted to external tables. > - Location not owned by "hive" user are converted to external table. > - Hive owned ORC format are converted to full ACID transactional table. > - Hive owned Non-ORC format are converted to MM transactional table. > REPL LOAD should apply these rules during bootstrap and incremental phases > and convert the tables accordingly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21021) Scalar subquery with only aggregate in subquery (no group by) has unnecessary sq_count_check branch
[
https://issues.apache.org/jira/browse/HIVE-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713553#comment-16713553
]
Hive QA commented on HIVE-21021:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15223/dev-support/hive-personality.sh
|
| git revision | master / d8c500b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15223/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Scalar subquery with only aggregate in subquery (no group by) has unnecessary
> sq_count_check branch
> ---
>
> Key: HIVE-21021
> URL: https://issues.apache.org/jira/browse/HIVE-21021
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21021.1.patch
>
>
> {code:sql}
> CREATE TABLE `store_sales`(
> `ss_sold_date_sk` int,
> `ss_quantity` int,
> `ss_list_price` decimal(7,2));
> CREATE TABLE `date_dim`(
> `d_date_sk` int,
> `d_year` int);
> explain cbo with avg_sales as
> (select avg(quantity*list_price) average_sales
> from (select ss_quantity quantity
> ,ss_list_price list_price
>from store_sales
>,date_dim
>where ss_sold_date_sk = d_date_sk
> and d_year between 1999 and 2001 ) x)
> select * from store_sales where ss_list_price > (select average_sales from
> avg_sales);
> {code}
> {noformat}
> CBO PLAN:
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none], cost=[{2.0
> rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[>($2, $3)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveTableScan(table=[[sub, store_sales]], table:alias=[store_sales])
> HiveProject($f0=[/($0, $1)])
> HiveAggregate(group=[{}], agg#0=[s
[jira] [Commented] (HIVE-21007) Semi join + Union can lead to wrong plans
[
https://issues.apache.org/jira/browse/HIVE-21007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713531#comment-16713531
]
Hive QA commented on HIVE-21007:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951070/HIVE-21007.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 45 failed/errored test(s), 15659 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterPartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterTable
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterTableCascade
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterViewParititon
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testColumnStatistics
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testComplexTable
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testComplexTypeApi
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testConcurrentMetastores
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testCreateAndGetTableWithDriver
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testCreateTableSettingId
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBLocationChange
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBOwner
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBOwnerChange
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabase
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabaseLocation
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabaseLocationWithPermissionProblems
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDropDatabaseCascadeMVMultiDB
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDropTable
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFilterLastPartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFilterSinglePartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFunctionWithResources
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetConfigValue
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetMetastoreUuid
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetPartitionsWithSpec
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetSchemaWithNoClassDefFoundError
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetTableObjects
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetUUIDInParallel
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testJDOPersistanceManagerCleanup
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitionNames
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitions
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitionsWihtLimitEnabled
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testNameMethods
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testPartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testPartitionFilter
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testRenamePartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testRetriableClientWithConnLifetime
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleFunction
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleTable
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleTypeApi
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testStatsFastTrivial
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSynchronized
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testTableDatabase
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testTableFilter
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testUpdatePartitionStat_doesNotUpdateStats
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testValidateTableCols
(batchId=227)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15222/tes
[jira] [Commented] (HIVE-21007) Semi join + Union can lead to wrong plans
[
https://issues.apache.org/jira/browse/HIVE-21007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713517#comment-16713517
]
Hive QA commented on HIVE-21007:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15222/dev-support/hive-personality.sh
|
| git revision | master / d8c500b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15222/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Semi join + Union can lead to wrong plans
> -
>
> Key: HIVE-21007
> URL: https://issues.apache.org/jira/browse/HIVE-21007
> Project: Hive
> Issue Type: Bug
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21007.1.patch, HIVE-21007.2.patch,
> HIVE-21007.3.patch, HIVE-21007.4.patch, HIVE-21007.5.patch, HIVE-21007.6.patch
>
>
> Tez compiler has the ability to push JOIN within UNION (by replicating join
> on each branch). If this JOIN had a SJ branch outgoing (or incoming) it could
> mess up the plan and end up generating incorrect or wrong plan.
> As a safe measure any SJ branch after UNION should be removed (until we
> improve the logic to better handle SJ branches)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20961) Retire NVL implementation
[
https://issues.apache.org/jira/browse/HIVE-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713509#comment-16713509
]
Hive QA commented on HIVE-20961:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951069/HIVE-20961.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 45 failed/errored test(s), 15659 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterPartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterTable
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterTableCascade
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterViewParititon
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testColumnStatistics
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testComplexTable
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testComplexTypeApi
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testConcurrentMetastores
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testCreateAndGetTableWithDriver
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testCreateTableSettingId
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBLocationChange
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBOwner
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBOwnerChange
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabase
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabaseLocation
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabaseLocationWithPermissionProblems
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDropDatabaseCascadeMVMultiDB
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDropTable
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFilterLastPartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFilterSinglePartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFunctionWithResources
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetConfigValue
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetMetastoreUuid
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetPartitionsWithSpec
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetSchemaWithNoClassDefFoundError
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetTableObjects
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetUUIDInParallel
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testJDOPersistanceManagerCleanup
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitionNames
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitions
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitionsWihtLimitEnabled
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testNameMethods
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testPartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testPartitionFilter
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testRenamePartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testRetriableClientWithConnLifetime
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleFunction
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleTable
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleTypeApi
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testStatsFastTrivial
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSynchronized
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testTableDatabase
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testTableFilter
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testUpdatePartitionStat_doesNotUpdateStats
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testValidateTableCols
(batchId=227)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15221/te
[jira] [Commented] (HIVE-20961) Retire NVL implementation
[
https://issues.apache.org/jira/browse/HIVE-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713499#comment-16713499
]
Hive QA commented on HIVE-20961:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 1 new + 1055 unchanged - 6
fixed = 1056 total (was 1061) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} ql generated 0 new + 2310 unchanged - 1 fixed = 2310
total (was 2311) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} accumulo-handler in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} hbase-handler in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15221/dev-support/hive-personality.sh
|
| git revision | master / d8c500b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15221/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql accumulo-handler hbase-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15221/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Retire NVL implementation
> -
>
> Key: HIVE-20961
> URL: https://issues.apache.org/jira/browse/HIVE-20961
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-20961.01.patch, HIVE-20961.02.patch,
> HIVE-20961.03.patch, HIVE-20961.04.patch
>
>
> Right now we have coalesce and nvl implemented separetly; it might be better
> t
[jira] [Commented] (HIVE-16100) Dynamic Sorted Partition optimizer loses sibling operators
[
https://issues.apache.org/jira/browse/HIVE-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713484#comment-16713484
]
Hive QA commented on HIVE-16100:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951066/HIVE-16100.8.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15659 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization]
(batchId=172)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15220/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15220/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15220/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951066 - PreCommit-HIVE-Build
> Dynamic Sorted Partition optimizer loses sibling operators
> --
>
> Key: HIVE-16100
> URL: https://issues.apache.org/jira/browse/HIVE-16100
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16100.1.patch, HIVE-16100.2.patch,
> HIVE-16100.2.patch, HIVE-16100.3.patch, HIVE-16100.4.patch,
> HIVE-16100.5.patch, HIVE-16100.6.patch, HIVE-16100.7.patch, HIVE-16100.8.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L173
> {code}
> // unlink connection between FS and its parent
> fsParent = fsOp.getParentOperators().get(0);
> fsParent.getChildOperators().clear();
> {code}
> The optimizer discards any cases where the fsParent has another SEL child
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21021) Scalar subquery with only aggregate in subquery (no group by) has unnecessary sq_count_check branch
[
https://issues.apache.org/jira/browse/HIVE-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21021:
---
Attachment: HIVE-21021.1.patch
> Scalar subquery with only aggregate in subquery (no group by) has unnecessary
> sq_count_check branch
> ---
>
> Key: HIVE-21021
> URL: https://issues.apache.org/jira/browse/HIVE-21021
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21021.1.patch
>
>
> {code:sql}
> CREATE TABLE `store_sales`(
> `ss_sold_date_sk` int,
> `ss_quantity` int,
> `ss_list_price` decimal(7,2));
> CREATE TABLE `date_dim`(
> `d_date_sk` int,
> `d_year` int);
> explain cbo with avg_sales as
> (select avg(quantity*list_price) average_sales
> from (select ss_quantity quantity
> ,ss_list_price list_price
>from store_sales
>,date_dim
>where ss_sold_date_sk = d_date_sk
> and d_year between 1999 and 2001 ) x)
> select * from store_sales where ss_list_price > (select average_sales from
> avg_sales);
> {code}
> {noformat}
> CBO PLAN:
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none], cost=[{2.0
> rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[>($2, $3)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveTableScan(table=[[sub, store_sales]], table:alias=[store_sales])
> HiveProject($f0=[/($0, $1)])
> HiveAggregate(group=[{}], agg#0=[sum($0)], agg#1=[count($0)])
> HiveProject($f0=[*(CAST($1):DECIMAL(10, 0), $2)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999, 2001), IS
> NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> HiveProject(cnt=[$0])
> HiveFilter(condition=[<=(sq_count_check($0), 1)])
> HiveProject(cnt=[$0])
> HiveAggregate(group=[{}], cnt=[COUNT()])
> HiveProject
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[count($0)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999,
> 2001), IS NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21021) Scalar subquery with only aggregate in subquery (no group by) has unnecessary sq_count_check branch
[
https://issues.apache.org/jira/browse/HIVE-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21021:
---
Status: Patch Available (was: Open)
> Scalar subquery with only aggregate in subquery (no group by) has unnecessary
> sq_count_check branch
> ---
>
> Key: HIVE-21021
> URL: https://issues.apache.org/jira/browse/HIVE-21021
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21021.1.patch
>
>
> {code:sql}
> CREATE TABLE `store_sales`(
> `ss_sold_date_sk` int,
> `ss_quantity` int,
> `ss_list_price` decimal(7,2));
> CREATE TABLE `date_dim`(
> `d_date_sk` int,
> `d_year` int);
> explain cbo with avg_sales as
> (select avg(quantity*list_price) average_sales
> from (select ss_quantity quantity
> ,ss_list_price list_price
>from store_sales
>,date_dim
>where ss_sold_date_sk = d_date_sk
> and d_year between 1999 and 2001 ) x)
> select * from store_sales where ss_list_price > (select average_sales from
> avg_sales);
> {code}
> {noformat}
> CBO PLAN:
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none], cost=[{2.0
> rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[>($2, $3)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveTableScan(table=[[sub, store_sales]], table:alias=[store_sales])
> HiveProject($f0=[/($0, $1)])
> HiveAggregate(group=[{}], agg#0=[sum($0)], agg#1=[count($0)])
> HiveProject($f0=[*(CAST($1):DECIMAL(10, 0), $2)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999, 2001), IS
> NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> HiveProject(cnt=[$0])
> HiveFilter(condition=[<=(sq_count_check($0), 1)])
> HiveProject(cnt=[$0])
> HiveAggregate(group=[{}], cnt=[COUNT()])
> HiveProject
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[count($0)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999,
> 2001), IS NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21021) Scalar subquery with only aggregate in subquery (no group by) has unnecessary sq_count_check branch
[
https://issues.apache.org/jira/browse/HIVE-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg reassigned HIVE-21021:
--
> Scalar subquery with only aggregate in subquery (no group by) has unnecessary
> sq_count_check branch
> ---
>
> Key: HIVE-21021
> URL: https://issues.apache.org/jira/browse/HIVE-21021
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> {code:sql}
> CREATE TABLE `store_sales`(
> `ss_sold_date_sk` int,
> `ss_quantity` int,
> `ss_list_price` decimal(7,2));
> CREATE TABLE `date_dim`(
> `d_date_sk` int,
> `d_year` int);
> explain cbo with avg_sales as
> (select avg(quantity*list_price) average_sales
> from (select ss_quantity quantity
> ,ss_list_price list_price
>from store_sales
>,date_dim
>where ss_sold_date_sk = d_date_sk
> and d_year between 1999 and 2001 ) x)
> select * from store_sales where ss_list_price > (select average_sales from
> avg_sales);
> {code}
> {noformat}
> CBO PLAN:
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none], cost=[{2.0
> rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[>($2, $3)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1], ss_list_price=[$2])
> HiveTableScan(table=[[sub, store_sales]], table:alias=[store_sales])
> HiveProject($f0=[/($0, $1)])
> HiveAggregate(group=[{}], agg#0=[sum($0)], agg#1=[count($0)])
> HiveProject($f0=[*(CAST($1):DECIMAL(10, 0), $2)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999, 2001), IS
> NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> HiveProject(cnt=[$0])
> HiveFilter(condition=[<=(sq_count_check($0), 1)])
> HiveProject(cnt=[$0])
> HiveAggregate(group=[{}], cnt=[COUNT()])
> HiveProject
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[count($0)])
> HiveJoin(condition=[=($0, $3)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_quantity=[$1],
> ss_list_price=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[sub, store_sales]],
> table:alias=[store_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(BETWEEN(false, $1, 1999,
> 2001), IS NOT NULL($0))])
> HiveTableScan(table=[[sub, date_dim]],
> table:alias=[date_dim])
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16100) Dynamic Sorted Partition optimizer loses sibling operators
[
https://issues.apache.org/jira/browse/HIVE-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713469#comment-16713469
]
Hive QA commented on HIVE-16100:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15220/dev-support/hive-personality.sh
|
| git revision | master / d8c500b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15220/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Dynamic Sorted Partition optimizer loses sibling operators
> --
>
> Key: HIVE-16100
> URL: https://issues.apache.org/jira/browse/HIVE-16100
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16100.1.patch, HIVE-16100.2.patch,
> HIVE-16100.2.patch, HIVE-16100.3.patch, HIVE-16100.4.patch,
> HIVE-16100.5.patch, HIVE-16100.6.patch, HIVE-16100.7.patch, HIVE-16100.8.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L173
> {code}
> // unlink connection between FS and its parent
> fsParent = fsOp.getParentOperators().get(0);
> fsParent.getChildOperators().clear();
> {code}
> The optimizer discards any cases where the fsParent has another SEL child
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21020) log which table/partition is being processed by a txn in Worker
[
https://issues.apache.org/jira/browse/HIVE-21020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-21020:
--
Description:
Make sure we have info in the log that ties cat.table.part with txnid of the
compactor
in Worker
{\{LOG.info("Starting " + ci.type.toString() + " compaction for " +
ci.getFullPartitionName());}}
was:
Make sure we have info in the log that ties cat.table.part with txnid of the
compactor
in Worker
{\{LOG.info("Starting " + ci.type.toString() + " compaction for " +
ci.getFullPartitionName());}}
> log which table/partition is being processed by a txn in Worker
> ---
>
> Key: HIVE-21020
> URL: https://issues.apache.org/jira/browse/HIVE-21020
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Priority: Major
>
> Make sure we have info in the log that ties cat.table.part with txnid of the
> compactor
> in Worker
> {\{LOG.info("Starting " + ci.type.toString() + " compaction for " +
> ci.getFullPartitionName());}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21020) log which table/partition is being processed by a txn in Worker
[
https://issues.apache.org/jira/browse/HIVE-21020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-21020:
--
Description:
Make sure we have info in the log that ties cat.table.part with txnid of the
compactor
in Worker
{\{LOG.info("Starting " + ci.type.toString() + " compaction for " +
ci.getFullPartitionName());}}
was:Make sure we have info in the log that ties cat.table.part with txnid of
the compactor
> log which table/partition is being processed by a txn in Worker
> ---
>
> Key: HIVE-21020
> URL: https://issues.apache.org/jira/browse/HIVE-21020
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Priority: Major
>
> Make sure we have info in the log that ties cat.table.part with txnid of the
> compactor
> in Worker
> {\{LOG.info("Starting " + ci.type.toString() + " compaction for " +
> ci.getFullPartitionName());}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20941) Compactor produces a delete_delta_x_y even if there are no input delete events
[
https://issues.apache.org/jira/browse/HIVE-20941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713456#comment-16713456
]
Hive QA commented on HIVE-20941:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951071/HIVE-20941.05.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15661 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=256)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=256)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15219/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15219/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15219/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951071 - PreCommit-HIVE-Build
> Compactor produces a delete_delta_x_y even if there are no input delete events
> --
>
> Key: HIVE-20941
> URL: https://issues.apache.org/jira/browse/HIVE-20941
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-20941.01.patch, HIVE-20941.02.patch,
> HIVE-20941.03.patch, HIVE-20941.04.patch, HIVE-20941.05.patch
>
>
> see example in HIVE-20901
>
> Probably change logic in CompactorMR.CompactorMap.map() which creates delete
> event writer
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20941) Compactor produces a delete_delta_x_y even if there are no input delete events
[
https://issues.apache.org/jira/browse/HIVE-20941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713441#comment-16713441
]
Hive QA commented on HIVE-20941:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 2 new + 599 unchanged - 3
fixed = 601 total (was 602) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 169
unchanged - 4 fixed = 169 total (was 173) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15219/dev-support/hive-personality.sh
|
| git revision | master / d8c500b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15219/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15219/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Compactor produces a delete_delta_x_y even if there are no input delete events
> --
>
> Key: HIVE-20941
> URL: https://issues.apache.org/jira/browse/HIVE-20941
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-20941.01.patch, HIVE-20941.02.patch,
> HIVE-20941.03.patch, HIVE-20941.04.patch, HIVE-20941.05.patch
>
>
> see example in HIVE-20901
>
> Probably change logic in CompactorMR.CompactorMap.map() which creates delete
> event writer
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21007) Semi join + Union can lead to wrong plans
[ https://issues.apache.org/jira/browse/HIVE-21007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21007: --- Attachment: HIVE-21007.6.patch > Semi join + Union can lead to wrong plans > - > > Key: HIVE-21007 > URL: https://issues.apache.org/jira/browse/HIVE-21007 > Project: Hive > Issue Type: Bug >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21007.1.patch, HIVE-21007.2.patch, > HIVE-21007.3.patch, HIVE-21007.4.patch, HIVE-21007.5.patch, HIVE-21007.6.patch > > > Tez compiler has the ability to push JOIN within UNION (by replicating join > on each branch). If this JOIN had a SJ branch outgoing (or incoming) it could > mess up the plan and end up generating incorrect or wrong plan. > As a safe measure any SJ branch after UNION should be removed (until we > improve the logic to better handle SJ branches) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21019) Fix autoColumnStats tests to make auto stats gather possible.
[ https://issues.apache.org/jira/browse/HIVE-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713414#comment-16713414 ] Vineet Garg edited comment on HIVE-21019 at 12/7/18 11:40 PM: -- [~ychena] I don't think it is a good idea to turn off the optimization in tests. We should fix the underlying issue. I am closing this and marking it as duplicate of HIVE-16100 since that fixes the problem. was (Author: vgarg): [~ychena] I don't think it is not a good idea to turn off the optimization in tests. We should fix the underlying issue. I am closing this and marking it as duplicate of HIVE-16100 since that fixes the problem. > Fix autoColumnStats tests to make auto stats gather possible. > - > > Key: HIVE-21019 > URL: https://issues.apache.org/jira/browse/HIVE-21019 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-21019.1.patch > > > Before https://issues.apache.org/jira/browse/HIVE-20915 , the optimizer sort > dynamic partitions is turn off for these tests. So these test can have group > by in the query plan which can trigger compute statistics. After the jira, > the optimizer is enabled, because of a a known bug with dynamic sorted > partition optimization (HIVE-16100), the computer statistics does not happen. > Fix the tests to disable the optimizer to test the same code path as before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20941) Compactor produces a delete_delta_x_y even if there are no input delete events
[ https://issues.apache.org/jira/browse/HIVE-20941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Kryvenko updated HIVE-20941: - Attachment: HIVE-20941.05.patch > Compactor produces a delete_delta_x_y even if there are no input delete events > -- > > Key: HIVE-20941 > URL: https://issues.apache.org/jira/browse/HIVE-20941 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Major > Attachments: HIVE-20941.01.patch, HIVE-20941.02.patch, > HIVE-20941.03.patch, HIVE-20941.04.patch, HIVE-20941.05.patch > > > see example in HIVE-20901 > > Probably change logic in CompactorMR.CompactorMap.map() which creates delete > event writer -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20955) Calcite Rule HiveExpandDistinctAggregatesRule seems throwing IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-20955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713418#comment-16713418
]
Vineet Garg commented on HIVE-20955:
[~bslim] Do you mind uploading the patch for this? I am still unable to run
druid tests locally to updated the test output.
> Calcite Rule HiveExpandDistinctAggregatesRule seems throwing
> IndexOutOfBoundsException
> --
>
> Key: HIVE-20955
> URL: https://issues.apache.org/jira/browse/HIVE-20955
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20955.1.patch, HIVE-20955.2.patch,
> HIVE-20955.3.patch
>
>
>
> Adde the following query to Druid test
> ql/src/test/queries/clientpositive/druidmini_expressions.q
> {code}
> select count(distinct `__time`, cint) from (select * from
> druid_table_alltypesorc) as src;
> {code}
> leads to error \{code} 2018-11-21T07:36:39,449 ERROR [main] QTestUtil: Client
> execution failed with error code = 4 running "\{code}
> with exception stack
> {code}
> 2018-11-21T07:36:39,443 ERROR [ecd48683-0286-4cb4-b0ad-e150fab51038 main]
> parse.CalcitePlanner: CBO failed, skipping CBO.
> java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:310)
> ~[guava-19.0.jar:?]
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:293)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:41)
> ~[guava-19.0.jar:?]
> at
> org.apache.calcite.rel.metadata.RelMdColumnOrigins.getColumnOrigins(RelMdColumnOrigins.java:77)
> ~[calcite-core-1.17.0.jar:1.17.0]
> at GeneratedMetadataHandler_ColumnOrigin.getColumnOrigins_$(Unknown Source)
> ~[?:?]
> at GeneratedMetadataHandler_ColumnOrigin.getColumnOrigins(Unknown Source)
> ~[?:?]
> at
> org.apache.calcite.rel.metadata.RelMetadataQuery.getColumnOrigins(RelMetadataQuery.java:345)
> ~[calcite-core-1.17.0.jar:1.17.0]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveExpandDistinctAggregatesRule.onMatch(HiveExpandDistinctAggregatesRule.java:168)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:315)
> ~[calcite-core-1.17.0.jar:1.17.0]
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
> ~[calcite-core-1.17.0.jar:1.17.0]
> at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:415)
> ~[calcite-core-1.17.0.jar:1.17.0]
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:280)
> ~[calcite-core-1.17.0.jar:1.17.0]
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74)
> ~[calcite-core-1.17.0.jar:1.17.0]
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:211)
> ~[calcite-core-1.17.0.jar:1.17.0]
> at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:198)
> ~[calcite-core-1.17.0.jar:1.17.0]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:2363)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:2314)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:2031)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1780)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1680)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118)
> ~[calcite-core-1.17.0.jar:1.17.0]
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1043)
> ~[calcite-core-1.17.0.jar:1.17.0]
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154)
> ~[calcite-core-1.17.0.jar:1.17.0]
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111)
> ~[calcite-core-1.17.0.jar:1.17.0]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1439)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:478)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.S
[jira] [Updated] (HIVE-21007) Semi join + Union can lead to wrong plans
[ https://issues.apache.org/jira/browse/HIVE-21007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21007: --- Status: Open (was: Patch Available) > Semi join + Union can lead to wrong plans > - > > Key: HIVE-21007 > URL: https://issues.apache.org/jira/browse/HIVE-21007 > Project: Hive > Issue Type: Bug >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21007.1.patch, HIVE-21007.2.patch, > HIVE-21007.3.patch, HIVE-21007.4.patch, HIVE-21007.5.patch, HIVE-21007.6.patch > > > Tez compiler has the ability to push JOIN within UNION (by replicating join > on each branch). If this JOIN had a SJ branch outgoing (or incoming) it could > mess up the plan and end up generating incorrect or wrong plan. > As a safe measure any SJ branch after UNION should be removed (until we > improve the logic to better handle SJ branches) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20961) Retire NVL implementation
[ https://issues.apache.org/jira/browse/HIVE-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-20961: Attachment: HIVE-20961.04.patch > Retire NVL implementation > - > > Key: HIVE-20961 > URL: https://issues.apache.org/jira/browse/HIVE-20961 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20961.01.patch, HIVE-20961.02.patch, > HIVE-20961.03.patch, HIVE-20961.04.patch > > > Right now we have coalesce and nvl implemented separetly; it might be better > to remove one of them as they are doing the same. Because Coalesce is in the > standard - I think NVL have to go...and became an alias to Coalesce. > Further benefit is: that optimizations which coalesce already recieves; will > be done on NVL calls as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21007) Semi join + Union can lead to wrong plans
[ https://issues.apache.org/jira/browse/HIVE-21007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21007: --- Status: Patch Available (was: Open) > Semi join + Union can lead to wrong plans > - > > Key: HIVE-21007 > URL: https://issues.apache.org/jira/browse/HIVE-21007 > Project: Hive > Issue Type: Bug >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21007.1.patch, HIVE-21007.2.patch, > HIVE-21007.3.patch, HIVE-21007.4.patch, HIVE-21007.5.patch, HIVE-21007.6.patch > > > Tez compiler has the ability to push JOIN within UNION (by replicating join > on each branch). If this JOIN had a SJ branch outgoing (or incoming) it could > mess up the plan and end up generating incorrect or wrong plan. > As a safe measure any SJ branch after UNION should be removed (until we > improve the logic to better handle SJ branches) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-21019) Fix autoColumnStats tests to make auto stats gather possible.
[ https://issues.apache.org/jira/browse/HIVE-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg resolved HIVE-21019. Resolution: Duplicate > Fix autoColumnStats tests to make auto stats gather possible. > - > > Key: HIVE-21019 > URL: https://issues.apache.org/jira/browse/HIVE-21019 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-21019.1.patch > > > Before https://issues.apache.org/jira/browse/HIVE-20915 , the optimizer sort > dynamic partitions is turn off for these tests. So these test can have group > by in the query plan which can trigger compute statistics. After the jira, > the optimizer is enabled, because of a a known bug with dynamic sorted > partition optimization (HIVE-16100), the computer statistics does not happen. > Fix the tests to disable the optimizer to test the same code path as before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21019) Fix autoColumnStats tests to make auto stats gather possible.
[ https://issues.apache.org/jira/browse/HIVE-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713414#comment-16713414 ] Vineet Garg commented on HIVE-21019: [~ychena] I don't think it is not a good idea to turn off the optimization in tests. We should fix the underlying issue. I am closing this and marking it as duplicate of HIVE-16100 since that fixes the problem. > Fix autoColumnStats tests to make auto stats gather possible. > - > > Key: HIVE-21019 > URL: https://issues.apache.org/jira/browse/HIVE-21019 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-21019.1.patch > > > Before https://issues.apache.org/jira/browse/HIVE-20915 , the optimizer sort > dynamic partitions is turn off for these tests. So these test can have group > by in the query plan which can trigger compute statistics. After the jira, > the optimizer is enabled, because of a a known bug with dynamic sorted > partition optimization (HIVE-16100), the computer statistics does not happen. > Fix the tests to disable the optimizer to test the same code path as before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16100) Dynamic Sorted Partition optimizer loses sibling operators
[
https://issues.apache.org/jira/browse/HIVE-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713410#comment-16713410
]
Vineet Garg commented on HIVE-16100:
I realized that earlier patch provided by [~gopalv] was trying to fix two
different problems 1) SDPO losing sibling and 2) reducer de-dup incorrectly
removing branch.
Fix for 2) issue is causing correctness issues in order by queries (e.g.
mm_all.q).
IMHO we should first fix 1) and then investigate and fix 2) in separate JIRA.
I have uploaded a new [^HIVE-16100.8.patch] which should fix stats auto gather
problem with SDPO. Let us use HIVE-17020 to fix 2)
> Dynamic Sorted Partition optimizer loses sibling operators
> --
>
> Key: HIVE-16100
> URL: https://issues.apache.org/jira/browse/HIVE-16100
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16100.1.patch, HIVE-16100.2.patch,
> HIVE-16100.2.patch, HIVE-16100.3.patch, HIVE-16100.4.patch,
> HIVE-16100.5.patch, HIVE-16100.6.patch, HIVE-16100.7.patch, HIVE-16100.8.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L173
> {code}
> // unlink connection between FS and its parent
> fsParent = fsOp.getParentOperators().get(0);
> fsParent.getChildOperators().clear();
> {code}
> The optimizer discards any cases where the fsParent has another SEL child
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20961) Retire NVL implementation
[
https://issues.apache.org/jira/browse/HIVE-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713407#comment-16713407
]
Hive QA commented on HIVE-20961:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951037/HIVE-20961.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15218/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15218/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15218/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12951037/HIVE-20961.03.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951037 - PreCommit-HIVE-Build
> Retire NVL implementation
> -
>
> Key: HIVE-20961
> URL: https://issues.apache.org/jira/browse/HIVE-20961
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-20961.01.patch, HIVE-20961.02.patch,
> HIVE-20961.03.patch
>
>
> Right now we have coalesce and nvl implemented separetly; it might be better
> to remove one of them as they are doing the same. Because Coalesce is in the
> standard - I think NVL have to go...and became an alias to Coalesce.
> Further benefit is: that optimizations which coalesce already recieves; will
> be done on NVL calls as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16100) Dynamic Sorted Partition optimizer loses sibling operators
[
https://issues.apache.org/jira/browse/HIVE-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16100:
---
Status: Open (was: Patch Available)
> Dynamic Sorted Partition optimizer loses sibling operators
> --
>
> Key: HIVE-16100
> URL: https://issues.apache.org/jira/browse/HIVE-16100
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.2.0, 2.1.1, 1.2.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16100.1.patch, HIVE-16100.2.patch,
> HIVE-16100.2.patch, HIVE-16100.3.patch, HIVE-16100.4.patch,
> HIVE-16100.5.patch, HIVE-16100.6.patch, HIVE-16100.7.patch, HIVE-16100.8.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L173
> {code}
> // unlink connection between FS and its parent
> fsParent = fsOp.getParentOperators().get(0);
> fsParent.getChildOperators().clear();
> {code}
> The optimizer discards any cases where the fsParent has another SEL child
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16100) Dynamic Sorted Partition optimizer loses sibling operators
[
https://issues.apache.org/jira/browse/HIVE-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16100:
---
Attachment: HIVE-16100.8.patch
> Dynamic Sorted Partition optimizer loses sibling operators
> --
>
> Key: HIVE-16100
> URL: https://issues.apache.org/jira/browse/HIVE-16100
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 1.2.1, 2.1.1, 2.2.0
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16100.1.patch, HIVE-16100.2.patch,
> HIVE-16100.2.patch, HIVE-16100.3.patch, HIVE-16100.4.patch,
> HIVE-16100.5.patch, HIVE-16100.6.patch, HIVE-16100.7.patch, HIVE-16100.8.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L173
> {code}
> // unlink connection between FS and its parent
> fsParent = fsOp.getParentOperators().get(0);
> fsParent.getChildOperators().clear();
> {code}
> The optimizer discards any cases where the fsParent has another SEL child
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20961) Retire NVL implementation
[
https://issues.apache.org/jira/browse/HIVE-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713404#comment-16713404
]
Hive QA commented on HIVE-20961:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951037/HIVE-20961.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15659 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[schemeAuthority2]
(batchId=192)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15217/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15217/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15217/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951037 - PreCommit-HIVE-Build
> Retire NVL implementation
> -
>
> Key: HIVE-20961
> URL: https://issues.apache.org/jira/browse/HIVE-20961
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-20961.01.patch, HIVE-20961.02.patch,
> HIVE-20961.03.patch
>
>
> Right now we have coalesce and nvl implemented separetly; it might be better
> to remove one of them as they are doing the same. Because Coalesce is in the
> standard - I think NVL have to go...and became an alias to Coalesce.
> Further benefit is: that optimizations which coalesce already recieves; will
> be done on NVL calls as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16100) Dynamic Sorted Partition optimizer loses sibling operators
[
https://issues.apache.org/jira/browse/HIVE-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16100:
---
Status: Patch Available (was: Open)
> Dynamic Sorted Partition optimizer loses sibling operators
> --
>
> Key: HIVE-16100
> URL: https://issues.apache.org/jira/browse/HIVE-16100
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.2.0, 2.1.1, 1.2.1
>Reporter: Gopal V
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16100.1.patch, HIVE-16100.2.patch,
> HIVE-16100.2.patch, HIVE-16100.3.patch, HIVE-16100.4.patch,
> HIVE-16100.5.patch, HIVE-16100.6.patch, HIVE-16100.7.patch, HIVE-16100.8.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java#L173
> {code}
> // unlink connection between FS and its parent
> fsParent = fsOp.getParentOperators().get(0);
> fsParent.getChildOperators().clear();
> {code}
> The optimizer discards any cases where the fsParent has another SEL child
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20942) Worker should heartbeat its own txn
[
https://issues.apache.org/jira/browse/HIVE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-20942:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Release Note: n/a
Status: Resolved (was: Patch Available)
committed to master
thanks Igor for the contribution
> Worker should heartbeat its own txn
> ---
>
> Key: HIVE-20942
> URL: https://issues.apache.org/jira/browse/HIVE-20942
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20942.01.patch, HIVE-20942.02.patch,
> HIVE-20942.03.patch, HIVE-20942.04.patch
>
>
> Since HIVE-20823 \{{Worker.java}} starts a txn - should either add a
> heartbeat thread to it or use HiveTxnManager to start txn which will set up
> heartbeat automatically. In the later case make sure it's properly cancelled
> on failures.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20942) Worker should heartbeat its own txn
[
https://issues.apache.org/jira/browse/HIVE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713392#comment-16713392
]
Eugene Koifman edited comment on HIVE-20942 at 12/7/18 11:06 PM:
-
committed to master HIVE-20942.04.patch
thanks Igor for the contribution
was (Author: ekoifman):
committed to master
thanks Igor for the contribution
> Worker should heartbeat its own txn
> ---
>
> Key: HIVE-20942
> URL: https://issues.apache.org/jira/browse/HIVE-20942
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20942.01.patch, HIVE-20942.02.patch,
> HIVE-20942.03.patch, HIVE-20942.04.patch
>
>
> Since HIVE-20823 \{{Worker.java}} starts a txn - should either add a
> heartbeat thread to it or use HiveTxnManager to start txn which will set up
> heartbeat automatically. In the later case make sure it's properly cancelled
> on failures.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20942) Worker should heartbeat its own txn
[
https://issues.apache.org/jira/browse/HIVE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713390#comment-16713390
]
Eugene Koifman commented on HIVE-20942:
---
+1
> Worker should heartbeat its own txn
> ---
>
> Key: HIVE-20942
> URL: https://issues.apache.org/jira/browse/HIVE-20942
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-20942.01.patch, HIVE-20942.02.patch,
> HIVE-20942.03.patch, HIVE-20942.04.patch
>
>
> Since HIVE-20823 \{{Worker.java}} starts a txn - should either add a
> heartbeat thread to it or use HiveTxnManager to start txn which will set up
> heartbeat automatically. In the later case make sure it's properly cancelled
> on failures.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20961) Retire NVL implementation
[
https://issues.apache.org/jira/browse/HIVE-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713387#comment-16713387
]
Hive QA commented on HIVE-20961:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 1 new + 1055 unchanged - 6
fixed = 1056 total (was 1061) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} ql generated 0 new + 2310 unchanged - 1 fixed = 2310
total (was 2311) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} accumulo-handler in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} hbase-handler in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15217/dev-support/hive-personality.sh
|
| git revision | master / aae392f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15217/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql accumulo-handler hbase-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15217/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Retire NVL implementation
> -
>
> Key: HIVE-20961
> URL: https://issues.apache.org/jira/browse/HIVE-20961
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-20961.01.patch, HIVE-20961.02.patch,
> HIVE-20961.03.patch
>
>
> Right now we have coalesce and nvl implemented separetly; it might be better
> to remove one of them
[jira] [Updated] (HIVE-21017) Throw proper exception in case of alter table add partition on an ACID table
[ https://issues.apache.org/jira/browse/HIVE-21017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-21017: -- Description: I don't think ACID is relevant here. The issue is adding existing data to a bucketed table via setting location. > Throw proper exception in case of alter table add partition on an ACID table > > > Key: HIVE-21017 > URL: https://issues.apache.org/jira/browse/HIVE-21017 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > > I don't think ACID is relevant here. The issue is adding existing data to a > bucketed table via setting location. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20966) Support bootstrap and incremental replication to a target with hive.strict.managed.tables enabled.
[
https://issues.apache.org/jira/browse/HIVE-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713357#comment-16713357
]
Hive QA commented on HIVE-20966:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951009/HIVE-20966.04.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 15646 tests
executed
*Failed tests:*
{noformat}
TestAlterTableMetadata - did not produce a TEST-*.xml file (likely timed out)
(batchId=248)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=248)
TestReplAcidTablesWithJsonMessage - did not produce a TEST-*.xml file (likely
timed out) (batchId=248)
TestReplIncrementalLoadAcidTablesWithJsonMessage - did not produce a TEST-*.xml
file (likely timed out) (batchId=248)
TestReplicationScenariosMigration - did not produce a TEST-*.xml file (likely
timed out) (batchId=248)
TestSemanticAnalyzerHookLoading - did not produce a TEST-*.xml file (likely
timed out) (batchId=248)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15216/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15216/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15216/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951009 - PreCommit-HIVE-Build
> Support bootstrap and incremental replication to a target with
> hive.strict.managed.tables enabled.
> --
>
> Key: HIVE-20966
> URL: https://issues.apache.org/jira/browse/HIVE-20966
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: DR
> Attachments: HIVE-20966.01.patch, HIVE-20966.02.patch,
> HIVE-20966.03.patch, HIVE-20966.04.patch, HIVE-20966.04.patch
>
>
> *Requirements:*
> Hive2 supports replication of managed tables. But in Hive3 with
> hive.strict.managed.tables=true, some of these managed tables are converted
> to ACID or MM tables. Also, some of them are converted to external tables
> based on below rules.
> - Avro format with external schema, Storage handlers, List bucketed tabled
> are converted to external tables.
> - Location not owned by "hive" user are converted to external table.
> - Hive owned ORC format are converted to full ACID transactional table.
> - Hive owned Non-ORC format are converted to MM transactional table.
> REPL LOAD should apply these rules during bootstrap and incremental phases
> and convert the tables accordingly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20966) Support bootstrap and incremental replication to a target with hive.strict.managed.tables enabled.
[
https://issues.apache.org/jira/browse/HIVE-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713320#comment-16713320
]
Hive QA commented on HIVE-20966:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
14s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
5s{color} | {color:blue} standalone-metastore/metastore-server in master has
184 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} hcatalog/server-extensions in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} common: The patch generated 1 new + 427 unchanged - 0
fixed = 428 total (was 427) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 55 new + 854 unchanged - 11
fixed = 909 total (was 865) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} itests/hive-unit: The patch generated 145 new + 658
unchanged - 58 fixed = 803 total (was 716) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 16 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red} standalone-metastore/metastore-server generated 4 new
+ 184 unchanged - 0 fixed = 188 total (was 184) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
3s{color} | {color:red} ql generated 1 new + 2310 unchanged - 1 fixed = 2311
total (was 2311) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
33s{color} | {color:red} hcatalog/server-extensions generated 2 new + 1
unchanged - 0 fixed = 3 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Boxed value is unboxed and then immediately reboxed in
org.apache.hadoop.hive.metastore.messaging.json.JSONAlte
[jira] [Commented] (HIVE-20613) CachedStore: Add more UT coverage (outside of .q files)
[
https://issues.apache.org/jira/browse/HIVE-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713281#comment-16713281
]
Hive QA commented on HIVE-20613:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951003/HIVE-20613.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15667 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15215/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15215/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15215/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951003 - PreCommit-HIVE-Build
> CachedStore: Add more UT coverage (outside of .q files)
> ---
>
> Key: HIVE-20613
> URL: https://issues.apache.org/jira/browse/HIVE-20613
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20613.1.patch, HIVE-20613.2.patch,
> HIVE-20613.2.patch
>
>
> 1. Add tests which will use the background thread for updating the cached
> data (database, table, partition, table stats, partition stats)
> 2. Add more tests for existing APIs: stats aggregation, listing partitions
> when partial specs are provided, testing the storage descriptor
> caching/deduplication (specially when tables/ptns are dropped/added), table
> col stats, partition col stats
> 3. Test 1. in a multithreaded scenario
> 4. Test whitelist/blacklist
> 5. Test prewarm memory limit estimation code
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20961) Retire NVL implementation
[ https://issues.apache.org/jira/browse/HIVE-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-20961: Attachment: HIVE-20961.03.patch > Retire NVL implementation > - > > Key: HIVE-20961 > URL: https://issues.apache.org/jira/browse/HIVE-20961 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20961.01.patch, HIVE-20961.02.patch, > HIVE-20961.03.patch > > > Right now we have coalesce and nvl implemented separetly; it might be better > to remove one of them as they are doing the same. Because Coalesce is in the > standard - I think NVL have to go...and became an alias to Coalesce. > Further benefit is: that optimizations which coalesce already recieves; will > be done on NVL calls as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20961) Retire NVL implementation
[ https://issues.apache.org/jira/browse/HIVE-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-20961: Attachment: (was: HIVE-20961.03.patch) > Retire NVL implementation > - > > Key: HIVE-20961 > URL: https://issues.apache.org/jira/browse/HIVE-20961 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20961.01.patch, HIVE-20961.02.patch > > > Right now we have coalesce and nvl implemented separetly; it might be better > to remove one of them as they are doing the same. Because Coalesce is in the > standard - I think NVL have to go...and became an alias to Coalesce. > Further benefit is: that optimizations which coalesce already recieves; will > be done on NVL calls as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20961) Retire NVL implementation
[ https://issues.apache.org/jira/browse/HIVE-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-20961: Attachment: HIVE-20961.03.patch > Retire NVL implementation > - > > Key: HIVE-20961 > URL: https://issues.apache.org/jira/browse/HIVE-20961 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20961.01.patch, HIVE-20961.02.patch, > HIVE-20961.03.patch > > > Right now we have coalesce and nvl implemented separetly; it might be better > to remove one of them as they are doing the same. Because Coalesce is in the > standard - I think NVL have to go...and became an alias to Coalesce. > Further benefit is: that optimizations which coalesce already recieves; will > be done on NVL calls as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20961) Retire NVL implementation
[ https://issues.apache.org/jira/browse/HIVE-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-20961: Attachment: (was: HIVE-20961.03.patch) > Retire NVL implementation > - > > Key: HIVE-20961 > URL: https://issues.apache.org/jira/browse/HIVE-20961 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20961.01.patch, HIVE-20961.02.patch > > > Right now we have coalesce and nvl implemented separetly; it might be better > to remove one of them as they are doing the same. Because Coalesce is in the > standard - I think NVL have to go...and became an alias to Coalesce. > Further benefit is: that optimizations which coalesce already recieves; will > be done on NVL calls as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20961) Retire NVL implementation
[ https://issues.apache.org/jira/browse/HIVE-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-20961: Attachment: HIVE-20961.03.patch > Retire NVL implementation > - > > Key: HIVE-20961 > URL: https://issues.apache.org/jira/browse/HIVE-20961 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20961.01.patch, HIVE-20961.02.patch > > > Right now we have coalesce and nvl implemented separetly; it might be better > to remove one of them as they are doing the same. Because Coalesce is in the > standard - I think NVL have to go...and became an alias to Coalesce. > Further benefit is: that optimizations which coalesce already recieves; will > be done on NVL calls as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20613) CachedStore: Add more UT coverage (outside of .q files)
[
https://issues.apache.org/jira/browse/HIVE-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713237#comment-16713237
]
Hive QA commented on HIVE-20613:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
2s{color} | {color:blue} standalone-metastore/metastore-server in master has
184 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 14 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} standalone-metastore/metastore-server generated 0 new
+ 183 unchanged - 1 fixed = 183 total (was 184) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15215/dev-support/hive-personality.sh
|
| git revision | master / aae392f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15215/yetus/whitespace-eol.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15215/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CachedStore: Add more UT coverage (outside of .q files)
> ---
>
> Key: HIVE-20613
> URL: https://issues.apache.org/jira/browse/HIVE-20613
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20613.1.patch, HIVE-20613.2.patch,
> HIVE-20613.2.patch
>
>
> 1. Add tests which will use the background thread for updating the cached
> data (database, table, partition, table stats, partition stats)
> 2. Add more tests for existing APIs: stats aggregation, listing partitions
> when partial specs are provided, testing the storage descriptor
> caching/deduplication (specially when tables/ptns are dropped/added), table
> col stats, partition col stats
> 3. Test 1. in a multithreaded scenario
> 4. Test whitelist/blacklist
> 5. Test prewarm memory limit estimation code
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20615) CachedStore: Background refresh thread bug fixes
[
https://issues.apache.org/jira/browse/HIVE-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713225#comment-16713225
]
Hive QA commented on HIVE-20615:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951004/HIVE-20615.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 51 failed/errored test(s), 15641 tests
executed
*Failed tests:*
{noformat}
TestCachedStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestCatalogCaching - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestDeadline - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestMetaStoreEventListenerOnlyOnCommit - did not produce a TEST-*.xml file
(likely timed out) (batchId=227)
TestMetaStoreListenersError - did not produce a TEST-*.xml file (likely timed
out) (batchId=227)
TestMetaStoreSchemaInfo - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterPartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterTable
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterTableCascade
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterViewParititon
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testColumnStatistics
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testComplexTable
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testComplexTypeApi
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testConcurrentMetastores
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testCreateAndGetTableWithDriver
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testCreateTableSettingId
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBLocationChange
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBOwner
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBOwnerChange
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabase
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabaseLocation
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabaseLocationWithPermissionProblems
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDropDatabaseCascadeMVMultiDB
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDropTable
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFilterLastPartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFilterSinglePartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFunctionWithResources
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetConfigValue
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetMetastoreUuid
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetPartitionsWithSpec
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetSchemaWithNoClassDefFoundError
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetTableObjects
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetUUIDInParallel
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testJDOPersistanceManagerCleanup
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitionNames
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitions
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitionsWihtLimitEnabled
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testNameMethods
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testPartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testPartitionFilter
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testRenamePartition
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testRetriableClientWithConnLifetime
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleFunction
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleTable
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleTypeApi
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testStatsFastTrivial
(batchId=227)
org.apache.had
[jira] [Updated] (HIVE-21013) JdbcStorageHandler fail to find partition column in Oracle
[
https://issues.apache.org/jira/browse/HIVE-21013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-21013:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Patch pushed to master. Thanks Jesus for review!
> JdbcStorageHandler fail to find partition column in Oracle
> --
>
> Key: HIVE-21013
> URL: https://issues.apache.org/jira/browse/HIVE-21013
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21013.1.patch
>
>
> Stack:
> {code}
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1543830849610_0048_1_00, diagnostics=[Task failed,
> taskId=task_1543830849610_0048_1_00_05, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1543830849610_0048_1_00_05_0:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException:
> org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException:
> Caught exception while trying to execute query:Cannot find salaries in sql
> query salaries
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException:
> org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException:
> Caught exception while trying to execute query:Cannot find salaries in sql
> query salaries
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:80)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> ... 16 more
> Caused by: java.io.IOException: java.io.IOException:
> org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException:
> Caught exception while trying to execute query:Cannot find salaries in sql
> query salaries
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> ... 18 more
> Caused by: java.io.IOException:
[jira] [Work started] (HIVE-21019) Fix autoColumnStats tests to make auto stats gather possible.
[ https://issues.apache.org/jira/browse/HIVE-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21019 started by Yongzhi Chen. --- > Fix autoColumnStats tests to make auto stats gather possible. > - > > Key: HIVE-21019 > URL: https://issues.apache.org/jira/browse/HIVE-21019 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > > Before https://issues.apache.org/jira/browse/HIVE-20915 , the optimizer sort > dynamic partitions is turn off for these tests. So these test can have group > by in the query plan which can trigger compute statistics. After the jira, > the optimizer is enabled, because of a a known bug with dynamic sorted > partition optimization (HIVE-16100), the computer statistics does not happen. > Fix the tests to disable the optimizer to test the same code path as before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21019) Fix autoColumnStats tests to make auto stats gather possible.
[ https://issues.apache.org/jira/browse/HIVE-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-21019: Attachment: HIVE-21019.1.patch > Fix autoColumnStats tests to make auto stats gather possible. > - > > Key: HIVE-21019 > URL: https://issues.apache.org/jira/browse/HIVE-21019 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-21019.1.patch > > > Before https://issues.apache.org/jira/browse/HIVE-20915 , the optimizer sort > dynamic partitions is turn off for these tests. So these test can have group > by in the query plan which can trigger compute statistics. After the jira, > the optimizer is enabled, because of a a known bug with dynamic sorted > partition optimization (HIVE-16100), the computer statistics does not happen. > Fix the tests to disable the optimizer to test the same code path as before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20615) CachedStore: Background refresh thread bug fixes
[
https://issues.apache.org/jira/browse/HIVE-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713184#comment-16713184
]
Hive QA commented on HIVE-20615:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
3s{color} | {color:blue} standalone-metastore/metastore-server in master has
184 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15214/dev-support/hive-personality.sh
|
| git revision | master / 4f3377c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15214/yetus/whitespace-eol.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15214/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CachedStore: Background refresh thread bug fixes
>
>
> Key: HIVE-20615
> URL: https://issues.apache.org/jira/browse/HIVE-20615
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20615.1.patch, HIVE-20615.1.patch,
> HIVE-20615.1.patch, HIVE-20615.1.patch, HIVE-20615.1.patch, HIVE-20615.1.patch
>
>
> Regression introduced in HIVE-18264. Fixes background thread starting and
> refreshing of the table cache.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21019) Fix autoColumnStats tests to make auto stats gather possible.
[ https://issues.apache.org/jira/browse/HIVE-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-21019: Description: Before https://issues.apache.org/jira/browse/HIVE-20915 , the optimizer sort dynamic partitions is turn off for these tests. So these test can have group by in the query plan which can trigger compute statistics. After the jira, the optimizer is enabled, because of a a known bug with dynamic sorted partition optimization (HIVE-16100), the computer statistics does not happen. Fix the tests to disable the optimizer to test the same code path as before. (was: Before https://issues.apache.org/jira/browse/HIVE-20915 , the optimizer sort dynamic partitions is turn off for these tests. So these test can have group by in the query plan which can trigger compute statistics. After the jira, the optimizer is enabled, because of a bug of ) > Fix autoColumnStats tests to make auto stats gather possible. > - > > Key: HIVE-21019 > URL: https://issues.apache.org/jira/browse/HIVE-21019 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > > Before https://issues.apache.org/jira/browse/HIVE-20915 , the optimizer sort > dynamic partitions is turn off for these tests. So these test can have group > by in the query plan which can trigger compute statistics. After the jira, > the optimizer is enabled, because of a a known bug with dynamic sorted > partition optimization (HIVE-16100), the computer statistics does not happen. > Fix the tests to disable the optimizer to test the same code path as before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21019) Fix autoColumnStats tests to make auto stats gather possible.
[ https://issues.apache.org/jira/browse/HIVE-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-21019: Description: Before https://issues.apache.org/jira/browse/HIVE-20915 , the optimizer sort dynamic partitions is turn off for these tests. So these test can have group by in the query plan which can trigger compute statistics. After the jira, the optimizer is enabled, because of a bug of (was: Before https://issues.apache.org/jira/browse/HIVE-20915 , the optimizer sort dynamic partitions is turn off for these tests. So these test can have group by in the query plan which can trigger compute statistics. After the jira, the optimizer is enabled, the query plan do not have group by, but a reduce sorting operation. In order to test the auto column stats gather feature, we should disable sort dynamic partitions for these tests. ) > Fix autoColumnStats tests to make auto stats gather possible. > - > > Key: HIVE-21019 > URL: https://issues.apache.org/jira/browse/HIVE-21019 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > > Before https://issues.apache.org/jira/browse/HIVE-20915 , the optimizer sort > dynamic partitions is turn off for these tests. So these test can have group > by in the query plan which can trigger compute statistics. After the jira, > the optimizer is enabled, because of a bug of -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21018) Grouping/distinct on more than 64 columns should be possible
[
https://issues.apache.org/jira/browse/HIVE-21018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713166#comment-16713166
]
Hive QA commented on HIVE-21018:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12950993/HIVE-21018.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 47 failed/errored test(s), 15661 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[distinct_66] (batchId=11)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTableCascade
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterViewParititon
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testColumnStatistics
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTypeApi
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testConcurrentMetastores
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateAndGetTableWithDriver
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateTableSettingId
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBLocationChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwner
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwnerChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabase
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocation
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocationWithPermissionProblems
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropDatabaseCascadeMVMultiDB
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterLastPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterSinglePartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFunctionWithResources
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetConfigValue
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetMetastoreUuid
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetPartitionsWithSpec
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetSchemaWithNoClassDefFoundError
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetTableObjects
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetUUIDInParallel
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testJDOPersistanceManagerCleanup
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionNames
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitions
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testListPartitionsWihtLimitEnabled
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testNameMethods
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartitionFilter
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRenamePartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testRetriableClientWithConnLifetime
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleFunction
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSimpleTypeApi
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testStatsFastTrivial
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testSyn
[jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713173#comment-16713173 ] Yongzhi Chen commented on HIVE-20915: - [~vgarg], thanks for clarifying. > Make dynamic sort partition optimization available to HoS and MR > > > Key: HIVE-20915 > URL: https://issues.apache.org/jira/browse/HIVE-20915 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch, > HIVE-20915.3.patch, HIVE-20915.4.patch, HIVE-20915.5.patch > > > HIVE-20703 put dynamic sort partition optimization under cost based decision, > but it also makes the optimizer only available to tez. > hive.optimize.sort.dynamic.partition works with other execution engines for a > long time, we should keep the optimizer available to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713128#comment-16713128 ] Vineet Garg commented on HIVE-20915: This is a known bug with dynamic sorted partition optimization. I am currently working on fixing it. (HIVE-16100). > Make dynamic sort partition optimization available to HoS and MR > > > Key: HIVE-20915 > URL: https://issues.apache.org/jira/browse/HIVE-20915 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch, > HIVE-20915.3.patch, HIVE-20915.4.patch, HIVE-20915.5.patch > > > HIVE-20703 put dynamic sort partition optimization under cost based decision, > but it also makes the optimizer only available to tez. > hive.optimize.sort.dynamic.partition works with other execution engines for a > long time, we should keep the optimizer available to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21018) Grouping/distinct on more than 64 columns should be possible
[
https://issues.apache.org/jira/browse/HIVE-21018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713126#comment-16713126
]
Hive QA commented on HIVE-21018:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} ql: The patch generated 0 new + 458 unchanged - 1
fixed = 458 total (was 459) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15213/dev-support/hive-personality.sh
|
| git revision | master / 4f3377c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15213/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Grouping/distinct on more than 64 columns should be possible
>
>
> Key: HIVE-21018
> URL: https://issues.apache.org/jira/browse/HIVE-21018
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21018.01.patch
>
>
> Earlier when more than 64 columns were in a group by or distinct an error was
> given
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21011) Upgrade MurmurHash 2.0 to 3.0 in vectorized map and reduce operators
[
https://issues.apache.org/jira/browse/HIVE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713103#comment-16713103
]
Hive QA commented on HIVE-21011:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12950980/HIVE-21011.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15659 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_limit]
(batchId=171)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15212/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15212/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15212/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12950980 - PreCommit-HIVE-Build
> Upgrade MurmurHash 2.0 to 3.0 in vectorized map and reduce operators
>
>
> Key: HIVE-21011
> URL: https://issues.apache.org/jira/browse/HIVE-21011
> Project: Hive
> Issue Type: Improvement
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21011.1.patch, HIVE-21011.2.patch
>
>
> HIVE-20873 improved map join performance by using MurmurHash 3.0. However,
> there's more operators that can use it. VectorMapJoinCommonOperator and
> VectorReduceSinkUniformHashOperator use MurmurHash 2.0, so it can be upgraded
> to MurmurHash 3.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21011) Upgrade MurmurHash 2.0 to 3.0 in vectorized map and reduce operators
[
https://issues.apache.org/jira/browse/HIVE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713059#comment-16713059
]
Hive QA commented on HIVE-21011:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} common: The patch generated 0 new + 3 unchanged - 3
fixed = 3 total (was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15212/dev-support/hive-personality.sh
|
| git revision | master / 4f3377c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15212/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade MurmurHash 2.0 to 3.0 in vectorized map and reduce operators
>
>
> Key: HIVE-21011
> URL: https://issues.apache.org/jira/browse/HIVE-21011
> Project: Hive
> Issue Type: Improvement
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21011.1.patch, HIVE-21011.2.patch
>
>
> HIVE-20873 improved map join performance by using MurmurHash 3.0. However,
> there's more operators that can use it. VectorMapJoinCommonOperator and
> VectorReduceSinkUniformHashOperator use MurmurHash 2.0, so it can be upgraded
> to MurmurHash 3.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713058#comment-16713058 ] Yongzhi Chen commented on HIVE-20915: - [~kgyrtkirk], the optimizer uses one reducer to sort the output and put it into one file (one for each partition, so no need for group by). The behavior is always like this. You can check old patch(not-checked in) which try to enable the optimizer by default, enable the optimizer will remove the auto gather thing. https://issues.apache.org/jira/secure/attachment/12898098/HIVE-17935.7.patch#file-5 > Make dynamic sort partition optimization available to HoS and MR > > > Key: HIVE-20915 > URL: https://issues.apache.org/jira/browse/HIVE-20915 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch, > HIVE-20915.3.patch, HIVE-20915.4.patch, HIVE-20915.5.patch > > > HIVE-20703 put dynamic sort partition optimization under cost based decision, > but it also makes the optimizer only available to tez. > hive.optimize.sort.dynamic.partition works with other execution engines for a > long time, we should keep the optimizer available to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21019) Fix autoColumnStats tests to make auto stats gather possible.
[ https://issues.apache.org/jira/browse/HIVE-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen reassigned HIVE-21019: --- > Fix autoColumnStats tests to make auto stats gather possible. > - > > Key: HIVE-21019 > URL: https://issues.apache.org/jira/browse/HIVE-21019 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > > Before https://issues.apache.org/jira/browse/HIVE-20915 , the optimizer sort > dynamic partitions is turn off for these tests. So these test can have group > by in the query plan which can trigger compute statistics. After the jira, > the optimizer is enabled, the query plan do not have group by, but a reduce > sorting operation. In order to test the auto column stats gather feature, we > should disable sort dynamic partitions for these tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[
https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713028#comment-16713028
]
Zoltan Haindrich commented on HIVE-20915:
-
{quote}I will create a jira to update the test to use set
hive.optimize.sort.dynamic.partition.threshold=-1, is that OK Zoltan Haindrich
?{quote}
Sorry, but I don't think that would be a good idea...I might be misinterpreting
your comments/point of view. cc: [~jcamachorodriguez], [~ashutoshc]
> Make dynamic sort partition optimization available to HoS and MR
>
>
> Key: HIVE-20915
> URL: https://issues.apache.org/jira/browse/HIVE-20915
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch,
> HIVE-20915.3.patch, HIVE-20915.4.patch, HIVE-20915.5.patch
>
>
> HIVE-20703 put dynamic sort partition optimization under cost based decision,
> but it also makes the optimizer only available to tez.
> hive.optimize.sort.dynamic.partition works with other execution engines for a
> long time, we should keep the optimizer available to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20942) Worker should heartbeat its own txn
[
https://issues.apache.org/jira/browse/HIVE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713027#comment-16713027
]
Hive QA commented on HIVE-20942:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12950969/HIVE-20942.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15650 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15211/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15211/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15211/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12950969 - PreCommit-HIVE-Build
> Worker should heartbeat its own txn
> ---
>
> Key: HIVE-20942
> URL: https://issues.apache.org/jira/browse/HIVE-20942
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-20942.01.patch, HIVE-20942.02.patch,
> HIVE-20942.03.patch, HIVE-20942.04.patch
>
>
> Since HIVE-20823 \{{Worker.java}} starts a txn - should either add a
> heartbeat thread to it or use HiveTxnManager to start txn which will set up
> heartbeat automatically. In the later case make sure it's properly cancelled
> on failures.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20966) Support bootstrap and incremental replication to a target with hive.strict.managed.tables enabled.
[ https://issues.apache.org/jira/browse/HIVE-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20966: --- Status: Patch Available (was: Open) > Support bootstrap and incremental replication to a target with > hive.strict.managed.tables enabled. > -- > > Key: HIVE-20966 > URL: https://issues.apache.org/jira/browse/HIVE-20966 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR > Attachments: HIVE-20966.01.patch, HIVE-20966.02.patch, > HIVE-20966.03.patch, HIVE-20966.04.patch, HIVE-20966.04.patch > > > *Requirements:* > Hive2 supports replication of managed tables. But in Hive3 with > hive.strict.managed.tables=true, some of these managed tables are converted > to ACID or MM tables. Also, some of them are converted to external tables > based on below rules. > - Avro format with external schema, Storage handlers, List bucketed tabled > are converted to external tables. > - Location not owned by "hive" user are converted to external table. > - Hive owned ORC format are converted to full ACID transactional table. > - Hive owned Non-ORC format are converted to MM transactional table. > REPL LOAD should apply these rules during bootstrap and incremental phases > and convert the tables accordingly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20966) Support bootstrap and incremental replication to a target with hive.strict.managed.tables enabled.
[ https://issues.apache.org/jira/browse/HIVE-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20966: --- Attachment: HIVE-20966.04.patch > Support bootstrap and incremental replication to a target with > hive.strict.managed.tables enabled. > -- > > Key: HIVE-20966 > URL: https://issues.apache.org/jira/browse/HIVE-20966 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR > Attachments: HIVE-20966.01.patch, HIVE-20966.02.patch, > HIVE-20966.03.patch, HIVE-20966.04.patch, HIVE-20966.04.patch > > > *Requirements:* > Hive2 supports replication of managed tables. But in Hive3 with > hive.strict.managed.tables=true, some of these managed tables are converted > to ACID or MM tables. Also, some of them are converted to external tables > based on below rules. > - Avro format with external schema, Storage handlers, List bucketed tabled > are converted to external tables. > - Location not owned by "hive" user are converted to external table. > - Hive owned ORC format are converted to full ACID transactional table. > - Hive owned Non-ORC format are converted to MM transactional table. > REPL LOAD should apply these rules during bootstrap and incremental phases > and convert the tables accordingly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713022#comment-16713022 ] Zoltan Haindrich commented on HIVE-20915: - sorry; but I don't understand what do you mean by that. * these tests are there to test "auto column stats collection" ; which pre-existed before this ticket. * I don't think any new feature is expected to break existing ones * removing "that groupby" along with that compute_stats() call is essential removes column stats gathering ; which makes it inoperable. > Make dynamic sort partition optimization available to HoS and MR > > > Key: HIVE-20915 > URL: https://issues.apache.org/jira/browse/HIVE-20915 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch, > HIVE-20915.3.patch, HIVE-20915.4.patch, HIVE-20915.5.patch > > > HIVE-20703 put dynamic sort partition optimization under cost based decision, > but it also makes the optimizer only available to tez. > hive.optimize.sort.dynamic.partition works with other execution engines for a > long time, we should keep the optimizer available to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20966) Support bootstrap and incremental replication to a target with hive.strict.managed.tables enabled.
[ https://issues.apache.org/jira/browse/HIVE-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20966: --- Status: Open (was: Patch Available) > Support bootstrap and incremental replication to a target with > hive.strict.managed.tables enabled. > -- > > Key: HIVE-20966 > URL: https://issues.apache.org/jira/browse/HIVE-20966 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR > Attachments: HIVE-20966.01.patch, HIVE-20966.02.patch, > HIVE-20966.03.patch, HIVE-20966.04.patch > > > *Requirements:* > Hive2 supports replication of managed tables. But in Hive3 with > hive.strict.managed.tables=true, some of these managed tables are converted > to ACID or MM tables. Also, some of them are converted to external tables > based on below rules. > - Avro format with external schema, Storage handlers, List bucketed tabled > are converted to external tables. > - Location not owned by "hive" user are converted to external table. > - Hive owned ORC format are converted to full ACID transactional table. > - Hive owned Non-ORC format are converted to MM transactional table. > REPL LOAD should apply these rules during bootstrap and incremental phases > and convert the tables accordingly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-20440: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks for the patch [~asinkovits] and [~stakiar], [~dkuzmenko] for the review! > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch, > HIVE-20440.09.patch, HIVE-20440.10.patch, HIVE-20440.11.patch, > HIVE-20440.12.patch, HIVE-20440.13.patch, HIVE-20440.14.patch.txt, > HIVE-20440.15.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713004#comment-16713004 ] Yongzhi Chen commented on HIVE-20915: - I will create a jira to update the test to use set hive.optimize.sort.dynamic.partition.threshold=-1, is that OK [~kgyrtkirk] ? > Make dynamic sort partition optimization available to HoS and MR > > > Key: HIVE-20915 > URL: https://issues.apache.org/jira/browse/HIVE-20915 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch, > HIVE-20915.3.patch, HIVE-20915.4.patch, HIVE-20915.5.patch > > > HIVE-20703 put dynamic sort partition optimization under cost based decision, > but it also makes the optimizer only available to tez. > hive.optimize.sort.dynamic.partition works with other execution engines for a > long time, we should keep the optimizer available to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712992#comment-16712992 ] Yongzhi Chen edited comment on HIVE-20915 at 12/7/18 3:46 PM: -- [~kgyrtkirk], This patch enables dynamic sort partition optimization, and current cost based algorithm and the default value of hive.optimize.sort.dynamic.partition.threshold=0 will make this test use this optimization (the query plan will be different). If you set hive.optimize.sort.dynamic.partition.threshold=-1, the query plan will be the same as before. With the optimizer, there is no group by step, so there is no compute stats part. was (Author: ychena): [~kgyrtkirk], it looks the similar stats information for me: https://github.com/apache/hive/commit/9f2e8e61f3cf456420cfb461504cdfa076b96f6e#diff-e5ca628c543044b9188f64378e677accR53 This patch enables dynamic sort partition optimization, and current cost based algorithm and the default value of hive.optimize.sort.dynamic.partition.threshold=0 will make this test use this optimization (the query plan will be different). If you set hive.optimize.sort.dynamic.partition.threshold=-1, the query plan will be the same as before. > Make dynamic sort partition optimization available to HoS and MR > > > Key: HIVE-20915 > URL: https://issues.apache.org/jira/browse/HIVE-20915 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch, > HIVE-20915.3.patch, HIVE-20915.4.patch, HIVE-20915.5.patch > > > HIVE-20703 put dynamic sort partition optimization under cost based decision, > but it also makes the optimizer only available to tez. > hive.optimize.sort.dynamic.partition works with other execution engines for a > long time, we should keep the optimizer available to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712992#comment-16712992 ] Yongzhi Chen commented on HIVE-20915: - [~kgyrtkirk], it looks the similar stats information for me: https://github.com/apache/hive/commit/9f2e8e61f3cf456420cfb461504cdfa076b96f6e#diff-e5ca628c543044b9188f64378e677accR53 This patch enables dynamic sort partition optimization, and current cost based algorithm and the default value of hive.optimize.sort.dynamic.partition.threshold=0 will make this test use this optimization (the query plan will be different). If you set hive.optimize.sort.dynamic.partition.threshold=-1, the query plan will be the same as before. > Make dynamic sort partition optimization available to HoS and MR > > > Key: HIVE-20915 > URL: https://issues.apache.org/jira/browse/HIVE-20915 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch, > HIVE-20915.3.patch, HIVE-20915.4.patch, HIVE-20915.5.patch > > > HIVE-20703 put dynamic sort partition optimization under cost based decision, > but it also makes the optimizer only available to tez. > hive.optimize.sort.dynamic.partition works with other execution engines for a > long time, we should keep the optimizer available to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21018) Grouping/distinct on more than 64 columns should be possible
[ https://issues.apache.org/jira/browse/HIVE-21018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712988#comment-16712988 ] Jesus Camacho Rodriguez commented on HIVE-21018: +1 (pending tests) Thanks [~kgyrtkirk]! > Grouping/distinct on more than 64 columns should be possible > > > Key: HIVE-21018 > URL: https://issues.apache.org/jira/browse/HIVE-21018 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21018.01.patch > > > Earlier when more than 64 columns were in a group by or distinct an error was > given -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20733) GenericUDFOPEqualNS may not use = in plan descriptions
[ https://issues.apache.org/jira/browse/HIVE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712982#comment-16712982 ] David Lavati commented on HIVE-20733: - q.out updates done, the 1 failed test seems unrelated. > GenericUDFOPEqualNS may not use = in plan descriptions > -- > > Key: HIVE-20733 > URL: https://issues.apache.org/jira/browse/HIVE-20733 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: David Lavati >Priority: Major > Attachments: HIVE-20733.2.patch, HIVE-20733.patch > > > right now GenericUDFOPEqualNS is displayed a "=" in explains; however it > should be "<=>" > this may cause some confusion... > related qtest: is_distinct_from.q > same: GenericUDFOPNotEqualNS -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20613) CachedStore: Add more UT coverage (outside of .q files)
[ https://issues.apache.org/jira/browse/HIVE-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-20613: --- Assignee: Vaibhav Gumashta (was: Zoltan Haindrich) > CachedStore: Add more UT coverage (outside of .q files) > --- > > Key: HIVE-20613 > URL: https://issues.apache.org/jira/browse/HIVE-20613 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-20613.1.patch, HIVE-20613.2.patch, > HIVE-20613.2.patch > > > 1. Add tests which will use the background thread for updating the cached > data (database, table, partition, table stats, partition stats) > 2. Add more tests for existing APIs: stats aggregation, listing partitions > when partial specs are provided, testing the storage descriptor > caching/deduplication (specially when tables/ptns are dropped/added), table > col stats, partition col stats > 3. Test 1. in a multithreaded scenario > 4. Test whitelist/blacklist > 5. Test prewarm memory limit estimation code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21015) HCatLoader can't provide statistics for tables no in default DB
[
https://issues.apache.org/jira/browse/HIVE-21015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712957#comment-16712957
]
Peter Vary commented on HIVE-21015:
---
+1
> HCatLoader can't provide statistics for tables no in default DB
> ---
>
> Key: HIVE-21015
> URL: https://issues.apache.org/jira/browse/HIVE-21015
> Project: Hive
> Issue Type: Bug
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21015.0.patch, HIVE-21015.1.patch,
> HIVE-21015.2.patch, HIVE-21015.3.patch
>
>
> This is due to a former change (HIVE-20330) that does not take database into
> consideration when retrieving the proper InputJobInfo for the loader.
> Found during testing:
> {code:java}
> 07:52:56 2018-12-05 07:52:16,599 [main] WARN
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
> - Couldn't get statistics from LoadFunc:
> org.apache.hive.hcatalog.pig.HCatLoader@492fa72a
> 07:52:56 java.io.IOException: java.io.IOException: Could not calculate input
> size for location (table) tpcds_3000_decimal_parquet.date_dim
> 07:52:56 at
> org.apache.hive.hcatalog.pig.HCatLoader.getStatistics(HCatLoader.java:281)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getInputSizeFromLoader(InputSizeReducerEstimator.java:171)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getTotalInputFileSize(InputSizeReducerEstimator.java:118)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getTotalInputFileSize(InputSizeReducerEstimator.java:97)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.estimateNumberOfReducers(InputSizeReducerEstimator.java:80)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.estimateNumberOfReducers(JobControlCompiler.java:1148)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.calculateRuntimeReducers(JobControlCompiler.java:1115)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:1063)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:564)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:333)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:221)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:293)
> 07:52:56 at org.apache.pig.PigServer.launchPlan(PigServer.java:1475)
> 07:52:56 at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1460)
> 07:52:56 at org.apache.pig.PigServer.storeEx(PigServer.java:1119)
> 07:52:56 at org.apache.pig.PigServer.store(PigServer.java:1082)
> 07:52:56 at org.apache.pig.PigServer.openIterator(PigServer.java:995)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:782)
> 07:52:56 at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> 07:52:56 at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> 07:52:56 at org.apache.pig.Main.run(Main.java:630)
> 07:52:56 at org.apache.pig.Main.main(Main.java:175)
> 07:52:56 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 07:52:56 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 07:52:56 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 07:52:56 at java.lang.reflect.Method.invoke(Method.java:498)
> 07:52:56 at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
> 07:52:56 at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
> 07:52:56 Caused by: java.io.IOException: Could not calculate input size for
> location (table) tpcds_3000_decimal_parquet.date_dim
> 07:52:56 at
> org.apache.hive.hcatalog.pig.HCatLoader.getStatistics(HCatLoader.java:276)
> 07:52:56 ... 29 more{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20615) CachedStore: Background refresh thread bug fixes
[ https://issues.apache.org/jira/browse/HIVE-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-20615: --- Assignee: Vaibhav Gumashta (was: Zoltan Haindrich) > CachedStore: Background refresh thread bug fixes > > > Key: HIVE-20615 > URL: https://issues.apache.org/jira/browse/HIVE-20615 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-20615.1.patch, HIVE-20615.1.patch, > HIVE-20615.1.patch, HIVE-20615.1.patch, HIVE-20615.1.patch, HIVE-20615.1.patch > > > Regression introduced in HIVE-18264. Fixes background thread starting and > refreshing of the table cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20961) Retire NVL implementation
[
https://issues.apache.org/jira/browse/HIVE-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712931#comment-16712931
]
Hive QA commented on HIVE-20961:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12950964/HIVE-20961.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 15650 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=262)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constantPropWhen]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_case] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_when] (batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_memcheck]
(batchId=45)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_ppd_key_range]
(batchId=103)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_pushdown]
(batchId=103)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15210/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15210/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15210/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12950964 - PreCommit-HIVE-Build
> Retire NVL implementation
> -
>
> Key: HIVE-20961
> URL: https://issues.apache.org/jira/browse/HIVE-20961
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-20961.01.patch, HIVE-20961.02.patch
>
>
> Right now we have coalesce and nvl implemented separetly; it might be better
> to remove one of them as they are doing the same. Because Coalesce is in the
> standard - I think NVL have to go...and became an alias to Coalesce.
> Further benefit is: that optimizations which coalesce already recieves; will
> be done on NVL calls as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20613) CachedStore: Add more UT coverage (outside of .q files)
[ https://issues.apache.org/jira/browse/HIVE-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-20613: Attachment: HIVE-20613.2.patch > CachedStore: Add more UT coverage (outside of .q files) > --- > > Key: HIVE-20613 > URL: https://issues.apache.org/jira/browse/HIVE-20613 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20613.1.patch, HIVE-20613.2.patch, > HIVE-20613.2.patch > > > 1. Add tests which will use the background thread for updating the cached > data (database, table, partition, table stats, partition stats) > 2. Add more tests for existing APIs: stats aggregation, listing partitions > when partial specs are provided, testing the storage descriptor > caching/deduplication (specially when tables/ptns are dropped/added), table > col stats, partition col stats > 3. Test 1. in a multithreaded scenario > 4. Test whitelist/blacklist > 5. Test prewarm memory limit estimation code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20615) CachedStore: Background refresh thread bug fixes
[ https://issues.apache.org/jira/browse/HIVE-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-20615: Attachment: HIVE-20615.1.patch > CachedStore: Background refresh thread bug fixes > > > Key: HIVE-20615 > URL: https://issues.apache.org/jira/browse/HIVE-20615 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20615.1.patch, HIVE-20615.1.patch, > HIVE-20615.1.patch, HIVE-20615.1.patch, HIVE-20615.1.patch, HIVE-20615.1.patch > > > Regression introduced in HIVE-18264. Fixes background thread starting and > refreshing of the table cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20613) CachedStore: Add more UT coverage (outside of .q files)
[ https://issues.apache.org/jira/browse/HIVE-20613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-20613: --- Assignee: Zoltan Haindrich (was: Vaibhav Gumashta) > CachedStore: Add more UT coverage (outside of .q files) > --- > > Key: HIVE-20613 > URL: https://issues.apache.org/jira/browse/HIVE-20613 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20613.1.patch, HIVE-20613.2.patch > > > 1. Add tests which will use the background thread for updating the cached > data (database, table, partition, table stats, partition stats) > 2. Add more tests for existing APIs: stats aggregation, listing partitions > when partial specs are provided, testing the storage descriptor > caching/deduplication (specially when tables/ptns are dropped/added), table > col stats, partition col stats > 3. Test 1. in a multithreaded scenario > 4. Test whitelist/blacklist > 5. Test prewarm memory limit estimation code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20615) CachedStore: Background refresh thread bug fixes
[ https://issues.apache.org/jira/browse/HIVE-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-20615: --- Assignee: Zoltan Haindrich (was: Vaibhav Gumashta) > CachedStore: Background refresh thread bug fixes > > > Key: HIVE-20615 > URL: https://issues.apache.org/jira/browse/HIVE-20615 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20615.1.patch, HIVE-20615.1.patch, > HIVE-20615.1.patch, HIVE-20615.1.patch, HIVE-20615.1.patch > > > Regression introduced in HIVE-18264. Fixes background thread starting and > refreshing of the table cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20942) Worker should heartbeat its own txn
[
https://issues.apache.org/jira/browse/HIVE-20942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712956#comment-16712956
]
Hive QA commented on HIVE-20942:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 3 new + 15 unchanged - 0 fixed
= 18 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15211/dev-support/hive-personality.sh
|
| git revision | master / 83d1fd2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15211/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15211/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Worker should heartbeat its own txn
> ---
>
> Key: HIVE-20942
> URL: https://issues.apache.org/jira/browse/HIVE-20942
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-20942.01.patch, HIVE-20942.02.patch,
> HIVE-20942.03.patch, HIVE-20942.04.patch
>
>
> Since HIVE-20823 \{{Worker.java}} starts a txn - should either add a
> heartbeat thread to it or use HiveTxnManager to start txn which will set up
> heartbeat automatically. In the later case make sure it's properly cancelled
> on failures.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712944#comment-16712944 ] Zoltan Haindrich commented on HIVE-20915: - [~ychena], [~ngangam] This patch have broken auto column stats collection. It doesn't seem to happen anymore! [ql/src/test/results/clientpositive/autoColumnStats_6.q.out|https://github.com/apache/hive/commit/9f2e8e61f3cf456420cfb461504cdfa076b96f6e#diff-e5ca628c543044b9188f64378e677accL64] Please reopen this ticket or open a new one to fix the regression! > Make dynamic sort partition optimization available to HoS and MR > > > Key: HIVE-20915 > URL: https://issues.apache.org/jira/browse/HIVE-20915 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch, > HIVE-20915.3.patch, HIVE-20915.4.patch, HIVE-20915.5.patch > > > HIVE-20703 put dynamic sort partition optimization under cost based decision, > but it also makes the optimizer only available to tez. > hive.optimize.sort.dynamic.partition works with other execution engines for a > long time, we should keep the optimizer available to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21018) Grouping/distinct on more than 64 columns should be possible
[ https://issues.apache.org/jira/browse/HIVE-21018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21018: Status: Patch Available (was: Open) > Grouping/distinct on more than 64 columns should be possible > > > Key: HIVE-21018 > URL: https://issues.apache.org/jira/browse/HIVE-21018 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21018.01.patch > > > Earlier when more than 64 columns were in a group by or distinct an error was > given -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21018) Grouping/distinct on more than 64 columns should be possible
[ https://issues.apache.org/jira/browse/HIVE-21018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21018: Attachment: HIVE-21018.01.patch > Grouping/distinct on more than 64 columns should be possible > > > Key: HIVE-21018 > URL: https://issues.apache.org/jira/browse/HIVE-21018 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21018.01.patch > > > Earlier when more than 64 columns were in a group by or distinct an error was > given -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21018) Grouping/distinct on more than 64 columns should be possible
[ https://issues.apache.org/jira/browse/HIVE-21018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-21018: --- > Grouping/distinct on more than 64 columns should be possible > > > Key: HIVE-21018 > URL: https://issues.apache.org/jira/browse/HIVE-21018 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > Earlier when more than 64 columns were in a group by or distinct an error was > given -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20961) Retire NVL implementation
[
https://issues.apache.org/jira/browse/HIVE-20961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712886#comment-16712886
]
Hive QA commented on HIVE-20961:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 1 new + 1055 unchanged - 6
fixed = 1056 total (was 1061) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} ql generated 0 new + 2311 unchanged - 1 fixed = 2311
total (was 2312) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15210/dev-support/hive-personality.sh
|
| git revision | master / 83d1fd2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15210/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15210/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Retire NVL implementation
> -
>
> Key: HIVE-20961
> URL: https://issues.apache.org/jira/browse/HIVE-20961
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-20961.01.patch, HIVE-20961.02.patch
>
>
> Right now we have coalesce and nvl implemented separetly; it might be better
> to remove one of them as they are doing the same. Because Coalesce is in the
> standard - I think NVL have to go...and became an alias to Coalesce.
> Further benefit is: that optimizations which coalesce already recieves; will
> be done on NVL calls as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20966) Support bootstrap and incremental replication to a target with hive.strict.managed.tables enabled.
[
https://issues.apache.org/jira/browse/HIVE-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712858#comment-16712858
]
Hive QA commented on HIVE-20966:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12950961/HIVE-20966.04.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 64 failed/errored test(s), 15637 tests
executed
*Failed tests:*
{noformat}
TestAlterTableMetadata - did not produce a TEST-*.xml file (likely timed out)
(batchId=248)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=248)
TestReplAcidTablesWithJsonMessage - did not produce a TEST-*.xml file (likely
timed out) (batchId=248)
TestReplIncrementalLoadAcidTablesWithJsonMessage - did not produce a TEST-*.xml
file (likely timed out) (batchId=248)
TestReplicationScenariosMigration - did not produce a TEST-*.xml file (likely
timed out) (batchId=248)
TestSemanticAnalyzerHookLoading - did not produce a TEST-*.xml file (likely
timed out) (batchId=248)
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
(batchId=276)
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testConcurrentDropPartitions
(batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=229)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTableCascade
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterViewParititon
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testColumnStatistics
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testComplexTypeApi
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testConcurrentMetastores
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateAndGetTableWithDriver
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testCreateTableSettingId
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBLocationChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwner
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDBOwnerChange
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabase
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocation
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDatabaseLocationWithPermissionProblems
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropDatabaseCascadeMVMultiDB
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testDropTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterLastPartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFilterSinglePartition
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testFunctionWithResources
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetConfigValue
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetMetastoreUuid
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetPartitionsWithSpec
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testGetSchemaWithNoClassDefFoundError
(batchId=229)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.tes
[jira] [Commented] (HIVE-20966) Support bootstrap and incremental replication to a target with hive.strict.managed.tables enabled.
[
https://issues.apache.org/jira/browse/HIVE-20966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712829#comment-16712829
]
Hive QA commented on HIVE-20966:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
11s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
184 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} hcatalog/server-extensions in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} common: The patch generated 1 new + 427 unchanged - 0
fixed = 428 total (was 427) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 55 new + 854 unchanged - 11
fixed = 909 total (was 865) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} itests/hive-unit: The patch generated 145 new + 658
unchanged - 58 fixed = 803 total (was 716) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 16 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
13s{color} | {color:red} standalone-metastore/metastore-server generated 4 new
+ 184 unchanged - 0 fixed = 188 total (was 184) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
48s{color} | {color:red} ql generated 1 new + 2311 unchanged - 1 fixed = 2312
total (was 2312) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
34s{color} | {color:red} hcatalog/server-extensions generated 2 new + 1
unchanged - 0 fixed = 3 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Boxed value is unboxed and then immediately reboxed in
org.apache.hadoop.hive.metastore.messaging.json.JSONAlte
[jira] [Updated] (HIVE-21017) Throw proper exception in case of alter table add partition on an ACID table
[ https://issues.apache.org/jira/browse/HIVE-21017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21017: Summary: Throw proper exception in case of alter table add partition on an ACID table (was: Throw proper exception in case of alter table add partition (old hashed files) to a murmur hashed bucketed table) > Throw proper exception in case of alter table add partition on an ACID table > > > Key: HIVE-21017 > URL: https://issues.apache.org/jira/browse/HIVE-21017 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21015) HCatLoader can't provide statistics for tables no in default DB
[
https://issues.apache.org/jira/browse/HIVE-21015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712757#comment-16712757
]
Hive QA commented on HIVE-21015:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12950960/HIVE-21015.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15208/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15208/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15208/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12950960/HIVE-21015.3.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12950960 - PreCommit-HIVE-Build
> HCatLoader can't provide statistics for tables no in default DB
> ---
>
> Key: HIVE-21015
> URL: https://issues.apache.org/jira/browse/HIVE-21015
> Project: Hive
> Issue Type: Bug
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21015.0.patch, HIVE-21015.1.patch,
> HIVE-21015.2.patch, HIVE-21015.3.patch
>
>
> This is due to a former change (HIVE-20330) that does not take database into
> consideration when retrieving the proper InputJobInfo for the loader.
> Found during testing:
> {code:java}
> 07:52:56 2018-12-05 07:52:16,599 [main] WARN
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
> - Couldn't get statistics from LoadFunc:
> org.apache.hive.hcatalog.pig.HCatLoader@492fa72a
> 07:52:56 java.io.IOException: java.io.IOException: Could not calculate input
> size for location (table) tpcds_3000_decimal_parquet.date_dim
> 07:52:56 at
> org.apache.hive.hcatalog.pig.HCatLoader.getStatistics(HCatLoader.java:281)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getInputSizeFromLoader(InputSizeReducerEstimator.java:171)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getTotalInputFileSize(InputSizeReducerEstimator.java:118)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getTotalInputFileSize(InputSizeReducerEstimator.java:97)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.estimateNumberOfReducers(InputSizeReducerEstimator.java:80)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.estimateNumberOfReducers(JobControlCompiler.java:1148)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.calculateRuntimeReducers(JobControlCompiler.java:1115)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:1063)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:564)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:333)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:221)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:293)
> 07:52:56 at org.apache.pig.PigServer.launchPlan(PigServer.java:1475)
> 07:52:56 at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1460)
> 07:52:56 at org.apache.pig.PigServer.storeEx(PigServer.java:1119)
> 07:52:56 at org.apache.pig.PigServer.store(PigServer.java:1082)
> 07:52:56 at org.apache.pig.PigServer.openIterator(PigServer.java:995)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:782)
> 07:52:56 at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> 07:52:56 at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> 07:52:56 at org.apache.pig.Main.run(Main.java:630)
> 07:52:56 at org.apache.pig.Main.main(Main.java:175)
> 07:52:56 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 07:52:56 at
> sun.reflec
[jira] [Commented] (HIVE-21015) HCatLoader can't provide statistics for tables no in default DB
[
https://issues.apache.org/jira/browse/HIVE-21015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712756#comment-16712756
]
Hive QA commented on HIVE-21015:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12950960/HIVE-21015.3.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15650 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15207/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15207/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15207/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12950960 - PreCommit-HIVE-Build
> HCatLoader can't provide statistics for tables no in default DB
> ---
>
> Key: HIVE-21015
> URL: https://issues.apache.org/jira/browse/HIVE-21015
> Project: Hive
> Issue Type: Bug
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21015.0.patch, HIVE-21015.1.patch,
> HIVE-21015.2.patch, HIVE-21015.3.patch
>
>
> This is due to a former change (HIVE-20330) that does not take database into
> consideration when retrieving the proper InputJobInfo for the loader.
> Found during testing:
> {code:java}
> 07:52:56 2018-12-05 07:52:16,599 [main] WARN
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
> - Couldn't get statistics from LoadFunc:
> org.apache.hive.hcatalog.pig.HCatLoader@492fa72a
> 07:52:56 java.io.IOException: java.io.IOException: Could not calculate input
> size for location (table) tpcds_3000_decimal_parquet.date_dim
> 07:52:56 at
> org.apache.hive.hcatalog.pig.HCatLoader.getStatistics(HCatLoader.java:281)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getInputSizeFromLoader(InputSizeReducerEstimator.java:171)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getTotalInputFileSize(InputSizeReducerEstimator.java:118)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.getTotalInputFileSize(InputSizeReducerEstimator.java:97)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator.estimateNumberOfReducers(InputSizeReducerEstimator.java:80)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.estimateNumberOfReducers(JobControlCompiler.java:1148)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.calculateRuntimeReducers(JobControlCompiler.java:1115)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:1063)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:564)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:333)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:221)
> 07:52:56 at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:293)
> 07:52:56 at org.apache.pig.PigServer.launchPlan(PigServer.java:1475)
> 07:52:56 at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1460)
> 07:52:56 at org.apache.pig.PigServer.storeEx(PigServer.java:1119)
> 07:52:56 at org.apache.pig.PigServer.store(PigServer.java:1082)
> 07:52:56 at org.apache.pig.PigServer.openIterator(PigServer.java:995)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:782)
> 07:52:56 at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:383)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
> 07:52:56 at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> 07:52:56 at org.apache.pig.tools
