[jira] [Commented] (HIVE-21116) HADOOP_CREDSTORE_PASSWORD is not populated under yarn.app.mapreduce.am.admin.user.env
[
https://issues.apache.org/jira/browse/HIVE-21116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745851#comment-16745851
]
Hive QA commented on HIVE-21116:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} common: The patch generated 0 new + 101 unchanged -
1 fixed = 101 total (was 102) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15678/dev-support/hive-personality.sh
|
| git revision | master / 67284b0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15678/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HADOOP_CREDSTORE_PASSWORD is not populated under
> yarn.app.mapreduce.am.admin.user.env
> --
>
> Key: HIVE-21116
> URL: https://issues.apache.org/jira/browse/HIVE-21116
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21116.1.patch, HIVE-21116.2.patch,
> HIVE-21116.3.patch, HIVE-21116.4.patch, HIVE-21116.5.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745833#comment-16745833
]
Hive QA commented on HIVE-21052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955320/HIVE-21052.7.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15700 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15677/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15677/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15677/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955320 - PreCommit-HIVE-Build
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Critical
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
> HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch
>
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745818#comment-16745818
]
Hive QA commented on HIVE-21052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
17s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
19s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
9s{color} | {color:blue} standalone-metastore/metastore-common in master has 29
extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
33s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} shims/common: The patch generated 1 new + 95 unchanged
- 0 fixed = 96 total (was 95) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} shims/0.23: The patch generated 5 new + 69 unchanged -
0 fixed = 74 total (was 69) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 12 new + 575 unchanged - 7
fixed = 587 total (was 582) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/hive-unit: The patch generated 10 new + 149
unchanged - 0 fixed = 159 total (was 149) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
18s{color} | {color:red} standalone-metastore/metastore-server generated 2 new
+ 188 unchanged - 0 fixed = 190 total (was 188) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
50s{color} | {color:red} ql generated 1 new + 2309 unchanged - 1 fixed = 2310
total (was 2310) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| |
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean()
may fail to close PreparedStatement At
Comp
[jira] [Commented] (HIVE-21077) Database and catalogs should have creation time
[
https://issues.apache.org/jira/browse/HIVE-21077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745797#comment-16745797
]
Bharathkrishna Guruvayoor Murali commented on HIVE-21077:
-
The changes for upgrade scripts from 3.1.0 to 3.2.0 looks good to me.
+1
> Database and catalogs should have creation time
> ---
>
> Key: HIVE-21077
> URL: https://issues.apache.org/jira/browse/HIVE-21077
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21077.01.patch, HIVE-21077.02.patch,
> HIVE-21077.03.patch, HIVE-21077.04.patch, HIVE-21077.05.patch,
> HIVE-21077.06.patch, HIVE-21077.07.patch, HIVE-21077.08.branch-3.patch,
> HIVE-21077.09.patch, HIVE-21077.10.patch
>
>
> Currently, database do not have creation time like we have for tables and
> partitions.
> {noformat}
> // namespace for tables
> struct Database {
> 1: string name,
> 2: string description,
> 3: string locationUri,
> 4: map parameters, // properties associated with the
> database
> 5: optional PrincipalPrivilegeSet privileges,
> 6: optional string ownerName,
> 7: optional PrincipalType ownerType,
> 8: optional string catalogName
> }
> {noformat}
> Currently, without creationTime there is no way to identify if the copy of
> Database which a client has is the same as the one on the server if the name
> is same. Without object ids creationTime value is the only way currently to
> identify uniquely a instance of metastore object. It would be good to have
> Database creation time as well.
> Same applies for catalogs as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21132) Semi join edge is not being removed despite max bloomfilter entries set to 1
[
https://issues.apache.org/jira/browse/HIVE-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745796#comment-16745796
]
Hive QA commented on HIVE-21132:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955311/HIVE-21132.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 170 failed/errored test(s), 15696 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[semijoin_reddedup]
(batchId=40)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniHiveKafkaCliDriver.testCliDriver[kafka_storage_handler]
(batchId=275)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_partition_pruning_2]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[global_limit]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mapreduce1]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mapreduce2]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[multi_count_distinct_null]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_struct_type_vectorization]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parallel_colstats]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_complex_types_vectorization]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_map_type_vectorization]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_struct_type_vectorization]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_types_vectorization]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_smb_mapjoin_14]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_9]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_groupby]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_gby_empty]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_limit]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_semijoin]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_udf_udaf]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_udf_udaf_stats_opt]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_semijoin]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_subq_not_in]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_udf_udaf]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[column_access_stats]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constraints_optimization]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer1]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer2]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer4]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[count_dist_rewrite]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_prod_1]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_prod_3]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[distinct_66]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_partition_pruning]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainanalyze_2]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_resolution]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[infer_bucket_sort
[jira] [Commented] (HIVE-21132) Semi join edge is not being removed despite max bloomfilter entries set to 1
[
https://issues.apache.org/jira/browse/HIVE-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745776#comment-16745776
]
Hive QA commented on HIVE-21132:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 54 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15676/dev-support/hive-personality.sh
|
| git revision | master / 67284b0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15676/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15676/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Semi join edge is not being removed despite max bloomfilter entries set to 1
>
>
> Key: HIVE-21132
> URL: https://issues.apache.org/jira/browse/HIVE-21132
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21132.1.patch
>
>
> * Reproducer
> {code:sql}
> --! qt:dataset:lineitem
> --! qt:dataset:part
> --! qt:dataset:src
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> --set hive.compute.query.using.stats=false;
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.tez.dynamic.partition.pruning=true;
> set hive.tez.dynamic.semijoin.reduction=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.stats.autogather=true;
> set hive.tez.bigtable.minsize.semijoin.reduction=1;
> set hive.tez.min.bloom.filter.entries=1;
> set hive.stats.fetch.column.stats=true;
> set hive.tez.bloom.filter.factor=1.0f;
> set hive.auto.convert.join=false;
> set hive.optimize.shared.work=false;
> create database tpch_test;
> use tpch_test;
> CREATE TABLE `customer`(
> `c_custkey` bigint,
> `c_name` string,
> `c_address` stri
[jira] [Commented] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745766#comment-16745766
]
Hive QA commented on HIVE-21107:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955310/HIVE-21107.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15696 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15675/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15675/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15675/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955310 - PreCommit-HIVE-Build
> Cannot find field" error during dynamically partitioned hash join
> -
>
> Key: HIVE-21107
> URL: https://issues.apache.org/jira/browse/HIVE-21107
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch,
> HIVE-21107.3.patch, HIVE-21107.4.patch, HIVE-21107.5.patch
>
>
> This occurs in non-CBO path with dynamic partitioned join + constant
> propagation ON.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745754#comment-16745754
]
Hive QA commented on HIVE-21107:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 74 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15675/dev-support/hive-personality.sh
|
| git revision | master / 67284b0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15675/yetus/whitespace-eol.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15675/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Cannot find field" error during dynamically partitioned hash join
> -
>
> Key: HIVE-21107
> URL: https://issues.apache.org/jira/browse/HIVE-21107
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch,
> HIVE-21107.3.patch, HIVE-21107.4.patch, HIVE-21107.5.patch
>
>
> This occurs in non-CBO path with dynamic partitioned join + constant
> propagation ON.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20776) Run HMS filterHooks on server-side in addition to client-side
[
https://issues.apache.org/jira/browse/HIVE-20776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745730#comment-16745730
]
Hive QA commented on HIVE-20776:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955304/HIVE-20776.012.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15695 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionsByNamesNullDbName[Embedded]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionsByNamesNullDbName[Remote]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionsByNamesNullTblName[Embedded]
(batchId=220)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionsByNamesNullTblName[Remote]
(batchId=220)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15674/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15674/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15674/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955304 - PreCommit-HIVE-Build
> Run HMS filterHooks on server-side in addition to client-side
> -
>
> Key: HIVE-20776
> URL: https://issues.apache.org/jira/browse/HIVE-20776
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Karthik Manamcheri
>Assignee: Na Li
>Priority: Major
> Attachments: HIVE-20776.001.patch, HIVE-20776.003.patch,
> HIVE-20776.004.patch, HIVE-20776.005.patch, HIVE-20776.006.patch,
> HIVE-20776.007.patch, HIVE-20776.007.patch, HIVE-20776.008.patch,
> HIVE-20776.009.patch, HIVE-20776.009.patch, HIVE-20776.010.patch,
> HIVE-20776.011.patch, HIVE-20776.011.patch, HIVE-20776.012.patch
>
>
> In HMS, I noticed that all the filter hooks are applied on the client side
> (in HiveMetaStoreClient.java). Is there any reason why we can't apply the
> filters on the server-side?
> Motivation: Some newer apache projects such as Kudu use HMS for metadata
> storage. Kudu is not completely Java-based and there are interaction points
> where they have C++ clients. In such cases, it would be ideal to have
> consistent behavior from HMS side as far as filters, etc are concerned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17020:
---
Attachment: HIVE-17020.7.patch
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch, HIVE-17020.4.patch, HIVE-17020.5.patch,
> HIVE-17020.6.patch, HIVE-17020.7.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17020:
---
Status: Patch Available (was: Open)
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch, HIVE-17020.4.patch, HIVE-17020.5.patch,
> HIVE-17020.6.patch, HIVE-17020.7.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17020:
---
Status: Open (was: Patch Available)
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch, HIVE-17020.4.patch, HIVE-17020.5.patch,
> HIVE-17020.6.patch, HIVE-17020.7.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745713#comment-16745713
]
Eugene Koifman commented on HIVE-21052:
---
I left some comments on RB.
I think the way Cleaner manages concurrency is not quite right.
Currently there is 1 Cleaner per HMS. You can have several HMS instances in
the cluster for HA. Eac Cleaner run is mutexed via {{handle =
txnHandler.getMutexAPI().acquireLock(TxnStore.MUTEX_KEY.Cleaner.name());}} so
only 1 is actually running at a time.
The (new) Cleaner seems to parallelize tasks too early and then has to mutex on
the HMS access.
I would suggest resolving the paths first and then enqueue parallel tasks into
the Priority queue to just to the deletes. I would make sure that 2 clean
operations of the same partition should not be allowed, nor 2 table level
cleans. (I'm not sure if table clean could run concurrently with partition
level clean of the same table - I suspect yes if {{markCleaned()}} is such that
the table clean and partition clean remove disjoint sets of TXN_COMPONENTS
entries. For 1st pass, I'd disallow it)
You could keep a (Concurrent) Map of locks which is thrown away at the end of
Cleaner.run(). the locks are either named after Table or Partition. To
acquire Partition level lock you 1st have acquire table level lock. This way
each {{CleanWork}} work can runs separately as long as it's not violating above
rules. In other words, cleans that are guaranteed to work on entities that are
not the same/related run in parallel - otherwise in sequence.
I suspect it may be useful to see if {{findReadyToClean()}} returns a very long
list it may be useful to create several RawStore connections to do the
'resolve' operations in parallel but I'd say this is pass 2 or later. This
would actually allow these to run in parallel.
Let me know what you think.
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Critical
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
> HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch
>
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21133) Add simulated materialized views useful for rewriting debugging
[
https://issues.apache.org/jira/browse/HIVE-21133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-21133:
--
> Add simulated materialized views useful for rewriting debugging
> ---
>
> Key: HIVE-21133
> URL: https://issues.apache.org/jira/browse/HIVE-21133
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Implement simulated materialized views, useful to check whether a certain
> rewriting will be triggered. Simulated materialized views definitions will be
> stored in the user session, and they will only be used when simulation mode
> is enabled and user runs {{explain cbo}} / {{explain cbo extended}}.
> {code}
> set hive.simulation.enable=true;
> create simulated materialized view mv1_n2 as
> select * from emps_n3 where empid < 150;
> explain cbo
> select *
> from (select * from emps_n3 where empid < 120) t
> join depts_n2 using (deptno);
> drop simulated materialized view mv1_n2;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21133) Add simulated materialized views useful for rewriting debugging
[
https://issues.apache.org/jira/browse/HIVE-21133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21133:
---
Attachment: HIVE-21133.patch
> Add simulated materialized views useful for rewriting debugging
> ---
>
> Key: HIVE-21133
> URL: https://issues.apache.org/jira/browse/HIVE-21133
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21133.patch
>
>
> Implement simulated materialized views, useful to check whether a certain
> rewriting will be triggered. Simulated materialized views definitions will be
> stored in the user session, and they will only be used when simulation mode
> is enabled and user runs {{explain cbo}} / {{explain cbo extended}}.
> {code}
> set hive.simulation.enable=true;
> create simulated materialized view mv1_n2 as
> select * from emps_n3 where empid < 150;
> explain cbo
> select *
> from (select * from emps_n3 where empid < 120) t
> join depts_n2 using (deptno);
> drop simulated materialized view mv1_n2;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-21133) Add simulated materialized views useful for rewriting debugging
[
https://issues.apache.org/jira/browse/HIVE-21133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-21133 started by Jesus Camacho Rodriguez.
--
> Add simulated materialized views useful for rewriting debugging
> ---
>
> Key: HIVE-21133
> URL: https://issues.apache.org/jira/browse/HIVE-21133
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Implement simulated materialized views, useful to check whether a certain
> rewriting will be triggered. Simulated materialized views definitions will be
> stored in the user session, and they will only be used when simulation mode
> is enabled and user runs {{explain cbo}} / {{explain cbo extended}}.
> {code}
> set hive.simulation.enable=true;
> create simulated materialized view mv1_n2 as
> select * from emps_n3 where empid < 150;
> explain cbo
> select *
> from (select * from emps_n3 where empid < 120) t
> join depts_n2 using (deptno);
> drop simulated materialized view mv1_n2;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21133) Add simulated materialized views useful for rewriting debugging
[
https://issues.apache.org/jira/browse/HIVE-21133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21133:
---
Status: Patch Available (was: In Progress)
> Add simulated materialized views useful for rewriting debugging
> ---
>
> Key: HIVE-21133
> URL: https://issues.apache.org/jira/browse/HIVE-21133
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Implement simulated materialized views, useful to check whether a certain
> rewriting will be triggered. Simulated materialized views definitions will be
> stored in the user session, and they will only be used when simulation mode
> is enabled and user runs {{explain cbo}} / {{explain cbo extended}}.
> {code}
> set hive.simulation.enable=true;
> create simulated materialized view mv1_n2 as
> select * from emps_n3 where empid < 150;
> explain cbo
> select *
> from (select * from emps_n3 where empid < 120) t
> join depts_n2 using (deptno);
> drop simulated materialized view mv1_n2;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20776) Run HMS filterHooks on server-side in addition to client-side
[
https://issues.apache.org/jira/browse/HIVE-20776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745712#comment-16745712
]
Hive QA commented on HIVE-20776:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
8s{color} | {color:blue} standalone-metastore/metastore-common in master has 29
extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15674/dev-support/hive-personality.sh
|
| git revision | master / 67284b0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15674/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Run HMS filterHooks on server-side in addition to client-side
> -
>
> Key: HIVE-20776
> URL: https://issues.apache.org/jira/browse/HIVE-20776
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Karthik Manamcheri
>Assignee: Na Li
>Priority: Major
> Attachments: HIVE-20776.001.patch, HIVE-20776.003.patch,
> HIVE-20776.004.patch, HIVE-20776.005.patch, HIVE-20776.006.patch,
> HIVE-20776.007.patch, HIVE-20776.007.patch, HIVE-20776.008.patch,
> HIVE-20776.009.patch, HIVE-20776.009.patch, HIVE-20776.010.patch,
> HIVE-20776.011.patch, HIVE-20776.011.patch, HIVE-20776.012.patch
>
>
> In HMS, I noticed that all the filter hooks are applied on the client side
> (in HiveMetaStoreClient.java). Is there any reason why we can't apply the
> filters on the server-side?
> Motivation: Some newer apache projects such as Kudu use HMS for metadata
> storage. Kudu is not completely Java-based and
[jira] [Commented] (HIVE-20546) Upgrade to Apache Druid 0.13.0-incubating
[
https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745697#comment-16745697
]
Hive QA commented on HIVE-20546:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955302/HIVE-20546.2.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15695 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_timeseries]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_topn] (batchId=3)
org.apache.hadoop.hive.druid.TestHiveDruidQueryBasedInputFormat.testTimeZone
(batchId=283)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15673/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15673/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15673/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955302 - PreCommit-HIVE-Build
> Upgrade to Apache Druid 0.13.0-incubating
> -
>
> Key: HIVE-20546
> URL: https://issues.apache.org/jira/browse/HIVE-20546
> Project: Hive
> Issue Type: Task
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20546.1.patch, HIVE-20546.2.patch, HIVE-20546.patch
>
>
> This task is to upgrade to druid 0.13.0 when it is released. Note that it
> will hopefully be first apache release for Druid.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20546) Upgrade to Apache Druid 0.13.0-incubating
[
https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745690#comment-16745690
]
Hive QA commented on HIVE-20546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} druid-handler in master has 3 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
18s{color} | {color:blue} itests/qtest-druid in master has 7 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
34s{color} | {color:red} druid-handler generated 1 new + 3 unchanged - 0 fixed
= 4 total (was 3) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:druid-handler |
| | org.apache.hadoop.hive.druid.json.KafkaSupervisorReport doesn't override
org.apache.druid.indexing.overlord.supervisor.SupervisorReport.equals(Object)
At KafkaSupervisorReport.java:At KafkaSupervisorReport.java:[line 1] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15673/dev-support/hive-personality.sh
|
| git revision | master / 67284b0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15673/yetus/new-findbugs-druid-handler.html
|
| modules | C: ql druid-handler . itests/qtest-druid U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15673/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade to Apache Druid 0.13.0-incubating
> -
>
> Key: HIVE-20546
> URL: https://issues.apache.org/jira/browse/HIVE-20546
> Project: Hive
> Issue Type: Task
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20546.1.patch, HIVE-20546.2.patch, HIVE-20546.
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Attachment: (was: HIVE-21052.7.patch) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Attachment: (was: HIVE-21052.8.patch) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Attachment: HIVE-21052.7.patch Status: Patch Available (was: Open) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21030) Add credential store env properties redaction in JobConf
[ https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21030: -- Attachment: HIVE-21030.3.branch-3.patch > Add credential store env properties redaction in JobConf > > > Key: HIVE-21030 > URL: https://issues.apache.org/jira/browse/HIVE-21030 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21030.1.patch, HIVE-21030.2.branch-3.patch, > HIVE-21030.2.patch, HIVE-21030.3.branch-3.patch, HIVE-21030.3.patch, > HIVE-21030.4.patch, HIVE-21030.5.patch, HIVE-21030.6.patch, > HIVE-21030.branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21116) HADOOP_CREDSTORE_PASSWORD is not populated under yarn.app.mapreduce.am.admin.user.env
[ https://issues.apache.org/jira/browse/HIVE-21116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21116: -- Attachment: HIVE-21116.5.patch > HADOOP_CREDSTORE_PASSWORD is not populated under > yarn.app.mapreduce.am.admin.user.env > -- > > Key: HIVE-21116 > URL: https://issues.apache.org/jira/browse/HIVE-21116 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21116.1.patch, HIVE-21116.2.patch, > HIVE-21116.3.patch, HIVE-21116.4.patch, HIVE-21116.5.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Status: Open (was: Patch Available) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch, HIVE-21052.8.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21077) Database and catalogs should have creation time
[
https://issues.apache.org/jira/browse/HIVE-21077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745651#comment-16745651
]
Hive QA commented on HIVE-21077:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955300/HIVE-21077.10.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15699 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15672/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15672/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15672/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955300 - PreCommit-HIVE-Build
> Database and catalogs should have creation time
> ---
>
> Key: HIVE-21077
> URL: https://issues.apache.org/jira/browse/HIVE-21077
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21077.01.patch, HIVE-21077.02.patch,
> HIVE-21077.03.patch, HIVE-21077.04.patch, HIVE-21077.05.patch,
> HIVE-21077.06.patch, HIVE-21077.07.patch, HIVE-21077.08.branch-3.patch,
> HIVE-21077.09.patch, HIVE-21077.10.patch
>
>
> Currently, database do not have creation time like we have for tables and
> partitions.
> {noformat}
> // namespace for tables
> struct Database {
> 1: string name,
> 2: string description,
> 3: string locationUri,
> 4: map parameters, // properties associated with the
> database
> 5: optional PrincipalPrivilegeSet privileges,
> 6: optional string ownerName,
> 7: optional PrincipalType ownerType,
> 8: optional string catalogName
> }
> {noformat}
> Currently, without creationTime there is no way to identify if the copy of
> Database which a client has is the same as the one on the server if the name
> is same. Without object ids creationTime value is the only way currently to
> identify uniquely a instance of metastore object. It would be good to have
> Database creation time as well.
> Same applies for catalogs as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Status: Open (was: Patch Available) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch, HIVE-21052.8.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21116) HADOOP_CREDSTORE_PASSWORD is not populated under yarn.app.mapreduce.am.admin.user.env
[ https://issues.apache.org/jira/browse/HIVE-21116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21116: -- Attachment: HIVE-21116.4.patch > HADOOP_CREDSTORE_PASSWORD is not populated under > yarn.app.mapreduce.am.admin.user.env > -- > > Key: HIVE-21116 > URL: https://issues.apache.org/jira/browse/HIVE-21116 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21116.1.patch, HIVE-21116.2.patch, > HIVE-21116.3.patch, HIVE-21116.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Attachment: HIVE-21052.8.patch Status: Patch Available (was: Open) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch, HIVE-21052.8.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21077) Database and catalogs should have creation time
[
https://issues.apache.org/jira/browse/HIVE-21077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745631#comment-16745631
]
Hive QA commented on HIVE-21077:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
11s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
59s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15672/dev-support/hive-personality.sh
|
| git revision | master / 67284b0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15672/yetus/whitespace-eol.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15672/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Database and catalogs should have creation time
> ---
>
> Key: HIVE-21077
> URL: https://issues.apache.org/jira/browse/HIVE-21077
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21077.01.patch, HIVE-21077.02.patch,
> HIVE-21077.03.patch, HIVE-21077.04.patch, HIVE-21077.05.patch,
> HIVE-21077.06.patch, HIVE-21077.07.patch, HIVE-21077.08.branch-3.patch,
> HIVE-21077.09.patch, HIVE-21077.10.patch
>
>
> Currently, database do not have creation time like we have for tables and
> partitions.
> {noformat}
> // namespace for tables
> struct Database {
> 1: string name,
> 2: string description,
> 3: string locationUri,
> 4: map
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745627#comment-16745627
]
Eugene Koifman commented on HIVE-21052:
---
there is 1 \{{writeId}} per (table, txnid) - HIVE-21118
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Critical
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
> HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch
>
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745618#comment-16745618 ] Jaume M commented on HIVE-21052: I was assuming it's possible there are different writeIds for the same table. Is this possible? In this case a several rows would be added with the same table and a different writeId. > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21132) Semi join edge is not being removed despite max bloomfilter entries set to 1
[
https://issues.apache.org/jira/browse/HIVE-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21132:
---
Status: Patch Available (was: Open)
> Semi join edge is not being removed despite max bloomfilter entries set to 1
>
>
> Key: HIVE-21132
> URL: https://issues.apache.org/jira/browse/HIVE-21132
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21132.1.patch
>
>
> * Reproducer
> {code:sql}
> --! qt:dataset:lineitem
> --! qt:dataset:part
> --! qt:dataset:src
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> --set hive.compute.query.using.stats=false;
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.tez.dynamic.partition.pruning=true;
> set hive.tez.dynamic.semijoin.reduction=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.stats.autogather=true;
> set hive.tez.bigtable.minsize.semijoin.reduction=1;
> set hive.tez.min.bloom.filter.entries=1;
> set hive.stats.fetch.column.stats=true;
> set hive.tez.bloom.filter.factor=1.0f;
> set hive.auto.convert.join=false;
> set hive.optimize.shared.work=false;
> create database tpch_test;
> use tpch_test;
> CREATE TABLE `customer`(
> `c_custkey` bigint,
> `c_name` string,
> `c_address` string,
> `c_nationkey` bigint,
> `c_phone` string,
> `c_acctbal` double,
> `c_mktsegment` string,
> `c_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026723');
> CREATE TABLE `lineitem`(
> `l_orderkey` bigint,
> `l_partkey` bigint,
> `l_suppkey` bigint,
> `l_linenumber` int,
> `l_quantity` double,
> `l_extendedprice` double,
> `l_discount` double,
> `l_tax` double,
> `l_returnflag` string,
> `l_linestatus` string,
> `l_shipdate` string,
> `l_commitdate` string,
> `l_receiptdate` string,
> `l_shipinstruct` string,
> `l_shipmode` string,
> `l_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543027179');
> CREATE TABLE `orders`(
> `o_orderkey` bigint,
> `o_custkey` bigint,
> `o_orderstatus` string,
> `o_totalprice` double,
> `o_orderdate` string,
> `o_orderpriority` string,
> `o_clerk` string,
> `o_shippriority` int,
> `o_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026824');
> alter table customer update statistics
> set('numRows'='15000','rawDataSize'='8633707142');
> alter table lineitem update statistics
> set('numRows'='589709','rawDataSize'='184245066955');
> alter table orders update statistics
> set('numRows'='15','rawDataSize'='46741318253');
> create view q18_tmp_cached as
> select l_orderkey, sum(l_quantity) as t_sum_quantity
> from lineitem
> where l_orderkey is not null
> group by l_orderkey;
> -- Set bloom filter size to huge number so we get any possible semijoin
> reductions
> set hive.tez.min.bloom.filter.entries=0;
> set hive.tez.max.bloom.filter.entries=1;
> create table q18_large_volume_customer_cached stored as orc tblproperties
> ('transactional'='true', 'transactional_properties'='default') as
> select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice,
> sum(l_quantity)
> from customer, orders, q18_tmp_cached t, lineitem l
> where
> c_custkey = o_custkey and o_orderkey = t.l_orderkey
> and o_orderkey is not null and t.t_sum_quantity > 300
> and o_orderkey = l.l_orderkey and l.l_orderkey is not null
> group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice
> order by o_totalprice desc, o_orderdate
> limit 100;
> drop database
[jira] [Updated] (HIVE-21132) Semi join edge is not being removed despite max bloomfilter entries set to 1
[
https://issues.apache.org/jira/browse/HIVE-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21132:
---
Attachment: HIVE-21132.1.patch
> Semi join edge is not being removed despite max bloomfilter entries set to 1
>
>
> Key: HIVE-21132
> URL: https://issues.apache.org/jira/browse/HIVE-21132
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21132.1.patch
>
>
> * Reproducer
> {code:sql}
> --! qt:dataset:lineitem
> --! qt:dataset:part
> --! qt:dataset:src
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> --set hive.compute.query.using.stats=false;
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.tez.dynamic.partition.pruning=true;
> set hive.tez.dynamic.semijoin.reduction=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.stats.autogather=true;
> set hive.tez.bigtable.minsize.semijoin.reduction=1;
> set hive.tez.min.bloom.filter.entries=1;
> set hive.stats.fetch.column.stats=true;
> set hive.tez.bloom.filter.factor=1.0f;
> set hive.auto.convert.join=false;
> set hive.optimize.shared.work=false;
> create database tpch_test;
> use tpch_test;
> CREATE TABLE `customer`(
> `c_custkey` bigint,
> `c_name` string,
> `c_address` string,
> `c_nationkey` bigint,
> `c_phone` string,
> `c_acctbal` double,
> `c_mktsegment` string,
> `c_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026723');
> CREATE TABLE `lineitem`(
> `l_orderkey` bigint,
> `l_partkey` bigint,
> `l_suppkey` bigint,
> `l_linenumber` int,
> `l_quantity` double,
> `l_extendedprice` double,
> `l_discount` double,
> `l_tax` double,
> `l_returnflag` string,
> `l_linestatus` string,
> `l_shipdate` string,
> `l_commitdate` string,
> `l_receiptdate` string,
> `l_shipinstruct` string,
> `l_shipmode` string,
> `l_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543027179');
> CREATE TABLE `orders`(
> `o_orderkey` bigint,
> `o_custkey` bigint,
> `o_orderstatus` string,
> `o_totalprice` double,
> `o_orderdate` string,
> `o_orderpriority` string,
> `o_clerk` string,
> `o_shippriority` int,
> `o_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026824');
> alter table customer update statistics
> set('numRows'='15000','rawDataSize'='8633707142');
> alter table lineitem update statistics
> set('numRows'='589709','rawDataSize'='184245066955');
> alter table orders update statistics
> set('numRows'='15','rawDataSize'='46741318253');
> create view q18_tmp_cached as
> select l_orderkey, sum(l_quantity) as t_sum_quantity
> from lineitem
> where l_orderkey is not null
> group by l_orderkey;
> -- Set bloom filter size to huge number so we get any possible semijoin
> reductions
> set hive.tez.min.bloom.filter.entries=0;
> set hive.tez.max.bloom.filter.entries=1;
> create table q18_large_volume_customer_cached stored as orc tblproperties
> ('transactional'='true', 'transactional_properties'='default') as
> select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice,
> sum(l_quantity)
> from customer, orders, q18_tmp_cached t, lineitem l
> where
> c_custkey = o_custkey and o_orderkey = t.l_orderkey
> and o_orderkey is not null and t.t_sum_quantity > 300
> and o_orderkey = l.l_orderkey and l.l_orderkey is not null
> group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice
> order by o_totalprice desc, o_orderdate
> limit 100;
> drop database tpch_t
[jira] [Commented] (HIVE-21132) Semi join edge is not being removed despite max bloomfilter entries set to 1
[
https://issues.apache.org/jira/browse/HIVE-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745608#comment-16745608
]
Vineet Garg commented on HIVE-21132:
Root cause is that Reduce de-duplication ends up messing up pattern of Semi
join branches (GB-RS-GB-RS)
> Semi join edge is not being removed despite max bloomfilter entries set to 1
>
>
> Key: HIVE-21132
> URL: https://issues.apache.org/jira/browse/HIVE-21132
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> * Reproducer
> {code:sql}
> --! qt:dataset:lineitem
> --! qt:dataset:part
> --! qt:dataset:src
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> --set hive.compute.query.using.stats=false;
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.tez.dynamic.partition.pruning=true;
> set hive.tez.dynamic.semijoin.reduction=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.stats.autogather=true;
> set hive.tez.bigtable.minsize.semijoin.reduction=1;
> set hive.tez.min.bloom.filter.entries=1;
> set hive.stats.fetch.column.stats=true;
> set hive.tez.bloom.filter.factor=1.0f;
> set hive.auto.convert.join=false;
> set hive.optimize.shared.work=false;
> create database tpch_test;
> use tpch_test;
> CREATE TABLE `customer`(
> `c_custkey` bigint,
> `c_name` string,
> `c_address` string,
> `c_nationkey` bigint,
> `c_phone` string,
> `c_acctbal` double,
> `c_mktsegment` string,
> `c_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026723');
> CREATE TABLE `lineitem`(
> `l_orderkey` bigint,
> `l_partkey` bigint,
> `l_suppkey` bigint,
> `l_linenumber` int,
> `l_quantity` double,
> `l_extendedprice` double,
> `l_discount` double,
> `l_tax` double,
> `l_returnflag` string,
> `l_linestatus` string,
> `l_shipdate` string,
> `l_commitdate` string,
> `l_receiptdate` string,
> `l_shipinstruct` string,
> `l_shipmode` string,
> `l_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543027179');
> CREATE TABLE `orders`(
> `o_orderkey` bigint,
> `o_custkey` bigint,
> `o_orderstatus` string,
> `o_totalprice` double,
> `o_orderdate` string,
> `o_orderpriority` string,
> `o_clerk` string,
> `o_shippriority` int,
> `o_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026824');
> alter table customer update statistics
> set('numRows'='15000','rawDataSize'='8633707142');
> alter table lineitem update statistics
> set('numRows'='589709','rawDataSize'='184245066955');
> alter table orders update statistics
> set('numRows'='15','rawDataSize'='46741318253');
> create view q18_tmp_cached as
> select l_orderkey, sum(l_quantity) as t_sum_quantity
> from lineitem
> where l_orderkey is not null
> group by l_orderkey;
> -- Set bloom filter size to huge number so we get any possible semijoin
> reductions
> set hive.tez.min.bloom.filter.entries=0;
> set hive.tez.max.bloom.filter.entries=1;
> create table q18_large_volume_customer_cached stored as orc tblproperties
> ('transactional'='true', 'transactional_properties'='default') as
> select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice,
> sum(l_quantity)
> from customer, orders, q18_tmp_cached t, lineitem l
> where
> c_custkey = o_custkey and o_orderkey = t.l_orderkey
> and o_orderkey is not null and t.t_sum_quantity > 300
> and o_orderkey = l.l_orderkey and l.l_orderkey is not null
> group by c_name, c_custkey, o_orderkey, o_orderdate, o_tota
[jira] [Updated] (HIVE-21132) Semi join edge is not being removed despite max bloomfilter entries set to 1
[
https://issues.apache.org/jira/browse/HIVE-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21132:
---
Description:
* Reproducer
{code:sql}
--! qt:dataset:lineitem
--! qt:dataset:part
--! qt:dataset:src
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
--set hive.compute.query.using.stats=false;
set hive.mapred.mode=nonstrict;
set hive.explain.user=false;
set hive.optimize.ppd=true;
set hive.ppd.remove.duplicatefilters=true;
set hive.tez.dynamic.partition.pruning=true;
set hive.tez.dynamic.semijoin.reduction=true;
set hive.optimize.metadataonly=false;
set hive.optimize.index.filter=true;
set hive.stats.autogather=true;
set hive.tez.bigtable.minsize.semijoin.reduction=1;
set hive.tez.min.bloom.filter.entries=1;
set hive.stats.fetch.column.stats=true;
set hive.tez.bloom.filter.factor=1.0f;
set hive.auto.convert.join=false;
set hive.optimize.shared.work=false;
create database tpch_test;
use tpch_test;
CREATE TABLE `customer`(
`c_custkey` bigint,
`c_name` string,
`c_address` string,
`c_nationkey` bigint,
`c_phone` string,
`c_acctbal` double,
`c_mktsegment` string,
`c_comment` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
TBLPROPERTIES (
'bucketing_version'='2',
'transactional'='true',
'transactional_properties'='default',
'transient_lastDdlTime'='1543026723');
CREATE TABLE `lineitem`(
`l_orderkey` bigint,
`l_partkey` bigint,
`l_suppkey` bigint,
`l_linenumber` int,
`l_quantity` double,
`l_extendedprice` double,
`l_discount` double,
`l_tax` double,
`l_returnflag` string,
`l_linestatus` string,
`l_shipdate` string,
`l_commitdate` string,
`l_receiptdate` string,
`l_shipinstruct` string,
`l_shipmode` string,
`l_comment` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
TBLPROPERTIES (
'bucketing_version'='2',
'transactional'='true',
'transactional_properties'='default',
'transient_lastDdlTime'='1543027179');
CREATE TABLE `orders`(
`o_orderkey` bigint,
`o_custkey` bigint,
`o_orderstatus` string,
`o_totalprice` double,
`o_orderdate` string,
`o_orderpriority` string,
`o_clerk` string,
`o_shippriority` int,
`o_comment` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
TBLPROPERTIES (
'bucketing_version'='2',
'transactional'='true',
'transactional_properties'='default',
'transient_lastDdlTime'='1543026824');
alter table customer update statistics
set('numRows'='15000','rawDataSize'='8633707142');
alter table lineitem update statistics
set('numRows'='589709','rawDataSize'='184245066955');
alter table orders update statistics
set('numRows'='15','rawDataSize'='46741318253');
create view q18_tmp_cached as
select l_orderkey, sum(l_quantity) as t_sum_quantity
from lineitem
where l_orderkey is not null
group by l_orderkey;
-- Set bloom filter size to huge number so we get any possible semijoin
reductions
set hive.tez.min.bloom.filter.entries=0;
set hive.tez.max.bloom.filter.entries=1;
create table q18_large_volume_customer_cached stored as orc tblproperties
('transactional'='true', 'transactional_properties'='default') as
select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity)
from customer, orders, q18_tmp_cached t, lineitem l
where
c_custkey = o_custkey and o_orderkey = t.l_orderkey
and o_orderkey is not null and t.t_sum_quantity > 300
and o_orderkey = l.l_orderkey and l.l_orderkey is not null
group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice
order by o_totalprice desc, o_orderdate
limit 100;
drop database tpch_test cascade;
{code}
To reproduce run the above as TestMiniLlapLocalCliDriver test
> Semi join edge is not being removed despite max bloomfilter entries set to 1
>
>
> Key: HIVE-21132
> URL: https://issues.apache.org/jira/browse/HIVE-21132
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> * Reproducer
> {code:sql}
> --! qt:dataset:lineitem
> --! qt:dataset:part
> --! qt:dataset:src
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> --
[jira] [Assigned] (HIVE-21132) Semi join edge is not being removed despite max bloomfilter entries set to 1
[ https://issues.apache.org/jira/browse/HIVE-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-21132: -- > Semi join edge is not being removed despite max bloomfilter entries set to 1 > > > Key: HIVE-21132 > URL: https://issues.apache.org/jira/browse/HIVE-21132 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21107: --- Attachment: HIVE-21107.5.patch > Cannot find field" error during dynamically partitioned hash join > - > > Key: HIVE-21107 > URL: https://issues.apache.org/jira/browse/HIVE-21107 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch, > HIVE-21107.3.patch, HIVE-21107.4.patch, HIVE-21107.5.patch > > > This occurs in non-CBO path with dynamic partitioned join + constant > propagation ON. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745604#comment-16745604
]
Hive QA commented on HIVE-21107:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955301/HIVE-21107.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 15696 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestActivePassiveHA.testActivePassiveHA (batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testClientConnectionsOnFailover
(batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testConnectionActivePassiveHAServiceDiscovery
(batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover (batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailoverUnauthorized
(batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testNoConnectionOnPassive (batchId=261)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15671/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15671/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15671/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955301 - PreCommit-HIVE-Build
> Cannot find field" error during dynamically partitioned hash join
> -
>
> Key: HIVE-21107
> URL: https://issues.apache.org/jira/browse/HIVE-21107
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch,
> HIVE-21107.3.patch, HIVE-21107.4.patch
>
>
> This occurs in non-CBO path with dynamic partitioned join + constant
> propagation ON.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745593#comment-16745593
]
Eugene Koifman commented on HIVE-21052:
---
[~jmarhuen], I'm not sure I understand your 1st 2 bullet points. we currently
only support auto-commit mode and all the locks for a given statement are
processed in a single call to {{lock(LockRequest rqst)}} so you should see the
full set of tables and corresponding {{writeID}}. So in the absence of retries
(of the HMS call), I'd expect TXN_COMPONENTS to have a single 'p' type row for
a given (table, txn) combination. (Implicitly, each table gets only 1
{{writeID}} within a given txn.)
Are we saying the same thing?
If retries cause multiple p-type entires for (table, txn) that should be
harmless. As you say, Initiator would only make 1 {{COMPACTION_QUEUE}} entry
and {{Cleaner}} will clean data for all aborted txns for a given table based on
that queue entry.
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Critical
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
> HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch
>
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21061) CTAS query fails with IllegalStateException for empty source
[
https://issues.apache.org/jira/browse/HIVE-21061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745586#comment-16745586
]
Vaibhav Gumashta edited comment on HIVE-21061 at 1/17/19 10:42 PM:
---
-+1 pending tests-
Actually, before I +1, possible to add a simple test case for this?
was (Author: vgumashta):
+1 pending tests
> CTAS query fails with IllegalStateException for empty source
>
>
> Key: HIVE-21061
> URL: https://issues.apache.org/jira/browse/HIVE-21061
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Attachments: HIVE-21061.patch
>
>
> Creating a table using CTAS from an empty source table with predicate
> condition evaluating to False
> {code}
> create table testctas1 (id int);
> create table testctas2 as select * from testctas1 where 1=2;
> {code}
> Fails with below exception:
> {code}
> Caused by: java.lang.IllegalStateException: null
> at com.google.common.base.Preconditions.checkState(Preconditions.java:159)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.verifyAndSetVectorPartDesc(Vectorizer.java:1312)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateInputFormatAndSchemaEvolution(Vectorizer.java:1654)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(Vectorizer.java:1865)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:1109)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:961)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:2442)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:717)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:258)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12443)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1863)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1810)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1805)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
> ... 36 more
> {code}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21061) CTAS query fails with IllegalStateException for empty source
[
https://issues.apache.org/jira/browse/HIVE-21061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745586#comment-16745586
]
Vaibhav Gumashta commented on HIVE-21061:
-
+1 pending tests
> CTAS query fails with IllegalStateException for empty source
>
>
> Key: HIVE-21061
> URL: https://issues.apache.org/jira/browse/HIVE-21061
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Attachments: HIVE-21061.patch
>
>
> Creating a table using CTAS from an empty source table with predicate
> condition evaluating to False
> {code}
> create table testctas1 (id int);
> create table testctas2 as select * from testctas1 where 1=2;
> {code}
> Fails with below exception:
> {code}
> Caused by: java.lang.IllegalStateException: null
> at com.google.common.base.Preconditions.checkState(Preconditions.java:159)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.verifyAndSetVectorPartDesc(Vectorizer.java:1312)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateInputFormatAndSchemaEvolution(Vectorizer.java:1654)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(Vectorizer.java:1865)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertMapWork(Vectorizer.java:1109)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:961)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:2442)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeTaskPlan(TezCompiler.java:717)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:258)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12443)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:358)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1863)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1810)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1805)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
> ... 36 more
> {code}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20776) Run HMS filterHooks on server-side in addition to client-side
[ https://issues.apache.org/jira/browse/HIVE-20776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated HIVE-20776: - Attachment: HIVE-20776.012.patch > Run HMS filterHooks on server-side in addition to client-side > - > > Key: HIVE-20776 > URL: https://issues.apache.org/jira/browse/HIVE-20776 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Karthik Manamcheri >Assignee: Na Li >Priority: Major > Attachments: HIVE-20776.001.patch, HIVE-20776.003.patch, > HIVE-20776.004.patch, HIVE-20776.005.patch, HIVE-20776.006.patch, > HIVE-20776.007.patch, HIVE-20776.007.patch, HIVE-20776.008.patch, > HIVE-20776.009.patch, HIVE-20776.009.patch, HIVE-20776.010.patch, > HIVE-20776.011.patch, HIVE-20776.011.patch, HIVE-20776.012.patch > > > In HMS, I noticed that all the filter hooks are applied on the client side > (in HiveMetaStoreClient.java). Is there any reason why we can't apply the > filters on the server-side? > Motivation: Some newer apache projects such as Kudu use HMS for metadata > storage. Kudu is not completely Java-based and there are interaction points > where they have C++ clients. In such cases, it would be ideal to have > consistent behavior from HMS side as far as filters, etc are concerned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20198) Constant time table drops/renames
[ https://issues.apache.org/jira/browse/HIVE-20198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745561#comment-16745561 ] Vihang Karajgaonkar commented on HIVE-20198: Thanks [~ekoifman] very interesting to know that. > Constant time table drops/renames > - > > Key: HIVE-20198 > URL: https://issues.apache.org/jira/browse/HIVE-20198 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Alexander Kolbasov >Assignee: Vihang Karajgaonkar >Priority: Major > > Currently table drops and table renames have O(P) performance (where P is the > number of partitions). When a managed table is deleted, the implementation > deletes table metadata and then deletes all partitions in HDFS. HDFS > operations are optimized and only do a sequential deletes for partitions > outside of table prefix. This operation is O(P)where Pis the number of > partitions. > Table rename goes through the list of partitions and modifies table name (and > potentially db name) in each partition. It also modifies each partition > location to match the new db/table name and renames directories (which is a > non-atomic and slow operation on S3). This is O(P) operation where P is the > number of partitions. > Basic idea is to do the following: > # Assign unique ID to each table > # Create directory name based on unique ID rather then the name > # Table rename then becomes metadata-only operation - there is no need to > change any location information. > # Table drop can become an asynchronous operation where the table is marked > as "deleted". Subsequent public metadata APIs should skip such tables. A > background cleaner thread may then go and clean up directories. > Since the table location is unique for each table, new tables will not reuse > existing locations. This change isn't compatible with the current behavior > where there is an assumption that table location is based on table name. We > can get around this by providing "opt-in" mechanism - special table property > that tells that the table can have such new behavior, so the improvement will > initially work for new tables created with this feature enabled. We may later > provide some tool to convert existing tables to the new scheme. > One complication is there in case where impersonation is enabled - the FS > operations should be performed using client UGI rather then server's, so the > cleaner thread should be able to use client UGIs. > Initially we can punt on this and do standard table drops when impersonation > is enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745559#comment-16745559
]
Hive QA commented on HIVE-21107:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 74 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15671/dev-support/hive-personality.sh
|
| git revision | master / 67284b0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15671/yetus/whitespace-eol.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15671/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Cannot find field" error during dynamically partitioned hash join
> -
>
> Key: HIVE-21107
> URL: https://issues.apache.org/jira/browse/HIVE-21107
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch,
> HIVE-21107.3.patch, HIVE-21107.4.patch
>
>
> This occurs in non-CBO path with dynamic partitioned join + constant
> propagation ON.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20707) Automatic partition management
[ https://issues.apache.org/jira/browse/HIVE-20707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745552#comment-16745552 ] Jesus Camacho Rodriguez commented on HIVE-20707: Thanks for letting me know [~prasanth_j]. HIVE-21072 already has the (small) part of the patch that would fix that issue described over there, hence I think we will just backport that. > Automatic partition management > -- > > Key: HIVE-20707 > URL: https://issues.apache.org/jira/browse/HIVE-20707 > Project: Hive > Issue Type: New Feature >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20702.3.patch, HIVE-20707-branch-3.patch, > HIVE-20707.1.patch, HIVE-20707.2.patch, HIVE-20707.4.patch, > HIVE-20707.5.patch, HIVE-20707.6.patch, HIVE-20707.6.patch, HIVE-20707.7.patch > > > In current scenario, to add partitions for external tables to metastore, MSCK > REPAIR command has to be executed manually. To avoid this manual step, > external tables can be specified a table property based on which a background > metastore thread can sync partitions periodically. Tables can also be > specified with partition retention period. Any partition whose age exceeds > the retention period will be dropped automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20198) Constant time table drops/renames
[
https://issues.apache.org/jira/browse/HIVE-20198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745538#comment-16745538
]
Eugene Koifman commented on HIVE-20198:
---
FYI, {{TBLS.TBL_ID}} is exposed via Thrift since HIVE-20556.
> Constant time table drops/renames
> -
>
> Key: HIVE-20198
> URL: https://issues.apache.org/jira/browse/HIVE-20198
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Alexander Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
>
> Currently table drops and table renames have O(P) performance (where P is the
> number of partitions). When a managed table is deleted, the implementation
> deletes table metadata and then deletes all partitions in HDFS. HDFS
> operations are optimized and only do a sequential deletes for partitions
> outside of table prefix. This operation is O(P)where Pis the number of
> partitions.
> Table rename goes through the list of partitions and modifies table name (and
> potentially db name) in each partition. It also modifies each partition
> location to match the new db/table name and renames directories (which is a
> non-atomic and slow operation on S3). This is O(P) operation where P is the
> number of partitions.
> Basic idea is to do the following:
> # Assign unique ID to each table
> # Create directory name based on unique ID rather then the name
> # Table rename then becomes metadata-only operation - there is no need to
> change any location information.
> # Table drop can become an asynchronous operation where the table is marked
> as "deleted". Subsequent public metadata APIs should skip such tables. A
> background cleaner thread may then go and clean up directories.
> Since the table location is unique for each table, new tables will not reuse
> existing locations. This change isn't compatible with the current behavior
> where there is an assumption that table location is based on table name. We
> can get around this by providing "opt-in" mechanism - special table property
> that tells that the table can have such new behavior, so the improvement will
> initially work for new tables created with this feature enabled. We may later
> provide some tool to convert existing tables to the new scheme.
> One complication is there in case where impersonation is enabled - the FS
> operations should be performed using client UGI rather then server's, so the
> cleaner thread should be able to use client UGIs.
> Initially we can punt on this and do standard table drops when impersonation
> is enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20546) Upgrade to Druid 0.13.0
[ https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20546: Attachment: HIVE-20546.2.patch > Upgrade to Druid 0.13.0 > --- > > Key: HIVE-20546 > URL: https://issues.apache.org/jira/browse/HIVE-20546 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20546.1.patch, HIVE-20546.2.patch, HIVE-20546.patch > > > This task is to upgrade to druid 0.13.0 when it is released. Note that it > will hopefully be first apache release for Druid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745533#comment-16745533 ] Vineet Garg commented on HIVE-21107: Latest patch addresses the review comment. > Cannot find field" error during dynamically partitioned hash join > - > > Key: HIVE-21107 > URL: https://issues.apache.org/jira/browse/HIVE-21107 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch, > HIVE-21107.3.patch, HIVE-21107.4.patch > > > This occurs in non-CBO path with dynamic partitioned join + constant > propagation ON. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20546) Upgrade to Apache Druid 0.13.0-incubating
[ https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20546: Summary: Upgrade to Apache Druid 0.13.0-incubating (was: Upgrade to Apache Druid 0.13.0) > Upgrade to Apache Druid 0.13.0-incubating > - > > Key: HIVE-20546 > URL: https://issues.apache.org/jira/browse/HIVE-20546 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20546.1.patch, HIVE-20546.2.patch, HIVE-20546.patch > > > This task is to upgrade to druid 0.13.0 when it is released. Note that it > will hopefully be first apache release for Druid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20546) Upgrade to Apache Druid 0.13.0
[ https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20546: Summary: Upgrade to Apache Druid 0.13.0 (was: Upgrade to Druid 0.13.0) > Upgrade to Apache Druid 0.13.0 > -- > > Key: HIVE-20546 > URL: https://issues.apache.org/jira/browse/HIVE-20546 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20546.1.patch, HIVE-20546.2.patch, HIVE-20546.patch > > > This task is to upgrade to druid 0.13.0 when it is released. Note that it > will hopefully be first apache release for Druid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21107: --- Status: Patch Available (was: Open) > Cannot find field" error during dynamically partitioned hash join > - > > Key: HIVE-21107 > URL: https://issues.apache.org/jira/browse/HIVE-21107 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch, > HIVE-21107.3.patch, HIVE-21107.4.patch > > > This occurs in non-CBO path with dynamic partitioned join + constant > propagation ON. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21107: --- Status: Open (was: Patch Available) > Cannot find field" error during dynamically partitioned hash join > - > > Key: HIVE-21107 > URL: https://issues.apache.org/jira/browse/HIVE-21107 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch, > HIVE-21107.3.patch, HIVE-21107.4.patch > > > This occurs in non-CBO path with dynamic partitioned join + constant > propagation ON. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21107: --- Attachment: HIVE-21107.4.patch > Cannot find field" error during dynamically partitioned hash join > - > > Key: HIVE-21107 > URL: https://issues.apache.org/jira/browse/HIVE-21107 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch, > HIVE-21107.3.patch, HIVE-21107.4.patch > > > This occurs in non-CBO path with dynamic partitioned join + constant > propagation ON. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21077) Database and catalogs should have creation time
[
https://issues.apache.org/jira/browse/HIVE-21077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-21077:
---
Attachment: HIVE-21077.10.patch
> Database and catalogs should have creation time
> ---
>
> Key: HIVE-21077
> URL: https://issues.apache.org/jira/browse/HIVE-21077
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21077.01.patch, HIVE-21077.02.patch,
> HIVE-21077.03.patch, HIVE-21077.04.patch, HIVE-21077.05.patch,
> HIVE-21077.06.patch, HIVE-21077.07.patch, HIVE-21077.08.branch-3.patch,
> HIVE-21077.09.patch, HIVE-21077.10.patch
>
>
> Currently, database do not have creation time like we have for tables and
> partitions.
> {noformat}
> // namespace for tables
> struct Database {
> 1: string name,
> 2: string description,
> 3: string locationUri,
> 4: map parameters, // properties associated with the
> database
> 5: optional PrincipalPrivilegeSet privileges,
> 6: optional string ownerName,
> 7: optional PrincipalType ownerType,
> 8: optional string catalogName
> }
> {noformat}
> Currently, without creationTime there is no way to identify if the copy of
> Database which a client has is the same as the one on the server if the name
> is same. Without object ids creationTime value is the only way currently to
> identify uniquely a instance of metastore object. It would be good to have
> Database creation time as well.
> Same applies for catalogs as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21077) Database and catalogs should have creation time
[
https://issues.apache.org/jira/browse/HIVE-21077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745529#comment-16745529
]
Vihang Karajgaonkar commented on HIVE-21077:
It looks like environment issue. The build works for me locally. Reattaaching
> Database and catalogs should have creation time
> ---
>
> Key: HIVE-21077
> URL: https://issues.apache.org/jira/browse/HIVE-21077
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21077.01.patch, HIVE-21077.02.patch,
> HIVE-21077.03.patch, HIVE-21077.04.patch, HIVE-21077.05.patch,
> HIVE-21077.06.patch, HIVE-21077.07.patch, HIVE-21077.08.branch-3.patch,
> HIVE-21077.09.patch, HIVE-21077.10.patch
>
>
> Currently, database do not have creation time like we have for tables and
> partitions.
> {noformat}
> // namespace for tables
> struct Database {
> 1: string name,
> 2: string description,
> 3: string locationUri,
> 4: map parameters, // properties associated with the
> database
> 5: optional PrincipalPrivilegeSet privileges,
> 6: optional string ownerName,
> 7: optional PrincipalType ownerType,
> 8: optional string catalogName
> }
> {noformat}
> Currently, without creationTime there is no way to identify if the copy of
> Database which a client has is the same as the one on the server if the name
> is same. Without object ids creationTime value is the only way currently to
> identify uniquely a instance of metastore object. It would be good to have
> Database creation time as well.
> Same applies for catalogs as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745523#comment-16745523
]
Hive QA commented on HIVE-20233:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955297/HIVE-20233.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15670/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15670/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15670/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-17 21:23:36.098
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15670/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 21:23:36.101
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
f09db52..67284b0 master -> origin/master
+ git reset --hard HEAD
HEAD is now at f09db52 HIVE-20170: Improve JoinOperator "rows for join key"
Logging (BELUGA BEHR via Peter Vary)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 67284b0 HIVE-18884 Simplify Logging in Hive Metastore Client
(Mani M via Alan Gates)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 21:23:36.970
+ rm -rf ../yetus_PreCommit-HIVE-Build-15670
+ mkdir ../yetus_PreCommit-HIVE-Build-15670
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15670
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15670/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java: does not
exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc265097300568903570.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc265097300568903570.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc7495757597411567234.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
log4j:WARN No appenders could be found for logger (DataNucleus.Persistence).
log4j:WARN Please initialize the log4j system properly.
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer completed with success for 41 classes.
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/ql/target/generated-sources/antlr3/org/a
[jira] [Assigned] (HIVE-21131) Document some of the static util methods in MetastoreUtils
[
https://issues.apache.org/jira/browse/HIVE-21131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar reassigned HIVE-21131:
--
> Document some of the static util methods in MetastoreUtils
> --
>
> Key: HIVE-21131
> URL: https://issues.apache.org/jira/browse/HIVE-21131
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Trivial
>
> {{MetastoreUtils}} has some methods like {{makePartNameMatcher}} which could
> use some javadoc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18884) Simplify Logging in Hive Metastore Client
[
https://issues.apache.org/jira/browse/HIVE-18884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-18884:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch 9 committed to master. I ran the metadata test that failed locally and
it was fine. Thanks Mani for the patch and for your patience with our
imperfect build system.
> Simplify Logging in Hive Metastore Client
> -
>
> Key: HIVE-18884
> URL: https://issues.apache.org/jira/browse/HIVE-18884
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Mani M
>Priority: Minor
> Labels: logging, newbie, noob
> Attachments: HIVE.18884.02.patch, HIVE.18884.03.patch,
> HIVE.18884.04.patch, HIVE.18884.05.patch, HIVE.18884.06.patch,
> HIVE.18884.07.patch, HIVE.18884.08.patch, HIVE.18884.09.patch,
> HIVE.18884.patch
>
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
> The current logging is:
> {code}
> 2018-02-26 07:02:44,883 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Trying to connect to metastore with URI
> thrift://host.company.com:9083
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Connected to metastore.
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Opened a connection to metastore, current connections: 2
> {code}
> Please simplify to something like:
> {code}
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Opened a connection to the Metastore Server (URI
> thrift://host.company.com:9083), current connections: 2
> ... or ...
> 2018-02-26 07:02:44,892 ERROR hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Failed to connect to the Metastore Server (URI
> thrift://host.company.com:9083)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20233:
---
Status: Patch Available (was: Open)
Last build failed seemingly related to a network issue. Trying again.
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch,
> HIVE-20233.3.patch, HIVE-20233.4.patch, HIVE-20233.4.patch, HIVE-20233.5.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20233:
---
Status: Open (was: Patch Available)
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch,
> HIVE-20233.3.patch, HIVE-20233.4.patch, HIVE-20233.4.patch, HIVE-20233.5.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20233:
---
Attachment: HIVE-20233.5.patch
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch,
> HIVE-20233.3.patch, HIVE-20233.4.patch, HIVE-20233.4.patch, HIVE-20233.5.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21034) Add option to schematool to drop Hive databases
[
https://issues.apache.org/jira/browse/HIVE-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745467#comment-16745467
]
Alan Gates commented on HIVE-21034:
---
Option 2 seems like a hassle for everyone, so I agree it's out.
Rather than an environment variable, why not just have the tool prompt and say
"are you sure y/[n]" with no as the default. A --yes option can be added to
the command line that answers all questions with yes so that the tool can still
be scripted. This seems to be the way tools like rpm usually handle this.
> Add option to schematool to drop Hive databases
> ---
>
> Key: HIVE-21034
> URL: https://issues.apache.org/jira/browse/HIVE-21034
> Project: Hive
> Issue Type: Improvement
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
>
> An option to remove all Hive managed data could be a useful addition to
> {{schematool}}.
> I propose to introduce a new flag {{-dropAllDatabases}} that would *drop all
> databases with CASCADE* to remove all data of managed tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21130) Mappers stuck after initializing MapOperator
[ https://issues.apache.org/jira/browse/HIVE-21130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muthu Manickam updated HIVE-21130: -- Description: Hive queries are stuck after initializing MapOperator. These hive queries are simple CTAS reading from a Hive table backed by rcfile format. This table has 7500 partitions and 110 columns with column data types restricted to string and int. This is being on run on a EMR cluster with 100 data nodes with enough memory/cores. After the query is submitted, YARN allocates the necessary containers. All the mapper tasks are in RUNNING state and all the map tasks reach the stage of initializing MapOperator and get stuck. Here is log message from the map tasks. 2019-01-17 *15:02:06,262* INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Initializing operator MAP[0] 2019-01-17 *15:08:22,093* INFO [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper: _*You can see that task is stuck for 6 minutes. This even gets to 20 minutes depending upon the number of parallel queries on the cluster. After the ExceMapper starts, the query completes in a minute.*_ I also noticed tread dumps in the logs that lead to a finding where it is spending all the time in this menthod *setReadNestedColumnPathConf().* "main" #1 prio=5 os_prio=0 tid=0x7f4cd805e800 nid=0x18074 runnable [0x7f4cded7f000] java.lang.Thread.State: RUNNABLE at java.lang.String.toLowerCase(String.java:2670) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.*setReadNestedColumnPathConf(ColumnProjectionUtils.java:223)* at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendNestedColumnPaths(ColumnProjectionUtils.java:145) at org.apache.hadoop.hive.ql.exec.MapOperator.cloneConfsForNestedColPruning(MapOperator.java:365) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:419) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:106) Based on the error, *I tried running the same query with 1 to 5 columns in the select clause and it runs quickly as expected. If I include more columns in the select clause, it falls in to the same issue with long pause times between MapOperator initialization and ExecMapper.* The error is very similar to the one in this Jira https://issues.apache.org/jira/browse/HIVE-16969 was: Hive queries are stuck after initializing MapOperator. These hive queries are simple CTAS reading from a Hive table backed by rcfile format. This table has 7500 partitions and 110 columns with column data types restricted to string and int. This is being on run on a EMR cluster with 100 data nodes with enough memory/cores. After the query is submitted, YARN allocates the necessary containers. All the mapper tasks are in RUNNING state and all the map tasks reach this stage of initializing MapOperator. Here is log message from the map tasks. 2019-01-17 *15:02:06,262* INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Initializing operator MAP[0] 2019-01-17 *15:08:22,093* INFO [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper: _*After this message, they are stuck for 6 minutes. This even gets to 20 minutes depending upon the number of parallel queries on the cluster. After the ExceMapper starts, the query completes in a minute.*_ I also noticed tread dumps in the logs that lead to a finding where it is spending all the time in this menthod *setReadNestedColumnPathConf().* "main" #1 prio=5 os_prio=0 tid=0x7f4cd805e800 nid=0x18074 runnable [0x7f4cded7f000] java.lang.Thread.State: RUNNABLE at java.lang.String.toLowerCase(String.java:2670) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.*setReadNestedColumnPathConf(ColumnProjectionUtils.java:223)* at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendNestedColumnPaths(ColumnProjectionUtils.java:145) at org.apache.hadoop.hive.ql.exec.MapOperator.cloneConfsForNestedColPruning(MapOperator.java:365) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:419) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:106) Based on the error, *I tried running the same query with 1 to 5 columns in the select clause and it runs quickly as expected. If I include more columns in the select clause, it falls in to the same issue with long pause times between MapOperator initialization and ExecMapper.* The error is very similar to the one in this Jira https://issues.apache.org/jira/browse/HIVE-16969 > Mappers stuck after initializing MapOperator > > > Key: HIVE-21130 > URL: https://issues.apache.org/jira/browse/HIVE-21130 > Project: Hive > Issue Type: Bug > Components: Operators >Affects Versions: 2.3.2 >Reporter: Muthu Manickam >Priority: Crit
[jira] [Commented] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745433#comment-16745433
]
Hive QA commented on HIVE-20233:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955290/HIVE-20233.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15669/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15669/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15669/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-17 19:50:57.731
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15669/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 19:50:57.735
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at f09db52 HIVE-20170: Improve JoinOperator "rows for join key"
Logging (BELUGA BEHR via Peter Vary)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at f09db52 HIVE-20170: Improve JoinOperator "rows for join key"
Logging (BELUGA BEHR via Peter Vary)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 19:50:58.735
+ rm -rf ../yetus_PreCommit-HIVE-Build-15669
+ mkdir ../yetus_PreCommit-HIVE-Build-15669
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15669
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15669/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java: does not
exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc77057751180628816.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc77057751180628816.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
[ERROR] Failed to execute goal on project hive-shims-0.23: Could not resolve
dependencies for project
org.apache.hive.shims:hive-shims-0.23:jar:4.0.0-SNAPSHOT: The following
artifacts could not be resolved:
org.eclipse.jetty:jetty-server:jar:9.3.25.v20180904,
org.eclipse.jetty:jetty-http:jar:9.3.25.v20180904,
org.eclipse.jetty:jetty-io:jar:9.3.25.v20180904,
org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:3.1.0,
org.apache.hadoop:hadoop-yarn-server-common:jar:3.1.0,
org.apache.hadoop:hadoop-yarn-registry:jar:3.1.0, dnsjava:dnsjava:jar:2.1.7,
org.apache.geronimo.specs:geronimo-jcache_1.0_spec:jar:1.0-alpha-1,
org.ehcache:ehcache:jar:3.3.1, com.zaxxer:HikariCP-java7:jar:2.4.12,
com.microsoft.sqlserver:mssql-jdbc:jar:6.2.1.jre7,
org.apache.hadoop:hadoop-yarn-server-applicationhistoryservice:jar:3.1.0,
de.ruedigermoeller:fst:jar:2.50, com.cedarsoftware:java-util:jar:1.9.0,
com.cedarsoftware:json-io:jar:2.5.1,
org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:3.1.0,
org.apache.hadoop:hadoop-yarn-server-tests:jar:tests:3.1.0,
org.apache.hadoop:h
[jira] [Updated] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20233:
---
Status: Patch Available (was: Open)
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch,
> HIVE-20233.3.patch, HIVE-20233.4.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20233:
---
Status: Patch Available (was: Open)
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch,
> HIVE-20233.3.patch, HIVE-20233.4.patch, HIVE-20233.4.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20233:
---
Status: Open (was: Patch Available)
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch,
> HIVE-20233.3.patch, HIVE-20233.4.patch, HIVE-20233.4.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20233:
---
Attachment: HIVE-20233.4.patch
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch,
> HIVE-20233.3.patch, HIVE-20233.4.patch, HIVE-20233.4.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745420#comment-16745420
]
Hive QA commented on HIVE-20233:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955287/HIVE-20233.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15668/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15668/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15668/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-17 19:33:26.653
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15668/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 19:33:26.656
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
8e7c3b3..f09db52 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 8e7c3b3 HIVE-21124: HPL/SQL does not support the CREATE TABLE
LIKE statement (Baoning He, reviewed by Daniel Dai)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at f09db52 HIVE-20170: Improve JoinOperator "rows for join key"
Logging (BELUGA BEHR via Peter Vary)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 19:33:28.051
+ rm -rf ../yetus_PreCommit-HIVE-Build-15668
+ mkdir ../yetus_PreCommit-HIVE-Build-15668
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15668
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15668/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java: does not
exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] Failed to execute goal on project hive-shims-common: Could not resolve
dependencies for project
org.apache.hive.shims:hive-shims-common:jar:4.0.0-SNAPSHOT: Could not find
artifact org.apache.zookeeper:zookeeper:jar:3.4.6 in datanucleus
(http://www.datanucleus.org/downloads/maven2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-shims-common
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-15668
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955287 - PreCommit-HIVE-Build
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch,
> HIVE-20233.3.patch, HIVE-20233.4.patch
>
>
> Various improvements
[jira] [Updated] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20233:
---
Attachment: HIVE-20233.4.patch
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch,
> HIVE-20233.3.patch, HIVE-20233.4.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20233:
---
Status: Open (was: Patch Available)
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch,
> HIVE-20233.3.patch, HIVE-20233.4.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20170) Improve JoinOperator "rows for join key" Logging
[
https://issues.apache.org/jira/browse/HIVE-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-20170:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for the patch [~belugabehr]!
> Improve JoinOperator "rows for join key" Logging
>
>
> Key: HIVE-20170
> URL: https://issues.apache.org/jira/browse/HIVE-20170
> Project: Hive
> Issue Type: Improvement
> Components: Operators
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20170.1.patch, HIVE-20170.10.patch,
> HIVE-20170.2.patch, HIVE-20170.3.patch, HIVE-20170.4.patch,
> HIVE-20170.5.patch, HIVE-20170.6.patch, HIVE-20170.7.patch,
> HIVE-20170.8.patch, HIVE-20170.9.patch
>
>
> {code:java}
> 2018-06-25 09:37:33,193 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5728000 rows
> for join key [333, 22]
> 2018-06-25 09:37:33,901 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5828000 rows
> for join key [333, 22]
> 2018-06-25 09:37:34,623 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5928000 rows
> for join key [333, 22]
> 2018-06-25 09:37:35,342 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 6028000 rows
> for join key [333, 22]
> {code}
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java#L120]
> This logging should use the same facilities as the other Operators for
> emitting this type of log message. HIVE-10078
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.18
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745293#comment-16745293
]
Hive QA commented on HIVE-21001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955267/HIVE-21001.09.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15667/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15667/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15667/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-17 17:02:53.768
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15667/source-prep.txt
+ [[ true == \t\r\u\e ]]
+ rm -rf ivy maven
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 17:02:54.542
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 8e7c3b3 HIVE-21124: HPL/SQL does not support the CREATE TABLE
LIKE statement (Baoning He, reviewed by Daniel Dai)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 8e7c3b3 HIVE-21124: HPL/SQL does not support the CREATE TABLE
LIKE statement (Baoning He, reviewed by Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 17:02:55.205
+ rm -rf ../yetus_PreCommit-HIVE-Build-15667
+ mkdir ../yetus_PreCommit-HIVE-Build-15667
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15667
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15667/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:1840: trailing whitespace.
null sort order:
/data/hiveptest/working/scratch/build.patch:1841: trailing whitespace.
sort order:
/data/hiveptest/working/scratch/build.patch:1862: trailing whitespace.
columns.comments
/data/hiveptest/working/scratch/build.patch:1877: trailing whitespace.
/data/hiveptest/working/scratch/build.patch:1887: trailing whitespace.
columns.comments
warning: squelched 68 whitespace errors
warning: 73 lines add whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc8561900869872189704.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc8561900869872189704.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
[ERROR] Failed to execute goal on project hive-service-rpc: Could not resolve
dependencies for project org.apache.hive:hive-service-rpc:jar:4.0.0-SNAPSHOT:
Failed to collect dependencies for [commons-codec:commons-codec:jar:1.7
(compile), commons-cli:commons-cli:jar:1.2 (compile),
org.apache.thrift:libfb303:jar:0.9.3 (compile),
org.apache.thrift:libthrift:jar:0.9.3 (compile), junit:junit:jar:4.11 (test),
org.apache.hive:hive-classification:jar:4.0.0-SNAPSHOT (compile),
org.slf4j:slf4
[jira] [Commented] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[
https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745291#comment-16745291
]
Hive QA commented on HIVE-21078:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955265/HIVE-21078.09.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15701 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=261)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=261)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15666/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15666/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15666/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955265 - PreCommit-HIVE-Build
> Replicate column and table level statistics for unpartitioned Hive tables
> -
>
> Key: HIVE-21078
> URL: https://issues.apache.org/jira/browse/HIVE-21078
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch,
> HIVE-21078.03.patch, HIVE-21078.04.patch, HIVE-21078.05.patch,
> HIVE-21078.06.patch, HIVE-21078.07.patch, HIVE-21078.08.patch,
> HIVE-21078.09.patch, HIVE-21078.09.patch, HIVE-21078.09.patch,
> HIVE-21078.09.patch, HIVE-21078.sameas.05.patch
>
>
> This task is for replicating column and table level statistics for
> unpartitioned tables. The same for partitioned tables will be worked upon in
> a separate sub-task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[
https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745270#comment-16745270
]
Hive QA commented on HIVE-21078:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
12s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
5s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 21 new + 759 unchanged - 1
fixed = 780 total (was 760) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 3 new + 771
unchanged - 0 fixed = 774 total (was 771) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} metastore-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} standalone-metastore_metastore-server generated 0
new + 49 unchanged - 1 fixed = 49 total (was 50) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} server-extensions in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} hive-unit in the patch passed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15666/dev-s
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.18
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.09.patch > Upgrade to calcite-1.18 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[ https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21078: -- Status: In Progress (was: Patch Available) > Replicate column and table level statistics for unpartitioned Hive tables > - > > Key: HIVE-21078 > URL: https://issues.apache.org/jira/browse/HIVE-21078 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch, > HIVE-21078.03.patch, HIVE-21078.04.patch, HIVE-21078.05.patch, > HIVE-21078.06.patch, HIVE-21078.07.patch, HIVE-21078.08.patch, > HIVE-21078.09.patch, HIVE-21078.09.patch, HIVE-21078.09.patch, > HIVE-21078.sameas.05.patch > > > This task is for replicating column and table level statistics for > unpartitioned tables. The same for partitioned tables will be worked upon in > a separate sub-task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[ https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21078: -- Attachment: HIVE-21078.09.patch Status: Patch Available (was: In Progress) > Replicate column and table level statistics for unpartitioned Hive tables > - > > Key: HIVE-21078 > URL: https://issues.apache.org/jira/browse/HIVE-21078 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch, > HIVE-21078.03.patch, HIVE-21078.04.patch, HIVE-21078.05.patch, > HIVE-21078.06.patch, HIVE-21078.07.patch, HIVE-21078.08.patch, > HIVE-21078.09.patch, HIVE-21078.09.patch, HIVE-21078.09.patch, > HIVE-21078.09.patch, HIVE-21078.sameas.05.patch > > > This task is for replicating column and table level statistics for > unpartitioned tables. The same for partitioned tables will be worked upon in > a separate sub-task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20879) Using null in a projection expression leads to CastException
[
https://issues.apache.org/jira/browse/HIVE-20879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745114#comment-16745114
]
Hive QA commented on HIVE-20879:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955241/HIVE-20879.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15696 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15665/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15665/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15665/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955241 - PreCommit-HIVE-Build
> Using null in a projection expression leads to CastException
>
>
> Key: HIVE-20879
> URL: https://issues.apache.org/jira/browse/HIVE-20879
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Attachments: HIVE-20879.2.patch, HIVE-20879.3.patch,
> HIVE-20879.4.patch, HIVE-20879.5.patch, HIVE-20879.6.patch,
> HIVE-20879.7.patch, HIVE-20879.patch
>
>
> repro:
> {code}
> create table cx1(bool0 boolean);
> select NULL or bool0 from cx1;
> {code}
> workaround(correct type of null):
> {code}
> select cast(NULL as boolean) or bool0 from cx1;
> {code}
> exception:
> {code}
> 2018-11-07T07:28:39,628 ERROR [3533166f-7174-45cd-9d9e-d487038cb6e0 main]
> ql.Driver: FAILED: ClassCastEx
> ception
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
> cannot be ca
> st to
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObject
> Inspector cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspect
> or
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.initialize(GenericUDFOPAnd.java:56)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:1
> 48)
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.ja
> va:260)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprN
> odeDesc(TypeCheckProcFactory.java:1251)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckPr
> ocFactory.java:1660)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:10
> 5)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20879) Using null in a projection expression leads to CastException
[
https://issues.apache.org/jira/browse/HIVE-20879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745057#comment-16745057
]
Hive QA commented on HIVE-20879:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
30s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15665/dev-support/hive-personality.sh
|
| git revision | master / 8e7c3b3 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15665/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Using null in a projection expression leads to CastException
>
>
> Key: HIVE-20879
> URL: https://issues.apache.org/jira/browse/HIVE-20879
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Attachments: HIVE-20879.2.patch, HIVE-20879.3.patch,
> HIVE-20879.4.patch, HIVE-20879.5.patch, HIVE-20879.6.patch,
> HIVE-20879.7.patch, HIVE-20879.patch
>
>
> repro:
> {code}
> create table cx1(bool0 boolean);
> select NULL or bool0 from cx1;
> {code}
> workaround(correct type of null):
> {code}
> select cast(NULL as boolean) or bool0 from cx1;
> {code}
> exception:
> {code}
> 2018-11-07T07:28:39,628 ERROR [3533166f-7174-45cd-9d9e-d487038cb6e0 main]
> ql.Driver: FAILED: ClassCastEx
> ception
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
> cannot be ca
> st to
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObject
> Inspector cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspect
> or
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.initialize(GenericUDFOPAnd.java:56)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:1
> 48)
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.ja
> va:260)
> at
> org.apache.hadoop.hive.ql.p
[jira] [Updated] (HIVE-20879) Using null in a projection expression leads to CastException
[
https://issues.apache.org/jira/browse/HIVE-20879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lavati updated HIVE-20879:
Status: In Progress (was: Patch Available)
> Using null in a projection expression leads to CastException
>
>
> Key: HIVE-20879
> URL: https://issues.apache.org/jira/browse/HIVE-20879
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Attachments: HIVE-20879.2.patch, HIVE-20879.3.patch,
> HIVE-20879.4.patch, HIVE-20879.5.patch, HIVE-20879.6.patch,
> HIVE-20879.7.patch, HIVE-20879.patch
>
>
> repro:
> {code}
> create table cx1(bool0 boolean);
> select NULL or bool0 from cx1;
> {code}
> workaround(correct type of null):
> {code}
> select cast(NULL as boolean) or bool0 from cx1;
> {code}
> exception:
> {code}
> 2018-11-07T07:28:39,628 ERROR [3533166f-7174-45cd-9d9e-d487038cb6e0 main]
> ql.Driver: FAILED: ClassCastEx
> ception
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
> cannot be ca
> st to
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObject
> Inspector cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspect
> or
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.initialize(GenericUDFOPAnd.java:56)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:1
> 48)
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.ja
> va:260)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprN
> odeDesc(TypeCheckProcFactory.java:1251)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckPr
> ocFactory.java:1660)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:10
> 5)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20879) Using null in a projection expression leads to CastException
[
https://issues.apache.org/jira/browse/HIVE-20879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lavati updated HIVE-20879:
Attachment: HIVE-20879.7.patch
Status: Patch Available (was: In Progress)
> Using null in a projection expression leads to CastException
>
>
> Key: HIVE-20879
> URL: https://issues.apache.org/jira/browse/HIVE-20879
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Attachments: HIVE-20879.2.patch, HIVE-20879.3.patch,
> HIVE-20879.4.patch, HIVE-20879.5.patch, HIVE-20879.6.patch,
> HIVE-20879.7.patch, HIVE-20879.patch
>
>
> repro:
> {code}
> create table cx1(bool0 boolean);
> select NULL or bool0 from cx1;
> {code}
> workaround(correct type of null):
> {code}
> select cast(NULL as boolean) or bool0 from cx1;
> {code}
> exception:
> {code}
> 2018-11-07T07:28:39,628 ERROR [3533166f-7174-45cd-9d9e-d487038cb6e0 main]
> ql.Driver: FAILED: ClassCastEx
> ception
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
> cannot be ca
> st to
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObject
> Inspector cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspect
> or
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.initialize(GenericUDFOPAnd.java:56)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:1
> 48)
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.ja
> va:260)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprN
> odeDesc(TypeCheckProcFactory.java:1251)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckPr
> ocFactory.java:1660)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:10
> 5)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20879) Using null in a projection expression leads to CastException
[
https://issues.apache.org/jira/browse/HIVE-20879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744987#comment-16744987
]
Hive QA commented on HIVE-20879:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955235/HIVE-20879.6.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15664/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15664/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15664/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-17 12:20:59.344
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15664/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 12:20:59.346
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 8e7c3b3 HIVE-21124: HPL/SQL does not support the CREATE TABLE
LIKE statement (Baoning He, reviewed by Daniel Dai)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 8e7c3b3 HIVE-21124: HPL/SQL does not support the CREATE TABLE
LIKE statement (Baoning He, reviewed by Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 12:21:00.425
+ rm -rf ../yetus_PreCommit-HIVE-Build-15664
+ mkdir ../yetus_PreCommit-HIVE-Build-15664
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15664
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15664/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc9181394851621024156.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc9181394851621024156.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc9036437233217703452.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
log4j:WARN No appenders could be found for logger (DataNucleus.Persistence).
log4j:WARN Please initialize the log4j system properly.
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer completed with success for 41 classes.
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/ql/target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveLexer.java
does not exist: must build
/data/hiveptest/working/apache-g
[jira] [Updated] (HIVE-20879) Using null in a projection expression leads to CastException
[
https://issues.apache.org/jira/browse/HIVE-20879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lavati updated HIVE-20879:
Status: In Progress (was: Patch Available)
> Using null in a projection expression leads to CastException
>
>
> Key: HIVE-20879
> URL: https://issues.apache.org/jira/browse/HIVE-20879
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Attachments: HIVE-20879.2.patch, HIVE-20879.3.patch,
> HIVE-20879.4.patch, HIVE-20879.5.patch, HIVE-20879.patch
>
>
> repro:
> {code}
> create table cx1(bool0 boolean);
> select NULL or bool0 from cx1;
> {code}
> workaround(correct type of null):
> {code}
> select cast(NULL as boolean) or bool0 from cx1;
> {code}
> exception:
> {code}
> 2018-11-07T07:28:39,628 ERROR [3533166f-7174-45cd-9d9e-d487038cb6e0 main]
> ql.Driver: FAILED: ClassCastEx
> ception
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
> cannot be ca
> st to
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObject
> Inspector cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspect
> or
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.initialize(GenericUDFOPAnd.java:56)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:1
> 48)
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.ja
> va:260)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprN
> odeDesc(TypeCheckProcFactory.java:1251)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckPr
> ocFactory.java:1660)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:10
> 5)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20879) Using null in a projection expression leads to CastException
[
https://issues.apache.org/jira/browse/HIVE-20879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lavati updated HIVE-20879:
Attachment: HIVE-20879.6.patch
Status: Patch Available (was: In Progress)
> Using null in a projection expression leads to CastException
>
>
> Key: HIVE-20879
> URL: https://issues.apache.org/jira/browse/HIVE-20879
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Attachments: HIVE-20879.2.patch, HIVE-20879.3.patch,
> HIVE-20879.4.patch, HIVE-20879.5.patch, HIVE-20879.6.patch, HIVE-20879.patch
>
>
> repro:
> {code}
> create table cx1(bool0 boolean);
> select NULL or bool0 from cx1;
> {code}
> workaround(correct type of null):
> {code}
> select cast(NULL as boolean) or bool0 from cx1;
> {code}
> exception:
> {code}
> 2018-11-07T07:28:39,628 ERROR [3533166f-7174-45cd-9d9e-d487038cb6e0 main]
> ql.Driver: FAILED: ClassCastEx
> ception
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObjectInspector
> cannot be ca
> st to
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableVoidObject
> Inspector cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspect
> or
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.initialize(GenericUDFOPAnd.java:56)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:1
> 48)
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.ja
> va:260)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprN
> odeDesc(TypeCheckProcFactory.java:1251)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckPr
> ocFactory.java:1660)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:10
> 5)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[
https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744962#comment-16744962
]
Hive QA commented on HIVE-21078:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955230/HIVE-21078.09.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15663/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15663/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15663/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-17 12:01:22.157
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15663/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 12:01:22.160
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 8e7c3b3 HIVE-21124: HPL/SQL does not support the CREATE TABLE
LIKE statement (Baoning He, reviewed by Daniel Dai)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 8e7c3b3 HIVE-21124: HPL/SQL does not support the CREATE TABLE
LIKE statement (Baoning He, reviewed by Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 12:01:22.776
+ rm -rf ../yetus_PreCommit-HIVE-Build-15663
+ mkdir ../yetus_PreCommit-HIVE-Build-15663
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15663
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15663/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java:
does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java:
does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java:
does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcrossInstances.java:
does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/WarehouseInstance.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsUpdateTask.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/FSTableEvent.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: does not
exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java: does not
exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/HiveWrapper.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop
[jira] [Updated] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[ https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21078: -- Attachment: HIVE-21078.09.patch Status: Patch Available (was: In Progress) The test TestTxnCommands passes for me locally. The metastore communication error looks to be transient. Also I haven't seen this test fail in previous runs with the same patch. Re-submitting the 05 patch renamed as 09. > Replicate column and table level statistics for unpartitioned Hive tables > - > > Key: HIVE-21078 > URL: https://issues.apache.org/jira/browse/HIVE-21078 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch, > HIVE-21078.03.patch, HIVE-21078.04.patch, HIVE-21078.05.patch, > HIVE-21078.06.patch, HIVE-21078.07.patch, HIVE-21078.08.patch, > HIVE-21078.09.patch, HIVE-21078.09.patch, HIVE-21078.09.patch, > HIVE-21078.sameas.05.patch > > > This task is for replicating column and table level statistics for > unpartitioned tables. The same for partitioned tables will be worked upon in > a separate sub-task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[ https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21078: -- Status: In Progress (was: Patch Available) > Replicate column and table level statistics for unpartitioned Hive tables > - > > Key: HIVE-21078 > URL: https://issues.apache.org/jira/browse/HIVE-21078 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch, > HIVE-21078.03.patch, HIVE-21078.04.patch, HIVE-21078.05.patch, > HIVE-21078.06.patch, HIVE-21078.07.patch, HIVE-21078.08.patch, > HIVE-21078.09.patch, HIVE-21078.09.patch, HIVE-21078.sameas.05.patch > > > This task is for replicating column and table level statistics for > unpartitioned tables. The same for partitioned tables will be worked upon in > a separate sub-task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20879) Using null in a projection expression leads to CastException
[
https://issues.apache.org/jira/browse/HIVE-20879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744942#comment-16744942
]
Hive QA commented on HIVE-20879:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955214/HIVE-20879.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15662/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15662/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15662/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-17 11:41:15.495
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15662/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 11:41:15.498
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 8e7c3b3 HIVE-21124: HPL/SQL does not support the CREATE TABLE
LIKE statement (Baoning He, reviewed by Daniel Dai)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 8e7c3b3 HIVE-21124: HPL/SQL does not support the CREATE TABLE
LIKE statement (Baoning He, reviewed by Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 11:41:16.825
+ rm -rf ../yetus_PreCommit-HIVE-Build-15662
+ mkdir ../yetus_PreCommit-HIVE-Build-15662
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15662
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15662/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] Failed to execute goal on project hive-shims-common: Could not resolve
dependencies for project
org.apache.hive.shims:hive-shims-common:jar:4.0.0-SNAPSHOT: The following
artifacts could not be resolved: commons-codec:commons-codec:jar:1.7,
commons-logging:commons-logging:jar:1.2,
org.apache.httpcomponents:httpcore:jar:4.4.4: Could not find artifact
commons-codec:commons-codec:jar:1.7 in datanucleus
(http://www.datanucleus.org/downloads/maven2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-shims-common
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-15662
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955214 - PreCommit-HIVE-Build
> Using null in a projection expression leads to CastException
>
>
> Key: HIVE-20879
> URL: https://issues.apache.org/jira/browse/HIVE-20879
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Attachments: HIVE-20879.2.patch, HIVE-20879.3.patch,
> HIVE-20879.4.patch, HIVE-20
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.18
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744940#comment-16744940
]
Hive QA commented on HIVE-21001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955208/HIVE-21001.09.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15661/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15661/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15661/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-17 11:39:37.268
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15661/source-prep.txt
+ [[ true == \t\r\u\e ]]
+ rm -rf ivy maven
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 11:39:37.764
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 8e7c3b3 HIVE-21124: HPL/SQL does not support the CREATE TABLE
LIKE statement (Baoning He, reviewed by Daniel Dai)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 8e7c3b3 HIVE-21124: HPL/SQL does not support the CREATE TABLE
LIKE statement (Baoning He, reviewed by Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-17 11:39:38.927
+ rm -rf ../yetus_PreCommit-HIVE-Build-15661
+ mkdir ../yetus_PreCommit-HIVE-Build-15661
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15661
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15661/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:1840: trailing whitespace.
null sort order:
/data/hiveptest/working/scratch/build.patch:1841: trailing whitespace.
sort order:
/data/hiveptest/working/scratch/build.patch:1862: trailing whitespace.
columns.comments
/data/hiveptest/working/scratch/build.patch:1877: trailing whitespace.
/data/hiveptest/working/scratch/build.patch:1887: trailing whitespace.
columns.comments
warning: squelched 68 whitespace errors
warning: 73 lines add whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc1368803581363050274.exe, --version]
protoc-jar: executing: [/tmp/protoc1368803581363050274.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
libprotoc 2.5.0
[ERROR] Failed to execute goal on project hive-service-rpc: Could not resolve
dependencies for project org.apache.hive:hive-service-rpc:jar:4.0.0-SNAPSHOT:
Failed to collect dependencies for [commons-codec:commons-codec:jar:1.7
(compile), commons-cli:commons-cli:jar:1.2 (compile),
org.apache.thrift:libfb303:jar:0.9.3 (compile),
org.apache.thrift:libthrift:jar:0.9.3 (compile), junit:junit:jar:4.11 (test),
org.apache.hive:hive-classification:jar:4.0.0-SNAPSHOT (compile),
org.slf4j:slf4
[jira] [Resolved] (HIVE-19295) Some multiple inserts do not work on MR engine
[
https://issues.apache.org/jira/browse/HIVE-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin resolved HIVE-19295.
-
Resolution: Won't Fix
> Some multiple inserts do not work on MR engine
> --
>
> Key: HIVE-19295
> URL: https://issues.apache.org/jira/browse/HIVE-19295
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT);
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Do multiple insert*
> {code}
> FROM
> `default`.`customer_target` `trg`
> RIGHT OUTER JOIN
> `default`.`customer_source` `src`
> ON `src`.`id` = `trg`.`id`
> INSERT INTO `default`.`customer_target`-- update clause
> select `trg`.`id`, `src`.`first_name`, `src`.`last_name`, `trg`.`age`
>WHERE `src`.`id` = `trg`.`id`
> sort by `trg`.id
> INSERT INTO `default`.`customer_target`-- insert clause
> select `src`.`id`, `src`.`first_name`, `src`.`last_name`, `src`.`age`
>WHERE `trg`.`id` IS NULL
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-25T05:31:50,407 ERROR [e502bb46-e832-4156-860f-aba875bbb10b main]
> ql.Driver: FAILED: SemanticException [Error 10087]: The same output cannot be
> present multiple times: customer_target
> org.apache.hadoop.hive.ql.parse.SemanticException: The same output cannot be
> present multiple times: customer_target
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6899)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9772)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9644)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10549)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10427)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:11125)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:481)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11138)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.
[jira] [Commented] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[
https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744939#comment-16744939
]
Hive QA commented on HIVE-21078:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955206/HIVE-21078.09.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15701 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnCommands.testSetClauseFakeColumn (batchId=327)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15660/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15660/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15660/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955206 - PreCommit-HIVE-Build
> Replicate column and table level statistics for unpartitioned Hive tables
> -
>
> Key: HIVE-21078
> URL: https://issues.apache.org/jira/browse/HIVE-21078
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch,
> HIVE-21078.03.patch, HIVE-21078.04.patch, HIVE-21078.05.patch,
> HIVE-21078.06.patch, HIVE-21078.07.patch, HIVE-21078.08.patch,
> HIVE-21078.09.patch, HIVE-21078.09.patch, HIVE-21078.sameas.05.patch
>
>
> This task is for replicating column and table level statistics for
> unpartitioned tables. The same for partitioned tables will be worked upon in
> a separate sub-task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin resolved HIVE-19286.
-
Resolution: Won't Fix
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Transactions
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities:
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger: method=deserializePlan start=1524568304448 end=1524568304538 duration=90
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
> Starting to launch local task to process map join; maximum memory =
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for
> trg created
> 2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
> child 37 HASHTABLESINK
> 2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
> operator HASHTABLESINK[37]
> 2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
> JVM Max Heap Size: 477626368
> 2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
> Map local work failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFrom
[jira] [Commented] (HIVE-21016) Duplicate column name in GROUP BY statement causing Vertex failures
[
https://issues.apache.org/jira/browse/HIVE-21016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744925#comment-16744925
]
Mani M commented on HIVE-21016:
---
HI [~pvary]
As per my initial analysis, I think we need to check for the duplicates in the
below source where the group by clause is generated
https://github.com/apache/hive/blob/8e7c3b340f36a3b76453338b04b8cda360eeaa70/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L4937
{code:java}
List grpByExprs = getGroupByForClause(parseInfo, dest);
//Need to check for the duplicates from the above derived list.
for (int i = 0; i < grpByExprs.size(); ++i) {
ASTNode grpbyExpr = grpByExprs.get(i);
ColumnInfo exprInfo = groupByInputRowResolver.getExpression(grpbyExpr);
if (exprInfo == null) {
throw new SemanticException(ErrorMsg.INVALID_COLUMN.getMsg(grpbyExpr));
}
{code}
Correct me if my understanding is wrong
> Duplicate column name in GROUP BY statement causing Vertex failures
> ---
>
> Key: HIVE-21016
> URL: https://issues.apache.org/jira/browse/HIVE-21016
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: Bjorn Olsen
>Assignee: Mani M
>Priority: Major
>
> Hive queries fail with "Vertex failure" messages when the user submits a
> query containing duplicate GROUP BY columns. The Hive query parser should
> detect and reject this scenario with a meaningful error message, rather than
> executing the query and failing with an obfuscated message. For complex
> queries this can result in a lot of debugging effort, whereas a simple error
> message could have saved some time.
> To repeat the issue, choose any table and perform a GROUP BY with a duplicate
> column name.
> {{For example:}}
> select count( * ), party_id from party {{group by party_id, party_id;}}
> Note the duplicate column in the GROUP BY.
> This will fail with messages similar to below:
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing vector batch (tag=0) ffb9-5fb1-3024-922a-10cc313a7c171
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:390)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:232)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing vector batch (tag=0)
> ffb9-5fb1-3024-922a-10cc313a7c171
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:454)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:381)
> ... 17 more
> *Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19295) Some multiple inserts do not work on MR engine
[
https://issues.apache.org/jira/browse/HIVE-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744924#comment-16744924
]
Oleksiy Sayankin commented on HIVE-19295:
-
Found this comment on related issue HIVE-16235 (see
[link|https://issues.apache.org/jira/browse/HIVE-16235?focusedCommentId=15954330&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15954330]).
{quote}
Won't fix because:
1. MR is already deprecated and will go away in next Hive release
2. workaround is available
a) turn off MR and use Tez, or
b) turn off MapJoin
{quote}
So closing because of the same reasons.
> Some multiple inserts do not work on MR engine
> --
>
> Key: HIVE-19295
> URL: https://issues.apache.org/jira/browse/HIVE-19295
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT);
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Do multiple insert*
> {code}
> FROM
> `default`.`customer_target` `trg`
> RIGHT OUTER JOIN
> `default`.`customer_source` `src`
> ON `src`.`id` = `trg`.`id`
> INSERT INTO `default`.`customer_target`-- update clause
> select `trg`.`id`, `src`.`first_name`, `src`.`last_name`, `trg`.`age`
>WHERE `src`.`id` = `trg`.`id`
> sort by `trg`.id
> INSERT INTO `default`.`customer_target`-- insert clause
> select `src`.`id`, `src`.`first_name`, `src`.`last_name`, `src`.`age`
>WHERE `trg`.`id` IS NULL
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-25T05:31:50,407 ERROR [e502bb46-e832-4156-860f-aba875bbb10b main]
> ql.Driver: FAILED: SemanticException [Error 10087]: The same output cannot be
> present multiple times: customer_target
> org.apache.hadoop.hive.ql.parse.SemanticException: The same output cannot be
> present multiple times: customer_target
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6899)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9772)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9644)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10549)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10427)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:11125)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:481)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11138)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hi
[jira] [Comment Edited] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744922#comment-16744922
]
Oleksiy Sayankin edited comment on HIVE-19286 at 1/17/19 11:29 AM:
---
Found this comment on related issue HIVE-16235 (see
[link|https://issues.apache.org/jira/browse/HIVE-16235?focusedCommentId=15954330&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15954330]).
{quote}
Won't fix because:
1. MR is already deprecated and will go away in next Hive release
2. workaround is available
a) turn off MR and use Tez, or
b) turn off MapJoin
{quote}
So closing because of the same reasons.
was (Author: osayankin):
Found this comment on related issue (see
[link|https://issues.apache.org/jira/browse/HIVE-16235?focusedCommentId=15954330&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15954330]).
{quote}
Won't fix because:
1. MR is already deprecated and will go away in next Hive release
2. workaround is available
a) turn off MR and use Tez, or
b) turn off MapJoin
{quote}
So closing because of the same reasons.
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Transactions
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities:
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger: method=deserializePlan start=1524568304448 end=1524568304538 duration=90
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
> Starting to launch local task to process map join; maximum memory =
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for
> trg created
> 2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializ
[jira] [Commented] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744922#comment-16744922
]
Oleksiy Sayankin commented on HIVE-19286:
-
Found this comment on related issue (see
[link|https://issues.apache.org/jira/browse/HIVE-16235?focusedCommentId=15954330&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15954330]).
{quote}
Won't fix because:
1. MR is already deprecated and will go away in next Hive release
2. workaround is available
a) turn off MR and use Tez, or
b) turn off MapJoin
{quote}
So closing because of the same reasons.
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Transactions
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities:
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger: method=deserializePlan start=1524568304448 end=1524568304538 duration=90
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
> Starting to launch local task to process map join; maximum memory =
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for
> trg created
> 2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
> child 37 HASHTABLESINK
> 2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
> operator HASHTABLESINK[37]
> 2018-0
[jira] [Commented] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[
https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744903#comment-16744903
]
Hive QA commented on HIVE-21078:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
14s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 21 new + 759 unchanged - 1
fixed = 780 total (was 760) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 3 new + 771
unchanged - 0 fixed = 774 total (was 771) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} metastore-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} standalone-metastore_metastore-server generated 0
new + 49 unchanged - 1 fixed = 49 total (was 50) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} server-extensions in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} hive-unit in the patch passed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15660/dev-s
