[jira] [Commented] (HIVE-21230) LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side (HiveJoinAddNotNullRule bails out for outer joins)
[
https://issues.apache.org/jira/browse/HIVE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805635#comment-16805635
]
Hive QA commented on HIVE-21230:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
19s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} ql: The patch generated 0 new + 1 unchanged - 9
fixed = 1 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16770/dev-support/hive-personality.sh
|
| git revision | master / 2ce775b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16770/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side
> (HiveJoinAddNotNullRule bails out for outer joins)
>
>
> Key: HIVE-21230
> URL: https://issues.apache.org/jira/browse/HIVE-21230
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21230.1.patch, HIVE-21230.2.patch,
> HIVE-21230.3.patch, HIVE-21230.4.patch, HIVE-21230.5.patch, HIVE-21230.6.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> LEFT JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 = t1.col0 AND t0.col1 = t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null in the right input and
> introduce the corresponding filter predicate. Currently, the rule just bails
> out if it is not an inner join.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java#L79
--
This message was
[jira] [Updated] (HIVE-21548) Statistics: String statistics computation is slower with vectorization enabled
[ https://issues.apache.org/jira/browse/HIVE-21548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-21548: --- Attachment: prof-hll-string-stats.svg > Statistics: String statistics computation is slower with vectorization enabled > -- > > Key: HIVE-21548 > URL: https://issues.apache.org/jira/browse/HIVE-21548 > Project: Hive > Issue Type: Improvement > Components: Statistics, Vectorization >Affects Versions: 4.0.0 >Reporter: Gopal V >Priority: Major > Attachments: prof-hll-string-stats.svg, string-stats-hll.png > > > Computing the hashCode of a String for HLL is going > byte[] -> Text -> String -> byte[] > to get the Murmur3 hashCode of the byte[] region. > !string-stats-hll.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21548) Statistics: String statistics computation is slower with vectorization enabled
[ https://issues.apache.org/jira/browse/HIVE-21548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-21548: --- Affects Version/s: 4.0.0 > Statistics: String statistics computation is slower with vectorization enabled > -- > > Key: HIVE-21548 > URL: https://issues.apache.org/jira/browse/HIVE-21548 > Project: Hive > Issue Type: Improvement > Components: Statistics, Vectorization >Affects Versions: 4.0.0 >Reporter: Gopal V >Priority: Major > Attachments: string-stats-hll.png > > > Computing the hashCode of a String for HLL is going > byte[] -> Text -> String -> byte[] > to get the Murmur3 hashCode of the byte[] region. > !string-stats-hll.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21537) Scalar query rewrite could be improved to not generate an extra join if subquery is guaranteed to produce atmost one row
[
https://issues.apache.org/jira/browse/HIVE-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805623#comment-16805623
]
Hive QA commented on HIVE-21537:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964250/HIVE-21537.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15853 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=166)
[list_bucket_dml_10.q,acid_globallimit.q,vector_acid3.q,delete_where_no_match.q,vector_reduce1.q,stats_only_null.q,vectorization_6.q,semijoin_hint.q,vector_groupby_grouping_sets4.q,materialized_view_create_rewrite_rebuild_dummy.q,tez_vector_dynpart_hashjoin_2.q,delete_all_non_partitioned.q,delete_all_partitioned.q,compare_double_bigint_2.q,vector_order_null.q,cbo_rp_windowing_2.q,cbo_rp_unionDistinct_2.q,vector_reduce3.q,vector_varchar_4.q,smb_cache.q,hybridgrace_hashjoin_1.q,union_assertion_type.q,ppd_union_view.q,vector_decimal_4.q,schema_evol_orc_acid_part.q,cross_join.q,parquet_predicate_pushdown.q,llap_vector_nohybridgrace.q,tez_smb_main.q,quotedid_smb.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16769/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16769/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16769/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12964250 - PreCommit-HIVE-Build
> Scalar query rewrite could be improved to not generate an extra join if
> subquery is guaranteed to produce atmost one row
>
>
> Key: HIVE-21537
> URL: https://issues.apache.org/jira/browse/HIVE-21537
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: sub-query
> Attachments: HIVE-21537.1.patch, HIVE-21537.2.patch,
> HIVE-21537.3.patch
>
>
> Currently Hive planner introduces this branch and later executes a rule to
> remove this branch if it could.
> Subquery remove rule itself could check if subquery will produce max one row
> (using relmetadat's getMaxRowCount) and avoid introducing this branch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21516) Fix spark downloading for q tests
[ https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-21516: Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Miklos! > Fix spark downloading for q tests > - > > Key: HIVE-21516 > URL: https://issues.apache.org/jira/browse/HIVE-21516 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21516.01.patch, HIVE-21516.02.patch, > HIVE-21516.03.patch, HIVE-21516.04.patch, HIVE-21516.05.patch, > HIVE-21516.06.patch > > > Currently itests/pom.xml declares a command to generated the download script > for spark, thus it is re-generated every time any maven command is executed > for any sub project of itests. AS a side effect it is leaving download.sh > files everywhere. The download.sh file is almost totally static, no need to > recreate it every time, just requires $spark.version as a parameter. > Also it is only working properly under linux, as it relies on the md5sum > program which is not present in OS X. This means that if the spark tarball is > partially downloaded on OS X, then it would never be re-downloaded. This > should be fixed by making it work as well using md5 on OS X. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21517) Fix AggregateStatsCache
[ https://issues.apache.org/jira/browse/HIVE-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-21517: Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Miklos! > Fix AggregateStatsCache > --- > > Key: HIVE-21517 > URL: https://issues.apache.org/jira/browse/HIVE-21517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21517.01.patch > > > Due to a bug AggregateStatsCache is not returning the best matching result. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21537) Scalar query rewrite could be improved to not generate an extra join if subquery is guaranteed to produce atmost one row
[
https://issues.apache.org/jira/browse/HIVE-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805611#comment-16805611
]
Hive QA commented on HIVE-21537:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 1 new + 180 unchanged - 1
fixed = 181 total (was 181) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
33s{color} | {color:red} ql generated 2 new + 2254 unchanged - 2 fixed = 2256
total (was 2256) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Switch statement found in
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSubQueryRemoveRule.apply(RelMetadataQuery,
RexSubQuery, Set, RelOptUtil$Logic, HiveSubQRemoveRelBuilder, int, int,
boolean) where one case falls through to the next case At
HiveSubQueryRemoveRule.java:RexSubQuery, Set, RelOptUtil$Logic,
HiveSubQRemoveRelBuilder, int, int, boolean) where one case falls through to
the next case At HiveSubQueryRemoveRule.java:[lines 426-429] |
| | Switch statement found in
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveSubQueryRemoveRule.apply(RelMetadataQuery,
RexSubQuery, Set, RelOptUtil$Logic, HiveSubQRemoveRelBuilder, int, int,
boolean) where default case is missing At
HiveSubQueryRemoveRule.java:RexSubQuery, Set, RelOptUtil$Logic,
HiveSubQRemoveRelBuilder, int, int, boolean) where default case is missing At
HiveSubQueryRemoveRule.java:[lines 286-317] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16769/dev-support/hive-personality.sh
|
| git revision | master / 146611b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16769/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16769/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16769/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Scalar query rewrite could be improved to not generate an extra join if
> subquery is guaranteed to produce atmost one row
>
>
> Key: HIVE-21537
> URL: https://issues.apache.org/jira/browse/HIVE-21537
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.19
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805606#comment-16805606
]
Hive QA commented on HIVE-21001:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
23s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 7 new + 342 unchanged - 45
fixed = 349 total (was 387) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
11s{color} | {color:red} root: The patch generated 7 new + 351 unchanged - 45
fixed = 358 total (was 396) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
24s{color} | {color:red} ql generated 1 new + 2256 unchanged - 0 fixed = 2257
total (was 2256) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 15m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Switch statement found in
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTBuilder.literal(RexLiteral)
where default case is missing At ASTBuilder.java:where default case is
missing At ASTBuilder.java:[lines 279-290] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16767/dev-support/hive-personality.sh
|
| git revision | master / 146611b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16767/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16767/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16767/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16767/yetus/new-findbugs-ql.html
|
| modules | C: ql accumulo-handler hbase-handler . U: . |
| Console output |
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.19
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805605#comment-16805605
]
Hive QA commented on HIVE-21001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964216/HIVE-21001.49.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 15684 tests
executed
*Failed tests:*
{noformat}
TestFMSketchSerialization - did not produce a TEST-*.xml file (likely timed
out) (batchId=231)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestHyperLogLog - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestRetriesInRetryingHMSHandler - did not produce a TEST-*.xml file (likely
timed out) (batchId=231)
TestSparseEncodeHash - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestStatsSetupConst - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16767/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16767/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16767/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12964216 - PreCommit-HIVE-Build
> Upgrade to calcite-1.19
> ---
>
> Key: HIVE-21001
> URL: https://issues.apache.org/jira/browse/HIVE-21001
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch,
> HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch,
> HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch,
> HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch,
> HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch,
> HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch,
> HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch,
> HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch,
> HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch,
> HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch,
> HIVE-21001.22.patch, HIVE-21001.23.patch, HIVE-21001.24.patch,
> HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.26.patch,
> HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.27.patch,
> HIVE-21001.28.patch, HIVE-21001.29.patch, HIVE-21001.29.patch,
> HIVE-21001.30.patch, HIVE-21001.31.patch, HIVE-21001.32.patch,
> HIVE-21001.34.patch, HIVE-21001.35.patch, HIVE-21001.36.patch,
> HIVE-21001.37.patch, HIVE-21001.38.patch, HIVE-21001.39.patch,
> HIVE-21001.40.patch, HIVE-21001.41.patch, HIVE-21001.42.patch,
> HIVE-21001.43.patch, HIVE-21001.44.patch, HIVE-21001.45.patch,
> HIVE-21001.45.patch, HIVE-21001.46.patch, HIVE-21001.47.patch,
> HIVE-21001.48.patch, HIVE-21001.48.patch, HIVE-21001.48.patch,
> HIVE-21001.48.patch, HIVE-21001.49.patch, HIVE-21001.49.patch
>
>
> XLEAR LIBRARY CACHE
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805596#comment-16805596
]
Hive QA commented on HIVE-21109:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964213/HIVE-21109.08.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16766/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16766/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16766/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-30 02:32:30.819
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16766/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-30 02:32:30.821
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
23ab7f2..146611b master -> origin/master
a91e6ec..4015a5a branch-3 -> origin/branch-3
71e5f5a..cf9bf8f branch-3.1 -> origin/branch-3.1
+ git reset --hard HEAD
HEAD is now at 23ab7f2 HIVE-21484 : Metastore API getVersion() should return
real version (Vihang Karajgaonkar reviewed by Naveen Gangam, Peter Vary)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 146611b HIVE-21544: Constant propagation corrupts
coalesce/case/when expressions during folding (Zoltan Haindrich, reviewed by
Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-30 02:32:32.128
+ rm -rf ../yetus_PreCommit-HIVE-Build-16766
+ mkdir ../yetus_PreCommit-HIVE-Build-16766
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16766
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16766/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestStatsReplicationScenarios.java:
does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestStatsReplicationScenariosNoAutogather.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/CreateTableDesc.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/CreateTableOperation.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsUpdateTask.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/FSTableEvent.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java: does not
exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java: does
not exist in
[jira] [Commented] (HIVE-21499) should not remove the function from registry if create command failed with AlreadyExistsException
[
https://issues.apache.org/jira/browse/HIVE-21499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805592#comment-16805592
]
Hive QA commented on HIVE-21499:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964208/HIVE-21499.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15882 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16765/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16765/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16765/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12964208 - PreCommit-HIVE-Build
> should not remove the function from registry if create command failed with
> AlreadyExistsException
> -
>
> Key: HIVE-21499
> URL: https://issues.apache.org/jira/browse/HIVE-21499
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
> Environment: Hive-3.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21499.01.patch, HIVE-21499.02.patch,
> HIVE-21499.patch
>
>
> As a part of HIVE-20953 we are removing the function if creation for same
> failed with any reason, this will yield into the following situation.
> 1. create function failed since function already exists
> 2. on #1 failure hive will clear the permanent function from the registry
> 3. this function will be of no use until hiveserver2 restarted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21499) should not remove the function from registry if create command failed with AlreadyExistsException
[
https://issues.apache.org/jira/browse/HIVE-21499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805589#comment-16805589
]
Hive QA commented on HIVE-21499:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
19s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 47
unchanged - 0 fixed = 48 total (was 47) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16765/dev-support/hive-personality.sh
|
| git revision | master / 6f18bbb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16765/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16765/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> should not remove the function from registry if create command failed with
> AlreadyExistsException
> -
>
> Key: HIVE-21499
> URL: https://issues.apache.org/jira/browse/HIVE-21499
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
> Environment: Hive-3.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21499.01.patch, HIVE-21499.02.patch,
> HIVE-21499.patch
>
>
> As a part of HIVE-20953 we are removing the function if creation for same
> failed with any reason, this will yield into the following situation.
> 1. create function failed since function already exists
> 2. on #1 failure hive will clear the permanent function from the registry
> 3. this function will be of no use until hiveserver2 restarted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21544) Constant propagation corrupts coalesce/case/when expressions during folding
[
https://issues.apache.org/jira/browse/HIVE-21544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21544:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~kgyrtkirk]
> Constant propagation corrupts coalesce/case/when expressions during folding
> ---
>
> Key: HIVE-21544
> URL: https://issues.apache.org/jira/browse/HIVE-21544
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21544.01.patch
>
>
> {code}
> set hive.fetch.task.conversion=none;
> set hive.optimize.ppd=false;
> create table t (s1 string,s2 string);
> insert into t values (null,null);
> explain
> select
> coalesce(s1, 'null_value' ), coalesce(s2, 'null_value' ),
> coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' ),
> case when coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' )
> then 'eq' else 'noteq' end
> from t;
> select
> coalesce(s1, 'null_value' ), coalesce(s2, 'null_value' ),
> coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' ),
> case when coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' )
> then 'eq' else 'noteq' end
> from t;
> {code}
> incorrect result is:
> {code}
> null_value null_value NULLnoteq
> {code}
> expected result:
> {code}
> null_value null_value trueeq
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21342) Analyze compute stats for column leave behind staging dir on hdfs
[ https://issues.apache.org/jira/browse/HIVE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21342: --- Fix Version/s: 3.1.2 3.2.0 > Analyze compute stats for column leave behind staging dir on hdfs > - > > Key: HIVE-21342 > URL: https://issues.apache.org/jira/browse/HIVE-21342 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 > Environment: hive-3.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Fix For: 4.0.0, 3.2.0, 3.1.2 > > Attachments: HIVE-21342.02.patch, HIVE-21342.patch > > > staging dir cleanup does not happen for the "analyze table .. compute > statistics for columns", this leads to stale directory on hdfs. > the problem seems to be with ColumnStatsSemanticAnalyzer which don't have > hdfscleanup set for the context. > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java#L310 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21342) Analyze compute stats for column leave behind staging dir on hdfs
[ https://issues.apache.org/jira/browse/HIVE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21342: --- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master, thanks [~Rajkumar Singh]! > Analyze compute stats for column leave behind staging dir on hdfs > - > > Key: HIVE-21342 > URL: https://issues.apache.org/jira/browse/HIVE-21342 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 > Environment: hive-3.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21342.02.patch, HIVE-21342.patch > > > staging dir cleanup does not happen for the "analyze table .. compute > statistics for columns", this leads to stale directory on hdfs. > the problem seems to be with ColumnStatsSemanticAnalyzer which don't have > hdfscleanup set for the context. > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java#L310 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21342) Analyze compute stats for column leave behind staging dir on hdfs
[ https://issues.apache.org/jira/browse/HIVE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805576#comment-16805576 ] Jesus Camacho Rodriguez commented on HIVE-21342: +1 > Analyze compute stats for column leave behind staging dir on hdfs > - > > Key: HIVE-21342 > URL: https://issues.apache.org/jira/browse/HIVE-21342 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 > Environment: hive-3.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21342.02.patch, HIVE-21342.patch > > > staging dir cleanup does not happen for the "analyze table .. compute > statistics for columns", this leads to stale directory on hdfs. > the problem seems to be with ColumnStatsSemanticAnalyzer which don't have > hdfscleanup set for the context. > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java#L310 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21342) Analyze compute stats for column leave behind staging dir on hdfs
[
https://issues.apache.org/jira/browse/HIVE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805575#comment-16805575
]
Hive QA commented on HIVE-21342:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964210/HIVE-21342.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15882 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16764/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16764/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16764/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12964210 - PreCommit-HIVE-Build
> Analyze compute stats for column leave behind staging dir on hdfs
> -
>
> Key: HIVE-21342
> URL: https://issues.apache.org/jira/browse/HIVE-21342
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
> Environment: hive-3.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21342.02.patch, HIVE-21342.patch
>
>
> staging dir cleanup does not happen for the "analyze table .. compute
> statistics for columns", this leads to stale directory on hdfs.
> the problem seems to be with ColumnStatsSemanticAnalyzer which don't have
> hdfscleanup set for the context.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java#L310
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21342) Analyze compute stats for column leave behind staging dir on hdfs
[
https://issues.apache.org/jira/browse/HIVE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805566#comment-16805566
]
Hive QA commented on HIVE-21342:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
33s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16764/dev-support/hive-personality.sh
|
| git revision | master / 23ab7f2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16764/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Analyze compute stats for column leave behind staging dir on hdfs
> -
>
> Key: HIVE-21342
> URL: https://issues.apache.org/jira/browse/HIVE-21342
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
> Environment: hive-3.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-21342.02.patch, HIVE-21342.patch
>
>
> staging dir cleanup does not happen for the "analyze table .. compute
> statistics for columns", this leads to stale directory on hdfs.
> the problem seems to be with ColumnStatsSemanticAnalyzer which don't have
> hdfscleanup set for the context.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java#L310
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21544) Constant propagation corrupts coalesce/case/when expressions during folding
[
https://issues.apache.org/jira/browse/HIVE-21544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805558#comment-16805558
]
Hive QA commented on HIVE-21544:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964211/HIVE-21544.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15883 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16763/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16763/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16763/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12964211 - PreCommit-HIVE-Build
> Constant propagation corrupts coalesce/case/when expressions during folding
> ---
>
> Key: HIVE-21544
> URL: https://issues.apache.org/jira/browse/HIVE-21544
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21544.01.patch
>
>
> {code}
> set hive.fetch.task.conversion=none;
> set hive.optimize.ppd=false;
> create table t (s1 string,s2 string);
> insert into t values (null,null);
> explain
> select
> coalesce(s1, 'null_value' ), coalesce(s2, 'null_value' ),
> coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' ),
> case when coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' )
> then 'eq' else 'noteq' end
> from t;
> select
> coalesce(s1, 'null_value' ), coalesce(s2, 'null_value' ),
> coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' ),
> case when coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' )
> then 'eq' else 'noteq' end
> from t;
> {code}
> incorrect result is:
> {code}
> null_value null_value NULLnoteq
> {code}
> expected result:
> {code}
> null_value null_value trueeq
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21544) Constant propagation corrupts coalesce/case/when expressions during folding
[
https://issues.apache.org/jira/browse/HIVE-21544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805544#comment-16805544
]
Hive QA commented on HIVE-21544:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
31s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} ql: The patch generated 0 new + 83 unchanged - 2
fixed = 83 total (was 85) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 6 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16763/dev-support/hive-personality.sh
|
| git revision | master / 23ab7f2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16763/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16763/yetus/whitespace-tabs.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16763/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Constant propagation corrupts coalesce/case/when expressions during folding
> ---
>
> Key: HIVE-21544
> URL: https://issues.apache.org/jira/browse/HIVE-21544
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21544.01.patch
>
>
> {code}
> set hive.fetch.task.conversion=none;
> set hive.optimize.ppd=false;
> create table t (s1 string,s2 string);
> insert into t values (null,null);
> explain
> select
> coalesce(s1, 'null_value' ), coalesce(s2, 'null_value' ),
> coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' ),
> case when coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' )
> then 'eq' else 'noteq' end
> from t;
> select
> coalesce(s1, 'null_value'
[jira] [Commented] (HIVE-21509) LLAP may cache corrupted column vectors and return wrong query result
[
https://issues.apache.org/jira/browse/HIVE-21509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805527#comment-16805527
]
Hive QA commented on HIVE-21509:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964206/HIVE-21509.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15883 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16762/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16762/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16762/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12964206 - PreCommit-HIVE-Build
> LLAP may cache corrupted column vectors and return wrong query result
> -
>
> Key: HIVE-21509
> URL: https://issues.apache.org/jira/browse/HIVE-21509
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21509.0.wip.patch, HIVE-21509.1.wip.patch,
> HIVE-21509.2.patch
>
>
> In some scenarios, LLAP might store column vectors in cache that are getting
> reused and reset just before their original content would be written.
> The issue is a concurrency issue and is thereby flaky. It is not easy to
> reproduce, but the odds of surfacing this issue can by improved by setting
> LLAP executor and IO thread counts this way:
> * set hive.llap.daemon.num.executors=32;
> * set hive.llap.io.threadpool.size=1;
> * using TPCDS input data of store_sales table, have at least a couple of
> 100k's of rows, and use text format:
> {code:java}
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|')
> STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'{code}
> * having more splits increases the issue showing itself, so it is worth to
> _set tez.grouping.min-size=1024; set tez.grouping.max-size=1024;_
> * run query on this this table: select min(ss_sold_date_sk) from store_sales;
> The first query result is correct (2450816 in my case). Repeating the query
> will trigger reading from LLAP cache and produce a wrong result: 0.
> If one wants to make sure of running into this issue, place a
> Thread.sleep(250) at the beginning of VectorDeserializeOrcWriter#run().
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21541) Fix missing asf headers from HIVE-15406
[ https://issues.apache.org/jira/browse/HIVE-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805513#comment-16805513 ] Laszlo Bodor edited comment on HIVE-21541 at 3/29/19 10:31 PM: --- thanks [~pvary], [~kgyrtkirk] ! I double-checked, and I found the root cause of duplications...I copied it from GenericUDFTrunc :) [https://github.com/apache/hive/blob/a7a9f516b3803830999f60d2bffb45e76c8a91ef/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrunc.java#L2-L3] so, seems like it's a pattern, which could have spread in the codebase, we should check that... was (Author: abstractdog): thanks [~pvary], [~kgyrtkirk] ! I double-checked, and I found the root cause of duplications...I copied it from GenericUDFTrunc :) [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrunc.java#L2-L3] so, seems like it's a pattern, which could have spread in the codebase, we should check that... > Fix missing asf headers from HIVE-15406 > --- > > Key: HIVE-21541 > URL: https://issues.apache.org/jira/browse/HIVE-21541 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21541.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21541) Fix missing asf headers from HIVE-15406
[ https://issues.apache.org/jira/browse/HIVE-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805513#comment-16805513 ] Laszlo Bodor commented on HIVE-21541: - thanks [~pvary], [~kgyrtkirk] ! I double-checked, and I found the root cause of duplications...I copied it from GenericUDFTrunc :) [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrunc.java#L2-L3] so, seems like it's a pattern, which could have spread in the codebase, we should check that... > Fix missing asf headers from HIVE-15406 > --- > > Key: HIVE-21541 > URL: https://issues.apache.org/jira/browse/HIVE-21541 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21541.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21283) Create Synonym mid for substr, position for locate
[ https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M reassigned HIVE-21283: - Assignee: Mani M > Create Synonym mid for substr, position for locate > > > Key: HIVE-21283 > URL: https://issues.apache.org/jira/browse/HIVE-21283 > Project: Hive > Issue Type: New Feature > Components: UDF >Reporter: Mani M >Assignee: Mani M >Priority: Minor > Labels: UDF, pull-request-available, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, > HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, > HIVE.21283.08.PATCH, HIVE.21283.09.PATCH, HIVE.21283.10.PATCH, > HIVE.21283.2.PATCH, HIVE.21283.PATCH, image-2019-03-16-21-31-15-541.png, > image-2019-03-16-21-33-18-898.png > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Create new synonym for the existing function > > Mid for substr > postiion for locate -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21509) LLAP may cache corrupted column vectors and return wrong query result
[
https://issues.apache.org/jira/browse/HIVE-21509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805481#comment-16805481
]
Hive QA commented on HIVE-21509:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
49s{color} | {color:blue} llap-server in master has 81 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
22s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 22s{color}
| {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} llap-server: The patch generated 4 new + 29 unchanged
- 1 fixed = 33 total (was 30) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16762/dev-support/hive-personality.sh
|
| git revision | master / 23ab7f2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16762/yetus/patch-mvninstall-llap-server.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16762/yetus/patch-compile-llap-server.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16762/yetus/patch-compile-llap-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16762/yetus/diff-checkstyle-llap-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16762/yetus/patch-findbugs-llap-server.txt
|
| modules | C: storage-api llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16762/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP may cache corrupted column vectors and return wrong query result
> -
>
> Key: HIVE-21509
> URL: https://issues.apache.org/jira/browse/HIVE-21509
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21509.0.wip.patch, HIVE-21509.1.wip.patch,
> HIVE-21509.2.patch
>
>
> In some scenarios, LLAP might store column vectors in cache that
[jira] [Updated] (HIVE-21537) Scalar query rewrite could be improved to not generate an extra join if subquery is guaranteed to produce atmost one row
[ https://issues.apache.org/jira/browse/HIVE-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21537: --- Attachment: HIVE-21537.3.patch > Scalar query rewrite could be improved to not generate an extra join if > subquery is guaranteed to produce atmost one row > > > Key: HIVE-21537 > URL: https://issues.apache.org/jira/browse/HIVE-21537 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: sub-query > Attachments: HIVE-21537.1.patch, HIVE-21537.2.patch, > HIVE-21537.3.patch > > > Currently Hive planner introduces this branch and later executes a rule to > remove this branch if it could. > Subquery remove rule itself could check if subquery will produce max one row > (using relmetadat's getMaxRowCount) and avoid introducing this branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21537) Scalar query rewrite could be improved to not generate an extra join if subquery is guaranteed to produce atmost one row
[ https://issues.apache.org/jira/browse/HIVE-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21537: --- Status: Open (was: Patch Available) > Scalar query rewrite could be improved to not generate an extra join if > subquery is guaranteed to produce atmost one row > > > Key: HIVE-21537 > URL: https://issues.apache.org/jira/browse/HIVE-21537 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: sub-query > Attachments: HIVE-21537.1.patch, HIVE-21537.2.patch, > HIVE-21537.3.patch > > > Currently Hive planner introduces this branch and later executes a rule to > remove this branch if it could. > Subquery remove rule itself could check if subquery will produce max one row > (using relmetadat's getMaxRowCount) and avoid introducing this branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21537) Scalar query rewrite could be improved to not generate an extra join if subquery is guaranteed to produce atmost one row
[ https://issues.apache.org/jira/browse/HIVE-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21537: --- Status: Patch Available (was: Open) > Scalar query rewrite could be improved to not generate an extra join if > subquery is guaranteed to produce atmost one row > > > Key: HIVE-21537 > URL: https://issues.apache.org/jira/browse/HIVE-21537 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: sub-query > Attachments: HIVE-21537.1.patch, HIVE-21537.2.patch, > HIVE-21537.3.patch > > > Currently Hive planner introduces this branch and later executes a rule to > remove this branch if it could. > Subquery remove rule itself could check if subquery will produce max one row > (using relmetadat's getMaxRowCount) and avoid introducing this branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21230) LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side (HiveJoinAddNotNullRule bails out for outer joins)
[
https://issues.apache.org/jira/browse/HIVE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21230:
---
Status: Patch Available (was: Open)
> LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side
> (HiveJoinAddNotNullRule bails out for outer joins)
>
>
> Key: HIVE-21230
> URL: https://issues.apache.org/jira/browse/HIVE-21230
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21230.1.patch, HIVE-21230.2.patch,
> HIVE-21230.3.patch, HIVE-21230.4.patch, HIVE-21230.5.patch, HIVE-21230.6.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> LEFT JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 = t1.col0 AND t0.col1 = t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null in the right input and
> introduce the corresponding filter predicate. Currently, the rule just bails
> out if it is not an inner join.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java#L79
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21230) LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side (HiveJoinAddNotNullRule bails out for outer joins)
[
https://issues.apache.org/jira/browse/HIVE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21230:
---
Attachment: HIVE-21230.6.patch
> LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side
> (HiveJoinAddNotNullRule bails out for outer joins)
>
>
> Key: HIVE-21230
> URL: https://issues.apache.org/jira/browse/HIVE-21230
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21230.1.patch, HIVE-21230.2.patch,
> HIVE-21230.3.patch, HIVE-21230.4.patch, HIVE-21230.5.patch, HIVE-21230.6.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> LEFT JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 = t1.col0 AND t0.col1 = t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null in the right input and
> introduce the corresponding filter predicate. Currently, the rule just bails
> out if it is not an inner join.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java#L79
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21230) LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side (HiveJoinAddNotNullRule bails out for outer joins)
[
https://issues.apache.org/jira/browse/HIVE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21230:
---
Status: Open (was: Patch Available)
> LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side
> (HiveJoinAddNotNullRule bails out for outer joins)
>
>
> Key: HIVE-21230
> URL: https://issues.apache.org/jira/browse/HIVE-21230
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21230.1.patch, HIVE-21230.2.patch,
> HIVE-21230.3.patch, HIVE-21230.4.patch, HIVE-21230.5.patch, HIVE-21230.6.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> LEFT JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 = t1.col0 AND t0.col1 = t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null in the right input and
> introduce the corresponding filter predicate. Currently, the rule just bails
> out if it is not an inner join.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java#L79
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21543) Use FilterHooks for show compactions
[
https://issues.apache.org/jira/browse/HIVE-21543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805466#comment-16805466
]
Hive QA commented on HIVE-21543:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964192/HIVE-21543.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15882 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16761/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16761/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16761/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12964192 - PreCommit-HIVE-Build
> Use FilterHooks for show compactions
>
>
> Key: HIVE-21543
> URL: https://issues.apache.org/jira/browse/HIVE-21543
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21543.01.patch
>
>
> Use FilterHooks for checking dbs/tables/partitions for showCompactions
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21367) Hive returns an incorrect result when using a simple select query
[
https://issues.apache.org/jira/browse/HIVE-21367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Thistlethwaite reassigned HIVE-21367:
---
Assignee: (was: S. Poortman19-88)
> Hive returns an incorrect result when using a simple select query
> -
>
> Key: HIVE-21367
> URL: https://issues.apache.org/jira/browse/HIVE-21367
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, JDBC, SQL
>Affects Versions: 3.1.0
> Environment: - HDP 3.1
> - Hive 3.1.0
> - Spark 2.3.2
> - Sqoop 1.4.7
>Reporter: LEMBARKI Mohamed Amine
>Priority: Blocker
> Attachments: mapred_input_dir_recursive.png
>
>
> Hive returns an incorrect result when using a simple select query with a
> where clause
> While with an aggregation it returns a correct result
> The problem arises for tables created by Spark or Sqoop
> Also when we use spark-shell with HiveWarehouseConnector it returns a correct
> result
>
> Workflow:
> - Loading data with sqoop to hive
> - Data processing with spark using HiveWarehouseConnector and Storage to
> Hive
>
> below the error log :
>
> */-*
> *1 - Executing Query : select code from db1.tbl1 where code = '123'*
> */-*
> {code:java}
> [data@data1 ~]$ hive -e "select code from db1.tbl1 where code = '123'"
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Connecting to
> jdbc:hive2://data2:2181,data1:2181/default;password=data;serviceDiscoveryMode=zooKeeper;user=data;zooKeeperNamespace=hiveserver2
> 19/03/01 10:31:36 [main]: INFO jdbc.HiveConnection: Connected to data2:1
> Connected to: Apache Hive (version 3.1.0.3.1.0.0-78)
> Driver: Hive JDBC (version 3.1.0.3.1.0.0-78)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> INFO : Compiling
> command(queryId=hive_20190301103129_d48e71f6-a8dd-490e-a574-04d8d4f893e2):
> select code from db1.tbl1 where code = '123'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:code,
> type:string, comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20190301103129_d48e71f6-a8dd-490e-a574-04d8d4f893e2);
> Time taken: 0.142 seconds
> INFO : Executing
> command(queryId=hive_20190301103129_d48e71f6-a8dd-490e-a574-04d8d4f893e2):
> select code from db1.tbl1 where code = '123'
> INFO : Completed executing
> command(queryId=hive_20190301103129_d48e71f6-a8dd-490e-a574-04d8d4f893e2);
> Time taken: 0.003 seconds
> INFO : OK
> +--+
> | code |
> +--+
> +--+
> No rows selected (4,307 seconds)
> Beeline version 3.1.0.3.1.0.0-78 by Apache Hive
> Closing: 0:
> jdbc:hive2://data2:2181,data1:2181/default;password=data;serviceDiscoveryMode=zooKeeper;user=data;zooKeeperNamespace=hiveserver2
> {code}
> */-*
> *2 - Executing Query using count :*
> *select count(code) from db1.tbl1 where code = '123'*
> */-*
> {code:java}
> [data@data1 ~]$ hive -e "select count(code) from db1.tbl1 where code = '123'"
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Connecting to
> jdbc:hive2://data2:2181,data1:2181/default;password=data;serviceDiscoveryMode=zooKeeper;user=data;zooKeeperNamespace=hiveserver2
> 19/03/01 10:31:56 [main]: INFO jdbc.HiveConnection: Connected to data2:1
> Connected to: Apache Hive (version 3.1.0.3.1.0.0-78)
> Driver: Hive JDBC (version 3.1.0.3.1.0.0-78)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> INFO : Compiling
> command(queryId=hive_20190301103149_90aa338b-b99b-4f1c-b7e5-6b285f64cb3e):
> select count(code) from db1.tbl1 where code = '123'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
[jira] [Issue Comment Deleted] (HIVE-20284) In strict mode, if constant propagation is enable, the partition filter may be folded before partition pruner lead to error "No partition predicate for Alia
[
https://issues.apache.org/jira/browse/HIVE-20284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Thistlethwaite updated HIVE-20284:
Comment: was deleted
(was: For my)
> In strict mode, if constant propagation is enable, the partition filter may
> be folded before partition pruner lead to error "No partition predicate for
> Alias"
>
>
> Key: HIVE-20284
> URL: https://issues.apache.org/jira/browse/HIVE-20284
> Project: Hive
> Issue Type: Improvement
> Components: CLI
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Priority: Trivial
> Fix For: 4.0.0
>
> Attachments: HIVE-20284.1.patch, HIVE-20284.2.patch,
> HIVE-20284.3.patch, HIVE-20284.patch
>
>
> In strict mode and the hive.optimize.constant.propagation is set to true, the
> following sql will failed:
> {code:java}
> hive> desc employee_part;
> OK
> col_name data_type comment
> eid int
> name string
> dept string
> year string
> month string
> # Partition Information
> # col_namedata_type comment
> year string
> month string
> Time taken: 0.564 seconds, Fetched: 11 row(s)
> hive> set hive.mapred.mode=strict;
> hive> select * from employee_part where false and concat(year,month)='201807';
> FAILED: SemanticException Queries against partitioned tables without a
> partition filter are disabled for safety reasons. If you know what you are
> doing, please sethive.strict.checks.large.query to false and that
> hive.mapred.mode is not set to 'strict' to proceed. Note that if you may get
> errors or incorrect results if you make a mistake while using some of the
> unsafe features. No partition predicate for Alias "employee_part" Table
> "employee_part"
> {code}
> The above error message is confusing because the expression
> concat(year,month)='201807' is a partition filter。
> The reason is during logic optimization, the ConstantPropagate optimizer is
> running before partitionPruner optimizer, when found a express like 'false
> and concat(year,month)=', the express is replaced with 'fasle' , and the
> partition filter is dropped. So the PartitionPruner can not get the partition
> filter.
> Users can remove the constant express that always has true/false values to
> work around.
> When views used, if some columns are constant values, users will be
> confusing.
> So we should add some more message in the error msg returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20284) In strict mode, if constant propagation is enable, the partition filter may be folded before partition pruner lead to error "No partition predicate for Alias"
[
https://issues.apache.org/jira/browse/HIVE-20284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Thistlethwaite reassigned HIVE-20284:
---
Assignee: (was: S. Poortman19-88)
> In strict mode, if constant propagation is enable, the partition filter may
> be folded before partition pruner lead to error "No partition predicate for
> Alias"
>
>
> Key: HIVE-20284
> URL: https://issues.apache.org/jira/browse/HIVE-20284
> Project: Hive
> Issue Type: Improvement
> Components: CLI
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Priority: Trivial
> Fix For: 4.0.0
>
> Attachments: HIVE-20284.1.patch, HIVE-20284.2.patch,
> HIVE-20284.3.patch, HIVE-20284.patch
>
>
> In strict mode and the hive.optimize.constant.propagation is set to true, the
> following sql will failed:
> {code:java}
> hive> desc employee_part;
> OK
> col_name data_type comment
> eid int
> name string
> dept string
> year string
> month string
> # Partition Information
> # col_namedata_type comment
> year string
> month string
> Time taken: 0.564 seconds, Fetched: 11 row(s)
> hive> set hive.mapred.mode=strict;
> hive> select * from employee_part where false and concat(year,month)='201807';
> FAILED: SemanticException Queries against partitioned tables without a
> partition filter are disabled for safety reasons. If you know what you are
> doing, please sethive.strict.checks.large.query to false and that
> hive.mapred.mode is not set to 'strict' to proceed. Note that if you may get
> errors or incorrect results if you make a mistake while using some of the
> unsafe features. No partition predicate for Alias "employee_part" Table
> "employee_part"
> {code}
> The above error message is confusing because the expression
> concat(year,month)='201807' is a partition filter。
> The reason is during logic optimization, the ConstantPropagate optimizer is
> running before partitionPruner optimizer, when found a express like 'false
> and concat(year,month)=', the express is replaced with 'fasle' , and the
> partition filter is dropped. So the PartitionPruner can not get the partition
> filter.
> Users can remove the constant express that always has true/false values to
> work around.
> When views used, if some columns are constant values, users will be
> confusing.
> So we should add some more message in the error msg returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21543) Use FilterHooks for show compactions
[
https://issues.apache.org/jira/browse/HIVE-21543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805438#comment-16805438
]
Hive QA commented on HIVE-21543:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
45s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
18s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 4 new + 194 unchanged - 0 fixed = 198 total (was 194) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
3s{color} | {color:red} standalone-metastore/metastore-common generated 1 new +
29 unchanged - 0 fixed = 30 total (was 29) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
3s{color} | {color:red} standalone-metastore_metastore-common generated 2 new +
49 unchanged - 0 fixed = 51 total (was 49) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-common |
| |
org.apache.hadoop.hive.metastore.utils.FilterUtils.filterCompactionsIfEnabled(boolean,
MetaStoreFilterHook, String, List) makes inefficient use of keySet iterator
instead of entrySet iterator At FilterUtils.java:inefficient use of keySet
iterator instead of entrySet iterator At FilterUtils.java:[line 428] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16761/dev-support/hive-personality.sh
|
| git revision | master / 23ab7f2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16761/yetus/diff-checkstyle-standalone-metastore_metastore-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16761/yetus/new-findbugs-standalone-metastore_metastore-common.html
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16761/yetus/diff-javadoc-javadoc-standalone-metastore_metastore-common.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16761/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use FilterHooks for show compactions
>
[jira] [Issue Comment Deleted] (HIVE-21455) Too verbose logging in AvroGenericRecordReader
[
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Thistlethwaite updated HIVE-21455:
Comment: was deleted
(was: Pleas)
> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0, 1.1.0, 2.0.0, 2.1.0, 3.0.0, 3.1.0
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-21455.2.patch, HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema
> and thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-21455) Too verbose logging in AvroGenericRecordReader
[
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Thistlethwaite updated HIVE-21455:
Comment: was deleted
(was: On S. Poortman)
> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0, 1.1.0, 2.0.0, 2.1.0, 3.0.0, 3.1.0
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-21455.2.patch, HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema
> and thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21283) Create Synonym mid for substr, position for locate
[ https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Thistlethwaite reassigned HIVE-21283: --- Assignee: (was: S. Poortman19-88) > Create Synonym mid for substr, position for locate > > > Key: HIVE-21283 > URL: https://issues.apache.org/jira/browse/HIVE-21283 > Project: Hive > Issue Type: New Feature > Components: UDF >Reporter: Mani M >Priority: Minor > Labels: UDF, pull-request-available, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, > HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, > HIVE.21283.08.PATCH, HIVE.21283.09.PATCH, HIVE.21283.10.PATCH, > HIVE.21283.2.PATCH, HIVE.21283.PATCH, image-2019-03-16-21-31-15-541.png, > image-2019-03-16-21-33-18-898.png > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Create new synonym for the existing function > > Mid for substr > postiion for locate -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21537) Scalar query rewrite could be improved to not generate an extra join if subquery is guaranteed to produce atmost one row
[ https://issues.apache.org/jira/browse/HIVE-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805429#comment-16805429 ] Vineet Garg commented on HIVE-21537: [~jcamachorodriguez] Can you take a look? [https://github.com/apache/hive/pull/583] > Scalar query rewrite could be improved to not generate an extra join if > subquery is guaranteed to produce atmost one row > > > Key: HIVE-21537 > URL: https://issues.apache.org/jira/browse/HIVE-21537 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: sub-query > Attachments: HIVE-21537.1.patch, HIVE-21537.2.patch > > > Currently Hive planner introduces this branch and later executes a rule to > remove this branch if it could. > Subquery remove rule itself could check if subquery will produce max one row > (using relmetadat's getMaxRowCount) and avoid introducing this branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21537) Scalar query rewrite could be improved to not generate an extra join if subquery is guaranteed to produce atmost one row
[ https://issues.apache.org/jira/browse/HIVE-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21537: --- Attachment: HIVE-21537.2.patch > Scalar query rewrite could be improved to not generate an extra join if > subquery is guaranteed to produce atmost one row > > > Key: HIVE-21537 > URL: https://issues.apache.org/jira/browse/HIVE-21537 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: sub-query > Attachments: HIVE-21537.1.patch, HIVE-21537.2.patch > > > Currently Hive planner introduces this branch and later executes a rule to > remove this branch if it could. > Subquery remove rule itself could check if subquery will produce max one row > (using relmetadat's getMaxRowCount) and avoid introducing this branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21537) Scalar query rewrite could be improved to not generate an extra join if subquery is guaranteed to produce atmost one row
[ https://issues.apache.org/jira/browse/HIVE-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21537: --- Status: Patch Available (was: Open) > Scalar query rewrite could be improved to not generate an extra join if > subquery is guaranteed to produce atmost one row > > > Key: HIVE-21537 > URL: https://issues.apache.org/jira/browse/HIVE-21537 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: sub-query > Attachments: HIVE-21537.1.patch, HIVE-21537.2.patch > > > Currently Hive planner introduces this branch and later executes a rule to > remove this branch if it could. > Subquery remove rule itself could check if subquery will produce max one row > (using relmetadat's getMaxRowCount) and avoid introducing this branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21537) Scalar query rewrite could be improved to not generate an extra join if subquery is guaranteed to produce atmost one row
[ https://issues.apache.org/jira/browse/HIVE-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21537: --- Attachment: (was: HIVE-21537.2.patch) > Scalar query rewrite could be improved to not generate an extra join if > subquery is guaranteed to produce atmost one row > > > Key: HIVE-21537 > URL: https://issues.apache.org/jira/browse/HIVE-21537 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: sub-query > Attachments: HIVE-21537.1.patch > > > Currently Hive planner introduces this branch and later executes a rule to > remove this branch if it could. > Subquery remove rule itself could check if subquery will produce max one row > (using relmetadat's getMaxRowCount) and avoid introducing this branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21537) Scalar query rewrite could be improved to not generate an extra join if subquery is guaranteed to produce atmost one row
[ https://issues.apache.org/jira/browse/HIVE-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21537: --- Status: Open (was: Patch Available) > Scalar query rewrite could be improved to not generate an extra join if > subquery is guaranteed to produce atmost one row > > > Key: HIVE-21537 > URL: https://issues.apache.org/jira/browse/HIVE-21537 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: sub-query > Attachments: HIVE-21537.1.patch, HIVE-21537.2.patch > > > Currently Hive planner introduces this branch and later executes a rule to > remove this branch if it could. > Subquery remove rule itself could check if subquery will produce max one row > (using relmetadat's getMaxRowCount) and avoid introducing this branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21537) Scalar query rewrite could be improved to not generate an extra join if subquery is guaranteed to produce atmost one row
[ https://issues.apache.org/jira/browse/HIVE-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21537: --- Status: Open (was: Patch Available) > Scalar query rewrite could be improved to not generate an extra join if > subquery is guaranteed to produce atmost one row > > > Key: HIVE-21537 > URL: https://issues.apache.org/jira/browse/HIVE-21537 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: sub-query > Attachments: HIVE-21537.1.patch, HIVE-21537.2.patch > > > Currently Hive planner introduces this branch and later executes a rule to > remove this branch if it could. > Subquery remove rule itself could check if subquery will produce max one row > (using relmetadat's getMaxRowCount) and avoid introducing this branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21537) Scalar query rewrite could be improved to not generate an extra join if subquery is guaranteed to produce atmost one row
[ https://issues.apache.org/jira/browse/HIVE-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21537: --- Attachment: HIVE-21537.2.patch > Scalar query rewrite could be improved to not generate an extra join if > subquery is guaranteed to produce atmost one row > > > Key: HIVE-21537 > URL: https://issues.apache.org/jira/browse/HIVE-21537 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: sub-query > Attachments: HIVE-21537.1.patch, HIVE-21537.2.patch > > > Currently Hive planner introduces this branch and later executes a rule to > remove this branch if it could. > Subquery remove rule itself could check if subquery will produce max one row > (using relmetadat's getMaxRowCount) and avoid introducing this branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21537) Scalar query rewrite could be improved to not generate an extra join if subquery is guaranteed to produce atmost one row
[ https://issues.apache.org/jira/browse/HIVE-21537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21537: --- Status: Patch Available (was: Open) > Scalar query rewrite could be improved to not generate an extra join if > subquery is guaranteed to produce atmost one row > > > Key: HIVE-21537 > URL: https://issues.apache.org/jira/browse/HIVE-21537 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: sub-query > Attachments: HIVE-21537.1.patch, HIVE-21537.2.patch > > > Currently Hive planner introduces this branch and later executes a rule to > remove this branch if it could. > Subquery remove rule itself could check if subquery will produce max one row > (using relmetadat's getMaxRowCount) and avoid introducing this branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21529) Hive support bootstrap of ACID/MM tables on an existing policy.
[
https://issues.apache.org/jira/browse/HIVE-21529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805417#comment-16805417
]
Hive QA commented on HIVE-21529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964174/HIVE-21529.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15881 tests
executed
*Failed tests:*
{noformat}
TestRemoteUGIHiveMetaStoreIpAddress - did not produce a TEST-*.xml file (likely
timed out) (batchId=228)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16760/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16760/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16760/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12964174 - PreCommit-HIVE-Build
> Hive support bootstrap of ACID/MM tables on an existing policy.
> ---
>
> Key: HIVE-21529
> URL: https://issues.apache.org/jira/browse/HIVE-21529
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Transactions
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21529.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If ACID/MM tables to be enabled (hive.repl.dump.include.acid.tables) on an
> existing repl policy, then need to combine bootstrap dump of these tables
> along with the ongoing incremental dump.
> Shall add a one time config "hive.repl.bootstrap.acid.tables" to include
> bootstrap in the given dump.
> The support for hive.repl.bootstrap.cleanup.type for ACID tables to clean-up
> partially bootstrapped tables in case of retry is already in place, thanks to
> the work done during external tables. Need to test that it actually works.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-21484:
---
Fix Version/s: 2.4.0
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 2.4.0, 4.0.0, 3.2.0
>
> Attachments: HIVE-21484.01.patch, HIVE-21484.02.patch,
> HIVE-21484.03.patch, HIVE-21484.04.branch-3.patch,
> HIVE-21484.05.branch-2.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This version number can be used to do a version handshake between client
> and server in the future to improve our cross-version compatibity story.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21529) Hive support bootstrap of ACID/MM tables on an existing policy.
[
https://issues.apache.org/jira/browse/HIVE-21529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805391#comment-16805391
]
Hive QA commented on HIVE-21529:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
40s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16760/dev-support/hive-personality.sh
|
| git revision | master / 23ab7f2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16760/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive support bootstrap of ACID/MM tables on an existing policy.
> ---
>
> Key: HIVE-21529
> URL: https://issues.apache.org/jira/browse/HIVE-21529
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Transactions
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-21529.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If ACID/MM tables to be enabled (hive.repl.dump.include.acid.tables) on an
> existing repl policy, then need to combine bootstrap dump of these tables
> along with the ongoing incremental dump.
> Shall add a one time config "hive.repl.bootstrap.acid.tables" to include
> bootstrap in the given dump.
> The support for hive.repl.bootstrap.cleanup.type for ACID tables to clean-up
> partially bootstrapped tables in case of retry is already in place, thanks to
> the work done during external tables. Need to test that it actually works.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805392#comment-16805392
]
Vihang Karajgaonkar commented on HIVE-21484:
merged into branch-2 as well.
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 2.4.0, 4.0.0, 3.2.0
>
> Attachments: HIVE-21484.01.patch, HIVE-21484.02.patch,
> HIVE-21484.03.patch, HIVE-21484.04.branch-3.patch,
> HIVE-21484.05.branch-2.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This version number can be used to do a version handshake between client
> and server in the future to improve our cross-version compatibity story.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-21484:
---
Attachment: HIVE-21484.05.branch-2.patch
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-21484.01.patch, HIVE-21484.02.patch,
> HIVE-21484.03.patch, HIVE-21484.04.branch-3.patch,
> HIVE-21484.05.branch-2.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This version number can be used to do a version handshake between client
> and server in the future to improve our cross-version compatibity story.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21532) RuntimeException due to AccessControlException during creating hive-staging-dir
[
https://issues.apache.org/jira/browse/HIVE-21532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805352#comment-16805352
]
Hive QA commented on HIVE-21532:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964152/HIVE-21532.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16758/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16758/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16758/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-29 19:04:43.134
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16758/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-29 19:04:43.138
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
ce0d3a6..23ab7f2 master -> origin/master
5a65c34..a91e6ec branch-3 -> origin/branch-3
+ git reset --hard HEAD
HEAD is now at ce0d3a6 HIVE-21541: Fix missing asf headers from HIVE-15406
(Laszlo Bodor reviewed by Peter Vary) addendum
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 23ab7f2 HIVE-21484 : Metastore API getVersion() should return
real version (Vihang Karajgaonkar reviewed by Naveen Gangam, Peter Vary)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-29 19:04:45.120
+ rm -rf ../yetus_PreCommit-HIVE-Build-16758
+ mkdir ../yetus_PreCommit-HIVE-Build-16758
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16758
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16758/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
does not exist in index
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7233
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java' with
conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7233
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java' with
conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-16758
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12964152 - PreCommit-HIVE-Build
> RuntimeException due to AccessControlException during creating
> hive-staging-dir
> ---
>
> Key: HIVE-21532
> URL: https://issues.apache.org/jira/browse/HIVE-21532
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksandr Polishchuk
>Assignee: Oleksandr Polishchuk
>Priority: Minor
> Attachments: HIVE-21532.1.patch, HIVE-21532.1.patch,
> HIVE-21532.2.patch
>
>
> The bug was found with environment - Hive-2.3.
> Steps lead to an exception:
> 1) Create user without root permissions on your node.
> 2) The {{hive-site.xml}} file has to contain the next properties:
> {code:java}
>
> hive.security.authorization.enabled
> true
>
>
>
[jira] [Commented] (HIVE-21316) Comparision of varchar column and string literal should happen in varchar
[
https://issues.apache.org/jira/browse/HIVE-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805349#comment-16805349
]
Hive QA commented on HIVE-21316:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964147/HIVE-21316.07.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 15806 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=230)
TestHiveKVResultCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=320)
TestHiveSparkClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=320)
TestLocalHiveSparkClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=320)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestPartitionProjectionEvaluator - did not produce a TEST-*.xml file (likely
timed out) (batchId=230)
TestSmallTableCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=320)
TestSparkInvalidFileFormat - did not produce a TEST-*.xml file (likely timed
out) (batchId=320)
TestSparkJobMonitor - did not produce a TEST-*.xml file (likely timed out)
(batchId=320)
TestSparkPlan - did not produce a TEST-*.xml file (likely timed out)
(batchId=320)
TestSparkSessionManagerImpl - did not produce a TEST-*.xml file (likely timed
out) (batchId=320)
TestSparkTask - did not produce a TEST-*.xml file (likely timed out)
(batchId=320)
TestSparkUtilities - did not produce a TEST-*.xml file (likely timed out)
(batchId=320)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables_sysdb]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=109)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16757/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16757/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16757/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12964147 - PreCommit-HIVE-Build
> Comparision of varchar column and string literal should happen in varchar
> -
>
> Key: HIVE-21316
> URL: https://issues.apache.org/jira/browse/HIVE-21316
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21316.01.patch, HIVE-21316.02.patch,
> HIVE-21316.03.patch, HIVE-21316.04.patch, HIVE-21316.05.patch,
> HIVE-21316.06.patch, HIVE-21316.06.patch, HIVE-21316.07.patch,
> HIVE-21316.07.patch
>
>
> this is most probably the root cause behind HIVE-21310 as well
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-21484:
---
Resolution: Fixed
Fix Version/s: 3.2.0
4.0.0
Status: Resolved (was: Patch Available)
patch merged into branch-3
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-21484.01.patch, HIVE-21484.02.patch,
> HIVE-21484.03.patch, HIVE-21484.04.branch-3.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This version number can be used to do a version handshake between client
> and server in the future to improve our cross-version compatibity story.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805283#comment-16805283
]
Vihang Karajgaonkar commented on HIVE-21484:
branch-3 precommit has issues with test timeouts as reported in HIVE-21180. I
tested the patch locally. The code compiles and I ran the
{{TestEmbeddedHiveMetaStore}} and {{TestRemoteHiveMetaStore}}
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-21484.01.patch, HIVE-21484.02.patch,
> HIVE-21484.03.patch, HIVE-21484.04.branch-3.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This version number can be used to do a version handshake between client
> and server in the future to improve our cross-version compatibity story.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21546) hiveserver2 - “mapred.FileInputFormat: Total input files to process” - why single threaded?
[
https://issues.apache.org/jira/browse/HIVE-21546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
t oo updated HIVE-21546:
Affects Version/s: 3.1.1
2.3.4
Component/s: StorageHandler
storage-api
File Formats
> hiveserver2 - “mapred.FileInputFormat: Total input files to process” - why
> single threaded?
> ---
>
> Key: HIVE-21546
> URL: https://issues.apache.org/jira/browse/HIVE-21546
> Project: Hive
> Issue Type: Bug
> Components: File Formats, storage-api, StorageHandler
>Affects Versions: 3.1.1, 2.3.4
>Reporter: t oo
>Priority: Major
>
> I have setup Hive (v2.3.4) on Spark (exec engine, but MR gets same issue),
> hadoop 2.7.6 (or hadoop 2.8.5). My external hive table is Parquet format on
> s3 across 100s of partitions. Below settings are set to 20:
> {\{hive.exec.input.listing.max.threads mapred.dfsclient.parallelism.max
> mapreduce.input.fileinputformat.list-status.num-threads }}
> Run a simple query:
> {\{select * from s.there h_code = 'KGD78' and h_no = '265' }}
> I can see the below in HiveServer2 logs (the logs continue for more than 1000
> lines listing all the different partitions). Why is the listing of files not
> being done in parallel? It takes more than 5mins just in the listing.
> {{2019-03-29T11:29:26,866 INFO [3fa82455-7853-4c4b-8964-847c00bec708
> HiveServer2-Handler-Pool: Thread-53] compress.CodecPool: Got brand-new
> decompressor [.snappy] 2019-03-29T11:29:27,283 INFO
> [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
> mapred.FileInputFormat: Total input files to process : 1
> 2019-03-29T11:29:27,797 INFO [3fa82455-7853-4c4b-8964-847c00bec708
> HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input
> files to process : 1 2019-03-29T11:29:28,374 INFO
> [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
> mapred.FileInputFormat: Total input files to process : 1
> 2019-03-29T11:29:28,919 INFO [3fa82455-7853-4c4b-8964-847c00bec708
> HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input
> files to process : 1 2019-03-29T11:29:29,483 INFO
> [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
> mapred.FileInputFormat: Total input files to process : 1
> 2019-03-29T11:29:30,003 INFO [3fa82455-7853-4c4b-8964-847c00bec708
> HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input
> files to process : 1 2019-03-29T11:29:30,518 INFO
> [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
> mapred.FileInputFormat: Total input files to process : 1
> 2019-03-29T11:29:31,001 INFO [3fa82455-7853-4c4b-8964-847c00bec708
> HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input
> files to process : 1 2019-03-29T11:29:31,549 INFO
> [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
> mapred.FileInputFormat: Total input files to process : 1
> 2019-03-29T11:29:32,048 INFO [3fa82455-7853-4c4b-8964-847c00bec708
> HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input
> files to process : 1 2019-03-29T11:29:32,574 INFO
> [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
> mapred.FileInputFormat: Total input files to process : 1
> 2019-03-29T11:29:33,130 INFO [3fa82455-7853-4c4b-8964-847c00bec708
> HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input
> files to process : 1 2019-03-29T11:29:33,639 INFO
> [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
> mapred.FileInputFormat: Total input files to process : 1
> 2019-03-29T11:29:34,189 INFO [3fa82455-7853-4c4b-8964-847c00bec708
> HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input
> files to process : 1 2019-03-29T11:29:34,743 INFO
> [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
> mapred.FileInputFormat: Total input files to process : 1
> 2019-03-29T11:29:35,208 INFO [3fa82455-7853-4c4b-8964-847c00bec708
> HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input
> files to process : 1 2019-03-29T11:29:35,701 INFO
> [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
> mapred.FileInputFormat: Total input files to process : 1
> 2019-03-29T11:29:36,183 INFO [3fa82455-7853-4c4b-8964-847c00bec708
> HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input
> files to process : 1 2019-03-29T11:29:36,662 INFO
> [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
> mapred.FileInputFormat: Total input files to process : 1
> 2019-03-29T11:29:37,154 INFO [3fa82455-7853-4c4b-8964-847c00bec708
>
[jira] [Updated] (HIVE-21546) hiveserver2 - “mapred.FileInputFormat: Total input files to process” - why single threaded?
[
https://issues.apache.org/jira/browse/HIVE-21546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
t oo updated HIVE-21546:
Description:
I have setup Hive (v2.3.4) on Spark (exec engine, but MR gets same issue),
hadoop 2.7.6 (or hadoop 2.8.5). My external hive table is Parquet format on s3
across 100s of partitions. Below settings are set to 20:
{\{hive.exec.input.listing.max.threads mapred.dfsclient.parallelism.max
mapreduce.input.fileinputformat.list-status.num-threads }}
Run a simple query:
{\{select * from s.there h_code = 'KGD78' and h_no = '265' }}
I can see the below in HiveServer2 logs (the logs continue for more than 1000
lines listing all the different partitions). Why is the listing of files not
being done in parallel? It takes more than 5mins just in the listing.
{{2019-03-29T11:29:26,866 INFO [3fa82455-7853-4c4b-8964-847c00bec708
HiveServer2-Handler-Pool: Thread-53] compress.CodecPool: Got brand-new
decompressor [.snappy] 2019-03-29T11:29:27,283 INFO
[3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
mapred.FileInputFormat: Total input files to process : 1
2019-03-29T11:29:27,797 INFO [3fa82455-7853-4c4b-8964-847c00bec708
HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files
to process : 1 2019-03-29T11:29:28,374 INFO
[3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
mapred.FileInputFormat: Total input files to process : 1
2019-03-29T11:29:28,919 INFO [3fa82455-7853-4c4b-8964-847c00bec708
HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files
to process : 1 2019-03-29T11:29:29,483 INFO
[3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
mapred.FileInputFormat: Total input files to process : 1
2019-03-29T11:29:30,003 INFO [3fa82455-7853-4c4b-8964-847c00bec708
HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files
to process : 1 2019-03-29T11:29:30,518 INFO
[3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
mapred.FileInputFormat: Total input files to process : 1
2019-03-29T11:29:31,001 INFO [3fa82455-7853-4c4b-8964-847c00bec708
HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files
to process : 1 2019-03-29T11:29:31,549 INFO
[3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
mapred.FileInputFormat: Total input files to process : 1
2019-03-29T11:29:32,048 INFO [3fa82455-7853-4c4b-8964-847c00bec708
HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files
to process : 1 2019-03-29T11:29:32,574 INFO
[3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
mapred.FileInputFormat: Total input files to process : 1
2019-03-29T11:29:33,130 INFO [3fa82455-7853-4c4b-8964-847c00bec708
HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files
to process : 1 2019-03-29T11:29:33,639 INFO
[3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
mapred.FileInputFormat: Total input files to process : 1
2019-03-29T11:29:34,189 INFO [3fa82455-7853-4c4b-8964-847c00bec708
HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files
to process : 1 2019-03-29T11:29:34,743 INFO
[3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
mapred.FileInputFormat: Total input files to process : 1
2019-03-29T11:29:35,208 INFO [3fa82455-7853-4c4b-8964-847c00bec708
HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files
to process : 1 2019-03-29T11:29:35,701 INFO
[3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
mapred.FileInputFormat: Total input files to process : 1
2019-03-29T11:29:36,183 INFO [3fa82455-7853-4c4b-8964-847c00bec708
HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files
to process : 1 2019-03-29T11:29:36,662 INFO
[3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
mapred.FileInputFormat: Total input files to process : 1
2019-03-29T11:29:37,154 INFO [3fa82455-7853-4c4b-8964-847c00bec708
HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files
to process : 1 2019-03-29T11:29:37,645 INFO
[3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53]
mapred.FileInputFormat: Total input files to process : 1 }}
I've tried
{\{hive.exec.input.listing.max.threads mapred.dfsclient.parallelism.max
mapreduce.input.fileinputformat.list-status.num-threads }}
with defaults, 1, 50...still same result
Hive 3.1.1/hadoop3.1.2 also has the issue:
2019-03-29T18:10:15,451 INFO [16b32706-3490-432d-b49e-67279ea88e15
HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: at row
0. reading next block
2019-03-29T18:10:15,461 INFO [16b32706-3490-432d-b49e-67279ea88e15
HiveServer2-Handler-Pool: Thread-30]
[jira] [Commented] (HIVE-21316) Comparision of varchar column and string literal should happen in varchar
[
https://issues.apache.org/jira/browse/HIVE-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805276#comment-16805276
]
Hive QA commented on HIVE-21316:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
46s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
48s{color} | {color:red} ql: The patch generated 6 new + 137 unchanged - 0
fixed = 143 total (was 137) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 9 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 5m
12s{color} | {color:red} ql generated 1 new + 2256 unchanged - 0 fixed = 2257
total (was 2256) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter$HiveNlsString
doesn't override org.apache.calcite.util.NlsString.equals(Object) At
RexNodeConverter.java:At RexNodeConverter.java:[line 1] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16757/dev-support/hive-personality.sh
|
| git revision | master / 1734f6b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16757/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16757/yetus/whitespace-tabs.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16757/yetus/new-findbugs-ql.html
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16757/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Comparision of varchar column and string literal should happen in varchar
> -
>
> Key: HIVE-21316
> URL: https://issues.apache.org/jira/browse/HIVE-21316
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21316.01.patch, HIVE-21316.02.patch,
> HIVE-21316.03.patch, HIVE-21316.04.patch, HIVE-21316.05.patch,
> HIVE-21316.06.patch, HIVE-21316.06.patch, HIVE-21316.07.patch,
> HIVE-21316.07.patch
>
>
> this is most probably the root cause behind HIVE-21310 as well
--
This message was sent by Atlassian
[jira] [Commented] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805269#comment-16805269
]
Vihang Karajgaonkar commented on HIVE-21484:
Adding a branch-3 patch.
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-21484.01.patch, HIVE-21484.02.patch,
> HIVE-21484.03.patch, HIVE-21484.04.branch-3.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This version number can be used to do a version handshake between client
> and server in the future to improve our cross-version compatibity story.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-21484:
---
Attachment: HIVE-21484.04.branch-3.patch
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-21484.01.patch, HIVE-21484.02.patch,
> HIVE-21484.03.patch, HIVE-21484.04.branch-3.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This version number can be used to do a version handshake between client
> and server in the future to improve our cross-version compatibity story.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21396) TestCliDriver#vector_groupby_reduce is flaky - rounding error
[
https://issues.apache.org/jira/browse/HIVE-21396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805267#comment-16805267
]
Vihang Karajgaonkar commented on HIVE-21396:
don't think my patch has anything do with the test failures since it doesn't
even involve any code changes.
> TestCliDriver#vector_groupby_reduce is flaky - rounding error
> -
>
> Key: HIVE-21396
> URL: https://issues.apache.org/jira/browse/HIVE-21396
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Laszlo Bodor
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21396.01.patch
>
>
> http://104.198.109.242/logs/PreCommit-HIVE-Build-16349/failed/61-TestCliDriver-multi_insert_partitioned.q-parquet_types.q-udf_to_unix_timestamp.q-and-27-more/TEST-61-TestCliDriver-multi_insert_partitioned.q-parquet_types.q-udf_to_unix_timestamp.q-and-27-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
> http://104.198.109.242/logs/PreCommit-HIVE-Build-16351/failed/61-TestCliDriver-multi_insert_partitioned.q-parquet_types.q-udf_to_unix_timestamp.q-and-27-more/TEST-61-TestCliDriver-multi_insert_partitioned.q-parquet_types.q-udf_to_unix_timestamp.q-and-27-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
> -5080.17 --> -5080.1699
> actual:
> {code:java}
> 1 85411 816 58.285714285714285 -5080.1699 -362.86928571428564
> 621.35 44.382142857142857143
> {code}
> expected:
> {code:java}
> 1 85411 816 58.285714285714285 -5080.17 -362.8692857142857
> 621.35 44.382142857142857143
> {code}
> https://github.com/apache/hive/blob/268a6e5af11e0fdc3887d570c1680035fd9426c3/ql/src/test/results/clientpositive/vector_groupby_reduce.q.out
> it's a result of sum (max(ss_net_profit) np)
> {code}
> select
> ss_ticket_number, sum(ss_item_sk), sum(q), avg(q), sum(np), avg(np),
> sum(decwc), avg(decwc)
> from
> (select
> ss_ticket_number, ss_item_sk, min(ss_quantity) q, max(ss_net_profit)
> np, max(ss_wholesale_cost_decimal) decwc
> from
> store_sales_n3
> where ss_ticket_number = 1
> group by ss_ticket_number, ss_item_sk) a
> group by ss_ticket_number
> order by ss_ticket_number
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21180) Fix branch-3 metastore test timeouts
[
https://issues.apache.org/jira/browse/HIVE-21180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar reassigned HIVE-21180:
--
Assignee: (was: Vihang Karajgaonkar)
> Fix branch-3 metastore test timeouts
>
>
> Key: HIVE-21180
> URL: https://issues.apache.org/jira/browse/HIVE-21180
> Project: Hive
> Issue Type: Test
>Affects Versions: 3.2.0
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> The module name below is wrong since metastore-server doesn't exist on
> branch-3. This is most likely the reason why test batches are timing out on
> branch-3
> {noformat}
> 2019-01-29 00:32:17,765 INFO [HostExecutor 3]
> HostExecutor.executeTestBatch:262 Drone [user=hiveptest,
> host=104.198.216.224, instance=0] executing UnitTestBatch
> [name=228_UTBatch_standalone-metastore__metastore-server_20_tests, id=228,
> moduleName=standalone-metastore/metastore-server, batchSize=20,
> isParallel=true, testList=[TestPartitionManagement,
> TestCatalogNonDefaultClient, TestCatalogOldClient, TestHiveAlterHandler,
> TestTxnHandlerNegative, TestTxnUtils, TestFilterHooks, TestRawStoreProxy,
> TestLockRequestBuilder, TestHiveMetastoreCli, TestCheckConstraint,
> TestAddPartitions, TestListPartitions, TestFunctions, TestGetTableMeta,
> TestTablesCreateDropAlterTruncate, TestRuntimeStats, TestDropPartitions,
> TestTablesList, TestUniqueConstraint]] with bash
> /home/hiveptest/104.198.216.224-hiveptest-0/scratch/hiveptest-228_UTBatch_standalone-metastore__metastore-server_20_tests.sh
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21180) Fix branch-3 metastore test timeouts
[
https://issues.apache.org/jira/browse/HIVE-21180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805263#comment-16805263
]
Vihang Karajgaonkar commented on HIVE-21180:
Don't think I will have enough time to work on this.
> Fix branch-3 metastore test timeouts
>
>
> Key: HIVE-21180
> URL: https://issues.apache.org/jira/browse/HIVE-21180
> Project: Hive
> Issue Type: Test
>Affects Versions: 3.2.0
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> The module name below is wrong since metastore-server doesn't exist on
> branch-3. This is most likely the reason why test batches are timing out on
> branch-3
> {noformat}
> 2019-01-29 00:32:17,765 INFO [HostExecutor 3]
> HostExecutor.executeTestBatch:262 Drone [user=hiveptest,
> host=104.198.216.224, instance=0] executing UnitTestBatch
> [name=228_UTBatch_standalone-metastore__metastore-server_20_tests, id=228,
> moduleName=standalone-metastore/metastore-server, batchSize=20,
> isParallel=true, testList=[TestPartitionManagement,
> TestCatalogNonDefaultClient, TestCatalogOldClient, TestHiveAlterHandler,
> TestTxnHandlerNegative, TestTxnUtils, TestFilterHooks, TestRawStoreProxy,
> TestLockRequestBuilder, TestHiveMetastoreCli, TestCheckConstraint,
> TestAddPartitions, TestListPartitions, TestFunctions, TestGetTableMeta,
> TestTablesCreateDropAlterTruncate, TestRuntimeStats, TestDropPartitions,
> TestTablesList, TestUniqueConstraint]] with bash
> /home/hiveptest/104.198.216.224-hiveptest-0/scratch/hiveptest-228_UTBatch_standalone-metastore__metastore-server_20_tests.sh
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805262#comment-16805262
]
Vihang Karajgaonkar commented on HIVE-21484:
Patch merged into master
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-21484.01.patch, HIVE-21484.02.patch,
> HIVE-21484.03.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This version number can be used to do a version handshake between client
> and server in the future to improve our cross-version compatibity story.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21386) Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache
[ https://issues.apache.org/jira/browse/HIVE-21386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805253#comment-16805253 ] Vineet Garg commented on HIVE-21386: Pull request: [https://github.com/apache/hive/pull/582] > Extend the fetch task enhancement done in HIVE-21279 to make it work with > query result cache > > > Key: HIVE-21386 > URL: https://issues.apache.org/jira/browse/HIVE-21386 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21386.1.patch, HIVE-21386.2.patch, > HIVE-21386.3.patch, HIVE-21386.4.patch > > > The improvement done in HIVE-21279 is disabled for query cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21279) Avoid moving/rename operation in FileSink op for SELECT queries
[ https://issues.apache.org/jira/browse/HIVE-21279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21279: -- Labels: pull-request-available (was: ) > Avoid moving/rename operation in FileSink op for SELECT queries > --- > > Key: HIVE-21279 > URL: https://issues.apache.org/jira/browse/HIVE-21279 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21279.1.patch, HIVE-21279.10.patch, > HIVE-21279.11.patch, HIVE-21279.12.patch, HIVE-21279.13.patch, > HIVE-21279.2.patch, HIVE-21279.3.patch, HIVE-21279.4.patch, > HIVE-21279.5.patch, HIVE-21279.6.patch, HIVE-21279.7.patch, > HIVE-21279.8.patch, HIVE-21279.9.patch > > > Currently at the end of a job FileSink operator moves/rename temp directory > to another directory from which FetchTask fetches result. This is done to > avoid fetching potential partial/invalid files by failed/runway tasks. This > operation is expensive for cloud storage. It could be avoided if FetchTask is > passed on set of files to read from instead of whole directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21279) Avoid moving/rename operation in FileSink op for SELECT queries
[ https://issues.apache.org/jira/browse/HIVE-21279?focusedWorklogId=220632=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-220632 ] ASF GitHub Bot logged work on HIVE-21279: - Author: ASF GitHub Bot Created on: 29/Mar/19 17:47 Start Date: 29/Mar/19 17:47 Worklog Time Spent: 10m Work Description: vineetgarg02 commented on pull request #582: Hive 21386: Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache URL: https://github.com/apache/hive/pull/582 This patch brings the following changes: * If result cache is enable for a query (select), results will be written to result cache dir instead of tmp scratch dir. * Query result cache will then cache the set of files if it can for the query, instead of moving and caching the result directory. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 220632) Time Spent: 10m Remaining Estimate: 0h > Avoid moving/rename operation in FileSink op for SELECT queries > --- > > Key: HIVE-21279 > URL: https://issues.apache.org/jira/browse/HIVE-21279 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21279.1.patch, HIVE-21279.10.patch, > HIVE-21279.11.patch, HIVE-21279.12.patch, HIVE-21279.13.patch, > HIVE-21279.2.patch, HIVE-21279.3.patch, HIVE-21279.4.patch, > HIVE-21279.5.patch, HIVE-21279.6.patch, HIVE-21279.7.patch, > HIVE-21279.8.patch, HIVE-21279.9.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently at the end of a job FileSink operator moves/rename temp directory > to another directory from which FetchTask fetches result. This is done to > avoid fetching potential partial/invalid files by failed/runway tasks. This > operation is expensive for cloud storage. It could be avoided if FetchTask is > passed on set of files to read from instead of whole directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21443) Better usability for SHOW COMPACTIONS
[ https://issues.apache.org/jira/browse/HIVE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805248#comment-16805248 ] Vineet Garg commented on HIVE-21443: No problem. I reverted the commit because it was blocking other patches (to get green run). I think it is better to fix all the tests and commit it once so that other patches don't push separate test failure inadvertently (like in pre-historic times of 3.0 development). > Better usability for SHOW COMPACTIONS > - > > Key: HIVE-21443 > URL: https://issues.apache.org/jira/browse/HIVE-21443 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Todd Lipcon >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21443.addendum.patch, HIVE-21443.patch > > > Currently on a test cluster the output of 'SHOW COMPACTIONS' has 117k rows. > This makes it basically useless to work with. > For better usability, we should support syntax like 'SHOW COMPACTIONS IN > ' or maybe 'SHOW COMPACTIONS ON ' (particular syntax to be > chosen for consistency with other operations I suppose). > Alternatively (or maybe in addition) it seems like it would be nice to expose > the same data in a queryable table (eg in information_schema or a system > namespace) so that I could do things like: SELECT dbname, state, count(*) > from compactions group by 1,2; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21443) Better usability for SHOW COMPACTIONS
[ https://issues.apache.org/jira/browse/HIVE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805245#comment-16805245 ] Peter Vary commented on HIVE-21443: --- Thanks [~vgarg]! Couldn't revert myself (fetching the kids). I always wait for the green run, but this time I made a mistake (juggling too many things in parallel). Sorry for the problems caused. :( Peter > Better usability for SHOW COMPACTIONS > - > > Key: HIVE-21443 > URL: https://issues.apache.org/jira/browse/HIVE-21443 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Todd Lipcon >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21443.addendum.patch, HIVE-21443.patch > > > Currently on a test cluster the output of 'SHOW COMPACTIONS' has 117k rows. > This makes it basically useless to work with. > For better usability, we should support syntax like 'SHOW COMPACTIONS IN > ' or maybe 'SHOW COMPACTIONS ON ' (particular syntax to be > chosen for consistency with other operations I suppose). > Alternatively (or maybe in addition) it seems like it would be nice to expose > the same data in a queryable table (eg in information_schema or a system > namespace) so that I could do things like: SELECT dbname, state, count(*) > from compactions group by 1,2; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21443) Better usability for SHOW COMPACTIONS
[ https://issues.apache.org/jira/browse/HIVE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805236#comment-16805236 ] Gopal V commented on HIVE-21443: the addendum isn't committed, because of sysdb.q result change (+6,-3 in there). > Better usability for SHOW COMPACTIONS > - > > Key: HIVE-21443 > URL: https://issues.apache.org/jira/browse/HIVE-21443 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Todd Lipcon >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21443.addendum.patch, HIVE-21443.patch > > > Currently on a test cluster the output of 'SHOW COMPACTIONS' has 117k rows. > This makes it basically useless to work with. > For better usability, we should support syntax like 'SHOW COMPACTIONS IN > ' or maybe 'SHOW COMPACTIONS ON ' (particular syntax to be > chosen for consistency with other operations I suppose). > Alternatively (or maybe in addition) it seems like it would be nice to expose > the same data in a queryable table (eg in information_schema or a system > namespace) so that I could do things like: SELECT dbname, state, count(*) > from compactions group by 1,2; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HIVE-21443) Better usability for SHOW COMPACTIONS
[ https://issues.apache.org/jira/browse/HIVE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reopened HIVE-21443: Reverted the commit since it is causing test failures. Please make sure to get clean run (green) before pushing to master. > Better usability for SHOW COMPACTIONS > - > > Key: HIVE-21443 > URL: https://issues.apache.org/jira/browse/HIVE-21443 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Todd Lipcon >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21443.addendum.patch, HIVE-21443.patch > > > Currently on a test cluster the output of 'SHOW COMPACTIONS' has 117k rows. > This makes it basically useless to work with. > For better usability, we should support syntax like 'SHOW COMPACTIONS IN > ' or maybe 'SHOW COMPACTIONS ON ' (particular syntax to be > chosen for consistency with other operations I suppose). > Alternatively (or maybe in addition) it seems like it would be nice to expose > the same data in a queryable table (eg in information_schema or a system > namespace) so that I could do things like: SELECT dbname, state, count(*) > from compactions group by 1,2; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21404) MSSQL upgrade script alters the wrong column
[
https://issues.apache.org/jira/browse/HIVE-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805229#comment-16805229
]
Hive QA commented on HIVE-21404:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964143/HIVE-21404.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15875 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables_sysdb]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=169)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16756/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16756/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16756/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12964143 - PreCommit-HIVE-Build
> MSSQL upgrade script alters the wrong column
>
>
> Key: HIVE-21404
> URL: https://issues.apache.org/jira/browse/HIVE-21404
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.2.0
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-21404.1.patch, HIVE-21404.2.patch,
> HIVE-21404.3.patch, HIVE-21404.4.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HIVE-20221 changes PARTITION_PARAMS, so the following command is modifying
> the wrong table:
> {{ALTER TABLE "SERDE_PARAMS" ALTER COLUMN "PARAM_VALUE" nvarchar(MAX);}}
> https://github.com/apache/hive/blob/d3b036920acde7bb04840697eb13038103b062b4/standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.1.0-to-3.2.0.mssql.sql#L21
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21539) GroupBy + where clause on same column results in incorrect query rewrite
[
https://issues.apache.org/jira/browse/HIVE-21539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg reassigned HIVE-21539:
--
Assignee: Vineet Garg
> GroupBy + where clause on same column results in incorrect query rewrite
>
>
> Key: HIVE-21539
> URL: https://issues.apache.org/jira/browse/HIVE-21539
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0
>Reporter: anishek
>Assignee: Vineet Garg
>Priority: Major
>
> {code}
> create table a (i int, j string);
> insert into a values ( 1, 'a'),(2,'b');
> explain extended select min(j) from a where j='a' group by j;
> ++
> | Explain |
> ++
> | OPTIMIZED SQL: SELECT MIN(TRUE) AS `_o__c0`|
> | FROM `default`.`a` |
> | WHERE `j` = 'a'|
> | GROUP BY TRUE |
> | STAGE DEPENDENCIES:|
> | Stage-1 is a root stage |
> | Stage-0 depends on stages: Stage-1 |
> ||
> | STAGE PLANS: |
> | Stage: Stage-1 |
> | Tez|
> | DagId:
> anagarwal_20190318153535_25c1f460-1986-475e-9995-9f6342029dd8:11 |
> | Edges: |
> | Reducer 2 <- Map 1 (SIMPLE_EDGE) |
> | DagName:
> anagarwal_20190318153535_25c1f460-1986-475e-9995-9f6342029dd8:11 |
> | Vertices:|
> | Map 1 |
> | Map Operator Tree: |
> | TableScan |
> | alias: a |
> | filterExpr: (j = 'a') (type: boolean) |
> | Statistics: Num rows: 2 Data size: 170 Basic stats:
> COMPLETE Column stats: COMPLETE |
> | GatherStats: false |
> | Filter Operator |
> | isSamplingPred: false |
> | predicate: (j = 'a') (type: boolean) |
> | Statistics: Num rows: 1 Data size: 85 Basic stats:
> COMPLETE Column stats: COMPLETE |
> | Select Operator|
> | Statistics: Num rows: 1 Data size: 85 Basic stats:
> COMPLETE Column stats: COMPLETE |
> | Group By Operator|
> | aggregations: min(true)|
> | keys: true (type: boolean) |
> | mode: hash |
> | outputColumnNames: _col0, _col1 |
> | Statistics: Num rows: 1 Data size: 8 Basic stats:
> COMPLETE Column stats: COMPLETE |
> | Reduce Output Operator |
> | key expressions: _col0 (type: boolean) |
> | null sort order: a |
> | sort order: +|
> | Map-reduce partition columns: _col0 (type:
> boolean) |
> | Statistics: Num rows: 1 Data size: 8 Basic stats:
> COMPLETE Column stats: COMPLETE |
> | tag: -1 |
> | value expressions: _col1 (type: boolean) |
> | auto parallelism: true |
> | Path -> Alias: |
> | hdfs://localhost:9000/tmp/hive/warehouse/a [a] |
> | Path -> Partition: |
> | hdfs://localhost:9000/tmp/hive/warehouse/a |
> | Partition |
> | base file name: a|
> | input format: org.apache.hadoop.mapred.TextInputFormat |
> | output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat |
> | properties: |
> | COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"i":"true","j":"true"}} |
> | bucket_count -1|
> | bucketing_version 2|
> | column.name.delimiter ,|
> | columns i,j|
> | columns.comments |
> | columns.types int:string |
>
[jira] [Commented] (HIVE-21443) Better usability for SHOW COMPACTIONS
[ https://issues.apache.org/jira/browse/HIVE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805210#comment-16805210 ] Peter Vary commented on HIVE-21443: --- The id of the compaction is changed so some other compaction is initiated on the driver. It will cause flakiness depending on the batch. Will fix on Monday. > Better usability for SHOW COMPACTIONS > - > > Key: HIVE-21443 > URL: https://issues.apache.org/jira/browse/HIVE-21443 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Todd Lipcon >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21443.addendum.patch, HIVE-21443.patch > > > Currently on a test cluster the output of 'SHOW COMPACTIONS' has 117k rows. > This makes it basically useless to work with. > For better usability, we should support syntax like 'SHOW COMPACTIONS IN > ' or maybe 'SHOW COMPACTIONS ON ' (particular syntax to be > chosen for consistency with other operations I suppose). > Alternatively (or maybe in addition) it seems like it would be nice to expose > the same data in a queryable table (eg in information_schema or a system > namespace) so that I could do things like: SELECT dbname, state, count(*) > from compactions group by 1,2; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21443) Better usability for SHOW COMPACTIONS
[ https://issues.apache.org/jira/browse/HIVE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-21443: -- Assignee: Peter Vary (was: Gopal V) > Better usability for SHOW COMPACTIONS > - > > Key: HIVE-21443 > URL: https://issues.apache.org/jira/browse/HIVE-21443 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Todd Lipcon >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21443.addendum.patch, HIVE-21443.patch > > > Currently on a test cluster the output of 'SHOW COMPACTIONS' has 117k rows. > This makes it basically useless to work with. > For better usability, we should support syntax like 'SHOW COMPACTIONS IN > ' or maybe 'SHOW COMPACTIONS ON ' (particular syntax to be > chosen for consistency with other operations I suppose). > Alternatively (or maybe in addition) it seems like it would be nice to expose > the same data in a queryable table (eg in information_schema or a system > namespace) so that I could do things like: SELECT dbname, state, count(*) > from compactions group by 1,2; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21443) Better usability for SHOW COMPACTIONS
[ https://issues.apache.org/jira/browse/HIVE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-21443: --- Attachment: HIVE-21443.addendum.patch > Better usability for SHOW COMPACTIONS > - > > Key: HIVE-21443 > URL: https://issues.apache.org/jira/browse/HIVE-21443 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Todd Lipcon >Assignee: Gopal V >Priority: Major > Attachments: HIVE-21443.addendum.patch, HIVE-21443.patch > > > Currently on a test cluster the output of 'SHOW COMPACTIONS' has 117k rows. > This makes it basically useless to work with. > For better usability, we should support syntax like 'SHOW COMPACTIONS IN > ' or maybe 'SHOW COMPACTIONS ON ' (particular syntax to be > chosen for consistency with other operations I suppose). > Alternatively (or maybe in addition) it seems like it would be nice to expose > the same data in a queryable table (eg in information_schema or a system > namespace) so that I could do things like: SELECT dbname, state, count(*) > from compactions group by 1,2; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21443) Better usability for SHOW COMPACTIONS
[ https://issues.apache.org/jira/browse/HIVE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-21443: -- Assignee: Gopal V (was: Peter Vary) > Better usability for SHOW COMPACTIONS > - > > Key: HIVE-21443 > URL: https://issues.apache.org/jira/browse/HIVE-21443 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Todd Lipcon >Assignee: Gopal V >Priority: Major > Attachments: HIVE-21443.addendum.patch, HIVE-21443.patch > > > Currently on a test cluster the output of 'SHOW COMPACTIONS' has 117k rows. > This makes it basically useless to work with. > For better usability, we should support syntax like 'SHOW COMPACTIONS IN > ' or maybe 'SHOW COMPACTIONS ON ' (particular syntax to be > chosen for consistency with other operations I suppose). > Alternatively (or maybe in addition) it seems like it would be nice to expose > the same data in a queryable table (eg in information_schema or a system > namespace) so that I could do things like: SELECT dbname, state, count(*) > from compactions group by 1,2; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21443) Better usability for SHOW COMPACTIONS
[ https://issues.apache.org/jira/browse/HIVE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805201#comment-16805201 ] Peter Vary commented on HIVE-21443: --- Revert please. Something else left some compaction there. Will be flaky. [~kgyrtkirk]: mixed up with another jira :( > Better usability for SHOW COMPACTIONS > - > > Key: HIVE-21443 > URL: https://issues.apache.org/jira/browse/HIVE-21443 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Todd Lipcon >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21443.patch > > > Currently on a test cluster the output of 'SHOW COMPACTIONS' has 117k rows. > This makes it basically useless to work with. > For better usability, we should support syntax like 'SHOW COMPACTIONS IN > ' or maybe 'SHOW COMPACTIONS ON ' (particular syntax to be > chosen for consistency with other operations I suppose). > Alternatively (or maybe in addition) it seems like it would be nice to expose > the same data in a queryable table (eg in information_schema or a system > namespace) so that I could do things like: SELECT dbname, state, count(*) > from compactions group by 1,2; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21544) Constant propagation corrupts coalesce/case/when expressions during folding
[
https://issues.apache.org/jira/browse/HIVE-21544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805194#comment-16805194
]
Jesus Camacho Rodriguez commented on HIVE-21544:
+1
> Constant propagation corrupts coalesce/case/when expressions during folding
> ---
>
> Key: HIVE-21544
> URL: https://issues.apache.org/jira/browse/HIVE-21544
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21544.01.patch
>
>
> {code}
> set hive.fetch.task.conversion=none;
> set hive.optimize.ppd=false;
> create table t (s1 string,s2 string);
> insert into t values (null,null);
> explain
> select
> coalesce(s1, 'null_value' ), coalesce(s2, 'null_value' ),
> coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' ),
> case when coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' )
> then 'eq' else 'noteq' end
> from t;
> select
> coalesce(s1, 'null_value' ), coalesce(s2, 'null_value' ),
> coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' ),
> case when coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' )
> then 'eq' else 'noteq' end
> from t;
> {code}
> incorrect result is:
> {code}
> null_value null_value NULLnoteq
> {code}
> expected result:
> {code}
> null_value null_value trueeq
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21443) Better usability for SHOW COMPACTIONS
[
https://issues.apache.org/jira/browse/HIVE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805191#comment-16805191
]
Gopal V commented on HIVE-21443:
[~pvary]: the failed tests are related.
{code}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
Failing for the past 1 build (Since Failed#16745 )
Took 59 sec.
Error Message
Client Execution succeeded but contained differences (error code = 1) after
executing sysdb.q
4810,4811c4810,4811
< 4 default default scr_txn NULL major initiated NULL NULL NULL NULL NULL NULL
NULL
< 5 default default scr_txn_2 NULL minor initiated NULL NULL NULL NULL NULL
NULL NULL
---
> 1 default default scr_txn NULL major initiated NULL NULL NULL NULL NULL NULL
> NULL
> 2 default default scr_txn_2 NULL minor initiated NULL NULL NULL NULL NULL
> NULL NULL
5047,5048c5047,5048
< 4 default default scr_txn NULL major initiated NULL NULL NULL NULL NULL NULL
NULL
< 5 default default scr_txn_2 NULL minor initiated NULL NULL NULL NULL NULL
NULL NULL
---
> 1 default default scr_txn NULL major initiated NULL NULL NULL NULL NULL NULL
> NULL
> 2 default default scr_txn_2 NULL minor initiated NULL NULL NULL NULL NULL
> NULL NULL
{code}
[~kgyrtkirk]: I'll check in an addendum.
> Better usability for SHOW COMPACTIONS
> -
>
> Key: HIVE-21443
> URL: https://issues.apache.org/jira/browse/HIVE-21443
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Todd Lipcon
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21443.patch
>
>
> Currently on a test cluster the output of 'SHOW COMPACTIONS' has 117k rows.
> This makes it basically useless to work with.
> For better usability, we should support syntax like 'SHOW COMPACTIONS IN
> ' or maybe 'SHOW COMPACTIONS ON ' (particular syntax to be
> chosen for consistency with other operations I suppose).
> Alternatively (or maybe in addition) it seems like it would be nice to expose
> the same data in a queryable table (eg in information_schema or a system
> namespace) so that I could do things like: SELECT dbname, state, count(*)
> from compactions group by 1,2;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21404) MSSQL upgrade script alters the wrong column
[
https://issues.apache.org/jira/browse/HIVE-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805190#comment-16805190
]
Hive QA commented on HIVE-21404:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 20 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16756/dev-support/hive-personality.sh
|
| git revision | master / ce0d3a6 |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16756/yetus/whitespace-tabs.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16756/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MSSQL upgrade script alters the wrong column
>
>
> Key: HIVE-21404
> URL: https://issues.apache.org/jira/browse/HIVE-21404
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.2.0
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-21404.1.patch, HIVE-21404.2.patch,
> HIVE-21404.3.patch, HIVE-21404.4.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HIVE-20221 changes PARTITION_PARAMS, so the following command is modifying
> the wrong table:
> {{ALTER TABLE "SERDE_PARAMS" ALTER COLUMN "PARAM_VALUE" nvarchar(MAX);}}
> https://github.com/apache/hive/blob/d3b036920acde7bb04840697eb13038103b062b4/standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.1.0-to-3.2.0.mssql.sql#L21
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21443) Better usability for SHOW COMPACTIONS
[ https://issues.apache.org/jira/browse/HIVE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805189#comment-16805189 ] Zoltan Haindrich commented on HIVE-21443: - where is the clean testrun??? > Better usability for SHOW COMPACTIONS > - > > Key: HIVE-21443 > URL: https://issues.apache.org/jira/browse/HIVE-21443 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Todd Lipcon >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21443.patch > > > Currently on a test cluster the output of 'SHOW COMPACTIONS' has 117k rows. > This makes it basically useless to work with. > For better usability, we should support syntax like 'SHOW COMPACTIONS IN > ' or maybe 'SHOW COMPACTIONS ON ' (particular syntax to be > chosen for consistency with other operations I suppose). > Alternatively (or maybe in addition) it seems like it would be nice to expose > the same data in a queryable table (eg in information_schema or a system > namespace) so that I could do things like: SELECT dbname, state, count(*) > from compactions group by 1,2; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.19
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805186#comment-16805186
]
Hive QA commented on HIVE-21001:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
41s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
50s{color} | {color:red} ql: The patch generated 7 new + 342 unchanged - 45
fixed = 349 total (was 387) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
21s{color} | {color:red} root: The patch generated 7 new + 351 unchanged - 45
fixed = 358 total (was 396) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 5m
22s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
20s{color} | {color:red} patch/accumulo-handler cannot run setBugDatabaseInfo
from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
10s{color} | {color:red} patch/hbase-handler cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 19m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16755/dev-support/hive-personality.sh
|
| git revision | master / ce0d3a6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16755/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16755/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16755/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16755/yetus/patch-findbugs-ql.txt
|
| findbugs |
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.19
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.49.patch > Upgrade to calcite-1.19 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch, > HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch, > HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch, > HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch, > HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch, > HIVE-21001.22.patch, HIVE-21001.23.patch, HIVE-21001.24.patch, > HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.26.patch, > HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.27.patch, > HIVE-21001.28.patch, HIVE-21001.29.patch, HIVE-21001.29.patch, > HIVE-21001.30.patch, HIVE-21001.31.patch, HIVE-21001.32.patch, > HIVE-21001.34.patch, HIVE-21001.35.patch, HIVE-21001.36.patch, > HIVE-21001.37.patch, HIVE-21001.38.patch, HIVE-21001.39.patch, > HIVE-21001.40.patch, HIVE-21001.41.patch, HIVE-21001.42.patch, > HIVE-21001.43.patch, HIVE-21001.44.patch, HIVE-21001.45.patch, > HIVE-21001.45.patch, HIVE-21001.46.patch, HIVE-21001.47.patch, > HIVE-21001.48.patch, HIVE-21001.48.patch, HIVE-21001.48.patch, > HIVE-21001.48.patch, HIVE-21001.49.patch, HIVE-21001.49.patch > > > XLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21517) Fix AggregateStatsCache
[ https://issues.apache.org/jira/browse/HIVE-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805176#comment-16805176 ] Miklos Gergely commented on HIVE-21517: --- [~vgumashta], could you please take a look, and merge if you agree? It's a really small patch :) > Fix AggregateStatsCache > --- > > Key: HIVE-21517 > URL: https://issues.apache.org/jira/browse/HIVE-21517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21517.01.patch > > > Due to a bug AggregateStatsCache is not returning the best matching result. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=220593=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-220593
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 29/Mar/19 16:12
Start Date: 29/Mar/19 16:12
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r270479016
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -987,10 +989,14 @@ public void createTable(Table tbl, boolean ifNotExists,
tTbl.setPrivileges(principalPrivs);
}
}
- // Set table snapshot to api.Table to make it persistent.
- TableSnapshot tableSnapshot = AcidUtils.getTableSnapshot(conf, tbl,
true);
- if (tableSnapshot != null) {
-tbl.getTTable().setWriteId(tableSnapshot.getWriteId());
+ // Set table snapshot to api.Table to make it persistent. A
transactional table being
+ // replicated may have a valid write Id copied from the source. Use that
instead of
+ // crafting one on the replica.
+ if (tTbl.getWriteId() <= 0) {
Review comment:
You are right. We do not need it at the creation time. We already have tests
for that and they are working fine i.e. the expected stats both the table level
and column level is getting replicated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 220593)
Time Spent: 7.5h (was: 7h 20m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch, HIVE-21109.07.patch, HIVE-21109.08.patch
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=220592=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-220592
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 29/Mar/19 16:10
Start Date: 29/Mar/19 16:10
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r270478318
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestStatsReplicationScenarios.java
##
@@ -359,17 +383,20 @@ private void testStatsReplicationCommon(boolean
parallelBootstrap, boolean metad
}
@Test
- public void testForNonAcidTables() throws Throwable {
+ public void testNonParallelBootstrapLoad() throws Throwable {
+LOG.info("Testing " + testName.getClass().getName() + "." +
testName.getMethodName());
testStatsReplicationCommon(false, false);
}
@Test
- public void testForNonAcidTablesParallelBootstrapLoad() throws Throwable {
-testStatsReplicationCommon(true, false);
+ public void testForParallelBootstrapLoad() throws Throwable {
+LOG.info("Testing " + testName.getClass().getName() + "." +
testName.getMethodName());
+testStatsReplicationCommon(true, false );
}
@Test
- public void testNonAcidMetadataOnlyDump() throws Throwable {
+ public void testMetadataOnlyDump() throws Throwable {
Review comment:
Added test for the first case. For second case, the events for parallel
inserts will be serialized and applied serially on repl side. So this should be
a problem on repl. We may test whether the events are generated in serialized
fashion and have same expected contents. But that should be done a test which
tests concurrent inserts (may be we already have it somewhere) and not in a
replication test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 220592)
Time Spent: 7h 20m (was: 7h 10m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch, HIVE-21109.07.patch, HIVE-21109.08.patch
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.19
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805160#comment-16805160
]
Hive QA commented on HIVE-21001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964140/HIVE-21001.49.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15876 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables_sysdb]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=169)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.org.apache.hadoop.hive.ql.TestWarehouseExternalDir
(batchId=243)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testExternalDefaultPaths
(batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16755/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16755/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16755/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12964140 - PreCommit-HIVE-Build
> Upgrade to calcite-1.19
> ---
>
> Key: HIVE-21001
> URL: https://issues.apache.org/jira/browse/HIVE-21001
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch,
> HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch,
> HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch,
> HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch,
> HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch,
> HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch,
> HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch,
> HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch,
> HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch,
> HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch,
> HIVE-21001.22.patch, HIVE-21001.23.patch, HIVE-21001.24.patch,
> HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.26.patch,
> HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.27.patch,
> HIVE-21001.28.patch, HIVE-21001.29.patch, HIVE-21001.29.patch,
> HIVE-21001.30.patch, HIVE-21001.31.patch, HIVE-21001.32.patch,
> HIVE-21001.34.patch, HIVE-21001.35.patch, HIVE-21001.36.patch,
> HIVE-21001.37.patch, HIVE-21001.38.patch, HIVE-21001.39.patch,
> HIVE-21001.40.patch, HIVE-21001.41.patch, HIVE-21001.42.patch,
> HIVE-21001.43.patch, HIVE-21001.44.patch, HIVE-21001.45.patch,
> HIVE-21001.45.patch, HIVE-21001.46.patch, HIVE-21001.47.patch,
> HIVE-21001.48.patch, HIVE-21001.48.patch, HIVE-21001.48.patch,
> HIVE-21001.48.patch, HIVE-21001.49.patch
>
>
> XLEAR LIBRARY CACHE
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=220591=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-220591
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 29/Mar/19 16:08
Start Date: 29/Mar/19 16:08
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r270477619
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -2950,21 +2956,33 @@ public Partition createPartition(Table tbl,
Map partSpec) throws
int size = addPartitionDesc.getPartitionCount();
List in =
new ArrayList(size);
-AcidUtils.TableSnapshot tableSnapshot = AcidUtils.getTableSnapshot(conf,
tbl, true);
long writeId;
String validWriteIdList;
-if (tableSnapshot != null && tableSnapshot.getWriteId() > 0) {
- writeId = tableSnapshot.getWriteId();
- validWriteIdList = tableSnapshot.getValidWriteIdList();
+
+// In case of replication, get the writeId from the source and use valid
write Id list
+// for replication.
+if (addPartitionDesc.getReplicationSpec() != null &&
+addPartitionDesc.getReplicationSpec().isInReplicationScope() &&
+addPartitionDesc.getPartition(0).getWriteId() > 0) {
+ writeId = addPartitionDesc.getPartition(0).getWriteId();
+ validWriteIdList =
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 220591)
Time Spent: 7h 10m (was: 7h)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch, HIVE-21109.07.patch, HIVE-21109.08.patch
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=220590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-220590
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 29/Mar/19 16:08
Start Date: 29/Mar/19 16:08
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r270477567
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnCommonUtils.java
##
@@ -84,6 +86,73 @@ public static ValidTxnList
createValidReadTxnList(GetOpenTxnsResponse txns, long
return new ValidReadTxnList(exceptions, outAbortedBits, highWaterMark,
minOpenTxnId);
}
+ /**
+ * Transform a {@link
org.apache.hadoop.hive.metastore.api.GetOpenTxnsResponse} to a
+ * {@link org.apache.hadoop.hive.common.ValidTxnList}. This assumes that
the caller intends to
+ * read the files, and thus treats both open and aborted transactions as
invalid.
+ *
+ * This API is used by Hive replication which may have multiple transactions
open at a time.
+ *
+ * @param txns open txn list from the metastore
+ * @param currentTxns Current transactions that the replication has opened.
If any of the
+ *transactions is greater than 0 it will be removed from
the exceptions
+ *list so that the replication sees its own transaction
as valid.
+ * @return a valid txn list.
+ */
+ public static ValidTxnList createValidReadTxnList(GetOpenTxnsResponse txns,
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 220590)
Time Spent: 7h (was: 6h 50m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch, HIVE-21109.07.patch, HIVE-21109.08.patch
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21109) Stats replication for ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805159#comment-16805159 ] Ashutosh Bapat commented on HIVE-21109: --- [~sankarh] I have updated PR with commits addressing your review comments. Please review. > Stats replication for ACID tables. > -- > > Key: HIVE-21109 > URL: https://issues.apache.org/jira/browse/HIVE-21109 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch, > HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch, > HIVE-21109.06.patch, HIVE-21109.07.patch, HIVE-21109.08.patch > > Time Spent: 6h 50m > Remaining Estimate: 0h > > Transactional tables require a writeID associated with the stats update. This > writeId needs to be in sync with the writeId on the source and hence needs to > be replicated from the source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21109) Stats replication for ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21109: -- Attachment: HIVE-21109.08.patch Status: Patch Available (was: In Progress) Last ptest run showed 4 failures out of which 3 were from past 4 builds. Last one didn't reproduce locally. Attaching rebased patch to trigger ptests. > Stats replication for ACID tables. > -- > > Key: HIVE-21109 > URL: https://issues.apache.org/jira/browse/HIVE-21109 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch, > HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch, > HIVE-21109.06.patch, HIVE-21109.07.patch, HIVE-21109.08.patch > > Time Spent: 6h 50m > Remaining Estimate: 0h > > Transactional tables require a writeID associated with the stats update. This > writeId needs to be in sync with the writeId on the source and hence needs to > be replicated from the source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21109) Stats replication for ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21109: -- Status: In Progress (was: Patch Available) > Stats replication for ACID tables. > -- > > Key: HIVE-21109 > URL: https://issues.apache.org/jira/browse/HIVE-21109 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch, > HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch, > HIVE-21109.06.patch, HIVE-21109.07.patch > > Time Spent: 6h 50m > Remaining Estimate: 0h > > Transactional tables require a writeID associated with the stats update. This > writeId needs to be in sync with the writeId on the source and hence needs to > be replicated from the source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16003) Blobstores should use fs.listFiles(path, recursive=true) rather than FileUtils.listStatusRecursively
[
https://issues.apache.org/jira/browse/HIVE-16003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805131#comment-16805131
]
Andrew Sherman commented on HIVE-16003:
---
I set assignee to unassigned as [~janulatha] is not working on this.
> Blobstores should use fs.listFiles(path, recursive=true) rather than
> FileUtils.listStatusRecursively
>
>
> Key: HIVE-16003
> URL: https://issues.apache.org/jira/browse/HIVE-16003
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Priority: Major
>
> {{FileUtils.listStatusRecursively}} can be slow on blobstores because
> {{listStatus}} calls are applied recursively to a given directory. This can
> be especially bad on tables with multiple levels of partitioning.
> The {{FileSystem}} API provides an optimized API called {{listFiles(path,
> recursive)}} that can be used to invoke an optimized recursive directory
> listing.
> The problem is that the {{listFiles(path, recursive)}} API doesn't provide a
> option to pass in a {{PathFilter}}, while {{FileUtils.listStatusRecursively}}
> uses a custom HIDDEN_FILES_PATH_FILTER.
> To fix this we could either:
> 1: Modify the FileSystem API to provide a {{listFiles(path, recursive,
> PathFilter)}} method (probably the cleanest solution)
> 2: Add conditional logic so that blobstores invoke {{listFiles(path,
> recursive)}} and the rest of the code uses the current implementation of
> {{FileUtils.listStatusRecursively}}
> 3: Replace the implementation of {{FileUtils.listStatusRecursively}} with
> {{listFiles(path, recursive)}} and apply the {{PathFilter}} on the results
> (not sure what optimizations can be made if {{PathFilter}} objects are passed
> into {{FileSystem}} methods - maybe {{PathFilter}} objects are pushed to the
> NameNode?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-16003) Blobstores should use fs.listFiles(path, recursive=true) rather than FileUtils.listStatusRecursively
[
https://issues.apache.org/jira/browse/HIVE-16003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-16003:
-
Assignee: (was: Janaki Lahorani)
> Blobstores should use fs.listFiles(path, recursive=true) rather than
> FileUtils.listStatusRecursively
>
>
> Key: HIVE-16003
> URL: https://issues.apache.org/jira/browse/HIVE-16003
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Priority: Major
>
> {{FileUtils.listStatusRecursively}} can be slow on blobstores because
> {{listStatus}} calls are applied recursively to a given directory. This can
> be especially bad on tables with multiple levels of partitioning.
> The {{FileSystem}} API provides an optimized API called {{listFiles(path,
> recursive)}} that can be used to invoke an optimized recursive directory
> listing.
> The problem is that the {{listFiles(path, recursive)}} API doesn't provide a
> option to pass in a {{PathFilter}}, while {{FileUtils.listStatusRecursively}}
> uses a custom HIDDEN_FILES_PATH_FILTER.
> To fix this we could either:
> 1: Modify the FileSystem API to provide a {{listFiles(path, recursive,
> PathFilter)}} method (probably the cleanest solution)
> 2: Add conditional logic so that blobstores invoke {{listFiles(path,
> recursive)}} and the rest of the code uses the current implementation of
> {{FileUtils.listStatusRecursively}}
> 3: Replace the implementation of {{FileUtils.listStatusRecursively}} with
> {{listFiles(path, recursive)}} and apply the {{PathFilter}} on the results
> (not sure what optimizations can be made if {{PathFilter}} objects are passed
> into {{FileSystem}} methods - maybe {{PathFilter}} objects are pushed to the
> NameNode?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21544) Constant propagation corrupts coalesce/case/when expressions during folding
[
https://issues.apache.org/jira/browse/HIVE-21544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-21544:
Status: Patch Available (was: Open)
[~ashutoshc], [~jcamachorodriguez] Could you please take a look?
> Constant propagation corrupts coalesce/case/when expressions during folding
> ---
>
> Key: HIVE-21544
> URL: https://issues.apache.org/jira/browse/HIVE-21544
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21544.01.patch
>
>
> {code}
> set hive.fetch.task.conversion=none;
> set hive.optimize.ppd=false;
> create table t (s1 string,s2 string);
> insert into t values (null,null);
> explain
> select
> coalesce(s1, 'null_value' ), coalesce(s2, 'null_value' ),
> coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' ),
> case when coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' )
> then 'eq' else 'noteq' end
> from t;
> select
> coalesce(s1, 'null_value' ), coalesce(s2, 'null_value' ),
> coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' ),
> case when coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' )
> then 'eq' else 'noteq' end
> from t;
> {code}
> incorrect result is:
> {code}
> null_value null_value NULLnoteq
> {code}
> expected result:
> {code}
> null_value null_value trueeq
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21544) Constant propagation corrupts coalesce/case/when expressions during folding
[
https://issues.apache.org/jira/browse/HIVE-21544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-21544:
Attachment: HIVE-21544.01.patch
> Constant propagation corrupts coalesce/case/when expressions during folding
> ---
>
> Key: HIVE-21544
> URL: https://issues.apache.org/jira/browse/HIVE-21544
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21544.01.patch
>
>
> {code}
> set hive.fetch.task.conversion=none;
> set hive.optimize.ppd=false;
> create table t (s1 string,s2 string);
> insert into t values (null,null);
> explain
> select
> coalesce(s1, 'null_value' ), coalesce(s2, 'null_value' ),
> coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' ),
> case when coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' )
> then 'eq' else 'noteq' end
> from t;
> select
> coalesce(s1, 'null_value' ), coalesce(s2, 'null_value' ),
> coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' ),
> case when coalesce(s1, 'null_value' )=coalesce(s2, 'null_value' )
> then 'eq' else 'noteq' end
> from t;
> {code}
> incorrect result is:
> {code}
> null_value null_value NULLnoteq
> {code}
> expected result:
> {code}
> null_value null_value trueeq
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21342) Analyze compute stats for column leave behind staging dir on hdfs
[ https://issues.apache.org/jira/browse/HIVE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-21342: -- Attachment: (was: HIVE-21499.01.patch) > Analyze compute stats for column leave behind staging dir on hdfs > - > > Key: HIVE-21342 > URL: https://issues.apache.org/jira/browse/HIVE-21342 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 > Environment: hive-3.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21342.02.patch, HIVE-21342.patch > > > staging dir cleanup does not happen for the "analyze table .. compute > statistics for columns", this leads to stale directory on hdfs. > the problem seems to be with ColumnStatsSemanticAnalyzer which don't have > hdfscleanup set for the context. > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java#L310 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21342) Analyze compute stats for column leave behind staging dir on hdfs
[ https://issues.apache.org/jira/browse/HIVE-21342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-21342: -- Attachment: HIVE-21342.02.patch Status: Patch Available (was: Open) > Analyze compute stats for column leave behind staging dir on hdfs > - > > Key: HIVE-21342 > URL: https://issues.apache.org/jira/browse/HIVE-21342 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 > Environment: hive-3.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-21342.02.patch, HIVE-21342.patch, > HIVE-21499.01.patch > > > staging dir cleanup does not happen for the "analyze table .. compute > statistics for columns", this leads to stale directory on hdfs. > the problem seems to be with ColumnStatsSemanticAnalyzer which don't have > hdfscleanup set for the context. > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java#L310 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
