[jira] [Commented] (HIVE-18423) Hive should support usage of external tables using jdbc
[
https://issues.apache.org/jira/browse/HIVE-18423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451554#comment-16451554
]
Hive QA commented on HIVE-18423:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} jdbc-handler: The patch generated 1 new + 47 unchanged
- 1 fixed = 48 total (was 48) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 1 new + 378 unchanged - 0
fixed = 379 total (was 378) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10473/dev-support/hive-personality.sh
|
| git revision | master / 63923e7 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10473/yetus/diff-checkstyle-jdbc-handler.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10473/yetus/diff-checkstyle-ql.txt
|
| modules | C: common jdbc-handler ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10473/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive should support usage of external tables using jdbc
> ---
>
> Key: HIVE-18423
> URL: https://issues.apache.org/jira/browse/HIVE-18423
> Project: Hive
> Issue Type: Improvement
>Reporter: Jonathan Doron
>Assignee: Jonathan Doron
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-18423.1.patch, HIVE-18423.2.patch,
> HIVE-18423.3.patch, HIVE-18423.4.patch, HIVE-18423.5.patch, HIVE-18423.6.patch
>
>
> Hive should support the usage of external jdbc tables(and not only external
> tables that hold queries), so an Hive user would be able to use the external
> table as an hive internal table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19293) Turn on hive.optimize.index.filter
[ https://issues.apache.org/jira/browse/HIVE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg resolved HIVE-19293. Resolution: Duplicate > Turn on hive.optimize.index.filter > -- > > Key: HIVE-19293 > URL: https://issues.apache.org/jira/browse/HIVE-19293 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Priority: Major > > HIVE-18448 has turned this off. This could cause performance regression. This > should be turned on by default -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19293) Turn on hive.optimize.index.filter
[ https://issues.apache.org/jira/browse/HIVE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19293: --- Issue Type: Task (was: Bug) > Turn on hive.optimize.index.filter > -- > > Key: HIVE-19293 > URL: https://issues.apache.org/jira/browse/HIVE-19293 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Priority: Major > > HIVE-18448 has turned this off. This could cause performance regression. This > should be turned on by default -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19108) Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q causes Wrong Query Results
[ https://issues.apache.org/jira/browse/HIVE-19108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19108: Attachment: HIVE-19108.01.patch > Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q > causes Wrong Query Results > --- > > Key: HIVE-19108 > URL: https://issues.apache.org/jira/browse/HIVE-19108 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19108.01.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19271) TestMiniLlapLocalCliDriver default_constraint and check_constraint failing
[ https://issues.apache.org/jira/browse/HIVE-19271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451573#comment-16451573 ] Vineet Garg commented on HIVE-19271: [~ashutoshc] can you take a look? > TestMiniLlapLocalCliDriver default_constraint and check_constraint failing > -- > > Key: HIVE-19271 > URL: https://issues.apache.org/jira/browse/HIVE-19271 > Project: Hive > Issue Type: Test >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-19271.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18862) qfiles: prepare .q files for using datasets
[ https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451595#comment-16451595 ] Rui Li commented on HIVE-18862: --- It's great if we can have this in branch-3, which makes it easier to cherry pick commits in master into branch-3. > qfiles: prepare .q files for using datasets > --- > > Key: HIVE-18862 > URL: https://issues.apache.org/jira/browse/HIVE-18862 > Project: Hive > Issue Type: Sub-task >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, > HIVE-18862.03.patch, HIVE-18862.04.patch, HIVE-18862.05.patch, > HIVE-18862.06.patch, HIVE-18862.07.patch, HIVE-18862.08.patch, > HIVE-18862.09.patch > > > # Parse .q files for source table usage > # Add needed dataset annotations > # Remove create table statements from "q_test_init.sql" like files > # Handle oncoming issues related to dataset introduction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19233) Add utility for acid 1.0 to 2.0 migration
[ https://issues.apache.org/jira/browse/HIVE-19233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19233: -- Status: Patch Available (was: Open) > Add utility for acid 1.0 to 2.0 migration > - > > Key: HIVE-19233 > URL: https://issues.apache.org/jira/browse/HIVE-19233 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-19233.01.patch, HIVE-19233.02.patch, > HIVE-19233.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19269) Vectorization: Turn On by Default
[
https://issues.apache.org/jira/browse/HIVE-19269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451355#comment-16451355
]
Hive QA commented on HIVE-19269:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10468/dev-support/hive-personality.sh
|
| git revision | master / 36ef274 |
| Default Java | 1.8.0_111 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10468/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Turn On by Default
> -
>
> Key: HIVE-19269
> URL: https://issues.apache.org/jira/browse/HIVE-19269
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19269.01.patch, HIVE-19269.02.patch
>
>
> Reflect that our most expected Hive deployment will be using vectorization
> and change the default of hive.vectorized.execution.enabled to true.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18448) Drop Support For Indexes From Apache Hive
[
https://issues.apache.org/jira/browse/HIVE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451376#comment-16451376
]
Vineet Garg commented on HIVE-18448:
[~kgyrtkirk] this patch turned {{hive.optimize.index.filter}} to false. This
has disabled ORC ppd by default. We should turn if on again as it might cause
performance regression
> Drop Support For Indexes From Apache Hive
> -
>
> Key: HIVE-18448
> URL: https://issues.apache.org/jira/browse/HIVE-18448
> Project: Hive
> Issue Type: Improvement
> Components: Indexing
>Reporter: BELUGA BEHR
>Assignee: Zoltan Haindrich
>Priority: Minor
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-18448.01.patch, HIVE-18448.01wip02.patch,
> HIVE-18448.01wip03.patch, HIVE-18448.01wip04.patch, HIVE-18448.01wip05.patch
>
>
> If a user needs to look up a small subset of records quickly, they can use
> Apache HBase, if they need fast retrieval of larger sets of data, or fast
> joins, aggregations, they can use Apache Impala. It seems to me that Hive
> indexes do not serve much of a role in the future of Hive.
> Even without moving workloads to other products, columnar file formats with
> their statistics achieve similar goals as Hive indexes.
> Please consider dropping Indexes from the Apache Hive project.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19186) Multi Table INSERT statements query has a flaw for partitioned table when INSERT INTO and INSERT OVERWRITE are used
[ https://issues.apache.org/jira/browse/HIVE-19186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451383#comment-16451383 ] Steve Yeom commented on HIVE-19186: --- Out of the 4 failed tests of age 1, every tests except results_cache_invalidation2.q is clear in my environment. results_cache_invalidation2.q fails with the same error in the code base without the patch of this jira. > Multi Table INSERT statements query has a flaw for partitioned table when > INSERT INTO and INSERT OVERWRITE are used > --- > > Key: HIVE-19186 > URL: https://issues.apache.org/jira/browse/HIVE-19186 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19186.01.patch, HIVE-19186.02.patch, > HIVE-19186.03.patch > > > One problem test case is: > create table intermediate(key int) partitioned by (p int) stored as orc; > insert into table intermediate partition(p='455') select distinct key from > src where key >= 0 order by key desc limit 2; > insert into table intermediate partition(p='456') select distinct key from > src where key is not null order by key asc limit 2; > insert into table intermediate partition(p='457') select distinct key from > src where key >= 100 order by key asc limit 2; > create table multi_partitioned (key int, key2 int) partitioned by (p int); > from intermediate > insert into table multi_partitioned partition(p=2) select p, key > insert overwrite table multi_partitioned partition(p=1) select key, p; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19277) Active/Passive HA web endpoints does not allow cross origin requests
[ https://issues.apache.org/jira/browse/HIVE-19277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19277: - Status: Patch Available (was: Open) > Active/Passive HA web endpoints does not allow cross origin requests > > > Key: HIVE-19277 > URL: https://issues.apache.org/jira/browse/HIVE-19277 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19277.1.patch > > > CORS is not allowed with web endpoints added for active/passive HA. Enable > CORS by default for all web endpoints. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19209) Streaming ingest record writers should accept input stream
[ https://issues.apache.org/jira/browse/HIVE-19209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451393#comment-16451393 ] Prasanth Jayachandran commented on HIVE-19209: -- Updated API doc. > Streaming ingest record writers should accept input stream > -- > > Key: HIVE-19209 > URL: https://issues.apache.org/jira/browse/HIVE-19209 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19209.1.patch, HIVE-19209.2.patch > > > Record writers in streaming ingest currently accepts byte[]. Provide an > option for clients to pass in input stream directly from which byte[] for > record can be constructed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-19293) Turn on hive.optimize.index.filter
[ https://issues.apache.org/jira/browse/HIVE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg resolved HIVE-19293. Resolution: Not A Problem My bad. It was already turned off. > Turn on hive.optimize.index.filter > -- > > Key: HIVE-19293 > URL: https://issues.apache.org/jira/browse/HIVE-19293 > Project: Hive > Issue Type: Bug >Reporter: Vineet Garg >Assignee: Zoltan Haindrich >Priority: Major > > HIVE-18448 has turned this off. This could cause performance regression. This > should be turned on by default -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18448) Drop Support For Indexes From Apache Hive
[ https://issues.apache.org/jira/browse/HIVE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451409#comment-16451409 ] Vineet Garg commented on HIVE-18448: [~kgyrtkirk] Never mind it was already turned off your patch didn't turn it off. I looked at wrong diff :) > Drop Support For Indexes From Apache Hive > - > > Key: HIVE-18448 > URL: https://issues.apache.org/jira/browse/HIVE-18448 > Project: Hive > Issue Type: Improvement > Components: Indexing >Reporter: BELUGA BEHR >Assignee: Zoltan Haindrich >Priority: Minor > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-18448.01.patch, HIVE-18448.01wip02.patch, > HIVE-18448.01wip03.patch, HIVE-18448.01wip04.patch, HIVE-18448.01wip05.patch > > > If a user needs to look up a small subset of records quickly, they can use > Apache HBase, if they need fast retrieval of larger sets of data, or fast > joins, aggregations, they can use Apache Impala. It seems to me that Hive > indexes do not serve much of a role in the future of Hive. > Even without moving workloads to other products, columnar file formats with > their statistics achieve similar goals as Hive indexes. > Please consider dropping Indexes from the Apache Hive project. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[
https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451486#comment-16451486
]
Hive QA commented on HIVE-19118:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920411/HIVE-19118.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 46 failed/errored test(s), 14292 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3
(batchId=300)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill
(batchId=242)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
[jira] [Updated] (HIVE-12369) Native Vector GroupBy
[
https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-12369:
Summary: Native Vector GroupBy (was: Native Vector GroupBy (Part 1))
> Native Vector GroupBy
> -
>
> Key: HIVE-12369
> URL: https://issues.apache.org/jira/browse/HIVE-12369
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch,
> HIVE-12369.05.patch, HIVE-12369.06.patch, HIVE-12369.091.patch,
> HIVE-12369.094.patch, HIVE-12369.095.patch, HIVE-12369.096.patch,
> HIVE-12369.097.patch, HIVE-12369.098.patch, HIVE-12369.099.patch,
> HIVE-12369.0991.patch, HIVE-12369.0992.patch, HIVE-12369.0993.patch
>
>
> Implement Native Vector GroupBy using fast hash table technology developed
> for Native Vector MapJoin, etc.
> Patch is currently limited to a single COUNT aggregation. Or, no aggregation
> also known as duplicate reduction.
> Here are examples of new kinds of classes introduces that stored the count in
> the slot table and don't allocate hash elements:
> {noformat}
> COUNT(column) VectorGroupByHashLongKeySingleCountColumnOperator
> COUNT(key) VectorGroupByHashLongKeySingleCountKeyOperator
> COUNT(*) VectorGroupByHashLongKeySingleCountStarOperator
> {noformat}
> And the duplicate reduction operator a single key. Example:
> {noformat}
> VectorGroupByHashLongKeyDuplicateReductionOperator
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18908) FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Summary: FULL OUTER JOIN to MapJoin (was: Add support for FULL OUTER JOIN to MapJoin) > FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: FULL OUTER MapJoin Code Changes.docx, > HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, > HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, > HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, > HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, JOIN to > MAPJOIN Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19264) Vectorization: Reenable vectorization in vector_adaptor_usage_mode.q
[ https://issues.apache.org/jira/browse/HIVE-19264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19264: Resolution: Fixed Status: Resolved (was: Patch Available) > Vectorization: Reenable vectorization in vector_adaptor_usage_mode.q > > > Key: HIVE-19264 > URL: https://issues.apache.org/jira/browse/HIVE-19264 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19264.01.patch, HIVE-19264.02.patch > > > [~vihangk1] observed vectorization had accidentally been turned off. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HIVE-19293) Turn on hive.optimize.index.filter
[ https://issues.apache.org/jira/browse/HIVE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reopened HIVE-19293: Assignee: (was: Zoltan Haindrich) > Turn on hive.optimize.index.filter > -- > > Key: HIVE-19293 > URL: https://issues.apache.org/jira/browse/HIVE-19293 > Project: Hive > Issue Type: Bug >Reporter: Vineet Garg >Priority: Major > > HIVE-18448 has turned this off. This could cause performance regression. This > should be turned on by default -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19118: Attachment: HIVE-19118.02.patch > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19118.01.patch, HIVE-19118.02.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19118: Status: Open (was: Patch Available) > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19118.01.patch, HIVE-19118.02.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17193) HoS: don't combine map works that are targets of different DPPs
[
https://issues.apache.org/jira/browse/HIVE-17193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451652#comment-16451652
]
Hive QA commented on HIVE-17193:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 1 new + 25 unchanged - 2 fixed
= 26 total (was 27) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10474/dev-support/hive-personality.sh
|
| git revision | master / 63923e7 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10474/yetus/diff-checkstyle-ql.txt
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10474/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HoS: don't combine map works that are targets of different DPPs
> ---
>
> Key: HIVE-17193
> URL: https://issues.apache.org/jira/browse/HIVE-17193
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-17193.1.patch, HIVE-17193.2.patch,
> HIVE-17193.3.patch, HIVE-17193.4.patch, HIVE-17193.5.patch
>
>
> Suppose {{srcpart}} is partitioned by {{ds}}. The following query can trigger
> the issue:
> {code}
> explain
> select * from
> (select srcpart.ds,srcpart.key from srcpart join src on srcpart.ds=src.key)
> a
> join
> (select srcpart.ds,srcpart.key from srcpart join src on
> srcpart.ds=src.value) b
> on a.key=b.key;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18766) Race condition during shutdown of RemoteDriver, error messages aren't always sent
[
https://issues.apache.org/jira/browse/HIVE-18766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-18766:
---
Assignee: Aihua Xu
> Race condition during shutdown of RemoteDriver, error messages aren't always
> sent
> -
>
> Key: HIVE-18766
> URL: https://issues.apache.org/jira/browse/HIVE-18766
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
>
> When we send an error during shutdown of the {{RemoteDriver}}, we don't wait
> for the error message to be sent. We just send it and then close the RPC
> channel. For a graceful shutdown, that doesn't seem ideal. We should at least
> wait a bit for the RPC message to be sent before shutting things down. I ran
> some tests locally and its pretty easy to hit a situation where the error
> message doesn't even get sent to the {{SparkClientImpl}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19232) results_cache_invalidation2 is failing
[ https://issues.apache.org/jira/browse/HIVE-19232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19232: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master/branch-3 > results_cache_invalidation2 is failing > -- > > Key: HIVE-19232 > URL: https://issues.apache.org/jira/browse/HIVE-19232 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Ashutosh Chauhan >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19232.1.patch, HIVE-19232.2.patch > > > TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2] > Fails with plan difference on both cli as well as minillaplocal. Plan diffs > looks concerning since its now longer using cache. > Also, it should run only on minillaplocal -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19232) results_cache_invalidation2 is failing
[ https://issues.apache.org/jira/browse/HIVE-19232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19232: -- Fix Version/s: 3.0.0 > results_cache_invalidation2 is failing > -- > > Key: HIVE-19232 > URL: https://issues.apache.org/jira/browse/HIVE-19232 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Ashutosh Chauhan >Assignee: Jason Dere >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19232.1.patch, HIVE-19232.2.patch > > > TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2] > Fails with plan difference on both cli as well as minillaplocal. Plan diffs > looks concerning since its now longer using cache. > Also, it should run only on minillaplocal -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19281) incorrect protocol name for LLAP AM plugin
[ https://issues.apache.org/jira/browse/HIVE-19281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451417#comment-16451417 ] Jason Dere commented on HIVE-19281: --- +1 > incorrect protocol name for LLAP AM plugin > -- > > Key: HIVE-19281 > URL: https://issues.apache.org/jira/browse/HIVE-19281 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19281.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing
[
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451537#comment-16451537
]
Hive QA commented on HIVE-18910:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920417/HIVE-18910.41.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10472/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10472/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10472/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-04-25 02:39:49.896
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-10472/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-04-25 02:39:49.899
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
e909448..63923e7 master -> origin/master
+ git reset --hard HEAD
HEAD is now at e909448 HIVE-19232: results_cache_invalidation2 is failing
(Jason Dere, reviewed by Vineet Garg)
+ git clean -f -d
Removing ${project.basedir}/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 63923e7 HIVE-19186 : Multi Table INSERT statements query has a
flaw for partitioned table when INSERT INTO and INSERT OVERWRITE are used
(Steve Yeom via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-04-25 02:39:57.405
+ rm -rf ../yetus_PreCommit-HIVE-Build-10472
+ mkdir ../yetus_PreCommit-HIVE-Build-10472
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-10472
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-10472/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: git apply: bad git-diff - inconsistent old filename on line 19053
error: ql/src/test/results/clientpositive/results_cache_invalidation2.q.out:
does not exist in index
error: src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not exist in
index
error: src/test/results/positive/external_table_ppd.q.out: does not exist in
index
error: src/test/results/positive/hbase_binary_storage_queries.q.out: does not
exist in index
error: src/test/results/positive/hbase_ddl.q.out: does not exist in index
error: src/test/results/positive/hbasestats.q.out: does not exist in index
error:
streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java:
does not exist in index
error:
streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/worker/BucketIdResolver.java:
does not exist in index
error:
streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/worker/MutatorCoordinator.java:
does not exist in index
error:
streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java: does
not exist in index
error:
streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/worker/TestBucketIdResolverImpl.java:
does not exist in index
error:
webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatTable.java:
does not exist in index
error:
webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java:
does not exist in index
error:
hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out:
does not exist in index
error: hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out:
does not exist in index
error:
hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out:
does not exist in index
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Attachment: HIVE-18908.096.patch > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: FULL OUTER MapJoin Code Changes.docx, > HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, > HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, > HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, > HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, JOIN to > MAPJOIN Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451539#comment-16451539 ] Matt McCline commented on HIVE-18908: - Putting this change into hibernation for now. Not a priority to work on it for now. > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: FULL OUTER MapJoin Code Changes.docx, > HIVE-18908.01.patch, HIVE-18908.02.patch, HIVE-18908.03.patch, > HIVE-18908.04.patch, HIVE-18908.05.patch, HIVE-18908.06.patch, > HIVE-18908.08.patch, HIVE-18908.09.patch, HIVE-18908.091.patch, > HIVE-18908.092.patch, HIVE-18908.093.patch, HIVE-18908.096.patch, JOIN to > MAPJOIN Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19269) Vectorization: Turn On by Default
[ https://issues.apache.org/jira/browse/HIVE-19269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19269: Status: In Progress (was: Patch Available) > Vectorization: Turn On by Default > - > > Key: HIVE-19269 > URL: https://issues.apache.org/jira/browse/HIVE-19269 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19269.01.patch, HIVE-19269.02.patch > > > Reflect that our most expected Hive deployment will be using vectorization > and change the default of hive.vectorized.execution.enabled to true. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19283) Select count(distinct()) a couple of times stuck in last reducer
[
https://issues.apache.org/jira/browse/HIVE-19283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451555#comment-16451555
]
Goun Na edited comment on HIVE-19283 at 4/25/18 3:13 AM:
-
[~gopalv], that is a great news! Thanks!
was (Author: goun):
[~gopalv], this is great! Thanks!
> Select count(distinct()) a couple of times stuck in last reducer
>
>
> Key: HIVE-19283
> URL: https://issues.apache.org/jira/browse/HIVE-19283
> Project: Hive
> Issue Type: Improvement
> Components: CBO, Logical Optimizer
>Affects Versions: 2.1.1
>Reporter: Goun Na
>Assignee: Ashutosh Chauhan
>Priority: Major
>
> Distinct count query performance is significantly improved due to
> HIVE-10568.
> {code:java}
> select count(distinct elevenst_id)
> from 11st.log_table
> where part_dt between '20180101' and '20180131'{code}
>
> However, some queries with several distinct counts are still slow. It starts
> with multiple mappers, but stuck in the last one reducer.
> {code:java}
> select
> count(distinct elevenst_id)
> , count(distinct member_id)
> , count(distinct user_id)
> , count(distinct action_id)
> , count(distinct other_id)
> from 11st.log_table
> where part_dt between '20180101' and '20180131'{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19247) StatsOptimizer: Missing stats fast-path for Date
[
https://issues.apache.org/jira/browse/HIVE-19247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-19247:
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Gopal!
[~vgarg] Please consider it for 3.0
> StatsOptimizer: Missing stats fast-path for Date
>
>
> Key: HIVE-19247
> URL: https://issues.apache.org/jira/browse/HIVE-19247
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 2.2.0, 3.0.0, 2.3.2
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19247.1.patch
>
>
> {code}
> 2018-04-19T18:57:24,268 DEBUG [67259108-c184-4c92-9e18-9e296922
> HiveServer2-Handler-Pool: Thread-73]: optimizer.StatsOptimizer
> (StatsOptimizer.java:process(614)) - Unsupported type: date encountered in
> metadata optimizer for column : jour
> {code}
> {code}
> if (udaf instanceof GenericUDAFMin) {
> ExprNodeColumnDesc colDesc =
> (ExprNodeColumnDesc)exprMap.get(((ExprNodeColumnDesc)aggr.getParameters().get(0)).getColumn());
> String colName = colDesc.getColumn();
> StatType type = getType(colDesc.getTypeString());
> if (!tbl.isPartitioned()) {
> if
> (!StatsSetupConst.areColumnStatsUptoDate(tbl.getParameters(), colName)) {
> Logger.debug("Stats for table : " + tbl.getTableName() + "
> column " + colName
> + " are not up to date.");
> return null;
> }
> ColumnStatisticsData statData =
> hive.getMSC().getTableColumnStatistics(
> tbl.getDbName(), tbl.getTableName(),
> Lists.newArrayList(colName))
> .get(0).getStatsData();
> String name = colDesc.getTypeString().toUpperCase();
> switch (type) {
> case Integeral: {
> LongSubType subType = LongSubType.valueOf(name);
> LongColumnStatsData lstats = statData.getLongStats();
> if (lstats.isSetLowValue()) {
> oneRow.add(subType.cast(lstats.getLowValue()));
> } else {
> oneRow.add(null);
> }
> break;
> }
> case Double: {
> DoubleSubType subType = DoubleSubType.valueOf(name);
> DoubleColumnStatsData dstats = statData.getDoubleStats();
> if (dstats.isSetLowValue()) {
> oneRow.add(subType.cast(dstats.getLowValue()));
> } else {
> oneRow.add(null);
> }
> break;
> }
> default: // unsupported type
> Logger.debug("Unsupported type: " + colDesc.getTypeString()
> + " encountered in " +
> "metadata optimizer for column : " + colName);
> return null;
> }
> }
> {code}
> {code}
> enum StatType{
> Integeral,
> Double,
> String,
> Boolean,
> Binary,
> Unsupported
> }
> enum LongSubType {
> BIGINT { @Override
> Object cast(long longValue) { return longValue; } },
> INT { @Override
> Object cast(long longValue) { return (int)longValue; } },
> SMALLINT { @Override
> Object cast(long longValue) { return (short)longValue; } },
> TINYINT { @Override
> Object cast(long longValue) { return (byte)longValue; } };
> abstract Object cast(long longValue);
> }
> {code}
> Date is stored in stats (& also the typo there).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19277) Active/Passive HA web endpoints does not allow cross origin requests
[
https://issues.apache.org/jira/browse/HIVE-19277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451380#comment-16451380
]
Prasanth Jayachandran commented on HIVE-19277:
--
I am not able to unit test this but from what I read if the response headers
have "Access-Control-Allow-Origin: *" then it will allow cross origin requests.
{code:title=without CORS response headers}
$ curl -H "Access-Control-Request-Method: GET" -H "Origin: https://dag.io;
--head http://localhost:10020/leader
HTTP/1.1 200 OK
Date: Tue, 24 Apr 2018 22:54:28 GMT
Transfer-Encoding: chunked
Server: Jetty(9.3.8.v20160314)
{code}
{code:title=with CORS response headers}
curl -H "Access-Control-Request-Method: GET" -H "Origin: https://dag.io; --head
http://localhost:10020/leader
HTTP/1.1 200 OK
Date: Tue, 24 Apr 2018 22:59:02 GMT
Content-Type: application/json; charset=utf8
Access-Control-Allow-Methods: GET,DELETE
Access-Control-Allow-Origin: *
Transfer-Encoding: chunked
Server: Jetty(9.3.8.v20160314)
{code}
[~sershe] can you please review this one? small patch
> Active/Passive HA web endpoints does not allow cross origin requests
>
>
> Key: HIVE-19277
> URL: https://issues.apache.org/jira/browse/HIVE-19277
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19277.1.patch
>
>
> CORS is not allowed with web endpoints added for active/passive HA. Enable
> CORS by default for all web endpoints.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19209) Streaming ingest record writers should accept input stream
[ https://issues.apache.org/jira/browse/HIVE-19209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19209: - Attachment: HIVE-19209.2.patch > Streaming ingest record writers should accept input stream > -- > > Key: HIVE-19209 > URL: https://issues.apache.org/jira/browse/HIVE-19209 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19209.1.patch, HIVE-19209.2.patch > > > Record writers in streaming ingest currently accepts byte[]. Provide an > option for clients to pass in input stream directly from which byte[] for > record can be constructed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18958) Fix Spark config warnings
[
https://issues.apache.org/jira/browse/HIVE-18958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451405#comment-16451405
]
Bharathkrishna Guruvayoor Murali commented on HIVE-18958:
-
Pending removing the warning for configuration key
'spark.yarn.driver.memoryOverhead'
{color:#33}Submitting the patch to run tests and also get review for the
removal of code that adds HIVE_SPARK_RSC_CONFIGS to SparkConf in
HiveSparkClientFactory.{color}
> Fix Spark config warnings
> -
>
> Key: HIVE-18958
> URL: https://issues.apache.org/jira/browse/HIVE-18958
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18958.01.patch
>
>
> Getting a few configuration warnings in the logs that we should fix:
> {code}
> 2018-03-14T10:06:19,164 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key 'spark.yarn.driver.memoryOverhead' has
> been deprecated as of Spark 2.3 and may be removed in the future. Please use
> the new key 'spark.driver.memoryOverhead' instead.
> 2018-03-14T10:06:19,165 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key spark.akka.logLifecycleEvents is not
> supported any more because Spark doesn't use Akka since 2.0
> 2018-03-14T10:06:19,165 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead'
> has been deprecated as of Spark 2.3 and may be removed in the future. Please
> use the new key 'spark.executor.memoryOverhead' instead.
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.server.connect.timeout=9
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.rpc.threads=8
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.connect.timeout=3
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.secret.bits=256
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.rpc.max.size=52428800
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18999) Filter operator does not work for List
[
https://issues.apache.org/jira/browse/HIVE-18999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451422#comment-16451422
]
Steve Yeom commented on HIVE-18999:
---
Talked with Ashutosh.
What he is proposing is that we can convert the type conversion while we are
converting
Hive filter expression AST tree to Calcite Rex node tree, not during execution,
> Filter operator does not work for List
> --
>
> Key: HIVE-18999
> URL: https://issues.apache.org/jira/browse/HIVE-18999
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18999.01.patch, HIVE-18999.02.patch,
> HIVE-18999.03.patch
>
>
> {code:sql}
> create table table1(col0 int, col1 bigint, col2 string, col3 bigint, col4
> bigint);
> insert into table1 values (1, 1, 'ccl',2014, 11);
> insert into table1 values (1, 1, 'ccl',2015, 11);
> insert into table1 values (1, 1, 'ccl',2014, 11);
> insert into table1 values (1, 1, 'ccl',2013, 11);
> -- INCORRECT
> SELECT COUNT(t1.col0) from table1 t1 where struct(col3, col4) in
> (struct(2014,11));
> -- CORRECT
> SELECT COUNT(t1.col0) from table1 t1 where struct(col3, col4) in
> (struct('2014','11'));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19293) Turn on hive.optimize.index.filter
[ https://issues.apache.org/jira/browse/HIVE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451517#comment-16451517 ] Gopal V commented on HIVE-19293: Now that the "CREATE INDEX" can no longer return incorrect results in queries, we should enable that flag=true, because that now only turns on the ORC/Parquet index filters only. > Turn on hive.optimize.index.filter > -- > > Key: HIVE-19293 > URL: https://issues.apache.org/jira/browse/HIVE-19293 > Project: Hive > Issue Type: Bug >Reporter: Vineet Garg >Assignee: Zoltan Haindrich >Priority: Major > > HIVE-18448 has turned this off. This could cause performance regression. This > should be turned on by default -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19274) Add an OpTreeSignature persistence checker hook
[
https://issues.apache.org/jira/browse/HIVE-19274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451523#comment-16451523
]
Hive QA commented on HIVE-19274:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
55s{color} | {color:red} root: The patch generated 1 new + 9 unchanged - 1
fixed = 10 total (was 10) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 1 new + 9 unchanged - 1 fixed
= 10 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml javac javadoc findbugs checkstyle
compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10471/dev-support/hive-personality.sh
|
| git revision | master / e909448 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10471/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10471/yetus/diff-checkstyle-ql.txt
|
| modules | C: . ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10471/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add an OpTreeSignature persistence checker hook
> ---
>
> Key: HIVE-19274
> URL: https://issues.apache.org/jira/browse/HIVE-19274
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19274.01.patch, HIVE-19274.01wip01.patch
>
>
> Adding a Hook to run during testing which checks that OpTreeSignatures are
> working as expected would be really usefull; it should run at least during
> the PerfCliDriver
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19293) Turn on hive.optimize.index.filter
[ https://issues.apache.org/jira/browse/HIVE-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451535#comment-16451535 ] Ashutosh Chauhan commented on HIVE-19293: - If someone is interested I have 1 MB outdated patch at HIVE-12342 which can be used as a starting point. : ) > Turn on hive.optimize.index.filter > -- > > Key: HIVE-19293 > URL: https://issues.apache.org/jira/browse/HIVE-19293 > Project: Hive > Issue Type: Bug >Reporter: Vineet Garg >Assignee: Zoltan Haindrich >Priority: Major > > HIVE-18448 has turned this off. This could cause performance regression. This > should be turned on by default -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[
https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451542#comment-16451542
]
Matt McCline commented on HIVE-12369:
-
Putting this change into hibernation for now. Not a priority to work on it for
now.
> Native Vector GroupBy
> -
>
> Key: HIVE-12369
> URL: https://issues.apache.org/jira/browse/HIVE-12369
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch,
> HIVE-12369.05.patch, HIVE-12369.06.patch, HIVE-12369.091.patch,
> HIVE-12369.094.patch, HIVE-12369.095.patch, HIVE-12369.096.patch,
> HIVE-12369.097.patch, HIVE-12369.098.patch, HIVE-12369.099.patch,
> HIVE-12369.0991.patch, HIVE-12369.0992.patch, HIVE-12369.0993.patch
>
>
> Implement Native Vector GroupBy using fast hash table technology developed
> for Native Vector MapJoin, etc.
> Patch is currently limited to a single COUNT aggregation. Or, no aggregation
> also known as duplicate reduction.
> Here are examples of new kinds of classes introduces that stored the count in
> the slot table and don't allocate hash elements:
> {noformat}
> COUNT(column) VectorGroupByHashLongKeySingleCountColumnOperator
> COUNT(key) VectorGroupByHashLongKeySingleCountKeyOperator
> COUNT(*) VectorGroupByHashLongKeySingleCountStarOperator
> {noformat}
> And the duplicate reduction operator a single key. Example:
> {noformat}
> VectorGroupByHashLongKeyDuplicateReductionOperator
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19269) Vectorization: Turn On by Default
[ https://issues.apache.org/jira/browse/HIVE-19269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451548#comment-16451548 ] Matt McCline commented on HIVE-19269: - Next step: Generate new Q output files for all Drivers. > Vectorization: Turn On by Default > - > > Key: HIVE-19269 > URL: https://issues.apache.org/jira/browse/HIVE-19269 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19269.01.patch, HIVE-19269.02.patch > > > Reflect that our most expected Hive deployment will be using vectorization > and change the default of hive.vectorized.execution.enabled to true. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19186) Multi Table INSERT statements query has a flaw for partitioned table when INSERT INTO and INSERT OVERWRITE are used
[ https://issues.apache.org/jira/browse/HIVE-19186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19186: --- Fix Version/s: 3.0.0 > Multi Table INSERT statements query has a flaw for partitioned table when > INSERT INTO and INSERT OVERWRITE are used > --- > > Key: HIVE-19186 > URL: https://issues.apache.org/jira/browse/HIVE-19186 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19186.01.patch, HIVE-19186.02.patch, > HIVE-19186.03.patch > > > One problem test case is: > create table intermediate(key int) partitioned by (p int) stored as orc; > insert into table intermediate partition(p='455') select distinct key from > src where key >= 0 order by key desc limit 2; > insert into table intermediate partition(p='456') select distinct key from > src where key is not null order by key asc limit 2; > insert into table intermediate partition(p='457') select distinct key from > src where key >= 100 order by key asc limit 2; > create table multi_partitioned (key int, key2 int) partitioned by (p int); > from intermediate > insert into table multi_partitioned partition(p=2) select p, key > insert overwrite table multi_partitioned partition(p=1) select key, p; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19186) Multi Table INSERT statements query has a flaw for partitioned table when INSERT INTO and INSERT OVERWRITE are used
[ https://issues.apache.org/jira/browse/HIVE-19186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451557#comment-16451557 ] Vineet Garg commented on HIVE-19186: Pushed to branch-3 > Multi Table INSERT statements query has a flaw for partitioned table when > INSERT INTO and INSERT OVERWRITE are used > --- > > Key: HIVE-19186 > URL: https://issues.apache.org/jira/browse/HIVE-19186 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19186.01.patch, HIVE-19186.02.patch, > HIVE-19186.03.patch > > > One problem test case is: > create table intermediate(key int) partitioned by (p int) stored as orc; > insert into table intermediate partition(p='455') select distinct key from > src where key >= 0 order by key desc limit 2; > insert into table intermediate partition(p='456') select distinct key from > src where key is not null order by key asc limit 2; > insert into table intermediate partition(p='457') select distinct key from > src where key >= 100 order by key asc limit 2; > create table multi_partitioned (key int, key2 int) partitioned by (p int); > from intermediate > insert into table multi_partitioned partition(p=2) select p, key > insert overwrite table multi_partitioned partition(p=1) select key, p; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17193) HoS: don't combine map works that are targets of different DPPs
[
https://issues.apache.org/jira/browse/HIVE-17193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451675#comment-16451675
]
Hive QA commented on HIVE-17193:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920422/HIVE-17193.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 44 failed/errored test(s), 14293 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3
(batchId=300)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative
(batchId=254)
{noformat}
Test results:
[jira] [Updated] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19118: Status: Patch Available (was: Open) Patch is ready. Please help review. > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19118.01.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19118: Attachment: (was: HIVE-19118.01.patch) > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19118.01.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19118: Attachment: HIVE-19118.01.patch > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19118.01.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19077) Handle duplicate ptests requests standing in queue at the same time
[
https://issues.apache.org/jira/browse/HIVE-19077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449372#comment-16449372
]
Zoltan Haindrich commented on HIVE-19077:
-
I think we should be in control of our jenkins jobs; but I don't have the right
to do it...
> Handle duplicate ptests requests standing in queue at the same time
> ---
>
> Key: HIVE-19077
> URL: https://issues.apache.org/jira/browse/HIVE-19077
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Blocker
> Fix For: 3.1.0
>
> Attachments: HIVE-19077.0.patch, HIVE-19077.1.patch,
> HIVE-19077.overrideoption.patch, HIVE-19077.sslFix.patch
>
>
> I've been keeping on eye on our {{PreCommit-HIVE-Build}} job, and what I
> noticed that sometimes huge queues can build up, that contain jira's more
> than once. (Yesterday I've seen a queue of 40, having 31 distinct jiras..)
> Simple scenario is that I upload a patch, it gets queued for ptest (already
> long queue), and 3 hours later I will update it, re-upload and re-queue. Now
> the current ptest infra seems to be smart enough to always deal with the
> latest patch, so what will happen is that the same patch will be tested 2
> times (with ~3 hours) diff, most probably with same result.
> I propose we do some deduplication - if ptest starts running the request for
> Jira X, then it can take a look on the current queue, and see if X is there
> again. If so, it can skip for now, it will be picked up later anyway.
> In practice this means that if you reconsider your patch and update it, your
> original place in the queue will be gone (like as a penalty for changing it),
> but overall it saves resources for the whole community.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15767) Hive On Spark is not working on secure clusters from Oozie
[
https://issues.apache.org/jira/browse/HIVE-15767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449392#comment-16449392
]
linwukang commented on HIVE-15767:
--
Hi [~gezapeti] , after apply this patch, i find that the Hive On Spark worked
with yarn, all tasks is finished successfully. but there's another error throws
at the end of the progress:
{code:java}
2018-04-24T14:28:46,409 INFO [116dbf89-2982-407d-9b64-4206b3bbe105 main]
lockmgr.DbTxnManager: Stopped heartbeat for query:
flowagent_20180424142839_be68e2b9-aca9-4023-89f8-6a18d53dd0c5
2018-04-24T14:28:46,409 INFO [116dbf89-2982-407d-9b64-4206b3bbe105 main]
lockmgr.DbLockManager: releaseLocks: [lockid:438
queryId=flowagent_20180424142839_be68e2b9-aca9-4023-89f8-6a18d53dd0c5 txnid:0]
2018-04-24T14:28:46,422 ERROR [116dbf89-2982-407d-9b64-4206b3bbe105 main]
CliDriver: Failed with exception
java.io.IOException:org.apache.hadoop.ipc.RemoteException(java.io.IOException):
Delegation Token can be issued only with kerberos or web authentication
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:6635)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:563)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:988)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1727)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
can be issued only with kerberos or web authentication
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:6635)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:563)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:988)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1727)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:521)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:428)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2208)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
{code}
> Hive On Spark is not working on secure clusters from Oozie
> --
>
> Key: HIVE-15767
> URL: https://issues.apache.org/jira/browse/HIVE-15767
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.1, 2.1.1
>Reporter: Peter Cseh
>Assignee: Peter Cseh
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-15767-001.patch, HIVE-15767-002.patch,
> HIVE-15767.1.patch
>
>
> When a HiveAction is launched form Oozie with Hive On Spark enabled, we're
> getting errors:
> {noformat}
> Caused by: java.io.IOException: Exception reading
>
[jira] [Updated] (HIVE-19274) Add an OpTreeSignature persistence checker hook
[ https://issues.apache.org/jira/browse/HIVE-19274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19274: Attachment: HIVE-19274.01.patch > Add an OpTreeSignature persistence checker hook > --- > > Key: HIVE-19274 > URL: https://issues.apache.org/jira/browse/HIVE-19274 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19274.01.patch, HIVE-19274.01wip01.patch > > > Adding a Hook to run during testing which checks that OpTreeSignatures are > working as expected would be really usefull; it should run at least during > the PerfCliDriver -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19280) Invalid error messages for UPDATE/DELETE on insert-only transactional tables
[
https://issues.apache.org/jira/browse/HIVE-19280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449416#comment-16449416
]
Hive QA commented on HIVE-19280:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10455/dev-support/hive-personality.sh
|
| git revision | master / 56c3a95 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10455/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Invalid error messages for UPDATE/DELETE on insert-only transactional tables
>
>
> Key: HIVE-19280
> URL: https://issues.apache.org/jira/browse/HIVE-19280
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19280.01.patch
>
>
> UPDATE/DELETE on MM tables fails with
> "FAILED: SemanticException Error 10297: Attempt to do update or delete on
> table tpch.tbl_default_mm that is not transactional".
> This is invalid since the MM table is transactional.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing
[ https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18910: -- Attachment: HIVE-18910.41.patch > Migrate to Murmur hash for shuffle and bucketing > > > Key: HIVE-18910 > URL: https://issues.apache.org/jira/browse/HIVE-18910 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18910.1.patch, HIVE-18910.10.patch, > HIVE-18910.11.patch, HIVE-18910.12.patch, HIVE-18910.13.patch, > HIVE-18910.14.patch, HIVE-18910.15.patch, HIVE-18910.16.patch, > HIVE-18910.17.patch, HIVE-18910.18.patch, HIVE-18910.19.patch, > HIVE-18910.2.patch, HIVE-18910.20.patch, HIVE-18910.21.patch, > HIVE-18910.22.patch, HIVE-18910.23.patch, HIVE-18910.24.patch, > HIVE-18910.25.patch, HIVE-18910.26.patch, HIVE-18910.27.patch, > HIVE-18910.28.patch, HIVE-18910.29.patch, HIVE-18910.3.patch, > HIVE-18910.30.patch, HIVE-18910.31.patch, HIVE-18910.32.patch, > HIVE-18910.33.patch, HIVE-18910.34.patch, HIVE-18910.35.patch, > HIVE-18910.36.patch, HIVE-18910.36.patch, HIVE-18910.37.patch, > HIVE-18910.38.patch, HIVE-18910.39.patch, HIVE-18910.4.patch, > HIVE-18910.40.patch, HIVE-18910.41.patch, HIVE-18910.5.patch, > HIVE-18910.6.patch, HIVE-18910.7.patch, HIVE-18910.8.patch, HIVE-18910.9.patch > > > Hive uses JAVA hash which is not as good as murmur for better distribution > and efficiency in bucketing a table. > Migrate to murmur hash but still keep backward compatibility for existing > users so that they dont have to reload the existing tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19118: Status: Patch Available (was: Open) Fixed the spaces and indents. Please help review. Thanks. > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19118.01.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19118: Attachment: HIVE-19118.01.patch > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19118.01.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19118: Status: Open (was: Patch Available) Just found there are a few incorrect indents. Will make a change and submit again. > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19118.01.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns
[
https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449398#comment-16449398
]
Hive QA commented on HIVE-18986:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920362/HIVE-18986.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 14286 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10454/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10454/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10454/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 33 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12920362 - PreCommit-HIVE-Build
> Table rename will run java.lang.StackOverflowError in dataNucleus if the
> table contains large number of columns
> ---
>
> Key: HIVE-18986
> URL: https://issues.apache.org/jira/browse/HIVE-18986
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Aihua Xu
>
[jira] [Commented] (HIVE-19274) Add an OpTreeSignature persistence checker hook
[ https://issues.apache.org/jira/browse/HIVE-19274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449396#comment-16449396 ] Zoltan Haindrich commented on HIVE-19274: - test failures are not related; attaching patch rebased to current master > Add an OpTreeSignature persistence checker hook > --- > > Key: HIVE-19274 > URL: https://issues.apache.org/jira/browse/HIVE-19274 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19274.01.patch, HIVE-19274.01wip01.patch > > > Adding a Hook to run during testing which checks that OpTreeSignatures are > working as expected would be really usefull; it should run at least during > the PerfCliDriver -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19280) Invalid error messages for UPDATE/DELETE on insert-only transactional tables
[
https://issues.apache.org/jira/browse/HIVE-19280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449453#comment-16449453
]
Hive QA commented on HIVE-19280:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920360/HIVE-19280.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 14298 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_ppr_all]
(batchId=60)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10455/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10455/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10455/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 33 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12920360 - PreCommit-HIVE-Build
> Invalid error messages for UPDATE/DELETE on insert-only transactional tables
>
>
> Key: HIVE-19280
> URL: https://issues.apache.org/jira/browse/HIVE-19280
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>
[jira] [Updated] (HIVE-18423) Hive should support usage of external tables using jdbc
[ https://issues.apache.org/jira/browse/HIVE-18423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Doron updated HIVE-18423: -- Attachment: HIVE-18423.6.patch > Hive should support usage of external tables using jdbc > --- > > Key: HIVE-18423 > URL: https://issues.apache.org/jira/browse/HIVE-18423 > Project: Hive > Issue Type: Improvement >Reporter: Jonathan Doron >Assignee: Jonathan Doron >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-18423.1.patch, HIVE-18423.2.patch, > HIVE-18423.3.patch, HIVE-18423.4.patch, HIVE-18423.5.patch, HIVE-18423.6.patch > > > Hive should support the usage of external jdbc tables(and not only external > tables that hold queries), so an Hive user would be able to use the external > table as an hive internal table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18988: Attachment: HIVE-18988.04.patch > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch, HIVE-18988.04.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Find the valid last replication state such that it points to event ID of > open_txn event of oldest on-going txn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail
[
https://issues.apache.org/jira/browse/HIVE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449471#comment-16449471
]
Hive QA commented on HIVE-19204:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 1 new + 280 unchanged - 2
fixed = 281 total (was 282) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10457/dev-support/hive-personality.sh
|
| git revision | master / 56c3a95 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10457/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10457/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Detailed errors from some tasks are not displayed to the client because the
> tasks don't set exception when they fail
>
>
> Key: HIVE-19204
> URL: https://issues.apache.org/jira/browse/HIVE-19204
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19204.1.patch, HIVE-19204.2.patch
>
>
> In TaskRunner.java, if the tasks have exception set, then the task result
> will have such exception set and Driver.java will get such details and
> display to the client. But some tasks don't set such exceptions so the client
> won't see such details unless you check the HS2 log.
>
> {noformat}
> public void runSequential() {
> int exitVal = -101;
> try {
> exitVal = tsk.executeTask(ss == null ? null : ss.getHiveHistory());
> } catch (Throwable t) {
> if (tsk.getException() == null) {
> tsk.setException(t);
> }
> LOG.error("Error in executeTask", t);
> }
> result.setExitVal(exitVal);
> if (tsk.getException() != null) {
> result.setTaskError(tsk.getException());
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17193) HoS: don't combine map works that are targets of different DPPs
[
https://issues.apache.org/jira/browse/HIVE-17193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-17193:
--
Attachment: HIVE-17193.5.patch
> HoS: don't combine map works that are targets of different DPPs
> ---
>
> Key: HIVE-17193
> URL: https://issues.apache.org/jira/browse/HIVE-17193
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-17193.1.patch, HIVE-17193.2.patch,
> HIVE-17193.3.patch, HIVE-17193.4.patch, HIVE-17193.5.patch
>
>
> Suppose {{srcpart}} is partitioned by {{ds}}. The following query can trigger
> the issue:
> {code}
> explain
> select * from
> (select srcpart.ds,srcpart.key from srcpart join src on srcpart.ds=src.key)
> a
> join
> (select srcpart.ds,srcpart.key from srcpart join src on
> srcpart.ds=src.value) b
> on a.key=b.key;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail
[
https://issues.apache.org/jira/browse/HIVE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449531#comment-16449531
]
Hive QA commented on HIVE-19204:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920364/HIVE-19204.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 14296 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.metastore.TestCatalogNonDefaultClient.dropPartitions
(batchId=209)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar.org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar
(batchId=229)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10457/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10457/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10457/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 34 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12920364 - PreCommit-HIVE-Build
> Detailed errors from some tasks are not displayed to the client because the
> tasks don't set exception when they fail
>
>
>
[jira] [Commented] (HIVE-19054) Function replication shall use "hive.repl.replica.functions.root.dir" as root
[
https://issues.apache.org/jira/browse/HIVE-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449562#comment-16449562
]
Hive QA commented on HIVE-19054:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10458/dev-support/hive-personality.sh
|
| git revision | master / 56c3a95 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10458/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Function replication shall use "hive.repl.replica.functions.root.dir" as root
> -
>
> Key: HIVE-19054
> URL: https://issues.apache.org/jira/browse/HIVE-19054
> Project: Hive
> Issue Type: Bug
> Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19054.1.patch, HIVE-19054.2.patch
>
>
> It's wrongly use fs.defaultFS as the root, ignore
> "hive.repl.replica.functions.root.dir" definition, thus prevent replicating
> to cloud destination.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19211) New streaming ingest API and support for dynamic partitioning
[ https://issues.apache.org/jira/browse/HIVE-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19211: - Attachment: HIVE-19211.5.patch > New streaming ingest API and support for dynamic partitioning > - > > Key: HIVE-19211 > URL: https://issues.apache.org/jira/browse/HIVE-19211 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19211.1.patch, HIVE-19211.2.patch, > HIVE-19211.3.patch, HIVE-19211.4.patch, HIVE-19211.5.patch > > > - New streaming API under new hive sub-module > - Dynamic partitioning support > - Auto-rollover transactions > - Automatic heartbeating -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19211) New streaming ingest API and support for dynamic partitioning
[ https://issues.apache.org/jira/browse/HIVE-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449594#comment-16449594 ] Prasanth Jayachandran commented on HIVE-19211: -- Addressed review comments - API is now connect -> beginTransaction -> commitTransaction -> close. - catalog is not supported as some DDL's still doesn't support and some txns related metastore APIs doesn't accept catalog yet - normalized names and added fully qualified names to debug, error msgs - added default partition support when partition value is null or empty - moved heartbeater thread inside transaction batch - added more tests - remove TransactionBatch interface - agentInfo is now username + hostname + threadName, fallback is UUID > New streaming ingest API and support for dynamic partitioning > - > > Key: HIVE-19211 > URL: https://issues.apache.org/jira/browse/HIVE-19211 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19211.1.patch, HIVE-19211.2.patch, > HIVE-19211.3.patch, HIVE-19211.4.patch, HIVE-19211.5.patch > > > - New streaming API under new hive sub-module > - Dynamic partitioning support > - Auto-rollover transactions > - Automatic heartbeating -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19284) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora reassigned HIVE-19284: > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19284 > URL: https://issues.apache.org/jira/browse/HIVE-19284 > Project: Hive > Issue Type: Improvement >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19054) Function replication shall use "hive.repl.replica.functions.root.dir" as root
[
https://issues.apache.org/jira/browse/HIVE-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449642#comment-16449642
]
Hive QA commented on HIVE-19054:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12917274/HIVE-19054.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 14296 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_windowspec4]
(batchId=68)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.repl.load.message.TestPrimaryToReplicaResourceFunction.createDestinationPath
(batchId=280)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10458/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10458/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10458/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 34 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12917274 - PreCommit-HIVE-Build
> Function replication shall use "hive.repl.replica.functions.root.dir" as root
> -
>
> Key: HIVE-19054
> URL: https://issues.apache.org/jira/browse/HIVE-19054
> Project: Hive
> Issue Type: Bug
>
[jira] [Updated] (HIVE-18423) Hive should support usage of external tables using jdbc
[ https://issues.apache.org/jira/browse/HIVE-18423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Doron updated HIVE-18423: -- Status: In Progress (was: Patch Available) > Hive should support usage of external tables using jdbc > --- > > Key: HIVE-18423 > URL: https://issues.apache.org/jira/browse/HIVE-18423 > Project: Hive > Issue Type: Improvement >Reporter: Jonathan Doron >Assignee: Jonathan Doron >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-18423.1.patch, HIVE-18423.2.patch, > HIVE-18423.3.patch, HIVE-18423.4.patch, HIVE-18423.5.patch, HIVE-18423.6.patch > > > Hive should support the usage of external jdbc tables(and not only external > tables that hold queries), so an Hive user would be able to use the external > table as an hive internal table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18423) Hive should support usage of external tables using jdbc
[ https://issues.apache.org/jira/browse/HIVE-18423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Doron updated HIVE-18423: -- Status: Patch Available (was: In Progress) > Hive should support usage of external tables using jdbc > --- > > Key: HIVE-18423 > URL: https://issues.apache.org/jira/browse/HIVE-18423 > Project: Hive > Issue Type: Improvement >Reporter: Jonathan Doron >Assignee: Jonathan Doron >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-18423.1.patch, HIVE-18423.2.patch, > HIVE-18423.3.patch, HIVE-18423.4.patch, HIVE-18423.5.patch, HIVE-18423.6.patch > > > Hive should support the usage of external jdbc tables(and not only external > tables that hold queries), so an Hive user would be able to use the external > table as an hive internal table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19285) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-19285: - Component/s: HiveServer2 > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19285 > URL: https://issues.apache.org/jira/browse/HIVE-19285 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19285) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-19285: - Affects Version/s: 3.1.0 > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19285 > URL: https://issues.apache.org/jira/browse/HIVE-19285 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19285) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-19285: - Attachment: HIVE-19285.1.patch > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19285 > URL: https://issues.apache.org/jira/browse/HIVE-19285 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > Attachments: HIVE-19285.1.patch > > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work stopped] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-18988 stopped by Sankar Hariappan. --- > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Find the valid last replication state such that it points to event ID of > open_txn event of oldest on-going txn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19285) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora reassigned HIVE-19285: > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19285 > URL: https://issues.apache.org/jira/browse/HIVE-19285 > Project: Hive > Issue Type: Improvement >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19285) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-19285: - Status: Patch Available (was: Open) > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19285 > URL: https://issues.apache.org/jira/browse/HIVE-19285 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > Attachments: HIVE-19285.1.patch > > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19211) New streaming ingest API and support for dynamic partitioning
[ https://issues.apache.org/jira/browse/HIVE-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19211: - Attachment: HIVE-19211.6.patch > New streaming ingest API and support for dynamic partitioning > - > > Key: HIVE-19211 > URL: https://issues.apache.org/jira/browse/HIVE-19211 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19211.1.patch, HIVE-19211.2.patch, > HIVE-19211.3.patch, HIVE-19211.4.patch, HIVE-19211.5.patch, HIVE-19211.6.patch > > > - New streaming API under new hive sub-module > - Dynamic partitioning support > - Auto-rollover transactions > - Automatic heartbeating -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19285) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-19285: - Affects Version/s: (was: 3.1.0) 3.0.0 > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19285 > URL: https://issues.apache.org/jira/browse/HIVE-19285 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18988: Status: Patch Available (was: Open) Added 04.patch with * Logic to timeout the open txns which are opened before triggering bootstrap. * Replicate the write ids state in target based on validWriteIdlist for each ACID/MM table getting replicated. > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch, HIVE-18988.04.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Set the valid last replication state such that it doesn't miss any open > txn started after triggering bootstrap dump. > - If any txns on-going which was opened before triggering bootstrap dump, > then it is not guaranteed that if open_txn event captured for these txns. > Also, if these txns are opened for streaming ingest case, then dumped ACID > table data may include data of open txns which impact snapshot isolation at > target. To avoid that, bootstrap dump should wait for timeout (new > configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, > just force abort those txns and continue. > - If any txns force aborted belongs to a streaming ingest case, then dumped > ACID table data may have aborted data too. So, it is necessary to replicate > the aborted write ids to target to mark those data invalid for any readers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18988: Attachment: HIVE-18988.04.patch > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch, HIVE-18988.04.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Set the valid last replication state such that it doesn't miss any open > txn started after triggering bootstrap dump. > - If any txns on-going which was opened before triggering bootstrap dump, > then it is not guaranteed that if open_txn event captured for these txns. > Also, if these txns are opened for streaming ingest case, then dumped ACID > table data may include data of open txns which impact snapshot isolation at > target. To avoid that, bootstrap dump should wait for timeout (new > configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, > just force abort those txns and continue. > - If any txns force aborted belongs to a streaming ingest case, then dumped > ACID table data may have aborted data too. So, it is necessary to replicate > the aborted write ids to target to mark those data invalid for any readers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18988: Attachment: (was: HIVE-18988.04.patch) > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Set the valid last replication state such that it doesn't miss any open > txn started after triggering bootstrap dump. > - If any txns on-going which was opened before triggering bootstrap dump, > then it is not guaranteed that if open_txn event captured for these txns. > Also, if these txns are opened for streaming ingest case, then dumped ACID > table data may include data of open txns which impact snapshot isolation at > target. To avoid that, bootstrap dump should wait for timeout (new > configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, > just force abort those txns and continue. > - If any txns force aborted belongs to a streaming ingest case, then dumped > ACID table data may have aborted data too. So, it is necessary to replicate > the aborted write ids to target to mark those data invalid for any readers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18988: Description: Bootstrapping of ACID tables, need special handling to replicate a stable state of data. - If ACID feature enables, then perform bootstrap dump for ACID tables with in read txn. -> Dump table/partition metadata. -> Get the list of valid data files for a table using same logic as read txn do. -> Dump latest ValidWriteIdList as per current read txn. - Set the valid last replication state such that it doesn't miss any open txn started after triggering bootstrap dump. - If any txns on-going which was opened before triggering bootstrap dump, then it is not guaranteed that if open_txn event captured for these txns. Also, if these txns are opened for streaming ingest case, then dumped ACID table data may include data of open txns which impact snapshot isolation at target. To avoid that, bootstrap dump should wait for timeout (new configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, just force abort those txns and continue. - If any txns force aborted belongs to a streaming ingest case, then dumped ACID table data may have aborted data too. So, it is necessary to replicate the aborted write ids to target to mark those data invalid for any readers. was: Bootstrapping of ACID tables, need special handling to replicate a stable state of data. - If ACID feature enables, then perform bootstrap dump for ACID tables with in read txn. -> Dump table/partition metadata. -> Get the list of valid data files for a table using same logic as read txn do. -> Dump latest ValidWriteIdList as per current read txn. - Find the valid last replication state such that it points to event ID of open_txn event of oldest on-going txn. > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Set the valid last replication state such that it doesn't miss any open > txn started after triggering bootstrap dump. > - If any txns on-going which was opened before triggering bootstrap dump, > then it is not guaranteed that if open_txn event captured for these txns. > Also, if these txns are opened for streaming ingest case, then dumped ACID > table data may include data of open txns which impact snapshot isolation at > target. To avoid that, bootstrap dump should wait for timeout (new > configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, > just force abort those txns and continue. > - If any txns force aborted belongs to a streaming ingest case, then dumped > ACID table data may have aborted data too. So, it is necessary to replicate > the aborted write ids to target to mark those data invalid for any readers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-19286:
Description:
*STEP 1. Create test data*
{code}
DROP TABLE IF EXISTS customer_target;
DROP TABLE IF EXISTS customer_source;
{code}
{code}
CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
('transactional'='true');
{code}
{code}
insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
SELECT id, first_name, last_name, age FROM customer_target;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 002 | Michael | Watson | 27 |
| 001 | John| Smith | 45 |
| 003 | Den | Brown | 33 |
+--+-++--+
{code}
{code}
CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
age INT);
insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
SELECT id, first_name, last_name, age FROM customer_source;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 088 | Robert | Dowson | 25 |
| 001 | Dorothi | Hogward| 77 |
| 007 | Alex| Bowee | 1|
+--+-++--+
{code}
*STEP 2. Merge data*
{code}
merge into customer_target trg using customer_source src on src.id = trg.id
when matched then update set first_name = src.first_name, last_name =
src.last_name when not matched then insert values (src.id, src.first_name,
src.last_name, src.age);
{code}
*ACTUAL RESULT*
{code}
2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities: Deserializing
MapredLocalWork using kryo
2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
Starting to launch local task to process map join; maximum memory =
477626368
2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
trg, children = [HASHTABLESINK[37]]
2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for trg
created
2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
operator TS[0]
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
Done 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
initialized
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
children of 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
child 37 HASHTABLESINK
2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
operator HASHTABLESINK[37]
2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
JVM Max Heap Size: 477626368
2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
Map local work failed
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:153)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:366)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:556)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:411)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:391)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:764)
[jira] [Work started] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-19286 started by Oleksiy Sayankin.
---
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities:
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger: method=deserializePlan start=1524568304448 end=1524568304538 duration=90
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
> Starting to launch local task to process map join; maximum memory =
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for
> trg created
> 2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
> child 37 HASHTABLESINK
> 2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
> operator HASHTABLESINK[37]
> 2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
> JVM Max Heap Size: 477626368
> 2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
> Map local work failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
>
[jira] [Updated] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-19286:
Description:
*General Info*
Hive version : 2.3.3
{code}
commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
Author: Daniel Dai
Date: Wed Mar 28 16:46:29 2018 -0700
Preparing for 2.3.3 release
{code}
Hadoop version: 2.7.2.
*STEP 1. Create test data*
{code}
DROP TABLE IF EXISTS customer_target;
DROP TABLE IF EXISTS customer_source;
{code}
{code}
CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
('transactional'='true');
{code}
{code}
insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
SELECT id, first_name, last_name, age FROM customer_target;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 002 | Michael | Watson | 27 |
| 001 | John| Smith | 45 |
| 003 | Den | Brown | 33 |
+--+-++--+
{code}
{code}
CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
age INT);
insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
SELECT id, first_name, last_name, age FROM customer_source;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 088 | Robert | Dowson | 25 |
| 001 | Dorothi | Hogward| 77 |
| 007 | Alex| Bowee | 1|
+--+-++--+
{code}
*STEP 2. Merge data*
{code}
merge into customer_target trg using customer_source src on src.id = trg.id
when matched then update set first_name = src.first_name, last_name =
src.last_name when not matched then insert values (src.id, src.first_name,
src.last_name, src.age);
{code}
*ACTUAL RESULT*
{code}
2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities: Deserializing
MapredLocalWork using kryo
2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
Starting to launch local task to process map join; maximum memory =
477626368
2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
trg, children = [HASHTABLESINK[37]]
2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for trg
created
2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
operator TS[0]
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
Done 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
initialized
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
children of 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
child 37 HASHTABLESINK
2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
operator HASHTABLESINK[37]
2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
JVM Max Heap Size: 477626368
2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
Map local work failed
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:153)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:366)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:556)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:411)
[jira] [Assigned] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin reassigned HIVE-19286:
---
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities:
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger: method=deserializePlan start=1524568304448 end=1524568304538 duration=90
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
> Starting to launch local task to process map join; maximum memory =
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for
> trg created
> 2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
> child 37 HASHTABLESINK
> 2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
> operator HASHTABLESINK[37]
> 2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
> JVM Max Heap Size: 477626368
> 2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
> Map local work failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:153)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:366)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:556)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
>
[jira] [Updated] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-19286:
Affects Version/s: 2.3.3
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities:
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger: method=deserializePlan start=1524568304448 end=1524568304538 duration=90
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
> Starting to launch local task to process map join; maximum memory =
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for
> trg created
> 2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
> child 37 HASHTABLESINK
> 2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
> operator HASHTABLESINK[37]
> 2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
> JVM Max Heap Size: 477626368
> 2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
> Map local work failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
>
[jira] [Commented] (HIVE-18423) Hive should support usage of external tables using jdbc
[ https://issues.apache.org/jira/browse/HIVE-18423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449716#comment-16449716 ] Jonathan Doron commented on HIVE-18423: --- [~jcamachorodriguez] I have fixed all failed test, please review. > Hive should support usage of external tables using jdbc > --- > > Key: HIVE-18423 > URL: https://issues.apache.org/jira/browse/HIVE-18423 > Project: Hive > Issue Type: Improvement >Reporter: Jonathan Doron >Assignee: Jonathan Doron >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-18423.1.patch, HIVE-18423.2.patch, > HIVE-18423.3.patch, HIVE-18423.4.patch, HIVE-18423.5.patch, HIVE-18423.6.patch > > > Hive should support the usage of external jdbc tables(and not only external > tables that hold queries), so an Hive user would be able to use the external > table as an hive internal table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter
[
https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16295:
Attachment: HIVE-16295.2.WIP.patch
> Add support for using Hadoop's S3A OutputCommitter
> --
>
> Key: HIVE-16295
> URL: https://issues.apache.org/jira/browse/HIVE-16295
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-16295.1.WIP.patch, HIVE-16295.2.WIP.patch
>
>
> Hive doesn't have integration with Hadoop's {{OutputCommitter}}, it uses a
> {{NullOutputCommitter}} and uses its own commit logic spread across
> {{FileSinkOperator}}, {{MoveTask}}, and {{Hive}}.
> The Hadoop community is building an {{OutputCommitter}} that integrates with
> S3Guard and does a safe, coordinate commit of data on S3 inside individual
> tasks (HADOOP-13786). If Hive can integrate with this new {{OutputCommitter}}
> there would be a lot of benefits to Hive-on-S3:
> * Data is only written once; directly committing data at a task level means
> no renames are necessary
> * The commit is done safely, in a coordinated manner; duplicate tasks (from
> task retries or speculative execution) should not step on each other
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19252) TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently
[
https://issues.apache.org/jira/browse/HIVE-19252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449654#comment-16449654
]
Hive QA commented on HIVE-19252:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10459/dev-support/hive-personality.sh
|
| git revision | master / 56c3a95 |
| Default Java | 1.8.0_111 |
| modules | C: itests/hive-minikdc U: itests/hive-minikdc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10459/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently
>
>
> Key: HIVE-19252
> URL: https://issues.apache.org/jira/browse/HIVE-19252
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Ashutosh Chauhan
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-19252.1.patch, HIVE-19252.1.patch
>
>
> For last 8 builds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-19286:
Description:
*General Info*
Hive version : 2.3.3
{code}
commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
Author: Daniel Dai
Date: Wed Mar 28 16:46:29 2018 -0700
Preparing for 2.3.3 release
{code}
Hadoop version: 2.7.2.
Engine
{code}
hive> set hive.execution.engine;
hive.execution.engine=mr
{code}
*STEP 1. Create test data*
{code}
DROP TABLE IF EXISTS customer_target;
DROP TABLE IF EXISTS customer_source;
{code}
{code}
CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
('transactional'='true');
{code}
{code}
insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
SELECT id, first_name, last_name, age FROM customer_target;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 002 | Michael | Watson | 27 |
| 001 | John| Smith | 45 |
| 003 | Den | Brown | 33 |
+--+-++--+
{code}
{code}
CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
age INT);
insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
SELECT id, first_name, last_name, age FROM customer_source;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 088 | Robert | Dowson | 25 |
| 001 | Dorothi | Hogward| 77 |
| 007 | Alex| Bowee | 1|
+--+-++--+
{code}
*STEP 2. Merge data*
{code}
merge into customer_target trg using customer_source src on src.id = trg.id
when matched then update set first_name = src.first_name, last_name =
src.last_name when not matched then insert values (src.id, src.first_name,
src.last_name, src.age);
{code}
*ACTUAL RESULT*
{code}
2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities: Deserializing
MapredLocalWork using kryo
2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
Starting to launch local task to process map join; maximum memory =
477626368
2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
trg, children = [HASHTABLESINK[37]]
2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for trg
created
2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
operator TS[0]
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
Done 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
initialized
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
children of 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
child 37 HASHTABLESINK
2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
operator HASHTABLESINK[37]
2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
JVM Max Heap Size: 477626368
2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
Map local work failed
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:153)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:366)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:556)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
[jira] [Updated] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-19286:
Description:
*General Info*
Hive version : 2.3.3
{code}
commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
Author: Daniel Dai
Date: Wed Mar 28 16:46:29 2018 -0700
Preparing for 2.3.3 release
{code}
Hadoop version: 2.7.2.
*STEP 1. Create test data*
{code}
DROP TABLE IF EXISTS customer_target;
DROP TABLE IF EXISTS customer_source;
{code}
{code}
CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
('transactional'='true');
{code}
{code}
insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
SELECT id, first_name, last_name, age FROM customer_target;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 002 | Michael | Watson | 27 |
| 001 | John| Smith | 45 |
| 003 | Den | Brown | 33 |
+--+-++--+
{code}
{code}
CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
age INT);
insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
SELECT id, first_name, last_name, age FROM customer_source;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 088 | Robert | Dowson | 25 |
| 001 | Dorothi | Hogward| 77 |
| 007 | Alex| Bowee | 1|
+--+-++--+
{code}
*STEP 2. Merge data*
{code}
merge into customer_target trg using customer_source src on src.id = trg.id
when matched then update set first_name = src.first_name, last_name =
src.last_name when not matched then insert values (src.id, src.first_name,
src.last_name, src.age);
{code}
*ACTUAL RESULT*
{code}
2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities: Deserializing
MapredLocalWork using kryo
2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
Starting to launch local task to process map join; maximum memory =
477626368
2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
trg, children = [HASHTABLESINK[37]]
2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for trg
created
2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
operator TS[0]
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
Done 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
initialized
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
children of 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
child 37 HASHTABLESINK
2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
operator HASHTABLESINK[37]
2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
JVM Max Heap Size: 477626368
2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
Map local work failed
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:153)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:366)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:556)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:411)
[jira] [Commented] (HIVE-19252) TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently
[
https://issues.apache.org/jira/browse/HIVE-19252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449738#comment-16449738
]
Hive QA commented on HIVE-19252:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920370/HIVE-19252.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 36 failed/errored test(s), 14296 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testAutoPurgeUnset (batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testPartitionedNoAutoPurge
(batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testTruncateInvalidAutoPurge
(batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testTruncateNoAutoPurge
(batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testTruncatePartitionedNoAutoPurge
(batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testTruncateUnsetAutoPurge
(batchId=233)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill
(batchId=242)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10459/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10459/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10459/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 36 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12920370 - PreCommit-HIVE-Build
> TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently
>
>
> Key: HIVE-19252
> URL:
[jira] [Commented] (HIVE-19247) StatsOptimizer: Missing stats fast-path for Date
[
https://issues.apache.org/jira/browse/HIVE-19247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450032#comment-16450032
]
Hive QA commented on HIVE-19247:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920372/HIVE-19247.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 14297 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_decimal_date]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_columnPruning]
(batchId=83)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.addNoSuchTable[Remote]
(batchId=211)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testDropTableWithoutPurge[Embedded]
(batchId=209)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10461/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10461/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10461/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12920372 - PreCommit-HIVE-Build
> StatsOptimizer: Missing stats fast-path for Date
>
>
> Key: HIVE-19247
> URL: https://issues.apache.org/jira/browse/HIVE-19247
> Project: Hive
>
[jira] [Commented] (HIVE-19283) Select count(distinct()) a couple of times stuck in last reducer
[
https://issues.apache.org/jira/browse/HIVE-19283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450041#comment-16450041
]
Gopal V commented on HIVE-19283:
This was fixed sometime during hive-3.x
https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/tez/multi_count_distinct.q.out
> Select count(distinct()) a couple of times stuck in last reducer
>
>
> Key: HIVE-19283
> URL: https://issues.apache.org/jira/browse/HIVE-19283
> Project: Hive
> Issue Type: Improvement
> Components: CBO, Logical Optimizer
>Affects Versions: 2.1.1
>Reporter: Goun Na
>Assignee: Ashutosh Chauhan
>Priority: Major
>
> Distinct count query performance is significantly improved due to
> HIVE-10568.
> {code:java}
> select count(distinct elevenst_id)
> from 11st.log_table
> where part_dt between '20180101' and '20180131'{code}
>
> However, some queries with several distinct counts are still slow. It starts
> with multiple mappers, but stuck in the last one reducer.
> {code:java}
> select
> count(distinct elevenst_id)
> , count(distinct member_id)
> , count(distinct user_id)
> , count(distinct action_id)
> , count(distinct other_id)
> from 11st.log_table
> where part_dt between '20180101' and '20180131'{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450045#comment-16450045
]
Oleksiy Sayankin commented on HIVE-19286:
-
PS : the above multiple {{INSERT}} query does not work even on
{{TextInputFormat}} tables with the same error:
{code}
FAILED: SemanticException [Error 10087]: The same output cannot be present
multiple times: customer_target
{code}
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities:
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger: method=deserializePlan start=1524568304448 end=1524568304538 duration=90
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
> Starting to launch local task to process map join; maximum memory =
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for
> trg created
> 2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
> child 37 HASHTABLESINK
> 2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
> operator HASHTABLESINK[37]
> 2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
> JVM Max Heap Size: 477626368
> 2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
> Map local work failed
> java.lang.NullPointerException
>
[jira] [Commented] (HIVE-19184) Hive 3.0.0 release branch preparation
[
https://issues.apache.org/jira/browse/HIVE-19184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450049#comment-16450049
]
Hive QA commented on HIVE-19184:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10462/dev-support/hive-personality.sh
|
| git revision | master / 56c3a95 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10462/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive 3.0.0 release branch preparation
> -
>
> Key: HIVE-19184
> URL: https://issues.apache.org/jira/browse/HIVE-19184
> Project: Hive
> Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19184.01-branch-3.patch
>
>
> Need to do bunch of things to prepare branch-3 for release e.g.
> * Update pom.xml to delete SNAPSHOT
> * Update .reviewboardrc
> * Remove storage-api module to build
> * Change storage-api depdency etc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449820#comment-16449820
]
Oleksiy Sayankin edited comment on HIVE-19286 at 4/24/18 1:15 PM:
--
My results after some debug. NPE happens because
{{inspector.getStructFieldRef(names[0]);}} returns {{null}}
{code}
@Override
public ObjectInspector initialize(ObjectInspector rowInspector) throws
HiveException {
// We need to support field names like KEY.0, VALUE.1 between
// map-reduce boundary.
String[] names = expr.getColumn().split("\\.");
String[] unionfields = names[0].split("\\:");
if (names.length == 1 && unionfields.length == 1) {
simpleCase = true;
inspector = (StructObjectInspector) rowInspector;
field = inspector.getStructFieldRef(names[0]);
return outputOI = field.getFieldObjectInspector();
}
{code}
in {{ExprNodeColumnEvaluator}}. Here {{names[0]}} == {{"ROW__ID"}}. Class
{{OrcStruct}} contains method
{code}
@Override
public StructField getStructFieldRef(String s) {
for(StructField field: fields) {
if (field.getFieldName().equalsIgnoreCase(s)) {
return field;
}
}
return null;
}
{code}
and array {{fields}} is initialized with {{StructField}} for only four columns:
{{id}}, {{first_name}}, {{last_name}}, {{age}}. So it returns {{null}} when
{{s}} == {{"ROW__ID"}} . Hive wants no insert {{"ROW__ID"}} because it
transforms {{MERGE}} into multiple {{INSERT}}:
{code}
FROM
`default`.`customer_target` `trg`
RIGHT OUTER JOIN
`default`.`customer_source` `src`
ON `src`.`id` = `trg`.`id`
INSERT INTO `default`.`customer_target`-- update clause
select `trg`.ROW__ID, `trg`.`id`, `src`.`first_name`, `src`.`last_name`,
`trg`.`age`
WHERE `src`.`id` = `trg`.`id`
sort by `trg`.ROW__ID
INSERT INTO `default`.`customer_target`-- insert clause
select `src`.`id`, `src`.`first_name`, `src`.`last_name`, `src`.`age`
WHERE `trg`.`id` IS NULL
INSERT INTO merge_tmp_table
SELECT cardinality_violation(`trg`.ROW__ID)
WHERE `src`.`id` = `trg`.`id` GROUP BY `trg`.ROW__ID HAVING count(*) > 1
{code}
was (Author: osayankin):
My results after some debug. NPE happens because
{{inspector.getStructFieldRef(names[0]);}} returns {{null}}
{code}
@Override
public ObjectInspector initialize(ObjectInspector rowInspector) throws
HiveException {
// We need to support field names like KEY.0, VALUE.1 between
// map-reduce boundary.
String[] names = expr.getColumn().split("\\.");
String[] unionfields = names[0].split("\\:");
if (names.length == 1 && unionfields.length == 1) {
simpleCase = true;
inspector = (StructObjectInspector) rowInspector;
field = inspector.getStructFieldRef(names[0]);
return outputOI = field.getFieldObjectInspector();
}
{code}
in {{ExprNodeColumnEvaluator}}. Here {{names[0] == "ROW__ID"}}. Class
{{OrcStruct}} contains method
{code}
@Override
public StructField getStructFieldRef(String s) {
for(StructField field: fields) {
if (field.getFieldName().equalsIgnoreCase(s)) {
return field;
}
}
return null;
}
{code}
and array {{fields}} is initialized with {{StructField}} for only four columns:
{{id}}, {{first_name}}, {{last_name}}, {{age}}. So it returns {{null}} when {{s
== "ROW__ID"}}. Hive wants no insert {{ROW__ID}} because it transforms
{{MERGE}} into multiple {{INSERT}}:
{code}
FROM
`default`.`customer_target` `trg`
RIGHT OUTER JOIN
`default`.`customer_source` `src`
ON `src`.`id` = `trg`.`id`
INSERT INTO `default`.`customer_target`-- update clause
select `trg`.ROW__ID, `trg`.`id`, `src`.`first_name`, `src`.`last_name`,
`trg`.`age`
WHERE `src`.`id` = `trg`.`id`
sort by `trg`.ROW__ID
INSERT INTO `default`.`customer_target`-- insert clause
select `src`.`id`, `src`.`first_name`, `src`.`last_name`, `src`.`age`
WHERE `trg`.`id` IS NULL
INSERT INTO merge_tmp_table
SELECT cardinality_violation(`trg`.ROW__ID)
WHERE `src`.`id` = `trg`.`id` GROUP BY `trg`.ROW__ID HAVING count(*) > 1
{code}
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1.
[jira] [Assigned] (HIVE-19287) parse error with semicolon in comment in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei reassigned HIVE-19287: - > parse error with semicolon in comment in file > - > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19287) parse error with semicolon in comment in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449912#comment-16449912 ] Laszlo Bodor commented on HIVE-19287: - [~qunyan] : the issue title could be a bit misleading, as the patch fixes the case when commented line starts with whitespace, so it's not about the semicolon, isn't it? > parse error with semicolon in comment in file > - > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
