[jira] [Commented] (HIVE-19725) Add ability to dump non-native tables in replication metadata dump
[
https://issues.apache.org/jira/browse/HIVE-19725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515430#comment-16515430
]
Hive QA commented on HIVE-19725:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11881/dev-support/hive-personality.sh
|
| git revision | master / ebd2c5f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/hive-unit ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11881/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add ability to dump non-native tables in replication metadata dump
> --
>
> Key: HIVE-19725
> URL: https://issues.apache.org/jira/browse/HIVE-19725
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: Repl, pull-request-available
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19725.01.patch, HIVE-19725.02.patch,
> HIVE-19725.03.patch, HIVE-19725.04.patch, HIVE-19725.05.patch,
> HIVE-19725.06-branch-3.patch, HIVE-19725.07.patch
>
>
> if hive.repl.dump.metadata.only is set to true, allow dumping non native
> tables also.
> Data dump for non-native tables should never be allowed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19909) qtests: retire hadoop_major version specific tests; and logics
[ https://issues.apache.org/jira/browse/HIVE-19909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515465#comment-16515465 ] Teddy Choi commented on HIVE-19909: --- LGTM +1. > qtests: retire hadoop_major version specific tests; and logics > -- > > Key: HIVE-19909 > URL: https://issues.apache.org/jira/browse/HIVE-19909 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19909.01.patch > > > these are just hanging around in QTestUtil...they don't even in use...since > they are looking for hadoop versions like "0.23" and "0.20" ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19530) Vectorization: Fix JDBCSerde and re-enable vectorization
[ https://issues.apache.org/jira/browse/HIVE-19530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515467#comment-16515467 ] Teddy Choi commented on HIVE-19530: --- LGTM +1, tests pending. > Vectorization: Fix JDBCSerde and re-enable vectorization > > > Key: HIVE-19530 > URL: https://issues.apache.org/jira/browse/HIVE-19530 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Major > Attachments: HIVE-19530.01.patch, HIVE-19530.02.patch > > > According to [~jcamachorodriguez] there is a big switch statement in the code > that has might have missing types. This can lead to the string types seen. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19725) Add ability to dump non-native tables in replication metadata dump
[
https://issues.apache.org/jira/browse/HIVE-19725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515476#comment-16515476
]
Hive QA commented on HIVE-19725:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12928073/HIVE-19725.07.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14546 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11881/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11881/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11881/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12928073 - PreCommit-HIVE-Build
> Add ability to dump non-native tables in replication metadata dump
> --
>
> Key: HIVE-19725
> URL: https://issues.apache.org/jira/browse/HIVE-19725
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: Repl, pull-request-available
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19725.01.patch, HIVE-19725.02.patch,
> HIVE-19725.03.patch, HIVE-19725.04.patch, HIVE-19725.05.patch,
> HIVE-19725.06-branch-3.patch, HIVE-19725.07.patch
>
>
> if hive.repl.dump.metadata.only is set to true, allow dumping non native
> tables also.
> Data dump for non-native tables should never be allowed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19890) ACID: Inherit bucket-id from original ROW_ID for delete deltas
[
https://issues.apache.org/jira/browse/HIVE-19890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515493#comment-16515493
]
Hive QA commented on HIVE-19890:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 1 new + 695 unchanged - 0
fixed = 696 total (was 695) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11882/dev-support/hive-personality.sh
|
| git revision | master / ebd2c5f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11882/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11882/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: Inherit bucket-id from original ROW_ID for delete deltas
> --
>
> Key: HIVE-19890
> URL: https://issues.apache.org/jira/browse/HIVE-19890
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-19890.1.patch
>
>
> The ACID delete deltas for unbucketed tables are written to arbitrary files,
> which should instead be shuffled using the bucket-id instead of hash(ROW__ID).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19909) qtests: retire hadoop_major version specific tests; and logics
[ https://issues.apache.org/jira/browse/HIVE-19909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19909: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) pushed to master. Thank you Teddy for reviewing the changes! > qtests: retire hadoop_major version specific tests; and logics > -- > > Key: HIVE-19909 > URL: https://issues.apache.org/jira/browse/HIVE-19909 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19909.01.patch > > > these are just hanging around in QTestUtil...they don't even in use...since > they are looking for hadoop versions like "0.23" and "0.20" ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19882) Fix QTestUtil session lifecycle
[
https://issues.apache.org/jira/browse/HIVE-19882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19882:
Attachment: HIVE-19882.03.patch
> Fix QTestUtil session lifecycle
> ---
>
> Key: HIVE-19882
> URL: https://issues.apache.org/jira/browse/HIVE-19882
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19882.01.patch, HIVE-19882.02.patch,
> HIVE-19882.03.patch
>
>
> there are a number of strange come and go failing tests; it was always
> strange to me that qtestutil cleans up at some questionable points - this
> seems to be leading to executing some commands with the previous qfiles
> session...
> ideally the session/etc should start/reused in {{before}}
> and it should be closed in {{after}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19896) Create view stores invalid, malformed version of original SQL text
[
https://issues.apache.org/jira/browse/HIVE-19896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliot West updated HIVE-19896:
---
Target Version/s: 2.3.3, 3.0.1 (was: 2.3.3)
> Create view stores invalid, malformed version of original SQL text
> --
>
> Key: HIVE-19896
> URL: https://issues.apache.org/jira/browse/HIVE-19896
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 1.2.1, 2.3.0
>Reporter: Elliot West
>Priority: Major
>
> We perform additional processing on some of our view definitions and we've
> found that in some cases Hive actually stores malformed representations of
> the original source SQL text definition of the view. We have examples where
> the stored original view text is in fact an unparsable statement. This
> prevents later processing of the view (recompiling for column adjustments for
> example) and is counter intuitive to the user.
> Specifically the failure appears to be focused on the removal of necessary
> back-tick quoting from the original text on aliases (but possibly other
> identifiers).
> h2. Steps to reproduce (abridged/formatted for clarity):
> Create a view that is subject to malformation:
> {code}
> hive> create view v as select * from (select 1) as `a.b`;
> OK
> {code}
> Describe view definition to obtain original text:
> {code}
> describe extended v;
> ...
> Detailed Table InformationTable(
> tableName:v,
> dbName:default,
> viewOriginalText:select * from (select 1) as a.b,
> viewExpandedText:select `a.b`.`_c0` from (select 1) as `a.b`,
> tableType:VIRTUAL_VIEW)
> {code}
> Note that {{viewOriginalText}} value does not equal the text supplied on the
> first step, specifically {{`a.b`}} → {{a.b}}.
> Executing {{viewOriginalText}} value results in failure:
> {code}
> hive> select * from (select 1) as a.b;
> NoViableAltException(17@[215:51: ( KW_AS )?])
> at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
> ...
> FAILED: ParseException line 1:29 cannot recognize input near 'as' 'a' '.' in
> subquery source
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19896) Create view stores invalid, malformed version of original SQL text
[
https://issues.apache.org/jira/browse/HIVE-19896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliot West updated HIVE-19896:
---
Target Version/s: 2.3.3 (was: 2.3.2)
> Create view stores invalid, malformed version of original SQL text
> --
>
> Key: HIVE-19896
> URL: https://issues.apache.org/jira/browse/HIVE-19896
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 1.2.1, 2.3.0
>Reporter: Elliot West
>Priority: Major
>
> We perform additional processing on some of our view definitions and we've
> found that in some cases Hive actually stores malformed representations of
> the original source SQL text definition of the view. We have examples where
> the stored original view text is in fact an unparsable statement. This
> prevents later processing of the view (recompiling for column adjustments for
> example) and is counter intuitive to the user.
> Specifically the failure appears to be focused on the removal of necessary
> back-tick quoting from the original text on aliases (but possibly other
> identifiers).
> h2. Steps to reproduce (abridged/formatted for clarity):
> Create a view that is subject to malformation:
> {code}
> hive> create view v as select * from (select 1) as `a.b`;
> OK
> {code}
> Describe view definition to obtain original text:
> {code}
> describe extended v;
> ...
> Detailed Table InformationTable(
> tableName:v,
> dbName:default,
> viewOriginalText:select * from (select 1) as a.b,
> viewExpandedText:select `a.b`.`_c0` from (select 1) as `a.b`,
> tableType:VIRTUAL_VIEW)
> {code}
> Note that {{viewOriginalText}} value does not equal the text supplied on the
> first step, specifically {{`a.b`}} → {{a.b}}.
> Executing {{viewOriginalText}} value results in failure:
> {code}
> hive> select * from (select 1) as a.b;
> NoViableAltException(17@[215:51: ( KW_AS )?])
> at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
> ...
> FAILED: ParseException line 1:29 cannot recognize input near 'as' 'a' '.' in
> subquery source
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19882) Fix QTestUtil session lifecycle
[
https://issues.apache.org/jira/browse/HIVE-19882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19882:
Description:
there are a number of strange come and go failing tests; it was always strange
to me that qtestutil cleans up at some questionable points - this seems to be
leading to executing some commands with the previous qfiles session...
ideally the session/etc should start/reused in {{before}}
and it should be closed in {{after}}
seems like configuration is handled probably incorrectly; saving the conf after
initialization - and restoring it for a new session should ensure consistency
was:
there are a number of strange come and go failing tests; it was always strange
to me that qtestutil cleans up at some questionable points - this seems to be
leading to executing some commands with the previous qfiles session...
ideally the session/etc should start/reused in {{before}}
and it should be closed in {{after}}
> Fix QTestUtil session lifecycle
> ---
>
> Key: HIVE-19882
> URL: https://issues.apache.org/jira/browse/HIVE-19882
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19882.01.patch, HIVE-19882.02.patch,
> HIVE-19882.03.patch
>
>
> there are a number of strange come and go failing tests; it was always
> strange to me that qtestutil cleans up at some questionable points - this
> seems to be leading to executing some commands with the previous qfiles
> session...
> ideally the session/etc should start/reused in {{before}}
> and it should be closed in {{after}}
> seems like configuration is handled probably incorrectly; saving the conf
> after initialization - and restoring it for a new session should ensure
> consistency
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19267) Create/Replicate ACID Write event
[ https://issues.apache.org/jira/browse/HIVE-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19267: --- Attachment: HIVE-19267.17.patch > Create/Replicate ACID Write event > - > > Key: HIVE-19267 > URL: https://issues.apache.org/jira/browse/HIVE-19267 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Attachments: HIVE-19267.01.patch, HIVE-19267.02.patch, > HIVE-19267.03.patch, HIVE-19267.04.patch, HIVE-19267.05.patch, > HIVE-19267.06.patch, HIVE-19267.07.patch, HIVE-19267.08.patch, > HIVE-19267.09.patch, HIVE-19267.10.patch, HIVE-19267.11.patch, > HIVE-19267.12.patch, HIVE-19267.13.patch, HIVE-19267.14.patch, > HIVE-19267.15.patch, HIVE-19267.16.patch, HIVE-19267.17.patch > > > > h1. Replicate ACID write Events > * Create new EVENT_WRITE event with related message format to log the write > operations with in a txn along with data associated. > * Log this event when perform any writes (insert into, insert overwrite, > load table, delete, update, merge, truncate) on table/partition. > * If a single MERGE/UPDATE/INSERT/DELETE statement operates on multiple > partitions, then need to log one event per partition. > * DbNotificationListener should log this type of event to special metastore > table named "MTxnWriteNotificationLog". > * This table should maintain a map of txn ID against list of > tables/partitions written by given txn. > * The entry for a given txn should be removed by the cleaner thread that > removes the expired events from EventNotificationTable. > h1. Replicate Commit Txn operation (with writes) > Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions > modified within the txn. > *Source warehouse:* > * This event should read the EVENT_WRITEs from "MTxnWriteNotificationLog" > metastore table to consolidate the list of tables/partitions modified within > this txn scope. > * Based on the list of tables/partitions modified and table Write ID, need > to compute the list of delta files added by this txn. > * Repl dump should read this message and dump the metadata and delta files > list. > *Target warehouse:* > * Ensure snapshot isolation at target for on-going read txns which shouldn't > view the data replicated from committed txn. (Ensured with open and allocate > write ID events). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19890) ACID: Inherit bucket-id from original ROW_ID for delete deltas
[
https://issues.apache.org/jira/browse/HIVE-19890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515521#comment-16515521
]
Hive QA commented on HIVE-19890:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12928074/HIVE-19890.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 14544 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=158)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testMultiInsertStatement
(batchId=303)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNoBuckets (batchId=291)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversionMultiBucket
(batchId=291)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testNoBuckets (batchId=291)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testToAcidConversionMultiBucket
(batchId=291)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11882/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11882/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11882/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12928074 - PreCommit-HIVE-Build
> ACID: Inherit bucket-id from original ROW_ID for delete deltas
> --
>
> Key: HIVE-19890
> URL: https://issues.apache.org/jira/browse/HIVE-19890
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-19890.1.patch
>
>
> The ACID delete deltas for unbucketed tables are written to arbitrary files,
> which should instead be shuffled using the bucket-id instead of hash(ROW__ID).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19882) Fix QTestUtil session lifecycle
[
https://issues.apache.org/jira/browse/HIVE-19882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19882:
Attachment: HIVE-19882.04.patch
> Fix QTestUtil session lifecycle
> ---
>
> Key: HIVE-19882
> URL: https://issues.apache.org/jira/browse/HIVE-19882
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19882.01.patch, HIVE-19882.02.patch,
> HIVE-19882.03.patch, HIVE-19882.04.patch
>
>
> there are a number of strange come and go failing tests; it was always
> strange to me that qtestutil cleans up at some questionable points - this
> seems to be leading to executing some commands with the previous qfiles
> session...
> ideally the session/etc should start/reused in {{before}}
> and it should be closed in {{after}}
> seems like configuration is handled probably incorrectly; saving the conf
> after initialization - and restoring it for a new session should ensure
> consistency
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18545) Add UDF to parse complex types from json
[ https://issues.apache.org/jira/browse/HIVE-18545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18545: Attachment: HIVE-18545.05.patch > Add UDF to parse complex types from json > > > Key: HIVE-18545 > URL: https://issues.apache.org/jira/browse/HIVE-18545 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-18545.02.patch, HIVE-18545.03.patch, > HIVE-18545.04.patch, HIVE-18545.05.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515552#comment-16515552
]
Hive QA commented on HIVE-19921:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11883/dev-support/hive-personality.sh
|
| git revision | master / 4ec256c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11883/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix perf duration and queue name in HiveProtoLoggingHook
>
>
> Key: HIVE-19921
> URL: https://issues.apache.org/jira/browse/HIVE-19921
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-19921.01.patch
>
>
> The perf log should return duration instead of end time.
> The queue name should be llap queue for llap queries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515574#comment-16515574
]
Hive QA commented on HIVE-19921:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12928081/HIVE-19921.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14531 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11883/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11883/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11883/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12928081 - PreCommit-HIVE-Build
> Fix perf duration and queue name in HiveProtoLoggingHook
>
>
> Key: HIVE-19921
> URL: https://issues.apache.org/jira/browse/HIVE-19921
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-19921.01.patch
>
>
> The perf log should return duration instead of end time.
> The queue name should be llap queue for llap queries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18140) Partitioned tables statistics can go wrong in basic stats mixed case
[
https://issues.apache.org/jira/browse/HIVE-18140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18140:
Attachment: HIVE-18140.02.patch
> Partitioned tables statistics can go wrong in basic stats mixed case
>
>
> Key: HIVE-18140
> URL: https://issues.apache.org/jira/browse/HIVE-18140
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18140.01.patch, HIVE-18140.01wip01.patch,
> HIVE-18140.01wip03.patch, HIVE-18140.01wip04.patch, HIVE-18140.02.patch,
> HIVE-18140.02.patch, HIVE-18140.02wip01.patch, HIVE-19140.02wip02.patch,
> HIVE-19727.02wip03.patch
>
>
> suppose the following scenario:
> * part1 has basic stats {{RC=10,DS=1K}}
> * all other partition has no basic stats (and a bunch of rows)
> then
> [this|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L378]
> condition would be false; which in turn produces estimations for the whole
> partitioned table: {{RC=10,DS=1K}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18140) Partitioned tables statistics can go wrong in basic stats mixed case
[
https://issues.apache.org/jira/browse/HIVE-18140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18140:
Attachment: (was: HIVE-18140.02.patch)
> Partitioned tables statistics can go wrong in basic stats mixed case
>
>
> Key: HIVE-18140
> URL: https://issues.apache.org/jira/browse/HIVE-18140
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18140.01.patch, HIVE-18140.01wip01.patch,
> HIVE-18140.01wip03.patch, HIVE-18140.01wip04.patch, HIVE-18140.02.patch,
> HIVE-18140.02wip01.patch, HIVE-18140.03.patch, HIVE-19140.02wip02.patch,
> HIVE-19727.02wip03.patch
>
>
> suppose the following scenario:
> * part1 has basic stats {{RC=10,DS=1K}}
> * all other partition has no basic stats (and a bunch of rows)
> then
> [this|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L378]
> condition would be false; which in turn produces estimations for the whole
> partitioned table: {{RC=10,DS=1K}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18140) Partitioned tables statistics can go wrong in basic stats mixed case
[
https://issues.apache.org/jira/browse/HIVE-18140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18140:
Attachment: HIVE-18140.03.patch
> Partitioned tables statistics can go wrong in basic stats mixed case
>
>
> Key: HIVE-18140
> URL: https://issues.apache.org/jira/browse/HIVE-18140
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18140.01.patch, HIVE-18140.01wip01.patch,
> HIVE-18140.01wip03.patch, HIVE-18140.01wip04.patch, HIVE-18140.02.patch,
> HIVE-18140.02wip01.patch, HIVE-18140.03.patch, HIVE-19140.02wip02.patch,
> HIVE-19727.02wip03.patch
>
>
> suppose the following scenario:
> * part1 has basic stats {{RC=10,DS=1K}}
> * all other partition has no basic stats (and a bunch of rows)
> then
> [this|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L378]
> condition would be false; which in turn produces estimations for the whole
> partitioned table: {{RC=10,DS=1K}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19922) TestMiniDruidKafkaCliDriver[druidkafkamini_basic started] to fail
[
https://issues.apache.org/jira/browse/HIVE-19922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515589#comment-16515589
]
Hive QA commented on HIVE-19922:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 24 unchanged - 3 fixed
= 25 total (was 27) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11884/dev-support/hive-personality.sh
|
| git revision | master / 4ec256c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11884/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11884/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestMiniDruidKafkaCliDriver[druidkafkamini_basic started] to fail
> -
>
> Key: HIVE-19922
> URL: https://issues.apache.org/jira/browse/HIVE-19922
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Priority: Major
> Attachments: HIVE-19922.patch
>
>
> Consistently failing in the last 4 runs.
> See:
> [https://builds.apache.org/job/PreCommit-HIVE-Build/11824/testReport/org.apache.hadoop.hive.cli/TestMiniDruidKafkaCliDriver/testCliDriver_druidkafkamini_basic_/history/]
> Can not reproduce the failure locally :(
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19880) Repl Load to return recoverable vs non-recoverable error codes
[ https://issues.apache.org/jira/browse/HIVE-19880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19880: --- Attachment: HIVE-19880.04-branch-3.patch > Repl Load to return recoverable vs non-recoverable error codes > -- > > Key: HIVE-19880 > URL: https://issues.apache.org/jira/browse/HIVE-19880 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19880.01.patch, HIVE-19880.04-branch-3.patch, > HIVE-19880.04.patch > > > To enable bootstrap of large databases, application has to have the ability > to keep retrying the bootstrap load till it encounters a fatal error. The > ability to identify if an error is fatal or not will be decided by hive and > communication of the same will happen to application via error codes. > So there should be different error codes for recoverable vs non-recoverable > failures which should be propagated to application as part of running the > repl load command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19725) Add ability to dump non-native tables in replication metadata dump
[ https://issues.apache.org/jira/browse/HIVE-19725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19725: --- Attachment: HIVE-19725.07-branch-3.patch > Add ability to dump non-native tables in replication metadata dump > -- > > Key: HIVE-19725 > URL: https://issues.apache.org/jira/browse/HIVE-19725 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.0.0, 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: Repl, pull-request-available > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19725.01.patch, HIVE-19725.02.patch, > HIVE-19725.03.patch, HIVE-19725.04.patch, HIVE-19725.05.patch, > HIVE-19725.06-branch-3.patch, HIVE-19725.07-branch-3.patch, > HIVE-19725.07.patch > > > if hive.repl.dump.metadata.only is set to true, allow dumping non native > tables also. > Data dump for non-native tables should never be allowed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19922) TestMiniDruidKafkaCliDriver[druidkafkamini_basic started] to fail
[ https://issues.apache.org/jira/browse/HIVE-19922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514801#comment-16514801 ] Peter Vary edited comment on HIVE-19922 at 6/18/18 11:33 AM: - Attaching a revert for [~daijy]'s last comment to check if this helps, or not was (Author: pvary): Addressing a revert for [~daijy]'s last comment to check if this helps, or not > TestMiniDruidKafkaCliDriver[druidkafkamini_basic started] to fail > - > > Key: HIVE-19922 > URL: https://issues.apache.org/jira/browse/HIVE-19922 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Priority: Major > Attachments: HIVE-19922.patch > > > Consistently failing in the last 4 runs. > See: > [https://builds.apache.org/job/PreCommit-HIVE-Build/11824/testReport/org.apache.hadoop.hive.cli/TestMiniDruidKafkaCliDriver/testCliDriver_druidkafkamini_basic_/history/] > Can not reproduce the failure locally :( > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19922) TestMiniDruidKafkaCliDriver[druidkafkamini_basic started] to fail
[
https://issues.apache.org/jira/browse/HIVE-19922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515616#comment-16515616
]
Hive QA commented on HIVE-19922:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12928082/HIVE-19922.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14531 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.client.TestRuntimeStats.testCleanup[Embedded]
(batchId=212)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11884/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11884/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11884/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12928082 - PreCommit-HIVE-Build
> TestMiniDruidKafkaCliDriver[druidkafkamini_basic started] to fail
> -
>
> Key: HIVE-19922
> URL: https://issues.apache.org/jira/browse/HIVE-19922
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Priority: Major
> Attachments: HIVE-19922.patch
>
>
> Consistently failing in the last 4 runs.
> See:
> [https://builds.apache.org/job/PreCommit-HIVE-Build/11824/testReport/org.apache.hadoop.hive.cli/TestMiniDruidKafkaCliDriver/testCliDriver_druidkafkamini_basic_/history/]
> Can not reproduce the failure locally :(
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19922) TestMiniDruidKafkaCliDriver[druidkafkamini_basic started] to fail
[
https://issues.apache.org/jira/browse/HIVE-19922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515617#comment-16515617
]
Hive QA commented on HIVE-19922:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12928082/HIVE-19922.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11885/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11885/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11885/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12928082/HIVE-19922.patch was
found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12928082 - PreCommit-HIVE-Build
> TestMiniDruidKafkaCliDriver[druidkafkamini_basic started] to fail
> -
>
> Key: HIVE-19922
> URL: https://issues.apache.org/jira/browse/HIVE-19922
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Priority: Major
> Attachments: HIVE-19922.patch
>
>
> Consistently failing in the last 4 runs.
> See:
> [https://builds.apache.org/job/PreCommit-HIVE-Build/11824/testReport/org.apache.hadoop.hive.cli/TestMiniDruidKafkaCliDriver/testCliDriver_druidkafkamini_basic_/history/]
> Can not reproduce the failure locally :(
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19922) TestMiniDruidKafkaCliDriver[druidkafkamini_basic started] to fail
[ https://issues.apache.org/jira/browse/HIVE-19922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515622#comment-16515622 ] Peter Vary commented on HIVE-19922: --- Looks like the test is flaky, and not related to [~daijy]'s patch... > TestMiniDruidKafkaCliDriver[druidkafkamini_basic started] to fail > - > > Key: HIVE-19922 > URL: https://issues.apache.org/jira/browse/HIVE-19922 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Priority: Major > Attachments: HIVE-19922.patch > > > Consistently failing in the last 4 runs. > See: > [https://builds.apache.org/job/PreCommit-HIVE-Build/11824/testReport/org.apache.hadoop.hive.cli/TestMiniDruidKafkaCliDriver/testCliDriver_druidkafkamini_basic_/history/] > Can not reproduce the failure locally :( > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19922) TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flaky
[ https://issues.apache.org/jira/browse/HIVE-19922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-19922: -- Summary: TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flaky (was: TestMiniDruidKafkaCliDriver[druidkafkamini_basic started] to fail) > TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flaky > -- > > Key: HIVE-19922 > URL: https://issues.apache.org/jira/browse/HIVE-19922 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Priority: Major > Attachments: HIVE-19922.patch > > > Consistently failing in the last 4 runs. > See: > [https://builds.apache.org/job/PreCommit-HIVE-Build/11824/testReport/org.apache.hadoop.hive.cli/TestMiniDruidKafkaCliDriver/testCliDriver_druidkafkamini_basic_/history/] > Can not reproduce the failure locally :( > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19780) Test failure : org.apache.hadoop.hive.ql.TestTxnNoBuckets fails with java.lang.AssertionError
[
https://issues.apache.org/jira/browse/HIVE-19780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515623#comment-16515623
]
Alisha Prabhu commented on HIVE-19780:
--
At line no. 195 of TestTxnNoBuckets.java, the values for the parameter a and b
in the below code obtained from "myctas2" table differ on different
environments.
{code:java}
rs = runStatementOnDriver("select ROW__ID, a, b, INPUT__FILE__NAME from myctas2
order by ROW__ID");
String expected2[][] = {
{"{\"writeid\":1,\"bucketid\":536870912,\"rowid\":0}\t1\t2",
"warehouse/myctas2/delta_001_001_/bucket_0"},
{"{\"writeid\":1,\"bucketid\":536936448,\"rowid\":0}\t3\t4",
"warehouse/myctas2/delta_001_001_/bucket_1"}
};
checkExpected(rs, expected2, "Unexpected row count after ctas from acid
table");
{code}
The above patch works when applied to certain environments locally. Again,
without which the test case results in the above mentioned error.
Looking forward to a possible solution for the issue.
Thank you.
> Test failure : org.apache.hadoop.hive.ql.TestTxnNoBuckets fails with
> java.lang.AssertionError
> -
>
> Key: HIVE-19780
> URL: https://issues.apache.org/jira/browse/HIVE-19780
> Project: Hive
> Issue Type: Bug
> Environment: uname -a
> Linux pts00607-vm1 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC
> 2016 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Alisha Prabhu
>Priority: Major
> Labels: pp64le, x86_64
> Attachments: HIVE-19780.1.patch
>
>
> Maven command used is mvn -Dtest=TestTxnNoBuckets#testCTAS test
> Error:
> {code:java}
> [INFO] Running org.apache.hadoop.hive.ql.TestTxnNoBuckets
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 34.582 s <<< FAILURE! - in org.apache.hadoop.hive.ql.TestTxnNoBuckets
> [ERROR] testCTAS(org.apache.hadoop.hive.ql.TestTxnNoBuckets) Time elapsed:
> 31.64 s <<< FAILURE!
> java.lang.AssertionError: Actual line (data) 0 data:
> {"writeid":1,"bucketid":536870912,"rowid":0} 3 4
> file:/var/lib/jenkins/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnNoBuckets-1528085280454/warehouse/myctas2/delta_001_001_/bucket_0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hive.ql.TxnCommandsBaseForTests.checkExpected(TxnCommandsBaseForTests.java:228)
> {code}
> org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS refers to the
> same test case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19780) Test failure : org.apache.hadoop.hive.ql.TestTxnNoBuckets fails with java.lang.AssertionError
[
https://issues.apache.org/jira/browse/HIVE-19780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515623#comment-16515623
]
Alisha Prabhu edited comment on HIVE-19780 at 6/18/18 11:55 AM:
At line no. 195 of TestTxnNoBuckets.java, the values for the parameter a and b
in the below code obtained from "myctas2" table differ on different
environments.
{code:java}
rs = runStatementOnDriver("select ROW__ID, a, b, INPUT__FILE__NAME from myctas2
order by ROW__ID");
String expected2[][] = {
{"{\"writeid\":1,\"bucketid\":536870912,\"rowid\":0}\t1\t2",
"warehouse/myctas2/delta_001_001_/bucket_0"},
{"{\"writeid\":1,\"bucketid\":536936448,\"rowid\":0}\t3\t4",
"warehouse/myctas2/delta_001_001_/bucket_1"}
};
checkExpected(rs, expected2, "Unexpected row count after ctas from acid
table");
{code}
The above patch works when applied on certain environments locally. Again,
without which the test case results in the above mentioned error.
Looking forward to a possible solution for the issue.
Thank you.
was (Author: alishap):
At line no. 195 of TestTxnNoBuckets.java, the values for the parameter a and b
in the below code obtained from "myctas2" table differ on different
environments.
{code:java}
rs = runStatementOnDriver("select ROW__ID, a, b, INPUT__FILE__NAME from myctas2
order by ROW__ID");
String expected2[][] = {
{"{\"writeid\":1,\"bucketid\":536870912,\"rowid\":0}\t1\t2",
"warehouse/myctas2/delta_001_001_/bucket_0"},
{"{\"writeid\":1,\"bucketid\":536936448,\"rowid\":0}\t3\t4",
"warehouse/myctas2/delta_001_001_/bucket_1"}
};
checkExpected(rs, expected2, "Unexpected row count after ctas from acid
table");
{code}
The above patch works when applied to certain environments locally. Again,
without which the test case results in the above mentioned error.
Looking forward to a possible solution for the issue.
Thank you.
> Test failure : org.apache.hadoop.hive.ql.TestTxnNoBuckets fails with
> java.lang.AssertionError
> -
>
> Key: HIVE-19780
> URL: https://issues.apache.org/jira/browse/HIVE-19780
> Project: Hive
> Issue Type: Bug
> Environment: uname -a
> Linux pts00607-vm1 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC
> 2016 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Alisha Prabhu
>Priority: Major
> Labels: pp64le, x86_64
> Attachments: HIVE-19780.1.patch
>
>
> Maven command used is mvn -Dtest=TestTxnNoBuckets#testCTAS test
> Error:
> {code:java}
> [INFO] Running org.apache.hadoop.hive.ql.TestTxnNoBuckets
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 34.582 s <<< FAILURE! - in org.apache.hadoop.hive.ql.TestTxnNoBuckets
> [ERROR] testCTAS(org.apache.hadoop.hive.ql.TestTxnNoBuckets) Time elapsed:
> 31.64 s <<< FAILURE!
> java.lang.AssertionError: Actual line (data) 0 data:
> {"writeid":1,"bucketid":536870912,"rowid":0} 3 4
> file:/var/lib/jenkins/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnNoBuckets-1528085280454/warehouse/myctas2/delta_001_001_/bucket_0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hive.ql.TxnCommandsBaseForTests.checkExpected(TxnCommandsBaseForTests.java:228)
> {code}
> org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS refers to the
> same test case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19922) TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flaky
[ https://issues.apache.org/jira/browse/HIVE-19922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515627#comment-16515627 ] Peter Vary commented on HIVE-19922: --- HIVE-19509 tried to disable this test, but failed :) > TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flaky > -- > > Key: HIVE-19922 > URL: https://issues.apache.org/jira/browse/HIVE-19922 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Priority: Major > Attachments: HIVE-19922.patch > > > Consistently failing in the last 4 runs. > See: > [https://builds.apache.org/job/PreCommit-HIVE-Build/11824/testReport/org.apache.hadoop.hive.cli/TestMiniDruidKafkaCliDriver/testCliDriver_druidkafkamini_basic_/history/] > Can not reproduce the failure locally :( > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19923) Follow up of HIVE-19615, use UnaryFunction instead of prefix
[
https://issues.apache.org/jira/browse/HIVE-19923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515638#comment-16515638
]
Hive QA commented on HIVE-19923:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
21s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 3 new + 6 unchanged - 0 fixed
= 9 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11886/dev-support/hive-personality.sh
|
| git revision | master / 4ec256c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11886/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11886/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Follow up of HIVE-19615, use UnaryFunction instead of prefix
>
>
> Key: HIVE-19923
> URL: https://issues.apache.org/jira/browse/HIVE-19923
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19923.patch
>
>
> Correct usage of Druid isnull function is {code} isnull(exp){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19923) Follow up of HIVE-19615, use UnaryFunction instead of prefix
[
https://issues.apache.org/jira/browse/HIVE-19923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515663#comment-16515663
]
Hive QA commented on HIVE-19923:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12928089/HIVE-19923.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14531 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11886/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11886/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11886/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12928089 - PreCommit-HIVE-Build
> Follow up of HIVE-19615, use UnaryFunction instead of prefix
>
>
> Key: HIVE-19923
> URL: https://issues.apache.org/jira/browse/HIVE-19923
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19923.patch
>
>
> Correct usage of Druid isnull function is {code} isnull(exp){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19920) Schematool fails in embedded mode when auth is on
[
https://issues.apache.org/jira/browse/HIVE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515666#comment-16515666
]
Hive QA commented on HIVE-19920:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12928091/HIVE-19920.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11888/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11888/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11888/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12928091/HIVE-19920.2.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12928091 - PreCommit-HIVE-Build
> Schematool fails in embedded mode when auth is on
> -
>
> Key: HIVE-19920
> URL: https://issues.apache.org/jira/browse/HIVE-19920
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19920.1.patch, HIVE-19920.2.patch
>
>
> This is a follow up of HIVE-19775. We need to override more properties in
> embedded hs2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19922) TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flaky
[ https://issues.apache.org/jira/browse/HIVE-19922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-19922: -- Attachment: HIVE-19922.2.patch > TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flaky > -- > > Key: HIVE-19922 > URL: https://issues.apache.org/jira/browse/HIVE-19922 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Priority: Major > Attachments: HIVE-19922.2.patch, HIVE-19922.patch > > > Consistently failing in the last 4 runs. > See: > [https://builds.apache.org/job/PreCommit-HIVE-Build/11824/testReport/org.apache.hadoop.hive.cli/TestMiniDruidKafkaCliDriver/testCliDriver_druidkafkamini_basic_/history/] > Can not reproduce the failure locally :( > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19725) Add ability to dump non-native tables in replication metadata dump
[ https://issues.apache.org/jira/browse/HIVE-19725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19725: Target Version/s: 3.1.0, 4.0.0 (was: 3.0.0, 3.1.0, 4.0.0) > Add ability to dump non-native tables in replication metadata dump > -- > > Key: HIVE-19725 > URL: https://issues.apache.org/jira/browse/HIVE-19725 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.0.0, 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: Repl, pull-request-available > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19725.01.patch, HIVE-19725.02.patch, > HIVE-19725.03.patch, HIVE-19725.04.patch, HIVE-19725.05.patch, > HIVE-19725.06-branch-3.patch, HIVE-19725.07-branch-3.patch, > HIVE-19725.07.patch > > > if hive.repl.dump.metadata.only is set to true, allow dumping non native > tables also. > Data dump for non-native tables should never be allowed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19725) Add ability to dump non-native tables in replication metadata dump
[ https://issues.apache.org/jira/browse/HIVE-19725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515690#comment-16515690 ] Sankar Hariappan commented on HIVE-19725: - +1 > Add ability to dump non-native tables in replication metadata dump > -- > > Key: HIVE-19725 > URL: https://issues.apache.org/jira/browse/HIVE-19725 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.0.0, 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: Repl, pull-request-available > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19725.01.patch, HIVE-19725.02.patch, > HIVE-19725.03.patch, HIVE-19725.04.patch, HIVE-19725.05.patch, > HIVE-19725.06-branch-3.patch, HIVE-19725.07-branch-3.patch, > HIVE-19725.07.patch > > > if hive.repl.dump.metadata.only is set to true, allow dumping non native > tables also. > Data dump for non-native tables should never be allowed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19725) Add ability to dump non-native tables in replication metadata dump
[ https://issues.apache.org/jira/browse/HIVE-19725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19725: Affects Version/s: (was: 3.0.0) > Add ability to dump non-native tables in replication metadata dump > -- > > Key: HIVE-19725 > URL: https://issues.apache.org/jira/browse/HIVE-19725 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: Repl, pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19725.01.patch, HIVE-19725.02.patch, > HIVE-19725.03.patch, HIVE-19725.04.patch, HIVE-19725.05.patch, > HIVE-19725.06-branch-3.patch, HIVE-19725.07-branch-3.patch, > HIVE-19725.07.patch > > > if hive.repl.dump.metadata.only is set to true, allow dumping non native > tables also. > Data dump for non-native tables should never be allowed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19725) Add ability to dump non-native tables in replication metadata dump
[ https://issues.apache.org/jira/browse/HIVE-19725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515694#comment-16515694 ] Sankar Hariappan commented on HIVE-19725: - 07.patch is committed to master! Thanks [~maheshk114] for the contribution! > Add ability to dump non-native tables in replication metadata dump > -- > > Key: HIVE-19725 > URL: https://issues.apache.org/jira/browse/HIVE-19725 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: Repl, pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19725.01.patch, HIVE-19725.02.patch, > HIVE-19725.03.patch, HIVE-19725.04.patch, HIVE-19725.05.patch, > HIVE-19725.06-branch-3.patch, HIVE-19725.07-branch-3.patch, > HIVE-19725.07.patch > > > if hive.repl.dump.metadata.only is set to true, allow dumping non native > tables also. > Data dump for non-native tables should never be allowed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19725) Add ability to dump non-native tables in replication metadata dump
[ https://issues.apache.org/jira/browse/HIVE-19725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19725: Fix Version/s: (was: 3.1.0) > Add ability to dump non-native tables in replication metadata dump > -- > > Key: HIVE-19725 > URL: https://issues.apache.org/jira/browse/HIVE-19725 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: Repl, pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19725.01.patch, HIVE-19725.02.patch, > HIVE-19725.03.patch, HIVE-19725.04.patch, HIVE-19725.05.patch, > HIVE-19725.06-branch-3.patch, HIVE-19725.07-branch-3.patch, > HIVE-19725.07.patch > > > if hive.repl.dump.metadata.only is set to true, allow dumping non native > tables also. > Data dump for non-native tables should never be allowed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19870) HCatalog dynamic partition query can fail, if the table path is managed by Sentry
[ https://issues.apache.org/jira/browse/HIVE-19870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-19870: -- Attachment: HIVE-19870.5.patch > HCatalog dynamic partition query can fail, if the table path is managed by > Sentry > - > > Key: HIVE-19870 > URL: https://issues.apache.org/jira/browse/HIVE-19870 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-19870.2.patch, HIVE-19870.3.patch, > HIVE-19870.4.patch, HIVE-19870.5.patch, HIVE-19870.patch > > > The central issue is that HCatalog is assuming it needs to handle the storage > based authorization features. When a job completes, in HCatalog's file > committing phase it tries to manually set the permissions for the table for > authorization's sake, which makes it go against auto-authorization managment > features provided by Sentry. > The offending code is specifically at > [https://github.com/apache/hive/blob/master/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java#L373-L374] > and > [https://github.com/apache/hive/blob/master/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java#L380-L385] > and > [https://github.com/apache/hive/blob/master/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java#L952-L954] > where you can notice that _after_ it moves the files created by the job into > their respective partition directories under the final table destination, it > goes onto trying to perform chmod/chgrp operations which will fail out -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19782) Flash out TestObjectStore.testDirectSQLDropParitionsCleanup
[
https://issues.apache.org/jira/browse/HIVE-19782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515707#comment-16515707
]
Peter Vary commented on HIVE-19782:
---
[~vihangk1], [~akolb]: Could you please review?
Thanks,
Peter
> Flash out TestObjectStore.testDirectSQLDropParitionsCleanup
> ---
>
> Key: HIVE-19782
> URL: https://issues.apache.org/jira/browse/HIVE-19782
> Project: Hive
> Issue Type: Test
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-19782.2.patch, HIVE-19782.patch
>
>
> {{TestObjectStore.testDirectSQLDropParitionsCleanup}} checks that the tables
> are empty after the drop. We should add some rows to every partition related
> table, to see that they are really cleaned up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-12342) Set default value of hive.optimize.index.filter to true
[
https://issues.apache.org/jira/browse/HIVE-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515710#comment-16515710
]
Hive QA commented on HIVE-12342:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 22 unchanged - 0 fixed
= 23 total (was 22) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11889/dev-support/hive-personality.sh
|
| git revision | master / 4ec256c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11889/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11889/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Set default value of hive.optimize.index.filter to true
> ---
>
> Key: HIVE-12342
> URL: https://issues.apache.org/jira/browse/HIVE-12342
> Project: Hive
> Issue Type: Task
> Components: Configuration
>Reporter: Ashutosh Chauhan
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-12342.05.patch, HIVE-12342.06.patch,
> HIVE-12342.07.patch, HIVE-12342.08.patch, HIVE-12342.09.patch,
> HIVE-12342.1.patch, HIVE-12342.10.patch, HIVE-12342.11.patch,
> HIVE-12342.12.patch, HIVE-12342.13.patch, HIVE-12342.14.patch,
> HIVE-12342.15.patch, HIVE-12342.16.patch, HIVE-12342.17.patch,
> HIVE-12342.2.patch, HIVE-12342.3.patch, HIVE-12342.4.patch, HIVE-12342.patch
>
>
> This configuration governs ppd for storage layer. When applicable, it will
> always help. It should be on by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19846) Removed Deprecated Calls From FileUtils-getJarFilesByPath
[ https://issues.apache.org/jira/browse/HIVE-19846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515722#comment-16515722 ] Peter Vary commented on HIVE-19846: --- +1, but we need full green runs before committing. Also if you do not change variable names and formatting where it is not necessary we will have easier time backporting the change > Removed Deprecated Calls From FileUtils-getJarFilesByPath > - > > Key: HIVE-19846 > URL: https://issues.apache.org/jira/browse/HIVE-19846 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-19846.1.patch, HIVE-19846.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19834) Clear Context Map of Paths to ContentSummary
[
https://issues.apache.org/jira/browse/HIVE-19834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515730#comment-16515730
]
Peter Vary commented on HIVE-19834:
---
I am not big fan of this change.
There are very few comments in Hive code, but here the class comment says:
{code:java}
/**
* Context for Semantic Analyzers. Usage: not reusable - construct a new one for
* each query should call clear() at end of use to remove temporary folders
*/
{code}
This explicitly states that it removes temporary folders. In most cases this is
the last method of the task, or execute method, so the class will be removed
without fuss. Do we have specific cases where we have seen this is causing
problems?
Thanks,
Peter
> Clear Context Map of Paths to ContentSummary
>
>
> Key: HIVE-19834
> URL: https://issues.apache.org/jira/browse/HIVE-19834
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.3.2, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-19834.1.patch
>
>
> The {{Context}} class has a {{clear}} method which is called. During the
> method, various files are deleted and in-memory maps are cleared. I would
> like to propose that we clear out an additional in-memory map structure that
> may contain a lot of data so that it can be GC'ed asap. This map contains
> mapping of "File Path"->"Content Summary". For a query with a large file
> set, this can be quite large.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19807) Add Useful Error Message To Table Header/Footer Parsing
[
https://issues.apache.org/jira/browse/HIVE-19807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515742#comment-16515742
]
Peter Vary commented on HIVE-19807:
---
[~belugabehr]: If you take a look at the callers of
{{Utilities.getHeaderCount}} you will see, that these methods expect
IOExceptions if there is a problem reading the files. In some cases they catch
every exception and they are creating an IOException from it anyway, but in
case of {{HiveInputFormat}} we might have some extra surprises... As we see in
the test failure (at least this is my guess :) ):
{code}
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[file_with_header_footer_negative]
(batchId=98)
{code}
Thanks,
Peter
> Add Useful Error Message To Table Header/Footer Parsing
> ---
>
> Key: HIVE-19807
> URL: https://issues.apache.org/jira/browse/HIVE-19807
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-19807.1.patch, HIVE-19807.2.patch
>
>
> Add some useful logging messages to invalid value parsing of
> {{skip.header.line.count}} and {{skip.footer.line.count}} for better
> troubleshooting.
> Also, parsing error resulted in {{java.io.IOException}} which:
> {quote}
> Signals that an I/O exception of some sort has occurred. This class is the
> general class of exceptions produced by failed or interrupted I/O operations.
> {quote}
> A parsing error is not the result of a failed or interrupted I/O operation so
> I dropped it and replaced with runtime exception {{IllegalArgumentException}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19711) Refactor Hive Schema Tool
[ https://issues.apache.org/jira/browse/HIVE-19711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515747#comment-16515747 ] Miklos Gergely commented on HIVE-19711: --- The test error is not related to this patch. [~ashutoshc] I suggest to merge this change. > Refactor Hive Schema Tool > - > > Key: HIVE-19711 > URL: https://issues.apache.org/jira/browse/HIVE-19711 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-19711.01.patch, HIVE-19711.02.patch, > HIVE-19711.03.patch, HIVE-19711.04.patch, HIVE-19711.05.patch > > > HiveSchemaTool is an 1500 lines long class trying to do everything It shold > be cut into multiple classes doing smaller components. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12342) Set default value of hive.optimize.index.filter to true
[
https://issues.apache.org/jira/browse/HIVE-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515754#comment-16515754
]
Hive QA commented on HIVE-12342:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12928094/HIVE-12342.17.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1430 failed/errored test(s), 14531 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=249)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[explain_outputs]
(batchId=257)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=257)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_10]
(batchId=257)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=257)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=257)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13]
(batchId=257)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_16]
(batchId=257)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1]
(batchId=257)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_2]
(batchId=257)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_3]
(batchId=257)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[allcolref_in_udf]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStatsPart]
(batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ambiguous_col]
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_tbl_date]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_deep_filters]
(batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_filter]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join_pkfk]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_part]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_select]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_9]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join0] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join10] (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join11] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join12] (batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join13] (batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join14] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join15] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join16] (batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join17] (batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19_inclause]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join1] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join20] (batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join21] (batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join22] (batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join23] (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join24] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join26] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join27] (batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join28] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join29] (batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join2] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join32] (batchId=89)
org.apache.hadoop.hive
[jira] [Commented] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"
[
https://issues.apache.org/jira/browse/HIVE-18902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515759#comment-16515759
]
Peter Vary commented on HIVE-18902:
---
[~bchupika], [~belugabehr]: +1 LGTM. Please rebase the patch, and have a clean
test run. If we have that, I can commit this change.
> Lower Logging Level for Cleaning Up "local RawStore"
>
>
> Key: HIVE-18902
> URL: https://issues.apache.org/jira/browse/HIVE-18902
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Bohdan Chupika
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18902.01.patch, HIVE-18902.02.patch,
> HIVE-18902.03.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L7756-L7768]
>
> {code:java}
> private static void cleanupRawStore() {
> try {
> RawStore rs = HMSHandler.getRawStore();
> if (rs != null) {
> HMSHandler.logInfo("Cleaning up thread local RawStore...");
> rs.shutdown();
> }
> } finally {
> HMSHandler handler = HMSHandler.threadLocalHMSHandler.get();
> if (handler != null) {
> handler.notifyMetaListenersOnShutDown();
> }
> HMSHandler.threadLocalHMSHandler.remove();
> HMSHandler.threadLocalConf.remove();
> HMSHandler.threadLocalModifiedConfig.remove();
> HMSHandler.removeRawStore();
> HMSHandler.logInfo("Done cleaning up thread local RawStore");
> }
> }
> {code}
> {code}
> 2018-03-03 17:21:49,832 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19:
> Cleaning up thread local RawStore...
> 2018-03-03 17:21:49,834 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19: Done
> cleaning up thread local RawStore
> {code}
> Not very helpful logging. Please change logging levels to _debug_ or even
> _trace_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[ https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19921: - Attachment: HIVE-19921.01-branch-3.patch > Fix perf duration and queue name in HiveProtoLoggingHook > > > Key: HIVE-19921 > URL: https://issues.apache.org/jira/browse/HIVE-19921 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19921.01-branch-3.patch, HIVE-19921.01.patch > > > The perf log should return duration instead of end time. > The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19919) HiveServer2 - expose queryable data dictionary (ie Oracles' ALL_TAB_COLUMNS)
[ https://issues.apache.org/jira/browse/HIVE-19919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515797#comment-16515797 ] Sergey Shelukhin commented on HIVE-19919: - Hive already has this, it's called sysdb. See sysdb.q test file on master. Unfortunately I think currently it needs to be initialized explicitly cc [~hagleitn] > HiveServer2 - expose queryable data dictionary (ie Oracles' ALL_TAB_COLUMNS) > > > Key: HIVE-19919 > URL: https://issues.apache.org/jira/browse/HIVE-19919 > Project: Hive > Issue Type: New Feature > Components: SQL >Affects Versions: 3.0.0, 2.3.2 >Reporter: t oo >Priority: Major > > All major db vendors have a table like information_schema.columns, > all_tab_columns or syscolumns containing table_name,column_name, data_type, > col_order. Adding this feature to HiveServer2 would be very convenient for > users. > This information is currently only available in the mysql metastore ie TBLS, > COLS but should be exposed up into the HiveServer2 1 port connection. > Thus saving users from having 2 connections (1 to see data, 1 to see > metadata). For security reason too, mysql can be firewalled from end-users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19882) Fix QTestUtil session lifecycle
[
https://issues.apache.org/jira/browse/HIVE-19882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19882:
Attachment: HIVE-19882.05.patch
> Fix QTestUtil session lifecycle
> ---
>
> Key: HIVE-19882
> URL: https://issues.apache.org/jira/browse/HIVE-19882
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19882.01.patch, HIVE-19882.02.patch,
> HIVE-19882.03.patch, HIVE-19882.04.patch, HIVE-19882.05.patch
>
>
> there are a number of strange come and go failing tests; it was always
> strange to me that qtestutil cleans up at some questionable points - this
> seems to be leading to executing some commands with the previous qfiles
> session...
> ideally the session/etc should start/reused in {{before}}
> and it should be closed in {{after}}
> seems like configuration is handled probably incorrectly; saving the conf
> after initialization - and restoring it for a new session should ensure
> consistency
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19103) Nested structure Projection Push Down in Hive with ORC
[
https://issues.apache.org/jira/browse/HIVE-19103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515802#comment-16515802
]
Hive QA commented on HIVE-19103:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 19 new + 151 unchanged - 0
fixed = 170 total (was 151) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11891/dev-support/hive-personality.sh
|
| git revision | master / 6a16a71 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11891/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11891/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Nested structure Projection Push Down in Hive with ORC
> --
>
> Key: HIVE-19103
> URL: https://issues.apache.org/jira/browse/HIVE-19103
> Project: Hive
> Issue Type: Improvement
> Components: Hive, ORC
>Reporter: Ashish Sharma
>Assignee: Ashish Sharma
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-19103.2.patch, HIVE-19103.patch
>
>
> Reading required columns only in nested structure schema
> Example -
> *Current state* -
> Schema - struct,g:string>>
> Query - select c.e.f from t where c.e.f > 10;
> Current state - read entire c struct from the file and then filter because
> "hive.io.file.readcolumn.ids" is referred due to which all the children
> column are select to read from the file.
> Conf -
> _hive.io.file.readcolumn.ids = "2"
> hive.io.file.readNestedColumn.paths = "c.e.f"_
> Result -
> boolean[ ] include = [true,false,false,true,true,true,true,true]
> *Expected state* -
> Schema - struct,g:string>>
> Query - select c.e.f from t where c.e.f > 10;
> Expected state - instead of reading entire c struct from the file just read
> only the f column by referring the " hive.io.file.readNestedColumn.paths".
> Conf -
> _hive.io.file.readcolumn.ids = "2"
> hive.io.file.readNestedColumn.paths = "c.e.f"_
> Result -
> boolean[ ] include = [true,false,false,true,false,true,true,false]
--
[jira] [Commented] (HIVE-19870) HCatalog dynamic partition query can fail, if the table path is managed by Sentry
[ https://issues.apache.org/jira/browse/HIVE-19870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515816#comment-16515816 ] Sahil Takiar commented on HIVE-19870: - +1 LGTM > HCatalog dynamic partition query can fail, if the table path is managed by > Sentry > - > > Key: HIVE-19870 > URL: https://issues.apache.org/jira/browse/HIVE-19870 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-19870.2.patch, HIVE-19870.3.patch, > HIVE-19870.4.patch, HIVE-19870.5.patch, HIVE-19870.patch > > > The central issue is that HCatalog is assuming it needs to handle the storage > based authorization features. When a job completes, in HCatalog's file > committing phase it tries to manually set the permissions for the table for > authorization's sake, which makes it go against auto-authorization managment > features provided by Sentry. > The offending code is specifically at > [https://github.com/apache/hive/blob/master/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java#L373-L374] > and > [https://github.com/apache/hive/blob/master/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java#L380-L385] > and > [https://github.com/apache/hive/blob/master/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java#L952-L954] > where you can notice that _after_ it moves the files created by the job into > their respective partition directories under the final table destination, it > goes onto trying to perform chmod/chgrp operations which will fail out -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19925) NPE in SparkTask#printConsoleMetrics
[
https://issues.apache.org/jira/browse/HIVE-19925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515819#comment-16515819
]
Sahil Takiar commented on HIVE-19925:
-
What env did you use to produce this error?
> NPE in SparkTask#printConsoleMetrics
>
>
> Key: HIVE-19925
> URL: https://issues.apache.org/jira/browse/HIVE-19925
> Project: Hive
> Issue Type: Improvement
>Reporter: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> When running a join query with HOS, as :
> {code:java}
> SELECT a.id FROM sample a JOIN sample b ON (a.id=b.id);{code}
> Got the following exception :
> {code:java}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.printConsoleMetrics(SparkTask.java:229)
> at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:166)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2678)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2330)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2001)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1701)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1695)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:87)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:315)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:328)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19923) Follow up of HIVE-19615, use UnaryFunction instead of prefix
[
https://issues.apache.org/jira/browse/HIVE-19923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19923:
---
Resolution: Fixed
Fix Version/s: 4.0.0
3.1.0
Status: Resolved (was: Patch Available)
Pushed to master, branch-3. Thanks [~bslim]
> Follow up of HIVE-19615, use UnaryFunction instead of prefix
>
>
> Key: HIVE-19923
> URL: https://issues.apache.org/jira/browse/HIVE-19923
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19923.patch
>
>
> Correct usage of Druid isnull function is {code} isnull(exp){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19821) Distributed HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-19821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-19821: Attachment: HIVE-19821_ Distributed HiveServer2.pdf > Distributed HiveServer2 > --- > > Key: HIVE-19821 > URL: https://issues.apache.org/jira/browse/HIVE-19821 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-19821.1.WIP.patch, HIVE-19821_ Distributed > HiveServer2.pdf > > > HS2 deployments often hit OOM issues due to a number of factors: (1) too many > concurrent connections, (2) query that scan a large number of partitions have > to pull a lot of metadata into memory (e.g. a query reading thousands of > partitions requires loading thousands of partitions into memory), (3) very > large queries can take up a lot of heap space, especially during query > parsing. There are a number of other factors that cause HiveServer2 to run > out of memory, these are just some of the more commons ones. > Distributed HS2 proposes to do all query parsing, compilation, planning, and > execution coordination inside a dedicated container. This should > significantly decrease memory pressure on HS2 and allow HS2 to scale to a > larger number of concurrent users. > For HoS (and I think Hive-on-Tez) this just requires moving all query > compilation, planning, etc. inside the application master for the > corresponding Hive session. > The main benefit here is isolation. A poorly written Hive query cannot bring > down an entire HiveServer2 instance and force all other queries to fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19821) Distributed HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-19821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515824#comment-16515824 ] Sahil Takiar commented on HIVE-19821: - Attached a high-level design doc, comments and feedback are welcome. > Distributed HiveServer2 > --- > > Key: HIVE-19821 > URL: https://issues.apache.org/jira/browse/HIVE-19821 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-19821.1.WIP.patch, HIVE-19821_ Distributed > HiveServer2.pdf > > > HS2 deployments often hit OOM issues due to a number of factors: (1) too many > concurrent connections, (2) query that scan a large number of partitions have > to pull a lot of metadata into memory (e.g. a query reading thousands of > partitions requires loading thousands of partitions into memory), (3) very > large queries can take up a lot of heap space, especially during query > parsing. There are a number of other factors that cause HiveServer2 to run > out of memory, these are just some of the more commons ones. > Distributed HS2 proposes to do all query parsing, compilation, planning, and > execution coordination inside a dedicated container. This should > significantly decrease memory pressure on HS2 and allow HS2 to scale to a > larger number of concurrent users. > For HoS (and I think Hive-on-Tez) this just requires moving all query > compilation, planning, etc. inside the application master for the > corresponding Hive session. > The main benefit here is isolation. A poorly written Hive query cannot bring > down an entire HiveServer2 instance and force all other queries to fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19919) HiveServer2 - expose queryable data dictionary (ie Oracles' ALL_TAB_COLUMNS)
[ https://issues.apache.org/jira/browse/HIVE-19919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515827#comment-16515827 ] Gunther Hagleitner commented on HIVE-19919: --- See HIVE-1010. hive-schema-3.0.0.hive.sql has the definitions. Can be initialized via SchemaTool. > HiveServer2 - expose queryable data dictionary (ie Oracles' ALL_TAB_COLUMNS) > > > Key: HIVE-19919 > URL: https://issues.apache.org/jira/browse/HIVE-19919 > Project: Hive > Issue Type: New Feature > Components: SQL >Affects Versions: 3.0.0, 2.3.2 >Reporter: t oo >Priority: Major > > All major db vendors have a table like information_schema.columns, > all_tab_columns or syscolumns containing table_name,column_name, data_type, > col_order. Adding this feature to HiveServer2 would be very convenient for > users. > This information is currently only available in the mysql metastore ie TBLS, > COLS but should be exposed up into the HiveServer2 1 port connection. > Thus saving users from having 2 connections (1 to see data, 1 to see > metadata). For security reason too, mysql can be firewalled from end-users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19602) Refactor inplace progress code in Hive-on-spark progress monitor to use ProgressMonitor instance
[ https://issues.apache.org/jira/browse/HIVE-19602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-19602: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. > Refactor inplace progress code in Hive-on-spark progress monitor to use > ProgressMonitor instance > > > Key: HIVE-19602 > URL: https://issues.apache.org/jira/browse/HIVE-19602 > Project: Hive > Issue Type: Bug >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19602.3.patch, HIVE-19602.4.patch, > HIVE-19602.5.patch > > > We can refactor the HOS inplace progress monitor code > (SparkJobMonitor#printStatusInPlace) to use InplaceUpdate#render. > We can create an instance of ProgressMonitor and use it to show the progress. > This would be similar to : > [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19786) RpcServer cancelTask log message is incorrect
[
https://issues.apache.org/jira/browse/HIVE-19786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515859#comment-16515859
]
Sahil Takiar commented on HIVE-19786:
-
+1 LGTM
> RpcServer cancelTask log message is incorrect
> -
>
> Key: HIVE-19786
> URL: https://issues.apache.org/jira/browse/HIVE-19786
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19786.1.patch, HIVE-19786.2.patch
>
>
> The log message inside the {{cancelTask}} of the {{RpcServer}}
> {{ChannelInitializer}} is incorrect. It states its measuring the timeout for
> the "test" message to be sent (basically a "hello" message to test the
> connection works). However, the {{cancelTask}} is actually used to timeout
> the SASL negotiation between the client and the server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19787) Log message when spark-submit has completed
[
https://issues.apache.org/jira/browse/HIVE-19787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-19787:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
> Log message when spark-submit has completed
> ---
>
> Key: HIVE-19787
> URL: https://issues.apache.org/jira/browse/HIVE-19787
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19787.1.patch
>
>
> If {{spark-submit}} runs successfully the "Driver" thread should log a
> message. Otherwise there is no way to know if {{spark-submit}} exited
> successfully. We should also rename the thread to some more informative than
> "Driver".
> Without this, debugging timeout exceptions of the RemoteDriver -> HS2
> connection is difficult, because there is no way to know if {{spark-submit}}
> finished or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19925) NPE in SparkTask#printConsoleMetrics
[
https://issues.apache.org/jira/browse/HIVE-19925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-19925:
Issue Type: Sub-task (was: Improvement)
Parent: HIVE-17718
> NPE in SparkTask#printConsoleMetrics
>
>
> Key: HIVE-19925
> URL: https://issues.apache.org/jira/browse/HIVE-19925
> Project: Hive
> Issue Type: Sub-task
>Reporter: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> When running a join query with HOS, as :
> {code:java}
> SELECT a.id FROM sample a JOIN sample b ON (a.id=b.id);{code}
> Got the following exception :
> {code:java}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.printConsoleMetrics(SparkTask.java:229)
> at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:166)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2678)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2330)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2001)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1701)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1695)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:87)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:315)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:328)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19786) RpcServer cancelTask log message is incorrect
[
https://issues.apache.org/jira/browse/HIVE-19786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-19786:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master given that the first patch returned with a clean Hive QA run
and that this patch just modifies a log statement.
> RpcServer cancelTask log message is incorrect
> -
>
> Key: HIVE-19786
> URL: https://issues.apache.org/jira/browse/HIVE-19786
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19786.1.patch, HIVE-19786.2.patch
>
>
> The log message inside the {{cancelTask}} of the {{RpcServer}}
> {{ChannelInitializer}} is incorrect. It states its measuring the timeout for
> the "test" message to be sent (basically a "hello" message to test the
> connection works). However, the {{cancelTask}} is actually used to timeout
> the SASL negotiation between the client and the server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19766) Show the number of rows inserted when execution engine is Spark
[ https://issues.apache.org/jira/browse/HIVE-19766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-19766: Issue Type: Sub-task (was: Improvement) Parent: HIVE-17718 > Show the number of rows inserted when execution engine is Spark > --- > > Key: HIVE-19766 > URL: https://issues.apache.org/jira/browse/HIVE-19766 > Project: Hive > Issue Type: Sub-task >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > Currently when insert query is run, the beeline output shows No rows affected. > The logic to show the number of rows inserted is now present when execution > engine is MR. > This Jira is to make this work with Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19925) NPE in SparkTask#printConsoleMetrics
[
https://issues.apache.org/jira/browse/HIVE-19925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515883#comment-16515883
]
Bharathkrishna Guruvayoor Murali commented on HIVE-19925:
-
I was using the setup on my local machine with following configs.
{code:java}
hive.execution.engine
spark
spark.master
local
spark.testing.memory
471859200
hive.aux.jars.path
/Users/bharathkrishna/hive-2/hive/itests/hive-unit/target/spark/jars/
{code}
> NPE in SparkTask#printConsoleMetrics
>
>
> Key: HIVE-19925
> URL: https://issues.apache.org/jira/browse/HIVE-19925
> Project: Hive
> Issue Type: Sub-task
>Reporter: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> When running a join query with HOS, as :
> {code:java}
> SELECT a.id FROM sample a JOIN sample b ON (a.id=b.id);{code}
> Got the following exception :
> {code:java}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.printConsoleMetrics(SparkTask.java:229)
> at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:166)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2678)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2330)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2001)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1701)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1695)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:87)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:315)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:328)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19103) Nested structure Projection Push Down in Hive with ORC
[
https://issues.apache.org/jira/browse/HIVE-19103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515885#comment-16515885
]
Hive QA commented on HIVE-19103:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12928101/HIVE-19103.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14533 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comma_in_column_name]
(batchId=5)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithDynamicPartition
(batchId=268)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11891/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11891/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11891/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12928101 - PreCommit-HIVE-Build
> Nested structure Projection Push Down in Hive with ORC
> --
>
> Key: HIVE-19103
> URL: https://issues.apache.org/jira/browse/HIVE-19103
> Project: Hive
> Issue Type: Improvement
> Components: Hive, ORC
>Reporter: Ashish Sharma
>Assignee: Ashish Sharma
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-19103.2.patch, HIVE-19103.patch
>
>
> Reading required columns only in nested structure schema
> Example -
> *Current state* -
> Schema - struct,g:string>>
> Query - select c.e.f from t where c.e.f > 10;
> Current state - read entire c struct from the file and then filter because
> "hive.io.file.readcolumn.ids" is referred due to which all the children
> column are select to read from the file.
> Conf -
> _hive.io.file.readcolumn.ids = "2"
> hive.io.file.readNestedColumn.paths = "c.e.f"_
> Result -
> boolean[ ] include = [true,false,false,true,true,true,true,true]
> *Expected state* -
> Schema - struct,g:string>>
> Query - select c.e.f from t where c.e.f > 10;
> Expected state - instead of reading entire c struct from the file just read
> only the f column by referring the " hive.io.file.readNestedColumn.paths".
> Conf -
> _hive.io.file.readcolumn.ids = "2"
> hive.io.file.readNestedColumn.paths = "c.e.f"_
> Result -
> boolean[ ] include = [true,false,false,true,false,true,true,false]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19925) NPE in SparkTask#printConsoleMetrics
[
https://issues.apache.org/jira/browse/HIVE-19925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515890#comment-16515890
]
Sahil Takiar commented on HIVE-19925:
-
hmmm odd, it might have something to do with the fact that spark.master is set
to local. Regardless, it would be good to make this code null-safe.
> NPE in SparkTask#printConsoleMetrics
>
>
> Key: HIVE-19925
> URL: https://issues.apache.org/jira/browse/HIVE-19925
> Project: Hive
> Issue Type: Sub-task
>Reporter: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> When running a join query with HOS, as :
> {code:java}
> SELECT a.id FROM sample a JOIN sample b ON (a.id=b.id);{code}
> Got the following exception :
> {code:java}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.printConsoleMetrics(SparkTask.java:229)
> at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:166)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2678)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2330)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2001)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1701)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1695)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:87)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:315)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:328)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19422) Create Docker env for running HoS locally
[
https://issues.apache.org/jira/browse/HIVE-19422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515892#comment-16515892
]
Sahil Takiar commented on HIVE-19422:
-
[~aihuaxu] yes any scripts or documentation to get HoS up and running locally
would be very helpful. The goal of the JIRA is to provide a way to get HoS to
run locally, easily. Docker is just a convenient way of doing this, but scripts
and documentation work too.
> Create Docker env for running HoS locally
> -
>
> Key: HIVE-19422
> URL: https://issues.apache.org/jira/browse/HIVE-19422
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
>
> It's really hard to run HoS on a locally installed distribution of Hive built
> using {{mvn package}}. The only way developers can really run HoS is via the
> Spark CLI Drivers. However, there are occasions where devs need to run HoS on
> a proper Hive distribution in order to validate some behavior.
> The docker image will also be useful to users who want to play around with
> HoS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19925) NPE in SparkTask#printConsoleMetrics
[
https://issues.apache.org/jira/browse/HIVE-19925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515927#comment-16515927
]
Bharathkrishna Guruvayoor Murali commented on HIVE-19925:
-
I saw that in LocalSparkJobStatus#getSparkStatistics method, the following code
returns null.
{code:java}
[...]
// add spark job metrics.
Map>> jobMetric =
jobMetricsListener.getJobMetric(jobId);
if (jobMetric == null) {
return null;
}
[...]{code}
Hence, the NPE. Is this something not expected to happen?
> NPE in SparkTask#printConsoleMetrics
>
>
> Key: HIVE-19925
> URL: https://issues.apache.org/jira/browse/HIVE-19925
> Project: Hive
> Issue Type: Sub-task
>Reporter: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> When running a join query with HOS, as :
> {code:java}
> SELECT a.id FROM sample a JOIN sample b ON (a.id=b.id);{code}
> Got the following exception :
> {code:java}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.printConsoleMetrics(SparkTask.java:229)
> at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:166)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2678)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2330)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2001)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1701)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1695)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:87)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:315)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:328)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18916) SparkClientImpl doesn't error out if spark-submit fails
[
https://issues.apache.org/jira/browse/HIVE-18916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515938#comment-16515938
]
Sahil Takiar edited comment on HIVE-18916 at 6/18/18 3:59 PM:
--
[~aihuaxu] we actually already do that. There is an {{List}} called
{{childErrorLog}} which contains the STDERR. Unfortunately, even the STDERR
contains some extraneous information. For example, for the test
{{spark_submit_negative_executor_memory.q}} the {{childErrLog}} log is below:
{code}
Run with --help for usage help or --verbose for debug output
Error: Executor Memory cores must be a positive number
{code}
Based on the Spark class {{org.apache.spark.util.CommandLineUtils}} errors
should usually start with "Error: "
was (Author: stakiar):
[~aihuaxu] we actually already do that. There is an {{List}} called
{{childErrorLog}} which contains the STDERR. Unfortunately, even the STDERR
contains some extraneous information. For example, for the test
{{spark_submit_negative_executor_memory.q}} the {{childErrLog}} log is below:
{code
Run with --help for usage help or --verbose for debug output
Error: Executor Memory cores must be a positive number
{code}
Based on the Spark class {{org.apache.spark.util.CommandLineUtils}} errors
should usually start with "Error: "
> SparkClientImpl doesn't error out if spark-submit fails
> ---
>
> Key: HIVE-18916
> URL: https://issues.apache.org/jira/browse/HIVE-18916
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18916.1.WIP.patch, HIVE-18916.2.patch,
> HIVE-18916.3.patch
>
>
> If {{spark-submit}} returns a non-zero exit code, {{SparkClientImpl}} will
> simply log the exit code, but won't throw an error. Eventually, the
> connection timeout will get triggered and an exception like {{Timed out
> waiting for client connection}} will be logged, which is pretty misleading.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18916) SparkClientImpl doesn't error out if spark-submit fails
[
https://issues.apache.org/jira/browse/HIVE-18916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515938#comment-16515938
]
Sahil Takiar commented on HIVE-18916:
-
[~aihuaxu] we actually already do that. There is an {{List}} called
{{childErrorLog}} which contains the STDERR. Unfortunately, even the STDERR
contains some extraneous information. For example, for the test
{{spark_submit_negative_executor_memory.q}} the {{childErrLog}} log is below:
{code
Run with --help for usage help or --verbose for debug output
Error: Executor Memory cores must be a positive number
{code}
Based on the Spark class {{org.apache.spark.util.CommandLineUtils}} errors
should usually start with "Error: "
> SparkClientImpl doesn't error out if spark-submit fails
> ---
>
> Key: HIVE-18916
> URL: https://issues.apache.org/jira/browse/HIVE-18916
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18916.1.WIP.patch, HIVE-18916.2.patch,
> HIVE-18916.3.patch
>
>
> If {{spark-submit}} returns a non-zero exit code, {{SparkClientImpl}} will
> simply log the exit code, but won't throw an error. Eventually, the
> connection timeout will get triggered and an exception like {{Timed out
> waiting for client connection}} will be logged, which is pretty misleading.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19812) Disable external table replication by default via a configuration property
[
https://issues.apache.org/jira/browse/HIVE-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515947#comment-16515947
]
Hive QA commented on HIVE-19812:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 110
unchanged - 0 fixed = 112 total (was 110) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11892/dev-support/hive-personality.sh
|
| git revision | master / 4810511 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11892/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: common itests/hive-unit ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11892/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Disable external table replication by default via a configuration property
> --
>
> Key: HIVE-19812
> URL: https://issues.apache.org/jira/browse/HIVE-19812
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19812.01.patch, HIVE-19812.02.patch,
> HIVE-19812.03.patch, HIVE-19812.04.patch, HIVE-19812.05.patch
>
>
> use a hive config property to allow external table replication. set this
> property by default to prevent external table replication.
> for metadata only hive
[jira] [Commented] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[ https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515950#comment-16515950 ] anishek commented on HIVE-19921: +1 > Fix perf duration and queue name in HiveProtoLoggingHook > > > Key: HIVE-19921 > URL: https://issues.apache.org/jira/browse/HIVE-19921 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19921.01-branch-3.patch, HIVE-19921.01.patch > > > The perf log should return duration instead of end time. > The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[ https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515954#comment-16515954 ] anishek commented on HIVE-19921: Patch HIVE-19921.01.patch is committed to master. Thanks [~harishjp]. > Fix perf duration and queue name in HiveProtoLoggingHook > > > Key: HIVE-19921 > URL: https://issues.apache.org/jira/browse/HIVE-19921 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19921.01-branch-3.patch, HIVE-19921.01.patch > > > The perf log should return duration instead of end time. > The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19925) NPE in SparkTask#printConsoleMetrics
[
https://issues.apache.org/jira/browse/HIVE-19925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515987#comment-16515987
]
Sahil Takiar commented on HIVE-19925:
-
hmm well {{jobMetricsListener.getJobMetric(jobId)}} probably shouldn't be
returning {{null}}, especially after the job has completed. That looks like a
bug.
> NPE in SparkTask#printConsoleMetrics
>
>
> Key: HIVE-19925
> URL: https://issues.apache.org/jira/browse/HIVE-19925
> Project: Hive
> Issue Type: Sub-task
>Reporter: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> When running a join query with HOS, as :
> {code:java}
> SELECT a.id FROM sample a JOIN sample b ON (a.id=b.id);{code}
> Got the following exception :
> {code:java}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.printConsoleMetrics(SparkTask.java:229)
> at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:166)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2678)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2330)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2001)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1701)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1695)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:87)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:315)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:328)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19812) Disable external table replication by default via a configuration property
[
https://issues.apache.org/jira/browse/HIVE-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16515993#comment-16515993
]
Hive QA commented on HIVE-19812:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12928102/HIVE-19812.05.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14537 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_basic]
(batchId=257)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_13_nonnative_import]
(batchId=95)
org.apache.hadoop.hive.metastore.client.TestRuntimeStats.testCleanup[Remote]
(batchId=212)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11892/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11892/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11892/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12928102 - PreCommit-HIVE-Build
> Disable external table replication by default via a configuration property
> --
>
> Key: HIVE-19812
> URL: https://issues.apache.org/jira/browse/HIVE-19812
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19812.01.patch, HIVE-19812.02.patch,
> HIVE-19812.03.patch, HIVE-19812.04.patch, HIVE-19812.05.patch
>
>
> use a hive config property to allow external table replication. set this
> property by default to prevent external table replication.
> for metadata only hive repl always export metadata for external tables.
>
> REPL_DUMP_EXTERNAL_TABLES("hive.repl.dump.include.external.tables", false,
> "Indicates if repl dump should include information about external tables. It
> should be \n"
> + "used in conjunction with 'hive.repl.dump.metadata.only' set to false. if
> 'hive.repl.dump.metadata.only' \n"
> + " is set to true then this config parameter has no effect as external table
> meta data is flushed \n"
> + " always by default.")
> This should be done for only replication dump and not for export
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19889) Wrong results due to PPD of non deterministic functions with CBO
[
https://issues.apache.org/jira/browse/HIVE-19889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janaki Lahorani updated HIVE-19889:
---
Description:
The following query can give wrong results when CBO is on:
{code}
select * from (
select part1,randum123
from (SELECT *, cast(rand() as double) AS randum123 FROM testA where part1='CA'
and part2 = 'ABC') a
where randum123 <= 0.5) s where s.randum123 > 0.25 limit 20;
The plan of the query is as follows:
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: testa
Statistics: Num rows: 2 Data size: 4580 Basic stats: COMPLETE
Column stats: NONE
Filter Operator
predicate: ((rand() <= 0.5D) and (rand() > 0.25D)) (type: boolean)
Statistics: Num rows: 1 Data size: 2290 Basic stats: COMPLETE
Column stats: NONE
Select Operator
expressions: 'CA' (type: string), rand() (type: double)
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 2290 Basic stats: COMPLETE
Column stats: NONE
Limit
Number of rows: 20
Statistics: Num rows: 1 Data size: 2290 Basic stats: COMPLETE
Column stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 1 Data size: 2290 Basic stats:
COMPLETE Column stats: NONE
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: 20
Processor Tree:
ListSink
{code}
The relevant part in the plan is the filter:
{code}
Filter Operator
predicate: ((rand() <= 0.5D) and (rand() > 0.25D)) (type: boolean)
{code}
The predicates randum123 <= 0.5 and s.randum123 > 0.25 were pushed down. And
randum123 was resolved to rand(). This is bad because it will result in
invocation of rand() two times and rand() UDF is non-deterministic. Both the
rand calls can generate values that can satisfy the predicates independently,
but not together, whereas the original intention of the query is to give
results when rand falls between 0.25 and 0.5.
A sample result:
{code}
CA 0.9191984370369802
CA 0.397933021566812
{code}
where the condition was not satisfied.
was:
The following query can give wrong results when CBO is on:
{code}
select * from (
select part1,randum123
from (SELECT *, cast(rand() as double) AS randum123 FROM testA where part1='CA'
and part2 = 'ABC') a
where randum123 <= 0.5) s where s.randum123 > 0.25 limit 20;
The plan of the query is as follows:
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: testa
Statistics: Num rows: 2 Data size: 4580 Basic stats: COMPLETE
Column stats: NONE
Filter Operator
predicate: ((rand() <= 0.5D) and (rand() > 0.25D)) (type: boolean)
Statistics: Num rows: 1 Data size: 2290 Basic stats: COMPLETE
Column stats: NONE
Select Operator
expressions: 'CA' (type: string), rand() (type: double)
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 2290 Basic stats: COMPLETE
Column stats: NONE
Limit
Number of rows: 20
Statistics: Num rows: 1 Data size: 2290 Basic stats: COMPLETE
Column stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 1 Data size: 2290 Basic stats:
COMPLETE Column stats: NONE
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: 20
Processor Tree:
ListSink
{code}
The relevant part in the plan is the filter:
{code}
Filter Operator
predicate: ((rand() <= 0.5D) and (rand() > 0.25D)) (type: boolean)
{code}
The predicates s.randum123 > 0.25 and s.randum123 > 0.25 were pushed down. And
randum123 was resolved to rand(). This is bad because it will result in
invocation of rand() two times and rand() UDF is non-deterministic. Both the
rand calls can generate values that can satisfy the predicates independently,
but not together, whereas the original intention of the query is to give
results when rand falls between 0.25 and 0.5.
A sample result:
{code}
CA 0.91
[jira] [Commented] (HIVE-19569) alter table db1.t1 rename db2.t2 generates MetaStoreEventListener.onDropTable()
[
https://issues.apache.org/jira/browse/HIVE-19569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516002#comment-16516002
]
Hive QA commented on HIVE-19569:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-11895/patches/PreCommit-HIVE-Build-11895.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11895/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> alter table db1.t1 rename db2.t2 generates
> MetaStoreEventListener.onDropTable()
> ---
>
> Key: HIVE-19569
> URL: https://issues.apache.org/jira/browse/HIVE-19569
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Standalone Metastore, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-19569.01-branch-3.patch, HIVE-19569.01.patch,
> HIVE-19569.02.patch, HIVE-19569.03.patch, HIVE-19569.04.patch
>
>
> When renaming a table within the same DB, this operation causes
> {{MetaStoreEventListener.onAlterTable()}} to fire but when changing DB name
> for a table it causes {{MetaStoreEventListener.onDropTable()}} +
> {{MetaStoreEventListener.onCreateTable()}}.
> The files from original table are moved to new table location.
> This creates confusing semantics since any logic in {{onDropTable()}} doesn't
> know about the larger context, i.e. that there will be a matching
> {{onCreateTable()}}.
> In particular, this causes a problem for Acid tables since files moved from
> old table use WriteIDs that are not meaningful with the context of new table.
> Current implementation is due to replication. This should ideally be changed
> to raise a "not supported" error for tables that are marked for replication.
> cc [~sankarh]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19929) Vectorization: Recheck for vectorization wrong results/execution failures
[ https://issues.apache.org/jira/browse/HIVE-19929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-19929: --- > Vectorization: Recheck for vectorization wrong results/execution failures > - > > Key: HIVE-19929 > URL: https://issues.apache.org/jira/browse/HIVE-19929 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > Use test variables hive.test.vectorized.execution.enabled.override=enable and > hive.test.vectorization.suppress.explain.execution.mode=true to look for > wrong results/execution failures when vectorization is forced ON and > "Execution mode: vectorized" is suppressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19886) Logs may be directed to 2 files if --hiveconf hive.log.file is used
[ https://issues.apache.org/jira/browse/HIVE-19886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-19886: -- Labels: pull-request-available (was: ) > Logs may be directed to 2 files if --hiveconf hive.log.file is used > --- > > Key: HIVE-19886 > URL: https://issues.apache.org/jira/browse/HIVE-19886 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 3.1.0, 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Jaume M >Priority: Major > Labels: pull-request-available > > hive launch script explicitly specific log4j2 configuration file to use. The > main() methods in HiveServer2 and HiveMetastore reconfigures the logger based > on user input via --hiveconf hive.log.file. This may cause logs to end up in > 2 different files. Initial logs goes to the file specified in > hive-log4j2.properties and after logger reconfiguration the rest of the logs > goes to the file specified via --hiveconf hive.log.file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19886) Logs may be directed to 2 files if --hiveconf hive.log.file is used
[ https://issues.apache.org/jira/browse/HIVE-19886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516011#comment-16516011 ] ASF GitHub Bot commented on HIVE-19886: --- GitHub user beltran opened a pull request: https://github.com/apache/hive/pull/375 HIVE-19886: Logs may be directed to 2 files if --hiveconf hive.log.fi… …le is used. This is still dumping some logs for the metastore before the reconfiguration is done: ``` 2018-06-18T16:39:17,109 INFO [main] conf.MetastoreConf: Found configuration file file:/hadoop-3.1.0/etc/hadoop/hive-site.xml 2018-06-18T16:39:17,364 INFO [main] conf.MetastoreConf: Unable to find config file hivemetastore-site.xml 2018-06-18T16:39:17,364 INFO [main] conf.MetastoreConf: Found configuration file null 2018-06-18T16:39:17,365 INFO [main] conf.MetastoreConf: Unable to find config file metastore-site.xml 2018-06-18T16:39:17,365 INFO [main] conf.MetastoreConf: Found configuration file null ``` (and that's it). Before, the loggers created before the logging configuration happened sent the logs to the wrong file forever. It's hard to get rid of those because `MetastoreConf` is used to create the configure logging(and therefore spits logs before the configuration happens) so some refactoring would have to be done. You can merge this pull request into a Git repository by running: $ git pull https://github.com/beltran/hive HIVE-19886 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/375.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #375 commit a4b0fbff76ff912823e912b7f2104fbd7db88116 Author: Jaume Marhuenda Date: 2018-06-18T16:44:13Z HIVE-19886: Logs may be directed to 2 files if --hiveconf hive.log.file is used > Logs may be directed to 2 files if --hiveconf hive.log.file is used > --- > > Key: HIVE-19886 > URL: https://issues.apache.org/jira/browse/HIVE-19886 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 3.1.0, 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Jaume M >Priority: Major > Labels: pull-request-available > > hive launch script explicitly specific log4j2 configuration file to use. The > main() methods in HiveServer2 and HiveMetastore reconfigures the logger based > on user input via --hiveconf hive.log.file. This may cause logs to end up in > 2 different files. Initial logs goes to the file specified in > hive-log4j2.properties and after logger reconfiguration the rest of the logs > goes to the file specified via --hiveconf hive.log.file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19929) Vectorization: Recheck for vectorization wrong results/execution failures
[ https://issues.apache.org/jira/browse/HIVE-19929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19929: Attachment: HIVE-19929.01.patch > Vectorization: Recheck for vectorization wrong results/execution failures > - > > Key: HIVE-19929 > URL: https://issues.apache.org/jira/browse/HIVE-19929 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19929.01.patch > > > Use test variables hive.test.vectorized.execution.enabled.override=enable and > hive.test.vectorization.suppress.explain.execution.mode=true to look for > wrong results/execution failures when vectorization is forced ON and > "Execution mode: vectorized" is suppressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19886) Logs may be directed to 2 files if --hiveconf hive.log.file is used
[ https://issues.apache.org/jira/browse/HIVE-19886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-19886: --- Attachment: HIVE-19886.patch Status: Patch Available (was: Open) > Logs may be directed to 2 files if --hiveconf hive.log.file is used > --- > > Key: HIVE-19886 > URL: https://issues.apache.org/jira/browse/HIVE-19886 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 3.1.0, 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Jaume M >Priority: Major > Labels: pull-request-available > Attachments: HIVE-19886.patch > > > hive launch script explicitly specific log4j2 configuration file to use. The > main() methods in HiveServer2 and HiveMetastore reconfigures the logger based > on user input via --hiveconf hive.log.file. This may cause logs to end up in > 2 different files. Initial logs goes to the file specified in > hive-log4j2.properties and after logger reconfiguration the rest of the logs > goes to the file specified via --hiveconf hive.log.file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19929) Vectorization: Recheck for vectorization wrong results/execution failures
[ https://issues.apache.org/jira/browse/HIVE-19929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19929: Status: Patch Available (was: Open) > Vectorization: Recheck for vectorization wrong results/execution failures > - > > Key: HIVE-19929 > URL: https://issues.apache.org/jira/browse/HIVE-19929 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19929.01.patch > > > Use test variables hive.test.vectorized.execution.enabled.override=enable and > hive.test.vectorization.suppress.explain.execution.mode=true to look for > wrong results/execution failures when vectorization is forced ON and > "Execution mode: vectorized" is suppressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516021#comment-16516021 ] Eugene Koifman commented on HIVE-17683: --- sorry for the delay. Will try get to this (and your other patch) next week. > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19920) Schematool fails in embedded mode when auth is on
[ https://issues.apache.org/jira/browse/HIVE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-19920: -- Attachment: HIVE-19920.3.patch > Schematool fails in embedded mode when auth is on > - > > Key: HIVE-19920 > URL: https://issues.apache.org/jira/browse/HIVE-19920 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19920.1.patch, HIVE-19920.2.patch, > HIVE-19920.3.patch > > > This is a follow up of HIVE-19775. We need to override more properties in > embedded hs2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19920) Schematool fails in embedded mode when auth is on
[ https://issues.apache.org/jira/browse/HIVE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516024#comment-16516024 ] Daniel Dai commented on HIVE-19920: --- Local test pass. Trigger rerun anyway. > Schematool fails in embedded mode when auth is on > - > > Key: HIVE-19920 > URL: https://issues.apache.org/jira/browse/HIVE-19920 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19920.1.patch, HIVE-19920.2.patch, > HIVE-19920.3.patch > > > This is a follow up of HIVE-19775. We need to override more properties in > embedded hs2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19532) fix tests for master-txnstats branch
[ https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Yeom updated HIVE-19532: -- Attachment: HIVE-19532.04.patch > fix tests for master-txnstats branch > > > Key: HIVE-19532 > URL: https://issues.apache.org/jira/browse/HIVE-19532 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch, > HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch, > HIVE-19532.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19532) fix tests for master-txnstats branch
[ https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516028#comment-16516028 ] Steve Yeom commented on HIVE-19532: --- Hi [~sershe] I think this patch 04 can be the base of a new project branch if we need. I definitely need to check the results of the p-commit test suite run. > fix tests for master-txnstats branch > > > Key: HIVE-19532 > URL: https://issues.apache.org/jira/browse/HIVE-19532 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch, > HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch, > HIVE-19532.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19867) Test and verify Concurrent INSERTS
[ https://issues.apache.org/jira/browse/HIVE-19867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516053#comment-16516053 ] Steve Yeom commented on HIVE-19867: --- I found the patch 03 does not address the cases of concurrent writes to a table. If writeId for transactional stats is saved into TBLS/PARTITIONS, then it can be used to figure out concurrent writes by comparing a write with its writeIdList. Simply by checking it is in the list or not. > Test and verify Concurrent INSERTS > > > Key: HIVE-19867 > URL: https://issues.apache.org/jira/browse/HIVE-19867 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19922) TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flaky
[ https://issues.apache.org/jira/browse/HIVE-19922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516055#comment-16516055 ] Daniel Dai commented on HIVE-19922: --- Thanks [~pvary], I also ran couple of tests to make sure the flaskiness is introduced by HIVE-19810 or not. So far not conclusive yet, I will run a few more times and update the ticket. > TestMiniDruidKafkaCliDriver[druidkafkamini_basic] is flaky > -- > > Key: HIVE-19922 > URL: https://issues.apache.org/jira/browse/HIVE-19922 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Priority: Major > Attachments: HIVE-19922.2.patch, HIVE-19922.patch > > > Consistently failing in the last 4 runs. > See: > [https://builds.apache.org/job/PreCommit-HIVE-Build/11824/testReport/org.apache.hadoop.hive.cli/TestMiniDruidKafkaCliDriver/testCliDriver_druidkafkamini_basic_/history/] > Can not reproduce the failure locally :( > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19865) Full ACID table stats has wrong rawDataSize
[ https://issues.apache.org/jira/browse/HIVE-19865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516057#comment-16516057 ] Steve Yeom commented on HIVE-19865: --- Not yet fixed. I have added "stats_sizebug.q" to address this issue. > Full ACID table stats has wrong rawDataSize > --- > > Key: HIVE-19865 > URL: https://issues.apache.org/jira/browse/HIVE-19865 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19931) stats_nonpart.q test run shows possibly wrong results.
[ https://issues.apache.org/jira/browse/HIVE-19931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Yeom reassigned HIVE-19931: - Assignee: Steve Yeom > stats_nonpart.q test run shows possibly wrong results. > -- > > Key: HIVE-19931 > URL: https://issues.apache.org/jira/browse/HIVE-19931 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19930) Resolve explain.out differences introduced by transactional metastore statistics
[ https://issues.apache.org/jira/browse/HIVE-19930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-19930: --- Summary: Resolve explain.out differences introduced by transactional metastore statistics (was: Resolve explain.out differences introduced by this fearture) > Resolve explain.out differences introduced by transactional metastore > statistics > > > Key: HIVE-19930 > URL: https://issues.apache.org/jira/browse/HIVE-19930 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19899) Support stored as JsonFile
[ https://issues.apache.org/jira/browse/HIVE-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-19899: Attachment: HIVE-19899.4.patch > Support stored as JsonFile > --- > > Key: HIVE-19899 > URL: https://issues.apache.org/jira/browse/HIVE-19899 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 > Environment: This is to add "stored as jsonfile" support for json > file format. >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-19899.1.patch, HIVE-19899.2.patch, > HIVE-19899.3.patch, HIVE-19899.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19899) Support stored as JsonFile
[ https://issues.apache.org/jira/browse/HIVE-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-19899: Attachment: HIVE-19899.4.patch > Support stored as JsonFile > --- > > Key: HIVE-19899 > URL: https://issues.apache.org/jira/browse/HIVE-19899 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 > Environment: This is to add "stored as jsonfile" support for json > file format. >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-19899.1.patch, HIVE-19899.2.patch, > HIVE-19899.3.patch, HIVE-19899.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19899) Support stored as JsonFile
[ https://issues.apache.org/jira/browse/HIVE-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-19899: Attachment: (was: HIVE-19899.4.patch) > Support stored as JsonFile > --- > > Key: HIVE-19899 > URL: https://issues.apache.org/jira/browse/HIVE-19899 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 > Environment: This is to add "stored as jsonfile" support for json > file format. >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-19899.1.patch, HIVE-19899.2.patch, > HIVE-19899.3.patch, HIVE-19899.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19899) Support stored as JsonFile
[ https://issues.apache.org/jira/browse/HIVE-19899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516105#comment-16516105 ] Aihua Xu commented on HIVE-19899: - patch-4: TestHCatStorer test failures are related. We have private storageFormat defined in child class which causes the issue. Removed duplicated private definition in TestHCatStorer. > Support stored as JsonFile > --- > > Key: HIVE-19899 > URL: https://issues.apache.org/jira/browse/HIVE-19899 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 > Environment: This is to add "stored as jsonfile" support for json > file format. >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-19899.1.patch, HIVE-19899.2.patch, > HIVE-19899.3.patch, HIVE-19899.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19703) GenericUDTFGetSplits never uses num splits argument
[
https://issues.apache.org/jira/browse/HIVE-19703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M reassigned HIVE-19703:
--
Assignee: Jaume M
> GenericUDTFGetSplits never uses num splits argument
> ---
>
> Key: HIVE-19703
> URL: https://issues.apache.org/jira/browse/HIVE-19703
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Eric Wohlstadter
>Assignee: Jaume M
>Priority: Major
>
> The description for GenericUDTFGetSplits says
> {code}
> Returns an array of length int serialized splits for the referenced tables
> string.
> {code}
> but the argument to control the number of splits is DOA.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19837) Setting to have different default location for external tables
[
https://issues.apache.org/jira/browse/HIVE-19837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516137#comment-16516137
]
Vineet Garg commented on HIVE-19837:
[~jdere] on branch-3 {{TestWarehouseExternalDir}} is failing. Can you please
take a look at it?
Ref: https://builds.apache.org/job/PreCommit-HIVE-Build/11874/testReport
> Setting to have different default location for external tables
> --
>
> Key: HIVE-19837
> URL: https://issues.apache.org/jira/browse/HIVE-19837
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19837.1.patch, HIVE-19837.3.patch
>
>
> Allow external tables to have a different default location than managed tables
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
