[jira] [Updated] (HIVE-20094) Update Druid to 0.12.1 version
[ https://issues.apache.org/jira/browse/HIVE-20094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-20094: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Slim! > Update Druid to 0.12.1 version > -- > > Key: HIVE-20094 > URL: https://issues.apache.org/jira/browse/HIVE-20094 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-20094.patch, HIVE-20094.patch, HIVE-20094.patch > > > As per Jira title. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19441) Use LLAP test Driver for Druid test suites
[
https://issues.apache.org/jira/browse/HIVE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541175#comment-16541175
]
Hive QA commented on HIVE-19441:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
48s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
34s{color} | {color:red} qtest in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/qtest: The patch generated 5 new + 3 unchanged
- 0 fixed = 8 total (was 3) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/util: The patch generated 5 new + 90 unchanged
- 0 fixed = 95 total (was 90) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12545/dev-support/hive-personality.sh
|
| git revision | master / 19c2ec4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12545/yetus/patch-mvninstall-itests_qtest.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12545/yetus/diff-checkstyle-itests_qtest.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12545/yetus/diff-checkstyle-itests_util.txt
|
| modules | C: ql itests itests/qtest itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12545/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use LLAP test Driver for Druid test suites
> --
>
> Key: HIVE-19441
> URL: https://issues.apache.org/jira/browse/HIVE-19441
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19441.10.patch, HIVE-19441.2.patch,
> HIVE-19441.3.patch, HIVE-19441.4.patch, HIVE-19441.5.patch,
> HIVE-19441.6.patch, HIVE-19441.7.patch, HIVE-19441.8.patch,
> HIVE-19441.8.patch, HIVE-19441.9.patch,
[jira] [Updated] (HIVE-20149) TestHiveCli failing/timing out
[ https://issues.apache.org/jira/browse/HIVE-20149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20149: --- Status: Patch Available (was: Open) > TestHiveCli failing/timing out > -- > > Key: HIVE-20149 > URL: https://issues.apache.org/jira/browse/HIVE-20149 > Project: Hive > Issue Type: Bug > Components: CLI >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20149.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20149) TestHiveCli failing/timing out
[
https://issues.apache.org/jira/browse/HIVE-20149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541173#comment-16541173
]
Vineet Garg commented on HIVE-20149:
HiveCli tries to run {{connect}} command, while running connect cli/beeline
checks if hive conf is initialized if not it is initialized using {{set}}
command. Initialization consists of executing {{set}} command and then using
the result set buffered rows are created which are further used to fill conf
objec, Buffered row/row needs to check the value of
{{hive.cli.print.escape.crlf}} and therefore it also end up calling the same
method which is trying to initialize hive conf resulting in downward spiral.
I think getEscapeCRLF should not try to initialize/create conf and uses an
existing one if it is initialized or otherwise just return false since
{{connect}} or some other command should already have initialized conf by the
time {{hive.cli.print.escape.crlf}} matters.
> TestHiveCli failing/timing out
> --
>
> Key: HIVE-20149
> URL: https://issues.apache.org/jira/browse/HIVE-20149
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20149.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20149) TestHiveCli failing/timing out
[ https://issues.apache.org/jira/browse/HIVE-20149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20149: --- Attachment: HIVE-20149.1.patch > TestHiveCli failing/timing out > -- > > Key: HIVE-20149 > URL: https://issues.apache.org/jira/browse/HIVE-20149 > Project: Hive > Issue Type: Bug > Components: CLI >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20149.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20019) Remove commons-logging and move to slf4j
[ https://issues.apache.org/jira/browse/HIVE-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541172#comment-16541172 ] Prasanth Jayachandran commented on HIVE-20019: -- log4j is already added to banned dependency at compile time in this patch. > Remove commons-logging and move to slf4j > > > Key: HIVE-20019 > URL: https://issues.apache.org/jira/browse/HIVE-20019 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20019.1.patch, HIVE-20019.2.patch, > HIVE-20019.3.patch, HIVE-20019.4.patch > > > Still seeing several references to commons-logging. We should move all > classes to slf4j instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20149) TestHiveCli failing/timing out
[ https://issues.apache.org/jira/browse/HIVE-20149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-20149: -- > TestHiveCli failing/timing out > -- > > Key: HIVE-20149 > URL: https://issues.apache.org/jira/browse/HIVE-20149 > Project: Hive > Issue Type: Bug > Components: CLI >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20019) Remove commons-logging and move to slf4j
[ https://issues.apache.org/jira/browse/HIVE-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541163#comment-16541163 ] Ashutosh Chauhan commented on HIVE-20019: - To future proof this.. we shall make log4j as banned dependency at compile time. > Remove commons-logging and move to slf4j > > > Key: HIVE-20019 > URL: https://issues.apache.org/jira/browse/HIVE-20019 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20019.1.patch, HIVE-20019.2.patch, > HIVE-20019.3.patch, HIVE-20019.4.patch > > > Still seeing several references to commons-logging. We should move all > classes to slf4j instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20094) Update Druid to 0.12.1 version
[
https://issues.apache.org/jira/browse/HIVE-20094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541146#comment-16541146
]
Hive QA commented on HIVE-20094:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931142/HIVE-20094.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14643 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12544/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12544/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12544/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931142 - PreCommit-HIVE-Build
> Update Druid to 0.12.1 version
> --
>
> Key: HIVE-20094
> URL: https://issues.apache.org/jira/browse/HIVE-20094
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Minor
> Attachments: HIVE-20094.patch, HIVE-20094.patch, HIVE-20094.patch
>
>
> As per Jira title.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20094) Update Druid to 0.12.1 version
[
https://issues.apache.org/jira/browse/HIVE-20094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541141#comment-16541141
]
Hive QA commented on HIVE-20094:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} druid-handler in master has 13 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12544/dev-support/hive-personality.sh
|
| git revision | master / 19c2ec4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: druid-handler . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12544/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Update Druid to 0.12.1 version
> --
>
> Key: HIVE-20094
> URL: https://issues.apache.org/jira/browse/HIVE-20094
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Minor
> Attachments: HIVE-20094.patch, HIVE-20094.patch, HIVE-20094.patch
>
>
> As per Jira title.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19820) add ACID stats support to background stats updater and fix bunch of edge cases found in SU tests
[ https://issues.apache.org/jira/browse/HIVE-19820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541133#comment-16541133 ] Sergey Shelukhin commented on HIVE-19820: - Adds support and tests for various alter table operations for txn stats. The failing tests above are mostly due to issues with: 1) Conversion to MM (no write ID yet it calls update stats for some reason). 2) Temporary tables (didn't look at specific problem yet). 3) Some NPE in alter (need to look at changes to code paths) > add ACID stats support to background stats updater and fix bunch of edge > cases found in SU tests > > > Key: HIVE-19820 > URL: https://issues.apache.org/jira/browse/HIVE-19820 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19820.01-master-txnstats.patch, > HIVE-19820.01.patch, HIVE-19820.02-master-txnstats.patch, > HIVE-19820.03-master-txnstats.patch, HIVE-19820.04-master-txnstats.patch, > HIVE-19820.patch, branch-19820.02.nogen.patch, branch-19820.nogen.patch, > branch-19820.nogen.patch > > > Follow-up from HIVE-19418. > Right now it checks whether stats are valid in an old-fashioned way... and > also gets ACID state, and discards it without using. > When ACID stats are implemented, ACID state needs to be used to do > version-aware valid stats checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19820) add ACID stats support to background stats updater and fix bunch of edge cases found in SU tests
[ https://issues.apache.org/jira/browse/HIVE-19820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19820: Attachment: branch-19820.02.nogen.patch > add ACID stats support to background stats updater and fix bunch of edge > cases found in SU tests > > > Key: HIVE-19820 > URL: https://issues.apache.org/jira/browse/HIVE-19820 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19820.01-master-txnstats.patch, > HIVE-19820.01.patch, HIVE-19820.02-master-txnstats.patch, > HIVE-19820.03-master-txnstats.patch, HIVE-19820.04-master-txnstats.patch, > HIVE-19820.patch, branch-19820.02.nogen.patch, branch-19820.nogen.patch, > branch-19820.nogen.patch > > > Follow-up from HIVE-19418. > Right now it checks whether stats are valid in an old-fashioned way... and > also gets ACID state, and discards it without using. > When ACID stats are implemented, ACID state needs to be used to do > version-aware valid stats checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20138) Sessions: Closing Tez sessions if opened outside of default sessions throws NPE
[
https://issues.apache.org/jira/browse/HIVE-20138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541119#comment-16541119
]
Sergey Shelukhin commented on HIVE-20138:
-
Looks like a dup of HIVE-19777
> Sessions: Closing Tez sessions if opened outside of default sessions throws
> NPE
> ---
>
> Key: HIVE-20138
> URL: https://issues.apache.org/jira/browse/HIVE-20138
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Gopal V
>Priority: Major
>
> On a busy cluster, if a session is currently in ACCEPTED state (i.e YARN
> hasn't started it yet), running a query will NPE.
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.getSession(TezSessionState.java:711)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:646)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:353)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:467)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.WorkloadManagerFederation.getUnmanagedSession(WorkloadManagerFederation.java:66)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.WorkloadManagerFederation.getSession(WorkloadManagerFederation.java:38)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:184)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17852) remove support for list bucketing "stored as directories" in 3.0
[
https://issues.apache.org/jira/browse/HIVE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541091#comment-16541091
]
Hive QA commented on HIVE-17852:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931139/HIVE-17852.19.patch
{color:green}SUCCESS:{color} +1 due to 36 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14640 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar.org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar
(batchId=235)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12543/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12543/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12543/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931139 - PreCommit-HIVE-Build
> remove support for list bucketing "stored as directories" in 3.0
>
>
> Key: HIVE-17852
> URL: https://issues.apache.org/jira/browse/HIVE-17852
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17852.01.patch, HIVE-17852.02.patch,
> HIVE-17852.03.patch, HIVE-17852.04.patch, HIVE-17852.05.patch,
> HIVE-17852.06.patch, HIVE-17852.07.patch, HIVE-17852.08.patch,
> HIVE-17852.09.patch, HIVE-17852.10.patch, HIVE-17852.11.patch,
> HIVE-17852.12.patch, HIVE-17852.13.patch, HIVE-17852.14.patch,
> HIVE-17852.15.patch, HIVE-17852.16.patch, HIVE-17852.17.patch,
> HIVE-17852.18.patch, HIVE-17852.19.patch
>
>
> From the email thread:
> 1) LB, when stored as directories, adds a lot of low-level complexity to Hive
> tables that has to be accounted for in many places in the code where the
> files are written or modified - from FSOP to ACID/replication/export.
> 2) While working on some FSOP code I noticed that some of that logic is
> broken - e.g. the duplicate file removal from tasks, a pretty fundamental
> correctness feature in Hive, may be broken. LB also doesn’t appear to be
> compatible with e.g. regular bucketing.
> 3) The feature hasn’t seen development activity in a while; it also doesn’t
> appear to be used a lot.
> Keeping with the theme of cleaning up “legacy” code for 3.0, I was proposing
> we remove it.
> (2) also suggested that, if needed, it might be easier to implement similar
> functionality by adding some flexibility to partitions (which LB directories
> look like anyway); that would also keep the logic on a higher level of
> abstraction (split generation, partition pruning) as opposed to many
> low-level places like FSOP, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20146) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.apache.hadoop.hive.serder2.OpenCSVSerde
[
https://issues.apache.org/jira/browse/HIVE-20146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541079#comment-16541079
]
gavin_shi commented on HIVE-20146:
--
thanks
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde:
> org.apache.hadoop.hive.serder2.OpenCSVSerde
> --
>
> Key: HIVE-20146
> URL: https://issues.apache.org/jira/browse/HIVE-20146
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.2
>Reporter: gavin_shi
>Priority: Major
>
> I am trying to create table in hive, but I am getting the following error
> {code:java}
> //代码占位符
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde:
> org.apache.hadoop.hive.serder2.OpenCSVSerde
> {code}
> the command that create the table :
> {code:java}
> //代码占位符
> create table csv_tab(id string,name string) row format serde
> 'org.apache.hadoop.hive.serder2.OpenCSVSerde' with
> serdeproperties("separatorChar"="\t","quoteChar"="'","escapeChar"="\\")
> stored as TEXTFILE;
> {code}
> hadoop version:2.6.0
> hive version:1.2.2
> please help me.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20032) Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541076#comment-16541076 ] Rui Li commented on HIVE-20032: --- Hi [~stakiar], thanks for working on this. There're other cases where RDD is cached, e.g. parallel order by. So you need to serialize the hash code in all these cases (maybe multi-insert is another one). Having separate SerDe for caching and shuffling would be good. But I guess that needs help from Spark side. And btw, have you run benchmarks to get the improvements of this change? > Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled > - > > Key: HIVE-20032 > URL: https://issues.apache.org/jira/browse/HIVE-20032 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-20032.1.patch, HIVE-20032.2.patch, > HIVE-20032.3.patch > > > Follow up on HIVE-15104, if we don't enable RDD cacheing or groupByShuffles, > then we don't need to serialize the hashCode when shuffling data in HoS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20095) Fix jdbc external table feature
[
https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541053#comment-16541053
]
Hive QA commented on HIVE-20095:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931132/HIVE-20095.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14629 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=105)
[update_orig_table.q,explainanalyze_3.q,orc_vectorization_ppd.q,vector_non_string_partition.q,multi_count_distinct.q,explainanalyze_1.q,explainuser_3.q,explainanalyze_4.q,hybridgrace_hashjoin_1.q,delete_orig_table.q,hybridgrace_hashjoin_2.q,tez-tag.q,orc_merge12.q,explainanalyze_5.q,vector_join_part_col_char.q]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables_sysdb]
(batchId=169)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12542/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12542/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12542/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931132 - PreCommit-HIVE-Build
> Fix jdbc external table feature
> ---
>
> Key: HIVE-20095
> URL: https://issues.apache.org/jira/browse/HIVE-20095
> Project: Hive
> Issue Type: Bug
>Reporter: Jonathan Doron
>Assignee: Jonathan Doron
>Priority: Major
> Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch,
> HIVE-20095.3.patch
>
>
> It seems like the committed code for HIVE-19161
> (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423
> updates, and therefore some of the external table queries are not working
> correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20148) Introduce rule to pushdown TopNKey through plan
[ https://issues.apache.org/jira/browse/HIVE-20148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541044#comment-16541044 ] Jesus Camacho Rodriguez commented on HIVE-20148: It seems to me initial implementation should support at least: - Push through Project if expression(s) in TopNKey can be mapped to expression(s) based on Project input. - Push through GroupBy. No grouping sets. If TopNKey expression is same as GroupBy expression, we can push it and remove it from above GroupBy. If expression in TopNKey shared common prefix with GroupBy, TopNKey could be pushed through GroupBy using that prefix and kept above it. - Push through ReduceSink. If TopNKey expression is same as ReduceSink expression and order is the same, we can push it and remove it from above ReduceSink. If expression in TopNKey shared common prefix with ReduceSink including same order, TopNKey could be pushed through ReduceSink using that prefix and kept above it. - Push through FOJ. Push TopNKey expression without keys to largest input. Keep on top of FOJ. - Push through LOJ. If TopNKey expression refers fully to expressions from left input, push with rewriting of expressions and remove from top of LOJ. If TopNKey expression has a prefix that refers to expressions from left input, push with rewriting of those expressions and keep on top of LOJ. - Push through ROJ. Same as for LOJ, but change left to right. [~gopalv], thoughts? Does it seem correct? Should we consider any other for the minimum implementation? > Introduce rule to pushdown TopNKey through plan > --- > > Key: HIVE-20148 > URL: https://issues.apache.org/jira/browse/HIVE-20148 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Teddy Choi >Priority: Major > > Follow-up for HIVE-17896, which introduces TopNKey operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20095) Fix jdbc external table feature
[
https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541023#comment-16541023
]
Hive QA commented on HIVE-20095:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} jdbc-handler in master has 8 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} jdbc-handler: The patch generated 30 new + 23
unchanged - 2 fixed = 53 total (was 25) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
30s{color} | {color:red} jdbc-handler generated 3 new + 8 unchanged - 0 fixed =
11 total (was 8) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:jdbc-handler |
| | Exception is caught when Exception is not thrown in
org.apache.hive.storage.jdbc.JdbcSerDe.initialize(Configuration, Properties)
At JdbcSerDe.java:is not thrown in
org.apache.hive.storage.jdbc.JdbcSerDe.initialize(Configuration, Properties)
At JdbcSerDe.java:[line 111] |
| |
org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnTypes(Configuration)
may fail to clean up java.sql.ResultSet Obligation to clean up resource
created at GenericJdbcDatabaseAccessor.java:up java.sql.ResultSet Obligation
to clean up resource created at GenericJdbcDatabaseAccessor.java:[line 115] is
not discharged |
| |
org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnTypes(Configuration)
may fail to clean up java.sql.Statement Obligation to clean up resource
created at GenericJdbcDatabaseAccessor.java:up java.sql.Statement Obligation
to clean up resource created at GenericJdbcDatabaseAccessor.java:[line 114] is
not discharged |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12542/dev-support/hive-personality.sh
|
| git revision | master / 19c2ec4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12542/yetus/diff-checkstyle-jdbc-handler.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12542/yetus/new-findbugs-jdbc-handler.html
|
| modules | C: ql jdbc-handler itests U: . |
| Console output |
[jira] [Commented] (HIVE-20131) SQL Script changes for creating txn write notification in 3.2.0 files
[
https://issues.apache.org/jira/browse/HIVE-20131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540991#comment-16540991
]
Hive QA commented on HIVE-20131:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931108/HIVE-20131.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14643 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12541/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12541/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12541/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931108 - PreCommit-HIVE-Build
> SQL Script changes for creating txn write notification in 3.2.0 files
> ---
>
> Key: HIVE-20131
> URL: https://issues.apache.org/jira/browse/HIVE-20131
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: ACID, DR, pull-request-available, replication
> Fix For: 4.0.0
>
> Attachments: HIVE-20131.01.patch
>
>
> 1. Change partition name size from 1024 to 767 . (mySQL 5.6 and before that
> supports max 767 length keys)
> 2. Remove the create txn_write_notification_log table creation from 3.1.0
> scripts and add a new scripts for 3.2.0
> 3. Remove the file 3.1.0-to-4.0.0 and instead add file for 3.2.0-to-4.0.0 and
> 3.1.0-to-3.2.0
> 4. Change in metastore init schema xml file to take 4.0.0 instead of 3.1.0
> as current version.
> h1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20131) SQL Script changes for creating txn write notification in 3.2.0 files
[
https://issues.apache.org/jira/browse/HIVE-20131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540971#comment-16540971
]
Hive QA commented on HIVE-20131:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 59 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12541/dev-support/hive-personality.sh
|
| git revision | master / 19c2ec4 |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12541/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12541/yetus/whitespace-tabs.txt
|
| modules | C: standalone-metastore metastore . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12541/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SQL Script changes for creating txn write notification in 3.2.0 files
> ---
>
> Key: HIVE-20131
> URL: https://issues.apache.org/jira/browse/HIVE-20131
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: ACID, DR, pull-request-available, replication
> Fix For: 4.0.0
>
> Attachments: HIVE-20131.01.patch
>
>
> 1. Change partition name size from 1024 to 767 . (mySQL 5.6 and before that
> supports max 767 length keys)
> 2. Remove the create txn_write_notification_log table creation from 3.1.0
> scripts and add a new scripts for 3.2.0
> 3. Remove the file 3.1.0-to-4.0.0 and instead add file for 3.2.0-to-4.0.0 and
> 3.1.0-to-3.2.0
> 4. Change in metastore init schema xml file to take 4.0.0 instead of 3.1.0
> as current version.
> h1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18807) Fix broken tests caused by HIVE-18493
[
https://issues.apache.org/jira/browse/HIVE-18807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540962#comment-16540962
]
Vineet Garg commented on HIVE-18807:
Bumped into this issue of TestHiveCli taking forever to run and causing
timeout. This is due to HIVE-18493 commit which added call to
{{beeLine.getOpts().getEscapeCRLF()}} in Rows. If it is not beeline
getEscapeCRLF tries to get hive conf using
{{beeLine.getCommands().getHiveConf(true)}} and eventually calling
{{Commands::getHiveConf}}. But this method again can call to Row:Row()
resulting in infinite loop. This is what is causing TestHiveCli timeout.
> Fix broken tests caused by HIVE-18493
> -
>
> Key: HIVE-18807
> URL: https://issues.apache.org/jira/browse/HIVE-18807
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Major
>
> Broken tests =
> org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB
> org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress
> org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19820) add ACID stats support to background stats updater and fix bunch of edge cases found in SU tests
[
https://issues.apache.org/jira/browse/HIVE-19820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540952#comment-16540952
]
Hive QA commented on HIVE-19820:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
15s{color} | {color:blue} standalone-metastore in master has 217 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} storage-api: The patch generated 1 new + 3 unchanged -
0 fixed = 4 total (was 3) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
54s{color} | {color:red} standalone-metastore: The patch generated 103 new +
2611 unchanged - 33 fixed = 2714 total (was 2644) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
11s{color} | {color:red} ql: The patch generated 29 new + 2168 unchanged - 10
fixed = 2197 total (was 2178) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 5m
19s{color} | {color:red} root: The patch generated 138 new + 5331 unchanged -
44 fixed = 5469 total (was 5375) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hcatalog-unit: The patch generated 5 new + 27
unchanged - 1 fixed = 32 total (was 28) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 403 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
10s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
10s{color} | {color:red} patch/storage-api cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
53s{color} | {color:red} patch/standalone-metastore cannot run
setBugDatabaseInfo from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 5m
8s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
0s{color} | {color:red} patch/itests/hive-unit cannot run setBugDatabaseInfo
from findbugs {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
39s{color} | {color:red} standalone-metastore generated 6 new + 54 unchanged -
0 fixed = 60 total (was 54) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 8m
11s{color} | {color:red} root generated 6 new + 371 unchanged - 0 fixed = 377
total (was 371) {color} |
|| || || || {color:brown} Other
[jira] [Comment Edited] (HIVE-18705) Improve HiveMetaStoreClient.dropDatabase
[
https://issues.apache.org/jira/browse/HIVE-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540939#comment-16540939
]
Alexander Kolbasov edited comment on HIVE-18705 at 7/12/18 12:43 AM:
-
[~szita] Can you post reviewboard link?
Also, what commit ID is your patch based on?
was (Author: akolb):
[~szita] Can you post reviewboard link?
Thanks,
- Sasha
> Improve HiveMetaStoreClient.dropDatabase
>
>
> Key: HIVE-18705
> URL: https://issues.apache.org/jira/browse/HIVE-18705
> Project: Hive
> Issue Type: Improvement
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-18705.0.patch, HIVE-18705.1.patch,
> HIVE-18705.2.patch, HIVE-18705.4.patch, HIVE-18705.5.patch,
> HIVE-18705.6.patch, HIVE-18705.7.patch, HIVE-18705.8.patch
>
>
> {{HiveMetaStoreClient.dropDatabase}} has a strange implementation to ensure
> dealing with client side hooks (for non-native tables e.g. HBase). Currently
> it starts by retrieving all the tables from HMS, and then sends {{dropTable}}
> calls to HMS table-by-table. At the end a {{dropDatabase}} just to be sure :)
> I believe this could be refactored so that it speeds up the dropDB in
> situations where the average table count per DB is very high.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18705) Improve HiveMetaStoreClient.dropDatabase
[
https://issues.apache.org/jira/browse/HIVE-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540939#comment-16540939
]
Alexander Kolbasov commented on HIVE-18705:
---
[~szita] Can you post reviewboard link?
Thanks,
- Sasha
> Improve HiveMetaStoreClient.dropDatabase
>
>
> Key: HIVE-18705
> URL: https://issues.apache.org/jira/browse/HIVE-18705
> Project: Hive
> Issue Type: Improvement
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-18705.0.patch, HIVE-18705.1.patch,
> HIVE-18705.2.patch, HIVE-18705.4.patch, HIVE-18705.5.patch,
> HIVE-18705.6.patch, HIVE-18705.7.patch, HIVE-18705.8.patch
>
>
> {{HiveMetaStoreClient.dropDatabase}} has a strange implementation to ensure
> dealing with client side hooks (for non-native tables e.g. HBase). Currently
> it starts by retrieving all the tables from HMS, and then sends {{dropTable}}
> calls to HMS table-by-table. At the end a {{dropDatabase}} just to be sure :)
> I believe this could be refactored so that it speeds up the dropDB in
> situations where the average table count per DB is very high.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18705) Improve HiveMetaStoreClient.dropDatabase
[
https://issues.apache.org/jira/browse/HIVE-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540934#comment-16540934
]
Vihang Karajgaonkar commented on HIVE-18705:
Hi [~szita] Sorry for taking long to respond. I was on vacation last week. I
have a (probably dumb :)) question. Where does the performance improvement come
from in this patch. I see you are reducing the number of getTable calls by
using batch iterator, but for "large" databases we still do a drop table
one-by-one and then do the dropDatabase. Am I missing something?
> Improve HiveMetaStoreClient.dropDatabase
>
>
> Key: HIVE-18705
> URL: https://issues.apache.org/jira/browse/HIVE-18705
> Project: Hive
> Issue Type: Improvement
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-18705.0.patch, HIVE-18705.1.patch,
> HIVE-18705.2.patch, HIVE-18705.4.patch, HIVE-18705.5.patch,
> HIVE-18705.6.patch, HIVE-18705.7.patch, HIVE-18705.8.patch
>
>
> {{HiveMetaStoreClient.dropDatabase}} has a strange implementation to ensure
> dealing with client side hooks (for non-native tables e.g. HBase). Currently
> it starts by retrieving all the tables from HMS, and then sends {{dropTable}}
> calls to HMS table-by-table. At the end a {{dropDatabase}} just to be sure :)
> I believe this could be refactored so that it speeds up the dropDB in
> situations where the average table count per DB is very high.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19375) "'transactional'='false' is no longer a valid property and will be ignored:
[
https://issues.apache.org/jira/browse/HIVE-19375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540931#comment-16540931
]
Jason Dere commented on HIVE-19375:
---
ok, +1
> "'transactional'='false' is no longer a valid property and will be ignored:
>
>
> Key: HIVE-19375
> URL: https://issues.apache.org/jira/browse/HIVE-19375
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Minor
> Attachments: HIVE-19375.01.patch
>
>
> from {{TransactionalValidationListener.handleCreateTableTransactionalProp()}}
> {noformat}
> if ("false".equalsIgnoreCase(transactional)) {
> // just drop transactional=false. For backward compatibility in case
> someone has scripts
> // with transactional=false
> LOG.info("'transactional'='false' is no longer a valid property and
> will be ignored: " +
> Warehouse.getQualifiedName(newTable));
> return;
> }
> {noformat}
> this msg is misleading since with metastore.create.as.acid=true, setting
> transactional=false is valid to make a flat table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19820) add ACID stats support to background stats updater and fix bunch of edge cases found in SU tests
[
https://issues.apache.org/jira/browse/HIVE-19820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540921#comment-16540921
]
Hive QA commented on HIVE-19820:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931232/HIVE-19820.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12540/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12540/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12540/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12931232/HIVE-19820.01.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931232 - PreCommit-HIVE-Build
> add ACID stats support to background stats updater and fix bunch of edge
> cases found in SU tests
>
>
> Key: HIVE-19820
> URL: https://issues.apache.org/jira/browse/HIVE-19820
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19820.01-master-txnstats.patch,
> HIVE-19820.01.patch, HIVE-19820.02-master-txnstats.patch,
> HIVE-19820.03-master-txnstats.patch, HIVE-19820.04-master-txnstats.patch,
> HIVE-19820.patch, branch-19820.nogen.patch, branch-19820.nogen.patch
>
>
> Follow-up from HIVE-19418.
> Right now it checks whether stats are valid in an old-fashioned way... and
> also gets ACID state, and discards it without using.
> When ACID stats are implemented, ACID state needs to be used to do
> version-aware valid stats checks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19820) add ACID stats support to background stats updater and fix bunch of edge cases found in SU tests
[
https://issues.apache.org/jira/browse/HIVE-19820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540919#comment-16540919
]
Hive QA commented on HIVE-19820:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931232/HIVE-19820.01.patch
{color:green}SUCCESS:{color} +1 due to 16 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 447 failed/errored test(s), 14652 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_subquery]
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_create_temp_table]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_10]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_7]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_or_replace_view]
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view_partitioned]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_4] (batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_tmp_table]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_tmp_table]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_10] (batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_9] (batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_acid_no_masking]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_iow_temp] (batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_analyze_decimal_compare]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_partial_size]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_display_colstats_tbllvl]
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_join1]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_options1]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_precedence]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_truncate]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_windowing_expressions]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_const]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_delete_orig_table]
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join_no_keys]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_mapjoin2]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_mapjoin3]
(batchId=12)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[autoColumnStats_10]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[autoColumnStats_2]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[change_allowincompatible_vectorization_false_date]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cte_mat_4]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[current_date_timestamp]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_tmp_table]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_3]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=167)
[jira] [Assigned] (HIVE-20148) Introduce rule to pushdown TopNKey through plan
[ https://issues.apache.org/jira/browse/HIVE-20148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20148: -- > Introduce rule to pushdown TopNKey through plan > --- > > Key: HIVE-20148 > URL: https://issues.apache.org/jira/browse/HIVE-20148 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Teddy Choi >Priority: Major > > Follow-up for HIVE-17896, which introduces TopNKey operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20102) Add a couple of additional tests for query parsing
[ https://issues.apache.org/jira/browse/HIVE-20102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20102: --- Attachment: HIVE-20102.02.patch > Add a couple of additional tests for query parsing > -- > > Key: HIVE-20102 > URL: https://issues.apache.org/jira/browse/HIVE-20102 > Project: Hive > Issue Type: Improvement > Components: Parser >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20102.01.patch, HIVE-20102.02.patch, > HIVE-20102.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20146) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.apache.hadoop.hive.serder2.OpenCSVSerde
[
https://issues.apache.org/jira/browse/HIVE-20146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540901#comment-16540901
]
BELUGA BEHR commented on HIVE-20146:
You have a typo in your statement
{code}
'serder2' => 'serde2'
'org.apache.hadoop.hive.serder2.OpenCSVSerde' =>
'org.apache.hadoop.hive.serde2.OpenCSVSerde'
{code}
https://cwiki.apache.org/confluence/display/Hive/CSV+Serde
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde:
> org.apache.hadoop.hive.serder2.OpenCSVSerde
> --
>
> Key: HIVE-20146
> URL: https://issues.apache.org/jira/browse/HIVE-20146
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.2
>Reporter: gavin_shi
>Priority: Major
>
> I am trying to create table in hive, but I am getting the following error
> {code:java}
> //代码占位符
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde:
> org.apache.hadoop.hive.serder2.OpenCSVSerde
> {code}
> the command that create the table :
> {code:java}
> //代码占位符
> create table csv_tab(id string,name string) row format serde
> 'org.apache.hadoop.hive.serder2.OpenCSVSerde' with
> serdeproperties("separatorChar"="\t","quoteChar"="'","escapeChar"="\\")
> stored as TEXTFILE;
> {code}
> hadoop version:2.6.0
> hive version:1.2.2
> please help me.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20147) Hive streaming ingest is contented on synchronized logging
[ https://issues.apache.org/jira/browse/HIVE-20147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20147: -- Component/s: Transactions > Hive streaming ingest is contented on synchronized logging > -- > > Key: HIVE-20147 > URL: https://issues.apache.org/jira/browse/HIVE-20147 > Project: Hive > Issue Type: Bug > Components: Streaming, Transactions >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: Screen Shot 2018-07-11 at 4.17.27 PM.png, > sync-logger-contention.svg > > > In one of the observed profile, >30% time spent on synchronized logging. See > attachment. > We should use async logging for hive streaming ingest by default. !Screen > Shot 2018-07-11 at 4.17.27 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19532) fix tests for master-txnstats branch
[ https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540900#comment-16540900 ] Eugene Koifman edited comment on HIVE-19532 at 7/11/18 11:44 PM: - vector_delete_orig_table.q - is a test issue - fixed in HIVE-20145 - rather in addendum to HIVE-20076 was (Author: ekoifman): vector_delete_orig_table.q - is a test issue - fixed in https://issues.apache.org/jira/browse/HIVE-20145 > fix tests for master-txnstats branch > > > Key: HIVE-19532 > URL: https://issues.apache.org/jira/browse/HIVE-19532 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch, > HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch, > HIVE-19532.04.patch, HIVE-19532.05.patch, HIVE-19532.07.patch, > HIVE-19532.08.patch, HIVE-19532.09.patch, HIVE-19532.10.patch, > HIVE-19532.11.patch, HIVE-19532.12.patch, HIVE-19532.13.patch, > HIVE-19532.14.patch, HIVE-19532.15.patch, HIVE-19532.16.patch, > HIVE-19532.17.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19532) fix tests for master-txnstats branch
[ https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540900#comment-16540900 ] Eugene Koifman commented on HIVE-19532: --- vector_delete_orig_table.q - is a test issue - fixed in https://issues.apache.org/jira/browse/HIVE-20145 > fix tests for master-txnstats branch > > > Key: HIVE-19532 > URL: https://issues.apache.org/jira/browse/HIVE-19532 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch, > HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch, > HIVE-19532.04.patch, HIVE-19532.05.patch, HIVE-19532.07.patch, > HIVE-19532.08.patch, HIVE-19532.09.patch, HIVE-19532.10.patch, > HIVE-19532.11.patch, HIVE-19532.12.patch, HIVE-19532.13.patch, > HIVE-19532.14.patch, HIVE-19532.15.patch, HIVE-19532.16.patch, > HIVE-19532.17.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20091) Tez: Add security credentials for FileSinkOperator output
[ https://issues.apache.org/jira/browse/HIVE-20091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20091: Attachment: HIVE-20091.08.patch > Tez: Add security credentials for FileSinkOperator output > - > > Key: HIVE-20091 > URL: https://issues.apache.org/jira/browse/HIVE-20091 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20091.01.patch, HIVE-20091.02.patch, > HIVE-20091.03.patch, HIVE-20091.04.patch, HIVE-20091.05.patch, > HIVE-20091.06.patch, HIVE-20091.07.patch, HIVE-20091.08.patch > > > DagUtils needs to add security credentials for the output for the > FileSinkOperator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20091) Tez: Add security credentials for FileSinkOperator output
[ https://issues.apache.org/jira/browse/HIVE-20091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20091: Status: Patch Available (was: In Progress) > Tez: Add security credentials for FileSinkOperator output > - > > Key: HIVE-20091 > URL: https://issues.apache.org/jira/browse/HIVE-20091 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20091.01.patch, HIVE-20091.02.patch, > HIVE-20091.03.patch, HIVE-20091.04.patch, HIVE-20091.05.patch, > HIVE-20091.06.patch, HIVE-20091.07.patch, HIVE-20091.08.patch > > > DagUtils needs to add security credentials for the output for the > FileSinkOperator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20091) Tez: Add security credentials for FileSinkOperator output
[ https://issues.apache.org/jira/browse/HIVE-20091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20091: Status: In Progress (was: Patch Available) > Tez: Add security credentials for FileSinkOperator output > - > > Key: HIVE-20091 > URL: https://issues.apache.org/jira/browse/HIVE-20091 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20091.01.patch, HIVE-20091.02.patch, > HIVE-20091.03.patch, HIVE-20091.04.patch, HIVE-20091.05.patch, > HIVE-20091.06.patch, HIVE-20091.07.patch > > > DagUtils needs to add security credentials for the output for the > FileSinkOperator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19532) fix tests for master-txnstats branch
[
https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540894#comment-16540894
]
Hive QA commented on HIVE-19532:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
4s{color} | {color:blue} standalone-metastore in master has 217 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
58s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} storage-api: The patch generated 1 new + 3 unchanged -
0 fixed = 4 total (was 3) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} standalone-metastore: The patch generated 79 new +
2330 unchanged - 17 fixed = 2409 total (was 2347) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
50s{color} | {color:red} ql: The patch generated 17 new + 1107 unchanged - 8
fixed = 1124 total (was 1115) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
42s{color} | {color:red} root: The patch generated 102 new + 3990 unchanged -
25 fixed = 4092 total (was 4015) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} itests/hcatalog-unit: The patch generated 5 new + 28
unchanged - 0 fixed = 33 total (was 28) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 85 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
30s{color} | {color:red} patch/storage-api cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 5m
8s{color} | {color:red} patch/standalone-metastore cannot run
setBugDatabaseInfo from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 6m
30s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
0s{color} | {color:red} patch/itests/hive-unit cannot run setBugDatabaseInfo
from findbugs {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
24s{color} | {color:red} standalone-metastore generated 5 new + 54 unchanged -
0 fixed = 59 total (was 54) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 8m
36s{color} | {color:red} root generated 5 new + 371 unchanged - 0 fixed = 376
total (was 371) {color} |
|| || || || {color:brown} Other Tests {color} ||
|
[jira] [Updated] (HIVE-20123) Fix masking tests after HIVE-19617
[ https://issues.apache.org/jira/browse/HIVE-20123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20123: --- Resolution: Fixed Fix Version/s: 4.0.0 3.1.0 Status: Resolved (was: Patch Available) Pushed to master, branch-3. Thanks [~vgarg] > Fix masking tests after HIVE-19617 > -- > > Key: HIVE-20123 > URL: https://issues.apache.org/jira/browse/HIVE-20123 > Project: Hive > Issue Type: Test >Affects Versions: 3.0.0, 3.1.0, 4.0.0, 3.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-20123.01.patch, HIVE-20123.patch > > > Masking tests results were changed inadvertently when HIVE-19617 went in, > since table names were changed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19532) fix tests for master-txnstats branch
[
https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540882#comment-16540882
]
Sergey Shelukhin edited comment on HIVE-19532 at 7/11/18 11:29 PM:
---
acid_no_buckets is a known bad one.
A concerning one (might be dup of acid_no_bucket; or might be caused by
HIVE-20076 somehow being merged wrong):
{noformat}
java.lang.AssertionError:
Client Execution succeeded but contained differences (error code = 1) after
executing vector_delete_orig_table.q
151c151
< 12280
---
> 12192
{noformat}
The error from TestCompactionTxnHandler looks weird, DBS table doesn't exist,
not sure why. Might require a local repro.
The rest look trivial.
was (Author: sershe):
acid_no_buckets is a known bad one.
A concerning one (might be dup of acid_no_bucket; or might be caused by
HIVE-20076):
{noformat}
java.lang.AssertionError:
Client Execution succeeded but contained differences (error code = 1) after
executing vector_delete_orig_table.q
151c151
< 12280
---
> 12192
{noformat}
The error from TestCompactionTxnHandler looks weird, DBS table doesn't exist,
not sure why. Might require a local repro.
The rest look trivial.
> fix tests for master-txnstats branch
>
>
> Key: HIVE-19532
> URL: https://issues.apache.org/jira/browse/HIVE-19532
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch,
> HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch,
> HIVE-19532.04.patch, HIVE-19532.05.patch, HIVE-19532.07.patch,
> HIVE-19532.08.patch, HIVE-19532.09.patch, HIVE-19532.10.patch,
> HIVE-19532.11.patch, HIVE-19532.12.patch, HIVE-19532.13.patch,
> HIVE-19532.14.patch, HIVE-19532.15.patch, HIVE-19532.16.patch,
> HIVE-19532.17.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19532) fix tests for master-txnstats branch
[
https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540882#comment-16540882
]
Sergey Shelukhin edited comment on HIVE-19532 at 7/11/18 11:28 PM:
---
acid_no_buckets is a known bad one.
A concerning one (might be dup of acid_no_bucket; or might be caused by
HIVE-20076):
{noformat}
java.lang.AssertionError:
Client Execution succeeded but contained differences (error code = 1) after
executing vector_delete_orig_table.q
151c151
< 12280
---
> 12192
{noformat}
The error from TestCompactionTxnHandler looks weird, DBS table doesn't exist,
not sure why. Might require a local repro.
The rest look trivial.
was (Author: sershe):
acid_no_buckets is a known bad one.
A concerning one (might be dup of acid_no_bucket; or might be caused by master
change for row IDs that somehow got merged wrong):
{noformat}
java.lang.AssertionError:
Client Execution succeeded but contained differences (error code = 1) after
executing vector_delete_orig_table.q
151c151
< 12280
---
> 12192
{noformat}
The error from TestCompactionTxnHandler looks weird, DBS table doesn't exist,
not sure why. Might require a local repro.
The rest look trivial.
> fix tests for master-txnstats branch
>
>
> Key: HIVE-19532
> URL: https://issues.apache.org/jira/browse/HIVE-19532
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch,
> HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch,
> HIVE-19532.04.patch, HIVE-19532.05.patch, HIVE-19532.07.patch,
> HIVE-19532.08.patch, HIVE-19532.09.patch, HIVE-19532.10.patch,
> HIVE-19532.11.patch, HIVE-19532.12.patch, HIVE-19532.13.patch,
> HIVE-19532.14.patch, HIVE-19532.15.patch, HIVE-19532.16.patch,
> HIVE-19532.17.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20147) Hive streaming ingest is contented on synchronized logging
[ https://issues.apache.org/jira/browse/HIVE-20147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20147: - Attachment: sync-logger-contention.svg > Hive streaming ingest is contented on synchronized logging > -- > > Key: HIVE-20147 > URL: https://issues.apache.org/jira/browse/HIVE-20147 > Project: Hive > Issue Type: Bug > Components: Streaming >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: Screen Shot 2018-07-11 at 4.17.27 PM.png, > sync-logger-contention.svg > > > In one of the observed profile, >30% time spent on synchronized logging. See > attachment. > We should use async logging for hive streaming ingest by default. !Screen > Shot 2018-07-11 at 4.17.27 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19532) fix tests for master-txnstats branch
[
https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540882#comment-16540882
]
Sergey Shelukhin commented on HIVE-19532:
-
A concerning one (might be dup of acid_no_bucket; or might be caused by master
change for row IDs that somehow got merged wrong):
{noformat}
java.lang.AssertionError:
Client Execution succeeded but contained differences (error code = 1) after
executing vector_delete_orig_table.q
151c151
< 12280
---
> 12192
{noformat}
> fix tests for master-txnstats branch
>
>
> Key: HIVE-19532
> URL: https://issues.apache.org/jira/browse/HIVE-19532
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch,
> HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch,
> HIVE-19532.04.patch, HIVE-19532.05.patch, HIVE-19532.07.patch,
> HIVE-19532.08.patch, HIVE-19532.09.patch, HIVE-19532.10.patch,
> HIVE-19532.11.patch, HIVE-19532.12.patch, HIVE-19532.13.patch,
> HIVE-19532.14.patch, HIVE-19532.15.patch, HIVE-19532.16.patch,
> HIVE-19532.17.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19532) fix tests for master-txnstats branch
[
https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540882#comment-16540882
]
Sergey Shelukhin edited comment on HIVE-19532 at 7/11/18 11:25 PM:
---
acid_no_buckets is a known bad one.
A concerning one (might be dup of acid_no_bucket; or might be caused by master
change for row IDs that somehow got merged wrong):
{noformat}
java.lang.AssertionError:
Client Execution succeeded but contained differences (error code = 1) after
executing vector_delete_orig_table.q
151c151
< 12280
---
> 12192
{noformat}
The error from TestCompactionTxnHandler looks weird, DBS table doesn't exist,
not sure why. Might require a local repro.
The rest look trivial.
was (Author: sershe):
A concerning one (might be dup of acid_no_bucket; or might be caused by master
change for row IDs that somehow got merged wrong):
{noformat}
java.lang.AssertionError:
Client Execution succeeded but contained differences (error code = 1) after
executing vector_delete_orig_table.q
151c151
< 12280
---
> 12192
{noformat}
> fix tests for master-txnstats branch
>
>
> Key: HIVE-19532
> URL: https://issues.apache.org/jira/browse/HIVE-19532
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch,
> HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch,
> HIVE-19532.04.patch, HIVE-19532.05.patch, HIVE-19532.07.patch,
> HIVE-19532.08.patch, HIVE-19532.09.patch, HIVE-19532.10.patch,
> HIVE-19532.11.patch, HIVE-19532.12.patch, HIVE-19532.13.patch,
> HIVE-19532.14.patch, HIVE-19532.15.patch, HIVE-19532.16.patch,
> HIVE-19532.17.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19940) Push predicates with deterministic UDFs with RBO
[
https://issues.apache.org/jira/browse/HIVE-19940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janaki Lahorani updated HIVE-19940:
---
Attachment: HIVE-19940.3.patch
> Push predicates with deterministic UDFs with RBO
>
>
> Key: HIVE-19940
> URL: https://issues.apache.org/jira/browse/HIVE-19940
> Project: Hive
> Issue Type: Improvement
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-19940.1.patch, HIVE-19940.2.patch,
> HIVE-19940.3.patch
>
>
> With RBO, predicates with any UDF doesn't get pushed down. It makes sense to
> not pushdown the predicates with non-deterministic function as the meaning of
> the query changes after the predicate is resolved to use the function. But
> pushing a deterministic function is beneficial.
> Test Case:
> {code}
> set hive.cbo.enable=false;
> CREATE TABLE `testb`(
>`cola` string COMMENT '',
>`colb` string COMMENT '',
>`colc` string COMMENT '')
> PARTITIONED BY (
>`part1` string,
>`part2` string,
>`part3` string)
> STORED AS AVRO;
> CREATE TABLE `testa`(
>`col1` string COMMENT '',
>`col2` string COMMENT '',
>`col3` string COMMENT '',
>`col4` string COMMENT '',
>`col5` string COMMENT '')
> PARTITIONED BY (
>`part1` string,
>`part2` string,
>`part3` string)
> STORED AS AVRO;
> insert into testA partition (part1='US', part2='ABC', part3='123')
> values ('12.34', '100', '200', '300', 'abc'),
> ('12.341', '1001', '2001', '3001', 'abcd');
> insert into testA partition (part1='UK', part2='DEF', part3='123')
> values ('12.34', '100', '200', '300', 'abc'),
> ('12.341', '1001', '2001', '3001', 'abcd');
> insert into testA partition (part1='US', part2='DEF', part3='200')
> values ('12.34', '100', '200', '300', 'abc'),
> ('12.341', '1001', '2001', '3001', 'abcd');
> insert into testA partition (part1='CA', part2='ABC', part3='300')
> values ('12.34', '100', '200', '300', 'abc'),
> ('12.341', '1001', '2001', '3001', 'abcd');
> insert into testB partition (part1='CA', part2='ABC', part3='300')
> values ('600', '700', 'abc'), ('601', '701', 'abcd');
> insert into testB partition (part1='CA', part2='ABC', part3='400')
> values ( '600', '700', 'abc'), ( '601', '701', 'abcd');
> insert into testB partition (part1='UK', part2='PQR', part3='500')
> values ('600', '700', 'abc'), ('601', '701', 'abcd');
> insert into testB partition (part1='US', part2='DEF', part3='200')
> values ( '600', '700', 'abc'), ('601', '701', 'abcd');
> insert into testB partition (part1='US', part2='PQR', part3='123')
> values ( '600', '700', 'abc'), ('601', '701', 'abcd');
> -- views with deterministic functions
> create view viewDeterministicUDFA partitioned on (vpart1, vpart2, vpart3) as
> select
> cast(col1 as decimal(38,18)) as vcol1,
> cast(col2 as decimal(38,18)) as vcol2,
> cast(col3 as decimal(38,18)) as vcol3,
> cast(col4 as decimal(38,18)) as vcol4,
> cast(col5 as char(10)) as vcol5,
> cast(part1 as char(2)) as vpart1,
> cast(part2 as char(3)) as vpart2,
> cast(part3 as char(3)) as vpart3
> from testa
> where part1 in ('US', 'CA');
> create view viewDeterministicUDFB partitioned on (vpart1, vpart2, vpart3) as
> select
> cast(cola as decimal(38,18)) as vcolA,
> cast(colb as decimal(38,18)) as vcolB,
> cast(colc as char(10)) as vcolC,
> cast(part1 as char(2)) as vpart1,
> cast(part2 as char(3)) as vpart2,
> cast(part3 as char(3)) as vpart3
> from testb
> where part1 in ('US', 'CA');
> explain
> select vcol1, vcol2, vcol3, vcola, vcolb
> from viewDeterministicUDFA a inner join viewDeterministicUDFB b
> on a.vpart1 = b.vpart1
> and a.vpart2 = b.vpart2
> and a.vpart3 = b.vpart3
> and a.vpart1 = 'US'
> and a.vpart2 = 'DEF'
> and a.vpart3 = '200';
> {code}
> Plan where the CAST is not pushed down.
> {code}
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: testa
> filterExpr: (part1) IN ('US', 'CA') (type: boolean)
> Statistics: Num rows: 6 Data size: 13740 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: CAST( col1 AS decimal(38,18)) (type:
> decimal(38,18)), CAST( col2 AS decimal(38,18)) (type: decimal(38,18)), CAST(
> col3 AS decimal(38,18)) (type: decimal(38,18)), CAST( part1 AS CHAR(2))
> (type: char(2)), CAST( part2 AS CHAR(3)) (type: char(3)), CAST( part3 AS
> CHAR(3)) (type: char(3))
> outputColumnNames: _col0, _col1, _col2, _col5, _col6, _col7
> Statistics: Num rows: 6 Data size: 13740 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: ((_col5 = 'US') and (_col6 = 'DEF') and (_col7 =
> '200')) (type: boolean)
> Statistics: Num
[jira] [Updated] (HIVE-20145) Fix wrong vector_delete_orig_table.q.out
[ https://issues.apache.org/jira/browse/HIVE-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20145: -- Resolution: Duplicate Status: Resolved (was: Patch Available) > Fix wrong vector_delete_orig_table.q.out > > > Key: HIVE-20145 > URL: https://issues.apache.org/jira/browse/HIVE-20145 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Minor > Attachments: HIVE-20145.2.patch, HIVE-20145.patch > > > HIVE-20076 changed the output of vector_delete_orig_table.q, but the patch > doesn't contains it. This patch will include a fixed output file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20076) ACID: Fix Synthetic ROW__ID generation for vectorized orc readers
[ https://issues.apache.org/jira/browse/HIVE-20076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20076: -- Resolution: Fixed Status: Resolved (was: Patch Available) > ACID: Fix Synthetic ROW__ID generation for vectorized orc readers > - > > Key: HIVE-20076 > URL: https://issues.apache.org/jira/browse/HIVE-20076 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-20076.2.patch, HIVE-20076.3.patch, > HIVE-20076.4.patch, HIVE-20076.patch > > > Delete on a partitioned table removes more rows than expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20147) Hive streaming ingest is contented on synchronized logging
[ https://issues.apache.org/jira/browse/HIVE-20147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-20147: > Hive streaming ingest is contented on synchronized logging > -- > > Key: HIVE-20147 > URL: https://issues.apache.org/jira/browse/HIVE-20147 > Project: Hive > Issue Type: Bug > Components: Streaming >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: Screen Shot 2018-07-11 at 4.17.27 PM.png > > > In one of the observed profile, >30% time spent on synchronized logging. See > attachment. > We should use async logging for hive streaming ingest by default. !Screen > Shot 2018-07-11 at 4.17.27 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20076) ACID: Fix Synthetic ROW__ID generation for vectorized orc readers
[ https://issues.apache.org/jira/browse/HIVE-20076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540870#comment-16540870 ] Teddy Choi commented on HIVE-20076: --- The third patch made a change in vector_delete_orig_table.q.out. So I made an addendum. The fourth patch includes the third patch and the addendum. It's pushed to branch-3.1, branch-3, and master branch. > ACID: Fix Synthetic ROW__ID generation for vectorized orc readers > - > > Key: HIVE-20076 > URL: https://issues.apache.org/jira/browse/HIVE-20076 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-20076.2.patch, HIVE-20076.3.patch, > HIVE-20076.4.patch, HIVE-20076.patch > > > Delete on a partitioned table removes more rows than expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20076) ACID: Fix Synthetic ROW__ID generation for vectorized orc readers
[ https://issues.apache.org/jira/browse/HIVE-20076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20076: -- Attachment: HIVE-20076.4.patch > ACID: Fix Synthetic ROW__ID generation for vectorized orc readers > - > > Key: HIVE-20076 > URL: https://issues.apache.org/jira/browse/HIVE-20076 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-20076.2.patch, HIVE-20076.3.patch, > HIVE-20076.4.patch, HIVE-20076.patch > > > Delete on a partitioned table removes more rows than expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-20064) Data cannot be inserted into a dynamic partition
[
https://issues.apache.org/jira/browse/HIVE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
gavin_shi resolved HIVE-20064.
--
Resolution: Fixed
> Data cannot be inserted into a dynamic partition
>
>
> Key: HIVE-20064
> URL: https://issues.apache.org/jira/browse/HIVE-20064
> Project: Hive
> Issue Type: Bug
> Components: SQL
>Reporter: gavin_shi
>Priority: Blocker
>
> Data cannot be inserted into a dynamic partition,The relevant table
> information is as follows：
> The source table:
> {code:java}
> //代码占位符
> create table customer(id string,name string,gender string,state string)
> partitioned by(country string);
> {code}
> The source table data：
>
> {code:java}
> //代码占位符
> +--++--+-+---+--+
> | customer.id | customer.name | customer.gender | customer.state |
> customer.country |
> +--++--+-+---+--+
> | 1 | gavin | 1 | shanghai | china |
> +--++--+-+---+--+
> {code}
>
> The target table:
>
> {code:java}
> //代码占位符
> create table customer_bak(id string,name string,gender string,state string)
> partitioned by(country string);
> {code}
>
> Now I use dynamic techniques to insert data into the customer_bak table：
> {code:java}
> //代码占位符
> set hive.exec.dynamic.partition=true;
> set hive.exec.dynamic.partition.mode=nonstrict;
> insert into table customer_bak partition(country) select
> id,name,gender,state,country from customer;
> {code}
> The program is always executing and neither success nor failure is shown. The
> information in the hive log is as follows:
> {code:java}
> //代码占位符
> 2018-06-28T16:01:22,900 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Executing
> command(queryId=hadoop_20180628160122_f80c5f8e-98a0-4286-bc55-c25efd7a60bc):
> insert into table customer_bak partition(country) select
> id,name,gender,state,country from customer
> 2018-06-28T16:01:22,901 WARN [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. tez,
> spark) or using Hive 1.X releases.
> 2018-06-28T16:01:22,902 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: WARNING: Hive-on-MR is deprecated in Hive 2 and may not be
> available in the future versions. Consider using a different execution engine
> (i.e. tez, spark) or using Hive 1.X releases.
> 2018-06-28T16:01:22,902 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Query ID =
> hadoop_20180628160122_f80c5f8e-98a0-4286-bc55-c25efd7a60bc
> 2018-06-28T16:01:22,903 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Total jobs = 3
> 2018-06-28T16:01:22,904 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Launching Job 1 out of 3
> 2018-06-28T16:01:22,907 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Starting task [Stage-1:MAPRED] in serial mode
> 2018-06-28T16:01:22,908 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: Number of reduce tasks is set to 0 since there's no reduce operator
> 2018-06-28T16:01:22,929 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Context: New scratch dir is
> hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3
> 2018-06-28T16:01:22,942 INFO [HiveServer2-Background-Pool: Thread-97]
> mr.ExecDriver: Using org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
> 2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Processing alias customer
> 2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Adding 2 inputs; the first input is
> hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
> 2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Content Summary not cached for
> hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
> 2018-06-28T16:01:22,945 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Content Summary not cached for
> hdfs://hadoop-cluster/user/hive/warehouse/customer/country=usa
> 2018-06-28T16:01:22,978 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Changed input file
> hdfs://hadoop-cluster/user/hive/warehouse/customer/country=usa to empty file
> hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036
> (false)
> 2018-06-28T16:01:22,979 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Context: New scratch dir is
>
[jira] [Resolved] (HIVE-18132) Acid Table: Exchange Partition
[ https://issues.apache.org/jira/browse/HIVE-18132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman resolved HIVE-18132. --- Resolution: Won't Fix > Acid Table: Exchange Partition > -- > > Key: HIVE-18132 > URL: https://issues.apache.org/jira/browse/HIVE-18132 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > This command currently renames a directory under 1 table to be under another > table's name space. This can't work for Acid since data itself has embedded > transaction info. > If src is not full acid, it could be added to target side like Load Data - > into delta/base, but if source side is also Acid the IDs won't make sense in > target table. It could match if using global txn ids but may not match with > per table write ids if some ID from src is committed but the same ID in > target is aborted. > Does this command currently work with bucketed tables? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18132) Acid Table: Exchange Partition
[ https://issues.apache.org/jira/browse/HIVE-18132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18132: - Assignee: Eugene Koifman > Acid Table: Exchange Partition > -- > > Key: HIVE-18132 > URL: https://issues.apache.org/jira/browse/HIVE-18132 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > This command currently renames a directory under 1 table to be under another > table's name space. This can't work for Acid since data itself has embedded > transaction info. > If src is not full acid, it could be added to target side like Load Data - > into delta/base, but if source side is also Acid the IDs won't make sense in > target table. It could match if using global txn ids but may not match with > per table write ids if some ID from src is committed but the same ID in > target is aborted. > Does this command currently work with bucketed tables? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20123) Fix masking tests after HIVE-19617
[ https://issues.apache.org/jira/browse/HIVE-20123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540842#comment-16540842 ] Vineet Garg commented on HIVE-20123: +1 > Fix masking tests after HIVE-19617 > -- > > Key: HIVE-20123 > URL: https://issues.apache.org/jira/browse/HIVE-20123 > Project: Hive > Issue Type: Test >Affects Versions: 3.0.0, 3.1.0, 4.0.0, 3.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20123.01.patch, HIVE-20123.patch > > > Masking tests results were changed inadvertently when HIVE-19617 went in, > since table names were changed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19532) fix tests for master-txnstats branch
[
https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540841#comment-16540841
]
Hive QA commented on HIVE-19532:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931101/HIVE-19532.17.patch
{color:green}SUCCESS:{color} +1 due to 13 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 14649 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_delete_orig_table]
(batchId=2)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[change_allowincompatible_vectorization_false_date]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_truncate]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sqlmerge_stats]
(batchId=176)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleanedCleansTxnsAndTxnComponents
(batchId=273)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12538/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12538/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12538/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931101 - PreCommit-HIVE-Build
> fix tests for master-txnstats branch
>
>
> Key: HIVE-19532
> URL: https://issues.apache.org/jira/browse/HIVE-19532
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch,
> HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch,
> HIVE-19532.04.patch, HIVE-19532.05.patch, HIVE-19532.07.patch,
> HIVE-19532.08.patch, HIVE-19532.09.patch, HIVE-19532.10.patch,
> HIVE-19532.11.patch, HIVE-19532.12.patch, HIVE-19532.13.patch,
> HIVE-19532.14.patch, HIVE-19532.15.patch, HIVE-19532.16.patch,
> HIVE-19532.17.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20145) Fix wrong vector_delete_orig_table.q.out
[ https://issues.apache.org/jira/browse/HIVE-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540823#comment-16540823 ] Teddy Choi commented on HIVE-20145: --- [~gopalv], thanks! > Fix wrong vector_delete_orig_table.q.out > > > Key: HIVE-20145 > URL: https://issues.apache.org/jira/browse/HIVE-20145 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Minor > Attachments: HIVE-20145.2.patch, HIVE-20145.patch > > > HIVE-20076 changed the output of vector_delete_orig_table.q, but the patch > doesn't contains it. This patch will include a fixed output file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20145) Fix wrong vector_delete_orig_table.q.out
[ https://issues.apache.org/jira/browse/HIVE-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20145: -- Attachment: HIVE-20145.2.patch > Fix wrong vector_delete_orig_table.q.out > > > Key: HIVE-20145 > URL: https://issues.apache.org/jira/browse/HIVE-20145 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Minor > Attachments: HIVE-20145.2.patch, HIVE-20145.patch > > > HIVE-20076 changed the output of vector_delete_orig_table.q, but the patch > doesn't contains it. This patch will include a fixed output file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20145) Fix wrong vector_delete_orig_table.q.out
[ https://issues.apache.org/jira/browse/HIVE-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540779#comment-16540779 ] Gopal V commented on HIVE-20145: LGTM - +1 [~teddy.choi]: you could commit this as an addendum to the HIVE-20076 instead of leaving the test broken right now. > Fix wrong vector_delete_orig_table.q.out > > > Key: HIVE-20145 > URL: https://issues.apache.org/jira/browse/HIVE-20145 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Minor > Attachments: HIVE-20145.patch > > > HIVE-20076 changed the output of vector_delete_orig_table.q, but the patch > doesn't contains it. This patch will include a fixed output file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19820) add ACID stats support to background stats updater and fix bunch of edge cases found in SU tests
[ https://issues.apache.org/jira/browse/HIVE-19820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540763#comment-16540763 ] Sergey Shelukhin commented on HIVE-19820: - Truncate support and test that combines truncate for table and partition, analyze, drop partition, etc. I think if HiveQA keeps being slow I will just commit this and fix tests later. I will try adding support for more exotic cases like rename partition and rename column for now. > add ACID stats support to background stats updater and fix bunch of edge > cases found in SU tests > > > Key: HIVE-19820 > URL: https://issues.apache.org/jira/browse/HIVE-19820 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19820.01-master-txnstats.patch, > HIVE-19820.01.patch, HIVE-19820.02-master-txnstats.patch, > HIVE-19820.03-master-txnstats.patch, HIVE-19820.04-master-txnstats.patch, > HIVE-19820.patch, branch-19820.nogen.patch, branch-19820.nogen.patch > > > Follow-up from HIVE-19418. > Right now it checks whether stats are valid in an old-fashioned way... and > also gets ACID state, and discards it without using. > When ACID stats are implemented, ACID state needs to be used to do > version-aware valid stats checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19820) add ACID stats support to background stats updater and fix bunch of edge cases found in SU tests
[ https://issues.apache.org/jira/browse/HIVE-19820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19820: Attachment: branch-19820.nogen.patch > add ACID stats support to background stats updater and fix bunch of edge > cases found in SU tests > > > Key: HIVE-19820 > URL: https://issues.apache.org/jira/browse/HIVE-19820 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19820.01-master-txnstats.patch, > HIVE-19820.01.patch, HIVE-19820.02-master-txnstats.patch, > HIVE-19820.03-master-txnstats.patch, HIVE-19820.04-master-txnstats.patch, > HIVE-19820.patch, branch-19820.nogen.patch, branch-19820.nogen.patch > > > Follow-up from HIVE-19418. > Right now it checks whether stats are valid in an old-fashioned way... and > also gets ACID state, and discards it without using. > When ACID stats are implemented, ACID state needs to be used to do > version-aware valid stats checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19820) add ACID stats support to background stats updater and fix bunch of edge cases found in SU tests
[ https://issues.apache.org/jira/browse/HIVE-19820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19820: Attachment: HIVE-19820.01.patch > add ACID stats support to background stats updater and fix bunch of edge > cases found in SU tests > > > Key: HIVE-19820 > URL: https://issues.apache.org/jira/browse/HIVE-19820 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19820.01-master-txnstats.patch, > HIVE-19820.01.patch, HIVE-19820.02-master-txnstats.patch, > HIVE-19820.03-master-txnstats.patch, HIVE-19820.04-master-txnstats.patch, > HIVE-19820.patch, branch-19820.nogen.patch, branch-19820.nogen.patch > > > Follow-up from HIVE-19418. > Right now it checks whether stats are valid in an old-fashioned way... and > also gets ACID state, and discards it without using. > When ACID stats are implemented, ACID state needs to be used to do > version-aware valid stats checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20145) Fix wrong vector_delete_orig_table.q.out
[ https://issues.apache.org/jira/browse/HIVE-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20145: -- Status: Patch Available (was: Open) > Fix wrong vector_delete_orig_table.q.out > > > Key: HIVE-20145 > URL: https://issues.apache.org/jira/browse/HIVE-20145 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Minor > Attachments: HIVE-20145.patch > > > HIVE-20076 changed the output of vector_delete_orig_table.q, but the patch > doesn't contains it. This patch will include a fixed output file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20145) Fix wrong vector_delete_orig_table.q.out
[ https://issues.apache.org/jira/browse/HIVE-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20145: -- Attachment: HIVE-20145.patch > Fix wrong vector_delete_orig_table.q.out > > > Key: HIVE-20145 > URL: https://issues.apache.org/jira/browse/HIVE-20145 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Minor > Attachments: HIVE-20145.patch > > > HIVE-20076 changed the output of vector_delete_orig_table.q, but the patch > doesn't contains it. This patch will include a fixed output file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20145) Fix wrong vector_delete_orig_table.q.out
[ https://issues.apache.org/jira/browse/HIVE-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20145: -- Summary: Fix wrong vector_delete_orig_table.q.out (was: Fix vector_delete_orig_table.q.out) > Fix wrong vector_delete_orig_table.q.out > > > Key: HIVE-20145 > URL: https://issues.apache.org/jira/browse/HIVE-20145 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Minor > Attachments: HIVE-20145.patch > > > HIVE-20076 changed the output of vector_delete_orig_table.q, but the patch > doesn't contains it. This patch will include a fixed output file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20145) Fix vector_delete_orig_table.q.out
[ https://issues.apache.org/jira/browse/HIVE-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi reassigned HIVE-20145: - > Fix vector_delete_orig_table.q.out > -- > > Key: HIVE-20145 > URL: https://issues.apache.org/jira/browse/HIVE-20145 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Minor > > HIVE-20076 changed the output of vector_delete_orig_table.q, but the patch > doesn't contains it. This patch will include a fixed output file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-18615) Prevent deserializing transaction list in the materializations invalidation cache multiple times
[ https://issues.apache.org/jira/browse/HIVE-18615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-18615. Resolution: Invalid > Prevent deserializing transaction list in the materializations invalidation > cache multiple times > > > Key: HIVE-18615 > URL: https://issues.apache.org/jira/browse/HIVE-18615 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19375) "'transactional'='false' is no longer a valid property and will be ignored:
[
https://issues.apache.org/jira/browse/HIVE-19375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19375:
--
Status: Patch Available (was: Open)
[~jdere] could you review please
> "'transactional'='false' is no longer a valid property and will be ignored:
>
>
> Key: HIVE-19375
> URL: https://issues.apache.org/jira/browse/HIVE-19375
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Minor
> Attachments: HIVE-19375.01.patch
>
>
> from {{TransactionalValidationListener.handleCreateTableTransactionalProp()}}
> {noformat}
> if ("false".equalsIgnoreCase(transactional)) {
> // just drop transactional=false. For backward compatibility in case
> someone has scripts
> // with transactional=false
> LOG.info("'transactional'='false' is no longer a valid property and
> will be ignored: " +
> Warehouse.getQualifiedName(newTable));
> return;
> }
> {noformat}
> this msg is misleading since with metastore.create.as.acid=true, setting
> transactional=false is valid to make a flat table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19375) "'transactional'='false' is no longer a valid property and will be ignored:
[
https://issues.apache.org/jira/browse/HIVE-19375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19375:
--
Attachment: HIVE-19375.01.patch
> "'transactional'='false' is no longer a valid property and will be ignored:
>
>
> Key: HIVE-19375
> URL: https://issues.apache.org/jira/browse/HIVE-19375
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19375.01.patch
>
>
> from {{TransactionalValidationListener.handleCreateTableTransactionalProp()}}
> {noformat}
> if ("false".equalsIgnoreCase(transactional)) {
> // just drop transactional=false. For backward compatibility in case
> someone has scripts
> // with transactional=false
> LOG.info("'transactional'='false' is no longer a valid property and
> will be ignored: " +
> Warehouse.getQualifiedName(newTable));
> return;
> }
> {noformat}
> this msg is misleading since with metastore.create.as.acid=true, setting
> transactional=false is valid to make a flat table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19375) "'transactional'='false' is no longer a valid property and will be ignored:
[
https://issues.apache.org/jira/browse/HIVE-19375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19375:
--
Priority: Minor (was: Major)
> "'transactional'='false' is no longer a valid property and will be ignored:
>
>
> Key: HIVE-19375
> URL: https://issues.apache.org/jira/browse/HIVE-19375
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Minor
> Attachments: HIVE-19375.01.patch
>
>
> from {{TransactionalValidationListener.handleCreateTableTransactionalProp()}}
> {noformat}
> if ("false".equalsIgnoreCase(transactional)) {
> // just drop transactional=false. For backward compatibility in case
> someone has scripts
> // with transactional=false
> LOG.info("'transactional'='false' is no longer a valid property and
> will be ignored: " +
> Warehouse.getQualifiedName(newTable));
> return;
> }
> {noformat}
> this msg is misleading since with metastore.create.as.acid=true, setting
> transactional=false is valid to make a flat table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20143) analyze doesn't mark partition column stats as accurate after truncate
[
https://issues.apache.org/jira/browse/HIVE-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-20143:
Description:
Discovered while looking at txn stats. This applies for non-txn tables; works
fine for truncate+analyze for non-partitioned tables, but not for partitions:
{noformat}
set hive.stats.dbclass=fs;
set hive.stats.fetch.column.stats=true;
set hive.stats.autogather=true;
set hive.stats.column.autogather=true;
set hive.compute.query.using.stats=true;
set hive.mapred.mode=nonstrict;
set hive.explain.user=false;
set hive.fetch.task.conversion=none;
set hive.query.results.cache.enabled=false;
create table stats_part1(key int,value string) partitioned by (p int);
insert into table stats_part1 partition(p=101) values (1, "foo");
insert into table stats_part1 partition(p=102) values (2, "bar");
explain select count(key) from stats_part1; -- from stats
truncate table stats_part1 partition(p=101);
explain select count(key) from stats_part1; -- not from stats, ok
analyze table stats_part1 partition(p) compute statistics for columns;
explain select count(key) from stats_part1; -- not from stats still
{noformat}
was:
Discovered while looking at txn stats. This works fine for truncate+analyze for
non-partitioned tables, but not for partitions:
{noformat}
set hive.stats.dbclass=fs;
set hive.stats.fetch.column.stats=true;
set hive.stats.autogather=true;
set hive.stats.column.autogather=true;
set hive.compute.query.using.stats=true;
set hive.mapred.mode=nonstrict;
set hive.explain.user=false;
set hive.fetch.task.conversion=none;
set hive.query.results.cache.enabled=false;
create table stats_part1(key int,value string) partitioned by (p int);
insert into table stats_part1 partition(p=101) values (1, "foo");
insert into table stats_part1 partition(p=102) values (2, "bar");
explain select count(key) from stats_part1; -- from stats
truncate table stats_part1 partition(p=101);
explain select count(key) from stats_part1; -- not from stats, ok
analyze table stats_part1 partition(p) compute statistics for columns;
explain select count(key) from stats_part1; -- not from stats still
{noformat}
> analyze doesn't mark partition column stats as accurate after truncate
> --
>
> Key: HIVE-20143
> URL: https://issues.apache.org/jira/browse/HIVE-20143
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Priority: Major
>
> Discovered while looking at txn stats. This applies for non-txn tables;
> works fine for truncate+analyze for non-partitioned tables, but not for
> partitions:
> {noformat}
> set hive.stats.dbclass=fs;
> set hive.stats.fetch.column.stats=true;
> set hive.stats.autogather=true;
> set hive.stats.column.autogather=true;
> set hive.compute.query.using.stats=true;
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.fetch.task.conversion=none;
> set hive.query.results.cache.enabled=false;
> create table stats_part1(key int,value string) partitioned by (p int);
> insert into table stats_part1 partition(p=101) values (1, "foo");
> insert into table stats_part1 partition(p=102) values (2, "bar");
> explain select count(key) from stats_part1; -- from stats
> truncate table stats_part1 partition(p=101);
> explain select count(key) from stats_part1; -- not from stats, ok
> analyze table stats_part1 partition(p) compute statistics for columns;
> explain select count(key) from stats_part1; -- not from stats still
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter
[
https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540730#comment-16540730
]
Hive QA commented on HIVE-16295:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931082/HIVE-16295.8.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14642 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12537/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12537/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12537/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931082 - PreCommit-HIVE-Build
> Add support for using Hadoop's S3A OutputCommitter
> --
>
> Key: HIVE-16295
> URL: https://issues.apache.org/jira/browse/HIVE-16295
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-16295.1.WIP.patch, HIVE-16295.2.WIP.patch,
> HIVE-16295.3.WIP.patch, HIVE-16295.4.patch, HIVE-16295.5.patch,
> HIVE-16295.6.patch, HIVE-16295.7.patch, HIVE-16295.8.patch
>
>
> Hive doesn't have integration with Hadoop's {{OutputCommitter}}, it uses a
> {{NullOutputCommitter}} and uses its own commit logic spread across
> {{FileSinkOperator}}, {{MoveTask}}, and {{Hive}}.
> The Hadoop community is building an {{OutputCommitter}} that integrates with
> S3Guard and does a safe, coordinate commit of data on S3 inside individual
> tasks (HADOOP-13786). If Hive can integrate with this new {{OutputCommitter}}
> there would be a lot of benefits to Hive-on-S3:
> * Data is only written once; directly committing data at a task level means
> no renames are necessary
> * The commit is done safely, in a coordinated manner; duplicate tasks (from
> task retries or speculative execution) should not step on each other
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20143) analyze doesn't mark partition column stats as accurate after truncate
[
https://issues.apache.org/jira/browse/HIVE-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-20143:
--
Component/s: Statistics
> analyze doesn't mark partition column stats as accurate after truncate
> --
>
> Key: HIVE-20143
> URL: https://issues.apache.org/jira/browse/HIVE-20143
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Priority: Major
>
> Discovered while looking at txn stats. This works fine for truncate+analyze
> for non-partitioned tables, but not for partitions:
> {noformat}
> set hive.stats.dbclass=fs;
> set hive.stats.fetch.column.stats=true;
> set hive.stats.autogather=true;
> set hive.stats.column.autogather=true;
> set hive.compute.query.using.stats=true;
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.fetch.task.conversion=none;
> set hive.query.results.cache.enabled=false;
> create table stats_part1(key int,value string) partitioned by (p int);
> insert into table stats_part1 partition(p=101) values (1, "foo");
> insert into table stats_part1 partition(p=102) values (2, "bar");
> explain select count(key) from stats_part1; -- from stats
> truncate table stats_part1 partition(p=101);
> explain select count(key) from stats_part1; -- not from stats, ok
> analyze table stats_part1 partition(p) compute statistics for columns;
> explain select count(key) from stats_part1; -- not from stats still
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20143) analyze doesn't mark partition column stats as accurate after truncate
[
https://issues.apache.org/jira/browse/HIVE-20143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540710#comment-16540710
]
Sergey Shelukhin commented on HIVE-20143:
-
cc [~jcamachorodriguez] [~ashutoshc] I don't if we care much about this
scenario. Could affect CBO if after truncate, the stats are never accurate.
> analyze doesn't mark partition column stats as accurate after truncate
> --
>
> Key: HIVE-20143
> URL: https://issues.apache.org/jira/browse/HIVE-20143
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> Discovered while looking at txn stats. This works fine for truncate+analyze
> for non-partitioned tables, but not for partitions:
> {noformat}
> set hive.stats.dbclass=fs;
> set hive.stats.fetch.column.stats=true;
> set hive.stats.autogather=true;
> set hive.stats.column.autogather=true;
> set hive.compute.query.using.stats=true;
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.fetch.task.conversion=none;
> set hive.query.results.cache.enabled=false;
> create table stats_part1(key int,value string) partitioned by (p int);
> insert into table stats_part1 partition(p=101) values (1, "foo");
> insert into table stats_part1 partition(p=102) values (2, "bar");
> explain select count(key) from stats_part1; -- from stats
> truncate table stats_part1 partition(p=101);
> explain select count(key) from stats_part1; -- not from stats, ok
> analyze table stats_part1 partition(p) compute statistics for columns;
> explain select count(key) from stats_part1; -- not from stats still
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20076) ACID: Fix Synthetic ROW__ID generation for vectorized orc readers
[ https://issues.apache.org/jira/browse/HIVE-20076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20076: -- Fix Version/s: (was: 3.2.0) 3.1.0 > ACID: Fix Synthetic ROW__ID generation for vectorized orc readers > - > > Key: HIVE-20076 > URL: https://issues.apache.org/jira/browse/HIVE-20076 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-20076.2.patch, HIVE-20076.3.patch, HIVE-20076.patch > > > Delete on a partitioned table removes more rows than expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20076) ACID: Fix Synthetic ROW__ID generation for vectorized orc readers
[ https://issues.apache.org/jira/browse/HIVE-20076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540694#comment-16540694 ] Teddy Choi commented on HIVE-20076: --- Committed to branch-3, branch-3.1 and master. Thanks [~ekoifman], [~gopalv], [~sershe]. > ACID: Fix Synthetic ROW__ID generation for vectorized orc readers > - > > Key: HIVE-20076 > URL: https://issues.apache.org/jira/browse/HIVE-20076 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-20076.2.patch, HIVE-20076.3.patch, HIVE-20076.patch > > > Delete on a partitioned table removes more rows than expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter
[
https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540692#comment-16540692
]
Hive QA commented on HIVE-16295:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
53s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 50 new + 343 unchanged - 23
fixed = 393 total (was 366) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 15
unchanged - 1 fixed = 16 total (was 16) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
56s{color} | {color:red} ql generated 2 new + 2285 unchanged - 2 fixed = 2287
total (was 2287) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Exception is caught when Exception is not thrown in
org.apache.hadoop.hive.ql.exec.HiveDataCommitter.moveFile(HiveConf, Path, Path,
boolean, boolean, boolean) At HiveDataCommitter.java:is not thrown in
org.apache.hadoop.hive.ql.exec.HiveDataCommitter.moveFile(HiveConf, Path, Path,
boolean, boolean, boolean) At HiveDataCommitter.java:[line 337] |
| |
org.apache.hadoop.hive.ql.exec.HiveDataCommitter.cleanUpOneDirectoryForReplace(Path,
FileSystem, PathFilter, HiveConf, boolean, boolean, Hive) concatenates strings
using + in a loop At HiveDataCommitter.java:boolean, Hive) concatenates
strings using + in a loop At HiveDataCommitter.java:[line 636] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12537/dev-support/hive-personality.sh
|
| git revision | master / 385a26a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12537/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
[jira] [Updated] (HIVE-20019) Remove commons-logging and move to slf4j
[ https://issues.apache.org/jira/browse/HIVE-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20019: - Attachment: HIVE-20019.4.patch > Remove commons-logging and move to slf4j > > > Key: HIVE-20019 > URL: https://issues.apache.org/jira/browse/HIVE-20019 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20019.1.patch, HIVE-20019.2.patch, > HIVE-20019.3.patch, HIVE-20019.4.patch > > > Still seeing several references to commons-logging. We should move all > classes to slf4j instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20076) ACID: Fix Synthetic ROW__ID generation for vectorized orc readers
[ https://issues.apache.org/jira/browse/HIVE-20076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20076: -- Fix Version/s: 3.2.0 > ACID: Fix Synthetic ROW__ID generation for vectorized orc readers > - > > Key: HIVE-20076 > URL: https://issues.apache.org/jira/browse/HIVE-20076 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-20076.2.patch, HIVE-20076.3.patch, HIVE-20076.patch > > > Delete on a partitioned table removes more rows than expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20097) Convert standalone-metastore to a submodule
[ https://issues.apache.org/jira/browse/HIVE-20097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-20097: -- Attachment: HIVE-20097.05.patch > Convert standalone-metastore to a submodule > --- > > Key: HIVE-20097 > URL: https://issues.apache.org/jira/browse/HIVE-20097 > Project: Hive > Issue Type: Sub-task > Components: Hive, Metastore, Standalone Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20097.01.patch, HIVE-20097.02.patch, > HIVE-20097.03.patch, HIVE-20097.04.patch, HIVE-20097.05.patch > > > This is a subtask to stage HIVE-17751 changes into several smaller phases. > The first part is moving existing code in hive-standalone-metastore to a > sub-module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20097) Convert standalone-metastore to a submodule
[
https://issues.apache.org/jira/browse/HIVE-20097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540675#comment-16540675
]
Alexander Kolbasov commented on HIVE-20097:
---
Test failures do not look related but I am retrying the patch. Patch 5 is
merged with
{code}
* commit 733c4f3169142d426995f8e12961c209688d29d0 (origin/master, origin/HEAD)
| Author: Eugene Koifman
| Date: Wed Jul 11 14:02:31 2018 -0700
|
| HIVE-19387: Truncate table for Acid tables conflicts with ResultSet cache
(Eugene Koifman, reviewed by Jason Dere)
{code}
> Convert standalone-metastore to a submodule
> ---
>
> Key: HIVE-20097
> URL: https://issues.apache.org/jira/browse/HIVE-20097
> Project: Hive
> Issue Type: Sub-task
> Components: Hive, Metastore, Standalone Metastore
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20097.01.patch, HIVE-20097.02.patch,
> HIVE-20097.03.patch, HIVE-20097.04.patch, HIVE-20097.05.patch
>
>
> This is a subtask to stage HIVE-17751 changes into several smaller phases.
> The first part is moving existing code in hive-standalone-metastore to a
> sub-module.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-14493) Partitioning support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-14493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-14493:
--
Assignee: Jesus Camacho Rodriguez
> Partitioning support for materialized views
> ---
>
> Key: HIVE-14493
> URL: https://issues.apache.org/jira/browse/HIVE-14493
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> We should support defining a partitioning specification for materialized
> views and that the results of the materialized view evaluation are stored
> meeting the partitioning spec.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [PARTITIONED ON (col_name, ...)] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19902) Provide Metastore micro-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-19902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540665#comment-16540665 ] Alexander Kolbasov commented on HIVE-19902: --- [~owen.omalley] Thank you for the example - I didn't know about Level.Trial annotations. I'll experiment with JMH more using this approach. I still have other concerns with JMH since it doesn't give me full control over the results dataset. One feature that I am using a lot is saving raw data in files and looking at gnuplot graphs. [~alangates] [~pvary] [~vihangk1] [~owen.omalley] I still think that the current approach (not using JMH) has its benefits - it is very simple and doesn't have any licensing issues. What do you think about going ahead with the current proposed implementation and then later possibly adding separate JMH-based submodule which isn't compiled by default? > Provide Metastore micro-benchmarks > -- > > Key: HIVE-19902 > URL: https://issues.apache.org/jira/browse/HIVE-19902 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-19902.01.patch, HIVE-19902.02.patch, > HIVE-19902.03.patch, HIVE-19902.04.patch > > > It would be very useful to have metastore benchmarks to be able to track perf > issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20106) Backport HIVE-20060 (HiveSchemaTool and MetastoreSchemaTool refactor) to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-20106: -- Resolution: Fixed Status: Resolved (was: Patch Available) Got a clean run locally. Pushed patch to branch-3. > Backport HIVE-20060 (HiveSchemaTool and MetastoreSchemaTool refactor) to > branch-3 > - > > Key: HIVE-20106 > URL: https://issues.apache.org/jira/browse/HIVE-20106 > Project: Hive > Issue Type: Task > Components: Beeline, Metastore >Affects Versions: 3.1.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-20106-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19387) Truncate table for Acid tables conflicts with ResultSet cache
[ https://issues.apache.org/jira/browse/HIVE-19387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540657#comment-16540657 ] Eugene Koifman commented on HIVE-19387: --- pushed to master thanks Jason for the review > Truncate table for Acid tables conflicts with ResultSet cache > - > > Key: HIVE-19387 > URL: https://issues.apache.org/jira/browse/HIVE-19387 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-19387.01-branch-3.patch, HIVE-19387.01.patch > > > How should this work? Should it work like Insert Overwrite T select * from T > where 1=2? > This should create a new empty base_x/ and thus operate w/o violating > Snapshot Isolation semantics. > This makes sense for specific partition or unpartitioned table. What about > "Truncate T" where T is partitioned? Is the expectation to wipe out all > partition info or to make each partition empty? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19387) Truncate table for Acid tables conflicts with ResultSet cache
[ https://issues.apache.org/jira/browse/HIVE-19387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19387: -- Attachment: HIVE-19387.01-branch-3.patch > Truncate table for Acid tables conflicts with ResultSet cache > - > > Key: HIVE-19387 > URL: https://issues.apache.org/jira/browse/HIVE-19387 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-19387.01-branch-3.patch, HIVE-19387.01.patch > > > How should this work? Should it work like Insert Overwrite T select * from T > where 1=2? > This should create a new empty base_x/ and thus operate w/o violating > Snapshot Isolation semantics. > This makes sense for specific partition or unpartitioned table. What about > "Truncate T" where T is partitioned? Is the expectation to wipe out all > partition info or to make each partition empty? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19387) Truncate table for Acid tables conflicts with ResultSet cache
[
https://issues.apache.org/jira/browse/HIVE-19387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540633#comment-16540633
]
Hive QA commented on HIVE-19387:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931078/HIVE-19387.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14643 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12536/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12536/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12536/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931078 - PreCommit-HIVE-Build
> Truncate table for Acid tables conflicts with ResultSet cache
> -
>
> Key: HIVE-19387
> URL: https://issues.apache.org/jira/browse/HIVE-19387
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-19387.01.patch
>
>
> How should this work? Should it work like Insert Overwrite T select * from T
> where 1=2?
> This should create a new empty base_x/ and thus operate w/o violating
> Snapshot Isolation semantics.
> This makes sense for specific partition or unpartitioned table. What about
> "Truncate T" where T is partitioned? Is the expectation to wipe out all
> partition info or to make each partition empty?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20106) Backport HIVE-20060 (HiveSchemaTool and MetastoreSchemaTool refactor) to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-20106: -- Attachment: HIVE-20106-branch-3.patch > Backport HIVE-20060 (HiveSchemaTool and MetastoreSchemaTool refactor) to > branch-3 > - > > Key: HIVE-20106 > URL: https://issues.apache.org/jira/browse/HIVE-20106 > Project: Hive > Issue Type: Task > Components: Beeline, Metastore >Affects Versions: 3.1.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-20106-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20106) Backport HIVE-20060 (HiveSchemaTool and MetastoreSchemaTool refactor) to branch-3
[ https://issues.apache.org/jira/browse/HIVE-20106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-20106: -- Fix Version/s: 3.2.0 Affects Version/s: 3.1.0 Status: Patch Available (was: Open) > Backport HIVE-20060 (HiveSchemaTool and MetastoreSchemaTool refactor) to > branch-3 > - > > Key: HIVE-20106 > URL: https://issues.apache.org/jira/browse/HIVE-20106 > Project: Hive > Issue Type: Task > Components: Beeline, Metastore >Affects Versions: 3.1.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-20106-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19984) Backport HIVE-15976 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-19984: -- Resolution: Fixed Status: Resolved (was: Patch Available) Got a clean run locally. Patch pushed to branch-3. > Backport HIVE-15976 to branch-3 > --- > > Key: HIVE-19984 > URL: https://issues.apache.org/jira/browse/HIVE-19984 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 3.1.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-19984-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20142) Semijoin Reduction : Peform cost based removal after rule based removal.
[ https://issues.apache.org/jira/browse/HIVE-20142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal reassigned HIVE-20142: - > Semijoin Reduction : Peform cost based removal after rule based removal. > > > Key: HIVE-20142 > URL: https://issues.apache.org/jira/browse/HIVE-20142 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > The semijoin reduction removal logic is spread out into multiple functions. > Currently, the cost based removal logic is applied before the rule > based(dumb) ones. > Instead, apply the rule based removal logic and then apply the cost based > removal. > > cc [~jdere] [~jcamachorodriguez] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19984) Backport HIVE-15976 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-19984: -- Fix Version/s: 3.2.0 Status: Patch Available (was: Open) > Backport HIVE-15976 to branch-3 > --- > > Key: HIVE-19984 > URL: https://issues.apache.org/jira/browse/HIVE-19984 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 3.1.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-19984-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19984) Backport HIVE-15976 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-19984: -- Attachment: HIVE-19984-branch-3.patch > Backport HIVE-15976 to branch-3 > --- > > Key: HIVE-19984 > URL: https://issues.apache.org/jira/browse/HIVE-19984 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 3.1.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-19984-branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19387) Truncate table for Acid tables conflicts with ResultSet cache
[
https://issues.apache.org/jira/browse/HIVE-19387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540600#comment-16540600
]
Hive QA commented on HIVE-19387:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12536/dev-support/hive-personality.sh
|
| git revision | master / 385a26a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12536/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Truncate table for Acid tables conflicts with ResultSet cache
> -
>
> Key: HIVE-19387
> URL: https://issues.apache.org/jira/browse/HIVE-19387
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-19387.01.patch
>
>
> How should this work? Should it work like Insert Overwrite T select * from T
> where 1=2?
> This should create a new empty base_x/ and thus operate w/o violating
> Snapshot Isolation semantics.
> This makes sense for specific partition or unpartitioned table. What about
> "Truncate T" where T is partitioned? Is the expectation to wipe out all
> partition info or to make each partition empty?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19360) CBO: Add an "optimizedSQL" to QueryPlan object
[
https://issues.apache.org/jira/browse/HIVE-19360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19360:
---
Attachment: HIVE-19360.5.patch
> CBO: Add an "optimizedSQL" to QueryPlan object
> ---
>
> Key: HIVE-19360
> URL: https://issues.apache.org/jira/browse/HIVE-19360
> Project: Hive
> Issue Type: Improvement
> Components: CBO, Diagnosability
>Affects Versions: 3.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19360.1.patch, HIVE-19360.2.patch,
> HIVE-19360.3.patch, HIVE-19360.4.patch, HIVE-19360.5.patch
>
>
> Calcite RelNodes can be converted back into SQL (as the new JDBC storage
> handler does), which allows Hive to print out the post CBO plan as a SQL
> query instead of having to guess the join orders from the subsequent Tez plan.
> The query generated might not be always valid SQL at this point, but is a
> world ahead of DAG plans in readability.
> Eg. tpc-ds Query4 CTEs gets expanded to
> {code}
> SELECT t16.$f3 customer_preferred_cust_flag
> FROM
> (SELECT t0.c_customer_id $f0,
>SUM((t2.ws_ext_list_price -
> t2.ws_ext_wholesale_cost - t2.ws_ext_discount_amt + t2.ws_ext_sales_price) /
> CAST(2 AS DECIMAL(10, 0))) $f8
>FROM
> (SELECT c_customer_sk,
> c_customer_id,
> c_first_name,
> c_last_name,
> c_preferred_cust_flag,
> c_birth_country,
> c_login,
> c_email_address
> FROM default.customer
> WHERE c_customer_sk IS NOT NULL
> AND c_customer_id IS NOT NULL) t0
>INNER JOIN (
> (SELECT ws_sold_date_sk,
> ws_bill_customer_sk,
> ws_ext_discount_amt,
> ws_ext_sales_price,
> ws_ext_wholesale_cost,
> ws_ext_list_price
> FROM default.web_sales
> WHERE ws_bill_customer_sk IS NOT NULL
> AND ws_sold_date_sk IS NOT NULL) t2
>INNER JOIN
> (SELECT d_date_sk,
> CAST(2002 AS INTEGER) d_year
> FROM default.date_dim
> WHERE d_year = 2002
> AND d_date_sk IS NOT NULL) t4 ON t2.ws_sold_date_sk =
> t4.d_date_sk) ON t0.c_customer_sk = t2.ws_bill_customer_sk
>GROUP BY t0.c_customer_id,
> t0.c_first_name,
> t0.c_last_name,
> t0.c_preferred_cust_flag,
> t0.c_birth_country,
> t0.c_login,
> t0.c_email_address) t7
> INNER JOIN (
> (SELECT t9.c_customer_id $f0,
>t9.c_preferred_cust_flag $f3,
>
> SUM((t11.ss_ext_list_price - t11.ss_ext_wholesale_cost -
> t11.ss_ext_discount_amt + t11.ss_ext_sales_price) / CAST(2 AS DECIMAL(10,
> 0))) $f8
>FROM
> (SELECT c_customer_sk,
> c_customer_id,
> c_first_name,
> c_last_name,
> c_preferred_cust_flag,
> c_birth_country,
> c_login,
> c_email_address
> FROM default.customer
> WHERE c_customer_sk IS NOT NULL
> AND c_customer_id IS NOT NULL) t9
>INNER JOIN (
> (SELECT ss_sold_date_sk,
> ss_customer_sk,
> ss_ext_discount_amt,
> ss_ext_sales_price,
> ss_ext_wholesale_cost,
> ss_ext_list_price
> FROM default.store_sales
> WHERE ss_customer_sk IS NOT NULL
> AND ss_sold_date_sk IS NOT NULL) t11
>INNER JOIN
> (SELECT d_date_sk,
> CAST(2002 AS INTEGER) d_year
> FROM default.date_dim
> WHERE d_year = 2002
> AND d_date_sk IS NOT NULL) t13 ON
> t11.ss_sold_date_sk = t13.d_date_sk) ON t9.c_customer_sk = t11.ss_customer_sk
>GROUP BY t9.c_customer_id,
> t9.c_first_name,
> t9.c_last_name,
> t9.c_preferred_cust_flag,
> t9.c_birth_country,
> t9.c_login,
[jira] [Commented] (HIVE-20108) Investigate alternatives to groupByKey
[
https://issues.apache.org/jira/browse/HIVE-20108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540599#comment-16540599
]
Sahil Takiar commented on HIVE-20108:
-
Another limitation of {{groupByKey}} is that it can't push any aggregation into
any of the shuffle logic. What we probably want here is something like
{{combineByKey}} or {{reduceByKey}}. These functions allow specifying an
{{Aggregator}} which is called in {{ExternalAppendOnlyMap}} to push aggregation
into the shuffle-reader (e.g. {{BlockStoreShuffleReader}}). This allows
aggregating data in memory, which avoids having to spill much data to disk.
{{groupByKey}} doesn't have this functionality and all data has to be stored in
the {{ExternalAppendOnlyMap}} before any aggregation by Hive can start, this
can result in a lot more spilled data.
Getting {{combineByKey}} / {{reduceByKey}} to work with HoS looks tricky, we
basically have to wrap the {{GroupByOperator}} into a function that Spark can
call. This should significantly decrease the chance of OOM that has been seen
in {{groupByKey}} since we are pushing the aggregation as far down as possible
which results in less data being stored in memory and spilled to disk.
> Investigate alternatives to groupByKey
> --
>
> Key: HIVE-20108
> URL: https://issues.apache.org/jira/browse/HIVE-20108
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> We use {{groupByKey}} for aggregations (or if
> {{hive.spark.use.groupby.shuffle}} is false we use
> {{repartitionAndSortWithinPartitions}}).
> {{groupByKey}} has its drawbacks because it can't spill records within a
> single key group. It also seems to be doing some unnecessary work in Spark's
> {{Aggregator}} (not positive about this part).
> {{repartitionAndSortWithinPartitions}} is better, but the sorting within
> partitions isn't necessary for aggregations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20090) Extend creation of semijoin reduction filters to be able to discover new opportunities
[
https://issues.apache.org/jira/browse/HIVE-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540584#comment-16540584
]
Deepak Jaiswal commented on HIVE-20090:
---
cc [~jdere]
> Extend creation of semijoin reduction filters to be able to discover new
> opportunities
> --
>
> Key: HIVE-20090
> URL: https://issues.apache.org/jira/browse/HIVE-20090
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20090.01.patch, HIVE-20090.02.patch,
> HIVE-20090.04.patch, HIVE-20090.05.patch
>
>
> Assume the following plan:
> {noformat}
> TS[0] - RS[1] - JOIN[4] - RS[5] - JOIN[8] - FS[9]
> TS[2] - RS[3] - JOIN[4]
> TS[6] - RS[7] - JOIN[8]
> {noformat}
> Currently, {{TS\[6\]}} may only be reduced with the output of {{RS\[5\]}},
> i.e., input to join between both subplans.
> However, it may be useful to consider other possibilities too, e.g., reduced
> by the output of {{RS\[1\]}} or {{RS\[3\]}}. For instance, this is important
> when, given a large plan, an edge between {{RS[5]}} and {{TS[0]}} would
> create a cycle, while an edge between {{RS[1]}} and {{TS[6]}} would not.
> This patch comprises two parts. First, it creates additional predicates when
> possible. Secondly, it removes duplicate semijoin reduction
> branches/predicates, e.g., if another semijoin that consumes the output of
> the same expression already reduces a certain table scan operator (heuristic,
> since this may not result in most efficient plan in all cases). Ultimately,
> the decision on whether to use one or another should be cost-driven
> (follow-up).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20123) Fix masking tests after HIVE-19617
[
https://issues.apache.org/jira/browse/HIVE-20123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540581#comment-16540581
]
Hive QA commented on HIVE-20123:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931071/HIVE-20123.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14642 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12535/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12535/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12535/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931071 - PreCommit-HIVE-Build
> Fix masking tests after HIVE-19617
> --
>
> Key: HIVE-20123
> URL: https://issues.apache.org/jira/browse/HIVE-20123
> Project: Hive
> Issue Type: Test
>Affects Versions: 3.0.0, 3.1.0, 4.0.0, 3.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20123.01.patch, HIVE-20123.patch
>
>
> Masking tests results were changed inadvertently when HIVE-19617 went in,
> since table names were changed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20123) Fix masking tests after HIVE-19617
[
https://issues.apache.org/jira/browse/HIVE-20123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540544#comment-16540544
]
Hive QA commented on HIVE-20123:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} itests/util: The patch generated 4 new + 5 unchanged -
0 fixed = 9 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12535/dev-support/hive-personality.sh
|
| git revision | master / 385a26a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12535/yetus/diff-checkstyle-itests_util.txt
|
| modules | C: ql itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12535/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix masking tests after HIVE-19617
> --
>
> Key: HIVE-20123
> URL: https://issues.apache.org/jira/browse/HIVE-20123
> Project: Hive
> Issue Type: Test
>Affects Versions: 3.0.0, 3.1.0, 4.0.0, 3.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20123.01.patch, HIVE-20123.patch
>
>
> Masking tests results were changed inadvertently when HIVE-19617 went in,
> since table names were changed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19387) Truncate table for Acid tables conflicts with ResultSet cache
[ https://issues.apache.org/jira/browse/HIVE-19387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540510#comment-16540510 ] Jason Dere commented on HIVE-19387: --- +1 > Truncate table for Acid tables conflicts with ResultSet cache > - > > Key: HIVE-19387 > URL: https://issues.apache.org/jira/browse/HIVE-19387 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-19387.01.patch > > > How should this work? Should it work like Insert Overwrite T select * from T > where 1=2? > This should create a new empty base_x/ and thus operate w/o violating > Snapshot Isolation semantics. > This makes sense for specific partition or unpartitioned table. What about > "Truncate T" where T is partitioned? Is the expectation to wipe out all > partition info or to make each partition empty? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
