[jira] [Commented] (HIVE-20291) Allow HiveStreamingConnection to receive a WriteId
[
https://issues.apache.org/jira/browse/HIVE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566447#comment-16566447
]
Hive QA commented on HIVE-20291:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12933991/HIVE-20291.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 14842 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_masking]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=193)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12996/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12996/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12996/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12933991 - PreCommit-HIVE-Build
> Allow HiveStreamingConnection to receive a WriteId
> --
>
> Key: HIVE-20291
> URL: https://issues.apache.org/jira/browse/HIVE-20291
> Project: Hive
> Issue Type: Improvement
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20291.1.patch
>
>
> If the writeId is received externally it won't need to open connections to
> the metastore. It won't be able to the commit in this case as well so it must
> be done by the entity passing the writeId.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20281) SharedWorkOptimizer fails with 'operator cache contents and actual plan differ'
[
https://issues.apache.org/jira/browse/HIVE-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566474#comment-16566474
]
Hive QA commented on HIVE-20281:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
11s{color} | {color:blue} ql in master has 2302 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12997/dev-support/hive-personality.sh
|
| git revision | master / 2cabb8d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12997/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SharedWorkOptimizer fails with 'operator cache contents and actual plan
> differ'
> ---
>
> Key: HIVE-20281
> URL: https://issues.apache.org/jira/browse/HIVE-20281
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Ashutosh Chauhan
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-20281.patch
>
>
> HIVE-18201 seems to trigger a latent bug in SW optimizer. Test
> {{subquery_in_having}} fails with:
> {code}
> 2018-07-31T08:42:57,328 DEBUG [b68f20cc-54d5-466d-b512-1540b3a43396 main]
> optimizer.SharedWorkOptimizer: After SharedWorkExtendedOptimizer:
> TS[0]-SEL[1]-MAPJOIN[131]-FIL[12]-SEL[13]-GBY[14]-RS[15]-GBY[16]-SEL[17]-MAPJOIN[136]-MAPJOIN[137]-FIL[103]-SEL[104]-FS[105]
>
> -FIL[113]-SEL[20]-RS[44]-MAPJOIN[133]-SEL[47]-GBY[48]-RS[49]-GBY[50]-SEL[51]-GBY[55]-RS[98]-MAPJOIN[136]
>
> -RS[88]-GBY[89]-SEL[120]-FIL[116]-SEL[91]-GBY[93]-RS[94]-GBY[95]-SEL[96]-RS[101]-MAPJOIN[137]
> TS[2]-FIL[112]-GBY[5]-RS[6]-GBY[7]-SEL[8]-RS[10]-MAPJOIN[131]
>
> -RS[31]-MAPJOIN[132]-FIL[33]-SEL[34]-GBY[35]-RS[36]-GBY[37]-SEL[38]-GBY[42]-MAPJOIN[133]
> TS[21]-FIL[114]-SEL[22]-MAPJOIN[132]
> 2018-07-31T08:42:57,329 ERROR [b68f20cc-54d5-466d-b512-1540b3a43396 main]
> ql.Driver: FAILED: SemanticException Error in shared work optimizer: operator
> cache contentsand actual plan differ
> org.apache.hadoop.hive.ql.parse.SemanticExce
[jira] [Updated] (HIVE-20121) investigate issues with TestReplicationScenariosAcidTables
[
https://issues.apache.org/jira/browse/HIVE-20121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-20121:
---
Status: Patch Available (was: In Progress)
> investigate issues with TestReplicationScenariosAcidTables
> --
>
> Key: HIVE-20121
> URL: https://issues.apache.org/jira/browse/HIVE-20121
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-20121.01.patch, HIVE-20121.02.patch,
> HIVE-20121.03.patch
>
>
> [~djaiswal] have noticed that somehow ptest execution are getting more and
> more problems lately... it seems to me that these problems are coming from
> this test
> {code}
> 2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable to
> shutdown metastore client. Will try closing transport directly.
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream
> {code}
> some links to more or less recent logs:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
> the hive.log is ~200M:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20121) investigate issues with TestReplicationScenariosAcidTables
[
https://issues.apache.org/jira/browse/HIVE-20121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-20121:
---
Attachment: HIVE-20121.03.patch
> investigate issues with TestReplicationScenariosAcidTables
> --
>
> Key: HIVE-20121
> URL: https://issues.apache.org/jira/browse/HIVE-20121
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-20121.01.patch, HIVE-20121.02.patch,
> HIVE-20121.03.patch
>
>
> [~djaiswal] have noticed that somehow ptest execution are getting more and
> more problems lately... it seems to me that these problems are coming from
> this test
> {code}
> 2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable to
> shutdown metastore client. Will try closing transport directly.
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream
> {code}
> some links to more or less recent logs:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
> the hive.log is ~200M:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20121) investigate issues with TestReplicationScenariosAcidTables
[
https://issues.apache.org/jira/browse/HIVE-20121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-20121:
---
Status: In Progress (was: Patch Available)
> investigate issues with TestReplicationScenariosAcidTables
> --
>
> Key: HIVE-20121
> URL: https://issues.apache.org/jira/browse/HIVE-20121
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-20121.01.patch, HIVE-20121.02.patch,
> HIVE-20121.03.patch
>
>
> [~djaiswal] have noticed that somehow ptest execution are getting more and
> more problems lately... it seems to me that these problems are coming from
> this test
> {code}
> 2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable to
> shutdown metastore client. Will try closing transport directly.
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream
> {code}
> some links to more or less recent logs:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
> the hive.log is ~200M:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20293) Support Replication of ACID table truncate operation
[ https://issues.apache.org/jira/browse/HIVE-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera reassigned HIVE-20293: -- > Support Replication of ACID table truncate operation > > > Key: HIVE-20293 > URL: https://issues.apache.org/jira/browse/HIVE-20293 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > > Add tags in jobconf for distcp related jobs started by replication. This will > allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon > issues a kill command. > * one of the tags should definitely be the query_id that starts the job : > With this flow beacon before retrying the bootstrap load, will issue a kill > command to hs2 with the query id of the previous issued command. hs2 will > then kill an running jobs on yarn tagged with the Query_id. > * To get around the additional failure point as mentioned above. The jobs > can be tagged with an additional unique tag_id provided by Beacon in the WITH > clause in repl load command to be used to tag distcp jobs ). Enhance the kill > api to take the tag as input and kill jobs associated with that tag. Problem > here is how do we validate the association of the tag with a hive query id to > make sure this api is not used to kill jobs run by other components, however > we can provide this capability to only admins and should be ok in that case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19924) Tag distcp jobs run by Repl Load
[ https://issues.apache.org/jira/browse/HIVE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19924: --- Target Version/s: 4.0.0, 3.2.0 (was: 3.1.0, 4.0.0) > Tag distcp jobs run by Repl Load > > > Key: HIVE-19924 > URL: https://issues.apache.org/jira/browse/HIVE-19924 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19924.01.patch, HIVE-19924.02.patch, > HIVE-19924.03.patch, HIVE-19924.04.patch, HIVE-19924.05.patch, > HIVE-19924.06.patch, HIVE-19924.07.patch, HIVE-19924.08.patch > > > Add tags in jobconf for distcp related jobs started by replication. This will > allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon > issues a kill command. > * one of the tags should definitely be the query_id that starts the job : > With this flow beacon before retrying the bootstrap load, will issue a kill > command to hs2 with the query id of the previous issued command. hs2 will > then kill an running jobs on yarn tagged with the Query_id. > * To get around the additional failure point as mentioned above. The jobs > can be tagged with an additional unique tag_id provided by Beacon in the WITH > clause in repl load command to be used to tag distcp jobs ). Enhance the kill > api to take the tag as input and kill jobs associated with that tag. Problem > here is how do we validate the association of the tag with a hive query id to > make sure this api is not used to kill jobs run by other components, however > we can provide this capability to only admins and should be ok in that case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20293) Support Replication of ACID table truncate operation
[ https://issues.apache.org/jira/browse/HIVE-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20293: --- Description: Support truncate acid table replication. 1. Write id allocation needs to be removed was: Add tags in jobconf for distcp related jobs started by replication. This will allow hive to kill these jobs in case beacon retries, or hs2 dies and beacon issues a kill command. * one of the tags should definitely be the query_id that starts the job : With this flow beacon before retrying the bootstrap load, will issue a kill command to hs2 with the query id of the previous issued command. hs2 will then kill an running jobs on yarn tagged with the Query_id. * To get around the additional failure point as mentioned above. The jobs can be tagged with an additional unique tag_id provided by Beacon in the WITH clause in repl load command to be used to tag distcp jobs ). Enhance the kill api to take the tag as input and kill jobs associated with that tag. Problem here is how do we validate the association of the tag with a hive query id to make sure this api is not used to kill jobs run by other components, however we can provide this capability to only admins and should be ok in that case. > Support Replication of ACID table truncate operation > > > Key: HIVE-20293 > URL: https://issues.apache.org/jira/browse/HIVE-20293 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > > Support truncate acid table replication. > 1. Write id allocation needs to be removed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20281) SharedWorkOptimizer fails with 'operator cache contents and actual plan differ'
[
https://issues.apache.org/jira/browse/HIVE-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566487#comment-16566487
]
Hive QA commented on HIVE-20281:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934002/HIVE-20281.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14841 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12997/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12997/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12997/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934002 - PreCommit-HIVE-Build
> SharedWorkOptimizer fails with 'operator cache contents and actual plan
> differ'
> ---
>
> Key: HIVE-20281
> URL: https://issues.apache.org/jira/browse/HIVE-20281
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Ashutosh Chauhan
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-20281.patch
>
>
> HIVE-18201 seems to trigger a latent bug in SW optimizer. Test
> {{subquery_in_having}} fails with:
> {code}
> 2018-07-31T08:42:57,328 DEBUG [b68f20cc-54d5-466d-b512-1540b3a43396 main]
> optimizer.SharedWorkOptimizer: After SharedWorkExtendedOptimizer:
> TS[0]-SEL[1]-MAPJOIN[131]-FIL[12]-SEL[13]-GBY[14]-RS[15]-GBY[16]-SEL[17]-MAPJOIN[136]-MAPJOIN[137]-FIL[103]-SEL[104]-FS[105]
>
> -FIL[113]-SEL[20]-RS[44]-MAPJOIN[133]-SEL[47]-GBY[48]-RS[49]-GBY[50]-SEL[51]-GBY[55]-RS[98]-MAPJOIN[136]
>
> -RS[88]-GBY[89]-SEL[120]-FIL[116]-SEL[91]-GBY[93]-RS[94]-GBY[95]-SEL[96]-RS[101]-MAPJOIN[137]
> TS[2]-FIL[112]-GBY[5]-RS[6]-GBY[7]-SEL[8]-RS[10]-MAPJOIN[131]
>
> -RS[31]-MAPJOIN[132]-FIL[33]-SEL[34]-GBY[35]-RS[36]-GBY[37]-SEL[38]-GBY[42]-MAPJOIN[133]
> TS[21]-FIL[114]-SEL[22]-MAPJOIN[132]
> 2018-07-31T08:42:57,329 ERROR [b68f20cc-54d5-466d-b512-1540b3a43396 main]
> ql.Driver: FAILED: SemanticException Error in shared work optimizer: operator
> cache contentsand actual plan differ
> org.apache.hadoop.hive.ql.parse.SemanticException: Error in shared work
> optimizer: operator cache contentsand actual plan differ
> at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.transform(SharedWorkOptimizer.java:524)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:185)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:146)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12361)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:356)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:165)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:663)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20294) Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
[ https://issues.apache.org/jira/browse/HIVE-20294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-20294: --- > Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT > > > Key: HIVE-20294 > URL: https://issues.apache.org/jira/browse/HIVE-20294 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized COALESCE and ELT. > Also, add tests for ARRAY and MAP indexing, IS [NOT] NULL and NOT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20294) Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
[ https://issues.apache.org/jira/browse/HIVE-20294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20294: Status: Patch Available (was: Open) > Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT > > > Key: HIVE-20294 > URL: https://issues.apache.org/jira/browse/HIVE-20294 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20294.01.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized COALESCE and ELT. > Also, add tests for ARRAY and MAP indexing, IS [NOT] NULL and NOT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20294) Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
[ https://issues.apache.org/jira/browse/HIVE-20294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20294: Attachment: HIVE-20294.01.patch > Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT > > > Key: HIVE-20294 > URL: https://issues.apache.org/jira/browse/HIVE-20294 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20294.01.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized COALESCE and ELT. > Also, add tests for ARRAY and MAP indexing, IS [NOT] NULL and NOT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20284) In strict mode, if constant propagation is enable, the partition filter may be folded before partition pruner lead to error "No partition predicate for Alias"
[
https://issues.apache.org/jira/browse/HIVE-20284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566517#comment-16566517
]
Hive QA commented on HIVE-20284:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
14s{color} | {color:blue} ql in master has 2302 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12998/dev-support/hive-personality.sh
|
| git revision | master / 2cabb8d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12998/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> In strict mode, if constant propagation is enable, the partition filter may
> be folded before partition pruner lead to error "No partition predicate for
> Alias"
>
>
> Key: HIVE-20284
> URL: https://issues.apache.org/jira/browse/HIVE-20284
> Project: Hive
> Issue Type: Improvement
> Components: CLI
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Trivial
> Fix For: 2.3.3, 4.0.0
>
> Attachments: HIVE-20284.1.patch, HIVE-20284.2.patch,
> HIVE-20284.3.patch, HIVE-20284.patch
>
>
> In strict mode and the hive.optimize.constant.propagation is set to true, the
> following sql will failed:
> {code:java}
> hive> desc employee_part;
> OK
> col_name data_type comment
> eid int
> name string
> dept string
> year string
> month string
> # Partition Information
> # col_namedata_type comment
>
[jira] [Commented] (HIVE-20284) In strict mode, if constant propagation is enable, the partition filter may be folded before partition pruner lead to error "No partition predicate for Alias"
[
https://issues.apache.org/jira/browse/HIVE-20284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566536#comment-16566536
]
Hive QA commented on HIVE-20284:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934014/HIVE-20284.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14842 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12998/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12998/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12998/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934014 - PreCommit-HIVE-Build
> In strict mode, if constant propagation is enable, the partition filter may
> be folded before partition pruner lead to error "No partition predicate for
> Alias"
>
>
> Key: HIVE-20284
> URL: https://issues.apache.org/jira/browse/HIVE-20284
> Project: Hive
> Issue Type: Improvement
> Components: CLI
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Trivial
> Fix For: 2.3.3, 4.0.0
>
> Attachments: HIVE-20284.1.patch, HIVE-20284.2.patch,
> HIVE-20284.3.patch, HIVE-20284.patch
>
>
> In strict mode and the hive.optimize.constant.propagation is set to true, the
> following sql will failed:
> {code:java}
> hive> desc employee_part;
> OK
> col_name data_type comment
> eid int
> name string
> dept string
> year string
> month string
> # Partition Information
> # col_namedata_type comment
> year string
> month string
> Time taken: 0.564 seconds, Fetched: 11 row(s)
> hive> set hive.mapred.mode=strict;
> hive> select * from employee_part where false and concat(year,month)='201807';
> FAILED: SemanticException Queries against partitioned tables without a
> partition filter are disabled for safety reasons. If you know what you are
> doing, please sethive.strict.checks.large.query to false and that
> hive.mapred.mode is not set to 'strict' to proceed. Note that if you may get
> errors or incorrect results if you make a mistake while using some of the
> unsafe features. No partition predicate for Alias "employee_part" Table
> "employee_part"
> {code}
> The above error message is confusing because the expression
> concat(year,month)='201807' is a partition filter。
> The reason is during logic optimization, the ConstantPropagate optimizer is
> running before partitionPruner optimizer, when found a express like 'false
> and concat(year,month)=', the express is replaced with 'fasle' , and the
> partition filter is dropped. So the PartitionPruner can not get the partition
> filter.
> Users can remove the constant express that always has true/false values to
> work around.
> When views used, if some columns are constant values, users will be
> confusing.
> So we should add some more message in the error msg returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20274) HiveServer2 ObjectInspectorFactory leaks for Struct and List object inspectors
[
https://issues.apache.org/jira/browse/HIVE-20274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566563#comment-16566563
]
Hive QA commented on HIVE-20274:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} serde in master has 195 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} serde: The patch generated 23 new + 13 unchanged - 8
fixed = 36 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12999/dev-support/hive-personality.sh
|
| git revision | master / 2cabb8d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12999/yetus/diff-checkstyle-serde.txt
|
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12999/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveServer2 ObjectInspectorFactory leaks for Struct and List object inspectors
> --
>
> Key: HIVE-20274
> URL: https://issues.apache.org/jira/browse/HIVE-20274
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20274.1.patch, HIVE-20274.2.patch,
> HIVE-20274.3.patch
>
>
> Fix in HIVE-19860 needs to be applied to
> {code}
> static ConcurrentHashMap
> cachedStandardListObjectInspector = new
> ConcurrentHashMap();
> ...
> static ConcurrentHashMap>, StandardStructObjectInspector>
> cachedStandardStructObjectInspector =
> new ConcurrentHashMap>,
> StandardStructObjectInspector>();
> ...
> static ConcurrentHashMap, ColumnarStructObjectInspector>
> cachedColumnarStructObjectInspector =
> new ConcurrentHashMap,
> ColumnarStructObjectInspector>();
> {code}
> And possibly for
> {code}
> static ConcurrentHashMap objectInspectorCache = new
> ConcurrentHashMap();
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20274) HiveServer2 ObjectInspectorFactory leaks for Struct and List object inspectors
[
https://issues.apache.org/jira/browse/HIVE-20274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566600#comment-16566600
]
Hive QA commented on HIVE-20274:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934018/HIVE-20274.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14842 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12999/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12999/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12999/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934018 - PreCommit-HIVE-Build
> HiveServer2 ObjectInspectorFactory leaks for Struct and List object inspectors
> --
>
> Key: HIVE-20274
> URL: https://issues.apache.org/jira/browse/HIVE-20274
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20274.1.patch, HIVE-20274.2.patch,
> HIVE-20274.3.patch
>
>
> Fix in HIVE-19860 needs to be applied to
> {code}
> static ConcurrentHashMap
> cachedStandardListObjectInspector = new
> ConcurrentHashMap();
> ...
> static ConcurrentHashMap>, StandardStructObjectInspector>
> cachedStandardStructObjectInspector =
> new ConcurrentHashMap>,
> StandardStructObjectInspector>();
> ...
> static ConcurrentHashMap, ColumnarStructObjectInspector>
> cachedColumnarStructObjectInspector =
> new ConcurrentHashMap,
> ColumnarStructObjectInspector>();
> {code}
> And possibly for
> {code}
> static ConcurrentHashMap objectInspectorCache = new
> ConcurrentHashMap();
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20296) Improve HivePointLookupOptimizerRule to be able to extract from more sophisticated contexts
[
https://issues.apache.org/jira/browse/HIVE-20296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-20296:
---
> Improve HivePointLookupOptimizerRule to be able to extract from more
> sophisticated contexts
> ---
>
> Key: HIVE-20296
> URL: https://issues.apache.org/jira/browse/HIVE-20296
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> Currently it could handle conditions in the form:
> {code}
> ( ( a=1 && b=1 && ... ) || (a=2 && b=2 && ...) || ... )
> {code}
> If there is some noise somewhere in the or like:
> {code}
> ( x=18 || ( a=1 && b=1 && ... ) || (a=2 && b=2 && ...) || ... )
> {code}
> it stops processing.
> https://github.com/apache/hive/blob/2cabb8da150b8fb980223fbd6c2c93b842ca3ee5/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java#L238
> HIVE-19097 right now unwinds some ors which can't be packed back; like query15
> https://github.com/apache/hive/blob/2cabb8da150b8fb980223fbd6c2c93b842ca3ee5/ql/src/test/queries/clientpositive/perf/query15.q#L14
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20260) NDV of a column shouldn't be scaled when row count is changed by filter on another column
[
https://issues.apache.org/jira/browse/HIVE-20260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20260:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
pushed to master. Thank you Ashutosh for reviewing the changes!
> NDV of a column shouldn't be scaled when row count is changed by filter on
> another column
> -
>
> Key: HIVE-20260
> URL: https://issues.apache.org/jira/browse/HIVE-20260
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Ashutosh Chauhan
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20260.01.patch, HIVE-20260.01.patch,
> HIVE-20260.01wip01.patch, HIVE-20260.01wip02.patch, HIVE-20260.01wip03.patch
>
>
> HIVE-17465 introduced progressive scaling of rowcounts in presence of
> multiple filters. HIVE-19500 improved on that by also scaling col stats (NDV)
> in such scenario. However, it should pay attention to column used in filter
> expression and not scale for all filters. eg.,
> consider filter a = 1 and b = 2 ndv of column b should not be scaled down by
> row count changes caused by a = 1
> Other way to say this that ndv of a particular column should be updated at
> the end of computation of row count for that operator.
> Here are the possible cases where our estimates can be accurate (or close to)
> {code}
> case 1 - (d_year = 2001 and d_moy=1)
> case 2 - (d_year = 2001 and d_year IN (2001, 2002))
> case 3 - (d_year = 2001 and d_moy = 1 and d_dom = 1)
> case 4 - (d_date IN ('1999-01-02', '1999-01-02'))
> case 5 - (d_date = '1999-01-01')
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19097:
Attachment: HIVE-19097.06.patch
> related equals and in operators may cause inaccurate stats estimations
> --
>
> Key: HIVE-19097
> URL: https://issues.apache.org/jira/browse/HIVE-19097
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19097.01.patch, HIVE-19097.02.patch,
> HIVE-19097.03.patch, HIVE-19097.04.patch, HIVE-19097.05.patch,
> HIVE-19097.06.patch, HIVE-19097.06wip01.patch, HIVE-19097.06wip02.patch,
> HIVE-19097.partial.patch
>
>
> tpcds#74 is optimized in a way that for date_dim the condition contains IN
> and = for the same column
> {code:java}
> | Map Operator Tree: |
> | TableScan |
> | alias: date_dim |
> | filterExpr: (((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) or ((d_year) IN (2001, 2002) and (d_year =
> 2001) and d_date_sk is not null)) (type: boolean) |
> | Statistics: Num rows: 73049 Data size: 876588 Basic
> stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator |
> | predicate: ((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) (type: boolean) |
> | Statistics: Num rows: 4 Data size: 48 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
> the "real" row count will be 365
> for separate {{IN}} and {{=}} the estimation is very good; but if both are
> present it becomes (very) underestimated.
> {code:java}
> set hive.query.results.cache.enabled=false;
> drop table if exists t1;
> drop table if exists t8;
> create table t1 (a integer,b integer);
> create table t8 like t1;
> insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5);
> insert into t8
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1 union all
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1
> ;
> analyze table t1 compute statistics for columns;
> analyze table t8 compute statistics for columns;
> explain analyze select sum(a) from t8 where b in (2,3) group by b;
> explain analyze select sum(a) from t8 where b=2 group by b;
> explain analyze select sum(a) from t1 where b in (2,3) and b=2 group by b;
> explain analyze select sum(a) from t8 where b in (2,3) and b=2 group by b;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20261) Expose inputPartitionList in QueryPlan
[
https://issues.apache.org/jira/browse/HIVE-20261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566631#comment-16566631
]
Hive QA commented on HIVE-20261:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
33s{color} | {color:blue} ql in master has 2302 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
48s{color} | {color:red} ql: The patch generated 1 new + 514 unchanged - 0
fixed = 515 total (was 514) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
39s{color} | {color:red} ql generated 2 new + 2302 unchanged - 0 fixed = 2304
total (was 2302) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | The field org.apache.hadoop.hive.ql.QueryPlan.inputPartitionList is
transient but isn't set by deserialization In QueryPlan.java:but isn't set by
deserialization In QueryPlan.java |
| | Redundant nullcheck of
org.apache.hadoop.hive.ql.metadata.Table.getPartCols(), which is known to be
non-null in org.apache.hadoop.hive.ql.metadata.Partition.getName() Redundant
null check at Partition.java:is known to be non-null in
org.apache.hadoop.hive.ql.metadata.Partition.getName() Redundant null check at
Partition.java:[line 200] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13000/dev-support/hive-personality.sh
|
| git revision | master / 2cabb8d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13000/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13000/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13000/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Expose inputPartitionList in QueryPlan
> --
>
> Key: HIVE-20261
> URL: https://issues.apache.org/jira/browse/HIVE-20261
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Zheng Shao
>Assignee: Zheng Shao
>Priority: Minor
> Attachments: HIVE-20261.1.patch, HIVE-20261.2.patch,
> HIVE-20261.3.patch, HIVE-20261.4.patch
>
>
> Having access to the list of input partitions for all historical Hive queries
> in a system provides a great opportunity to insights on data access frequency
> and potential storage tiering.
> This task aims to expose that via QueryPlan so that a Hive Hook can pick it
> up and store the information for an
[jira] [Updated] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19097:
Attachment: HIVE-19097.07.patch
> related equals and in operators may cause inaccurate stats estimations
> --
>
> Key: HIVE-19097
> URL: https://issues.apache.org/jira/browse/HIVE-19097
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19097.01.patch, HIVE-19097.02.patch,
> HIVE-19097.03.patch, HIVE-19097.04.patch, HIVE-19097.05.patch,
> HIVE-19097.06.patch, HIVE-19097.06wip01.patch, HIVE-19097.06wip02.patch,
> HIVE-19097.07.patch, HIVE-19097.partial.patch
>
>
> tpcds#74 is optimized in a way that for date_dim the condition contains IN
> and = for the same column
> {code:java}
> | Map Operator Tree: |
> | TableScan |
> | alias: date_dim |
> | filterExpr: (((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) or ((d_year) IN (2001, 2002) and (d_year =
> 2001) and d_date_sk is not null)) (type: boolean) |
> | Statistics: Num rows: 73049 Data size: 876588 Basic
> stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator |
> | predicate: ((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) (type: boolean) |
> | Statistics: Num rows: 4 Data size: 48 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
> the "real" row count will be 365
> for separate {{IN}} and {{=}} the estimation is very good; but if both are
> present it becomes (very) underestimated.
> {code:java}
> set hive.query.results.cache.enabled=false;
> drop table if exists t1;
> drop table if exists t8;
> create table t1 (a integer,b integer);
> create table t8 like t1;
> insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5);
> insert into t8
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1 union all
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1
> ;
> analyze table t1 compute statistics for columns;
> analyze table t8 compute statistics for columns;
> explain analyze select sum(a) from t8 where b in (2,3) group by b;
> explain analyze select sum(a) from t8 where b=2 group by b;
> explain analyze select sum(a) from t1 where b in (2,3) and b=2 group by b;
> explain analyze select sum(a) from t8 where b in (2,3) and b=2 group by b;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20121) investigate issues with TestReplicationScenariosAcidTables
[
https://issues.apache.org/jira/browse/HIVE-20121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1659#comment-1659
]
Hive QA commented on HIVE-20121:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/hive-unit: The patch generated 31 new + 85
unchanged - 0 fixed = 116 total (was 85) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13001/dev-support/hive-personality.sh
|
| git revision | master / 5c02fee |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13001/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: itests/hive-unit U: itests/hive-unit |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13001/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> investigate issues with TestReplicationScenariosAcidTables
> --
>
> Key: HIVE-20121
> URL: https://issues.apache.org/jira/browse/HIVE-20121
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-20121.01.patch, HIVE-20121.02.patch,
> HIVE-20121.03.patch
>
>
> [~djaiswal] have noticed that somehow ptest execution are getting more and
> more problems lately... it seems to me that these problems are coming from
> this test
> {code}
> 2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable to
> shutdown metastore client. Will try closing transport directly.
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream
> {code}
> some links to more or less recent logs:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
> the hive.log is ~200M:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20121) investigate issues with TestReplicationScenariosAcidTables
[
https://issues.apache.org/jira/browse/HIVE-20121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566720#comment-16566720
]
Hive QA commented on HIVE-20121:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934026/HIVE-20121.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14854 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13001/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13001/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13001/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934026 - PreCommit-HIVE-Build
> investigate issues with TestReplicationScenariosAcidTables
> --
>
> Key: HIVE-20121
> URL: https://issues.apache.org/jira/browse/HIVE-20121
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-20121.01.patch, HIVE-20121.02.patch,
> HIVE-20121.03.patch
>
>
> [~djaiswal] have noticed that somehow ptest execution are getting more and
> more problems lately... it seems to me that these problems are coming from
> this test
> {code}
> 2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable to
> shutdown metastore client. Will try closing transport directly.
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream
> {code}
> some links to more or less recent logs:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
> the hive.log is ~200M:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19097:
Attachment: HIVE-19097.08.patch
> related equals and in operators may cause inaccurate stats estimations
> --
>
> Key: HIVE-19097
> URL: https://issues.apache.org/jira/browse/HIVE-19097
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19097.01.patch, HIVE-19097.02.patch,
> HIVE-19097.03.patch, HIVE-19097.04.patch, HIVE-19097.05.patch,
> HIVE-19097.06.patch, HIVE-19097.06wip01.patch, HIVE-19097.06wip02.patch,
> HIVE-19097.07.patch, HIVE-19097.08.patch, HIVE-19097.partial.patch
>
>
> tpcds#74 is optimized in a way that for date_dim the condition contains IN
> and = for the same column
> {code:java}
> | Map Operator Tree: |
> | TableScan |
> | alias: date_dim |
> | filterExpr: (((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) or ((d_year) IN (2001, 2002) and (d_year =
> 2001) and d_date_sk is not null)) (type: boolean) |
> | Statistics: Num rows: 73049 Data size: 876588 Basic
> stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator |
> | predicate: ((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) (type: boolean) |
> | Statistics: Num rows: 4 Data size: 48 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
> the "real" row count will be 365
> for separate {{IN}} and {{=}} the estimation is very good; but if both are
> present it becomes (very) underestimated.
> {code:java}
> set hive.query.results.cache.enabled=false;
> drop table if exists t1;
> drop table if exists t8;
> create table t1 (a integer,b integer);
> create table t8 like t1;
> insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5);
> insert into t8
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1 union all
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1
> ;
> analyze table t1 compute statistics for columns;
> analyze table t8 compute statistics for columns;
> explain analyze select sum(a) from t8 where b in (2,3) group by b;
> explain analyze select sum(a) from t8 where b=2 group by b;
> explain analyze select sum(a) from t1 where b in (2,3) and b=2 group by b;
> explain analyze select sum(a) from t8 where b in (2,3) and b=2 group by b;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19097:
Attachment: HIVE-19097.08.patch
> related equals and in operators may cause inaccurate stats estimations
> --
>
> Key: HIVE-19097
> URL: https://issues.apache.org/jira/browse/HIVE-19097
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19097.01.patch, HIVE-19097.02.patch,
> HIVE-19097.03.patch, HIVE-19097.04.patch, HIVE-19097.05.patch,
> HIVE-19097.06.patch, HIVE-19097.06wip01.patch, HIVE-19097.06wip02.patch,
> HIVE-19097.07.patch, HIVE-19097.08.patch, HIVE-19097.08.patch,
> HIVE-19097.partial.patch
>
>
> tpcds#74 is optimized in a way that for date_dim the condition contains IN
> and = for the same column
> {code:java}
> | Map Operator Tree: |
> | TableScan |
> | alias: date_dim |
> | filterExpr: (((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) or ((d_year) IN (2001, 2002) and (d_year =
> 2001) and d_date_sk is not null)) (type: boolean) |
> | Statistics: Num rows: 73049 Data size: 876588 Basic
> stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator |
> | predicate: ((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) (type: boolean) |
> | Statistics: Num rows: 4 Data size: 48 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
> the "real" row count will be 365
> for separate {{IN}} and {{=}} the estimation is very good; but if both are
> present it becomes (very) underestimated.
> {code:java}
> set hive.query.results.cache.enabled=false;
> drop table if exists t1;
> drop table if exists t8;
> create table t1 (a integer,b integer);
> create table t8 like t1;
> insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5);
> insert into t8
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1 union all
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1
> ;
> analyze table t1 compute statistics for columns;
> analyze table t8 compute statistics for columns;
> explain analyze select sum(a) from t8 where b in (2,3) group by b;
> explain analyze select sum(a) from t8 where b=2 group by b;
> explain analyze select sum(a) from t1 where b in (2,3) and b=2 group by b;
> explain analyze select sum(a) from t8 where b in (2,3) and b=2 group by b;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20294) Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
[
https://issues.apache.org/jira/browse/HIVE-20294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566758#comment-16566758
]
Hive QA commented on HIVE-20294:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
20s{color} | {color:blue} ql in master has 2302 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
52s{color} | {color:red} ql: The patch generated 320 new + 645 unchanged - 5
fixed = 965 total (was 650) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
33s{color} | {color:green} ql generated 0 new + 2300 unchanged - 2 fixed = 2300
total (was 2302) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13002/dev-support/hive-personality.sh
|
| git revision | master / 5c02fee |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13002/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13002/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13002/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
>
>
> Key: HIVE-20294
> URL: https://issues.apache.org/jira/browse/HIVE-20294
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20294.01.patch
>
>
> Write new UT tests that use random data and intentional isRepeating batches
> to checks for NULL and Wrong Results for vectorized COALESCE and ELT.
> Also, add tests for ARRAY and MAP indexing, IS [NOT] NULL and NOT
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20297) Column Level Stats for Druid Tables
[ https://issues.apache.org/jira/browse/HIVE-20297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa reassigned HIVE-20297: --- > Column Level Stats for Druid Tables > --- > > Key: HIVE-20297 > URL: https://issues.apache.org/jira/browse/HIVE-20297 > Project: Hive > Issue Type: Improvement >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > > This task is to have correct column level stats for druid in hive metastore. > - Stats like min/max/cardinality can be gathered using a Druid Segment > Metadata Query. > - Druid Query planning we need to ensure that the filters/Aggregations pushed > inside DruidQuery are accounted for. > Having correct stats would also help optimizer ensure proper join orderings > when doing federated complex joins between hive/druid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20294) Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
[
https://issues.apache.org/jira/browse/HIVE-20294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566828#comment-16566828
]
Hive QA commented on HIVE-20294:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934032/HIVE-20294.01.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14846 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_map_type_vectorization]
(batchId=89)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_map_type_vectorization]
(batchId=154)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13002/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13002/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13002/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934032 - PreCommit-HIVE-Build
> Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
>
>
> Key: HIVE-20294
> URL: https://issues.apache.org/jira/browse/HIVE-20294
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20294.01.patch
>
>
> Write new UT tests that use random data and intentional isRepeating batches
> to checks for NULL and Wrong Results for vectorized COALESCE and ELT.
> Also, add tests for ARRAY and MAP indexing, IS [NOT] NULL and NOT
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19097:
Attachment: HIVE-19097.09.patch
> related equals and in operators may cause inaccurate stats estimations
> --
>
> Key: HIVE-19097
> URL: https://issues.apache.org/jira/browse/HIVE-19097
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19097.01.patch, HIVE-19097.02.patch,
> HIVE-19097.03.patch, HIVE-19097.04.patch, HIVE-19097.05.patch,
> HIVE-19097.06.patch, HIVE-19097.06wip01.patch, HIVE-19097.06wip02.patch,
> HIVE-19097.07.patch, HIVE-19097.08.patch, HIVE-19097.08.patch,
> HIVE-19097.09.patch, HIVE-19097.partial.patch
>
>
> tpcds#74 is optimized in a way that for date_dim the condition contains IN
> and = for the same column
> {code:java}
> | Map Operator Tree: |
> | TableScan |
> | alias: date_dim |
> | filterExpr: (((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) or ((d_year) IN (2001, 2002) and (d_year =
> 2001) and d_date_sk is not null)) (type: boolean) |
> | Statistics: Num rows: 73049 Data size: 876588 Basic
> stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator |
> | predicate: ((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) (type: boolean) |
> | Statistics: Num rows: 4 Data size: 48 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
> the "real" row count will be 365
> for separate {{IN}} and {{=}} the estimation is very good; but if both are
> present it becomes (very) underestimated.
> {code:java}
> set hive.query.results.cache.enabled=false;
> drop table if exists t1;
> drop table if exists t8;
> create table t1 (a integer,b integer);
> create table t8 like t1;
> insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5);
> insert into t8
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1 union all
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1
> ;
> analyze table t1 compute statistics for columns;
> analyze table t8 compute statistics for columns;
> explain analyze select sum(a) from t8 where b in (2,3) group by b;
> explain analyze select sum(a) from t8 where b=2 group by b;
> explain analyze select sum(a) from t1 where b in (2,3) and b=2 group by b;
> explain analyze select sum(a) from t8 where b in (2,3) and b=2 group by b;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566863#comment-16566863
]
Hive QA commented on HIVE-19097:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
41s{color} | {color:blue} ql in master has 2302 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
51s{color} | {color:red} ql: The patch generated 12 new + 497 unchanged - 18
fixed = 509 total (was 515) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
35s{color} | {color:red} ql generated 4 new + 2298 unchanged - 4 fixed = 2302
total (was 2302) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretNodeAs(PrimitiveTypeInfo,
ExprNodeDesc) invokes inefficient new Byte(String) constructor; use
Byte.valueOf(String) instead At TypeCheckProcFactory.java:Byte(String)
constructor; use Byte.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1259] |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretNodeAs(PrimitiveTypeInfo,
ExprNodeDesc) invokes inefficient new Integer(String) constructor; use
Integer.valueOf(String) instead At TypeCheckProcFactory.java:Integer(String)
constructor; use Integer.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1251] |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretNodeAs(PrimitiveTypeInfo,
ExprNodeDesc) invokes inefficient new Long(String) constructor; use
Long.valueOf(String) instead At TypeCheckProcFactory.java:Long(String)
constructor; use Long.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1253] |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretNodeAs(PrimitiveTypeInfo,
ExprNodeDesc) invokes inefficient new Short(String) constructor; use
Short.valueOf(String) instead At TypeCheckProcFactory.java:Short(String)
constructor; use Short.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1261] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13003/dev-support/hive-personali
[jira] [Updated] (HIVE-19368) Metastore: log a warning with table-name + partition-count when get_partitions returns >10k partitions
[ https://issues.apache.org/jira/browse/HIVE-19368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated HIVE-19368: Status: Open (was: Patch Available) > Metastore: log a warning with table-name + partition-count when > get_partitions returns >10k partitions > -- > > Key: HIVE-19368 > URL: https://issues.apache.org/jira/browse/HIVE-19368 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Gopal V >Assignee: Anurag Mantripragada >Priority: Major > Attachments: HIVE-19368.1.patch > > > Ran into this particular letter from the trenches & would like a normal WARN > log for it. > https://www.slideshare.net/Hadoop_Summit/hive-at-yahoo-letters-from-the-trenches/24 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19368) Metastore: log a warning with table-name + partition-count when get_partitions returns >10k partitions
[ https://issues.apache.org/jira/browse/HIVE-19368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated HIVE-19368: Attachment: HIVE-19368.1.patch Status: Patch Available (was: Open) > Metastore: log a warning with table-name + partition-count when > get_partitions returns >10k partitions > -- > > Key: HIVE-19368 > URL: https://issues.apache.org/jira/browse/HIVE-19368 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Gopal V >Assignee: Anurag Mantripragada >Priority: Major > Attachments: HIVE-19368.1.patch > > > Ran into this particular letter from the trenches & would like a normal WARN > log for it. > https://www.slideshare.net/Hadoop_Summit/hive-at-yahoo-letters-from-the-trenches/24 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19368) Metastore: log a warning with table-name + partition-count when get_partitions returns >10k partitions
[ https://issues.apache.org/jira/browse/HIVE-19368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated HIVE-19368: Attachment: (was: HIVE-19368.1.patch) > Metastore: log a warning with table-name + partition-count when > get_partitions returns >10k partitions > -- > > Key: HIVE-19368 > URL: https://issues.apache.org/jira/browse/HIVE-19368 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Gopal V >Assignee: Anurag Mantripragada >Priority: Major > Attachments: HIVE-19368.1.patch > > > Ran into this particular letter from the trenches & would like a normal WARN > log for it. > https://www.slideshare.net/Hadoop_Summit/hive-at-yahoo-letters-from-the-trenches/24 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566906#comment-16566906
]
Hive QA commented on HIVE-19097:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934080/HIVE-19097.08.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14845 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_drill]
(batchId=13)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testAddPartitionLocks
(batchId=312)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13003/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13003/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13003/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934080 - PreCommit-HIVE-Build
> related equals and in operators may cause inaccurate stats estimations
> --
>
> Key: HIVE-19097
> URL: https://issues.apache.org/jira/browse/HIVE-19097
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19097.01.patch, HIVE-19097.02.patch,
> HIVE-19097.03.patch, HIVE-19097.04.patch, HIVE-19097.05.patch,
> HIVE-19097.06.patch, HIVE-19097.06wip01.patch, HIVE-19097.06wip02.patch,
> HIVE-19097.07.patch, HIVE-19097.08.patch, HIVE-19097.08.patch,
> HIVE-19097.09.patch, HIVE-19097.partial.patch
>
>
> tpcds#74 is optimized in a way that for date_dim the condition contains IN
> and = for the same column
> {code:java}
> | Map Operator Tree: |
> | TableScan |
> | alias: date_dim |
> | filterExpr: (((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) or ((d_year) IN (2001, 2002) and (d_year =
> 2001) and d_date_sk is not null)) (type: boolean) |
> | Statistics: Num rows: 73049 Data size: 876588 Basic
> stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator |
> | predicate: ((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) (type: boolean) |
> | Statistics: Num rows: 4 Data size: 48 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
> the "real" row count will be 365
> for separate {{IN}} and {{=}} the estimation is very good; but if both are
> present it becomes (very) underestimated.
> {code:java}
> set hive.query.results.cache.enabled=false;
> drop table if exists t1;
> drop table if exists t8;
> create table t1 (a integer,b integer);
> create table t8 like t1;
> insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5);
> insert into t8
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1 union all
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1
> ;
> analyze table t1 compute statistics for columns;
> analyze table t8 compute statistics for columns;
> explain analyze select sum(a) from t8 where b in (2,3) group by b;
> explain analyze select sum(a) from t8 where b=2 group by b;
> explain analyze select sum(a) from t1 where b in (2,3) and b=2 group by b;
> explain analyze select sum(a) from t8 where b in (2,3) and b=2 group by b;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20239) Do Not Print StackTraces to STDERR in MapJoinProcessor
[
https://issues.apache.org/jira/browse/HIVE-20239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566924#comment-16566924
]
Vihang Karajgaonkar commented on HIVE-20239:
+1
> Do Not Print StackTraces to STDERR in MapJoinProcessor
> --
>
> Key: HIVE-20239
> URL: https://issues.apache.org/jira/browse/HIVE-20239
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20239.1.patch, HIVE-20239.2.patch
>
>
> {code:java|title=MapJoinProcessor.java}
> } catch (Exception e) {
> e.printStackTrace();
> throw new SemanticException("Failed to generate new mapJoin operator " +
> "by exception : " + e.getMessage());
> }
> {code}
> Please change to... something like...
> {code}
> } catch (Exception e) {
> throw new SemanticException("Failed to generate new mapJoin operator",
> e);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20166) LazyBinaryStruct Warn Level Logging
[
https://issues.apache.org/jira/browse/HIVE-20166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566932#comment-16566932
]
Vihang Karajgaonkar commented on HIVE-20166:
+1
> LazyBinaryStruct Warn Level Logging
> ---
>
> Key: HIVE-20166
> URL: https://issues.apache.org/jira/browse/HIVE-20166
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: newbie, noob
> Attachments: HIVE-20166.1.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java#L177-L180
> {code}
> // Extra bytes at the end?
> if (!extraFieldWarned && lastFieldByteEnd < structByteEnd) {
> extraFieldWarned = true;
> LOG.warn("Extra bytes detected at the end of the row! " +
>"Last field end " + lastFieldByteEnd + " and serialize buffer end
> " + structByteEnd + ". " +
>"Ignoring similar problems.");
> }
> // Missing fields?
> if (!missingFieldWarned && lastFieldByteEnd > structByteEnd) {
> missingFieldWarned = true;
> LOG.info("Missing fields! Expected " + fields.length + " fields but " +
> "only got " + fieldId + "! " +
> "Last field end " + lastFieldByteEnd + " and serialize buffer end "
> + structByteEnd + ". " +
> "Ignoring similar problems.");
> }
> {code}
> The first log statement is a 'warn' level logging, the second is an 'info'
> level logging. Please change the second log to also be a 'warn'. This seems
> like it could be a problem that the user would like to know about.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566956#comment-16566956
]
Hive QA commented on HIVE-19097:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
45s{color} | {color:blue} ql in master has 2302 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
51s{color} | {color:red} ql: The patch generated 13 new + 614 unchanged - 19
fixed = 627 total (was 633) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
45s{color} | {color:red} ql generated 4 new + 2298 unchanged - 4 fixed = 2302
total (was 2302) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretNodeAs(PrimitiveTypeInfo,
ExprNodeDesc) invokes inefficient new Byte(String) constructor; use
Byte.valueOf(String) instead At TypeCheckProcFactory.java:Byte(String)
constructor; use Byte.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1259] |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretNodeAs(PrimitiveTypeInfo,
ExprNodeDesc) invokes inefficient new Integer(String) constructor; use
Integer.valueOf(String) instead At TypeCheckProcFactory.java:Integer(String)
constructor; use Integer.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1251] |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretNodeAs(PrimitiveTypeInfo,
ExprNodeDesc) invokes inefficient new Long(String) constructor; use
Long.valueOf(String) instead At TypeCheckProcFactory.java:Long(String)
constructor; use Long.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1253] |
| |
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretNodeAs(PrimitiveTypeInfo,
ExprNodeDesc) invokes inefficient new Short(String) constructor; use
Short.valueOf(String) instead At TypeCheckProcFactory.java:Short(String)
constructor; use Short.valueOf(String) instead At
TypeCheckProcFactory.java:[line 1261] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13004/dev-support/hive-personali
[jira] [Commented] (HIVE-20121) investigate issues with TestReplicationScenariosAcidTables
[
https://issues.apache.org/jira/browse/HIVE-20121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566962#comment-16566962
]
mahesh kumar behera commented on HIVE-20121:
[~sankarh]
Please check if this can be committed
> investigate issues with TestReplicationScenariosAcidTables
> --
>
> Key: HIVE-20121
> URL: https://issues.apache.org/jira/browse/HIVE-20121
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-20121.01.patch, HIVE-20121.02.patch,
> HIVE-20121.03.patch
>
>
> [~djaiswal] have noticed that somehow ptest execution are getting more and
> more problems lately... it seems to me that these problems are coming from
> this test
> {code}
> 2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable to
> shutdown metastore client. Will try closing transport directly.
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream
> {code}
> some links to more or less recent logs:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
> the hive.log is ~200M:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20239) Do Not Print StackTraces to STDERR in MapJoinProcessor
[
https://issues.apache.org/jira/browse/HIVE-20239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-20239:
---
Resolution: Fixed
Target Version/s: 4.0.0, 3.2.0
Status: Resolved (was: Patch Available)
Patch merged into master and branch-3. Thanks for your contribution
[~anuragmantri]
> Do Not Print StackTraces to STDERR in MapJoinProcessor
> --
>
> Key: HIVE-20239
> URL: https://issues.apache.org/jira/browse/HIVE-20239
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20239.1.patch, HIVE-20239.2.patch
>
>
> {code:java|title=MapJoinProcessor.java}
> } catch (Exception e) {
> e.printStackTrace();
> throw new SemanticException("Failed to generate new mapJoin operator " +
> "by exception : " + e.getMessage());
> }
> {code}
> Please change to... something like...
> {code}
> } catch (Exception e) {
> throw new SemanticException("Failed to generate new mapJoin operator",
> e);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20291) Allow HiveStreamingConnection to receive a WriteId
[ https://issues.apache.org/jira/browse/HIVE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566979#comment-16566979 ] Eugene Koifman commented on HIVE-20291: --- Is there a design description somewhere that explains what this is for and how it will be used? In particular, RandomStatementIdChooser? If statement id is ever reused this will lead to a data loss. > Allow HiveStreamingConnection to receive a WriteId > -- > > Key: HIVE-20291 > URL: https://issues.apache.org/jira/browse/HIVE-20291 > Project: Hive > Issue Type: Improvement >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20291.1.patch > > > If the writeId is received externally it won't need to open connections to > the metastore. It won't be able to the commit in this case as well so it must > be done by the entity passing the writeId. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566984#comment-16566984
]
Hive QA commented on HIVE-19097:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934087/HIVE-19097.09.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 14845 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_drill]
(batchId=13)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query53]
(batchId=265)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query63]
(batchId=265)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query89]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query53]
(batchId=263)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query89]
(batchId=263)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13004/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13004/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13004/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934087 - PreCommit-HIVE-Build
> related equals and in operators may cause inaccurate stats estimations
> --
>
> Key: HIVE-19097
> URL: https://issues.apache.org/jira/browse/HIVE-19097
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19097.01.patch, HIVE-19097.02.patch,
> HIVE-19097.03.patch, HIVE-19097.04.patch, HIVE-19097.05.patch,
> HIVE-19097.06.patch, HIVE-19097.06wip01.patch, HIVE-19097.06wip02.patch,
> HIVE-19097.07.patch, HIVE-19097.08.patch, HIVE-19097.08.patch,
> HIVE-19097.09.patch, HIVE-19097.partial.patch
>
>
> tpcds#74 is optimized in a way that for date_dim the condition contains IN
> and = for the same column
> {code:java}
> | Map Operator Tree: |
> | TableScan |
> | alias: date_dim |
> | filterExpr: (((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) or ((d_year) IN (2001, 2002) and (d_year =
> 2001) and d_date_sk is not null)) (type: boolean) |
> | Statistics: Num rows: 73049 Data size: 876588 Basic
> stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator |
> | predicate: ((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) (type: boolean) |
> | Statistics: Num rows: 4 Data size: 48 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
> the "real" row count will be 365
> for separate {{IN}} and {{=}} the estimation is very good; but if both are
> present it becomes (very) underestimated.
> {code:java}
> set hive.query.results.cache.enabled=false;
> drop table if exists t1;
> drop table if exists t8;
> create table t1 (a integer,b integer);
> create table t8 like t1;
> insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5);
> insert into t8
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1 union all
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1
> ;
> analyze table t1 compute statistics for columns;
> analyze table t8 compute statistics for columns;
> explain analyze select sum(a) from t8 where b in (2,3) group by b;
> explain analyze select sum(a) from t8 where b=2 group by b;
> explain analyze select sum(a) from t1 where b in (2,3) and b=2 group by b;
> explain analyze select sum(a) from t8 where b in (2,3) and b=2 group by b;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566986#comment-16566986
]
Hive QA commented on HIVE-19097:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934087/HIVE-19097.09.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13005/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13005/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13005/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12934087/HIVE-19097.09.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934087 - PreCommit-HIVE-Build
> related equals and in operators may cause inaccurate stats estimations
> --
>
> Key: HIVE-19097
> URL: https://issues.apache.org/jira/browse/HIVE-19097
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19097.01.patch, HIVE-19097.02.patch,
> HIVE-19097.03.patch, HIVE-19097.04.patch, HIVE-19097.05.patch,
> HIVE-19097.06.patch, HIVE-19097.06wip01.patch, HIVE-19097.06wip02.patch,
> HIVE-19097.07.patch, HIVE-19097.08.patch, HIVE-19097.08.patch,
> HIVE-19097.09.patch, HIVE-19097.partial.patch
>
>
> tpcds#74 is optimized in a way that for date_dim the condition contains IN
> and = for the same column
> {code:java}
> | Map Operator Tree: |
> | TableScan |
> | alias: date_dim |
> | filterExpr: (((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) or ((d_year) IN (2001, 2002) and (d_year =
> 2001) and d_date_sk is not null)) (type: boolean) |
> | Statistics: Num rows: 73049 Data size: 876588 Basic
> stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator |
> | predicate: ((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) (type: boolean) |
> | Statistics: Num rows: 4 Data size: 48 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
> the "real" row count will be 365
> for separate {{IN}} and {{=}} the estimation is very good; but if both are
> present it becomes (very) underestimated.
> {code:java}
> set hive.query.results.cache.enabled=false;
> drop table if exists t1;
> drop table if exists t8;
> create table t1 (a integer,b integer);
> create table t8 like t1;
> insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5);
> insert into t8
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1 union all
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1
> ;
> analyze table t1 compute statistics for columns;
> analyze table t8 compute statistics for columns;
> explain analyze select sum(a) from t8 where b in (2,3) group by b;
> explain analyze select sum(a) from t8 where b=2 group by b;
> explain analyze select sum(a) from t1 where b in (2,3) and b=2 group by b;
> explain analyze select sum(a) from t8 where b in (2,3) and b=2 group by b;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566988#comment-16566988
]
Hive QA commented on HIVE-19097:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934087/HIVE-19097.09.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13006/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13006/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13006/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12934087/HIVE-19097.09.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934087 - PreCommit-HIVE-Build
> related equals and in operators may cause inaccurate stats estimations
> --
>
> Key: HIVE-19097
> URL: https://issues.apache.org/jira/browse/HIVE-19097
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19097.01.patch, HIVE-19097.02.patch,
> HIVE-19097.03.patch, HIVE-19097.04.patch, HIVE-19097.05.patch,
> HIVE-19097.06.patch, HIVE-19097.06wip01.patch, HIVE-19097.06wip02.patch,
> HIVE-19097.07.patch, HIVE-19097.08.patch, HIVE-19097.08.patch,
> HIVE-19097.09.patch, HIVE-19097.partial.patch
>
>
> tpcds#74 is optimized in a way that for date_dim the condition contains IN
> and = for the same column
> {code:java}
> | Map Operator Tree: |
> | TableScan |
> | alias: date_dim |
> | filterExpr: (((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) or ((d_year) IN (2001, 2002) and (d_year =
> 2001) and d_date_sk is not null)) (type: boolean) |
> | Statistics: Num rows: 73049 Data size: 876588 Basic
> stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator |
> | predicate: ((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) (type: boolean) |
> | Statistics: Num rows: 4 Data size: 48 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
> the "real" row count will be 365
> for separate {{IN}} and {{=}} the estimation is very good; but if both are
> present it becomes (very) underestimated.
> {code:java}
> set hive.query.results.cache.enabled=false;
> drop table if exists t1;
> drop table if exists t8;
> create table t1 (a integer,b integer);
> create table t8 like t1;
> insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5);
> insert into t8
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1 union all
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1
> ;
> analyze table t1 compute statistics for columns;
> analyze table t8 compute statistics for columns;
> explain analyze select sum(a) from t8 where b in (2,3) group by b;
> explain analyze select sum(a) from t8 where b=2 group by b;
> explain analyze select sum(a) from t1 where b in (2,3) and b=2 group by b;
> explain analyze select sum(a) from t8 where b in (2,3) and b=2 group by b;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20281) SharedWorkOptimizer fails with 'operator cache contents and actual plan differ'
[
https://issues.apache.org/jira/browse/HIVE-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567013#comment-16567013
]
Jesus Camacho Rodriguez commented on HIVE-20281:
[~ashutoshc], can you take a look?
The problem was that when we are trying to merge two (sub)trees and we are
gathering the operators that we need to remove, these are divided into two
sets: {{discardableOps}} and {{discardableInputOps}}. The former gathers the
operators that we are traversing while checking, while the latter gathers the
inputs to those operators (obviously it also checks whether those inputs are
the same). This distinction is useful later on when we actually perform the
merge operation. {{discardableInputOps}} should not include {{discardableOps}}.
However, for extended shared work optimizer I had introduced a boolean that
does exactly that. Because we have those duplicate operators, we end up with
inconsistent state that leads to additional operators in the cache (plan is
still correct btw, though I am not sure whether this could lead to incorrect
plan in some cases). Looking back at the code, it does not make sense to have
that boolean / distinction, I think maybe I made the assumption while coding
that I needed to keep them in both.
> SharedWorkOptimizer fails with 'operator cache contents and actual plan
> differ'
> ---
>
> Key: HIVE-20281
> URL: https://issues.apache.org/jira/browse/HIVE-20281
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Ashutosh Chauhan
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-20281.patch
>
>
> HIVE-18201 seems to trigger a latent bug in SW optimizer. Test
> {{subquery_in_having}} fails with:
> {code}
> 2018-07-31T08:42:57,328 DEBUG [b68f20cc-54d5-466d-b512-1540b3a43396 main]
> optimizer.SharedWorkOptimizer: After SharedWorkExtendedOptimizer:
> TS[0]-SEL[1]-MAPJOIN[131]-FIL[12]-SEL[13]-GBY[14]-RS[15]-GBY[16]-SEL[17]-MAPJOIN[136]-MAPJOIN[137]-FIL[103]-SEL[104]-FS[105]
>
> -FIL[113]-SEL[20]-RS[44]-MAPJOIN[133]-SEL[47]-GBY[48]-RS[49]-GBY[50]-SEL[51]-GBY[55]-RS[98]-MAPJOIN[136]
>
> -RS[88]-GBY[89]-SEL[120]-FIL[116]-SEL[91]-GBY[93]-RS[94]-GBY[95]-SEL[96]-RS[101]-MAPJOIN[137]
> TS[2]-FIL[112]-GBY[5]-RS[6]-GBY[7]-SEL[8]-RS[10]-MAPJOIN[131]
>
> -RS[31]-MAPJOIN[132]-FIL[33]-SEL[34]-GBY[35]-RS[36]-GBY[37]-SEL[38]-GBY[42]-MAPJOIN[133]
> TS[21]-FIL[114]-SEL[22]-MAPJOIN[132]
> 2018-07-31T08:42:57,329 ERROR [b68f20cc-54d5-466d-b512-1540b3a43396 main]
> ql.Driver: FAILED: SemanticException Error in shared work optimizer: operator
> cache contentsand actual plan differ
> org.apache.hadoop.hive.ql.parse.SemanticException: Error in shared work
> optimizer: operator cache contentsand actual plan differ
> at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.transform(SharedWorkOptimizer.java:524)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:185)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:146)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12361)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:356)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:165)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:663)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20281) SharedWorkOptimizer fails with 'operator cache contents and actual plan differ'
[
https://issues.apache.org/jira/browse/HIVE-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567016#comment-16567016
]
Ashutosh Chauhan commented on HIVE-20281:
-
+1
> SharedWorkOptimizer fails with 'operator cache contents and actual plan
> differ'
> ---
>
> Key: HIVE-20281
> URL: https://issues.apache.org/jira/browse/HIVE-20281
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Ashutosh Chauhan
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-20281.patch
>
>
> HIVE-18201 seems to trigger a latent bug in SW optimizer. Test
> {{subquery_in_having}} fails with:
> {code}
> 2018-07-31T08:42:57,328 DEBUG [b68f20cc-54d5-466d-b512-1540b3a43396 main]
> optimizer.SharedWorkOptimizer: After SharedWorkExtendedOptimizer:
> TS[0]-SEL[1]-MAPJOIN[131]-FIL[12]-SEL[13]-GBY[14]-RS[15]-GBY[16]-SEL[17]-MAPJOIN[136]-MAPJOIN[137]-FIL[103]-SEL[104]-FS[105]
>
> -FIL[113]-SEL[20]-RS[44]-MAPJOIN[133]-SEL[47]-GBY[48]-RS[49]-GBY[50]-SEL[51]-GBY[55]-RS[98]-MAPJOIN[136]
>
> -RS[88]-GBY[89]-SEL[120]-FIL[116]-SEL[91]-GBY[93]-RS[94]-GBY[95]-SEL[96]-RS[101]-MAPJOIN[137]
> TS[2]-FIL[112]-GBY[5]-RS[6]-GBY[7]-SEL[8]-RS[10]-MAPJOIN[131]
>
> -RS[31]-MAPJOIN[132]-FIL[33]-SEL[34]-GBY[35]-RS[36]-GBY[37]-SEL[38]-GBY[42]-MAPJOIN[133]
> TS[21]-FIL[114]-SEL[22]-MAPJOIN[132]
> 2018-07-31T08:42:57,329 ERROR [b68f20cc-54d5-466d-b512-1540b3a43396 main]
> ql.Driver: FAILED: SemanticException Error in shared work optimizer: operator
> cache contentsand actual plan differ
> org.apache.hadoop.hive.ql.parse.SemanticException: Error in shared work
> optimizer: operator cache contentsand actual plan differ
> at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.transform(SharedWorkOptimizer.java:524)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:185)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:146)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12361)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:356)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:165)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:663)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19368) Metastore: log a warning with table-name + partition-count when get_partitions returns >10k partitions
[
https://issues.apache.org/jira/browse/HIVE-19368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567021#comment-16567021
]
Hive QA commented on HIVE-19368:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} metastore-server in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13007/dev-support/hive-personality.sh
|
| git revision | master / 6917821 |
| Default Java | 1.8.0_111 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13007/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13007/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13007/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Metastore: log a warning with table-name + partition-count when
> get_partitions returns >10k partitions
> --
>
> Key: HIVE-19368
> URL: https://issues.apache.org/jira/browse/HIVE-19368
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.0
>Reporter: Gopal V
>Assignee: Anurag Mantripragada
>Priority: Major
> Attachments: HIVE-19368.1.patch
>
>
> Ran into this particular letter from the trenches & would like a normal WARN
> log for it.
> https://www.slideshare.net/Hadoop_Summit/hive-at-yahoo-letters-from-the-trenches/24
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20121) investigate issues with TestReplicationScenariosAcidTables
[
https://issues.apache.org/jira/browse/HIVE-20121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567030#comment-16567030
]
Sankar Hariappan commented on HIVE-20121:
-
+1
> investigate issues with TestReplicationScenariosAcidTables
> --
>
> Key: HIVE-20121
> URL: https://issues.apache.org/jira/browse/HIVE-20121
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-20121.01.patch, HIVE-20121.02.patch,
> HIVE-20121.03.patch
>
>
> [~djaiswal] have noticed that somehow ptest execution are getting more and
> more problems lately... it seems to me that these problems are coming from
> this test
> {code}
> 2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable to
> shutdown metastore client. Will try closing transport directly.
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream
> {code}
> some links to more or less recent logs:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
> the hive.log is ~200M:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20121) investigate issues with TestReplicationScenariosAcidTables
[
https://issues.apache.org/jira/browse/HIVE-20121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20121:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> investigate issues with TestReplicationScenariosAcidTables
> --
>
> Key: HIVE-20121
> URL: https://issues.apache.org/jira/browse/HIVE-20121
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20121.01.patch, HIVE-20121.02.patch,
> HIVE-20121.03.patch
>
>
> [~djaiswal] have noticed that somehow ptest execution are getting more and
> more problems lately... it seems to me that these problems are coming from
> this test
> {code}
> 2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable to
> shutdown metastore client. Will try closing transport directly.
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream
> {code}
> some links to more or less recent logs:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
> the hive.log is ~200M:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20121) investigate issues with TestReplicationScenariosAcidTables
[
https://issues.apache.org/jira/browse/HIVE-20121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567033#comment-16567033
]
Sankar Hariappan commented on HIVE-20121:
-
03.patch committed to master!
> investigate issues with TestReplicationScenariosAcidTables
> --
>
> Key: HIVE-20121
> URL: https://issues.apache.org/jira/browse/HIVE-20121
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20121.01.patch, HIVE-20121.02.patch,
> HIVE-20121.03.patch
>
>
> [~djaiswal] have noticed that somehow ptest execution are getting more and
> more problems lately... it seems to me that these problems are coming from
> this test
> {code}
> 2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable to
> shutdown metastore client. Will try closing transport directly.
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream
> {code}
> some links to more or less recent logs:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
> the hive.log is ~200M:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20121) investigate issues with TestReplicationScenariosAcidTables
[
https://issues.apache.org/jira/browse/HIVE-20121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20121:
Fix Version/s: 4.0.0
> investigate issues with TestReplicationScenariosAcidTables
> --
>
> Key: HIVE-20121
> URL: https://issues.apache.org/jira/browse/HIVE-20121
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20121.01.patch, HIVE-20121.02.patch,
> HIVE-20121.03.patch
>
>
> [~djaiswal] have noticed that somehow ptest execution are getting more and
> more problems lately... it seems to me that these problems are coming from
> this test
> {code}
> 2018-07-08T22:07:33,461 DEBUG [main] metastore.HiveMetaStoreClient: Unable to
> shutdown metastore client. Will try closing transport directly.
> org.apache.thrift.transport.TTransportException: Cannot write to null
> outputStream
> {code}
> some links to more or less recent logs:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/maven-test.txt
> the hive.log is ~200M:
> http://104.198.109.242/logs/PreCommit-HIVE-Build-12481/failed/240_UTBatch_itests__hive-unit_9_tests/logs/hive.log
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20281) SharedWorkOptimizer fails with 'operator cache contents and actual plan differ'
[
https://issues.apache.org/jira/browse/HIVE-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20281:
---
Resolution: Fixed
Fix Version/s: 3.2.0
4.0.0
Status: Resolved (was: Patch Available)
Pushed to master, branch-3. Thanks for reviewing [~ashutoshc]
> SharedWorkOptimizer fails with 'operator cache contents and actual plan
> differ'
> ---
>
> Key: HIVE-20281
> URL: https://issues.apache.org/jira/browse/HIVE-20281
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Ashutosh Chauhan
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20281.patch
>
>
> HIVE-18201 seems to trigger a latent bug in SW optimizer. Test
> {{subquery_in_having}} fails with:
> {code}
> 2018-07-31T08:42:57,328 DEBUG [b68f20cc-54d5-466d-b512-1540b3a43396 main]
> optimizer.SharedWorkOptimizer: After SharedWorkExtendedOptimizer:
> TS[0]-SEL[1]-MAPJOIN[131]-FIL[12]-SEL[13]-GBY[14]-RS[15]-GBY[16]-SEL[17]-MAPJOIN[136]-MAPJOIN[137]-FIL[103]-SEL[104]-FS[105]
>
> -FIL[113]-SEL[20]-RS[44]-MAPJOIN[133]-SEL[47]-GBY[48]-RS[49]-GBY[50]-SEL[51]-GBY[55]-RS[98]-MAPJOIN[136]
>
> -RS[88]-GBY[89]-SEL[120]-FIL[116]-SEL[91]-GBY[93]-RS[94]-GBY[95]-SEL[96]-RS[101]-MAPJOIN[137]
> TS[2]-FIL[112]-GBY[5]-RS[6]-GBY[7]-SEL[8]-RS[10]-MAPJOIN[131]
>
> -RS[31]-MAPJOIN[132]-FIL[33]-SEL[34]-GBY[35]-RS[36]-GBY[37]-SEL[38]-GBY[42]-MAPJOIN[133]
> TS[21]-FIL[114]-SEL[22]-MAPJOIN[132]
> 2018-07-31T08:42:57,329 ERROR [b68f20cc-54d5-466d-b512-1540b3a43396 main]
> ql.Driver: FAILED: SemanticException Error in shared work optimizer: operator
> cache contentsand actual plan differ
> org.apache.hadoop.hive.ql.parse.SemanticException: Error in shared work
> optimizer: operator cache contentsand actual plan differ
> at
> org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.transform(SharedWorkOptimizer.java:524)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:185)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:146)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12361)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:356)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:165)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:663)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20292) Bad join ordering in tpcds query93 with primary constraint defined
[ https://issues.apache.org/jira/browse/HIVE-20292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20292: --- Attachment: HIVE-20292.2.patch > Bad join ordering in tpcds query93 with primary constraint defined > -- > > Key: HIVE-20292 > URL: https://issues.apache.org/jira/browse/HIVE-20292 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20292.1.patch, HIVE-20292.2.patch > > > Query 93 has join (including outer) b/w store_sales, store_return and reason. > Without constraints store_return is joined with reason and then with > store_sales. > But if a primary key is added on store_return (alter table store_returns add > constraint tpcds_pk_sr primary key (sr_item_sk, sr_ticket_number) disable > novalidate rely) join order becomes ((store_sales, store_return), reason) > which is very inefficient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20292) Bad join ordering in tpcds query93 with primary constraint defined
[ https://issues.apache.org/jira/browse/HIVE-20292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20292: --- Status: Open (was: Patch Available) > Bad join ordering in tpcds query93 with primary constraint defined > -- > > Key: HIVE-20292 > URL: https://issues.apache.org/jira/browse/HIVE-20292 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20292.1.patch, HIVE-20292.2.patch > > > Query 93 has join (including outer) b/w store_sales, store_return and reason. > Without constraints store_return is joined with reason and then with > store_sales. > But if a primary key is added on store_return (alter table store_returns add > constraint tpcds_pk_sr primary key (sr_item_sk, sr_ticket_number) disable > novalidate rely) join order becomes ((store_sales, store_return), reason) > which is very inefficient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20292) Bad join ordering in tpcds query93 with primary constraint defined
[ https://issues.apache.org/jira/browse/HIVE-20292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20292: --- Status: Patch Available (was: Open) > Bad join ordering in tpcds query93 with primary constraint defined > -- > > Key: HIVE-20292 > URL: https://issues.apache.org/jira/browse/HIVE-20292 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20292.1.patch, HIVE-20292.2.patch > > > Query 93 has join (including outer) b/w store_sales, store_return and reason. > Without constraints store_return is joined with reason and then with > store_sales. > But if a primary key is added on store_return (alter table store_returns add > constraint tpcds_pk_sr primary key (sr_item_sk, sr_ticket_number) disable > novalidate rely) join order becomes ((store_sales, store_return), reason) > which is very inefficient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19368) Metastore: log a warning with table-name + partition-count when get_partitions returns >10k partitions
[
https://issues.apache.org/jira/browse/HIVE-19368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567080#comment-16567080
]
Hive QA commented on HIVE-19368:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934088/HIVE-19368.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 41 failed/errored test(s), 14842 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testACIDwithSchemaEvolutionAndCompaction
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testAcidOrcWritePreservesFieldNames
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testAcidWithSchemaEvolution
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testAlterTable
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testBucketCodec
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testBucketizedInputFormat
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testCleanerForTxnToWriteId
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testCompactWithDelete
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testDeleteIn
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testDynamicPartitionsMerge
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testDynamicPartitionsMerge2
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testETLSplitStrategyForACID
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testEmptyInTblproperties
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testFailHeartbeater
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testFileSystemUnCaching
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInitiatorWithMultipleFailedCompactions
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInsertOverwrite1
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInsertOverwrite2
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInsertOverwriteWithSelfJoin
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testMerge
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testMerge2
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testMerge3
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testMergeWithPredicate
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testMmTableCompaction
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testMultiInsert
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testMultiInsertStatement
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNoHistory
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidInsert
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion02
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion2
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testOpenTxnsCounter
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testOrcNoPPD
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testOrcPPD
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testOriginalFileReaderWhenNonAcidConvertedToAcid
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testSimpleRead
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testUpdateMixedCase
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testValidTxnsBookkeeping
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpd
[jira] [Updated] (HIVE-20293) Support Replication of ACID table truncate operation
[ https://issues.apache.org/jira/browse/HIVE-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20293: -- Component/s: Transactions > Support Replication of ACID table truncate operation > > > Key: HIVE-20293 > URL: https://issues.apache.org/jira/browse/HIVE-20293 > Project: Hive > Issue Type: Task > Components: repl, Transactions >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, replication > Fix For: 4.0.0, 3.2.0 > > > Support truncate acid table replication. > 1. Write id allocation needs to be removed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20169) Print Final Rows Processed in MapOperator
[
https://issues.apache.org/jira/browse/HIVE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-20169:
---
Resolution: Fixed
Fix Version/s: 3.2.0
4.0.0
Status: Resolved (was: Patch Available)
Patch merged into master and branch-3. Thanks [~bharos92]
> Print Final Rows Processed in MapOperator
> -
>
> Key: HIVE-20169
> URL: https://issues.apache.org/jira/browse/HIVE-20169
> Project: Hive
> Issue Type: Improvement
> Components: Operators
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20169.1.patch, HIVE-20169.2.patch,
> HIVE-20169.3.patch, HIVE-20169.4.patch
>
>
> https://github.com/apache/hive/blob/ac6b2a3fb195916e22b2e5f465add2ffbcdc7430/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java#L573-L582
> This class emits a log message every time it a certain number of records are
> processed, but it does not print a final count.
> Overload the {{MapOperator}} class's {{closeOp}} method to print a final log
> message providing the total number of rows read by this mapper.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20166) LazyBinaryStruct Warn Level Logging
[
https://issues.apache.org/jira/browse/HIVE-20166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-20166:
---
Resolution: Fixed
Fix Version/s: 3.2.0
4.0.0
Status: Resolved (was: Patch Available)
Patch merged into master and branch-3. Thanks for your contribution
[~anuragmantri]
> LazyBinaryStruct Warn Level Logging
> ---
>
> Key: HIVE-20166
> URL: https://issues.apache.org/jira/browse/HIVE-20166
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20166.1.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java#L177-L180
> {code}
> // Extra bytes at the end?
> if (!extraFieldWarned && lastFieldByteEnd < structByteEnd) {
> extraFieldWarned = true;
> LOG.warn("Extra bytes detected at the end of the row! " +
>"Last field end " + lastFieldByteEnd + " and serialize buffer end
> " + structByteEnd + ". " +
>"Ignoring similar problems.");
> }
> // Missing fields?
> if (!missingFieldWarned && lastFieldByteEnd > structByteEnd) {
> missingFieldWarned = true;
> LOG.info("Missing fields! Expected " + fields.length + " fields but " +
> "only got " + fieldId + "! " +
> "Last field end " + lastFieldByteEnd + " and serialize buffer end "
> + structByteEnd + ". " +
> "Ignoring similar problems.");
> }
> {code}
> The first log statement is a 'warn' level logging, the second is an 'info'
> level logging. Please change the second log to also be a 'warn'. This seems
> like it could be a problem that the user would like to know about.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20162) Do Not Print StackTraces to STDERR in AbstractJoinTaskDispatcher
[
https://issues.apache.org/jira/browse/HIVE-20162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567290#comment-16567290
]
Anurag Mantripragada commented on HIVE-20162:
-
[~ngangam] Could you please commit this? Thanks!
> Do Not Print StackTraces to STDERR in AbstractJoinTaskDispatcher
>
>
> Key: HIVE-20162
> URL: https://issues.apache.org/jira/browse/HIVE-20162
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Anurag Mantripragada
>Priority: Minor
> Labels: newbie, noob
> Attachments: HIVE-20162.1.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/AbstractJoinTaskDispatcher.java
> {code}
> } catch (Exception e) {
> e.printStackTrace();
> throw new SemanticException("Generate Map Join Task Error: " +
> e.getMessage());
> }
> {code}
> Remove the call to {{printStackTrace}} and just throw the error. If the
> stack trace really is needed (doubtful), then pass it to the
> {{SemanticException}} constructor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19368) Metastore: log a warning with table-name + partition-count when get_partitions returns >10k partitions
[
https://issues.apache.org/jira/browse/HIVE-19368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567295#comment-16567295
]
Vihang Karajgaonkar commented on HIVE-19368:
Test failures are not related. Looks like some other patch broke this test.
Couple of comments on the patch.
1. Can you create private static method which takes in the size of partitionIds
and reuse the method at both the places?
2. Also I think it would be useful to add the database name and API name to the
message as well. So you probably will have to pass them too. Consider using the
following msg which I think is more useful {{Log.warn("{} API is processing {}
partitions for {}.{}. Consider using a filter to reduce the number of
partitions scanned.", methodname, databaseName, tableName)}}
> Metastore: log a warning with table-name + partition-count when
> get_partitions returns >10k partitions
> --
>
> Key: HIVE-19368
> URL: https://issues.apache.org/jira/browse/HIVE-19368
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.1.0
>Reporter: Gopal V
>Assignee: Anurag Mantripragada
>Priority: Major
> Attachments: HIVE-19368.1.patch
>
>
> Ran into this particular letter from the trenches & would like a normal WARN
> log for it.
> https://www.slideshare.net/Hadoop_Summit/hive-at-yahoo-letters-from-the-trenches/24
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20292) Bad join ordering in tpcds query93 with primary constraint defined
[ https://issues.apache.org/jira/browse/HIVE-20292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20292: --- Status: Open (was: Patch Available) > Bad join ordering in tpcds query93 with primary constraint defined > -- > > Key: HIVE-20292 > URL: https://issues.apache.org/jira/browse/HIVE-20292 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20292.1.patch, HIVE-20292.2.patch, > HIVE-20292.3.patch > > > Query 93 has join (including outer) b/w store_sales, store_return and reason. > Without constraints store_return is joined with reason and then with > store_sales. > But if a primary key is added on store_return (alter table store_returns add > constraint tpcds_pk_sr primary key (sr_item_sk, sr_ticket_number) disable > novalidate rely) join order becomes ((store_sales, store_return), reason) > which is very inefficient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20292) Bad join ordering in tpcds query93 with primary constraint defined
[ https://issues.apache.org/jira/browse/HIVE-20292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20292: --- Status: Patch Available (was: Open) > Bad join ordering in tpcds query93 with primary constraint defined > -- > > Key: HIVE-20292 > URL: https://issues.apache.org/jira/browse/HIVE-20292 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20292.1.patch, HIVE-20292.2.patch, > HIVE-20292.3.patch > > > Query 93 has join (including outer) b/w store_sales, store_return and reason. > Without constraints store_return is joined with reason and then with > store_sales. > But if a primary key is added on store_return (alter table store_returns add > constraint tpcds_pk_sr primary key (sr_item_sk, sr_ticket_number) disable > novalidate rely) join order becomes ((store_sales, store_return), reason) > which is very inefficient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20292) Bad join ordering in tpcds query93 with primary constraint defined
[ https://issues.apache.org/jira/browse/HIVE-20292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20292: --- Attachment: HIVE-20292.3.patch > Bad join ordering in tpcds query93 with primary constraint defined > -- > > Key: HIVE-20292 > URL: https://issues.apache.org/jira/browse/HIVE-20292 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20292.1.patch, HIVE-20292.2.patch, > HIVE-20292.3.patch > > > Query 93 has join (including outer) b/w store_sales, store_return and reason. > Without constraints store_return is joined with reason and then with > store_sales. > But if a primary key is added on store_return (alter table store_returns add > constraint tpcds_pk_sr primary key (sr_item_sk, sr_ticket_number) disable > novalidate rely) join order becomes ((store_sales, store_return), reason) > which is very inefficient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20298:
--
Attachment: HIVE-20298.1.patch
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Priority: Major
> Attachments: HIVE-20298.1.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
> at
> org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.
[jira] [Commented] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567302#comment-16567302
]
Jason Dere commented on HIVE-20298:
---
Attaching a patch that got around the problem for me. [~sershe] [~ekoifman]
does this look ok, or does WRITE_ID require a default value when the column is
added to the TBLS metastore table?
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Priority: Major
> Attachments: HIVE-20298.1.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
>
[jira] [Commented] (HIVE-20291) Allow HiveStreamingConnection to receive a WriteId
[
https://issues.apache.org/jira/browse/HIVE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567304#comment-16567304
]
Jaume M commented on HIVE-20291:
This intent of using this is in a similar manner to the added test. The
HiveStreamingConnection would receive a writeId instead of figuring it out by
itself. Several HiveStreamingConnection may receive the same writeId and this
seems to result in an error for all except the first HiveStreamingConnection,
therefore the statementId was added.
We can probably remove the StatementIdChooser part and directly pass a the
statementId to the Writer. What do you think of this [~ekoifman]?
bq. If statement id is ever reused this will lead to a data loss.
Wouldn't this lead to an exception in the {{OrcRecordUpdater}} saying that the
file already exists?
> Allow HiveStreamingConnection to receive a WriteId
> --
>
> Key: HIVE-20291
> URL: https://issues.apache.org/jira/browse/HIVE-20291
> Project: Hive
> Issue Type: Improvement
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20291.1.patch
>
>
> If the writeId is received externally it won't need to open connections to
> the metastore. It won't be able to the commit in this case as well so it must
> be done by the entity passing the writeId.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20292) Bad join ordering in tpcds query93 with primary constraint defined
[
https://issues.apache.org/jira/browse/HIVE-20292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567312#comment-16567312
]
Hive QA commented on HIVE-20292:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934102/HIVE-20292.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 36 failed/errored test(s), 14854 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer5]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join28] (batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_subquery2]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_subquery]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_repeated_alias]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[runtime_skewjoin_mapjoin_spark]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_25]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join6]
(batchId=44)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[column_access_stats]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer2]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer3]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_sw]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join32_lessSize]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb_ptf]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_self_join]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_nested_mapjoin]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_use_op_stats]
(batchId=186)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=147)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[column_access_stats]
(batchId=134)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join28]
(batchId=146)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join32_lessSize]
(batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin_subquery2]
(batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin_subquery]
(batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[runtime_skewjoin_mapjoin_spark]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_25]
(batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_nested_mapjoin]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query2]
(batchId=265)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query59]
(batchId=265)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query95]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query2]
(batchId=263)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query59]
(batchId=263)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query93]
(batchId=263)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query95]
(batchId=263)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13008/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13008/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13008/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 36 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934102 - PreCommit-HIVE-Build
> Bad join ordering in tpcds query93 with primary constraint defined
> --
>
> Key: HIVE-20292
> URL: https://issues.apache.org/jira/browse/HIVE-20292
> Project: Hive
> Issue Type: Improvement
> Components: Query Plan
[jira] [Commented] (HIVE-19443) Issue with Druid timestamp with timezone handling
[
https://issues.apache.org/jira/browse/HIVE-19443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567356#comment-16567356
]
Jesus Camacho Rodriguez commented on HIVE-19443:
This test has already been added as {{druid_timestamptz2.q}}.
> Issue with Druid timestamp with timezone handling
> -
>
> Key: HIVE-19443
> URL: https://issues.apache.org/jira/browse/HIVE-19443
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: test_resutls.out, test_timestamp.q
>
>
> As you can see at the attached file [^test_resutls.out] when switching
> current timezone to UTC the insert of values from Hive table into Druid table
> does miss some rows.
> You can use this to reproduce it.
> [^test_timestamp.q]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19443) Issue with Druid timestamp with timezone handling
[ https://issues.apache.org/jira/browse/HIVE-19443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-19443. Resolution: Duplicate > Issue with Druid timestamp with timezone handling > - > > Key: HIVE-19443 > URL: https://issues.apache.org/jira/browse/HIVE-19443 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: test_resutls.out, test_timestamp.q > > > As you can see at the attached file [^test_resutls.out] when switching > current timezone to UTC the insert of values from Hive table into Druid table > does miss some rows. > You can use this to reproduce it. > [^test_timestamp.q] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19949) Clean up logic to check locks in Driver.isValidTxnListState
[ https://issues.apache.org/jira/browse/HIVE-19949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-19949: -- Assignee: (was: Jesus Camacho Rodriguez) > Clean up logic to check locks in Driver.isValidTxnListState > --- > > Key: HIVE-19949 > URL: https://issues.apache.org/jira/browse/HIVE-19949 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0, 4.0.0 >Reporter: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-19949.01.patch > > > Follow-up for HIVE-19876. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18620) Update materialized views when dropping its source tables
[ https://issues.apache.org/jira/browse/HIVE-18620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567361#comment-16567361 ] Jesus Camacho Rodriguez commented on HIVE-18620: [~mgergely], were you taking care of this? Please let me know whether that is the case, and if there is a new issue for it or you would be taking over this one. Thanks > Update materialized views when dropping its source tables > - > > Key: HIVE-18620 > URL: https://issues.apache.org/jira/browse/HIVE-18620 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > > When we want to drop a table used by a materialized view, we should prevent > dropping that table unless explicitly specified that materialized views based > on that table can be dropped too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20292) Bad join ordering in tpcds query93 with primary constraint defined
[
https://issues.apache.org/jira/browse/HIVE-20292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567364#comment-16567364
]
Hive QA commented on HIVE-20292:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
20s{color} | {color:blue} ql in master has 2302 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13009/dev-support/hive-personality.sh
|
| git revision | master / fd2d73f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13009/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Bad join ordering in tpcds query93 with primary constraint defined
> --
>
> Key: HIVE-20292
> URL: https://issues.apache.org/jira/browse/HIVE-20292
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20292.1.patch, HIVE-20292.2.patch,
> HIVE-20292.3.patch
>
>
> Query 93 has join (including outer) b/w store_sales, store_return and reason.
> Without constraints store_return is joined with reason and then with
> store_sales.
> But if a primary key is added on store_return (alter table store_returns add
> constraint tpcds_pk_sr primary key (sr_item_sk, sr_ticket_number) disable
> novalidate rely) join order becomes ((store_sales, store_return), reason)
> which is very inefficient.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20291) Allow HiveStreamingConnection to receive a WriteId
[ https://issues.apache.org/jira/browse/HIVE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20291: --- Status: Open (was: Patch Available) > Allow HiveStreamingConnection to receive a WriteId > -- > > Key: HIVE-20291 > URL: https://issues.apache.org/jira/browse/HIVE-20291 > Project: Hive > Issue Type: Improvement >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20291.1.patch > > > If the writeId is received externally it won't need to open connections to > the metastore. It won't be able to the commit in this case as well so it must > be done by the entity passing the writeId. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20292) Bad join ordering in tpcds query93 with primary constraint defined
[
https://issues.apache.org/jira/browse/HIVE-20292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567397#comment-16567397
]
Hive QA commented on HIVE-20292:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934161/HIVE-20292.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 38 failed/errored test(s), 14850 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer5]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join28] (batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_subquery2]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_subquery]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_repeated_alias]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[q93_with_constraints]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[runtime_skewjoin_mapjoin_spark]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_25]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join6]
(batchId=44)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[column_access_stats]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer2]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer3]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_sw]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join32_lessSize]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb_ptf]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_self_join]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_nested_mapjoin]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_use_op_stats]
(batchId=186)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=147)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[column_access_stats]
(batchId=134)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join28]
(batchId=146)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join32_lessSize]
(batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin_subquery2]
(batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin_subquery]
(batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[runtime_skewjoin_mapjoin_spark]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_25]
(batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_nested_mapjoin]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query2]
(batchId=265)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query59]
(batchId=265)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query95]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query2]
(batchId=263)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query59]
(batchId=263)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query93]
(batchId=263)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query95]
(batchId=263)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13009/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13009/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13009/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 38 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934161 - PreCommit-HIVE-Build
> Bad join ordering in
[jira] [Updated] (HIVE-18684) Race condition in RemoteSparkJobMonitor
[
https://issues.apache.org/jira/browse/HIVE-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18684:
Status: Open (was: Patch Available)
> Race condition in RemoteSparkJobMonitor
> ---
>
> Key: HIVE-18684
> URL: https://issues.apache.org/jira/browse/HIVE-18684
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18684.1.patch, HIVE-18684.2.patch,
> HIVE-18684.3.patch
>
>
> There is a race condition in {{RemoteSparkJobMonitor}}. Sometimes the info in
> {{RemoteSparkJobMonitor#startMonitor.STARTED}} gets printed out, sometimes it
> doesn't. This can be easily verified by running a qtest on
> {{TestMiniSparkOnYarnCliDriver}} and counting the number of times {{Query
> Hive on Spark job}} is printed vs. the number of times {{Finished
> successfully in}} gets printed.
> The issue is that {{RemoteSparkJobMonitor}} runs every one second, and checks
> the state of {{JobHandle}}. Depending on the state, it prints out some
> logging info. The content of the logs contain an implicit assumption that
> logs in the {{STARTED}} state are printed before the logs in the
> {{SUCCEEDED}} state. However, this isn't always the case. The state
> transitions are driven by how long the remote Spark job takes to run, and it
> it finishes within one second then the logs in the {{STARTED}} state never
> printed.
> This can be confusing to users, and there is key debugging information that
> is printed in the {{STARTED}} state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19937) Intern fields in MapWork on deserialization
[
https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567402#comment-16567402
]
Sahil Takiar commented on HIVE-19937:
-
Ping [~vihangk1]
> Intern fields in MapWork on deserialization
> ---
>
> Key: HIVE-19937
> URL: https://issues.apache.org/jira/browse/HIVE-19937
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19937.1.patch, HIVE-19937.2.patch,
> HIVE-19937.3.patch, HIVE-19937.4.patch, HIVE-19937.5.patch,
> post-patch-report.html, report.html
>
>
> When fixing HIVE-16395, we decided that each new Spark task should clone the
> {{JobConf}} object to prevent any {{ConcurrentModificationException}} from
> being thrown. However, setting this variable comes at a cost of storing a
> duplicate {{JobConf}} object for each Spark task. These objects can take up a
> significant amount of memory, we should intern them so that Spark tasks
> running in the same JVM don't store duplicate copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19008) Improve Spark session id logging
[ https://issues.apache.org/jira/browse/HIVE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567404#comment-16567404 ] Sahil Takiar commented on HIVE-19008: - Ping [~aihuaxu] > Improve Spark session id logging > > > Key: HIVE-19008 > URL: https://issues.apache.org/jira/browse/HIVE-19008 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-19008.1.patch, HIVE-19008.2.patch > > > HoS users have two session ids, one id for the Hive session and another id > for the Spark session, both are UUIDs. > I think some improvements could be made here: > The Spark session id could just be a counter that is incremented for each new > Spark session within a Hive session. Each Spark session is still globally > identifiable by its associated Hive session id + its own counter. This may > make more sense since the Hive session - Spark session has a 1-to-many > relationship, as in a single Hive session can contain multiple Spark > sessions, and each Spark session must belong to a Hive session. > Furthermore, we should include both the Hive session id and Spark session id > in the console logs + the Spark Web UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20273) Spark jobs aren't cancelled if getSparkJobInfo or getSparkStagesInfo
[
https://issues.apache.org/jira/browse/HIVE-20273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-20273:
Attachment: HIVE-20273.1.patch
> Spark jobs aren't cancelled if getSparkJobInfo or getSparkStagesInfo
>
>
> Key: HIVE-20273
> URL: https://issues.apache.org/jira/browse/HIVE-20273
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20273.1.patch
>
>
> HIVE-19053 and HIVE-19733 add handling of {{InterruptedException}} to
> {{#getSparkJobInfo}} and {{#getSparkStagesInfo}} in {{RemoteSparkJobStatus}},
> but that means the {{InterruptedException}} is wrapped in a {{HiveException}}
> and then thrown. The {{HiveException}} is then cause in
> {{RemoteSparkJobMonitor}} and then wrapped in another Hive exception. The
> double nesting of hive exception causes the logic in
> {{SparkTask#setSparkException}} to break, and it doesn't kill the job if an
> interrupted exception is thrown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20273) Spark jobs aren't cancelled if getSparkJobInfo or getSparkStagesInfo
[
https://issues.apache.org/jira/browse/HIVE-20273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-20273:
Status: Patch Available (was: Open)
> Spark jobs aren't cancelled if getSparkJobInfo or getSparkStagesInfo
>
>
> Key: HIVE-20273
> URL: https://issues.apache.org/jira/browse/HIVE-20273
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20273.1.patch
>
>
> HIVE-19053 and HIVE-19733 add handling of {{InterruptedException}} to
> {{#getSparkJobInfo}} and {{#getSparkStagesInfo}} in {{RemoteSparkJobStatus}},
> but that means the {{InterruptedException}} is wrapped in a {{HiveException}}
> and then thrown. The {{HiveException}} is then cause in
> {{RemoteSparkJobMonitor}} and then wrapped in another Hive exception. The
> double nesting of hive exception causes the logic in
> {{SparkTask#setSparkException}} to break, and it doesn't kill the job if an
> interrupted exception is thrown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20294) Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
[ https://issues.apache.org/jira/browse/HIVE-20294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20294: Status: In Progress (was: Patch Available) > Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT > > > Key: HIVE-20294 > URL: https://issues.apache.org/jira/browse/HIVE-20294 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20294.01.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized COALESCE and ELT. > Also, add tests for ARRAY and MAP indexing, IS [NOT] NULL and NOT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20294) Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
[ https://issues.apache.org/jira/browse/HIVE-20294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20294: Status: Patch Available (was: In Progress) > Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT > > > Key: HIVE-20294 > URL: https://issues.apache.org/jira/browse/HIVE-20294 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20294.01.patch, HIVE-20294.02.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized COALESCE and ELT. > Also, add tests for ARRAY and MAP indexing, IS [NOT] NULL and NOT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20294) Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
[ https://issues.apache.org/jira/browse/HIVE-20294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-20294: Attachment: HIVE-20294.02.patch > Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT > > > Key: HIVE-20294 > URL: https://issues.apache.org/jira/browse/HIVE-20294 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20294.01.patch, HIVE-20294.02.patch > > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized COALESCE and ELT. > Also, add tests for ARRAY and MAP indexing, IS [NOT] NULL and NOT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[ https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-14162: Attachment: HIVE-14162.5.patch > Allow disabling of long running job on Hive On Spark On YARN > > > Key: HIVE-14162 > URL: https://issues.apache.org/jira/browse/HIVE-14162 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Thomas Scott >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-14162.1.patch, HIVE-14162.2.patch, > HIVE-14162.3.patch, HIVE-14162.4.patch, HIVE-14162.5.patch > > > Hive On Spark launches a long running process on the first query to handle > all queries for that user session. In some use cases this is not desired, for > instance when using Hue with large intervals between query executions. > Could we have a property that would cause long running spark jobs to be > terminated after each query execution and started again for the next one? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567485#comment-16567485
]
Jason Dere commented on HIVE-20298:
---
Actually looking at the definition of the TBLS table in
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql,
this column is meant to have a default value
{noformat}
`WRITE_ID` bigint(20) DEFAULT 0,
{noformat}
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Priority: Major
> Attachments: HIVE-20298.1.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.g
[jira] [Commented] (HIVE-20273) Spark jobs aren't cancelled if getSparkJobInfo or getSparkStagesInfo
[
https://issues.apache.org/jira/browse/HIVE-20273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567486#comment-16567486
]
Hive QA commented on HIVE-20273:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2302 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 4 new + 14 unchanged - 2 fixed
= 18 total (was 16) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13010/dev-support/hive-personality.sh
|
| git revision | master / fd2d73f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13010/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13010/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Spark jobs aren't cancelled if getSparkJobInfo or getSparkStagesInfo
>
>
> Key: HIVE-20273
> URL: https://issues.apache.org/jira/browse/HIVE-20273
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20273.1.patch
>
>
> HIVE-19053 and HIVE-19733 add handling of {{InterruptedException}} to
> {{#getSparkJobInfo}} and {{#getSparkStagesInfo}} in {{RemoteSparkJobStatus}},
> but that means the {{InterruptedException}} is wrapped in a {{HiveException}}
> and then thrown. The {{HiveException}} is then cause in
> {{RemoteSparkJobMonitor}} and then wrapped in another Hive exception. The
> double nesting of hive exception causes the logic in
> {{SparkTask#setSparkException}} to break, and it doesn't kill the job if an
> interrupted exception is thrown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20259: -- Attachment: HIVE-20259.1.patch > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20259.1.patch > > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567500#comment-16567500 ] Jason Dere commented on HIVE-20259: --- Attached patch with utility DirectoryMarkerUpdate/Cleanup classes to create .cacheupdate files in the cache directory, to indicate that this directory should not be cleaned up by any other process performing DirectoryMarkerCleanup. This uses the last modify date of the .cacheupdate file to determine whether the file should be cleaned up, if the instance running cleanup determines this date is too old then the directory will be deleted. Another option, rather than relying on the lastModifyDate of the .cacheupdate file, would be for the .cacheupdate file contents to contain a long string indicating when this directory should be considered stale and safe to delete. This would have the benefit that the determination of when the directory should be cleaned up would depend on the settings of the application that wrote the .cacheupdate file, rather than on the settings of the application instance that is performing the cleanup. Though it means more file operations - it would involve having to read the file rather than just dealing with the file metadata. > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20259.1.patch > > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20259: -- Status: Patch Available (was: Open) > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20259.1.patch > > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20273) Spark jobs aren't cancelled if getSparkJobInfo or getSparkStagesInfo
[
https://issues.apache.org/jira/browse/HIVE-20273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567517#comment-16567517
]
Hive QA commented on HIVE-20273:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934171/HIVE-20273.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14858 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13010/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13010/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13010/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934171 - PreCommit-HIVE-Build
> Spark jobs aren't cancelled if getSparkJobInfo or getSparkStagesInfo
>
>
> Key: HIVE-20273
> URL: https://issues.apache.org/jira/browse/HIVE-20273
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20273.1.patch
>
>
> HIVE-19053 and HIVE-19733 add handling of {{InterruptedException}} to
> {{#getSparkJobInfo}} and {{#getSparkStagesInfo}} in {{RemoteSparkJobStatus}},
> but that means the {{InterruptedException}} is wrapped in a {{HiveException}}
> and then thrown. The {{HiveException}} is then cause in
> {{RemoteSparkJobMonitor}} and then wrapped in another Hive exception. The
> double nesting of hive exception causes the logic in
> {{SparkTask#setSparkException}} to break, and it doesn't kill the job if an
> interrupted exception is thrown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20118) SessionStateUserAuthenticator.getGroupNames() is always empty
[ https://issues.apache.org/jira/browse/HIVE-20118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567528#comment-16567528 ] Thejas M Nair commented on HIVE-20118: -- +1 > SessionStateUserAuthenticator.getGroupNames() is always empty > - > > Key: HIVE-20118 > URL: https://issues.apache.org/jira/browse/HIVE-20118 > Project: Hive > Issue Type: Improvement > Components: Authorization >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20118.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[
https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567538#comment-16567538
]
Hive QA commented on HIVE-14162:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2302 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 13 new + 156 unchanged - 0
fixed = 169 total (was 156) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
0s{color} | {color:red} ql generated 1 new + 2301 unchanged - 1 fixed = 2302
total (was 2302) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.close()
does not release lock on all paths At SparkSessionImpl.java:on all paths At
SparkSessionImpl.java:[line 190] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13011/dev-support/hive-personality.sh
|
| git revision | master / fd2d73f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13011/yetus/diff-checkstyle-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13011/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13011/yetus/new-findbugs-ql.html
|
| modules | C: common itests/hive-unit ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13011/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Allow disabling of long running job on Hive On Spark On YARN
>
>
> Key: HIVE-14162
> URL: https://issues
[jira] [Commented] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567551#comment-16567551
]
Sergey Shelukhin commented on HIVE-20298:
-
Yeah, it's not supposed to be nullable.
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Priority: Major
> Attachments: HIVE-20298.1.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
> at
> org.datanucleus.s
[jira] [Updated] (HIVE-20299) potential race in LLAP signer unit test
[ https://issues.apache.org/jira/browse/HIVE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20299: Attachment: HIVE-20299.patch > potential race in LLAP signer unit test > --- > > Key: HIVE-20299 > URL: https://issues.apache.org/jira/browse/HIVE-20299 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20299.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20299) potential race in LLAP signer unit test
[ https://issues.apache.org/jira/browse/HIVE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20299: Status: Patch Available (was: Open) > potential race in LLAP signer unit test > --- > > Key: HIVE-20299 > URL: https://issues.apache.org/jira/browse/HIVE-20299 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20299.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20299) potential race in LLAP signer unit test
[ https://issues.apache.org/jira/browse/HIVE-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567554#comment-16567554 ] Prasanth Jayachandran commented on HIVE-20299: -- +1 > potential race in LLAP signer unit test > --- > > Key: HIVE-20299 > URL: https://issues.apache.org/jira/browse/HIVE-20299 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20299.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[
https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567557#comment-16567557
]
Hive QA commented on HIVE-14162:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934177/HIVE-14162.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14857 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSparkSessionTimeout
(batchId=244)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13011/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13011/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13011/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934177 - PreCommit-HIVE-Build
> Allow disabling of long running job on Hive On Spark On YARN
>
>
> Key: HIVE-14162
> URL: https://issues.apache.org/jira/browse/HIVE-14162
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Thomas Scott
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-14162.1.patch, HIVE-14162.2.patch,
> HIVE-14162.3.patch, HIVE-14162.4.patch, HIVE-14162.5.patch
>
>
> Hive On Spark launches a long running process on the first query to handle
> all queries for that user session. In some use cases this is not desired, for
> instance when using Hue with large intervals between query executions.
> Could we have a property that would cause long running spark jobs to be
> terminated after each query execution and started again for the next one?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20294) Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
[
https://issues.apache.org/jira/browse/HIVE-20294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567576#comment-16567576
]
Hive QA commented on HIVE-20294:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2302 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 323 new + 647 unchanged - 5
fixed = 970 total (was 652) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
9s{color} | {color:red} ql generated 1 new + 2300 unchanged - 2 fixed = 2301
total (was 2302) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Class
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFMapIndexDecimalScalar
defines non-transient non-serializable instance field key In
VectorUDFMapIndexDecimalScalar.java:instance field key In
VectorUDFMapIndexDecimalScalar.java |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13012/dev-support/hive-personality.sh
|
| git revision | master / fd2d73f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13012/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13012/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13012/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13012/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
>
>
> Key: HIVE-20294
> URL: https://issues.apache.org/jira/browse/HIVE-20294
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20294.01.patch, HIVE-20294.02.patch
>
>
> Write new UT tests that use random data and intentional isRepeating batches
> to checks for NULL and Wrong Results for vectorized COALESCE and ELT.
> Also, add tests for ARRAY and MAP indexing, IS [NOT] NULL and NOT
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19008) Improve Spark session id logging
[ https://issues.apache.org/jira/browse/HIVE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567591#comment-16567591 ] Aihua Xu commented on HIVE-19008: - I like this idea. +1. > Improve Spark session id logging > > > Key: HIVE-19008 > URL: https://issues.apache.org/jira/browse/HIVE-19008 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-19008.1.patch, HIVE-19008.2.patch > > > HoS users have two session ids, one id for the Hive session and another id > for the Spark session, both are UUIDs. > I think some improvements could be made here: > The Spark session id could just be a counter that is incremented for each new > Spark session within a Hive session. Each Spark session is still globally > identifiable by its associated Hive session id + its own counter. This may > make more sense since the Hive session - Spark session has a 1-to-many > relationship, as in a single Hive session can contain multiple Spark > sessions, and each Spark session must belong to a Hive session. > Furthermore, we should include both the Hive session id and Spark session id > in the console logs + the Spark Web UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20294) Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
[
https://issues.apache.org/jira/browse/HIVE-20294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567608#comment-16567608
]
Hive QA commented on HIVE-20294:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934175/HIVE-20294.02.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14853 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_map_type_vectorization]
(batchId=154)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13012/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13012/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13012/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12934175 - PreCommit-HIVE-Build
> Vectorization: Fix NULL / Wrong Results issues in COALESCE / ELT
>
>
> Key: HIVE-20294
> URL: https://issues.apache.org/jira/browse/HIVE-20294
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20294.01.patch, HIVE-20294.02.patch
>
>
> Write new UT tests that use random data and intentional isRepeating batches
> to checks for NULL and Wrong Results for vectorized COALESCE and ELT.
> Also, add tests for ARRAY and MAP indexing, IS [NOT] NULL and NOT
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20259) Cleanup of results cache directory
[
https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567637#comment-16567637
]
Hive QA commented on HIVE-20259:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
53s{color} | {color:blue} ql in master has 2302 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} common: The patch generated 3 new + 424 unchanged - 0
fixed = 427 total (was 424) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13013/dev-support/hive-personality.sh
|
| git revision | master / fd2d73f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13013/yetus/diff-checkstyle-common.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13013/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Cleanup of results cache directory
> --
>
> Key: HIVE-20259
> URL: https://issues.apache.org/jira/browse/HIVE-20259
> Project: Hive
> Issue Type: Sub-task
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20259.1.patch
>
>
> The query results cache directory is currently deleted at process exit. This
> does not work in the case of a kill -9 or a sudden process exit of Hive.
> There should be some cleanup mechanism in place to take care of any old cache
> directories that were not deleted at process exit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
