[jira] [Commented] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[
https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056671#comment-17056671
]
Hive QA commented on HIVE-23005:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996305/HIVE-23005.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 216 failed/errored test(s), 18095 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[colstats_all_nulls]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[desc_table_formatted]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[explain_outputs]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[mapjoin2]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[select_dummy_source]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_10]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_16]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_2]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_3]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_7]
(batchId=306)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[udf_unix_timestamp]
(batchId=306)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testMoreComplexQueryWithDirectSql
(batchId=254)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testQueryWithFallbackToORM1
(batchId=254)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testQueryWithFallbackToORM2
(batchId=254)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testQueryWithFallbackToORM3
(batchId=254)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testQueryWithInWithFallbackToORM
(batchId=254)
org.apache.hadoop.hive.ql.TestMetaStoreLimitPartitionRequest.testSimpleQueryWithDirectSql
(batchId=254)
org.apache.hive.beeline.TestBeeLineWithArgs.testBackslashInLiteral (batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineConnectEnvVar
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineHiveConfVariable
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineHiveVariable
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineMultiHiveVariable
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineReconnect (batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineWithForce (batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testBeelineWithSilentAndReport
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testCSV2OutputWithDoubleQuotes
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutput (batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutputWithDoubleQuotes
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testConnectionWithURLParams
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testCustomDelimiter (batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testCustomDelimiterBeelineCmd
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testCustomDelimiterWithMultiQuery
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testCustomMultiCharDelimiter
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutput (batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutputWithDoubleQuotes
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testDescribeExtended (batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testEmbeddedBeelineConnection
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testEscapeSemiColonInEnclosedQuery
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testEscapeSemiColonInQueries
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testHiveVarSubstitution
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testLastLineCmdInScriptFile
(batchId=286)
org.apache.hive.beeline.TestBeeLineWithArgs.testMultiCommandsInOneEnclosedQuery
(batchId=286)
[jira] [Updated] (HIVE-23011) Shared work optimizer should check residual predicates when comparing joins
[ https://issues.apache.org/jira/browse/HIVE-23011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-23011: --- Attachment: HIVE-23011.patch > Shared work optimizer should check residual predicates when comparing joins > --- > > Key: HIVE-23011 > URL: https://issues.apache.org/jira/browse/HIVE-23011 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-23011.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23011) Shared work optimizer should check residual predicates when comparing joins
[ https://issues.apache.org/jira/browse/HIVE-23011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23011 started by Jesus Camacho Rodriguez. -- > Shared work optimizer should check residual predicates when comparing joins > --- > > Key: HIVE-23011 > URL: https://issues.apache.org/jira/browse/HIVE-23011 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-23011.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23011) Shared work optimizer should check residual predicates when comparing joins
[ https://issues.apache.org/jira/browse/HIVE-23011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-23011: --- Status: Patch Available (was: In Progress) > Shared work optimizer should check residual predicates when comparing joins > --- > > Key: HIVE-23011 > URL: https://issues.apache.org/jira/browse/HIVE-23011 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-23011.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23011) Shared work optimizer should check residual predicates when comparing joins
[ https://issues.apache.org/jira/browse/HIVE-23011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-23011: -- > Shared work optimizer should check residual predicates when comparing joins > --- > > Key: HIVE-23011 > URL: https://issues.apache.org/jira/browse/HIVE-23011 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22865) Include data in replication staging directory
[ https://issues.apache.org/jira/browse/HIVE-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anishek Agarwal updated HIVE-22865: --- Resolution: Fixed Status: Resolved (was: Patch Available) Patch committed to master, Thanks [~pkumarsinha] > Include data in replication staging directory > - > > Key: HIVE-22865 > URL: https://issues.apache.org/jira/browse/HIVE-22865 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22865.1.patch, HIVE-22865.10.patch, > HIVE-22865.11.patch, HIVE-22865.12.patch, HIVE-22865.13.patch, > HIVE-22865.14.patch, HIVE-22865.15.patch, HIVE-22865.16.patch, > HIVE-22865.17.patch, HIVE-22865.18.patch, HIVE-22865.19.patch, > HIVE-22865.2.patch, HIVE-22865.20.patch, HIVE-22865.21.patch, > HIVE-22865.22.patch, HIVE-22865.23.patch, HIVE-22865.24.patch, > HIVE-22865.25.patch, HIVE-22865.3.patch, HIVE-22865.4.patch, > HIVE-22865.5.patch, HIVE-22865.6.patch, HIVE-22865.7.patch, > HIVE-22865.8.patch, HIVE-22865.9.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[
https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056645#comment-17056645
]
Hive QA commented on HIVE-23005:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} jdbc: The patch generated 0 new + 31 unchanged - 2
fixed = 31 total (was 33) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21061/dev-support/hive-personality.sh
|
| git revision | master / 1a3b9bb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21061/yetus/patch-asflicense-problems.txt
|
| modules | C: jdbc U: jdbc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21061/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Consider Default JDBC Fetch Size From HS2
> -
>
> Key: HIVE-23005
> URL: https://issues.apache.org/jira/browse/HIVE-23005
> Project: Hive
> Issue Type: Sub-task
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23005.1.patch
>
>
> # Create two fetch sizes: and "init" fetch size and a "default" fetch size.
> The "init" fetch size comes from the JDBC connection string _fetchSize_ (if
> present) and the "default" fetch size comes from
> _HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE_ the server response to the
> open session request.
> # When a {{Statement}} is created, its starting fetch size is the "init"
> fetch size (may be 0)
> # Manually setting the fetch size on the {{Statement}} to 0, sets the fetch
> size to be the server default
> Setting to zero defaults to the server's instructed default and adheres to:
> bq. If the value specified is zero, then the hint is ignored. The default
> value is zero.
> That is to say, if the fetch size is 0, the default from the server is used,
> otherwise the user can pass a 'hint' and that will be the number of rows
> fetched instead.
> https://docs.oracle.com/javase/8/docs/api/java/sql/Statement.html#setFetchSize-int-
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22865) Include data in replication staging directory
[ https://issues.apache.org/jira/browse/HIVE-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056646#comment-17056646 ] Anishek Agarwal commented on HIVE-22865: +1 > Include data in replication staging directory > - > > Key: HIVE-22865 > URL: https://issues.apache.org/jira/browse/HIVE-22865 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22865.1.patch, HIVE-22865.10.patch, > HIVE-22865.11.patch, HIVE-22865.12.patch, HIVE-22865.13.patch, > HIVE-22865.14.patch, HIVE-22865.15.patch, HIVE-22865.16.patch, > HIVE-22865.17.patch, HIVE-22865.18.patch, HIVE-22865.19.patch, > HIVE-22865.2.patch, HIVE-22865.20.patch, HIVE-22865.21.patch, > HIVE-22865.22.patch, HIVE-22865.23.patch, HIVE-22865.24.patch, > HIVE-22865.25.patch, HIVE-22865.3.patch, HIVE-22865.4.patch, > HIVE-22865.5.patch, HIVE-22865.6.patch, HIVE-22865.7.patch, > HIVE-22865.8.patch, HIVE-22865.9.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23003) CliDriver leaves the session id in the threadname on failure
[
https://issues.apache.org/jira/browse/HIVE-23003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056638#comment-17056638
]
Hive QA commented on HIVE-23003:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996286/HIVE-23003.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18094 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21060/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21060/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21060/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996286 - PreCommit-HIVE-Build
> CliDriver leaves the session id in the threadname on failure
>
>
> Key: HIVE-23003
> URL: https://issues.apache.org/jira/browse/HIVE-23003
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23003.01.patch
>
>
> in case the command execution ends with an Exception; the line before the
> reurrn which supposed to remove the sessionid from the thread name is not
> removed
> https://github.com/apache/hive/blob/b5de84a6ce27904562bb15d7fe800b6ae3c32670/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java#L209
> impact: in case a lot of failed commands are executed (TestNegativeCliDriver)
> the sessionid-s are accumulated in the threadname causing "No more space
> left on device"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23003) CliDriver leaves the session id in the threadname on failure
[
https://issues.apache.org/jira/browse/HIVE-23003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056611#comment-17056611
]
Hive QA commented on HIVE-23003:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} cli in master has 9 extant Findbugs warnings. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
36s{color} | {color:red} cli generated 1 new + 8 unchanged - 1 fixed = 9 total
(was 9) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:cli |
| | org.apache.hadoop.hive.cli.CliDriver.processCmd1(String) invokes
System.exit(...), which shuts down the entire virtual machine At
CliDriver.java:down the entire virtual machine At CliDriver.java:[line 146] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21060/dev-support/hive-personality.sh
|
| git revision | master / 1a3b9bb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21060/yetus/new-findbugs-cli.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21060/yetus/patch-asflicense-problems.txt
|
| modules | C: cli U: cli |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21060/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CliDriver leaves the session id in the threadname on failure
>
>
> Key: HIVE-23003
> URL: https://issues.apache.org/jira/browse/HIVE-23003
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23003.01.patch
>
>
> in case the command execution ends with an Exception; the line before the
> reurrn which supposed to remove the sessionid from the thread name is not
> removed
> https://github.com/apache/hive/blob/b5de84a6ce27904562bb15d7fe800b6ae3c32670/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java#L209
> impact: in case a lot of failed commands are executed (TestNegativeCliDriver)
> the sessionid-s are accumulated in the threadname causing "No more space
> left on device"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23007) Server Should Return Default Fetch Size If One Is Not Sent By Client
[ https://issues.apache.org/jira/browse/HIVE-23007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23007: -- Status: Patch Available (was: Open) > Server Should Return Default Fetch Size If One Is Not Sent By Client > > > Key: HIVE-23007 > URL: https://issues.apache.org/jira/browse/HIVE-23007 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23007.1.patch, HIVE-23007.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23007) Server Should Return Default Fetch Size If One Is Not Sent By Client
[ https://issues.apache.org/jira/browse/HIVE-23007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23007: -- Attachment: HIVE-23007.2.patch > Server Should Return Default Fetch Size If One Is Not Sent By Client > > > Key: HIVE-23007 > URL: https://issues.apache.org/jira/browse/HIVE-23007 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23007.1.patch, HIVE-23007.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22985) Failed compaction always throws TxnAbortedException
[
https://issues.apache.org/jira/browse/HIVE-22985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056605#comment-17056605
]
Hive QA commented on HIVE-22985:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996274/HIVE-22985.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18094 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21059/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21059/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21059/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996274 - PreCommit-HIVE-Build
> Failed compaction always throws TxnAbortedException
> ---
>
> Key: HIVE-22985
> URL: https://issues.apache.org/jira/browse/HIVE-22985
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: compaction
> Attachments: HIVE-22985.01.patch, HIVE-22985.02.patch,
> HIVE-22985.02.patch
>
>
> If compaction fails, its txn is aborted, however Worker attempts to commit it
> again in a finally statement. This results in a TxnAbortedException [1]
> thrown from TxnHandler#commitTxn
> We need to add a check and only try to commit at the end if the txn is not
> aborted.(TxnHandler#commitTxn does nothing if txn is already committed.)
> [1]
> {code:java}
> ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler -
> TxnAbortedException(message:Transaction txnid:16 already aborted)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.raiseTxnUnexpectedState(TxnHandler.java:4843)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(TxnHandler.java:1141)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.commit_txn(HiveMetaStore.java:8101)
> ...
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.commitTxn(Worker.java:291)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:269)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23007) Server Should Return Default Fetch Size If One Is Not Sent By Client
[ https://issues.apache.org/jira/browse/HIVE-23007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23007: -- Status: Open (was: Patch Available) > Server Should Return Default Fetch Size If One Is Not Sent By Client > > > Key: HIVE-23007 > URL: https://issues.apache.org/jira/browse/HIVE-23007 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23007.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22985) Failed compaction always throws TxnAbortedException
[
https://issues.apache.org/jira/browse/HIVE-22985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056585#comment-17056585
]
Hive QA commented on HIVE-22985:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21059/dev-support/hive-personality.sh
|
| git revision | master / 1a3b9bb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21059/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21059/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Failed compaction always throws TxnAbortedException
> ---
>
> Key: HIVE-22985
> URL: https://issues.apache.org/jira/browse/HIVE-22985
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: compaction
> Attachments: HIVE-22985.01.patch, HIVE-22985.02.patch,
> HIVE-22985.02.patch
>
>
> If compaction fails, its txn is aborted, however Worker attempts to commit it
> again in a finally statement. This results in a TxnAbortedException [1]
> thrown from TxnHandler#commitTxn
> We need to add a check and only try to commit at the end if the txn is not
> aborted.(TxnHandler#commitTxn does nothing if txn is already committed.)
> [1]
> {code:java}
> ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler -
> TxnAbortedException(message:Transaction txnid:16 already aborted)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.raiseTxnUnexpectedState(TxnHandler.java:4843)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(TxnHandler.java:1141)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.commit_txn(HiveMetaStore.java:8101)
> ...
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.commitTxn(Worker.java:291)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:269)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23002) Optimise LazyBinaryUtils.writeVLong
[ https://issues.apache.org/jira/browse/HIVE-23002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23002: Status: Patch Available (was: Open) > Optimise LazyBinaryUtils.writeVLong > --- > > Key: HIVE-23002 > URL: https://issues.apache.org/jira/browse/HIVE-23002 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23002.1.patch, Screenshot 2020-03-10 at 5.01.34 > AM.jpg > > > [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java#L420] > It would be good to add a method which accepts scratch bytes. > > !Screenshot 2020-03-10 at 5.01.34 AM.jpg|width=452,height=321! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23002) Optimise LazyBinaryUtils.writeVLong
[ https://issues.apache.org/jira/browse/HIVE-23002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23002: Attachment: HIVE-23002.1.patch > Optimise LazyBinaryUtils.writeVLong > --- > > Key: HIVE-23002 > URL: https://issues.apache.org/jira/browse/HIVE-23002 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23002.1.patch, Screenshot 2020-03-10 at 5.01.34 > AM.jpg > > > [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java#L420] > It would be good to add a method which accepts scratch bytes. > > !Screenshot 2020-03-10 at 5.01.34 AM.jpg|width=452,height=321! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23004) Support Decimal64 operations across multiple vertices
[
https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056570#comment-17056570
]
Hive QA commented on HIVE-23004:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996332/HIVE-23004.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 413 failed/errored test(s), 18095 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[convert_decimal64_to_decimal]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_decimal_vectorized]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mergejoin] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_struct_type_vectorization]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_complex_types_vectorization]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_map_type_vectorization]
(batchId=97)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_no_row_serde]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_struct_type_vectorization]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_10]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_11]
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_12]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_13]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_14]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_15]
(batchId=100)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_16]
(batchId=94)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_17]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_1]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_2]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_3]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_4]
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_5]
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_6]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_7]
(batchId=98)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_8]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_9]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_decimal_date]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_limit]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_offset_limit]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[query_result_fileformat]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_aggregate_9]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_aggregate_without_gby]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_between_columns]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_binary_join_groupby]
(batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_case_when_1]
(batchId=98)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_case_when_2]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_cast_constant]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_2]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_mapjoin1]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_simple]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_coalesce]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_coalesce_2]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_coalesce_3]
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_coalesce_4]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_count]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_data_types]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_date_1]
(batchId=25)
[jira] [Commented] (HIVE-23009) SEL operator created by DynamicPartitionPruningOptimization does not populate colExprMap
[ https://issues.apache.org/jira/browse/HIVE-23009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056568#comment-17056568 ] Prasanth Jayachandran commented on HIVE-23009: -- Tested this locally with a repro. Looks good to me. +1 > SEL operator created by DynamicPartitionPruningOptimization does not populate > colExprMap > > > Key: HIVE-23009 > URL: https://issues.apache.org/jira/browse/HIVE-23009 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-23009.patch > > > This can lead to incorrect column stats propagation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23004) Support Decimal64 operations across multiple vertices
[
https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056547#comment-17056547
]
Hive QA commented on HIVE-23004:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
17s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21058/dev-support/hive-personality.sh
|
| git revision | master / 1a3b9bb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21058/yetus/whitespace-tabs.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21058/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21058/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support Decimal64 operations across multiple vertices
> -
>
> Key: HIVE-23004
> URL: https://issues.apache.org/jira/browse/HIVE-23004
> Project: Hive
> Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch
>
>
> Support Decimal64 operations across multiple vertices
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22955) PreUpgradeTool can fail because access to CharsetDecoder is not synchronized
[
https://issues.apache.org/jira/browse/HIVE-22955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056527#comment-17056527
]
Hive QA commented on HIVE-22955:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996271/HIVE-22955.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18095 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestCatalogNonDefaultClient.alterPartitions
(batchId=230)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21057/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21057/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21057/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996271 - PreCommit-HIVE-Build
> PreUpgradeTool can fail because access to CharsetDecoder is not synchronized
>
>
> Key: HIVE-22955
> URL: https://issues.apache.org/jira/browse/HIVE-22955
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Hankó Gergely
>Assignee: Hankó Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22955.1.patch, HIVE-22955.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> 2020-02-26 20:22:49,683 ERROR [main] acid.PreUpgradeTool
> (PreUpgradeTool.java:main(150)) - PreUpgradeTool failed
> org.apache.hadoop.hive.ql.metadata.HiveException at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:283)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:146)
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.IllegalStateException: Current state = RESET, new state = FLUSHED
> ...
> Caused by: java.lang.IllegalStateException: Current state = RESET, new state
> = FLUSHED at
> java.nio.charset.CharsetDecoder.throwIllegalStateException(CharsetDecoder.java:992)
> at java.nio.charset.CharsetDecoder.flush(CharsetDecoder.java:675) at
> java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:804) at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.needsCompaction(PreUpgradeTool.java:606)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.needsCompaction(PreUpgradeTool.java:567)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getCompactionCommands(PreUpgradeTool.java:464)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.processTable(PreUpgradeTool.java:374)
> {code}
> This is probably caused by HIVE-21948.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23010) IllegalStateException in tez.ReduceRecordProcessor when containers are being reused
[
https://issues.apache.org/jira/browse/HIVE-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sebastian Klemke updated HIVE-23010:
Description:
When executing a query in Hive that runs a filesink, mergejoin and two group by
operators in a single reduce vertex (reducer 2 in [^simplified-explain.txt]),
the following exception occurs non-deterministically:
{code:java}
java.lang.RuntimeException: java.lang.IllegalStateException: Was expecting
dummy store operator but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Was expecting dummy store operator
but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:421)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:148)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
... 16 more
{code}
Looking at Yarn logs, IllegalStateException occurs in a container if and only if
* the container has been running a task attempt of "Reducer 2" successfully
before
* the container is then being reused for another task attempt of the same
"Reducer 2" vertex
The same query runs fine with tez.am.container.reuse.enabled=false.
Apparently, this error occurs deterministically within a container that is
being reused for multiple task attempts of the same reduce vertex.
We have not been able to reproduce this error deterministically or with a
smaller execution plan due to low probability of container reuse for same
vertex.
was:
When executing a query in Hive that runs a filesink, mergejoin and two group by
operators in a single reduce vertex (reducer 2 in [^simplified-explain.txt]),
the following exception occurs non-deterministically:
{code:java}
java.lang.RuntimeException: java.lang.IllegalStateException: Was expecting
dummy store operator but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
[jira] [Updated] (HIVE-23010) IllegalStateException in tez.ReduceRecordProcessor when containers are being reused
[
https://issues.apache.org/jira/browse/HIVE-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sebastian Klemke updated HIVE-23010:
Description:
When executing a query in Hive that runs a filesink, mergejoin and two group by
operators in a single reduce vertex (reducer 2 in [^simplified-explain.txt]),
the following exception occurs non-deterministically:
{code:java}
java.lang.RuntimeException: java.lang.IllegalStateException: Was expecting
dummy store operator but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Was expecting dummy store operator
but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:421)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:148)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
... 16 more
{code}
Looking at Yarn logs, IllegalStateException occurs in a container if and only if
* the container has been running a task attempt of "Reducer 2" successfully
before
* the container is then being reused for another task attempt of the same
"Reducer 2" vertex
The same query runs fine with tez.am.container.reuse.enabled=false.
Apparently, this error occurs deterministically within a container that is
being reused for multiple task attempts of the same reduce vertex.
We have not been able to reproduce this error deterministically or with a
smaller execution plan due low probability of container reuse for same vertex.
was:
When executing a query in Hive that runs a filesink, mergejoin and two group by
operators in a single reduce vertex (reducer 2 in [^simplified-explain.txt]),
the following exception occurs non-deterministically:
{code:java}
java.lang.RuntimeException: java.lang.IllegalStateException: Was expecting
dummy store operator but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
[jira] [Updated] (HIVE-23010) IllegalStateException in tez.ReduceRecordProcessor when containers are being reused
[
https://issues.apache.org/jira/browse/HIVE-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sebastian Klemke updated HIVE-23010:
Description:
When executing a query in Hive that runs a filesink, mergejoin and two group by
operators in a single reduce vertex (reducer 2 in [^simplified-explain.txt]),
the following exception occurs non-deterministically:
{code:java}
java.lang.RuntimeException: java.lang.IllegalStateException: Was expecting
dummy store operator but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Was expecting dummy store operator
but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:421)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:148)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
... 16 more
{code}
Looking at Yarn logs, IllegalStateException occurs in a container if and only if
* the container has been running a task attempt of "Reducer 2" successfully
before
* the container is then being reused for another task attempt of the same
"Reducer 2" vertex
The same query runs fine with tez.am.container.reuse.enabled=false.
Apparently, this error occurs deterministically within a container that is
being reused for multiple task attempts of the same reduce vertex.
We have not been able to reproduce this error deterministically or with a
smaller execution plan.
was:
When executing a query in Hive that runs a filesink, mergejoin and two group by
operators in a single reduce vertex (reducer 2 in [^simplified-explain.txt]),
the following exception occurs non-deterministically:
{code:java}
java.lang.RuntimeException: java.lang.IllegalStateException: Was expecting
dummy store operator but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
[jira] [Updated] (HIVE-23010) IllegalStateException in tez.ReduceRecordProcessor when containers are being reused
[
https://issues.apache.org/jira/browse/HIVE-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sebastian Klemke updated HIVE-23010:
Description:
When executing a query in Hive that runs a filesink, mergejoin and two group by
operators in a single reduce vertex (reducer 2 in , the following exception
occurs non-deterministically:
{code}
java.lang.RuntimeException: java.lang.IllegalStateException: Was expecting
dummy store operator but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Was expecting dummy store operator
but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:421)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:148)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
... 16 more
{code}
Looking at Yarn logs, IllegalStateException occurs in a container if and only
if
* the container has been running a task attempt of the Mergejoin/Groupby
reducer successfully before
* the container is then being reused for another task attempt of the same
reduce vertex
The same query runs fine with tez.am.container.reuse.enabled=false.
Apparently, this error occurs deterministically within a container that is
being reused for multiple task attempts of the same reduce vertex.
We have not been able to reproduce this error deterministically or with a
smaller execution plan.
was:
When executing a query in Hive that runs a filesink, mergejoin and two group by
operators in a single reduce vertex, the following exception occurs
non-deterministically:
{code}
java.lang.RuntimeException: java.lang.IllegalStateException: Was expecting
dummy store operator but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
[jira] [Updated] (HIVE-23010) IllegalStateException in tez.ReduceRecordProcessor when containers are being reused
[
https://issues.apache.org/jira/browse/HIVE-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sebastian Klemke updated HIVE-23010:
Attachment: simplified-explain.txt
> IllegalStateException in tez.ReduceRecordProcessor when containers are being
> reused
> ---
>
> Key: HIVE-23010
> URL: https://issues.apache.org/jira/browse/HIVE-23010
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Sebastian Klemke
>Priority: Major
> Attachments: simplified-explain.txt
>
>
> When executing a query in Hive that runs a filesink, mergejoin and two group
> by operators in a single reduce vertex, the following exception occurs
> non-deterministically:
> {code}
> java.lang.RuntimeException: java.lang.IllegalStateException: Was expecting
> dummy store operator but found: FS[17]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalStateException: Was expecting dummy store
> operator but found: FS[17]
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:421)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:148)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 16 more
> {code}
> Looking at Yarn logs, IllegalStateException occurs in a container if and only
> if
> * the container has been running a task attempt of the Mergejoin/Groupby
> reducer successfully before
> * the container is then being reused for another task attempt of the same
> reduce vertex
> The same query runs fine with tez.am.container.reuse.enabled=false.
> Apparently, this error occurs deterministically within a container that is
> being reused for multiple task attempts of the same reduce vertex.
> We have not been able to reproduce this error deterministically or with a
> smaller execution plan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23010) IllegalStateException in tez.ReduceRecordProcessor when containers are being reused
[
https://issues.apache.org/jira/browse/HIVE-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sebastian Klemke updated HIVE-23010:
Description:
When executing a query in Hive that runs a filesink, mergejoin and two group by
operators in a single reduce vertex (reducer 2 in [^simplified-explain.txt]),
the following exception occurs non-deterministically:
{code:java}
java.lang.RuntimeException: java.lang.IllegalStateException: Was expecting
dummy store operator but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Was expecting dummy store operator
but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:421)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.getJoinParentOp(ReduceRecordProcessor.java:425)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:148)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
... 16 more
{code}
Looking at Yarn logs, IllegalStateException occurs in a container if and only if
* the container has been running a task attempt of the Mergejoin/Groupby
reducer successfully before
* the container is then being reused for another task attempt of the same
reduce vertex
The same query runs fine with tez.am.container.reuse.enabled=false.
Apparently, this error occurs deterministically within a container that is
being reused for multiple task attempts of the same reduce vertex.
We have not been able to reproduce this error deterministically or with a
smaller execution plan.
was:
When executing a query in Hive that runs a filesink, mergejoin and two group by
operators in a single reduce vertex (reducer 2 in , the following exception
occurs non-deterministically:
{code}
java.lang.RuntimeException: java.lang.IllegalStateException: Was expecting
dummy store operator but found: FS[17]
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
[jira] [Commented] (HIVE-22955) PreUpgradeTool can fail because access to CharsetDecoder is not synchronized
[
https://issues.apache.org/jira/browse/HIVE-22955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056489#comment-17056489
]
Hive QA commented on HIVE-22955:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} upgrade-acid/pre-upgrade in master has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} upgrade-acid/pre-upgrade: The patch generated 0 new
+ 15 unchanged - 38 fixed = 15 total (was 53) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21057/dev-support/hive-personality.sh
|
| git revision | master / 1a3b9bb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21057/yetus/patch-asflicense-problems.txt
|
| modules | C: upgrade-acid/pre-upgrade U: upgrade-acid/pre-upgrade |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21057/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> PreUpgradeTool can fail because access to CharsetDecoder is not synchronized
>
>
> Key: HIVE-22955
> URL: https://issues.apache.org/jira/browse/HIVE-22955
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Hankó Gergely
>Assignee: Hankó Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22955.1.patch, HIVE-22955.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> 2020-02-26 20:22:49,683 ERROR [main] acid.PreUpgradeTool
> (PreUpgradeTool.java:main(150)) - PreUpgradeTool failed
> org.apache.hadoop.hive.ql.metadata.HiveException at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:283)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:146)
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.IllegalStateException: Current state = RESET, new state = FLUSHED
> ...
> Caused by: java.lang.IllegalStateException: Current state = RESET, new state
> = FLUSHED at
> java.nio.charset.CharsetDecoder.throwIllegalStateException(CharsetDecoder.java:992)
> at
[jira] [Updated] (HIVE-23009) SEL operator created by DynamicPartitionPruningOptimization does not populate colExprMap
[ https://issues.apache.org/jira/browse/HIVE-23009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-23009: --- Status: Patch Available (was: In Progress) > SEL operator created by DynamicPartitionPruningOptimization does not populate > colExprMap > > > Key: HIVE-23009 > URL: https://issues.apache.org/jira/browse/HIVE-23009 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-23009.patch > > > This can lead to incorrect column stats propagation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23009) SEL operator created by DynamicPartitionPruningOptimization does not populate colExprMap
[ https://issues.apache.org/jira/browse/HIVE-23009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-23009: --- Attachment: HIVE-23009.patch > SEL operator created by DynamicPartitionPruningOptimization does not populate > colExprMap > > > Key: HIVE-23009 > URL: https://issues.apache.org/jira/browse/HIVE-23009 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-23009.patch > > > This can lead to incorrect column stats propagation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23009) SEL operator created by DynamicPartitionPruningOptimization does not populate colExprMap
[ https://issues.apache.org/jira/browse/HIVE-23009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23009 started by Jesus Camacho Rodriguez. -- > SEL operator created by DynamicPartitionPruningOptimization does not populate > colExprMap > > > Key: HIVE-23009 > URL: https://issues.apache.org/jira/browse/HIVE-23009 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-23009.patch > > > This can lead to incorrect column stats propagation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23009) SEL operator created by DynamicPartitionPruningOptimization does not populate colExprMap
[ https://issues.apache.org/jira/browse/HIVE-23009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-23009: -- > SEL operator created by DynamicPartitionPruningOptimization does not populate > colExprMap > > > Key: HIVE-23009 > URL: https://issues.apache.org/jira/browse/HIVE-23009 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > This can lead to incorrect column stats propagation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22762) Leap day is incorrectly parsed during cast in Hive
[
https://issues.apache.org/jira/browse/HIVE-22762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056473#comment-17056473
]
Hive QA commented on HIVE-22762:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996267/HIVE-22762.10.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18094 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21056/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21056/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21056/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996267 - PreCommit-HIVE-Build
> Leap day is incorrectly parsed during cast in Hive
> --
>
> Key: HIVE-22762
> URL: https://issues.apache.org/jira/browse/HIVE-22762
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22762.01.patch, HIVE-22762.01.patch,
> HIVE-22762.01.patch, HIVE-22762.01.patch, HIVE-22762.02.patch,
> HIVE-22762.03.patch, HIVE-22762.03.patch, HIVE-22762.04.patch,
> HIVE-22762.05.patch, HIVE-22762.06.patch, HIVE-22762.07.patch,
> HIVE-22762.08.patch, HIVE-22762.09.patch, HIVE-22762.10.patch
>
>
> While casting a string to a date with a custom date format having day token
> before year and moth tokens, the date is parsed incorrectly for leap days.
> h3. How to reproduce
> Execute {code}select cast("29 02 0" as date format "dd mm rr"){code} with
> Hive. The query results in *2020-02-28*, incorrectly.
>
> Executing the another cast with a slightly modified representation of the
> date (day is preceded by year and moth) is however correctly parsed:
> {code}select cast("0 02 29" as date format "rr mm dd"){code}
> It returns *2020-02-29*.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22762) Leap day is incorrectly parsed during cast in Hive
[
https://issues.apache.org/jira/browse/HIVE-22762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056426#comment-17056426
]
Hive QA commented on HIVE-22762:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} common: The patch generated 6 new + 0 unchanged - 0
fixed = 6 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21056/dev-support/hive-personality.sh
|
| git revision | master / 1a3b9bb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21056/yetus/diff-checkstyle-common.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21056/yetus/patch-asflicense-problems.txt
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21056/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Leap day is incorrectly parsed during cast in Hive
> --
>
> Key: HIVE-22762
> URL: https://issues.apache.org/jira/browse/HIVE-22762
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22762.01.patch, HIVE-22762.01.patch,
> HIVE-22762.01.patch, HIVE-22762.01.patch, HIVE-22762.02.patch,
> HIVE-22762.03.patch, HIVE-22762.03.patch, HIVE-22762.04.patch,
> HIVE-22762.05.patch, HIVE-22762.06.patch, HIVE-22762.07.patch,
> HIVE-22762.08.patch, HIVE-22762.09.patch, HIVE-22762.10.patch
>
>
> While casting a string to a date with a custom date format having day token
> before year and moth tokens, the date is parsed incorrectly for leap days.
> h3. How to reproduce
> Execute {code}select cast("29 02 0" as date format "dd mm rr"){code} with
> Hive. The query results in *2020-02-28*, incorrectly.
>
> Executing the another cast with a slightly modified representation of the
> date (day is preceded by year and moth) is however correctly parsed:
> {code}select cast("0 02 29" as date format "rr mm dd"){code}
> It returns *2020-02-29*.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23008) UDAFExampleMaxMinNUtil.sortedMerge must be able to handle all inputs
[ https://issues.apache.org/jira/browse/HIVE-23008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23008: Status: Patch Available (was: Open) > UDAFExampleMaxMinNUtil.sortedMerge must be able to handle all inputs > > > Key: HIVE-23008 > URL: https://issues.apache.org/jira/browse/HIVE-23008 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23008.01.patch > > > in case both inputs contain the same input values; the merge may stop with an > infinite loop > TestContribCliDriver may trigger this -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23008) UDAFExampleMaxMinNUtil.sortedMerge must be able to handle all inputs
[ https://issues.apache.org/jira/browse/HIVE-23008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23008: Attachment: HIVE-23008.01.patch > UDAFExampleMaxMinNUtil.sortedMerge must be able to handle all inputs > > > Key: HIVE-23008 > URL: https://issues.apache.org/jira/browse/HIVE-23008 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23008.01.patch > > > in case both inputs contain the same input values; the merge may stop with an > infinite loop > TestContribCliDriver may trigger this -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23008) UDAFExampleMaxMinNUtil.sortedMerge must be able to handle all inputs
[ https://issues.apache.org/jira/browse/HIVE-23008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-23008: --- > UDAFExampleMaxMinNUtil.sortedMerge must be able to handle all inputs > > > Key: HIVE-23008 > URL: https://issues.apache.org/jira/browse/HIVE-23008 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > in case both inputs contain the same input values; the merge may stop with an > infinite loop > TestContribCliDriver may trigger this -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22875) Refactor query creation in QueryCompactor implementations
[
https://issues.apache.org/jira/browse/HIVE-22875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056404#comment-17056404
]
Hive QA commented on HIVE-22875:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996263/HIVE-22875.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18093 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21055/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21055/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21055/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996263 - PreCommit-HIVE-Build
> Refactor query creation in QueryCompactor implementations
> -
>
> Key: HIVE-22875
> URL: https://issues.apache.org/jira/browse/HIVE-22875
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22875.01.patch, HIVE-22875.02.patch,
> HIVE-22875.02.patch, HIVE-22875.03.patch
>
>
> There is a lot of repetition where creation/compaction/drop queries are
> created in MajorQueryCompactor, MinorQueryCompactor, MmMajorQueryCompactor
> and MmMinorQueryCompactor.
> Initial idea is to create a CompactionQueryBuilder that all 4 implementations
> would use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22875) Refactor query creation in QueryCompactor implementations
[
https://issues.apache.org/jira/browse/HIVE-22875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056327#comment-17056327
]
Hive QA commented on HIVE-22875:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21055/dev-support/hive-personality.sh
|
| git revision | master / 6539e20 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21055/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21055/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Refactor query creation in QueryCompactor implementations
> -
>
> Key: HIVE-22875
> URL: https://issues.apache.org/jira/browse/HIVE-22875
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22875.01.patch, HIVE-22875.02.patch,
> HIVE-22875.02.patch, HIVE-22875.03.patch
>
>
> There is a lot of repetition where creation/compaction/drop queries are
> created in MajorQueryCompactor, MinorQueryCompactor, MmMajorQueryCompactor
> and MmMinorQueryCompactor.
> Initial idea is to create a CompactionQueryBuilder that all 4 implementations
> would use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21660) Wrong result when union all and later view with explode is used
[
https://issues.apache.org/jira/browse/HIVE-21660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-21660:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Ganesha!
> Wrong result when union all and later view with explode is used
> ---
>
> Key: HIVE-21660
> URL: https://issues.apache.org/jira/browse/HIVE-21660
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.1.1
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21660.1.patch, HIVE-21660.2.patch, HIVE-21660.patch
>
>
> There is a data loss when the data is inserted to a partitioned table using
> union all and lateral view with explode.
>
> *Steps to reproduce:*
>
> {code:java}
> create table t1 (id int, dt string);
> insert into t1 values (2, '2019-04-01');
> create table t2( id int, dates array);
> insert into t2 select 1 as id, array('2019-01-01','2019-01-02','2019-01-03')
> as dates;
> create table dst (id int) partitioned by (dt string);
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.exec.dynamic.partition=true;
> insert overwrite table dst partition (dt)
> select t.id, t.dt from (
> select id, dt from t1
> union all
> select id, dts as dt from t2 tt2 lateral view explode(tt2.dates) dd as dts )
> t;
> select * from dst;
> {code}
>
>
> *Actual Result:*
> {code:java}
> +--+--+
> | 2| 2019-04-01 |
> +--+--+{code}
>
> *Expected Result* (Run only the select part from the above insert query)*:*
> {code:java}
> +---++
> | 2 | 2019-04-01 |
> | 1 | 2019-01-01 |
> | 1 | 2019-01-02 |
> | 1 | 2019-01-03 |
> +---++{code}
>
> Data retrieved using union all and lateral view with explode from second
> table is missing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21660) Wrong result when union all and later view with explode is used
[
https://issues.apache.org/jira/browse/HIVE-21660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056299#comment-17056299
]
Ashutosh Chauhan commented on HIVE-21660:
-
+1
> Wrong result when union all and later view with explode is used
> ---
>
> Key: HIVE-21660
> URL: https://issues.apache.org/jira/browse/HIVE-21660
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.1.1
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21660.1.patch, HIVE-21660.2.patch, HIVE-21660.patch
>
>
> There is a data loss when the data is inserted to a partitioned table using
> union all and lateral view with explode.
>
> *Steps to reproduce:*
>
> {code:java}
> create table t1 (id int, dt string);
> insert into t1 values (2, '2019-04-01');
> create table t2( id int, dates array);
> insert into t2 select 1 as id, array('2019-01-01','2019-01-02','2019-01-03')
> as dates;
> create table dst (id int) partitioned by (dt string);
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.exec.dynamic.partition=true;
> insert overwrite table dst partition (dt)
> select t.id, t.dt from (
> select id, dt from t1
> union all
> select id, dts as dt from t2 tt2 lateral view explode(tt2.dates) dd as dts )
> t;
> select * from dst;
> {code}
>
>
> *Actual Result:*
> {code:java}
> +--+--+
> | 2| 2019-04-01 |
> +--+--+{code}
>
> *Expected Result* (Run only the select part from the above insert query)*:*
> {code:java}
> +---++
> | 2 | 2019-04-01 |
> | 1 | 2019-01-01 |
> | 1 | 2019-01-02 |
> | 1 | 2019-01-03 |
> +---++{code}
>
> Data retrieved using union all and lateral view with explode from second
> table is missing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22978) Fix decimal precision and scale inference for aggregate rewriting in Calcite
[
https://issues.apache.org/jira/browse/HIVE-22978?focusedWorklogId=400939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400939
]
ASF GitHub Bot logged work on HIVE-22978:
-
Author: ASF GitHub Bot
Created on: 10/Mar/20 19:19
Start Date: 10/Mar/20 19:19
Worklog Time Spent: 10m
Work Description: asfgit commented on pull request #938: HIVE-22978
URL: https://github.com/apache/hive/pull/938
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400939)
Time Spent: 50m (was: 40m)
> Fix decimal precision and scale inference for aggregate rewriting in Calcite
>
>
> Key: HIVE-22978
> URL: https://issues.apache.org/jira/browse/HIVE-22978
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22978.01.patch, HIVE-22978.02.patch,
> HIVE-22978.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Calcite rules can do rewritings of aggregate functions, e.g., {{avg}} into
> {{sum/count}}. When type of {{avg}} is decimal, inference of intermediate
> precision and scale for the division is not done correctly. The reason is
> that we miss support for some types in method {{getDefaultPrecision}} in
> {{HiveTypeSystemImpl}}. Additionally, {{deriveSumType}} should be overridden
> in {{HiveTypeSystemImpl}} to abide by the Hive semantics for sum aggregate
> type inference.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22978) Fix decimal precision and scale inference for aggregate rewriting in Calcite
[
https://issues.apache.org/jira/browse/HIVE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-22978:
---
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix decimal precision and scale inference for aggregate rewriting in Calcite
>
>
> Key: HIVE-22978
> URL: https://issues.apache.org/jira/browse/HIVE-22978
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22978.01.patch, HIVE-22978.02.patch,
> HIVE-22978.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Calcite rules can do rewritings of aggregate functions, e.g., {{avg}} into
> {{sum/count}}. When type of {{avg}} is decimal, inference of intermediate
> precision and scale for the division is not done correctly. The reason is
> that we miss support for some types in method {{getDefaultPrecision}} in
> {{HiveTypeSystemImpl}}. Additionally, {{deriveSumType}} should be overridden
> in {{HiveTypeSystemImpl}} to abide by the Hive semantics for sum aggregate
> type inference.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22964) MM table split computation is very slow
[
https://issues.apache.org/jira/browse/HIVE-22964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056293#comment-17056293
]
Hive QA commented on HIVE-22964:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996260/HIVE-22964.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18093 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21054/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21054/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21054/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996260 - PreCommit-HIVE-Build
> MM table split computation is very slow
> ---
>
> Key: HIVE-22964
> URL: https://issues.apache.org/jira/browse/HIVE-22964
> Project: Hive
> Issue Type: Improvement
>Reporter: Aditya Shah
>Assignee: Aditya Shah
>Priority: Major
> Attachments: HIVE-22964.1.patch, HIVE-22964.patch
>
>
> Since for MM table we process the paths prior to inputFormat.getSplits() we
> end up doing listing on the whole table at once. This could be optimized.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23007) Server Should Return Default Fetch Size If One Is Not Sent By Client
[ https://issues.apache.org/jira/browse/HIVE-23007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23007: -- Status: Patch Available (was: Open) > Server Should Return Default Fetch Size If One Is Not Sent By Client > > > Key: HIVE-23007 > URL: https://issues.apache.org/jira/browse/HIVE-23007 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23007.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23007) Server Should Return Default Fetch Size If One Is Not Sent By Client
[ https://issues.apache.org/jira/browse/HIVE-23007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-23007: - > Server Should Return Default Fetch Size If One Is Not Sent By Client > > > Key: HIVE-23007 > URL: https://issues.apache.org/jira/browse/HIVE-23007 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23007.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23007) Server Should Return Default Fetch Size If One Is Not Sent By Client
[ https://issues.apache.org/jira/browse/HIVE-23007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23007: -- Attachment: HIVE-23007.1.patch > Server Should Return Default Fetch Size If One Is Not Sent By Client > > > Key: HIVE-23007 > URL: https://issues.apache.org/jira/browse/HIVE-23007 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23007.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Attachment: HIVE-22990.patch Status: Patch Available (was: Open) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Status: Open (was: Patch Available) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Attachment: HIVE-23004.2.patch Status: Patch Available (was: Open) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-22990: -- Labels: pull-request-available (was: ) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=400912=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400912 ] ASF GitHub Bot logged work on HIVE-22990: - Author: ASF GitHub Bot Created on: 10/Mar/20 18:29 Start Date: 10/Mar/20 18:29 Worklog Time Spent: 10m Work Description: aasha commented on pull request #949: HIVE-22990 Add file based ack for replication URL: https://github.com/apache/hive/pull/949 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 400912) Remaining Estimate: 0h Time Spent: 10m > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Status: Patch Available (was: Open)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch, HIVE-22785.5.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Attachment: HIVE-22785.5.patch
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch, HIVE-22785.5.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22964) MM table split computation is very slow
[
https://issues.apache.org/jira/browse/HIVE-22964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056223#comment-17056223
]
Hive QA commented on HIVE-22964:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 3 new + 184 unchanged - 0
fixed = 187 total (was 184) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21054/dev-support/hive-personality.sh
|
| git revision | master / 1e15791 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21054/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21054/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21054/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MM table split computation is very slow
> ---
>
> Key: HIVE-22964
> URL: https://issues.apache.org/jira/browse/HIVE-22964
> Project: Hive
> Issue Type: Improvement
>Reporter: Aditya Shah
>Assignee: Aditya Shah
>Priority: Major
> Attachments: HIVE-22964.1.patch, HIVE-22964.patch
>
>
> Since for MM table we process the paths prior to inputFormat.getSplits() we
> end up doing listing on the whole table at once. This could be optimized.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Attachment: (was: HIVE-22785.5.patch)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Status: Open (was: Patch Available)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Attachment: HIVE-22785.5.patch
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch, HIVE-22785.5.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Status: Patch Available (was: Open)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch, HIVE-22785.5.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22785) Update/delete/merge statements not optimized through CBO
[
https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22785:
--
Status: Open (was: Patch Available)
> Update/delete/merge statements not optimized through CBO
>
>
> Key: HIVE-22785
> URL: https://issues.apache.org/jira/browse/HIVE-22785
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Critical
> Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch,
> HIVE-22785.2.patch, HIVE-22785.3.patch, HIVE-22785.4.patch, HIVE-22785.5.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three
> main tasks: 1) support for sort in Calcite planner, 2) support for SORT in
> AST converter, and 3) {{RewriteSemanticAnalyzer}} should extend
> {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22974) Metastore's table location check should be applied when location changed
[ https://issues.apache.org/jira/browse/HIVE-22974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22974: - Status: Open (was: Patch Available) > Metastore's table location check should be applied when location changed > > > Key: HIVE-22974 > URL: https://issues.apache.org/jira/browse/HIVE-22974 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22974.2.patch, HIVE-22974.3.patch, > HIVE-22974.4.patch > > > In HIVE-22189 a check was introduced to make sure managed and external tables > are located at the proper space. This condition cannot be satisfied during an > upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22974) Metastore's table location check should be applied when location changed
[ https://issues.apache.org/jira/browse/HIVE-22974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22974: - Attachment: HIVE-22974.4.patch > Metastore's table location check should be applied when location changed > > > Key: HIVE-22974 > URL: https://issues.apache.org/jira/browse/HIVE-22974 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22974.2.patch, HIVE-22974.3.patch, > HIVE-22974.4.patch > > > In HIVE-22189 a check was introduced to make sure managed and external tables > are located at the proper space. This condition cannot be satisfied during an > upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22974) Metastore's table location check should be applied when location changed
[ https://issues.apache.org/jira/browse/HIVE-22974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22974: - Status: Patch Available (was: Open) > Metastore's table location check should be applied when location changed > > > Key: HIVE-22974 > URL: https://issues.apache.org/jira/browse/HIVE-22974 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22974.2.patch, HIVE-22974.3.patch, > HIVE-22974.4.patch > > > In HIVE-22189 a check was introduced to make sure managed and external tables > are located at the proper space. This condition cannot be satisfied during an > upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22974) Metastore's table location check should be applied when location changed
[ https://issues.apache.org/jira/browse/HIVE-22974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22974: - Attachment: (was: HIVE-22974.4.patch) > Metastore's table location check should be applied when location changed > > > Key: HIVE-22974 > URL: https://issues.apache.org/jira/browse/HIVE-22974 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22974.2.patch, HIVE-22974.3.patch, > HIVE-22974.4.patch > > > In HIVE-22189 a check was introduced to make sure managed and external tables > are located at the proper space. This condition cannot be satisfied during an > upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22974) Metastore's table location check should be applied when location changed
[
https://issues.apache.org/jira/browse/HIVE-22974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056180#comment-17056180
]
Hive QA commented on HIVE-22974:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996252/HIVE-22974.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18094 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_ingest]
(batchId=185)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21053/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21053/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21053/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996252 - PreCommit-HIVE-Build
> Metastore's table location check should be applied when location changed
>
>
> Key: HIVE-22974
> URL: https://issues.apache.org/jira/browse/HIVE-22974
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22974.2.patch, HIVE-22974.3.patch,
> HIVE-22974.4.patch
>
>
> In HIVE-22189 a check was introduced to make sure managed and external tables
> are located at the proper space. This condition cannot be satisfied during an
> upgrade.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-18252) Limit the size of the object inspector caches
[ https://issues.apache.org/jira/browse/HIVE-18252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18252: -- Resolution: Duplicate Status: Resolved (was: Patch Available) Looks like this is fixed by HIVE-20274 and HIVE-19860 > Limit the size of the object inspector caches > - > > Key: HIVE-18252 > URL: https://issues.apache.org/jira/browse/HIVE-18252 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18252.1.patch, HIVE-18252.2.patch > > > Was running some tests that had a lot of queries with constant values, and > noticed that ObjectInspectorFactory.cachedStandardStructObjectInspector > started using up a lot of memory. > It appears that StructObjectInspector caching does not work properly with > constant values. Constant ObjectInspectors are not cached, so each constant > expression creates a new constant ObjectInspector. And since object > inspectors do not override equals(), object inspector comparison relies on > object instance comparison. So even if the values are exactly the same as > what is already in the cache, the StructObjectInspector cache lookup would > fail, and Hive would create a new object inspector and add it to the cache, > creating another entry that would never be used. Plus, there is no max cache > size - it's just a map that is allowed to grow as long as values keep getting > added to it. > Some possible solutions I can think of: > 1. Limit the size of the object inspector caches, rather than growing without > bound. > 2. Try to fix the caching to work with constant values. This would require > implementing equals() on the constant object inspectors (which could be slow > in nested cases), or else we would have to start caching constant object > inspectors, which could be expensive in terms of memory usage. Could be used > in combination with (1). By itself this is not a great solution because this > still has the unbounded cache growth issue. > 3. Disable caching in the case of constant object inspectors since this > scenario currently doesn't work. This could be used in combination with (1). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-22888:
--
Attachment: (was: Lock Perf.xlsx)
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.2.patch,
> HIVE-22888.3.patch, HIVE-22888.4.patch, HIVE-22888.5.patch,
> HIVE-22888.6.patch, HIVE-22888.8.patch
>
>
> - Created extra (db, tbl, part) index on HIVE_LOCKS table;
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE,
> HL_LOCK_TYPE FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 333) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE
> HL_LOCK_EXT_ID = 333) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LBC.HL_TABLE IS NULL OR LS.HL_TABLE =
> LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LBC.HL_PARTITION IS NULL OR
> LS.HL_PARTITION = LBC.HL_PARTITION))
> WHERE (LBC.HL_TXNID = 0 OR LS.HL_TXNID != LBC.HL_TXNID)
> AND (LBC.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NULL AND LS.HL_LOCK_TYPE='r' AND LBC.HL_TABLE
> IS NOT NULL )
> OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
> OR LBC.HL_LOCK_TYPE='r' AND LS.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NOT NULL AND LBC.HL_TABLE IS NULL))
> LIMIT 1;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-22888:
--
Attachment: Lock Perf.pdf
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.2.patch,
> HIVE-22888.3.patch, HIVE-22888.4.patch, HIVE-22888.5.patch,
> HIVE-22888.6.patch, HIVE-22888.8.patch, Lock Perf.pdf
>
>
> - Created extra (db, tbl, part) index on HIVE_LOCKS table;
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE,
> HL_LOCK_TYPE FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 333) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE
> HL_LOCK_EXT_ID = 333) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LBC.HL_TABLE IS NULL OR LS.HL_TABLE =
> LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LBC.HL_PARTITION IS NULL OR
> LS.HL_PARTITION = LBC.HL_PARTITION))
> WHERE (LBC.HL_TXNID = 0 OR LS.HL_TXNID != LBC.HL_TXNID)
> AND (LBC.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NULL AND LS.HL_LOCK_TYPE='r' AND LBC.HL_TABLE
> IS NOT NULL )
> OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
> OR LBC.HL_LOCK_TYPE='r' AND LS.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NOT NULL AND LBC.HL_TABLE IS NULL))
> LIMIT 1;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-22888:
--
Attachment: Lock Perf.xlsx
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.2.patch,
> HIVE-22888.3.patch, HIVE-22888.4.patch, HIVE-22888.5.patch,
> HIVE-22888.6.patch, HIVE-22888.8.patch
>
>
> - Created extra (db, tbl, part) index on HIVE_LOCKS table;
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE,
> HL_LOCK_TYPE FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 333) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE
> HL_LOCK_EXT_ID = 333) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LBC.HL_TABLE IS NULL OR LS.HL_TABLE =
> LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LBC.HL_PARTITION IS NULL OR
> LS.HL_PARTITION = LBC.HL_PARTITION))
> WHERE (LBC.HL_TXNID = 0 OR LS.HL_TXNID != LBC.HL_TXNID)
> AND (LBC.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NULL AND LS.HL_LOCK_TYPE='r' AND LBC.HL_TABLE
> IS NOT NULL )
> OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
> OR LBC.HL_LOCK_TYPE='r' AND LS.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NOT NULL AND LBC.HL_TABLE IS NULL))
> LIMIT 1;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-22888:
--
Attachment: (was: Lock Perf.numbers)
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.2.patch,
> HIVE-22888.3.patch, HIVE-22888.4.patch, HIVE-22888.5.patch,
> HIVE-22888.6.patch, HIVE-22888.8.patch
>
>
> - Created extra (db, tbl, part) index on HIVE_LOCKS table;
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE,
> HL_LOCK_TYPE FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 333) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE
> HL_LOCK_EXT_ID = 333) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LBC.HL_TABLE IS NULL OR LS.HL_TABLE =
> LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LBC.HL_PARTITION IS NULL OR
> LS.HL_PARTITION = LBC.HL_PARTITION))
> WHERE (LBC.HL_TXNID = 0 OR LS.HL_TXNID != LBC.HL_TXNID)
> AND (LBC.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NULL AND LS.HL_LOCK_TYPE='r' AND LBC.HL_TABLE
> IS NOT NULL )
> OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
> OR LBC.HL_LOCK_TYPE='r' AND LS.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NOT NULL AND LBC.HL_TABLE IS NULL))
> LIMIT 1;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati reassigned HIVE-22126: --- Assignee: Eugene Chung (was: David Lavati) > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch, > HIVE-22126.03.patch, HIVE-22126.04.patch, HIVE-22126.05.patch, > HIVE-22126.06.patch, HIVE-22126.07.patch, HIVE-22126.08.patch, > HIVE-22126.09.patch, HIVE-22126.09.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati reassigned HIVE-22126: --- Assignee: David Lavati (was: Eugene Chung) > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch, > HIVE-22126.03.patch, HIVE-22126.04.patch, HIVE-22126.05.patch, > HIVE-22126.06.patch, HIVE-22126.07.patch, HIVE-22126.08.patch, > HIVE-22126.09.patch, HIVE-22126.09.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23006) Compiler support for Probe MapJoin
[ https://issues.apache.org/jira/browse/HIVE-23006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis reassigned HIVE-23006: - > Compiler support for Probe MapJoin > -- > > Key: HIVE-23006 > URL: https://issues.apache.org/jira/browse/HIVE-23006 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-22126: Attachment: HIVE-22126.09.patch > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch, > HIVE-22126.03.patch, HIVE-22126.04.patch, HIVE-22126.05.patch, > HIVE-22126.06.patch, HIVE-22126.07.patch, HIVE-22126.08.patch, > HIVE-22126.09.patch, HIVE-22126.09.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22974) Metastore's table location check should be applied when location changed
[
https://issues.apache.org/jira/browse/HIVE-22974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056123#comment-17056123
]
Hive QA commented on HIVE-22974:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 2 new + 375 unchanged - 0 fixed = 377 total (was 375) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 190
unchanged - 0 fixed = 191 total (was 190) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21053/dev-support/hive-personality.sh
|
| git revision | master / 1e15791 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21053/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21053/yetus/diff-checkstyle-itests_hive-unit.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21053/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore/metastore-server itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21053/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Metastore's table location check should be applied when location changed
>
>
> Key: HIVE-22974
> URL: https://issues.apache.org/jira/browse/HIVE-22974
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22974.2.patch,
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22980:
Attachment: HIVE-22980.2.patch
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch
>
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22980:
Status: In Progress (was: Patch Available)
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch
>
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22980) Support custom path filter for ORC tables
[
https://issues.apache.org/jira/browse/HIVE-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-22980:
Status: Patch Available (was: In Progress)
> Support custom path filter for ORC tables
> -
>
> Key: HIVE-22980
> URL: https://issues.apache.org/jira/browse/HIVE-22980
> Project: Hive
> Issue Type: New Feature
> Components: ORC
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-22980.1.patch, HIVE-22980.2.patch
>
>
> The customer is looking for an option to specify custom path filter for ORC
> tables. Please find the details below from customer requirement.
> Problem Statement/Approach in customer words :
> {quote}
> Currently, Orc file input format does not take in path filters set in the
> property "mapreduce.input.pathfilter.class" OR "
> mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc
> files.
> AcidUtils class has a static filter called "hiddenFilters" which is used by
> ORC to filter input paths. If we can pass the custom filter classes(set in
> the property mentioned above) to AcidUtils and replace hiddenFilter with a
> filter that does an "and" operation over hiddenFilter+customFilters, the
> filters would work well.
> On local testing, mapreduce.input.pathfilter.class seems to be working for
> Text tables but not for ORC tables.
> {quote}
> Our analysis:
> {{OrcInputFormat}} and {{FileInputFormat}} are different implementations for
> {{Inputformat}} interface. Property "{{mapreduce.input.pathfilter.class}}" is
> only respected by {{FileInputFormat}}, but not by any other implementations
> of {{InputFormat}}. The customer wants to have the ability to filter out rows
> based on path/filenames, current ORC features like bloomfilters and indexes
> are not good enough for them to minimize number of disk read operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22901) Variable substitution can lead to OOM on circular references
[
https://issues.apache.org/jira/browse/HIVE-22901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056095#comment-17056095
]
Hive QA commented on HIVE-22901:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996253/HIVE-22901.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18094 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21052/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21052/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21052/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996253 - PreCommit-HIVE-Build
> Variable substitution can lead to OOM on circular references
>
>
> Key: HIVE-22901
> URL: https://issues.apache.org/jira/browse/HIVE-22901
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.2
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-22901.1.patch, HIVE-22901.2.patch,
> HIVE-22901.2.patch, HIVE-22901.2.patch, HIVE-22901.3.patch
>
>
> {{SystemVariables#substitute()}} is dealing with circular references between
> variables by only doing the substitution 40 times by default. If the
> substituted part is sufficiently large though, it's possible that the
> substitution will produce a string bigger than the heap size within the 40
> executions.
> Take the following test case that fails with OOM in current master (third
> round of execution would need 10G heap, while running with only 2G):
> {code}
> @Test
> public void testSubstitute() {
> String randomPart = RandomStringUtils.random(100_000);
> String reference = "${hiveconf:myTestVariable}";
> StringBuilder longStringWithReferences = new StringBuilder();
> for(int i = 0; i < 10; i ++) {
> longStringWithReferences.append(randomPart).append(reference);
> }
> SystemVariables uut = new SystemVariables();
> HiveConf conf = new HiveConf();
> conf.set("myTestVariable", longStringWithReferences.toString());
> uut.substitute(conf, longStringWithReferences.toString(), 40);
> }
> {code}
> Produces:
> {code}
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.apache.hadoop.hive.conf.SystemVariables.substitute(SystemVariables.java:110)
> at
> org.apache.hadoop.hive.conf.SystemVariablesTest.testSubstitute(SystemVariablesTest.java:27)
> {code}
> We should check the size of the substituted query and bail out earlier.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22940) Make the datasketches functions available as predefined functions
[
https://issues.apache.org/jira/browse/HIVE-22940?focusedWorklogId=400802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400802
]
ASF GitHub Bot logged work on HIVE-22940:
-
Author: ASF GitHub Bot
Created on: 10/Mar/20 16:12
Start Date: 10/Mar/20 16:12
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #930: HIVE-22940
datasketches functions
URL: https://github.com/apache/hive/pull/930#discussion_r390434617
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/DataSketchesFunctions.java
##
@@ -0,0 +1,218 @@
+package org.apache.hadoop.hive.ql.exec;
+
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
+
+public class DataSketchesFunctions {
+
+ private static final String DATA_TO_SKETCH = "datatosketch";
Review comment:
the final name of the method is:
* `ds_theta_gen_sketch`
* `ds_theta_to_sketch`
* `ds_theta_build_sketch`
* `ds_theta_build`
* `ds_theta_sketch`
I right now feel these last 2 the best; they are easy to remember and the
naming of these methods will be structured anyway - so the `ds_theta_` prefix
somewhat adds a context for the function we may not need to repeat "sketch"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400802)
Time Spent: 1h 40m (was: 1.5h)
> Make the datasketches functions available as predefined functions
> --
>
> Key: HIVE-22940
> URL: https://issues.apache.org/jira/browse/HIVE-22940
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22971) Eliminate file rename in insert-only compactor
[ https://issues.apache.org/jira/browse/HIVE-22971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-22971: - Attachment: HIVE-22971.02.patch > Eliminate file rename in insert-only compactor > -- > > Key: HIVE-22971 > URL: https://issues.apache.org/jira/browse/HIVE-22971 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: ACID, compaction > Attachments: HIVE-22971.01.patch, HIVE-22971.02.patch > > > File rename is expensive for object stores, so MM (insert-only) compaction > should skip that step when committing and write directly to base_x_cZ or > delta_x_y_cZ. > This also fixes the issue that for MM QB compaction the temp tables were > stored under the table directory, and these temp dirs were never cleaned up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22940) Make the datasketches functions available as predefined functions
[
https://issues.apache.org/jira/browse/HIVE-22940?focusedWorklogId=400790=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400790
]
ASF GitHub Bot logged work on HIVE-22940:
-
Author: ASF GitHub Bot
Created on: 10/Mar/20 16:03
Start Date: 10/Mar/20 16:03
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #930: HIVE-22940
datasketches functions
URL: https://github.com/apache/hive/pull/930#discussion_r390409296
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/DataSketchesFunctions.java
##
@@ -0,0 +1,218 @@
+package org.apache.hadoop.hive.ql.exec;
+
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
+
+public class DataSketchesFunctions {
+
+ private static final String DATA_TO_SKETCH = "datatosketch";
+ private static final String SKETCH_TO_ESTIMATE_WITH_ERROR_BOUNDS =
"sketchToEstimateWithErrorBounds";
Review comment:
`get_estimate_bounds` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400790)
Time Spent: 1h 20m (was: 1h 10m)
> Make the datasketches functions available as predefined functions
> --
>
> Key: HIVE-22940
> URL: https://issues.apache.org/jira/browse/HIVE-22940
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22940) Make the datasketches functions available as predefined functions
[
https://issues.apache.org/jira/browse/HIVE-22940?focusedWorklogId=400793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400793
]
ASF GitHub Bot logged work on HIVE-22940:
-
Author: ASF GitHub Bot
Created on: 10/Mar/20 16:03
Start Date: 10/Mar/20 16:03
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #930: HIVE-22940
datasketches functions
URL: https://github.com/apache/hive/pull/930#discussion_r390411646
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/DataSketchesFunctions.java
##
@@ -0,0 +1,218 @@
+package org.apache.hadoop.hive.ql.exec;
+
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
+
+public class DataSketchesFunctions {
+
+ private static final String DATA_TO_SKETCH = "datatosketch";
+ private static final String SKETCH_TO_ESTIMATE_WITH_ERROR_BOUNDS =
"sketchToEstimateWithErrorBounds";
+ // FIXME: consider to rename it to simply "estimate" or "evaluate" - in case
of the counting sketches the "sketchto..." doesnt add value
+ private static final String SKETCH_TO_ESTIMATE = "sketchToEstimate";
Review comment:
I think it might make sense to remove the `get_` prefix - but in that case
remove it from all of the functions...
what do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400793)
Time Spent: 1.5h (was: 1h 20m)
> Make the datasketches functions available as predefined functions
> --
>
> Key: HIVE-22940
> URL: https://issues.apache.org/jira/browse/HIVE-22940
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22940) Make the datasketches functions available as predefined functions
[
https://issues.apache.org/jira/browse/HIVE-22940?focusedWorklogId=400792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400792
]
ASF GitHub Bot logged work on HIVE-22940:
-
Author: ASF GitHub Bot
Created on: 10/Mar/20 16:03
Start Date: 10/Mar/20 16:03
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #930: HIVE-22940
datasketches functions
URL: https://github.com/apache/hive/pull/930#discussion_r390410296
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/DataSketchesFunctions.java
##
@@ -0,0 +1,218 @@
+package org.apache.hadoop.hive.ql.exec;
+
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
+
+public class DataSketchesFunctions {
+
+ private static final String DATA_TO_SKETCH = "datatosketch";
Review comment:
this was something which was really consistent; but I think:
* `to_sketch`
* `gen_sketch`
* `build_sketch`
would be probably better - I changed it to `gen_sketch` for now...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400792)
Time Spent: 1h 20m (was: 1h 10m)
> Make the datasketches functions available as predefined functions
> --
>
> Key: HIVE-22940
> URL: https://issues.apache.org/jira/browse/HIVE-22940
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22940) Make the datasketches functions available as predefined functions
[
https://issues.apache.org/jira/browse/HIVE-22940?focusedWorklogId=400794=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400794
]
ASF GitHub Bot logged work on HIVE-22940:
-
Author: ASF GitHub Bot
Created on: 10/Mar/20 16:03
Start Date: 10/Mar/20 16:03
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #930: HIVE-22940
datasketches functions
URL: https://github.com/apache/hive/pull/930#discussion_r390417433
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/DataSketchesFunctions.java
##
@@ -0,0 +1,218 @@
+package org.apache.hadoop.hive.ql.exec;
+
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
+
+public class DataSketchesFunctions {
+
+ private static final String DATA_TO_SKETCH = "datatosketch";
+ private static final String SKETCH_TO_ESTIMATE_WITH_ERROR_BOUNDS =
"sketchToEstimateWithErrorBounds";
+ // FIXME: consider to rename it to simply "estimate" or "evaluate" - in case
of the counting sketches the "sketchto..." doesnt add value
+ private static final String SKETCH_TO_ESTIMATE = "sketchToEstimate";
+ private static final String SKETCH_TO_STRING = "sketchToString";
+ // FIXME: probably use simply "union" instead unionSketch?
+ private static final String UNION_SKETCH = "unionSketch";
+ private static final String GET_N = "getN";
+ private static final String GET_CDF = "getCdf";
+ private static final String GET_PMF = "getPmf";
+ private static final String GET_QUANTILES = "GetQuantiles";
+ private static final String GET_QUANTILE = "GetQuantile";
+ private static final String GET_RANK = "getRank";
+ private static final String INTERSECT_SKETCH = "intersection";
+ private static final String EXCLUDE_SKETCH = "exclude";
+ private static final String GET_K = "getK";
+ private static final String GET_FREQUENT_ITEMS = "getFrequentItems";
+ private static final String T_TEST = "TTest";
+ private static final String SKETCH_TO_MEANS = "sketchtomeans";
+ private static final String SKETCH_TO_NUMBER_OF_RETAINED_ENTRIES =
"sketchtonumberofretainedentries";
Review comment:
I right now think that we could change this to:
* `get_n_retained`
* `n_retained`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400794)
Time Spent: 1.5h (was: 1h 20m)
> Make the datasketches functions available as predefined functions
> --
>
> Key: HIVE-22940
> URL: https://issues.apache.org/jira/browse/HIVE-22940
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22940) Make the datasketches functions available as predefined functions
[
https://issues.apache.org/jira/browse/HIVE-22940?focusedWorklogId=400791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400791
]
ASF GitHub Bot logged work on HIVE-22940:
-
Author: ASF GitHub Bot
Created on: 10/Mar/20 16:03
Start Date: 10/Mar/20 16:03
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #930: HIVE-22940
datasketches functions
URL: https://github.com/apache/hive/pull/930#discussion_r390419219
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/DataSketchesFunctions.java
##
@@ -0,0 +1,218 @@
+package org.apache.hadoop.hive.ql.exec;
+
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver2;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
+
+public class DataSketchesFunctions {
+
+ private static final String DATA_TO_SKETCH = "datatosketch";
+ private static final String SKETCH_TO_ESTIMATE_WITH_ERROR_BOUNDS =
"sketchToEstimateWithErrorBounds";
+ // FIXME: consider to rename it to simply "estimate" or "evaluate" - in case
of the counting sketches the "sketchto..." doesnt add value
+ private static final String SKETCH_TO_ESTIMATE = "sketchToEstimate";
+ private static final String SKETCH_TO_STRING = "sketchToString";
+ // FIXME: probably use simply "union" instead unionSketch?
+ private static final String UNION_SKETCH = "unionSketch";
+ private static final String GET_N = "getN";
+ private static final String GET_CDF = "getCdf";
+ private static final String GET_PMF = "getPmf";
+ private static final String GET_QUANTILES = "GetQuantiles";
+ private static final String GET_QUANTILE = "GetQuantile";
+ private static final String GET_RANK = "getRank";
+ private static final String INTERSECT_SKETCH = "intersection";
+ private static final String EXCLUDE_SKETCH = "exclude";
+ private static final String GET_K = "getK";
+ private static final String GET_FREQUENT_ITEMS = "getFrequentItems";
+ private static final String T_TEST = "TTest";
+ private static final String SKETCH_TO_MEANS = "sketchtomeans";
+ private static final String SKETCH_TO_NUMBER_OF_RETAINED_ENTRIES =
"sketchtonumberofretainedentries";
+ private static final String SKETCH_TO_QUANTILES_SKETCH =
"sketchToQuantilesSketch";
+ private static final String SKETCH_TO_VALUES = "sketchToValues";
+ private static final String SKETCH_TO_VARIANCES = "sketchToVariances";
+ private static final String SKETCH_TO_PERCENTILE = "sketchToPercentile";
+ private static final String UNION_SKETCH1 = "unionSketch1";
+ private static final String INTERSECT_SKETCH1 = "intersect";
+
+ private final Registry system;
+
+ public DataSketchesFunctions(Registry system) {
+this.system = system;
+ }
+
+ public static void register(Registry system) {
+DataSketchesFunctions dsf = new DataSketchesFunctions(system);
+// FIXME: what this should be approx, ds ... other?
+String prefix = "ds";
Review comment:
@jcamachor what do you think about this "ds" prefix ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400791)
Time Spent: 1h 20m (was: 1h 10m)
> Make the datasketches functions available as predefined functions
> --
>
> Key: HIVE-22940
> URL: https://issues.apache.org/jira/browse/HIVE-22940
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[
https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23005:
--
Attachment: HIVE-23005.1.patch
> Consider Default JDBC Fetch Size From HS2
> -
>
> Key: HIVE-23005
> URL: https://issues.apache.org/jira/browse/HIVE-23005
> Project: Hive
> Issue Type: Sub-task
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23005.1.patch
>
>
> # Create two fetch sizes: and "init" fetch size and a "default" fetch size.
> The "init" fetch size comes from the JDBC connection string _fetchSize_ (if
> present) and the "default" fetch size comes from
> _HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE_ the server response to the
> open session request.
> # When a {{Statement}} is created, its starting fetch size is the "init"
> fetch size (may be 0)
> # Manually setting the fetch size on the {{Statement}} to 0, sets the fetch
> size to be the server default
> Setting to zero defaults to the server's instructed default and adheres to:
> bq. If the value specified is zero, then the hint is ignored. The default
> value is zero.
> That is to say, if the fetch size is 0, the default from the server is used,
> otherwise the user can pass a 'hint' and that will be the number of rows
> fetched instead.
> https://docs.oracle.com/javase/8/docs/api/java/sql/Statement.html#setFetchSize-int-
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[
https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23005:
--
Attachment: (was: HIVE-23005.1.patch)
> Consider Default JDBC Fetch Size From HS2
> -
>
> Key: HIVE-23005
> URL: https://issues.apache.org/jira/browse/HIVE-23005
> Project: Hive
> Issue Type: Sub-task
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> # Create two fetch sizes: and "init" fetch size and a "default" fetch size.
> The "init" fetch size comes from the JDBC connection string _fetchSize_ (if
> present) and the "default" fetch size comes from
> _HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE_ the server response to the
> open session request.
> # When a {{Statement}} is created, its starting fetch size is the "init"
> fetch size (may be 0)
> # Manually setting the fetch size on the {{Statement}} to 0, sets the fetch
> size to be the server default
> Setting to zero defaults to the server's instructed default and adheres to:
> bq. If the value specified is zero, then the hint is ignored. The default
> value is zero.
> That is to say, if the fetch size is 0, the default from the server is used,
> otherwise the user can pass a 'hint' and that will be the number of rows
> fetched instead.
> https://docs.oracle.com/javase/8/docs/api/java/sql/Statement.html#setFetchSize-int-
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[
https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23005:
--
Description:
# Create two fetch sizes: and "init" fetch size and a "default" fetch size. The
"init" fetch size comes from the JDBC connection string _fetchSize_ (if
present) and the "default" fetch size comes from
_HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE_ the server response to the
open session request.
# When a {{Statement}} is created, its starting fetch size is the "init" fetch
size (may be 0)
# Manually setting the fetch size on the {{Statement}} to 0, sets the fetch
size to be the server default
Setting to zero defaults to the server's instructed default and adheres to:
bq. If the value specified is zero, then the hint is ignored. The default value
is zero.
That is to say, if the fetch size is 0, the default from the server is used,
otherwise the user can pass a 'hint' and that will be the number of rows
fetched instead.
https://docs.oracle.com/javase/8/docs/api/java/sql/Statement.html#setFetchSize-int-
was:
# Create two fetch sizes: and "init" fetch size and a "default" fetch size. The
"init" fetch size comes from the JDBC connection string _fetchSize_ (if
present) and the "default" fetch size comes from
_HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE_ the server response to the
open session request.
# When a {{Statement}} is created, its starting fetch size is the "init" fetch
size (may be 0)
# Manually setting the fetch size on the {{Statement}} to 0, sets the fetch
size to be the server default
> Consider Default JDBC Fetch Size From HS2
> -
>
> Key: HIVE-23005
> URL: https://issues.apache.org/jira/browse/HIVE-23005
> Project: Hive
> Issue Type: Sub-task
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23005.1.patch
>
>
> # Create two fetch sizes: and "init" fetch size and a "default" fetch size.
> The "init" fetch size comes from the JDBC connection string _fetchSize_ (if
> present) and the "default" fetch size comes from
> _HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE_ the server response to the
> open session request.
> # When a {{Statement}} is created, its starting fetch size is the "init"
> fetch size (may be 0)
> # Manually setting the fetch size on the {{Statement}} to 0, sets the fetch
> size to be the server default
> Setting to zero defaults to the server's instructed default and adheres to:
> bq. If the value specified is zero, then the hint is ignored. The default
> value is zero.
> That is to say, if the fetch size is 0, the default from the server is used,
> otherwise the user can pass a 'hint' and that will be the number of rows
> fetched instead.
> https://docs.oracle.com/javase/8/docs/api/java/sql/Statement.html#setFetchSize-int-
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22992) ZkRegistryBase caching mechanism only caches the first instance
[ https://issues.apache.org/jira/browse/HIVE-22992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-22992: --- Attachment: HIVE-22992.03.patch > ZkRegistryBase caching mechanism only caches the first instance > --- > > Key: HIVE-22992 > URL: https://issues.apache.org/jira/browse/HIVE-22992 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Minor > Attachments: HIVE-22992.01.patch, HIVE-22992.02.patch, > HIVE-22992.03.patch > > > ZkRegistryBase caching mechanism only caches the first instance of the llap > node running on the same host. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[
https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23005:
--
Description:
# Create two fetch sizes: and "init" fetch size and a "default" fetch size. The
"init" fetch size comes from the JDBC connection string _fetchSize_ (if
present) and the "default" fetch size comes from
_HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE_ the server response to the
open session request.
# When a {{Statement}} is created, its starting fetch size is the "init" fetch
size (may be 0)
# Manually setting the fetch size on the {{Statement}} to 0, sets the fetch
size to be the server default
> Consider Default JDBC Fetch Size From HS2
> -
>
> Key: HIVE-23005
> URL: https://issues.apache.org/jira/browse/HIVE-23005
> Project: Hive
> Issue Type: Sub-task
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23005.1.patch
>
>
> # Create two fetch sizes: and "init" fetch size and a "default" fetch size.
> The "init" fetch size comes from the JDBC connection string _fetchSize_ (if
> present) and the "default" fetch size comes from
> _HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE_ the server response to the
> open session request.
> # When a {{Statement}} is created, its starting fetch size is the "init"
> fetch size (may be 0)
> # Manually setting the fetch size on the {{Statement}} to 0, sets the fetch
> size to be the server default
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23000) Improve travis.yml
[
https://issues.apache.org/jira/browse/HIVE-23000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056050#comment-17056050
]
Zoltan Haindrich commented on HIVE-23000:
-
+1
> Improve travis.yml
> --
>
> Key: HIVE-23000
> URL: https://issues.apache.org/jira/browse/HIVE-23000
> Project: Hive
> Issue Type: Improvement
>Reporter: Philipp Dallig
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23000.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> travis.yml used an [unsupported operating system|http://example.com/] (trusty
> - 14.04), updated to bionic (18.04).
> Switched to openjdk8, because OracleJDK8 gets no updates (current version
> [1.8.0_151|https://travis-ci.org/apache/hive/builds/660081028#L441])
> Removed a unnecessary workaround, because of a travis fix in 2017.
> {{install: skip}} is the right way to [skip the installation
> phase|https://docs.travis-ci.com/user/job-lifecycle/#skipping-the-installation-phase]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[ https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23005: -- Status: Patch Available (was: Open) > Consider Default JDBC Fetch Size From HS2 > - > > Key: HIVE-23005 > URL: https://issues.apache.org/jira/browse/HIVE-23005 > Project: Hive > Issue Type: Sub-task > Components: JDBC >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23005.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[ https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23005: -- Attachment: HIVE-23005.1.patch > Consider Default JDBC Fetch Size From HS2 > - > > Key: HIVE-23005 > URL: https://issues.apache.org/jira/browse/HIVE-23005 > Project: Hive > Issue Type: Sub-task > Components: JDBC >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23005.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[ https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23005: -- Status: Open (was: Patch Available) > Consider Default JDBC Fetch Size From HS2 > - > > Key: HIVE-23005 > URL: https://issues.apache.org/jira/browse/HIVE-23005 > Project: Hive > Issue Type: Sub-task > Components: JDBC >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[ https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23005: -- Attachment: (was: HIVE-23005.1.patch) > Consider Default JDBC Fetch Size From HS2 > - > > Key: HIVE-23005 > URL: https://issues.apache.org/jira/browse/HIVE-23005 > Project: Hive > Issue Type: Sub-task > Components: JDBC >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[ https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23005: -- Status: Patch Available (was: Open) > Consider Default JDBC Fetch Size From HS2 > - > > Key: HIVE-23005 > URL: https://issues.apache.org/jira/browse/HIVE-23005 > Project: Hive > Issue Type: Sub-task > Components: JDBC >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[ https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23005: -- Attachment: HIVE-23005.1.patch > Consider Default JDBC Fetch Size From HS2 > - > > Key: HIVE-23005 > URL: https://issues.apache.org/jira/browse/HIVE-23005 > Project: Hive > Issue Type: Sub-task > Components: JDBC >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23005.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22901) Variable substitution can lead to OOM on circular references
[
https://issues.apache.org/jira/browse/HIVE-22901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056040#comment-17056040
]
Hive QA commented on HIVE-22901:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} common: The patch generated 4 new + 382 unchanged - 0
fixed = 386 total (was 382) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21052/dev-support/hive-personality.sh
|
| git revision | master / 1e15791 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21052/yetus/diff-checkstyle-common.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21052/yetus/patch-asflicense-problems.txt
|
| modules | C: common itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21052/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Variable substitution can lead to OOM on circular references
>
>
> Key: HIVE-22901
> URL: https://issues.apache.org/jira/browse/HIVE-22901
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.2
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-22901.1.patch, HIVE-22901.2.patch,
> HIVE-22901.2.patch, HIVE-22901.2.patch, HIVE-22901.3.patch
>
>
> {{SystemVariables#substitute()}} is dealing with circular references between
> variables by only doing the substitution 40 times by default. If the
> substituted part is sufficiently large though, it's possible that the
> substitution will produce a string bigger than the heap size within the 40
> executions.
> Take the following test case that fails with OOM in current master (third
>
[jira] [Assigned] (HIVE-23005) Consider Default JDBC Fetch Size From HS2
[ https://issues.apache.org/jira/browse/HIVE-23005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-23005: - > Consider Default JDBC Fetch Size From HS2 > - > > Key: HIVE-23005 > URL: https://issues.apache.org/jira/browse/HIVE-23005 > Project: Hive > Issue Type: Sub-task > Components: JDBC >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
