[jira] [Commented] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729851#comment-16729851
]
Hive QA commented on HIVE-21073:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12953169/HIVE-21073.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15759 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15442/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15442/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15442/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12953169 - PreCommit-HIVE-Build
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-21073.1.patch, HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729812#comment-16729812
]
Hive QA commented on HIVE-21073:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} ql generated 0 new + 2309 unchanged - 2 fixed = 2309
total (was 2311) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15442/dev-support/hive-personality.sh
|
| git revision | master / 926c1e8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15442/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-21073.1.patch, HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21073:
---
Status: Patch Available (was: Open)
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.1, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-21073.1.patch, HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21073:
---
Status: Open (was: Patch Available)
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.1, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-21073.1.patch, HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21073:
---
Attachment: HIVE-21073.1.patch
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-21073.1.patch, HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729781#comment-16729781
]
Hive QA commented on HIVE-21073:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12953162/HIVE-21073.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15759 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=262)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=262)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=262)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15441/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15441/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15441/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12953162 - PreCommit-HIVE-Build
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729760#comment-16729760
]
Hive QA commented on HIVE-21073:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} ql generated 0 new + 2309 unchanged - 2 fixed = 2309
total (was 2311) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15441/dev-support/hive-personality.sh
|
| git revision | master / 926c1e8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15441/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729759#comment-16729759
]
Andrew Sherman commented on HIVE-21073:
---
I see, thanks [~belugabehr]
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[
https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729749#comment-16729749
]
Hive QA commented on HIVE-21033:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12953155/HIVE-21033.5.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15759 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15440/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15440/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15440/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12953155 - PreCommit-HIVE-Build
> Forgetting to close operation cuts off any more HiveServer2 output
> --
>
> Key: HIVE-21033
> URL: https://issues.apache.org/jira/browse/HIVE-21033
> Project: Hive
> Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch,
> HIVE-21033.4.patch, HIVE-21033.5.patch, HIVE-21033.patch
>
>
> We had a custom client that did not handle closing the operations, until the
> end of the session. it is a mistake in the client, but it reveals kind of a
> vulnerability in HiveServer2
> This happens if you have a session with (1) HiveCommandOperation and (2)
> SQLOperation and don't close them right after. For example a session that
> does the operations (set a=b; select * from foobar; ).
> When SQLOperation runs , it set SessionState.out and err to be System.out and
> System.err . Ref:
> [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139]
> Then the client closes the session, or disconnects which triggers
> closeSession() on the Thrift side. In this case, the closeSession closes all
> the operations, starting with HiveCommandOperation. This closes all the
> streams, which is System.out and System.err as set by SQLOperation earlier.
> Ref:
> [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101]
>
> After this, no more HiveServer2 output appears as System.out and System.err
> are closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729748#comment-16729748
]
BELUGA BEHR commented on HIVE-21073:
[~asherman] Ya, exactly, and they return a String object which is then being
explicitly wrapped here in a new String object.
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[
https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729741#comment-16729741
]
Hive QA commented on HIVE-21033:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} cli in master has 13 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} hcatalog/core in master has 30 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} common: The patch generated 2 new + 1 unchanged - 2
fixed = 3 total (was 3) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 3 new + 58 unchanged - 3 fixed
= 61 total (was 61) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} service: The patch generated 1 new + 19 unchanged - 2
fixed = 20 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15440/dev-support/hive-personality.sh
|
| git revision | master / 926c1e8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Updated] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21073:
---
Attachment: HIVE-21073.1.patch
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21073:
---
Status: Patch Available (was: Open)
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.1, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HIVE-21073:
--
Assignee: BELUGA BEHR
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729732#comment-16729732
]
Andrew Sherman commented on HIVE-21073:
---
>From Java 8 those string concatenations [are actually done with a StringBuffer
>under the
>covers|http://www.pellegrino.link/2015/08/22/string-concatenation-with-java-8.html].
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21073:
---
Status: Patch Available (was: Open)
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.1, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21073:
---
Attachment: (was: HIVE-21073.1.patch)
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21073:
---
Attachment: HIVE-21073.1.patch
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21073:
---
Status: Open (was: Patch Available)
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.1, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Attachment: HIVE-21033.5.patch > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, > HIVE-21033.4.patch, HIVE-21033.5.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[
https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729667#comment-16729667
]
Hive QA commented on HIVE-21033:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12953150/HIVE-21033.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15439/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15439/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15439/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Closeable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/AutoCloseable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Flushable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(javax/xml/bind/annotation/XmlRootElement.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar(org/apache/commons/exec/ExecuteException.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/security/PrivilegedExceptionAction.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ExecutionException.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/TimeoutException.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/fs/FileSystem.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShimsSecure.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/ShimLoader.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShims.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShims$WebHCatJTShim.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/util/ToolRunner.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/CancellationException.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/RejectedExecutionException.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/SynchronousQueue.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ThreadPoolExecutor.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/TimeUnit.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/Future.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/conf/Configured.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/io/NullWritable.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/io/Text.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapred/JobClient.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapred/JobConf.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapreduce/Job.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapreduce/JobID.class)]]
[loading
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729655#comment-16729655 ] Szehon Ho commented on HIVE-21033: -- Fix checkstyle > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, > HIVE-21033.4.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Attachment: HIVE-21033.4.patch > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, > HIVE-21033.4.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21071) Improve getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729623#comment-16729623
]
BELUGA BEHR commented on HIVE-21071:
{{./ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:2442:
@VisibleForTesting:3: warning: Method length is 161 lines (max allowed is
150).}}
That's not going to be corrected with this patch. That would require a larger
overhaul. Please accept the provided patch for inclusion into the project.
> Improve getInputSummary
> ---
>
> Key: HIVE-21071
> URL: https://issues.apache.org/jira/browse/HIVE-21071
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21071.1.patch
>
>
> There is a global lock in the {{getInptSummary}} code, so it is important
> that it be fast. The current implementation has quite a bit of overhead that
> can be re-engineered.
> For example, the current implementation keeps a map of File Path to
> ContentSummary object. This map is populated by several threads
> concurrently. The method then loops through the map, in a single thread, at
> the end to add up all of the ContentSummary objects and ignores the paths.
> The code can be be re-engineered to not use a map, or a collection at all, to
> store the results and instead just keep a running tally. By keeping a tally,
> there is no {{O\(n)}} operation at the end to perform the addition.
> There are other things can be improved. The method returns an object which
> is never used anywhere, so change method to void return type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21071) Improve getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729623#comment-16729623
]
BELUGA BEHR edited comment on HIVE-21071 at 12/27/18 2:08 PM:
--
Checkstyle: {{./ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:2442:
@VisibleForTesting:3: warning: Method length is 161 lines (max allowed is
150).}}
That's not going to be corrected with this patch. That would require a larger
overhaul. Please accept the provided patch for inclusion into the project.
was (Author: belugabehr):
{{./ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:2442:
@VisibleForTesting:3: warning: Method length is 161 lines (max allowed is
150).}}
That's not going to be corrected with this patch. That would require a larger
overhaul. Please accept the provided patch for inclusion into the project.
> Improve getInputSummary
> ---
>
> Key: HIVE-21071
> URL: https://issues.apache.org/jira/browse/HIVE-21071
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21071.1.patch
>
>
> There is a global lock in the {{getInptSummary}} code, so it is important
> that it be fast. The current implementation has quite a bit of overhead that
> can be re-engineered.
> For example, the current implementation keeps a map of File Path to
> ContentSummary object. This map is populated by several threads
> concurrently. The method then loops through the map, in a single thread, at
> the end to add up all of the ContentSummary objects and ignores the paths.
> The code can be be re-engineered to not use a map, or a collection at all, to
> store the results and instead just keep a running tally. By keeping a tally,
> there is no {{O\(n)}} operation at the end to perform the addition.
> There are other things can be improved. The method returns an object which
> is never used anywhere, so change method to void return type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21071) Improve getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21071:
---
Description:
There is a global lock in the {{getInptSummary}} code, so it is important that
it be fast. The current implementation has quite a bit of overhead that can be
re-engineered.
For example, the current implementation keeps a map of File Path to
ContentSummary object. This map is populated by several threads concurrently.
The method then loops through the map, in a single thread, at the end to add up
all of the ContentSummary objects and ignores the paths. The code can be be
re-engineered to not use a map, or a collection at all, to store the results
and instead just keep a running tally. By keeping a tally, there is no
{{O\(n)}} operation at the end to perform the addition.
There are other things can be improved. The method returns an object which is
never used anywhere, so change method to void return type.
was:
There is a global lock in the {{getInptSummary}} code, so it is important that
it be fast. The current implementation has quite a bit of overhead that can be
re-engineered.
For example, the current implementation keeps a map of File Path to
ContentSummary object. This map is populated by several threads concurrently.
The method then loops through the map, in a single thread, at the end to add up
all of the ContentSummary objects and ignores the paths. The code can be be
re-engineered to not use a map, or a collection at all, to store the results
and instead just keep a running tally. By keeping a tally, there is no O(n)
operation at the end to perform the addition.
There are other things can be improved. The method returns an object which is
never used anywhere, so change method to void return type.
> Improve getInputSummary
> ---
>
> Key: HIVE-21071
> URL: https://issues.apache.org/jira/browse/HIVE-21071
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HIVE-21071.1.patch
>
>
> There is a global lock in the {{getInptSummary}} code, so it is important
> that it be fast. The current implementation has quite a bit of overhead that
> can be re-engineered.
> For example, the current implementation keeps a map of File Path to
> ContentSummary object. This map is populated by several threads
> concurrently. The method then loops through the map, in a single thread, at
> the end to add up all of the ContentSummary objects and ignores the paths.
> The code can be be re-engineered to not use a map, or a collection at all, to
> store the results and instead just keep a running tally. By keeping a tally,
> there is no {{O\(n)}} operation at the end to perform the addition.
> There are other things can be improved. The method returns an object which
> is never used anywhere, so change method to void return type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729603#comment-16729603 ] Szehon Ho commented on HIVE-21033: -- Patch creates new kind of PrintStream for SessionState, which I call 'SessionStream' to be able to handle not closing of System.out and System.err. As consequence, a lot of test classes need to change a little bit. > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729601#comment-16729601 ] Szehon Ho commented on HIVE-21033: -- [~aihuaxu] Do you have time to look at this patch? > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[
https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729543#comment-16729543
]
Hive QA commented on HIVE-21033:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12953132/HIVE-21033.3.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15759 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15438/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15438/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15438/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12953132 - PreCommit-HIVE-Build
> Forgetting to close operation cuts off any more HiveServer2 output
> --
>
> Key: HIVE-21033
> URL: https://issues.apache.org/jira/browse/HIVE-21033
> Project: Hive
> Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>Priority: Major
> Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, HIVE-21033.patch
>
>
> We had a custom client that did not handle closing the operations, until the
> end of the session. it is a mistake in the client, but it reveals kind of a
> vulnerability in HiveServer2
> This happens if you have a session with (1) HiveCommandOperation and (2)
> SQLOperation and don't close them right after. For example a session that
> does the operations (set a=b; select * from foobar; ).
> When SQLOperation runs , it set SessionState.out and err to be System.out and
> System.err . Ref:
> [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139]
> Then the client closes the session, or disconnects which triggers
> closeSession() on the Thrift side. In this case, the closeSession closes all
> the operations, starting with HiveCommandOperation. This closes all the
> streams, which is System.out and System.err as set by SQLOperation earlier.
> Ref:
> [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101]
>
> After this, no more HiveServer2 output appears as System.out and System.err
> are closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[
https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729533#comment-16729533
]
Hive QA commented on HIVE-21033:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} cli in master has 13 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} hcatalog/core in master has 30 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} common: The patch generated 5 new + 1 unchanged - 2
fixed = 6 total (was 3) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 4 new + 58 unchanged - 3 fixed
= 62 total (was 61) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} service: The patch generated 2 new + 19 unchanged - 2
fixed = 21 total (was 21) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hcatalog/core: The patch generated 1 new + 21
unchanged - 0 fixed = 22 total (was 21) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 19
unchanged - 0 fixed = 20 total (was 19) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} itests/util: The patch generated 1 new + 53 unchanged
- 0 fixed = 54 total (was 53) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
|
[jira] [Commented] (HIVE-21072) NPE when running partitioned CTAS statements
[
https://issues.apache.org/jira/browse/HIVE-21072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729532#comment-16729532
]
ASF GitHub Bot commented on HIVE-21072:
---
GitHub user viirya opened a pull request:
https://github.com/apache/hive/pull/510
HIVE-21072: Fix NPE when running partitioned CTAS statements
HIVE-20241 adds support of partitioned CTAS statements:
```sql
CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
SELECT value, key FROM src where key > 200 and key < 300;{code}
```
However, I've tried this feature by checking out latest branch-3, and
encountered NPE:
```
hive> CREATE TABLE t PARTITIONED BY (part) AS SELECT 1 as id, "a" as part;
FAILED: NullPointerException null
```
For the added `partition_ctas.q`, it is ran when using
`TestMiniLlapLocalCliDriver`. But the problematic code path is not ran if using
`TestMiniLlapLocalCliDriver`. When I go to test it with TestCliDriver manually,
it also throws NullPointerException:
{code}
2018-12-25T05:58:22,221 ERROR [a96009a7-3dda-4d95-9536-e2e16d976856 main]
ql.Driver: FAILED: NullPointerException null
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.usePartitionColumns(GenMapRedUtils.java:2103)
at
org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.createMRWorkForMergingFiles(GenMapRedUtils.java:1323)
at
org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.process(GenMRFileSink1.java:113)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
at
org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:54)
at
org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
at
org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
at
org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
at
org.apache.hadoop.hive.ql.parse.MapReduceCompiler.generateTaskTree(MapReduceCompiler.java:323)
at
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:244)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12503)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:357)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:166)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1854)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1801)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1796)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
```
This patch fixes the NPE issue.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/hive HIVE-21072
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hive/pull/510.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #510
commit 20c397d2a54179bc42fd604f93e6559f0a80e4e6
Author: Liang-Chi Hsieh
Date: 2018-12-27T10:28:17Z
Fix NPE when running partitioned CTAS statements.
> NPE when running partitioned CTAS statements
>
>
> Key: HIVE-21072
> URL: https://issues.apache.org/jira/browse/HIVE-21072
> Project: Hive
> Issue Type: Bug
>Reporter: Liang-Chi Hsieh
>Priority: Major
> Labels: pull-request-available
>
> HIVE-20241 adds support of partitioned CTAS statements:
> {code:sql}
> CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
> SELECT value, key FROM src where key > 200 and key < 300;{code}
>
> However, I've tried this feature by checking out latest branch-3, and
>
[jira] [Updated] (HIVE-21072) NPE when running partitioned CTAS statements
[
https://issues.apache.org/jira/browse/HIVE-21072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-21072:
--
Labels: pull-request-available (was: )
> NPE when running partitioned CTAS statements
>
>
> Key: HIVE-21072
> URL: https://issues.apache.org/jira/browse/HIVE-21072
> Project: Hive
> Issue Type: Bug
>Reporter: Liang-Chi Hsieh
>Priority: Major
> Labels: pull-request-available
>
> HIVE-20241 adds support of partitioned CTAS statements:
> {code:sql}
> CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
> SELECT value, key FROM src where key > 200 and key < 300;{code}
>
> However, I've tried this feature by checking out latest branch-3, and
> encountered NPE:
> {code:java}
> hive> CREATE TABLE t PARTITIONED BY (part) AS SELECT 1 as id, "a" as part;
> FAILED: NullPointerException null
> {code}
> I also ran the query test partition_ctas.q. The test passes when using
> TestMiniLlapLocalCliDriver, but when I go to test it with TestCliDriver
> manually, it also throws NullPointerException:
> {code}
> 2018-12-25T05:58:22,221 ERROR [a96009a7-3dda-4d95-9536-e2e16d976856 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.usePartitionColumns(GenMapRedUtils.java:2103)
> at
> org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.createMRWorkForMergingFiles(GenMapRedUtils.java:1323)
> at
> org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.process(GenMRFileSink1.java:113)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:54)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.generateTaskTree(MapReduceCompiler.java:323)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:244)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12503)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:357)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:166)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1854)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1801)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1796)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Attachment: (was: HIVE-21033.2.patch) > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Attachment: HIVE-21033.2.patch > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.2.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-21033: - Attachment: HIVE-21033.3.patch > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16976) DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN
[
https://issues.apache.org/jira/browse/HIVE-16976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729475#comment-16729475
]
Hive QA commented on HIVE-16976:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12953122/HIVE-16976.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 15759 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_gby]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_gby]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_limit]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_prod_1]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_groupingset_bug]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select]
(batchId=170)
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=258)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15437/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15437/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15437/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12953122 - PreCommit-HIVE-Build
> DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN
>
>
> Key: HIVE-16976
> URL: https://issues.apache.org/jira/browse/HIVE-16976
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.1.1, 3.0.0
>Reporter: Gopal V
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-16976.1.patch, HIVE-16976.2.patch,
> HIVE-16976.3.patch
>
>
> Tez DPP does not kick in for scenarios where a user wants to run a comparison
> clause instead of a JOIN/IN clause.
> {code}
> explain select count(1) from store_sales where ss_sold_date_sk > (select
> max(d_Date_sk) from date_dim where d_year = 2017);
> Warning: Map Join MAPJOIN[21][bigTable=?] in task 'Map 1' is a cross product
> OK
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Reducer 4 (BROADCAST_EDGE)
> Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
> Reducer 4 <- Map 3 (CUSTOM_SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 2 vectorized, llap
> File Output Operator [FS_36]
> Group By Operator [GBY_35] (rows=1 width=8)
> Output:["_col0"],aggregations:["count(VALUE._col0)"]
> <-Map 1 [CUSTOM_SIMPLE_EDGE] vectorized, llap
> PARTITION_ONLY_SHUFFLE [RS_34]
> Group By Operator [GBY_33] (rows=1 width=8)
> Output:["_col0"],aggregations:["count(1)"]
> Select Operator [SEL_32] (rows=9600142089 width=16)
> Filter Operator [FIL_31] (rows=9600142089 width=16)
> predicate:(_col0 > _col1)
> Map Join Operator [MAPJOIN_30] (rows=28800426268 width=16)
> Conds:(Inner),Output:["_col0","_col1"]
> <-Reducer 4 [BROADCAST_EDGE] vectorized, llap
> BROADCAST [RS_28]
> Group By Operator [GBY_27] (rows=1 width=8)
> Output:["_col0"],aggregations:["max(VALUE._col0)"]
> <-Map 3 [CUSTOM_SIMPLE_EDGE] vectorized, llap
> PARTITION_ONLY_SHUFFLE [RS_26]
> Group By Operator [GBY_25] (rows=1 width=8)
> Output:["_col0"],aggregations:["max(d_date_sk)"]
> Select Operator [SEL_24] (rows=652 width=12)
> Output:["d_date_sk"]
> Filter Operator [FIL_23] (rows=652 width=12)
>
[jira] [Commented] (HIVE-16976) DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN
[
https://issues.apache.org/jira/browse/HIVE-16976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729446#comment-16729446
]
Hive QA commented on HIVE-16976:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 23 new + 58 unchanged - 3
fixed = 81 total (was 61) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
52s{color} | {color:red} ql generated 1 new + 2311 unchanged - 0 fixed = 2312
total (was 2311) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Possible null pointer dereference of table in
org.apache.hadoop.hive.ql.optimizer.DynamicPartitionPruningOptimization.process(Node,
Stack, NodeProcessorCtx, Object[]) Dereferenced at
DynamicPartitionPruningOptimization.java:table in
org.apache.hadoop.hive.ql.optimizer.DynamicPartitionPruningOptimization.process(Node,
Stack, NodeProcessorCtx, Object[]) Dereferenced at
DynamicPartitionPruningOptimization.java:[line 192] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15437/dev-support/hive-personality.sh
|
| git revision | master / 926c1e8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15437/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15437/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15437/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN
>
>
> Key: HIVE-16976
> URL: https://issues.apache.org/jira/browse/HIVE-16976
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.1.1, 3.0.0
>Reporter: Gopal V
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-16976.1.patch, HIVE-16976.2.patch,
> HIVE-16976.3.patch
>
>
> Tez DPP does not kick in for scenarios where a user wants to run a comparison
> clause instead of a JOIN/IN clause.
> {code}
> explain select count(1) from store_sales where ss_sold_date_sk > (select
> max(d_Date_sk) from date_dim where d_year = 2017);
> Warning: Map Join MAPJOIN[21][bigTable=?] in task 'Map 1' is a cross product
> OK
> Plan optimized by CBO.
> Vertex dependency
[jira] [Updated] (HIVE-16976) DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN
[
https://issues.apache.org/jira/browse/HIVE-16976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Jaiswal updated HIVE-16976:
--
Attachment: HIVE-16976.3.patch
> DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN
>
>
> Key: HIVE-16976
> URL: https://issues.apache.org/jira/browse/HIVE-16976
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.1.1, 3.0.0
>Reporter: Gopal V
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-16976.1.patch, HIVE-16976.2.patch,
> HIVE-16976.3.patch
>
>
> Tez DPP does not kick in for scenarios where a user wants to run a comparison
> clause instead of a JOIN/IN clause.
> {code}
> explain select count(1) from store_sales where ss_sold_date_sk > (select
> max(d_Date_sk) from date_dim where d_year = 2017);
> Warning: Map Join MAPJOIN[21][bigTable=?] in task 'Map 1' is a cross product
> OK
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Reducer 4 (BROADCAST_EDGE)
> Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
> Reducer 4 <- Map 3 (CUSTOM_SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 2 vectorized, llap
> File Output Operator [FS_36]
> Group By Operator [GBY_35] (rows=1 width=8)
> Output:["_col0"],aggregations:["count(VALUE._col0)"]
> <-Map 1 [CUSTOM_SIMPLE_EDGE] vectorized, llap
> PARTITION_ONLY_SHUFFLE [RS_34]
> Group By Operator [GBY_33] (rows=1 width=8)
> Output:["_col0"],aggregations:["count(1)"]
> Select Operator [SEL_32] (rows=9600142089 width=16)
> Filter Operator [FIL_31] (rows=9600142089 width=16)
> predicate:(_col0 > _col1)
> Map Join Operator [MAPJOIN_30] (rows=28800426268 width=16)
> Conds:(Inner),Output:["_col0","_col1"]
> <-Reducer 4 [BROADCAST_EDGE] vectorized, llap
> BROADCAST [RS_28]
> Group By Operator [GBY_27] (rows=1 width=8)
> Output:["_col0"],aggregations:["max(VALUE._col0)"]
> <-Map 3 [CUSTOM_SIMPLE_EDGE] vectorized, llap
> PARTITION_ONLY_SHUFFLE [RS_26]
> Group By Operator [GBY_25] (rows=1 width=8)
> Output:["_col0"],aggregations:["max(d_date_sk)"]
> Select Operator [SEL_24] (rows=652 width=12)
> Output:["d_date_sk"]
> Filter Operator [FIL_23] (rows=652 width=12)
> predicate:(d_year = 2017)
> TableScan [TS_2] (rows=73049 width=12)
>
> tpcds_bin_partitioned_newschema_orc_1@date_dim,date_dim,Tbl:COMPLETE,Col:COMPLETE,Output:["d_date_sk","d_year"]
> <-Select Operator [SEL_29] (rows=28800426268 width=8)
> Output:["_col0"]
> TableScan [TS_0] (rows=28800426268 width=172)
>
> tpcds_bin_partitioned_newschema_orc_1@store_sales,store_sales,Tbl:COMPLETE,Col:COMPLETE
> {code}
> The SyntheticJoinPredicate is only injected for equi joins, not for < or >
> scalar subqueries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
