[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981770#comment-15981770 ] Sahil Takiar commented on HIVE-7224: This didn't go into the 2.2 release, seems its going into the 2.3 release. I've updated the wiki to reflect this: https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-BeelineCommandOptions > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch, HIVE-7224.4.patch, HIVE-7224.5.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532054#comment-15532054 ] Lefty Leverenz commented on HIVE-7224: -- Done. Welcome to the Hive wiki team, [~stakiar]. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch, HIVE-7224.4.patch, HIVE-7224.5.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531225#comment-15531225 ] Sahil Takiar commented on HIVE-7224: Hey [~leftylev] could you grant me permission to edit the wiki page? Username is "stakiar" > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch, HIVE-7224.4.patch, HIVE-7224.5.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528498#comment-15528498 ] Lefty Leverenz commented on HIVE-7224: -- Doc note: The change of default for --incremental needs to be documented in the wiki for Hive 2.2.0. * [HiveServer2 Clients -- Beeline Command Options | https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-BeelineCommandOptions] Added a TODOC2.2 label. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch, HIVE-7224.4.patch, HIVE-7224.5.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527645#comment-15527645 ] Sergio Peña commented on HIVE-7224: --- Sure. +1 > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch, HIVE-7224.4.patch, HIVE-7224.5.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527624#comment-15527624 ] Sahil Takiar commented on HIVE-7224: [~spena] can this be merged? The change simply sets Beeline's incremental property to true. There were a few subtasks that needed to be completed before this could be merged, but they are all resolved now. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch, HIVE-7224.4.patch, HIVE-7224.5.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[
https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15466087#comment-15466087
]
Hive QA commented on HIVE-7224:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827107/HIVE-7224.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10443 tests
executed
*Failed tests:*
{noformat}
TestBeeLineWithArgs - did not produce a TEST-*.xml file
TestHiveCli - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build//testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build//console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12827107 - PreCommit-HIVE-MASTER-Build
> Set incremental printing to true by default in Beeline
> --
>
> Key: HIVE-7224
> URL: https://issues.apache.org/jira/browse/HIVE-7224
> Project: Hive
> Issue Type: Bug
> Components: Beeline, Clients, JDBC
>Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Sahil Takiar
> Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch,
> HIVE-7224.3.patch, HIVE-7224.4.patch, HIVE-7224.5.patch
>
>
> See HIVE-7221.
> By default beeline tries to buffer the entire output relation before printing
> it on stdout. This can cause OOM when the output relation is large. However,
> beeline has the option of incremental prints. We should keep that as the
> default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[
https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15465971#comment-15465971
]
Hive QA commented on HIVE-7224:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827105/HIVE-7224.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1110/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1110/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1110/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-1110/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at e870ae1 HIVE-14697: Can not access kerberized HS2 Web UI (Chaoyu
Tang, reviewed by Jimmy Xiang)
+ git clean -f -d
Removing ql/src/test/queries/clientpositive/double_precision.q
Removing ql/src/test/results/clientpositive/double_precision.q.out
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at e870ae1 HIVE-14697: Can not access kerberized HS2 Web UI (Chaoyu
Tang, reviewed by Jimmy Xiang)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
error: patch failed:
beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java:71
error: beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java: patch does
not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12827105 - PreCommit-HIVE-MASTER-Build
> Set incremental printing to true by default in Beeline
> --
>
> Key: HIVE-7224
> URL: https://issues.apache.org/jira/browse/HIVE-7224
> Project: Hive
> Issue Type: Bug
> Components: Beeline, Clients, JDBC
>Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Sahil Takiar
> Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch,
> HIVE-7224.3.patch, HIVE-7224.4.patch
>
>
> See HIVE-7221.
> By default beeline tries to buffer the entire output relation before printing
> it on stdout. This can cause OOM when the output relation is large. However,
> beeline has the option of incremental prints. We should keep that as the
> default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15465960#comment-15465960 ] Sahil Takiar commented on HIVE-7224: [~thejas] and [~vgumashta] the subtasks we discussed for this JIRA have been resolved. I've reattached the patch to see if there are new test failures reported by Hive QA. Assuming there are no test failures, are there any other items needed before merging this change? > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch, HIVE-7224.4.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364826#comment-15364826 ] Sahil Takiar commented on HIVE-7224: Sounds good! Thanks for all the suggestions! I just created two sub-tasks to encapsulate this work. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364739#comment-15364739 ] Thejas M Nair commented on HIVE-7224: - [~stakiar] That sounds good to me. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364745#comment-15364745 ] Thejas M Nair commented on HIVE-7224: - You can probably add a method like isWidthFormatted in OutputFormat which TableOutputFormat would then set to true. Then pass the value to the Row constructors. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[
https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364704#comment-15364704
]
Sahil Takiar commented on HIVE-7224:
Thanks for the explanation [~thejas]!
So if my understanding is correct, one approach would be to:
* Modify the logic so that the width is only calculated if
{{TableOutputFormat}} is used
** This will save CPU cycles when the outputformat = a delimiter separated
value format
* Modify {{IncrementalRows}} so that it re-calculates the width every 1000 rows
** I think the advantage of doing it every 1000 rows is just for human
readability purposes
Does that sound reasonable to you?
> Set incremental printing to true by default in Beeline
> --
>
> Key: HIVE-7224
> URL: https://issues.apache.org/jira/browse/HIVE-7224
> Project: Hive
> Issue Type: Bug
> Components: Beeline, Clients, JDBC
>Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Sahil Takiar
> Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch,
> HIVE-7224.3.patch
>
>
> See HIVE-7221.
> By default beeline tries to buffer the entire output relation before printing
> it on stdout. This can cause OOM when the output relation is large. However,
> beeline has the option of incremental prints. We should keep that as the
> default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363688#comment-15363688 ] Thejas M Nair commented on HIVE-7224: - bq. To clarify, what would happen if Beeline uses the first 1000 rows to calculate the width, but then row 1001th is longer than that width. If 1001th row has column larger than the precomputed column width, that particular row would have the column with larger width to accommodate it. This would mean some rows have the separator "|" out of alignment with previous row. However, even if we recompute every 1000 rows, we could still have misalignment every 1000 rows. I looked at where the Row width gets used. The width is getting used only when --outputformat=table (ie TableOutputFormat class) is used . If someone is working on very large outputs, it is likely to be processed by other applications and not human eyes, and a *sv (eg csv) format is likely to be used. It doesn't make any sense waste cpu cycles computing the width in those cases. This is also the case where performance impact of this computation would be more visible. ie, If we can selectively enable buffering and width calculation only for TableOutputFormat, I don't think it would matter if we stick to column width based on first 1000 rows or recompute every 1000 rows. Looks like the Row subclasses have access to beeline options and would be able to determine what the output format is. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[
https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363295#comment-15363295
]
Sahil Takiar commented on HIVE-7224:
{quote}
I think it is better to keep the column width to be consistent across all rows
in the output, rather than adjust it every 1000 rows.
{quote}
Thanks for the input [~thejas]! To clarify, what would happen if Beeline uses
the first 1000 rows to calculate the width, but then row 1001th is longer than
that width. In that case the width would have to be widened for that row,
right? I think that is what the current implementation of {{--incremental}} is
doing.
I like the idea of keeping the width the same for all rows, even in incremental
mode, but I'm not sure how it would work. Thoughts?
> Set incremental printing to true by default in Beeline
> --
>
> Key: HIVE-7224
> URL: https://issues.apache.org/jira/browse/HIVE-7224
> Project: Hive
> Issue Type: Bug
> Components: Beeline, Clients, JDBC
>Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Sahil Takiar
> Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch,
> HIVE-7224.3.patch
>
>
> See HIVE-7221.
> By default beeline tries to buffer the entire output relation before printing
> it on stdout. This can cause OOM when the output relation is large. However,
> beeline has the option of incremental prints. We should keep that as the
> default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363180#comment-15363180 ] Thejas M Nair commented on HIVE-7224: - bq. I think a better approach for the IncrementalRows class would be to instead buffer 1000 rows at a time (by default, this value can be configurable), this way it can optimally set the column width for each set of 1000 rows. I think it is better to keep the column width to be consistent across all rows in the output, rather than adjust it every 1000 rows. The purpose of using optimal display size is primarily to make it easier to read. If we change the column width periodically, it doesn't help with that. Also, if there is some application that assumes the column width remains the same across rows, that might break. ie, beeline could buffer only the first 1000 rows and use that to determine the optimal column width, and then output the rest unbuffered. Not having to buffer the remaining rows could also be better for performance. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363091#comment-15363091 ] Vaibhav Gumashta commented on HIVE-7224: [~stakiar] I agree that buffering would improve usability. Would you like to take a shot at it? > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[
https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363001#comment-15363001
]
Sahil Takiar commented on HIVE-7224:
[~vgumashta] is seems the behavior you are seeing is by design. Looking at
https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-BeelineCommandOptions
the following explanation of the {{--incremental}} property suggests that this
is expected:
{quote}
Defaults to false. When set to false, the entire result set is fetched and
buffered before being displayed, yielding optimal display column sizing. When
set to true, result rows are displayed immediately as they are fetched,
yielding lower latency and memory usage at the price of extra display column
padding. Setting --incremental=true is recommended if you encounter an
OutOfMemory on the client side (due to the fetched result set size being large).
{quote}
So it seems there is a tradeoff when using {{--incremental}} that the column
padding won't be optimal, but memory usage will be better. This makes sense
since the {{IncrementalRows}} class that controls this logic doesn't do any
buffering of rows, so it cannot predict what the optimal column width should be
since it only looks at one row at a time.
I think a better approach for the {{IncrementalRows}} class would be to instead
buffer 1000 rows at a time (by default, this value can be configurable), this
way it can optimally set the column width for each set of 1000 rows. This
shouldn't introduce memory issues unless each row is huge, in which case the
use can decrease the buffer size to say 100 or 10.
What do you think?
> Set incremental printing to true by default in Beeline
> --
>
> Key: HIVE-7224
> URL: https://issues.apache.org/jira/browse/HIVE-7224
> Project: Hive
> Issue Type: Bug
> Components: Beeline, Clients, JDBC
>Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Sahil Takiar
> Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch,
> HIVE-7224.3.patch
>
>
> See HIVE-7221.
> By default beeline tries to buffer the entire output relation before printing
> it on stdout. This can cause OOM when the output relation is large. However,
> beeline has the option of incremental prints. We should keep that as the
> default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362947#comment-15362947 ] Sahil Takiar commented on HIVE-7224: Thanks [~vgumashta] I'll dig into it some more and see what I can find. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Beeline, Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[
https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362826#comment-15362826
]
Vaibhav Gumashta commented on HIVE-7224:
[~stakiar] There seems to be an issue with the column width estimation when
incremental printing is enabled by default. An example:
With incremental true:
{code}
| search_engine | string | |
| exclude_hit | string | |
| hier1 | string | |
| hier2 | string | |
| hier3 | string | |
| hier4 | string | |
| hier5 | string | |
| browser | string | |
| post_browser_height | string | |
| post_browser_width | string | |
| post_cookies| string | |
| post_java_enabled | string | |
| post_persistent_cookie | string | |
| color | string | |
| connection_type | string | |
| country | string | |
| domain | string | |
| post_t_time_info | string | |
| javascript | string | |
| language| string | |
| os | string | |
| plugins | string | |
| resolution | string | |
| last_hit_time_gmt | string | |
| first_hit_time_gmt | string | |
| visit_start_time_gmt | string | |
| last_purchase_time_gmt | string | |
+-+-+-+--+
|col_name |data_type| comment |
+-+-+-+--+
| last_purchase_num | string | |
| first_hit_page_url | string | |
| first_hit_pagename | string | |
| visit_start_page_url | string | |
| visit_start_pagename | string | |
| first_hit_referrer | string | |
| visit_referrer | string | |
| visit_search_engine | string | |
| visit_num | string | |
| visit_page_num | string | |
| prev_page | string | |
| geo_city| string | |
| geo_country | string | |
| geo_region | string | |
| duplicate_purchase | string | |
{code}
With incremental false:
{code}
| search_engine| string| |
| exclude_hit | string| |
| hier1| string| |
| hier2| string| |
| hier3| string| |
| hier4| string| |
| hier5| string| |
| browser | string| |
| post_browser_height | string| |
| post_browser_width | string| |
| post_cookies | string| |
| post_java_enabled| string| |
| post_persistent_cookie | string| |
| color| string| |
| connection_type | string| |
| country | string| |
| domain | string| |
| post_t_time_info | string| |
| javascript | string| |
| language | string| |
| os | string| |
| plugins | string| |
| resolution | string| |
| last_hit_time_gmt| string| |
| first_hit_time_gmt | string| |
| visit_start_time_gmt | string
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361610#comment-15361610 ] Sahil Takiar commented on HIVE-7224: Thanks [~vgumashta]! Let me know if you need any help with the manual testing. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359920#comment-15359920 ] Vaibhav Gumashta commented on HIVE-7224: [~stakiar] I'll do some quick manual testing and post my feedback. I think this looks good to go in. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359914#comment-15359914 ] Vaibhav Gumashta commented on HIVE-7224: Actually it does seem those tests were run and pass now: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/330/testReport/org.apache.hive.beeline/TestBeeLineWithArgs/ > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359913#comment-15359913 ] Vaibhav Gumashta commented on HIVE-7224: [~stakiar] Thanks a lot for taking this up. In the previous iterations, we've had issues with TestBeeLineWithArgs which was failing. From the tests listed here: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/330/testReport, doesn't look like those tests were run. Can you verify? > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358180#comment-15358180 ] Sahil Takiar commented on HIVE-7224: None of these test failures seem related to the given change. All Beeline tests are passing. The new patch I attached is a rebased version of the previous patch. Is there anything else blocking this from being merged? > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[
https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358108#comment-15358108
]
Hive QA commented on HIVE-7224:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813155/HIVE-7224.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 10288 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_deleteAnalyze
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part_all_complex
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_udf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_string_concat
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_varchar_mapjoin1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_17
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_limit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_math_funcs
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/330/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/330/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-330/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813155 - PreCommit-HIVE-MASTER-Build
> Set incremental printing to true by default in Beeline
> --
>
> Key: HIVE-7224
> URL: https://issues.apache.org/jira/browse/HIVE-7224
> Project: Hive
> Issue Type: Bug
> Components: Clients, JDBC
>Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Sahil Takiar
> Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch,
> HIVE-7224.3.patch
>
>
> See HIVE-7221.
> By default beeline tries to buffer the entire output relation before printing
> it on stdout. This can cause OOM when the output relation is large. However,
> beeline has the option of incremental prints. We should keep that as the
> default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348937#comment-15348937 ] Vaibhav Gumashta commented on HIVE-7224: [~stakiar] Sure, thanks for taking it up. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348886#comment-15348886 ] Sahil Takiar commented on HIVE-7224: [~vgumashta] I wanted to help moving this ticket forward, it seems there were some test failures related to this change. Is it ok if I assign this ticket to myself and start working on resolving the test failures? > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348880#comment-15348880 ] Sahil Takiar commented on HIVE-7224: Attaching a new, re-based patch in order to trigger build and see if the test failures are still present. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Sahil Takiar > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch, > HIVE-7224.3.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14791681#comment-14791681 ] Vaibhav Gumashta commented on HIVE-7224: [~xuefuz] Sure, I'll look into the test failures. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14791342#comment-14791342 ] Xuefu Zhang commented on HIVE-7224: --- [~vgumashta], it looks like that the patch has caused some test failure. Could you please investigate? Thanks. > Set incremental printing to true by default in Beeline > -- > > Key: HIVE-7224 > URL: https://issues.apache.org/jira/browse/HIVE-7224 > Project: Hive > Issue Type: Bug > Components: Clients, JDBC >Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch > > > See HIVE-7221. > By default beeline tries to buffer the entire output relation before printing > it on stdout. This can cause OOM when the output relation is large. However, > beeline has the option of incremental prints. We should keep that as the > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[
https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742077#comment-14742077
]
Hive QA commented on HIVE-7224:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12755464/HIVE-7224.2.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 9424 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
org.apache.hive.beeline.TestBeeLineWithArgs.testCSV2OutputWithDoubleQuotes
org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutput
org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutputWithDoubleQuotes
org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutput
org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutputWithDoubleQuotes
org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmpty
org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmptyCmdArg
org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2Output
org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2OutputWithDoubleQuotes
org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutput
org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutputWithDoubleQuotes
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5255/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5255/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5255/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12755464 - PreCommit-HIVE-TRUNK-Build
> Set incremental printing to true by default in Beeline
> --
>
> Key: HIVE-7224
> URL: https://issues.apache.org/jira/browse/HIVE-7224
> Project: Hive
> Issue Type: Bug
> Components: Clients, JDBC
>Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch, HIVE-7224.2.patch
>
>
> See HIVE-7221.
> By default beeline tries to buffer the entire output relation before printing
> it on stdout. This can cause OOM when the output relation is large. However,
> beeline has the option of incremental prints. We should keep that as the
> default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[
https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694553#comment-14694553
]
Hive QA commented on HIVE-7224:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12750158/HIVE-7224.2.patch
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 9348 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeeLineWithArgs.testCSV2OutputWithDoubleQuotes
org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutput
org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutputWithDoubleQuotes
org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutput
org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutputWithDoubleQuotes
org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmpty
org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmptyCmdArg
org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2Output
org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2OutputWithDoubleQuotes
org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutput
org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutputWithDoubleQuotes
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testSparkQuery
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4941/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4941/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4941/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12750158 - PreCommit-HIVE-TRUNK-Build
Set incremental printing to true by default in Beeline
--
Key: HIVE-7224
URL: https://issues.apache.org/jira/browse/HIVE-7224
Project: Hive
Issue Type: Bug
Components: Clients, JDBC
Affects Versions: 0.13.0, 1.0.0, 1.2.0, 1.1.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Attachments: HIVE-7224.1.patch, HIVE-7224.2.patch
See HIVE-7221.
By default beeline tries to buffer the entire output relation before printing
it on stdout. This can cause OOM when the output relation is large. However,
beeline has the option of incremental prints. We should keep that as the
default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7224) Set incremental printing to true by default in Beeline
[ https://issues.apache.org/jira/browse/HIVE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681110#comment-14681110 ] Thejas M Nair commented on HIVE-7224: - [~vgumashta] can you please rebase ? Set incremental printing to true by default in Beeline -- Key: HIVE-7224 URL: https://issues.apache.org/jira/browse/HIVE-7224 Project: Hive Issue Type: Bug Components: Clients, JDBC Affects Versions: 0.13.0 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Labels: TODOC1.2 Attachments: HIVE-7224.1.patch See HIVE-7221. By default beeline tries to buffer the entire output relation before printing it on stdout. This can cause OOM when the output relation is large. However, beeline has the option of incremental prints. We should keep that as the default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
