[jira] [Commented] (HIVE-14169) Honor --incremental flag only if TableOutputFormat is used
[ https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384925#comment-15384925 ] Sahil Takiar commented on HIVE-14169: - Hey [~taoli-hwx] * Yes, by default it is still false * For non-table formats we came to the conclusion that there is no real benefit to using BufferedRows. It only really makes sense if the table output format is used. The reason is that if table output format is used along with BufferedRows, then the BufferedRows can calculate the optimal sizing for each row that it prints out. However, this isn't applicable for non-table formats. This is why I made the change to stop honoring the value of incremental if a non-table format is used. Also, I am going to close this JIRA and mark it as a duplicate of HIVE-14170 - since it doesn't make sense to commit these changes without HIVE-14170 along with it. > Honor --incremental flag only if TableOutputFormat is used > -- > > Key: HIVE-14169 > URL: https://issues.apache.org/jira/browse/HIVE-14169 > Project: Hive > Issue Type: Sub-task > Components: Beeline >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-14169.1.patch > > > * When Beeline prints out a {{ResultSet}} to stdout it uses the > {{BeeLine.print}} method > * This method takes the {{ResultSet}} from the completed query and uses a > specified {{OutputFormat}} to print the rows (by default it uses > {{TableOutputFormat}}) > * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class > (either a {{IncrementalRows}} or a {{BufferedRows}} class) > The advantage of {{BufferedRows}} is that it can do a global calculation of > the column width, however, this is only useful for {{TableOutputFormat}}. So > there is no need to buffer all the rows if a different {{OutputFormat}} is > used. This JIRA will change the behavior of the {{--incremental}} flag so > that it is only honored if {{TableOutputFormat}} is used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14169) Honor --incremental flag only if TableOutputFormat is used
[ https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383363#comment-15383363 ] Tao Li commented on HIVE-14169: --- Hi Sahil, 2 quick questions: 1. I think the default setting for "--incremental" is still false with your change, right? If that's true, we still go into the code path of buffered rows if the "--incremental" is not specified, which I think is a typical use case. 2. Looks like we always do the incremental rows regardless of the setting for "--incremental" for non-table formats. What if the user specifies a non-table format and also "--incremental=false"? Do we want to do buffered rows in this case? Thanks. > Honor --incremental flag only if TableOutputFormat is used > -- > > Key: HIVE-14169 > URL: https://issues.apache.org/jira/browse/HIVE-14169 > Project: Hive > Issue Type: Sub-task > Components: Beeline >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-14169.1.patch > > > * When Beeline prints out a {{ResultSet}} to stdout it uses the > {{BeeLine.print}} method > * This method takes the {{ResultSet}} from the completed query and uses a > specified {{OutputFormat}} to print the rows (by default it uses > {{TableOutputFormat}}) > * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class > (either a {{IncrementalRows}} or a {{BufferedRows}} class) > The advantage of {{BufferedRows}} is that it can do a global calculation of > the column width, however, this is only useful for {{TableOutputFormat}}. So > there is no need to buffer all the rows if a different {{OutputFormat}} is > used. This JIRA will change the behavior of the {{--incremental}} flag so > that it is only honored if {{TableOutputFormat}} is used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14169) Honor --incremental flag only if TableOutputFormat is used
[ https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382902#comment-15382902 ] Sahil Takiar commented on HIVE-14169: - [~thejas] any comments on this JIRA or HIVE-14170? Thanks! --Sahil > Honor --incremental flag only if TableOutputFormat is used > -- > > Key: HIVE-14169 > URL: https://issues.apache.org/jira/browse/HIVE-14169 > Project: Hive > Issue Type: Sub-task > Components: Beeline >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-14169.1.patch > > > * When Beeline prints out a {{ResultSet}} to stdout it uses the > {{BeeLine.print}} method > * This method takes the {{ResultSet}} from the completed query and uses a > specified {{OutputFormat}} to print the rows (by default it uses > {{TableOutputFormat}}) > * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class > (either a {{IncrementalRows}} or a {{BufferedRows}} class) > The advantage of {{BufferedRows}} is that it can do a global calculation of > the column width, however, this is only useful for {{TableOutputFormat}}. So > there is no need to buffer all the rows if a different {{OutputFormat}} is > used. This JIRA will change the behavior of the {{--incremental}} flag so > that it is only honored if {{TableOutputFormat}} is used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14169) Honor --incremental flag only if TableOutputFormat is used
[ https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371136#comment-15371136 ] Sahil Takiar commented on HIVE-14169: - Test failures look unrelated, [~thejas] and [~spena] can you review? > Honor --incremental flag only if TableOutputFormat is used > -- > > Key: HIVE-14169 > URL: https://issues.apache.org/jira/browse/HIVE-14169 > Project: Hive > Issue Type: Sub-task > Components: Beeline >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-14169.1.patch > > > * When Beeline prints out a {{ResultSet}} to stdout it uses the > {{BeeLine.print}} method > * This method takes the {{ResultSet}} from the completed query and uses a > specified {{OutputFormat}} to print the rows (by default it uses > {{TableOutputFormat}}) > * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class > (either a {{IncrementalRows}} or a {{BufferedRows}} class) > The advantage of {{BufferedRows}} is that it can do a global calculation of > the column width, however, this is only useful for {{TableOutputFormat}}. So > there is no need to buffer all the rows if a different {{OutputFormat}} is > used. This JIRA will change the behavior of the {{--incremental}} flag so > that it is only honored if {{TableOutputFormat}} is used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14169) Honor --incremental flag only if TableOutputFormat is used
[ https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369277#comment-15369277 ] Hive QA commented on HIVE-14169: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12816907/HIVE-14169.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10296 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/444/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/444/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-444/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12816907 - PreCommit-HIVE-MASTER-Build > Honor --incremental flag only if TableOutputFormat is used > -- > > Key: HIVE-14169 > URL: https://issues.apache.org/jira/browse/HIVE-14169 > Project: Hive > Issue Type: Sub-task > Components: Beeline >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-14169.1.patch > > > * When Beeline prints out a {{ResultSet}} to stdout it uses the > {{BeeLine.print}} method > * This method takes the {{ResultSet}} from the completed query and uses a > specified {{OutputFormat}} to print the rows (by default it uses > {{TableOutputFormat}}) > * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class > (either a {{IncrementalRows}} or a {{BufferedRows}} class) > The advantage of {{BufferedRows}} is that it can do a global calculation of > the column width, however, this is only useful for {{TableOutputFormat}}. So > there is no need to buffer all the rows if a different {{OutputFormat}} is > used. This JIRA will change the behavior of the {{--incremental}} flag so > that it is only honored if {{TableOutputFormat}} is used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)