[jira] [Commented] (HIVE-13533) Remove AST dump
[ https://issues.apache.org/jira/browse/HIVE-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866977#comment-15866977 ] Sergey Shelukhin commented on HIVE-13533: - AST dump is useful in some cases... perhaps there should be a separate explain that would output it? > Remove AST dump > --- > > Key: HIVE-13533 > URL: https://issues.apache.org/jira/browse/HIVE-13533 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-13533.patch, HIVE-13533.patch > > > For very large queries, dumping the AST can lead to OOM errors. Currently > there are two places where we dump the AST: > - CalcitePlanner if we are running in DEBUG mode (line 300). > - ExplainTask if we use extended explain (line 179). > I guess the original reason to add the dump was to check whether the AST > conversion from CBO was working properly, but I think we are past that stage > now. > We will remove the logic to dump the AST in explain extended. For debug mode > in CalcitePlanner, we will lower the level to LOG.TRACE. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-13533) Remove AST dump
[ https://issues.apache.org/jira/browse/HIVE-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867496#comment-15867496 ] Jesus Camacho Rodriguez commented on HIVE-13533: [~sershe], if you feel that it might still be useful in some cases and should be exposed through explain, maybe we could indeed add an option _--include-ast_ for the extended case? What do you think? Cc [~ashutoshc] > Remove AST dump > --- > > Key: HIVE-13533 > URL: https://issues.apache.org/jira/browse/HIVE-13533 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-13533.patch, HIVE-13533.patch > > > For very large queries, dumping the AST can lead to OOM errors. Currently > there are two places where we dump the AST: > - CalcitePlanner if we are running in DEBUG mode (line 300). > - ExplainTask if we use extended explain (line 179). > I guess the original reason to add the dump was to check whether the AST > conversion from CBO was working properly, but I think we are past that stage > now. > We will remove the logic to dump the AST in explain extended. For debug mode > in CalcitePlanner, we will lower the level to LOG.TRACE. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-13533) Remove AST dump
[ https://issues.apache.org/jira/browse/HIVE-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15868390#comment-15868390 ] Sergey Shelukhin commented on HIVE-13533: - I was thinking along the lines of "explain ast", we have many explain commands these days ;) > Remove AST dump > --- > > Key: HIVE-13533 > URL: https://issues.apache.org/jira/browse/HIVE-13533 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-13533.patch, HIVE-13533.patch > > > For very large queries, dumping the AST can lead to OOM errors. Currently > there are two places where we dump the AST: > - CalcitePlanner if we are running in DEBUG mode (line 300). > - ExplainTask if we use extended explain (line 179). > I guess the original reason to add the dump was to check whether the AST > conversion from CBO was working properly, but I think we are past that stage > now. > We will remove the logic to dump the AST in explain extended. For debug mode > in CalcitePlanner, we will lower the level to LOG.TRACE. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-13533) Remove AST dump
[ https://issues.apache.org/jira/browse/HIVE-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15248276#comment-15248276 ] Ashutosh Chauhan commented on HIVE-13533: - +1 pending tests This likely will require golden file updates to many files. > Remove AST dump > --- > > Key: HIVE-13533 > URL: https://issues.apache.org/jira/browse/HIVE-13533 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13533.patch > > > For very large queries, dumping the AST can lead to OOM errors. Currently > there are two places where we dump the AST: > - CalcitePlanner if we are running in DEBUG mode (line 300). > - ExplainTask if we use extended explain (line 179). > I guess the original reason to add the dump was to check whether the AST > conversion from CBO was working properly, but I think we are past that stage > now. > We will remove the logic to dump the AST in explain extended. For debug mode > in CalcitePlanner, we will lower the level to LOG.TRACE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13533) Remove AST dump
[
https://issues.apache.org/jira/browse/HIVE-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255513#comment-15255513
]
Hive QA commented on HIVE-13533:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800257/HIVE-13533.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 339 failed/errored test(s), 9952 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguitycheck
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_table_null_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binary_output_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_outer_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_partlvl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_tbllvl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_display_colstats_tbllvl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_multi_partitions
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_partition_skip_default
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_full
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial_ndv
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_
