[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970009#action_12970009 ] Namit Jain commented on HIVE-1648: -- 1. QBParseInfo: add setDestToLimit() for symmetry() 2. I am not sure any of your tests are working - set hive.stats.autogather = false before you create the tables for which you want the stats to be populated while reading. Clearly, this is the reason why piggyback_part.q is working. 2. piggyback_join.q End: show table extended like piggy_table3; drop table piggy_table; Add: show table extended like piggy_table1; show table extended like piggy_table2; Also, add a test where you are joining: piggyback_table1 a join piggyback_table2 b on a.key = b.key join piggyback_table3 c b.key = c.key and then show table extended all the 3 tables. 3. piggyback_limit.q add: show table extended like piggy_table1; before the end. It should have no stats 4. piggbyback_subq.q and _union.q are wrong - you need to create new tables, and then show table extended them at the end, just like other tests. 5. > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang >Assignee: Paul Butler > Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.4.patch, > HIVE-1648.5.patch, HIVE-1648.patch, hive-1648.svn.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969878#action_12969878 ] Paul Butler commented on HIVE-1648: --- Added an SVN patch (5) that applies to latest. > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang >Assignee: Paul Butler > Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.4.patch, > HIVE-1648.5.patch, HIVE-1648.patch, hive-1648.svn.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968854#action_12968854 ] He Yongqiang commented on HIVE-1648: Actually the git patch can not be applied cleanly. Paul, can you upload a new patch using svn? > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang >Assignee: Paul Butler > Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.4.patch, > HIVE-1648.patch, hive-1648.svn.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968841#action_12968841 ] Namit Jain commented on HIVE-1648: -- @Yongqiang, you have missed the test changes in the patch - can you add them also ? > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang >Assignee: Paul Butler > Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.4.patch, > HIVE-1648.patch, hive-1648.svn.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966690#action_12966690 ] Paul Butler commented on HIVE-1648: --- The three-way join test was added to piggyback_join.q.out. No other new tests were added. I also changed the tests to create new tables and use SHOW TABLE EXTENDED, which I modified to print numRows if available. > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang >Assignee: Paul Butler > Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.4.patch, > HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1290#action_1290 ] Namit Jain commented on HIVE-1648: -- I dont see any new tests > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang >Assignee: Paul Butler > Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.4.patch, > HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966354#action_12966354 ] Paul Butler commented on HIVE-1648: --- Changes made. Note that subqueries are not piggybacked, but tests are there to make sure they still run when hive.stats.autogather=true. > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang >Assignee: Paul Butler > Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.4.patch, > HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935551#action_12935551 ] Paul Butler commented on HIVE-1648: --- Namit, it looks like show table extended like ``; doesn't print the number of rows. Unless there's a way to make it do that, I'll have to stick with desc extended. I sent you an email for clarification on the ConditionalTasks also. > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang >Assignee: Paul Butler > Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934683#action_12934683 ] Namit Jain commented on HIVE-1648: -- Otherwise, it looks OK > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang >Assignee: Paul Butler > Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[
https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934682#action_12934682
]
Namit Jain commented on HIVE-1648:
--
In SemanticAnalyzer:addStatsTask:
} else {
6177 List children = (List) op.getChildren();
6178 if (children != null) {
6179for (Node child : children) {
6180 opsToProcess.add((Operator)
child);
6181}
6182 }
why is the above code block needed ? TableScan can only be at the top.
Also, can you check for Conditional Tasks in addition to MapRedTask ?
> Automatically gathering stats when reading a table/partition
>
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ning Zhang
>Assignee: Paul Butler
> Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to
> gathering stats. This requires additional scan of the data. Stats gathering
> can be piggy-backed on TableScanOperator whenever a table/partition is
> scanned (given not LIMIT operator).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934619#action_12934619 ] Namit Jain commented on HIVE-1648: -- I haven't taken a look at the code, but here are the comments for the tests Instead of: desc extended in the tests, please use show table extended like ``; This will dump stats in a new line and can be easily compared. The non-deterministic stats are ignored. Add a test for limit in the sub-query. Dont select from existing tables: src/src1 for your stats tests. Create new tables and then set hive.stats.autogather.read to true. This was, you are sure that the remaining tests will not be affected. Add another test for 3-way join where the join keys are not the same: something like: select .. from A join B on A.key1 = B.key1 join C on B.key2 = C.key2 where > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang >Assignee: Paul Butler > Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930326#action_12930326 ] Paul Butler commented on HIVE-1648: --- I get a bunch of tests failing when I build the latest trunk, even without applying my patch. I'm trying to figure out what's wrong with those first. > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang >Assignee: Paul Butler > Attachments: HIVE-1648.2.patch, HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930321#action_12930321 ] Namit Jain commented on HIVE-1648: -- Paul, any updates on the unit tests ? > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang >Assignee: Paul Butler > Attachments: HIVE-1648.2.patch, HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928836#action_12928836 ] Paul Butler commented on HIVE-1648: --- Attached a new patch which applies on top of HIVE-1750. Also fixed the limits as suggested. Working on unit tests now. -- Paul > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang >Assignee: Paul Butler > Attachments: HIVE-1648.2.patch, HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928804#action_12928804 ] John Sichi commented on HIVE-1648: -- Obtained Paul Butler's account name from the JIRA change history...I'll add him as contributor. > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang > Attachments: HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928737#action_12928737 ] Namit Jain commented on HIVE-1648: -- Also, add tests for all the cases above. You should be able to compare the plan via a explain extended > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang > Attachments: HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928733#action_12928733 ] Namit Jain commented on HIVE-1648: -- getOuterQueryLimit is not used correctly. qb maintains a mapping of limit per destination case HiveParser.TOK_LIMIT: qbp.setDestLimit(ctx_1.dest, new Integer(ast.getChild(0).getText())); break; Since you are browsing the input tables, you dont care about the destination. Go over: private final HashMap destToLimit; in QBParseInfo. If that is not-empty, do not gather stats > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang > Attachments: HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928709#action_12928709 ] Namit Jain commented on HIVE-1648: -- Can you regenerate the patch - it conflicted with HIVE-1750 and may need some cleanup. > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang > Attachments: HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition
[ https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928706#action_12928706 ] Namit Jain commented on HIVE-1648: -- Paul, do you have an apache account ? Can you send it to me - I will add you as a hive contributor. Also, can you 'Submit Patch' when you add a patch - this way everyone knows that the patch is ready for review > Automatically gathering stats when reading a table/partition > > > Key: HIVE-1648 > URL: https://issues.apache.org/jira/browse/HIVE-1648 > Project: Hive > Issue Type: Sub-task >Reporter: Ning Zhang > Attachments: HIVE-1648.patch > > > HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to > gathering stats. This requires additional scan of the data. Stats gathering > can be piggy-backed on TableScanOperator whenever a table/partition is > scanned (given not LIMIT operator). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
