[
https://issues.apache.org/jira/browse/PIG-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167592#comment-13167592
]
Jie Li commented on PIG-2397:
-----------------------------
As we summarized in the slides, there are many factors affecting the overall
performance:
1. Reorder JOINs properly
Not sure if Pig is moving on to a cost-based optimizer that figures out a
better join order/type.
2&3. Multi query optimization, e.g. using COGROUP for JOIN + GROUP && using
FLATTEN for self-join + GROUP
First, can such rules can be implemented as query rewriting rules? Seems they
are always better.
Second, without query rewriting, it's still possible to take advantage of the
common key between join/group. Hive is taking such efforts now
(https://issues.apache.org/jira/browse/HIVE-1772).
4. Project before (CO)GROUP (https://issues.apache.org/jira/browse/PIG-1324)
This issue will affect the performance of GROUP/COGROUP. Currently users have
to explicitly drop redundant columns before (CO)GROUP, which is not convenient
and not easy to be done completely.
5. Remove redundant types in LOAD
(https://issues.apache.org/jira/browse/PIG-410)
Again not convenient for users. This should be done by Pig itself. For columns
with types, is pig possible to delay the conversion?
6. hash-based aggregation
This feature should significantly improve the group-by. Not sure it can be
further used for multi-query optimization. For example, for X (whatever job) +
group-by, we can use a hash-based combiner at the end of X, as Hive does
already.
Except the first factor (reorder joins) which Hive is unable as well, all the
other factors can contribute to Hive's outperformance. Glad to see that Pig has
already implemented the hash-based aggregation. We're excited to repeat the
benchmark with this novel feature.
> Running TPC-H on Pig
> --------------------
>
> Key: PIG-2397
> URL: https://issues.apache.org/jira/browse/PIG-2397
> Project: Pig
> Issue Type: Task
> Reporter: Jie Li
> Attachments: TPC-H_on_Pig.tgz, pig_tpch.ppt
>
>
> For a class project we developed a whole set of Pig scripts for TPC-H. Our
> goals are:
> 1) identifying the bottlenecks of Pig's performance especially of its
> relational operators,
> 2) studying how to write efficient scripts by making full use of Pig Latin's
> features,
> 3) comparing with Hive's TPC-H results for verifying both 1) and 2).
> We will update the JIRA with our scripts, results and analysis soon.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
