[
https://issues.apache.org/jira/browse/PIG-4958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393203#comment-15393203
]
Rohini Palaniswamy commented on PIG-4958:
-----------------------------------------
bq. Also this might overload the RM in case there are many such tasks.
Only one task for each order by statement in the pig script. So should not
cause overload.
bq. to pass consolidated stats information from the vertex manager to all the
tasks.
bq. I think this will be better achieved by making changes in Tez to make
specific information available via stats
Looking for some approach that can be used now till feature support is added
in Tez.
bq. Tez can be configured to generate counters per output.
We don't enable that in our clusters as it adds a lot of overhead to AM.
Patch can definitely be refined to fetch per output counter if available. Was
planning to address that when I did the patch for the skewed join case.
bq. It's a private Tez API, a lot of load to the RM, and it's likely to be
extremely slow since AMs run with a single RPC thread
private Tez API is a concern. But not worried about the load. Currently Pig
client is hitting AM with dagClient.getDAGStatus() every second to check
completion status and have not seen issues with AM. (Different issue - 1 sec
is too low. Should be at least 5 secs)
> Tez autoparallelism estimation for order by is higher than mapreduce
> --------------------------------------------------------------------
>
> Key: PIG-4958
> URL: https://issues.apache.org/jira/browse/PIG-4958
> Project: Pig
> Issue Type: Bug
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.17.0
>
> Attachments: PIG-4958-withoutsecurity.patch
>
>
> The input size is calculated from the size of the samples in memory. Size
> in memory is usually 4x or more than the serialized size. Mapreduce estimates
> the number of reducers based on serialized size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
