[
https://issues.apache.org/jira/browse/PIG-4958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392880#comment-15392880
]
Siddharth Seth commented on PIG-4958:
-------------------------------------
bq. If there are multiple outputs in sampler vertex, then the OUTPUT_BYTES
counter value might be high as we are not getting counter per output.
Tez can be configured to generate counters per output.
I don't think creating an instance of DAGClent from tasks will work very well.
It's a private Tez API, a lot of load to the RM, and it's likely to be
extremely slow since AMs run with a single RPC thread (which can block on the
getStatus call).
In terms of the counters - I think this will be better achieved by making
changes in Tez to make specific information available via stats - and make the
stats available in additional places. All counters don't necessarily need to be
made available everywhere (and some of them can be confusing).
> Tez autoparallelism estimation for order by is higher than mapreduce
> --------------------------------------------------------------------
>
> Key: PIG-4958
> URL: https://issues.apache.org/jira/browse/PIG-4958
> Project: Pig
> Issue Type: Bug
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.17.0
>
> Attachments: PIG-4958-withoutsecurity.patch
>
>
> The input size is calculated from the size of the samples in memory. Size
> in memory is usually 4x or more than the serialized size. Mapreduce estimates
> the number of reducers based on serialized size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
