[jira] [Commented] (PIG-3913) Pig should use job's jobClient wherever possible (fixes local mode counters)
[ https://issues.apache.org/jira/browse/PIG-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14003734#comment-14003734 ] Aniket Mokashi commented on PIG-3913: - Committed to trunk! (review updated and patch attached). > Pig should use job's jobClient wherever possible (fixes local mode counters) > > > Key: PIG-3913 > URL: https://issues.apache.org/jira/browse/PIG-3913 > Project: Pig > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Aniket Mokashi >Assignee: Aniket Mokashi > Fix For: 0.13.0 > > Attachments: PIG-3913-1.patch, PIG-3913-3.patch, PIG-3913-5.patch, > PIG-3913-6.patch, PIG-3913-7.patch, PIG-3913-MRv2.v01.patch > > > MapreduceLauncher initializes a statsJobClient to poll counter information of > jobs. This works fine in mapreduce mode but it reports incorrect information > in local (auto-local) mode. Pig code should try to use > org.apache.hadoop.mapred.jobcontrol.Job's getJobClient api to get handle to > jobClient wherever possible. statsJobClient (and wherever its references are > passed) should be deprecated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-3913) Pig should use job's jobClient wherever possible (fixes local mode counters)
[ https://issues.apache.org/jira/browse/PIG-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14002486#comment-14002486 ] Aniket Mokashi commented on PIG-3913: - Thanks [~jira.shegalov] and [~cheolsoo] for the review. I will incorporate your comments and commit this to trunk! > Pig should use job's jobClient wherever possible (fixes local mode counters) > > > Key: PIG-3913 > URL: https://issues.apache.org/jira/browse/PIG-3913 > Project: Pig > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Aniket Mokashi >Assignee: Aniket Mokashi > Fix For: 0.13.0 > > Attachments: PIG-3913-1.patch, PIG-3913-3.patch, PIG-3913-5.patch, > PIG-3913-6.patch, PIG-3913-MRv2.v01.patch > > > MapreduceLauncher initializes a statsJobClient to poll counter information of > jobs. This works fine in mapreduce mode but it reports incorrect information > in local (auto-local) mode. Pig code should try to use > org.apache.hadoop.mapred.jobcontrol.Job's getJobClient api to get handle to > jobClient wherever possible. statsJobClient (and wherever its references are > passed) should be deprecated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-3913) Pig should use job's jobClient wherever possible (fixes local mode counters)
[
https://issues.apache.org/jira/browse/PIG-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14002474#comment-14002474
]
Cheolsoo Park commented on PIG-3913:
+1. Let's commit this.
Btw, the patch needs to be rebased to trunk. In addition,
{{hadoop23/HadoopShims.java}} has unused imports. Please fix them when
committing the patch.
Thanks!
> Pig should use job's jobClient wherever possible (fixes local mode counters)
>
>
> Key: PIG-3913
> URL: https://issues.apache.org/jira/browse/PIG-3913
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Aniket Mokashi
>Assignee: Aniket Mokashi
> Fix For: 0.13.0
>
> Attachments: PIG-3913-1.patch, PIG-3913-3.patch, PIG-3913-5.patch,
> PIG-3913-6.patch, PIG-3913-MRv2.v01.patch
>
>
> MapreduceLauncher initializes a statsJobClient to poll counter information of
> jobs. This works fine in mapreduce mode but it reports incorrect information
> in local (auto-local) mode. Pig code should try to use
> org.apache.hadoop.mapred.jobcontrol.Job's getJobClient api to get handle to
> jobClient wherever possible. statsJobClient (and wherever its references are
> passed) should be deprecated.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (PIG-3913) Pig should use job's jobClient wherever possible (fixes local mode counters)
[ https://issues.apache.org/jira/browse/PIG-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993148#comment-13993148 ] Aniket Mokashi commented on PIG-3913: - Looking at this, it seems main difference is - in hadoop 1, jobcontrol.Job does not support an api to get the mapreduce/mapred.Job object from it (in fact it does not even store it). In hadoop 2, jobcontrol.Job is just a wrapper around ControlledJob which stores a pointer to the real mapreduce.Job. In hadoop 1 for querying stats, we should get jobClient of the job and ask for the runningJob given the JobID for the job. In hadoop 2, we should directly the real job using getJob of jobcontrol.Job and query it for stats. I will construct necessary shims to hide this behind interfaces. > Pig should use job's jobClient wherever possible (fixes local mode counters) > > > Key: PIG-3913 > URL: https://issues.apache.org/jira/browse/PIG-3913 > Project: Pig > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Aniket Mokashi >Assignee: Gera Shegalov > Fix For: 0.13.0 > > Attachments: PIG-3913-1.patch, PIG-3913-MRv2.v01.patch > > > MapreduceLauncher initializes a statsJobClient to poll counter information of > jobs. This works fine in mapreduce mode but it reports incorrect information > in local (auto-local) mode. Pig code should try to use > org.apache.hadoop.mapred.jobcontrol.Job's getJobClient api to get handle to > jobClient wherever possible. statsJobClient (and wherever its references are > passed) should be deprecated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-3913) Pig should use job's jobClient wherever possible (fixes local mode counters)
[ https://issues.apache.org/jira/browse/PIG-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999581#comment-13999581 ] Gera Shegalov commented on PIG-3913: +1 (non-binding) modulo nits on RB > Pig should use job's jobClient wherever possible (fixes local mode counters) > > > Key: PIG-3913 > URL: https://issues.apache.org/jira/browse/PIG-3913 > Project: Pig > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Aniket Mokashi >Assignee: Aniket Mokashi > Fix For: 0.13.0 > > Attachments: PIG-3913-1.patch, PIG-3913-3.patch, PIG-3913-5.patch, > PIG-3913-6.patch, PIG-3913-MRv2.v01.patch > > > MapreduceLauncher initializes a statsJobClient to poll counter information of > jobs. This works fine in mapreduce mode but it reports incorrect information > in local (auto-local) mode. Pig code should try to use > org.apache.hadoop.mapred.jobcontrol.Job's getJobClient api to get handle to > jobClient wherever possible. statsJobClient (and wherever its references are > passed) should be deprecated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-3913) Pig should use job's jobClient wherever possible (fixes local mode counters)
[ https://issues.apache.org/jira/browse/PIG-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997002#comment-13997002 ] Jonathan Coveney commented on PIG-3913: --- Can you throw this in the apache reviewboard? #highmaintenace > Pig should use job's jobClient wherever possible (fixes local mode counters) > > > Key: PIG-3913 > URL: https://issues.apache.org/jira/browse/PIG-3913 > Project: Pig > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Aniket Mokashi >Assignee: Aniket Mokashi > Fix For: 0.13.0 > > Attachments: PIG-3913-1.patch, PIG-3913-3.patch, PIG-3913-5.patch, > PIG-3913-MRv2.v01.patch > > > MapreduceLauncher initializes a statsJobClient to poll counter information of > jobs. This works fine in mapreduce mode but it reports incorrect information > in local (auto-local) mode. Pig code should try to use > org.apache.hadoop.mapred.jobcontrol.Job's getJobClient api to get handle to > jobClient wherever possible. statsJobClient (and wherever its references are > passed) should be deprecated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-3913) Pig should use job's jobClient wherever possible (fixes local mode counters)
[ https://issues.apache.org/jira/browse/PIG-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997880#comment-13997880 ] Aniket Mokashi commented on PIG-3913: - #GoodIdea: https://reviews.apache.org/r/21454/ > Pig should use job's jobClient wherever possible (fixes local mode counters) > > > Key: PIG-3913 > URL: https://issues.apache.org/jira/browse/PIG-3913 > Project: Pig > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Aniket Mokashi >Assignee: Aniket Mokashi > Fix For: 0.13.0 > > Attachments: PIG-3913-1.patch, PIG-3913-3.patch, PIG-3913-5.patch, > PIG-3913-6.patch, PIG-3913-MRv2.v01.patch > > > MapreduceLauncher initializes a statsJobClient to poll counter information of > jobs. This works fine in mapreduce mode but it reports incorrect information > in local (auto-local) mode. Pig code should try to use > org.apache.hadoop.mapred.jobcontrol.Job's getJobClient api to get handle to > jobClient wherever possible. statsJobClient (and wherever its references are > passed) should be deprecated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-3913) Pig should use job's jobClient wherever possible (fixes local mode counters)
[ https://issues.apache.org/jira/browse/PIG-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994005#comment-13994005 ] Aniket Mokashi commented on PIG-3913: - Added a new patch for this. > Pig should use job's jobClient wherever possible (fixes local mode counters) > > > Key: PIG-3913 > URL: https://issues.apache.org/jira/browse/PIG-3913 > Project: Pig > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Aniket Mokashi >Assignee: Aniket Mokashi > Fix For: 0.13.0 > > Attachments: PIG-3913-1.patch, PIG-3913-3.patch, > PIG-3913-MRv2.v01.patch > > > MapreduceLauncher initializes a statsJobClient to poll counter information of > jobs. This works fine in mapreduce mode but it reports incorrect information > in local (auto-local) mode. Pig code should try to use > org.apache.hadoop.mapred.jobcontrol.Job's getJobClient api to get handle to > jobClient wherever possible. statsJobClient (and wherever its references are > passed) should be deprecated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-3913) Pig should use job's jobClient wherever possible (fixes local mode counters)
[ https://issues.apache.org/jira/browse/PIG-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994315#comment-13994315 ] Gera Shegalov commented on PIG-3913: [~aniket486], PIG-3913-3 patch is incomplete, DowngradeHelper.java is missing. > Pig should use job's jobClient wherever possible (fixes local mode counters) > > > Key: PIG-3913 > URL: https://issues.apache.org/jira/browse/PIG-3913 > Project: Pig > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Aniket Mokashi >Assignee: Aniket Mokashi > Fix For: 0.13.0 > > Attachments: PIG-3913-1.patch, PIG-3913-3.patch, > PIG-3913-MRv2.v01.patch > > > MapreduceLauncher initializes a statsJobClient to poll counter information of > jobs. This works fine in mapreduce mode but it reports incorrect information > in local (auto-local) mode. Pig code should try to use > org.apache.hadoop.mapred.jobcontrol.Job's getJobClient api to get handle to > jobClient wherever possible. statsJobClient (and wherever its references are > passed) should be deprecated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-3913) Pig should use job's jobClient wherever possible (fixes local mode counters)
[ https://issues.apache.org/jira/browse/PIG-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995297#comment-13995297 ] Aniket Mokashi commented on PIG-3913: - I've attached PIG-3913-5.patch. > Pig should use job's jobClient wherever possible (fixes local mode counters) > > > Key: PIG-3913 > URL: https://issues.apache.org/jira/browse/PIG-3913 > Project: Pig > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Aniket Mokashi >Assignee: Aniket Mokashi > Fix For: 0.13.0 > > Attachments: PIG-3913-1.patch, PIG-3913-3.patch, PIG-3913-5.patch, > PIG-3913-MRv2.v01.patch > > > MapreduceLauncher initializes a statsJobClient to poll counter information of > jobs. This works fine in mapreduce mode but it reports incorrect information > in local (auto-local) mode. Pig code should try to use > org.apache.hadoop.mapred.jobcontrol.Job's getJobClient api to get handle to > jobClient wherever possible. statsJobClient (and wherever its references are > passed) should be deprecated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-3913) Pig should use job's jobClient wherever possible (fixes local mode counters)
[ https://issues.apache.org/jira/browse/PIG-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985851#comment-13985851 ] Aniket Mokashi commented on PIG-3913: - Thanks for the review, [~julienledem]. I realized that we have some code in pig that will avoiding printing counters in local mode (since they were broken). Let me clean up that code so that counters are available again with local mode. Also, I will test with hadoop 2 to see if this fix works with it. [~daijy], any known issues I should be aware of here? > Pig should use job's jobClient wherever possible (fixes local mode counters) > > > Key: PIG-3913 > URL: https://issues.apache.org/jira/browse/PIG-3913 > Project: Pig > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Aniket Mokashi >Assignee: Aniket Mokashi > Fix For: 0.13.0 > > Attachments: PIG-3913-1.patch > > > MapreduceLauncher initializes a statsJobClient to poll counter information of > jobs. This works fine in mapreduce mode but it reports incorrect information > in local (auto-local) mode. Pig code should try to use > org.apache.hadoop.mapred.jobcontrol.Job's getJobClient api to get handle to > jobClient wherever possible. statsJobClient (and wherever its references are > passed) should be deprecated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-3913) Pig should use job's jobClient wherever possible (fixes local mode counters)
[ https://issues.apache.org/jira/browse/PIG-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985005#comment-13985005 ] Julien Le Dem commented on PIG-3913: This looks good to me. Please add javadoc for deprecated methods with information about the new way of doing the same thing. +1 > Pig should use job's jobClient wherever possible (fixes local mode counters) > > > Key: PIG-3913 > URL: https://issues.apache.org/jira/browse/PIG-3913 > Project: Pig > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Aniket Mokashi >Assignee: Aniket Mokashi > Fix For: 0.13.0 > > Attachments: PIG-3913-1.patch > > > MapreduceLauncher initializes a statsJobClient to poll counter information of > jobs. This works fine in mapreduce mode but it reports incorrect information > in local (auto-local) mode. Pig code should try to use > org.apache.hadoop.mapred.jobcontrol.Job's getJobClient api to get handle to > jobClient wherever possible. statsJobClient (and wherever its references are > passed) should be deprecated. -- This message was sent by Atlassian JIRA (v6.2#6252)
