[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108080#comment-16108080
]
Junping Du commented on MAPREDUCE-5875:
---
bq. At a minimum, -beta1 will need to include a note that says it was removed
or, preferably, removed but replaced with something better.
I believe MAPREDUCE-6924 already address this.
> Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
> ---
>
> Key: MAPREDUCE-5875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
> Fix For: 3.0.0-alpha1
>
> Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
> MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
> MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
> MAPREDUCE-5875.v09.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108068#comment-16108068
]
Allen Wittenauer commented on MAPREDUCE-5875:
-
bq. To be clear, this fix never land in any non-alpha release, so we shouldn't
claim the fix is released or any regression due to revert of this fix.
Doesn't matter. The release notes for alpha1 said that this patch is there. At
a minimum, -beta1 will need to include a note that says it was removed or,
preferably, removed but replaced with something better.
> Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
> ---
>
> Key: MAPREDUCE-5875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
> Fix For: 3.0.0-alpha1
>
> Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
> MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
> MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
> MAPREDUCE-5875.v09.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108052#comment-16108052
]
Junping Du commented on MAPREDUCE-5875:
---
Drop 2.9.0 as the patch get reverted from branch-2. Just clone MAPREDUCE-6924
for improved fix.
To be clear, this fix never land in any non-alpha release, so we shouldn't
claim the fix is released or any regression due to revert of this fix.
> Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
> ---
>
> Key: MAPREDUCE-5875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
> Fix For: 3.0.0-alpha1
>
> Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
> MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
> MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
> MAPREDUCE-5875.v09.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108045#comment-16108045
]
Allen Wittenauer commented on MAPREDUCE-5875:
-
It's closed because it was already in a release. You absolutely need to open a
new JIRA for it.
> Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
> ---
>
> Key: MAPREDUCE-5875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
> MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
> MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
> MAPREDUCE-5875.v09.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108026#comment-16108026
]
Robert Kanter commented on MAPREDUCE-5875:
--
[~djp], it's "Closed" so I don't think we can reopen it. We should probably
just open up a new JIRA to address this issue.
> Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
> ---
>
> Key: MAPREDUCE-5875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
> MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
> MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
> MAPREDUCE-5875.v09.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108008#comment-16108008
]
Hudson commented on MAPREDUCE-5875:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12080 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12080/])
Revert "MAPREDUCE-5875. Make Counter limits consistent across JobClient,
(junping_du: rev fbb7d6bcbb887ce52ab1e9d5a1fed67a7f8a4be8)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryViewer.java
> Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
> ---
>
> Key: MAPREDUCE-5875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
> MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
> MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
> MAPREDUCE-5875.v09.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107986#comment-16107986
]
Junping Du commented on MAPREDUCE-5875:
---
As discussed in MAPREDUCE-6288, I have revert this patch from trunk and
branch-2. We need to reopen this jira for an improved solution. But it looks
like I cannot reopen it - can someone able to do this?
> Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
> ---
>
> Key: MAPREDUCE-5875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
> MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
> MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
> MAPREDUCE-5875.v09.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784235#comment-15784235
]
Junping Du commented on MAPREDUCE-5875:
---
Revert from branch-2.8 given discussion in MAPREDUCE-6288.
> Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
> ---
>
> Key: MAPREDUCE-5875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
> MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
> MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
> MAPREDUCE-5875.v09.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15260027#comment-15260027
]
Harsh J commented on MAPREDUCE-5875:
Uh sorry, the commit message timeline is confusing on this. This is indeed
present in the trunk and branch-2.8/branch-2, was only reverted in branch-2.7.
My bad - sorry for the noise - the addendum vs. 2.7 notes confused me :(
Resetting to earlier state.
> Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
> ---
>
> Key: MAPREDUCE-5875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
> MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
> MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
> MAPREDUCE-5875.v09.patch
>
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390425#comment-14390425
]
Hudson commented on MAPREDUCE-5875:
---
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #150 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/150/])
Reverted MAPREDUCE-6286, MAPREDUCE-6199, and MAPREDUCE-5875 from branch-2.7.
Editing CHANGES.txt to reflect this. (vinodkv: rev
e428fea73029ea0c3494c71a50c5f6c994888fd2)
* hadoop-mapreduce-project/CHANGES.txt
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Fix For: 2.8.0
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390437#comment-14390437
]
Hudson commented on MAPREDUCE-5875:
---
FAILURE: Integrated in Hadoop-Yarn-trunk #884 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/884/])
Reverted MAPREDUCE-6286, MAPREDUCE-6199, and MAPREDUCE-5875 from branch-2.7.
Editing CHANGES.txt to reflect this. (vinodkv: rev
e428fea73029ea0c3494c71a50c5f6c994888fd2)
* hadoop-mapreduce-project/CHANGES.txt
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Fix For: 2.8.0
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390617#comment-14390617
]
Hudson commented on MAPREDUCE-5875:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk #2082 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2082/])
Reverted MAPREDUCE-6286, MAPREDUCE-6199, and MAPREDUCE-5875 from branch-2.7.
Editing CHANGES.txt to reflect this. (vinodkv: rev
e428fea73029ea0c3494c71a50c5f6c994888fd2)
* hadoop-mapreduce-project/CHANGES.txt
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Fix For: 2.8.0
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390633#comment-14390633
]
Hudson commented on MAPREDUCE-5875:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #150 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/150/])
Reverted MAPREDUCE-6286, MAPREDUCE-6199, and MAPREDUCE-5875 from branch-2.7.
Editing CHANGES.txt to reflect this. (vinodkv: rev
e428fea73029ea0c3494c71a50c5f6c994888fd2)
* hadoop-mapreduce-project/CHANGES.txt
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Fix For: 2.8.0
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390655#comment-14390655
]
Hudson commented on MAPREDUCE-5875:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #141 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/141/])
Reverted MAPREDUCE-6286, MAPREDUCE-6199, and MAPREDUCE-5875 from branch-2.7.
Editing CHANGES.txt to reflect this. (vinodkv: rev
e428fea73029ea0c3494c71a50c5f6c994888fd2)
* hadoop-mapreduce-project/CHANGES.txt
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Fix For: 2.8.0
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391183#comment-14391183
]
Hudson commented on MAPREDUCE-5875:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2100 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2100/])
Reverted MAPREDUCE-6286, MAPREDUCE-6199, and MAPREDUCE-5875 from branch-2.7.
Editing CHANGES.txt to reflect this. (vinodkv: rev
e428fea73029ea0c3494c71a50c5f6c994888fd2)
* hadoop-mapreduce-project/CHANGES.txt
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Fix For: 2.8.0
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389368#comment-14389368
]
Hudson commented on MAPREDUCE-5875:
---
FAILURE: Integrated in Hadoop-trunk-Commit #7478 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7478/])
Reverted MAPREDUCE-6286, MAPREDUCE-6199, and MAPREDUCE-5875 from branch-2.7.
Editing CHANGES.txt to reflect this. (vinodkv: rev
e428fea73029ea0c3494c71a50c5f6c994888fd2)
* hadoop-mapreduce-project/CHANGES.txt
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Fix For: 2.8.0
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377308#comment-14377308
]
Gera Shegalov commented on MAPREDUCE-5875:
--
[~rkanter], Oh my, we messed something up with this JIRA. Both MAPREDUCE-6288
and MAPREDUCE-6286 are broken because we dropped the logic in v08 during rebase
to v09.
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Fix For: 2.7.0
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376897#comment-14376897
]
Robert Kanter commented on MAPREDUCE-5875:
--
It looks like this patch breaks {{mapred job -status job-id}}. See
MAPREDUCE-6288.
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Fix For: 2.7.0
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168613#comment-14168613
]
Hudson commented on MAPREDUCE-5875:
---
SUCCESS: Integrated in Hadoop-Yarn-trunk #709 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/709/])
MAPREDUCE-5875. Make Counter limits consistent across JobClient, MRAppMaster,
and YarnChild. (Gera Shegalov via kasha) (kasha: rev
e8a31f2e1c34514fba2f480e8db652f6e2ed65d8)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryViewer.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Fix For: 2.7.0
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168662#comment-14168662
]
Hudson commented on MAPREDUCE-5875:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk #1899 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1899/])
MAPREDUCE-5875. Make Counter limits consistent across JobClient, MRAppMaster,
and YarnChild. (Gera Shegalov via kasha) (kasha: rev
e8a31f2e1c34514fba2f480e8db652f6e2ed65d8)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryViewer.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Fix For: 2.7.0
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168676#comment-14168676
]
Hudson commented on MAPREDUCE-5875:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1924 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1924/])
MAPREDUCE-5875. Make Counter limits consistent across JobClient, MRAppMaster,
and YarnChild. (Gera Shegalov via kasha) (kasha: rev
e8a31f2e1c34514fba2f480e8db652f6e2ed65d8)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryViewer.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Fix For: 2.7.0
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168371#comment-14168371
]
Hadoop QA commented on MAPREDUCE-5875:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674373/MAPREDUCE-5875.v09.patch
against trunk revision 554250c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.pipes.TestPipeApplication
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4954//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4954//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4954//console
This message is automatically generated.
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168375#comment-14168375
]
Karthik Kambatla commented on MAPREDUCE-5875:
-
+1, pending Jenkins
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168420#comment-14168420
]
Gera Shegalov commented on MAPREDUCE-5875:
--
TestPipeApplication failure is MAPREDUCE-6115. File .keep causing the RAT
plugin check failure is from YARN-2652 .
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168535#comment-14168535
]
Karthik Kambatla commented on MAPREDUCE-5875:
-
Committing this.
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168536#comment-14168536
]
Hudson commented on MAPREDUCE-5875:
---
FAILURE: Integrated in Hadoop-trunk-Commit #6245 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6245/])
MAPREDUCE-5875. Make Counter limits consistent across JobClient, MRAppMaster,
and YarnChild. (Gera Shegalov via kasha) (kasha: rev
e8a31f2e1c34514fba2f480e8db652f6e2ed65d8)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryViewer.java
* hadoop-mapreduce-project/CHANGES.txt
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch,
MAPREDUCE-5875.v09.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14165982#comment-14165982
]
Gera Shegalov commented on MAPREDUCE-5875:
--
[~kasha], did you have more comments for this patch?
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14165983#comment-14165983
]
Hadoop QA commented on MAPREDUCE-5875:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12655029/MAPREDUCE-5875.v08.patch
against trunk revision 596702a.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4949//console
This message is automatically generated.
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057819#comment-14057819
]
Hadoop QA commented on MAPREDUCE-5875:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12655029/MAPREDUCE-5875.v08.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4725//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4725//console
This message is automatically generated.
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch, MAPREDUCE-5875.v08.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5875) Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054432#comment-14054432
]
Karthik Kambatla commented on MAPREDUCE-5875:
-
Looks close. Couple of comments.
# Instead of doing string manipulation inline, can we add a helper method
preferably where we are constructing these History URLs.
{code}
final Path jobConfPath = new Path(jobFile.getParent(), jobDetails[0]
+ _ + jobDetails[1] + _ + jobDetails[2] + _conf.xml);
Limits.reset(new JobConf(conf).loadFile(fs, jobConfPath));
{code}
# Nit: I am okay with not addressing this. We should probably skip isEnabled
checks for INFO and WARN levels as the code doesn't have it anywhere else.
Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
---
Key: MAPREDUCE-5875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5875
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, client, task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5875.v01.patch, MAPREDUCE-5875.v02.patch,
MAPREDUCE-5875.v03.patch, MAPREDUCE-5875.v04.patch, MAPREDUCE-5875.v05.patch,
MAPREDUCE-5875.v06.patch, MAPREDUCE-5875.v07.patch
Currently, counter limits mapreduce.job.counters.* handled by
{{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
it's taken into account in YarnChild.
It would be good to make the Limits job-configurable, such that max
counters/groups is only increased when needed. With the current Limits
implementation relying on static constants, it's going to be challenging for
tools that submit jobs concurrently without resorting to class loading
isolation.
The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
