[jira] [Resolved] (MAPREDUCE-3048) Fix test-patch to run tests via "mvn clean install test"

[ https://issues.apache.org/jira/browse/MAPREDUCE-3048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-3048. Resolution: Fixed Assignee: Vinod Kumar Vavilapalli I don

Re: [jira] [Commented] (MAPREDUCE-3065) ApplicationMaster killed by NodeManager due to excessive virtual memory consumption

Wow, this is new to me. Chris, I just created MAPREDUCE-3068and assigned it to you. Please reassign if you don't have cycles. Thanks, +Vinod On Thu, Sep 22, 2011 at 5:35 AM, Chris Riccomini (JIRA) wrote: > >[ > https://issues.apache.org

[jira] [Created] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers

Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers --- Key: MAPREDUCE-3068 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 Project: Hadoop Map/Reduce

Research on Hadoop Design

Hi everyone, We are a group of researchers at DePaul University investigating the use of tactics in open-source, high-performance, fault-tolerant software systems. Obviously we are not nearly as familiar with Hadoop as all of the developers are, so we hoped you could help us. We have focused p

[jira] [Created] (MAPREDUCE-3067) Container exit status not set properly to launched process's exit code on successful completion of process

Container exit status not set properly to launched process's exit code on successful completion of process -- Key: MAPREDUCE-3067 URL: https://issues.apache.or

Hadoop-Mapreduce-trunk-Commit - Build # 953 - Still Failing

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/953/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14024 lines...] [junit] Running org.apache

Hadoop-Mapreduce-trunk-Commit - Build # 952 - Still Failing

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/952/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14024 lines...] [junit] Running org.apache

[jira] [Created] (MAPREDUCE-3066) YARN NM fails to start

YARN NM fails to start -- Key: MAPREDUCE-3066 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3066 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 0.23.0 En

[jira] [Created] (MAPREDUCE-3065) ApplicationMaster killed by NodeManager due to excessive virtual memory consumption

ApplicationMaster killed by NodeManager due to excessive virtual memory consumption --- Key: MAPREDUCE-3065 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3065 Proj

Hadoop-Mapreduce-trunk-Commit - Build # 951 - Still Failing

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/951/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14024 lines...] [junit] Running org.apache

Re: ApplicationMaster Memory Usage

Thanks Chris, please do open a jira and we can further discuss. On Sep 21, 2011, at 1:51 PM, Chris Riccomini wrote: > Hey Vinod, > > OK, so I have a little more clarity into this. > > When I bump my resource request for my AM to 4096, it runs. The important > line in the NM logs is: > > 2011-

RE: ApplicationMaster Memory Usage

Hey Vinod, OK, so I have a little more clarity into this. When I bump my resource request for my AM to 4096, it runs. The important line in the NM logs is: 2011-09-21 13:43:44,366 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(402)) - Memory usage of ProcessTree 25656 for

Re: ApplicationMaster Memory Usage

For the record, I bumped to 4096 for memory resource request, and it works. :( On 9/21/11 1:32 PM, "Chris Riccomini" wrote: > Hey Vinod, > > So, I ran my application master directly from the CLI. I commented out the > YARN-specific code. It runs fine without leaking memory. > > I then ran it

RE: ApplicationMaster Memory Usage

Hey Vinod, So, I ran my application master directly from the CLI. I commented out the YARN-specific code. It runs fine without leaking memory. I then ran it from YARN, with all YARN-specific code commented it. It again ran fine. I then uncommented JUST my registerWithResourceManager call. It t

Re: ApplicationMaster Memory Usage

Actually MAPREDUCE-2998 is only related to MRV2, so that isn't related. Somehow, your JVM itself is taking so much of virtual memory. Are you loading some native libs? And how many containers have already been allocated by the time the AM crashes. May be you are accumulating some per-container da

Hadoop-Mapreduce-trunk-Commit - Build # 950 - Still Failing

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/950/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14024 lines...] [junit] Running org.apache

RE: ApplicationMaster Memory Usage

Hey Vinod, I svn up'd, and rebuilt. My application's task (container) now runs! Unfortunately, my application master eventually gets killed by the NodeManager anyway, and I'm still not clear as to why. The AM is just running a loop, asking for a container, and executing a command in the contain

[jira] [Resolved] (MAPREDUCE-3063) Mapreduce trunk Commit builds are failing

[ https://issues.apache.org/jira/browse/MAPREDUCE-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved MAPREDUCE-3063. -- Resolution: Duplicate this is duplicate of MAPREDUCE-3064 > Mapreduce trunk Commit

[jira] [Created] (MAPREDUCE-3064) 27 unit test failures with Invalid "mapreduce.jobtracker.address" configuration value for JobTracker: "local"

27 unit test failures with Invalid "mapreduce.jobtracker.address" configuration value for JobTracker: "local" -- Key: MAPREDUCE-3064 URL: https://issues.a

[jira] [Created] (MAPREDUCE-3063) Mapreduce trunk Commit builds are failing

Mapreduce trunk Commit builds are failing - Key: MAPREDUCE-3063 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3063 Project: Hadoop Map/Reduce Issue Type: Bug Components: build

Re: LocalResource and Container

Hey Vinod, Got it, thanks. Updated the WritingYarnApps FAQ. Cheers, Chris On 9/21/11 10:18 AM, "Vinod Kumar Vavilapalli" wrote: > Container needs to specify its own resources to be localized (Juts like the > AM Container). So you should just set the same tgz that is there on DFS as > one of t

[jira] [Created] (MAPREDUCE-3062) YARN NM/RM fail to start

YARN NM/RM fail to start Key: MAPREDUCE-3062 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3062 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager, resourcemanager Affects Version

Hadoop-Mapreduce-trunk-Commit - Build # 949 - Still Failing

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/949/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14026 lines...] [junit] Test org.apache.ha

Hadoop-Mapreduce-0.23-Build - Build # 20 - Still Failing

See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/20/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 19028 lines...] [INFO] Executing tasks main: [IN

Re: LocalResource and Container

Container needs to specify its own resources to be localized (Juts like the AM Container). So you should just set the same tgz that is there on DFS as one of the resources for the container. +Vinod On Wed, Sep 21, 2011 at 10:41 PM, Chris Riccomini wrote: > Hey Guys, > > I'm currently setting a

[jira] [Created] (MAPREDUCE-3061) Generic application history service

Generic application history service --- Key: MAPREDUCE-3061 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3061 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects V

LocalResource and Container

Hey Guys, I'm currently setting a local resource for my ApplicationMaster's container ctx when my client sets up its request. What surprised me is that the containers that my ApplicationMaster starts up don't have those local resources as well. When I find . on the container working directory,

Re: ApplicationMaster Memory Usage

Yes, the process-dump clearly tells that this is MAPREDUCE-2998. +Vinod (With a smirk to see his container-memory-monitoring code in action) On Wed, Sep 21, 2011 at 10:26 PM, Arun C Murthy wrote: > I'll bet you are hitting MR-2998. > > From the changelog: > >MAPREDUCE-2998. Fixed a bug in

Re: ApplicationMaster Memory Usage

Appears likely! Will svn up and try. Glad it got committed today. :) On 9/21/11 9:56 AM, "Arun C Murthy" wrote: > I'll bet you are hitting MR-2998. > > From the changelog: > > MAPREDUCE-2998. Fixed a bug in TaskAttemptImpl which caused it to fork > bin/mapred too many times. Contributed b

Re: ApplicationMaster Memory Usage

I'll bet you are hitting MR-2998. From the changelog: MAPREDUCE-2998. Fixed a bug in TaskAttemptImpl which caused it to fork bin/mapred too many times. Contributed by Vinod K V. Arun On Sep 21, 2011, at 9:52 AM, Chris Riccomini wrote: > Hey Guys, > > My ApplicationMaster is being killed

[jira] [Resolved] (MAPREDUCE-2252) XSS injection in JobHistoryParser

[ https://issues.apache.org/jira/browse/MAPREDUCE-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko resolved MAPREDUCE-2252. Resolution: Not A Problem Sounds like it's not a problem anymore. > XS

ApplicationMaster Memory Usage

Hey Guys, My ApplicationMaster is being killed by the NodeManager because of memory consumption, and I don't understand why. I'm using -Xmx512M, and setting my resource request to 2048. .addCommand("java -Xmx512M -cp './package/*' kafka.yarn.ApplicationMaster " ... ... private v

[jira] [Resolved] (MAPREDUCE-2939) Ant setup on hadoop7 jenkins host

[ https://issues.apache.org/jira/browse/MAPREDUCE-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko resolved MAPREDUCE-2939. Resolution: Fixed Jenkins build has been modified to point to the new t

[jira] [Created] (MAPREDUCE-3060) Generic shuffle service

Generic shuffle service --- Key: MAPREDUCE-3060 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3060 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0

Hadoop-Mapreduce-trunk-Commit - Build # 948 - Still Failing

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/948/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 33650 lines...] [junit] at org.apache

Hadoop-Mapreduce-trunk-Commit - Build # 947 - Still Failing

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/947/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 33572 lines...] [junit] at org.apache

[jira] [Created] (MAPREDUCE-3059) Resourcemanager metrices does not have aggregate containers allocated and containers eeleased Metrics

Resourcemanager metrices does not have aggregate containers allocated and containers eeleased Metrics - Key: MAPREDUCE-3059 URL: https://issues.apache.org/jira/bro

Hadoop-Mapreduce-0.23-Build - Build # 19 - Still Failing

See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/19/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 19030 lines...] main: [INFO] Executed tasks [INFO

[jira] [Created] (MAPREDUCE-3058) Sometimes task keeps on running while its Syslog says that it is shutdown

Sometimes task keeps on running while its Syslog says that it is shutdown - Key: MAPREDUCE-3058 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3058 Project: Hadoop Map/Reduc

[jira] [Created] (MAPREDUCE-3057) Job History Server goes of OutOfMemory with 1200 Jobs and Heap Size set to 10 GB

Job History Server goes of OutOfMemory with 1200 Jobs and Heap Size set to 10 GB Key: MAPREDUCE-3057 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3057 Project: Ha

[jira] [Created] (MAPREDUCE-3056) Jobs are failing when those are submitted by other users

Jobs are failing when those are submitted by other users Key: MAPREDUCE-3056 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3056 Project: Hadoop Map/Reduce Issue Type: Bug

Hadoop-Mapreduce-trunk-Commit - Build # 946 - Failure

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/946/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 33619 lines...] [junit] at org.apache