[jira] [Updated] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4278: -- Attachment: MAPREDUCE-4278-branch1.patch > cannot run two local jobs in parallel from the same gateway. > > > Key: MAPREDUCE-4278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Araceli Henley > Attachments: MAPREDUCE-4278-branch1.patch > > > I cannot run two local mode jobs from Pig in parallel from the same gateway, > this is a typical use case. If I re-run the tests sequentially, then the test > pass. This seems to be a problem from Hadoop. > Additionally, the pig harness, expects to be able to run > Pig-version-undertest against Pig-version-stable from the same gateway. > To replicate the error: > I have two clusters running from the same gateway. > If I run the Pig regression suites nightly.conf in local mode in paralell - > once on each cluster. Conflicts in M/R local mode result in failures in the > tests. > ERROR1: > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > output/file.out in any of the configured local directories > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) > at > org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) > at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) > at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) > at org.apache.hadoop.mapred.Task.done(Task.java:875) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) > --- > ERROR2: > 2012-05-17 20:25:36,762 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - > HadoopJobId: job_local_0001 > 2012-05-17 20:25:36,778 [Thread-3] INFO org.apache.hadoop.mapred.Task - > Using ResourceCalculatorPlugin : org.apache. > hadoop.util.LinuxResourceCalculatorPlugin@ffa490e > 2012-05-17 20:25:36,837 [Thread-3] WARN > org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 > java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 > at java.util.ArrayList.RangeCheck(ArrayList.java:547) > at java.util.ArrayList.get(ArrayList.java:322) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java > :153) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm > at.java:106) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.java:489) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) > 2012-05-17 20:25:41,291 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4278: -- Affects Version/s: (was: 1.0.2) (was: 0.23.1) Status: Patch Available (was: Open) > cannot run two local jobs in parallel from the same gateway. > > > Key: MAPREDUCE-4278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Araceli Henley > Attachments: MAPREDUCE-4278-branch1.patch > > > I cannot run two local mode jobs from Pig in parallel from the same gateway, > this is a typical use case. If I re-run the tests sequentially, then the test > pass. This seems to be a problem from Hadoop. > Additionally, the pig harness, expects to be able to run > Pig-version-undertest against Pig-version-stable from the same gateway. > To replicate the error: > I have two clusters running from the same gateway. > If I run the Pig regression suites nightly.conf in local mode in paralell - > once on each cluster. Conflicts in M/R local mode result in failures in the > tests. > ERROR1: > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > output/file.out in any of the configured local directories > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) > at > org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) > at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) > at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) > at org.apache.hadoop.mapred.Task.done(Task.java:875) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) > --- > ERROR2: > 2012-05-17 20:25:36,762 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - > HadoopJobId: job_local_0001 > 2012-05-17 20:25:36,778 [Thread-3] INFO org.apache.hadoop.mapred.Task - > Using ResourceCalculatorPlugin : org.apache. > hadoop.util.LinuxResourceCalculatorPlugin@ffa490e > 2012-05-17 20:25:36,837 [Thread-3] WARN > org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 > java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 > at java.util.ArrayList.RangeCheck(ArrayList.java:547) > at java.util.ArrayList.get(ArrayList.java:322) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java > :153) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm > at.java:106) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.java:489) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) > 2012-05-17 20:25:41,291 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4278: -- Attachment: MAPREDUCE-4278-trunk.patch > cannot run two local jobs in parallel from the same gateway. > > > Key: MAPREDUCE-4278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Araceli Henley > Attachments: MAPREDUCE-4278-2-branch1.patch, > MAPREDUCE-4278-branch1.patch, MAPREDUCE-4278-trunk.patch > > > I cannot run two local mode jobs from Pig in parallel from the same gateway, > this is a typical use case. If I re-run the tests sequentially, then the test > pass. This seems to be a problem from Hadoop. > Additionally, the pig harness, expects to be able to run > Pig-version-undertest against Pig-version-stable from the same gateway. > To replicate the error: > I have two clusters running from the same gateway. > If I run the Pig regression suites nightly.conf in local mode in paralell - > once on each cluster. Conflicts in M/R local mode result in failures in the > tests. > ERROR1: > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > output/file.out in any of the configured local directories > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) > at > org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) > at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) > at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) > at org.apache.hadoop.mapred.Task.done(Task.java:875) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) > --- > ERROR2: > 2012-05-17 20:25:36,762 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - > HadoopJobId: job_local_0001 > 2012-05-17 20:25:36,778 [Thread-3] INFO org.apache.hadoop.mapred.Task - > Using ResourceCalculatorPlugin : org.apache. > hadoop.util.LinuxResourceCalculatorPlugin@ffa490e > 2012-05-17 20:25:36,837 [Thread-3] WARN > org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 > java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 > at java.util.ArrayList.RangeCheck(ArrayList.java:547) > at java.util.ArrayList.get(ArrayList.java:322) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java > :153) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm > at.java:106) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.java:489) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) > 2012-05-17 20:25:41,291 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4278: -- Attachment: MAPREDUCE-4278-2-branch1.patch > cannot run two local jobs in parallel from the same gateway. > > > Key: MAPREDUCE-4278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Araceli Henley > Attachments: MAPREDUCE-4278-2-branch1.patch, > MAPREDUCE-4278-branch1.patch, MAPREDUCE-4278-trunk.patch > > > I cannot run two local mode jobs from Pig in parallel from the same gateway, > this is a typical use case. If I re-run the tests sequentially, then the test > pass. This seems to be a problem from Hadoop. > Additionally, the pig harness, expects to be able to run > Pig-version-undertest against Pig-version-stable from the same gateway. > To replicate the error: > I have two clusters running from the same gateway. > If I run the Pig regression suites nightly.conf in local mode in paralell - > once on each cluster. Conflicts in M/R local mode result in failures in the > tests. > ERROR1: > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > output/file.out in any of the configured local directories > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) > at > org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) > at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) > at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) > at org.apache.hadoop.mapred.Task.done(Task.java:875) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) > --- > ERROR2: > 2012-05-17 20:25:36,762 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - > HadoopJobId: job_local_0001 > 2012-05-17 20:25:36,778 [Thread-3] INFO org.apache.hadoop.mapred.Task - > Using ResourceCalculatorPlugin : org.apache. > hadoop.util.LinuxResourceCalculatorPlugin@ffa490e > 2012-05-17 20:25:36,837 [Thread-3] WARN > org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 > java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 > at java.util.ArrayList.RangeCheck(ArrayList.java:547) > at java.util.ArrayList.get(ArrayList.java:322) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java > :153) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm > at.java:106) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.java:489) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) > 2012-05-17 20:25:41,291 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4278: -- Attachment: MAPREDUCE-4278-trunk.patch > cannot run two local jobs in parallel from the same gateway. > > > Key: MAPREDUCE-4278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Araceli Henley > Attachments: MAPREDUCE-4278-2-branch1.patch, > MAPREDUCE-4278-branch1.patch, MAPREDUCE-4278-trunk.patch, > MAPREDUCE-4278-trunk.patch > > > I cannot run two local mode jobs from Pig in parallel from the same gateway, > this is a typical use case. If I re-run the tests sequentially, then the test > pass. This seems to be a problem from Hadoop. > Additionally, the pig harness, expects to be able to run > Pig-version-undertest against Pig-version-stable from the same gateway. > To replicate the error: > I have two clusters running from the same gateway. > If I run the Pig regression suites nightly.conf in local mode in paralell - > once on each cluster. Conflicts in M/R local mode result in failures in the > tests. > ERROR1: > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > output/file.out in any of the configured local directories > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) > at > org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) > at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) > at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) > at org.apache.hadoop.mapred.Task.done(Task.java:875) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) > --- > ERROR2: > 2012-05-17 20:25:36,762 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - > HadoopJobId: job_local_0001 > 2012-05-17 20:25:36,778 [Thread-3] INFO org.apache.hadoop.mapred.Task - > Using ResourceCalculatorPlugin : org.apache. > hadoop.util.LinuxResourceCalculatorPlugin@ffa490e > 2012-05-17 20:25:36,837 [Thread-3] WARN > org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 > java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 > at java.util.ArrayList.RangeCheck(ArrayList.java:547) > at java.util.ArrayList.get(ArrayList.java:322) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java > :153) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm > at.java:106) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.java:489) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) > 2012-05-17 20:25:41,291 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4278: -- Attachment: MAPREDUCE-4278-3-branch1.patch > cannot run two local jobs in parallel from the same gateway. > > > Key: MAPREDUCE-4278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Araceli Henley >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4278-2-branch1.patch, > MAPREDUCE-4278-3-branch1.patch, MAPREDUCE-4278-branch1.patch, > MAPREDUCE-4278-trunk.patch, MAPREDUCE-4278-trunk.patch > > > I cannot run two local mode jobs from Pig in parallel from the same gateway, > this is a typical use case. If I re-run the tests sequentially, then the test > pass. This seems to be a problem from Hadoop. > Additionally, the pig harness, expects to be able to run > Pig-version-undertest against Pig-version-stable from the same gateway. > To replicate the error: > I have two clusters running from the same gateway. > If I run the Pig regression suites nightly.conf in local mode in paralell - > once on each cluster. Conflicts in M/R local mode result in failures in the > tests. > ERROR1: > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > output/file.out in any of the configured local directories > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) > at > org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) > at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) > at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) > at org.apache.hadoop.mapred.Task.done(Task.java:875) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) > --- > ERROR2: > 2012-05-17 20:25:36,762 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - > HadoopJobId: job_local_0001 > 2012-05-17 20:25:36,778 [Thread-3] INFO org.apache.hadoop.mapred.Task - > Using ResourceCalculatorPlugin : org.apache. > hadoop.util.LinuxResourceCalculatorPlugin@ffa490e > 2012-05-17 20:25:36,837 [Thread-3] WARN > org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 > java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 > at java.util.ArrayList.RangeCheck(ArrayList.java:547) > at java.util.ArrayList.get(ArrayList.java:322) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java > :153) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm > at.java:106) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.java:489) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) > 2012-05-17 20:25:41,291 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-4278: - Resolution: Fixed Fix Version/s: 2.0.3-alpha 1.2.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 I just committed this. Thanks, Sandy! > cannot run two local jobs in parallel from the same gateway. > > > Key: MAPREDUCE-4278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Araceli Henley >Assignee: Sandy Ryza > Fix For: 1.2.0, 2.0.3-alpha > > Attachments: MAPREDUCE-4278-2-branch1.patch, > MAPREDUCE-4278-3-branch1.patch, MAPREDUCE-4278-branch1.patch, > MAPREDUCE-4278-trunk.patch, MAPREDUCE-4278-trunk.patch > > > I cannot run two local mode jobs from Pig in parallel from the same gateway, > this is a typical use case. If I re-run the tests sequentially, then the test > pass. This seems to be a problem from Hadoop. > Additionally, the pig harness, expects to be able to run > Pig-version-undertest against Pig-version-stable from the same gateway. > To replicate the error: > I have two clusters running from the same gateway. > If I run the Pig regression suites nightly.conf in local mode in paralell - > once on each cluster. Conflicts in M/R local mode result in failures in the > tests. > ERROR1: > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > output/file.out in any of the configured local directories > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) > at > org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) > at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) > at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) > at org.apache.hadoop.mapred.Task.done(Task.java:875) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) > --- > ERROR2: > 2012-05-17 20:25:36,762 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - > HadoopJobId: job_local_0001 > 2012-05-17 20:25:36,778 [Thread-3] INFO org.apache.hadoop.mapred.Task - > Using ResourceCalculatorPlugin : org.apache. > hadoop.util.LinuxResourceCalculatorPlugin@ffa490e > 2012-05-17 20:25:36,837 [Thread-3] WARN > org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 > java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 > at java.util.ArrayList.RangeCheck(ArrayList.java:547) > at java.util.ArrayList.get(ArrayList.java:322) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java > :153) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm > at.java:106) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.java:489) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) > 2012-05-17 20:25:41,291 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4278: - Fix Version/s: 0.23.7 merged to branch-0.23 > cannot run two local jobs in parallel from the same gateway. > > > Key: MAPREDUCE-4278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Araceli Henley >Assignee: Sandy Ryza > Fix For: 1.2.0, 2.0.3-alpha, 0.23.7 > > Attachments: MAPREDUCE-4278-2-branch1.patch, > MAPREDUCE-4278-3-branch1.patch, MAPREDUCE-4278-branch1.patch, > MAPREDUCE-4278-trunk.patch, MAPREDUCE-4278-trunk.patch > > > I cannot run two local mode jobs from Pig in parallel from the same gateway, > this is a typical use case. If I re-run the tests sequentially, then the test > pass. This seems to be a problem from Hadoop. > Additionally, the pig harness, expects to be able to run > Pig-version-undertest against Pig-version-stable from the same gateway. > To replicate the error: > I have two clusters running from the same gateway. > If I run the Pig regression suites nightly.conf in local mode in paralell - > once on each cluster. Conflicts in M/R local mode result in failures in the > tests. > ERROR1: > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > output/file.out in any of the configured local directories > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) > at > org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) > at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) > at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) > at org.apache.hadoop.mapred.Task.done(Task.java:875) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) > --- > ERROR2: > 2012-05-17 20:25:36,762 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - > HadoopJobId: job_local_0001 > 2012-05-17 20:25:36,778 [Thread-3] INFO org.apache.hadoop.mapred.Task - > Using ResourceCalculatorPlugin : org.apache. > hadoop.util.LinuxResourceCalculatorPlugin@ffa490e > 2012-05-17 20:25:36,837 [Thread-3] WARN > org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 > java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 > at java.util.ArrayList.RangeCheck(ArrayList.java:547) > at java.util.ArrayList.get(ArrayList.java:322) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java > :153) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm > at.java:106) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.java:489) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) > 2012-05-17 20:25:41,291 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
