[jira] [Commented] (MAPREDUCE-5069) add concrete common implementations of CombineFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733999#comment-13733999 ] Derek Farren commented on MAPREDUCE-5069: - You're right. My apologies. > add concrete common implementations of CombineFileInputFormat > - > > Key: MAPREDUCE-5069 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5069 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2 >Affects Versions: 2.0.3-alpha >Reporter: Sangjin Lee >Priority: Minor > Fix For: 3.0.0, 2.1.0-beta > > Attachments: MAPREDUCE-5069-1.patch, MAPREDUCE-5069-2.patch, > MAPREDUCE-5069-3.patch, MAPREDUCE-5069-4.patch, MAPREDUCE-5069-5.patch, > MAPREDUCE-5069-6.patch, MAPREDUCE-5069.patch > > > CombineFileInputFormat is abstract, and its specific equivalents to > TextInputFormat, SequenceFileInputFormat, etc. are currently not in the > hadoop code base. > These sound like very common need wherever CombineFileInputFormat is used, > and different folks would write the same code over and over to achieve the > same goal. It sounds very natural for hadoop to provide at least the text and > sequence file implementations of the CombineFileInputFormat class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5069) add concrete common implementations of CombineFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733962#comment-13733962 ] Jason Lowe commented on MAPREDUCE-5069: --- SequenceFileInputFormat extends FileInputFormat, and CombineSequenceFileInputFormat.SequenceFileRecordReaderWrapper is an example where the original CombineFileRecordReaderWrapper constructor is being called with a SequenceFileInputFormat instance. Could you elaborate on why you believe this doesn't work on sequence files or provide an example? > add concrete common implementations of CombineFileInputFormat > - > > Key: MAPREDUCE-5069 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5069 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2 >Affects Versions: 2.0.3-alpha >Reporter: Sangjin Lee >Priority: Minor > Fix For: 3.0.0, 2.1.0-beta > > Attachments: MAPREDUCE-5069-1.patch, MAPREDUCE-5069-2.patch, > MAPREDUCE-5069-3.patch, MAPREDUCE-5069-4.patch, MAPREDUCE-5069-5.patch, > MAPREDUCE-5069-6.patch, MAPREDUCE-5069.patch > > > CombineFileInputFormat is abstract, and its specific equivalents to > TextInputFormat, SequenceFileInputFormat, etc. are currently not in the > hadoop code base. > These sound like very common need wherever CombineFileInputFormat is used, > and different folks would write the same code over and over to achieve the > same goal. It sounds very natural for hadoop to provide at least the text and > sequence file implementations of the CombineFileInputFormat class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5069) add concrete common implementations of CombineFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733881#comment-13733881 ] Derek Farren commented on MAPREDUCE-5069: - It doesn't work on sequence files because the CombineFileRecordReaderWrapper constructor deals only with first argument type FileInputFormat. It needs another constructor that can deal with SequenceFileInputFormat. This is what we have: CombineFileRecordReaderWrapper(FileInputFormat inputFormat, CombineFileSplit split, TaskAttemptContext context, Integer idx) This is what we need to add: CombineFileRecordReaderWrapper(SequenceFileInputFormat sequenceFileInputFormat, CombineFileSplit split, TaskAttemptContext context, Integer idx) > add concrete common implementations of CombineFileInputFormat > - > > Key: MAPREDUCE-5069 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5069 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2 >Affects Versions: 2.0.3-alpha >Reporter: Sangjin Lee >Priority: Minor > Fix For: 3.0.0, 2.1.0-beta > > Attachments: MAPREDUCE-5069-1.patch, MAPREDUCE-5069-2.patch, > MAPREDUCE-5069-3.patch, MAPREDUCE-5069-4.patch, MAPREDUCE-5069-5.patch, > MAPREDUCE-5069-6.patch, MAPREDUCE-5069.patch > > > CombineFileInputFormat is abstract, and its specific equivalents to > TextInputFormat, SequenceFileInputFormat, etc. are currently not in the > hadoop code base. > These sound like very common need wherever CombineFileInputFormat is used, > and different folks would write the same code over and over to achieve the > same goal. It sounds very natural for hadoop to provide at least the text and > sequence file implementations of the CombineFileInputFormat class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733836#comment-13733836 ] Sandy Ryza commented on MAPREDUCE-5450: --- Thanks Arun > Unnecessary Configuration instantiation in IFileInputStream slows down merge > - Port to branch-1 > --- > > Key: MAPREDUCE-5450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.0 >Reporter: Stanislav Barton >Assignee: Stanislav Barton >Priority: Blocker > Fix For: 1.2.1 > > Attachments: MAPREDUCE-5450-1.1.0.txt, mapreduce-5450.txt > > > We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from > 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job > in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on > input each about 100MB) and 6 000 reducers (one reducer per table region). I > was trying to figure out what at which phase the slow down appears (firstly I > suspected that the slow gathering of the 1 map output files is the > culprit) and found out that the problem is not reading the map output (the > shuffle) but the sort/merge phase that follows - the last and actual reduce > phase is fast. I have tried to up the io.sort.factor because I thought the > lots of small files are being merged on disk, but again upping that to 1000 > didnt do any difference. I have then printed the stack trace and found out > that the problem is initialization of the > org.apache.hadoop.mapred.IFileInputStream namely the creation of the > Configuration object which is not propagated along from earlier context, see > the stack trace: > Thread 13332: (state = IN_NATIVE) > - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 > (Compiled frame; information may be imprecise) > - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 > (Compiled frame) > - java.io.File.exists() @bci=20, line=733 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) > @bci=136, line=999 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) > @bci=3, line=966 (Compiled frame) > - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, > line=146 (Compiled frame) > - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction, > java.security.AccessControlContext) @bci=0 (Compiled frame) > - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 > (Compiled frame) > - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 > (Compiled frame) > - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, > line=1192 (Compiled frame) > - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction) > @bci=0 (Compiled frame) > - > javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, > java.lang.String) @bci=10, line=89 (Compiled frame) > - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) > @bci=38, line=250 (Interpreted frame) > - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) > @bci=273, line=223 (Interpreted frame) > - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, > org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, > java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, > line=712 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, > line=731 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) > @bci=2, line=1047 (Interpreted frame) > - org.apache.hadoop.mapred.IFileInputStream.(java.io.InputStream, > long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted > frame) > - > org.apache.hadoop.mapred.IFile$Reader.(org.apache.hadoop.conf.Configuration, > org.apache.hadoop.fs.FSDataInputStream, long, > org.apache.hadoop.io.compress.CompressionCodec, > org.apache.hadoop.mapred.Counters$Counter)
[jira] [Updated] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5450: - Fix Version/s: (was: 1.2.1) 1.2.2 > Unnecessary Configuration instantiation in IFileInputStream slows down merge > - Port to branch-1 > --- > > Key: MAPREDUCE-5450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.0 >Reporter: Stanislav Barton >Assignee: Stanislav Barton >Priority: Blocker > Fix For: 1.2.2 > > Attachments: MAPREDUCE-5450-1.1.0.txt, mapreduce-5450.txt > > > We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from > 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job > in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on > input each about 100MB) and 6 000 reducers (one reducer per table region). I > was trying to figure out what at which phase the slow down appears (firstly I > suspected that the slow gathering of the 1 map output files is the > culprit) and found out that the problem is not reading the map output (the > shuffle) but the sort/merge phase that follows - the last and actual reduce > phase is fast. I have tried to up the io.sort.factor because I thought the > lots of small files are being merged on disk, but again upping that to 1000 > didnt do any difference. I have then printed the stack trace and found out > that the problem is initialization of the > org.apache.hadoop.mapred.IFileInputStream namely the creation of the > Configuration object which is not propagated along from earlier context, see > the stack trace: > Thread 13332: (state = IN_NATIVE) > - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 > (Compiled frame; information may be imprecise) > - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 > (Compiled frame) > - java.io.File.exists() @bci=20, line=733 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) > @bci=136, line=999 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) > @bci=3, line=966 (Compiled frame) > - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, > line=146 (Compiled frame) > - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction, > java.security.AccessControlContext) @bci=0 (Compiled frame) > - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 > (Compiled frame) > - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 > (Compiled frame) > - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, > line=1192 (Compiled frame) > - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction) > @bci=0 (Compiled frame) > - > javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, > java.lang.String) @bci=10, line=89 (Compiled frame) > - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) > @bci=38, line=250 (Interpreted frame) > - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) > @bci=273, line=223 (Interpreted frame) > - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, > org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, > java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, > line=712 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, > line=731 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) > @bci=2, line=1047 (Interpreted frame) > - org.apache.hadoop.mapred.IFileInputStream.(java.io.InputStream, > long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted > frame) > - > org.apache.hadoop.mapred.IFile$Reader.(org.apache.hadoop.conf.Configuration, > org.apache.hadoop.fs.FSDataInputStream, long, > org.apache.hadoop.io.compress.CompressionCodec, > org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=
[jira] [Commented] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733834#comment-13733834 ] Arun C Murthy commented on MAPREDUCE-5450: -- I merged this to branch-1.2 too. Thanks guys! > Unnecessary Configuration instantiation in IFileInputStream slows down merge > - Port to branch-1 > --- > > Key: MAPREDUCE-5450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.0 >Reporter: Stanislav Barton >Assignee: Stanislav Barton >Priority: Blocker > Fix For: 1.2.1 > > Attachments: MAPREDUCE-5450-1.1.0.txt, mapreduce-5450.txt > > > We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from > 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job > in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on > input each about 100MB) and 6 000 reducers (one reducer per table region). I > was trying to figure out what at which phase the slow down appears (firstly I > suspected that the slow gathering of the 1 map output files is the > culprit) and found out that the problem is not reading the map output (the > shuffle) but the sort/merge phase that follows - the last and actual reduce > phase is fast. I have tried to up the io.sort.factor because I thought the > lots of small files are being merged on disk, but again upping that to 1000 > didnt do any difference. I have then printed the stack trace and found out > that the problem is initialization of the > org.apache.hadoop.mapred.IFileInputStream namely the creation of the > Configuration object which is not propagated along from earlier context, see > the stack trace: > Thread 13332: (state = IN_NATIVE) > - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 > (Compiled frame; information may be imprecise) > - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 > (Compiled frame) > - java.io.File.exists() @bci=20, line=733 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) > @bci=136, line=999 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) > @bci=3, line=966 (Compiled frame) > - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, > line=146 (Compiled frame) > - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction, > java.security.AccessControlContext) @bci=0 (Compiled frame) > - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 > (Compiled frame) > - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 > (Compiled frame) > - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, > line=1192 (Compiled frame) > - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction) > @bci=0 (Compiled frame) > - > javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, > java.lang.String) @bci=10, line=89 (Compiled frame) > - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) > @bci=38, line=250 (Interpreted frame) > - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) > @bci=273, line=223 (Interpreted frame) > - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, > org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, > java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, > line=712 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, > line=731 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) > @bci=2, line=1047 (Interpreted frame) > - org.apache.hadoop.mapred.IFileInputStream.(java.io.InputStream, > long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted > frame) > - > org.apache.hadoop.mapred.IFile$Reader.(org.apache.hadoop.conf.Configuration, > org.apache.hadoop.fs.FSDataInputStream, long, > org.apache.hadoop.io.compress.CompressionCodec, > org
[jira] [Updated] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5450: - Fix Version/s: (was: 1.3.0) 1.2.1 > Unnecessary Configuration instantiation in IFileInputStream slows down merge > - Port to branch-1 > --- > > Key: MAPREDUCE-5450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.0 >Reporter: Stanislav Barton >Assignee: Stanislav Barton >Priority: Blocker > Fix For: 1.2.1 > > Attachments: MAPREDUCE-5450-1.1.0.txt, mapreduce-5450.txt > > > We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from > 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job > in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on > input each about 100MB) and 6 000 reducers (one reducer per table region). I > was trying to figure out what at which phase the slow down appears (firstly I > suspected that the slow gathering of the 1 map output files is the > culprit) and found out that the problem is not reading the map output (the > shuffle) but the sort/merge phase that follows - the last and actual reduce > phase is fast. I have tried to up the io.sort.factor because I thought the > lots of small files are being merged on disk, but again upping that to 1000 > didnt do any difference. I have then printed the stack trace and found out > that the problem is initialization of the > org.apache.hadoop.mapred.IFileInputStream namely the creation of the > Configuration object which is not propagated along from earlier context, see > the stack trace: > Thread 13332: (state = IN_NATIVE) > - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 > (Compiled frame; information may be imprecise) > - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 > (Compiled frame) > - java.io.File.exists() @bci=20, line=733 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) > @bci=136, line=999 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) > @bci=3, line=966 (Compiled frame) > - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, > line=146 (Compiled frame) > - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction, > java.security.AccessControlContext) @bci=0 (Compiled frame) > - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 > (Compiled frame) > - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 > (Compiled frame) > - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, > line=1192 (Compiled frame) > - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction) > @bci=0 (Compiled frame) > - > javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, > java.lang.String) @bci=10, line=89 (Compiled frame) > - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) > @bci=38, line=250 (Interpreted frame) > - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) > @bci=273, line=223 (Interpreted frame) > - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, > org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, > java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, > line=712 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, > line=731 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) > @bci=2, line=1047 (Interpreted frame) > - org.apache.hadoop.mapred.IFileInputStream.(java.io.InputStream, > long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted > frame) > - > org.apache.hadoop.mapred.IFile$Reader.(org.apache.hadoop.conf.Configuration, > org.apache.hadoop.fs.FSDataInputStream, long, > org.apache.hadoop.io.compress.CompressionCodec, > org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=
[jira] [Commented] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733826#comment-13733826 ] Arun C Murthy commented on MAPREDUCE-5450: -- Sorry, I meant for 1.2.2 (if necessary). > Unnecessary Configuration instantiation in IFileInputStream slows down merge > - Port to branch-1 > --- > > Key: MAPREDUCE-5450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.0 >Reporter: Stanislav Barton >Assignee: Stanislav Barton >Priority: Blocker > Fix For: 1.3.0 > > Attachments: MAPREDUCE-5450-1.1.0.txt, mapreduce-5450.txt > > > We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from > 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job > in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on > input each about 100MB) and 6 000 reducers (one reducer per table region). I > was trying to figure out what at which phase the slow down appears (firstly I > suspected that the slow gathering of the 1 map output files is the > culprit) and found out that the problem is not reading the map output (the > shuffle) but the sort/merge phase that follows - the last and actual reduce > phase is fast. I have tried to up the io.sort.factor because I thought the > lots of small files are being merged on disk, but again upping that to 1000 > didnt do any difference. I have then printed the stack trace and found out > that the problem is initialization of the > org.apache.hadoop.mapred.IFileInputStream namely the creation of the > Configuration object which is not propagated along from earlier context, see > the stack trace: > Thread 13332: (state = IN_NATIVE) > - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 > (Compiled frame; information may be imprecise) > - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 > (Compiled frame) > - java.io.File.exists() @bci=20, line=733 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) > @bci=136, line=999 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) > @bci=3, line=966 (Compiled frame) > - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, > line=146 (Compiled frame) > - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction, > java.security.AccessControlContext) @bci=0 (Compiled frame) > - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 > (Compiled frame) > - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 > (Compiled frame) > - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, > line=1192 (Compiled frame) > - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction) > @bci=0 (Compiled frame) > - > javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, > java.lang.String) @bci=10, line=89 (Compiled frame) > - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) > @bci=38, line=250 (Interpreted frame) > - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) > @bci=273, line=223 (Interpreted frame) > - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, > org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, > java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, > line=712 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, > line=731 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) > @bci=2, line=1047 (Interpreted frame) > - org.apache.hadoop.mapred.IFileInputStream.(java.io.InputStream, > long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted > frame) > - > org.apache.hadoop.mapred.IFile$Reader.(org.apache.hadoop.conf.Configuration, > org.apache.hadoop.fs.FSDataInputStream, long, > org.apache.hadoop.io.compress.CompressionCodec, > org.apac
[jira] [Commented] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733823#comment-13733823 ] Arun C Murthy commented on MAPREDUCE-5450: -- Let's get [~stanislav.barton]'s original patch into branch-1.2 for 1.2.1. Any objections? > Unnecessary Configuration instantiation in IFileInputStream slows down merge > - Port to branch-1 > --- > > Key: MAPREDUCE-5450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.1.0 >Reporter: Stanislav Barton >Assignee: Stanislav Barton >Priority: Blocker > Fix For: 1.3.0 > > Attachments: MAPREDUCE-5450-1.1.0.txt, mapreduce-5450.txt > > > We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from > 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job > in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on > input each about 100MB) and 6 000 reducers (one reducer per table region). I > was trying to figure out what at which phase the slow down appears (firstly I > suspected that the slow gathering of the 1 map output files is the > culprit) and found out that the problem is not reading the map output (the > shuffle) but the sort/merge phase that follows - the last and actual reduce > phase is fast. I have tried to up the io.sort.factor because I thought the > lots of small files are being merged on disk, but again upping that to 1000 > didnt do any difference. I have then printed the stack trace and found out > that the problem is initialization of the > org.apache.hadoop.mapred.IFileInputStream namely the creation of the > Configuration object which is not propagated along from earlier context, see > the stack trace: > Thread 13332: (state = IN_NATIVE) > - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 > (Compiled frame; information may be imprecise) > - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 > (Compiled frame) > - java.io.File.exists() @bci=20, line=733 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) > @bci=136, line=999 (Compiled frame) > - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) > @bci=3, line=966 (Compiled frame) > - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, > line=146 (Compiled frame) > - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction, > java.security.AccessControlContext) @bci=0 (Compiled frame) > - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 > (Compiled frame) > - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 > (Compiled frame) > - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, > line=1192 (Compiled frame) > - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) > - > java.security.AccessController.doPrivileged(java.security.PrivilegedAction) > @bci=0 (Compiled frame) > - > javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, > java.lang.String) @bci=10, line=89 (Compiled frame) > - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) > @bci=38, line=250 (Interpreted frame) > - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) > @bci=273, line=223 (Interpreted frame) > - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, > org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, > java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 > (Compiled frame) > - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, > line=712 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, > line=731 (Compiled frame) > - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) > @bci=2, line=1047 (Interpreted frame) > - org.apache.hadoop.mapred.IFileInputStream.(java.io.InputStream, > long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted > frame) > - > org.apache.hadoop.mapred.IFile$Reader.(org.apache.hadoop.conf.Configuration, > org.apache.hadoop.fs.FSDataInputStream, long, > org.apac
[jira] [Commented] (MAPREDUCE-5425) Junit in TestJobHistoryServer failing in jdk 7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733543#comment-13733543 ] Hudson commented on MAPREDUCE-5425: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1512 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1512/]) MAPREDUCE-5425. Junit in TestJobHistoryServer failing in jdk 7. Contributed by Robert Parker (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1511464) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryServer.java > Junit in TestJobHistoryServer failing in jdk 7 > -- > > Key: MAPREDUCE-5425 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5425 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.0.4-alpha >Reporter: Ashwin Shankar >Assignee: Robert Parker > Fix For: 3.0.0, 2.1.1-beta > > Attachments: MAPREDUCE-5425-2.patch, MAPREDUCE-5425-3.patch, > MAPREDUCE-5425.patch > > > We get the following exception when we run the unit tests of > TestJobHistoryServer with jdk 7: > Caused by: java.net.BindException: Problem binding to [0.0.0.0:10033] > java.net.BindException: Address already in use; For more details see: > http://wiki.apache.org/hadoop/BindException > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:719) > at org.apache.hadoop.ipc.Server.bind(Server.java:423) > at org.apache.hadoop.ipc.Server$Listener.(Server.java:535) > at org.apache.hadoop.ipc.Server.(Server.java:2202) > at org.apache.hadoop.ipc.RPC$Server.(RPC.java:901) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:505) > at > org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:480) > at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:746) > at > org.apache.hadoop.mapreduce.v2.hs.server.HSAdminServer.serviceInit(HSAdminServer.java:100) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > This is happening because testMainMethod starts the history server and doesnt > stop it. This worked in jdk 6 because tests executed sequentially and this > test was last one and didnt affect other tests,but in jdk 7 it fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1176) Contribution: FixedLengthInputFormat and FixedLengthRecordReader
[
https://issues.apache.org/jira/browse/MAPREDUCE-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733518#comment-13733518
]
BitsOfInfo commented on MAPREDUCE-1176:
---
Thanks, anyways, the implementation I originally threw together worked great
for our needs. If this is an optimization for it in just computing the split
size, then go ahead.
> Contribution: FixedLengthInputFormat and FixedLengthRecordReader
>
>
> Key: MAPREDUCE-1176
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1176
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Affects Versions: 0.20.1, 0.20.2
> Environment: Any
>Reporter: BitsOfInfo
>Assignee: Mariappan Asokan
> Attachments: MAPREDUCE-1176-v1.patch, MAPREDUCE-1176-v2.patch,
> MAPREDUCE-1176-v3.patch, MAPREDUCE-1176-v4.patch
>
>
> Hello,
> I would like to contribute the following two classes for incorporation into
> the mapreduce.lib.input package. These two classes can be used when you need
> to read data from files containing fixed length (fixed width) records. Such
> files have no CR/LF (or any combination thereof), no delimiters etc, but each
> record is a fixed length, and extra data is padded with spaces. The data is
> one gigantic line within a file.
> Provided are two classes first is the FixedLengthInputFormat and its
> corresponding FixedLengthRecordReader. When creating a job that specifies
> this input format, the job must have the
> "mapreduce.input.fixedlengthinputformat.record.length" property set as follows
> myJobConf.setInt("mapreduce.input.fixedlengthinputformat.record.length",[myFixedRecordLength]);
> OR
> myJobConf.setInt(FixedLengthInputFormat.FIXED_RECORD_LENGTH,
> [myFixedRecordLength]);
> This input format overrides computeSplitSize() in order to ensure that
> InputSplits do not contain any partial records since with fixed records there
> is no way to determine where a record begins if that were to occur. Each
> InputSplit passed to the FixedLengthRecordReader will start at the beginning
> of a record, and the last byte in the InputSplit will be the last byte of a
> record. The override of computeSplitSize() delegates to FileInputFormat's
> compute method, and then adjusts the returned split size by doing the
> following: (Math.floor(fileInputFormatsComputedSplitSize / fixedRecordLength)
> * fixedRecordLength)
> This suite of fixed length input format classes, does not support compressed
> files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5425) Junit in TestJobHistoryServer failing in jdk 7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733477#comment-13733477 ] Hudson commented on MAPREDUCE-5425: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1485 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1485/]) MAPREDUCE-5425. Junit in TestJobHistoryServer failing in jdk 7. Contributed by Robert Parker (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1511464) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryServer.java > Junit in TestJobHistoryServer failing in jdk 7 > -- > > Key: MAPREDUCE-5425 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5425 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.0.4-alpha >Reporter: Ashwin Shankar >Assignee: Robert Parker > Fix For: 3.0.0, 2.1.1-beta > > Attachments: MAPREDUCE-5425-2.patch, MAPREDUCE-5425-3.patch, > MAPREDUCE-5425.patch > > > We get the following exception when we run the unit tests of > TestJobHistoryServer with jdk 7: > Caused by: java.net.BindException: Problem binding to [0.0.0.0:10033] > java.net.BindException: Address already in use; For more details see: > http://wiki.apache.org/hadoop/BindException > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:719) > at org.apache.hadoop.ipc.Server.bind(Server.java:423) > at org.apache.hadoop.ipc.Server$Listener.(Server.java:535) > at org.apache.hadoop.ipc.Server.(Server.java:2202) > at org.apache.hadoop.ipc.RPC$Server.(RPC.java:901) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:505) > at > org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:480) > at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:746) > at > org.apache.hadoop.mapreduce.v2.hs.server.HSAdminServer.serviceInit(HSAdminServer.java:100) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > This is happening because testMainMethod starts the history server and doesnt > stop it. This worked in jdk 6 because tests executed sequentially and this > test was last one and didnt affect other tests,but in jdk 7 it fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5425) Junit in TestJobHistoryServer failing in jdk 7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733370#comment-13733370 ] Hudson commented on MAPREDUCE-5425: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #295 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/295/]) MAPREDUCE-5425. Junit in TestJobHistoryServer failing in jdk 7. Contributed by Robert Parker (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1511464) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryServer.java > Junit in TestJobHistoryServer failing in jdk 7 > -- > > Key: MAPREDUCE-5425 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5425 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.0.4-alpha >Reporter: Ashwin Shankar >Assignee: Robert Parker > Fix For: 3.0.0, 2.1.1-beta > > Attachments: MAPREDUCE-5425-2.patch, MAPREDUCE-5425-3.patch, > MAPREDUCE-5425.patch > > > We get the following exception when we run the unit tests of > TestJobHistoryServer with jdk 7: > Caused by: java.net.BindException: Problem binding to [0.0.0.0:10033] > java.net.BindException: Address already in use; For more details see: > http://wiki.apache.org/hadoop/BindException > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:719) > at org.apache.hadoop.ipc.Server.bind(Server.java:423) > at org.apache.hadoop.ipc.Server$Listener.(Server.java:535) > at org.apache.hadoop.ipc.Server.(Server.java:2202) > at org.apache.hadoop.ipc.RPC$Server.(RPC.java:901) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:505) > at > org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:480) > at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:746) > at > org.apache.hadoop.mapreduce.v2.hs.server.HSAdminServer.serviceInit(HSAdminServer.java:100) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > This is happening because testMainMethod starts the history server and doesnt > stop it. This worked in jdk 6 because tests executed sequentially and this > test was last one and didnt affect other tests,but in jdk 7 it fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
