[jira] [Commented] (MAPREDUCE-5329) APPLICATION_INIT is never sent to AuxServices other than the builtin ShuffleHandler
[ https://issues.apache.org/jira/browse/MAPREDUCE-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795015#comment-13795015 ] Avner BenHanoch commented on MAPREDUCE-5329: Thanks for all Siddharth, It was a pleasure working with you. I appreciate all your help and the prompt commits to branch-2 and branch-2.2. > APPLICATION_INIT is never sent to AuxServices other than the builtin > ShuffleHandler > --- > > Key: MAPREDUCE-5329 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5329 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.1.0-beta, 2.0.6-alpha >Reporter: Avner BenHanoch >Assignee: Avner BenHanoch > Fix For: trunk, 2.3.0, 2.2.1 > > Attachments: MAPREDUCE-5329.patch, MAPREDUCE-5329.patch, > MAPREDUCE-5329.patch > > > APPLICATION_INIT is never sent to AuxServices other than the built-in > ShuffleHandler. This means that 3rd party ShuffleProvider(s) will not be > able to function, because APPLICATION_INIT enables the AuxiliaryService to > map jobId->userId. This is needed for properly finding the MOFs of a job per > reducers' requests. > NOTE: The built-in ShuffleHandler does get APPLICATION_INIT events due to > hard-coded expression in hadoop code. The current TaskAttemptImpl.java code > explicitly call: serviceData.put (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID, > ...) and ignores any additional AuxiliaryService. As a result, only the > built-in ShuffleHandler will get APPLICATION_INIT events. Any 3rd party > AuxillaryService will never get APPLICATION_INIT events. > I think a solution can be in one of two ways: > 1. Change TaskAttemptImpl.java to loop on all Auxiliary Services and register > each of them, by calling serviceData.put (…) in loop. > 2. Change AuxServices.java similar to the fix in: MAPREDUCE-2668 > "APPLICATION_STOP is never sent to AuxServices". This means that in case the > 'handle' method gets APPLICATION_INIT event it will demultiplex it to all Aux > Services regardless of the value in event.getServiceID(). > I prefer the 2nd solution. I am welcoming any ideas. I can provide the > needed patch for any option that people like. > See [Pluggable Shuffle in Hadoop > documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html] -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5329) APPLICATION_INIT is never sent to AuxServices other than the builtin ShuffleHandler
[ https://issues.apache.org/jira/browse/MAPREDUCE-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795017#comment-13795017 ] Avner BenHanoch commented on MAPREDUCE-5329: Please let me know if you want me to open an issue for updating the [Pluggable Shuffle and Pluggable Sort documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html] with the new config - *mapreduce.job.shuffle.provider.services*, and if you want me to handle that task (I may need some explanation on the documentation creation). > APPLICATION_INIT is never sent to AuxServices other than the builtin > ShuffleHandler > --- > > Key: MAPREDUCE-5329 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5329 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.1.0-beta, 2.0.6-alpha >Reporter: Avner BenHanoch >Assignee: Avner BenHanoch > Fix For: trunk, 2.3.0, 2.2.1 > > Attachments: MAPREDUCE-5329.patch, MAPREDUCE-5329.patch, > MAPREDUCE-5329.patch > > > APPLICATION_INIT is never sent to AuxServices other than the built-in > ShuffleHandler. This means that 3rd party ShuffleProvider(s) will not be > able to function, because APPLICATION_INIT enables the AuxiliaryService to > map jobId->userId. This is needed for properly finding the MOFs of a job per > reducers' requests. > NOTE: The built-in ShuffleHandler does get APPLICATION_INIT events due to > hard-coded expression in hadoop code. The current TaskAttemptImpl.java code > explicitly call: serviceData.put (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID, > ...) and ignores any additional AuxiliaryService. As a result, only the > built-in ShuffleHandler will get APPLICATION_INIT events. Any 3rd party > AuxillaryService will never get APPLICATION_INIT events. > I think a solution can be in one of two ways: > 1. Change TaskAttemptImpl.java to loop on all Auxiliary Services and register > each of them, by calling serviceData.put (…) in loop. > 2. Change AuxServices.java similar to the fix in: MAPREDUCE-2668 > "APPLICATION_STOP is never sent to AuxServices". This means that in case the > 'handle' method gets APPLICATION_INIT event it will demultiplex it to all Aux > Services regardless of the value in event.getServiceID(). > I prefer the 2nd solution. I am welcoming any ideas. I can provide the > needed patch for any option that people like. > See [Pluggable Shuffle in Hadoop > documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html] -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5546) mapred.cmd on Windows set HADOOP_OPTS incorrectly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795072#comment-13795072 ] Hudson commented on MAPREDUCE-5546: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #363 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/363/]) MAPREDUCE-5546. mapred.cmd on Windows set HADOOP_OPTS incorrectly. Contributed by Chuan Liu. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1532016) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred.cmd > mapred.cmd on Windows set HADOOP_OPTS incorrectly > - > > Key: MAPREDUCE-5546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5546 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.0 >Reporter: Chuan Liu >Assignee: Chuan Liu > Fix For: 3.0.0, 2.2.1 > > Attachments: MAPREDUCE-5546-trunk.patch > > > The mapred command on Windows does not set HADOOP_OPTS correctly. As a > result, some options and settings will miss in the final command, and this > will lead to some desired behavior missing. One example is the logging file > setting will miss, i.e. even if one set HADOOP_ROOT_LOGGER to DRFA, there is > no history server log at HADOOP_LOGFILE. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5518) Fix typo "can't read paritions file"
[
https://issues.apache.org/jira/browse/MAPREDUCE-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795068#comment-13795068
]
Hudson commented on MAPREDUCE-5518:
---
SUCCESS: Integrated in Hadoop-Yarn-trunk #363 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/363/])
MAPREDUCE-5518. Fixed typo "can't read paritions file". Contributed by Albert
Chu. (devaraj:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1532208)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSort.java
> Fix typo "can't read paritions file"
>
>
> Key: MAPREDUCE-5518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5518
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: examples
>Affects Versions: 3.0.0
>Reporter: Albert Chu
>Assignee: Albert Chu
>Priority: Trivial
> Fix For: 2.2.1
>
> Attachments: MAPREDUCE-5518.patch
>
>
> Noticed a spelling error when I saw this error message
> {noformat}
> 13/09/19 13:25:08 INFO mapreduce.Job: Task Id :
> attempt_1379622083112_0002_m_000114_0, Status : FAILED
> Error: java.lang.IllegalArgumentException: can't read paritions file
> {noformat}
> "paritions" should be "partitions"
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5546) mapred.cmd on Windows set HADOOP_OPTS incorrectly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795176#comment-13795176 ] Hudson commented on MAPREDUCE-5546: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #1553 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1553/]) MAPREDUCE-5546. mapred.cmd on Windows set HADOOP_OPTS incorrectly. Contributed by Chuan Liu. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1532016) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred.cmd > mapred.cmd on Windows set HADOOP_OPTS incorrectly > - > > Key: MAPREDUCE-5546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5546 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.0 >Reporter: Chuan Liu >Assignee: Chuan Liu > Fix For: 3.0.0, 2.2.1 > > Attachments: MAPREDUCE-5546-trunk.patch > > > The mapred command on Windows does not set HADOOP_OPTS correctly. As a > result, some options and settings will miss in the final command, and this > will lead to some desired behavior missing. One example is the logging file > setting will miss, i.e. even if one set HADOOP_ROOT_LOGGER to DRFA, there is > no history server log at HADOOP_LOGFILE. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5518) Fix typo "can't read paritions file"
[
https://issues.apache.org/jira/browse/MAPREDUCE-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795172#comment-13795172
]
Hudson commented on MAPREDUCE-5518:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk #1553 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1553/])
MAPREDUCE-5518. Fixed typo "can't read paritions file". Contributed by Albert
Chu. (devaraj:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1532208)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSort.java
> Fix typo "can't read paritions file"
>
>
> Key: MAPREDUCE-5518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5518
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: examples
>Affects Versions: 3.0.0
>Reporter: Albert Chu
>Assignee: Albert Chu
>Priority: Trivial
> Fix For: 2.2.1
>
> Attachments: MAPREDUCE-5518.patch
>
>
> Noticed a spelling error when I saw this error message
> {noformat}
> 13/09/19 13:25:08 INFO mapreduce.Job: Task Id :
> attempt_1379622083112_0002_m_000114_0, Status : FAILED
> Error: java.lang.IllegalArgumentException: can't read paritions file
> {noformat}
> "paritions" should be "partitions"
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5546) mapred.cmd on Windows set HADOOP_OPTS incorrectly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795225#comment-13795225 ] Hudson commented on MAPREDUCE-5546: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1579 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1579/]) MAPREDUCE-5546. mapred.cmd on Windows set HADOOP_OPTS incorrectly. Contributed by Chuan Liu. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1532016) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred.cmd > mapred.cmd on Windows set HADOOP_OPTS incorrectly > - > > Key: MAPREDUCE-5546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5546 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.0 >Reporter: Chuan Liu >Assignee: Chuan Liu > Fix For: 3.0.0, 2.2.1 > > Attachments: MAPREDUCE-5546-trunk.patch > > > The mapred command on Windows does not set HADOOP_OPTS correctly. As a > result, some options and settings will miss in the final command, and this > will lead to some desired behavior missing. One example is the logging file > setting will miss, i.e. even if one set HADOOP_ROOT_LOGGER to DRFA, there is > no history server log at HADOOP_LOGFILE. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5518) Fix typo "can't read paritions file"
[
https://issues.apache.org/jira/browse/MAPREDUCE-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795221#comment-13795221
]
Hudson commented on MAPREDUCE-5518:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1579 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1579/])
MAPREDUCE-5518. Fixed typo "can't read paritions file". Contributed by Albert
Chu. (devaraj:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1532208)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSort.java
> Fix typo "can't read paritions file"
>
>
> Key: MAPREDUCE-5518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5518
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: examples
>Affects Versions: 3.0.0
>Reporter: Albert Chu
>Assignee: Albert Chu
>Priority: Trivial
> Fix For: 2.2.1
>
> Attachments: MAPREDUCE-5518.patch
>
>
> Noticed a spelling error when I saw this error message
> {noformat}
> 13/09/19 13:25:08 INFO mapreduce.Job: Task Id :
> attempt_1379622083112_0002_m_000114_0, Status : FAILED
> Error: java.lang.IllegalArgumentException: can't read paritions file
> {noformat}
> "paritions" should be "partitions"
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795228#comment-13795228 ] Jason Lowe commented on MAPREDUCE-5583: --- Not in a general way. Different jobs can have different limits, and queue/user limits are too granular a tool to handle that appropriately. We're either creating a ton of queues for the various scenarios which is a huge pain from the usability and admin point of view, or we're artificially constricting jobs that don't have a need for those limits that happen to run in a queue that was shrunk for other jobs. For example, take the case where we need to increase the memory for map tasks. If we took the use-the-queue-as-the-limit route we now have less tasks running simultaneously than we did before which is undesirable, and the queue needs to be changed each time the job grows or shrinks. If we could limit it per-job in the AM it would have run with the appropriate parallelism, assuming the original queue had the capacity. Having per-job limits allows the user to tune their jobs in a much more intuitive way and without requiring admins to assist in that tuning. > Ability to limit running map and reduce tasks > - > > Key: MAPREDUCE-5583 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.9, 2.1.1-beta >Reporter: Jason Lowe > > It would be nice if users could specify a limit to the number of map or > reduce tasks that are running simultaneously. Occasionally users are > performing operations in tasks that can lead to DDoS scenarios if too many > tasks run simultaneously (e.g.: accessing a database, web service, etc.). > Having the ability to throttle the number of tasks simultaneously running > would provide users a way to mitigate issues with too many tasks on a large > cluster attempting to access a serivce at any one time. > This is similar to the functionality requested by MAPREDUCE-224 and > implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5584) ShuffleHandler becomes unresponsive during gridmix runs and can leak file descriptors
Jason Lowe created MAPREDUCE-5584: - Summary: ShuffleHandler becomes unresponsive during gridmix runs and can leak file descriptors Key: MAPREDUCE-5584 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5584 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.3.0 Reporter: Jason Lowe While running gridmix on 2.3 we noticed that jobs are running much slower than normal. We tracked this down to reducers having difficulties shuffling data from maps. Details to follow. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5584) ShuffleHandler becomes unresponsive during gridmix runs and can leak file descriptors
[
https://issues.apache.org/jira/browse/MAPREDUCE-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-5584:
--
Priority: Blocker (was: Major)
The reducers were timing out attempting to contact certain nodes for their map
inputs. Simple GET probes to the shuffle port on these nodes showed that they
were indeed totally unresponsive. Examination of the nodes showed that they
had leaked a significant number of file descriptors with sockets in the
CLOSE_WAIT state.
The jstacks of the NodeManager processes on these nodes also showed that all of
the Netty handlers were stuck somewhere in
LocalDirAllocator.getLocalPathToRead. They were either stuck on the
synchronized lock or waiting for the results of fs.exists() to return which now
forks and execs {{stat}} since HADOOP-9652.
> ShuffleHandler becomes unresponsive during gridmix runs and can leak file
> descriptors
> -
>
> Key: MAPREDUCE-5584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5584
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Jason Lowe
>Priority: Blocker
>
> While running gridmix on 2.3 we noticed that jobs are running much slower
> than normal. We tracked this down to reducers having difficulties shuffling
> data from maps. Details to follow.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5584) ShuffleHandler becomes unresponsive during gridmix runs and can leak file descriptors
[ https://issues.apache.org/jira/browse/MAPREDUCE-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5584: -- Target Version/s: 2.3.0 > ShuffleHandler becomes unresponsive during gridmix runs and can leak file > descriptors > - > > Key: MAPREDUCE-5584 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5584 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Priority: Blocker > > While running gridmix on 2.3 we noticed that jobs are running much slower > than normal. We tracked this down to reducers having difficulties shuffling > data from maps. Details to follow. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5584) ShuffleHandler becomes unresponsive during gridmix runs and can leak file descriptors
[
https://issues.apache.org/jira/browse/MAPREDUCE-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795315#comment-13795315
]
Jason Lowe commented on MAPREDUCE-5584:
---
The CLOSE_WAIT issue is a problem with the ShuffleHandler not closing
connections under some error conditions. A sample backtrace:
{noformat}
2013-10-03 21:15:07,307 [New I/O worker #31] DEBUG
org.apache.hadoop.mapred.ShuffleHandler: Ignoring closed channel error
java.nio.channels.ClosedChannelException
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.discard(ChunkedWriteHandler.java:168)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:192)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:121)
at org.jboss.netty.channel.Channels.write(Channels.java:704)
at org.jboss.netty.channel.Channels.write(Channels.java:671)
at
org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
at
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.sendMapOutput(ShuffleHandler.java:612)
at
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:503)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
at
org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:485)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
{noformat}
> ShuffleHandler becomes unresponsive during gridmix runs and can leak file
> descriptors
> -
>
> Key: MAPREDUCE-5584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5584
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Jason Lowe
>Priority: Blocker
>
> While running gridmix on 2.3 we noticed that jobs are running much slower
> than normal. We tracked this down to reducers having difficulties shuffling
> data from maps. Details to follow.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795399#comment-13795399 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5583: Yes, +1 for this. This is how HADOOP-5170 is supposed to have been addressed, but couldn't in MRv1. > Ability to limit running map and reduce tasks > - > > Key: MAPREDUCE-5583 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.9, 2.1.1-beta >Reporter: Jason Lowe > > It would be nice if users could specify a limit to the number of map or > reduce tasks that are running simultaneously. Occasionally users are > performing operations in tasks that can lead to DDoS scenarios if too many > tasks run simultaneously (e.g.: accessing a database, web service, etc.). > Having the ability to throttle the number of tasks simultaneously running > would provide users a way to mitigate issues with too many tasks on a large > cluster attempting to access a serivce at any one time. > This is similar to the functionality requested by MAPREDUCE-224 and > implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795429#comment-13795429 ] Arun C Murthy commented on MAPREDUCE-5583: -- Looks like we are rehashing the discussion on HADOOP-5170. > Ability to limit running map and reduce tasks > - > > Key: MAPREDUCE-5583 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.9, 2.1.1-beta >Reporter: Jason Lowe > > It would be nice if users could specify a limit to the number of map or > reduce tasks that are running simultaneously. Occasionally users are > performing operations in tasks that can lead to DDoS scenarios if too many > tasks run simultaneously (e.g.: accessing a database, web service, etc.). > Having the ability to throttle the number of tasks simultaneously running > would provide users a way to mitigate issues with too many tasks on a large > cluster attempting to access a serivce at any one time. > This is similar to the functionality requested by MAPREDUCE-224 and > implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795427#comment-13795427 ] Arun C Murthy commented on MAPREDUCE-5583: -- I'm very concerned about allowing this sort of control to the end user without checks/balances. If every one does the same I want max of n tasks, all resources in the cluster can deadlock. This is one of the main reasons we haven't supported this. I'm still against it. > Ability to limit running map and reduce tasks > - > > Key: MAPREDUCE-5583 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.9, 2.1.1-beta >Reporter: Jason Lowe > > It would be nice if users could specify a limit to the number of map or > reduce tasks that are running simultaneously. Occasionally users are > performing operations in tasks that can lead to DDoS scenarios if too many > tasks run simultaneously (e.g.: accessing a database, web service, etc.). > Having the ability to throttle the number of tasks simultaneously running > would provide users a way to mitigate issues with too many tasks on a large > cluster attempting to access a serivce at any one time. > This is similar to the functionality requested by MAPREDUCE-224 and > implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795433#comment-13795433 ] Sandy Ryza commented on MAPREDUCE-5583: --- [~acmurthy], how would this cause deadlock? > Ability to limit running map and reduce tasks > - > > Key: MAPREDUCE-5583 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.9, 2.1.1-beta >Reporter: Jason Lowe > > It would be nice if users could specify a limit to the number of map or > reduce tasks that are running simultaneously. Occasionally users are > performing operations in tasks that can lead to DDoS scenarios if too many > tasks run simultaneously (e.g.: accessing a database, web service, etc.). > Having the ability to throttle the number of tasks simultaneously running > would provide users a way to mitigate issues with too many tasks on a large > cluster attempting to access a serivce at any one time. > This is similar to the functionality requested by MAPREDUCE-224 and > implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795443#comment-13795443 ] Arun C Murthy commented on MAPREDUCE-5583: -- Cluster with 100,000 containers, 1,000 jobs, each with 10 tasks, and specifies that they can only run 5 tasks. So, you are now only using 5% of the cluster and no one makes progress leading to very poor utilization and peanut-buttering effect. Admittedly it's a contrived example and yes, I agree a user can hack his own AM to do this - but let's not make this trivial for normal users. This leads to all sorts of bad side-effects by supporting it out of the box. Some form of admin control (e.g. queue with a max-cap) for a small number of use-cases where you *actually* need this feature is much safer. > Ability to limit running map and reduce tasks > - > > Key: MAPREDUCE-5583 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.9, 2.1.1-beta >Reporter: Jason Lowe > > It would be nice if users could specify a limit to the number of map or > reduce tasks that are running simultaneously. Occasionally users are > performing operations in tasks that can lead to DDoS scenarios if too many > tasks run simultaneously (e.g.: accessing a database, web service, etc.). > Having the ability to throttle the number of tasks simultaneously running > would provide users a way to mitigate issues with too many tasks on a large > cluster attempting to access a serivce at any one time. > This is similar to the functionality requested by MAPREDUCE-224 and > implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5577) Allow querying the JobHistoryServer by job arrival time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795470#comment-13795470 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5577: bq. However, jobs don't necessarily arrive in order of their finish time, a client who wants to stay on top of all completed jobs needs to query large time intervals to make sure they're not missing anything. Exposing functionality to allow querying by the time a job lands at the JobHistoryServer would allow clients to set the start of their query interval to the time of their last query. Trying to understand this. Why can't clients simply look at job-finish time? - Jobs that finished in the last one hour/or last one day. Why won't that work? > Allow querying the JobHistoryServer by job arrival time > --- > > Key: MAPREDUCE-5577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5577 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5577.patch > > > The JobHistoryServer REST APIs currently allow querying by job submit time > and finish time. However, jobs don't necessarily arrive in order of their > finish time, meaning that a client who wants to stay on top of all completed > jobs needs to query large time intervals to make sure they're not missing > anything. Exposing functionality to allow querying by the time a job lands > at the JobHistoryServer would allow clients to set the start of their query > interval to the time of their last query. > The arrival time of a job would be defined as the time that it lands in the > done directory and can be picked up using the last modified date on history > files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5585) TestCopyCommitter#testNoCommitAction Fails on JDK7
Jonathan Eagles created MAPREDUCE-5585: -- Summary: TestCopyCommitter#testNoCommitAction Fails on JDK7 Key: MAPREDUCE-5585 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5585 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.9, 3.0.0, 2.3.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles TestCopyCommitter#testNoCommitAction fails on JDK7 when run after testAtomicCommitMissingFinal or testAtomicCommitExistingFinal. Config settings are from atomic tests are being accidentally used for testNoCommitAction. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5585) TestCopyCommitter#testNoCommitAction Fails on JDK7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5585: --- Status: Patch Available (was: Open) > TestCopyCommitter#testNoCommitAction Fails on JDK7 > -- > > Key: MAPREDUCE-5585 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5585 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.9, 3.0.0, 2.3.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-5585.patch > > > TestCopyCommitter#testNoCommitAction fails on JDK7 when run after > testAtomicCommitMissingFinal or testAtomicCommitExistingFinal. Config > settings are from atomic tests are being accidentally used for > testNoCommitAction. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5585) TestCopyCommitter#testNoCommitAction Fails on JDK7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5585: --- Attachment: MAPREDUCE-5585.patch > TestCopyCommitter#testNoCommitAction Fails on JDK7 > -- > > Key: MAPREDUCE-5585 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5585 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.0.0, 0.23.9, 2.3.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-5585.patch > > > TestCopyCommitter#testNoCommitAction fails on JDK7 when run after > testAtomicCommitMissingFinal or testAtomicCommitExistingFinal. Config > settings are from atomic tests are being accidentally used for > testNoCommitAction. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5585) TestCopyCommitter#testNoCommitAction Fails on JDK7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795557#comment-13795557
]
Hadoop QA commented on MAPREDUCE-5585:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608556/MAPREDUCE-5585.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-tools/hadoop-distcp:
org.apache.hadoop.tools.mapred.TestCopyMapper
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4119//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4119//console
This message is automatically generated.
> TestCopyCommitter#testNoCommitAction Fails on JDK7
> --
>
> Key: MAPREDUCE-5585
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5585
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0, 0.23.9, 2.3.0
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: MAPREDUCE-5585.patch
>
>
> TestCopyCommitter#testNoCommitAction fails on JDK7 when run after
> testAtomicCommitMissingFinal or testAtomicCommitExistingFinal. Config
> settings are from atomic tests are being accidentally used for
> testNoCommitAction.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5585) TestCopyCommitter#testNoCommitAction Fails on JDK7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5585: --- Attachment: MAPREDUCE-5585.patch > TestCopyCommitter#testNoCommitAction Fails on JDK7 > -- > > Key: MAPREDUCE-5585 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5585 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.0.0, 0.23.9, 2.3.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-5585.patch, MAPREDUCE-5585.patch > > > TestCopyCommitter#testNoCommitAction fails on JDK7 when run after > testAtomicCommitMissingFinal or testAtomicCommitExistingFinal. Config > settings are from atomic tests are being accidentally used for > testNoCommitAction. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5585) TestCopyCommitter#testNoCommitAction Fails on JDK7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795710#comment-13795710
]
Hadoop QA commented on MAPREDUCE-5585:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608579/MAPREDUCE-5585.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-tools/hadoop-distcp:
org.apache.hadoop.tools.mapred.TestCopyMapper
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4120//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4120//console
This message is automatically generated.
> TestCopyCommitter#testNoCommitAction Fails on JDK7
> --
>
> Key: MAPREDUCE-5585
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5585
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0, 0.23.9, 2.3.0
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: MAPREDUCE-5585.patch, MAPREDUCE-5585.patch
>
>
> TestCopyCommitter#testNoCommitAction fails on JDK7 when run after
> testAtomicCommitMissingFinal or testAtomicCommitExistingFinal. Config
> settings are from atomic tests are being accidentally used for
> testNoCommitAction.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-4282) Convert Forrest docs to APT
[ https://issues.apache.org/jira/browse/MAPREDUCE-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796222#comment-13796222 ] Akira AJISAKA commented on MAPREDUCE-4282: -- Hi, I want to try converting incrementally. Please tell me what document is good to start? > Convert Forrest docs to APT > --- > > Key: MAPREDUCE-4282 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4282 > Project: Hadoop Map/Reduce > Issue Type: Task >Affects Versions: 2.0.0-alpha >Reporter: Eli Collins >Assignee: Eli Reisman > Labels: newbie > Attachments: bad-forrest2apt-attempt1.tar, MAPREDUCE-4282-1.patch, > MAPREDUCE-4282-2.patch > > > MR side of HADOOP-8427. Not all of the old forrest docs in > src/documentation/content/xdocs have been converted over to APT yet, let's do > that and remove the forrest docs. > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5577) Allow querying the JobHistoryServer by job arrival time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796228#comment-13796228 ] Sandy Ryza commented on MAPREDUCE-5577: --- The goal is to make things easier for clients that are trying to track all jobs that go through the JHS. Without this, they must always query the largest interval that a job could conceivably come in after its finish time (which could be minutes with things like GC pauses). This means a lot of redundant job data transferred and more work for the client, as it must keep track of all the jobs it's received in that time interval to filter out what's new. > Allow querying the JobHistoryServer by job arrival time > --- > > Key: MAPREDUCE-5577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5577 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5577.patch > > > The JobHistoryServer REST APIs currently allow querying by job submit time > and finish time. However, jobs don't necessarily arrive in order of their > finish time, meaning that a client who wants to stay on top of all completed > jobs needs to query large time intervals to make sure they're not missing > anything. Exposing functionality to allow querying by the time a job lands > at the JobHistoryServer would allow clients to set the start of their query > interval to the time of their last query. > The arrival time of a job would be defined as the time that it lands in the > done directory and can be picked up using the last modified date on history > files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-4282) Convert Forrest docs to APT
[ https://issues.apache.org/jira/browse/MAPREDUCE-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796232#comment-13796232 ] Sandy Ryza commented on MAPREDUCE-4282: --- The MapReduce Tutorial might be a good place to start? > Convert Forrest docs to APT > --- > > Key: MAPREDUCE-4282 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4282 > Project: Hadoop Map/Reduce > Issue Type: Task >Affects Versions: 2.0.0-alpha >Reporter: Eli Collins >Assignee: Eli Reisman > Labels: newbie > Attachments: bad-forrest2apt-attempt1.tar, MAPREDUCE-4282-1.patch, > MAPREDUCE-4282-2.patch > > > MR side of HADOOP-8427. Not all of the old forrest docs in > src/documentation/content/xdocs have been converted over to APT yet, let's do > that and remove the forrest docs. > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-4282) Convert Forrest docs to APT
[ https://issues.apache.org/jira/browse/MAPREDUCE-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796234#comment-13796234 ] Akira AJISAKA commented on MAPREDUCE-4282: -- [~sandyr], thanks! I'll try it. > Convert Forrest docs to APT > --- > > Key: MAPREDUCE-4282 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4282 > Project: Hadoop Map/Reduce > Issue Type: Task >Affects Versions: 2.0.0-alpha >Reporter: Eli Collins >Assignee: Eli Reisman > Labels: newbie > Attachments: bad-forrest2apt-attempt1.tar, MAPREDUCE-4282-1.patch, > MAPREDUCE-4282-2.patch > > > MR side of HADOOP-8427. Not all of the old forrest docs in > src/documentation/content/xdocs have been converted over to APT yet, let's do > that and remove the forrest docs. > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5585) TestCopyCommitter#testNoCommitAction Fails on JDK7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5585: --- Status: Open (was: Patch Available) Looking into the Copy Mapper test failure > TestCopyCommitter#testNoCommitAction Fails on JDK7 > -- > > Key: MAPREDUCE-5585 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5585 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.9, 3.0.0, 2.3.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-5585.patch, MAPREDUCE-5585.patch > > > TestCopyCommitter#testNoCommitAction fails on JDK7 when run after > testAtomicCommitMissingFinal or testAtomicCommitExistingFinal. Config > settings are from atomic tests are being accidentally used for > testNoCommitAction. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5586) TestCopyMapper#testCopyFailOnBlockSizeDifference fails when run from hadoop-tools/hadoop-distcp directory
Jonathan Eagles created MAPREDUCE-5586: -- Summary: TestCopyMapper#testCopyFailOnBlockSizeDifference fails when run from hadoop-tools/hadoop-distcp directory Key: MAPREDUCE-5586 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5586 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5585) TestCopyCommitter#testNoCommitAction Fails on JDK7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796389#comment-13796389 ] Jonathan Eagles commented on MAPREDUCE-5585: TestCopyMapper test failure is going to be covered by MAPREDUCE-5586 > TestCopyCommitter#testNoCommitAction Fails on JDK7 > -- > > Key: MAPREDUCE-5585 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5585 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.0.0, 0.23.9, 2.3.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-5585.patch, MAPREDUCE-5585.patch > > > TestCopyCommitter#testNoCommitAction fails on JDK7 when run after > testAtomicCommitMissingFinal or testAtomicCommitExistingFinal. Config > settings are from atomic tests are being accidentally used for > testNoCommitAction. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5577) Allow querying the JobHistoryServer by job arrival time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Zeyliger updated MAPREDUCE-5577: --- Description: The JobHistoryServer REST APIs currently allow querying by job submit time and finish time. However, jobs don't necessarily arrive in order of their finish time, meaning that a client who wants to stay on top of all completed jobs needs to query large time intervals to make sure they're not missing anything. Exposing functionality to allow querying by the time a job lands at the JobHistoryServer would allow clients to set the start of their query interval to the time of their last query. The arrival time of a job would be defined as the time that it lands in the done directory and can be picked up using the last modified date on history files. was: The JobHistoryServer REST APIs currently allow querying by job submit time and finish time. However, jobs don't necessarily arrive in order of their finish time, meaning that a client who wants to stay on top of all completed jobs needs to query large time intervals to make sure they're not missing anything. Exposing functionality to allow querying by the time a job lands at the JobHistoryServer would allow clients to set the start of their query interval to the time of their last query. The arrival time of a job would be defined as the time that it lands in the done directory and can be picked up using the last modified date on history files. > Allow querying the JobHistoryServer by job arrival time > --- > > Key: MAPREDUCE-5577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5577 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5577.patch > > > The JobHistoryServer REST APIs currently allow querying by job submit time > and finish time. However, jobs don't necessarily arrive in order of their > finish time, meaning that a client who wants to stay on top of all completed > jobs needs to query large time intervals to make sure they're not missing > anything. Exposing functionality to allow querying by the time a job lands > at the JobHistoryServer would allow clients to set the start of their query > interval to the time of their last query. > The arrival time of a job would be defined as the time that it lands in the > done directory and can be picked up using the last modified date on history > files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5541) Improved algorithm for whether need speculative task
[ https://issues.apache.org/jira/browse/MAPREDUCE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796429#comment-13796429 ] Benoy Antony commented on MAPREDUCE-5541: - +1. Looks good. Solves a problem seen in our clusters. Please review and commit.' John, Is there a trunk version for this improvement ? > Improved algorithm for whether need speculative task > > > Key: MAPREDUCE-5541 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5541 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 1.2.1 >Reporter: zhaoyunjiong >Assignee: zhaoyunjiong > Fix For: 1.2.2 > > Attachments: MAPREDUCE-5541-branch-1.2.patch, > MAPREDUCE-5541-branch-1.2.patch > > > Most of time, tasks won't start running at same time. > In this case hasSpeculativeTask in TaskInProgress not working very well. > Some times, some tasks just start running, and scheduler already decide it > need speculative task to run. > And this waste a lot of resource. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5541) Improved algorithm for whether need speculative task
[ https://issues.apache.org/jira/browse/MAPREDUCE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796462#comment-13796462 ] zhaoyunjiong commented on MAPREDUCE-5541: - Trunk version is very different, and already rewrite. > Improved algorithm for whether need speculative task > > > Key: MAPREDUCE-5541 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5541 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 1.2.1 >Reporter: zhaoyunjiong >Assignee: zhaoyunjiong > Fix For: 1.2.2 > > Attachments: MAPREDUCE-5541-branch-1.2.patch, > MAPREDUCE-5541-branch-1.2.patch > > > Most of time, tasks won't start running at same time. > In this case hasSpeculativeTask in TaskInProgress not working very well. > Some times, some tasks just start running, and scheduler already decide it > need speculative task to run. > And this waste a lot of resource. -- This message was sent by Atlassian JIRA (v6.1#6144)
